Jensen's Inequality | Varsity Tutors

SECTION 1

Historical Context & Motivation

The relationship between convex functions and averages has fascinated mathematicians for centuries, but it was the Danish mathematician Johan Ludwig William Valdemar Jensen who, in 1906, crystallized this relationship into the elegant inequality that bears his name. Jensen was not a professional academic — he spent most of his career as an engineer at the Copenhagen Telephone Company — yet his mathematical contributions proved foundational. His inequality generalized earlier observations about the arithmetic mean–geometric mean inequality and Hölder's inequality into a single, sweeping result about convex functions. The inequality answered a deceptively simple question: if you apply a curved function to the average of some numbers, how does the result compare to the average of the function applied to each number individually? This question turns out to be central to probability theory, optimization, and information theory alike.

1839

AM–GM Inequality Formalized

Augustin-Louis Cauchy provided a rigorous induction proof that the arithmetic mean of positive reals is at least as large as their geometric mean, a special case of what Jensen would later generalize.

1889

Hölder's Inequality

Otto Hölder published his celebrated inequality for Lᵖ spaces, providing another instance of convexity-based bounds that would later be subsumed by Jensen's framework.

1906

Jensen's Original Paper

Jensen published 'Sur les fonctions convexes et les inégalités entre les valeurs moyennes' in Acta Mathematica, introducing the general inequality for convex functions and finite sums with weights.

1929

Measure-Theoretic Extension

With the maturation of Lebesgue integration and Kolmogorov's axiomatization of probability, Jensen's inequality was extended to general probability measures, becoming a cornerstone of modern probability theory.

1948

Information-Theoretic Applications

Claude Shannon's foundational work on information theory used Jensen's inequality to establish key properties of entropy, including the non-negativity of Kullback–Leibler divergence.

The central question Jensen's inequality addresses is this: given a random variable X and a convex function φ, what is the relationship between φ(E[X]) and E[φ(X)]? The answer — that convexity forces the function of the mean to lie below the mean of the function — has profound consequences across mathematics and its applications.

SECTION 2

Core Principles & Definitions

Before stating Jensen's inequality precisely, we must establish the key definitions that underpin it. The inequality rests on the interplay between convexity of a function and the expectation operator from probability theory. Convexity captures the idea that the chord connecting any two points on a function's graph lies on or above the graph itself. When this geometric condition is satisfied, taking expectations commutes with the function in a specific, predictable direction.

Convex Function

A function φ : I → ℝ is convex on an interval I if for all x, y ∈ I and all λ ∈ [0, 1], we have φ(λx + (1 − λ)y) ≤ λφ(x) + (1 − λ)φ(y). Intuitively, the function curves upward — secant lines always lie above the graph.

Concave Function

A function φ is concave if −φ is convex. Equivalently, φ(λx + (1 − λ)y) ≥ λφ(x) + (1 − λ)φ(y). Jensen's inequality reverses direction for concave functions: φ(E[X]) ≥ E[φ(X)].

Supporting Hyperplane

At every point in the domain of a convex function, there exists an affine function (a tangent line) that lies entirely at or below the function. This geometric property is the engine behind the standard proof of Jensen's inequality.

Integrability Condition

Jensen's inequality requires E[|X|] < ∞ and E[|φ(X)|] < ∞ so that all expectations are well-defined. Without integrability, the inequality statement is vacuous or undefined.

Equality Condition

Equality φ(E[X]) = E[φ(X)] holds if and only if X is constant almost surely or φ is affine on the support of X. Understanding when equality holds is crucial for characterizing extremal distributions.

✦ KEY TAKEAWAY

Think of Jensen's inequality like heating a metal bar. If you measure the temperature at many points and then compute the average temperature, that average is a single number. Now imagine passing the average temperature through a nonlinear response function — say, the rate of thermal expansion, which accelerates at higher temperatures (a convex function). Jensen's inequality tells you that the expansion rate at the average temperature will underestimate the average expansion rate across all points. The curvature of the response function amplifies the contributions of hotter regions more than it dampens those of cooler ones, making the mean of the outputs exceed the output of the mean.

SECTION 3

Visual Explanation

The geometric intuition behind Jensen's inequality is best understood graphically. For a convex function φ, the graph bows downward (or equivalently, the epigraph is a convex set). When we evaluate φ at the mean of two points, we land on the curve itself, but the weighted average of the function values at those two points lands on the chord above the curve. The following diagram illustrates this core geometric insight for the convex function φ(x) = x² and two points x₁ and x₂ with equal weight.

For the convex function φ(x) = x², the green point φ(E[X]) sits on the curve at the mean, while the pink point E[φ(X)] sits on the chord above. The orange gap between them is always non-negative for convex φ.

The diagram above captures the essence of Jensen's inequality in the finite, discrete case. The convex curve bows downward toward the x-axis, so any chord connecting two points on the curve lies above the curve in between. When we compute the expected value E[X] = (x₁ + x₂)/2, the value of the function at that mean — the green point — is strictly below the midpoint of the chord — the pink point. This vertical gap is precisely the quantity E[φ(X)] − φ(E[X]) ≥ 0. The more curvature the function has (the larger φ″), and the more spread X has, the larger this gap becomes.

SECTION 4

Mathematical Framework

We now state Jensen's inequality in full generality and sketch its proof using the supporting hyperplane characterization of convex functions. Let (Ω, ℱ, P) be a probability space and let X : Ω → ℝ be an integrable random variable taking values in an interval I ⊆ ℝ. Let φ : I → ℝ be a convex function. Jensen's inequality asserts the following.

JENSEN'S INEQUALITY (GENERAL FORM)

φ(E[X]) ≤ E[φ(X)]

where X is an integrable random variable with E[X] ∈ I, φ is convex on I, and E[|φ(X)|] < ∞. If φ is concave, the inequality reverses: φ(E[X]) ≥ E[φ(X]).

Proof via Supporting Hyperplane

The proof is elegant and relies on a single key fact: for a convex function φ and any point μ in the interior of its domain, there exists a constant a (a subgradient) such that φ(x) ≥ φ(μ) + a(x − μ) for all x ∈ I. This is the supporting hyperplane property. If φ is differentiable at μ, then a = φ′(μ). Set μ = E[X]. Then for all ω ∈ Ω we have φ(X(ω)) ≥ φ(E[X]) + a(X(ω) − E[X]). Taking expectations of both sides and noting that E[X − E[X]] = 0, we obtain E[φ(X)] ≥ φ(E[X]) + a · 0 = φ(E[X]), completing the proof.

SUPPORTING HYPERPLANE (SUBGRADIENT CONDITION)

φ(x) ≥ φ(μ) + a(x − μ) for all x ∈ I

Here μ = E[X] and a ∈ ∂φ(μ) is a subgradient. If φ is differentiable, a = φ′(μ). This affine lower bound is the tangent line at μ.

CONDITIONAL JENSEN'S INEQUALITY

φ(E[X | 𝒢]) ≤ E[φ(X) | 𝒢] a.s.

This conditional version holds for any sub-σ-algebra 𝒢 ⊆ ℱ. It is used extensively in martingale theory: if (Mₙ) is a martingale and φ is convex, then (φ(Mₙ)) is a submartingale.

DISCRETE FINITE FORM

φ(Σᵢ λᵢ xᵢ) ≤ Σᵢ λᵢ φ(xᵢ), where λᵢ ≥ 0, Σᵢ λᵢ = 1

This is the finite weighted version. The weights λᵢ can be interpreted as probabilities for a discrete random variable taking values x₁, …, xₙ.

SECTION 5

Key Applications & Classification of Consequences

Jensen's inequality is remarkable not only for its elegance but for the sheer breadth of its consequences. Many classical inequalities in analysis, probability, and statistics can be derived as special cases by choosing a particular convex (or concave) function φ. The following diagram organizes the most important applications by domain, showing how a single inequality spawns results across disparate fields.

A taxonomy of consequences flowing from Jensen's inequality. Each branch corresponds to a particular choice of the convex function φ. The classical inequalities arise from simple φ choices, while information-theoretic and statistical applications require more subtle function selections.

Common instantiations of Jensen's inequality

Choose φ(x) =	Convex / Concave	Result	Named Inequality
`x²`	Convex	(E[X])² ≤ E[X²]	Variance non-negativity
`\|x\|ᵖ (p ≥ 1)`	Convex	\|E[X]\|ᵖ ≤ E[\|X\|ᵖ]	Lyapunov's inequality
`−ln(x)`	Convex	−ln(E[X]) ≤ E[−ln(X)]	AM–GM inequality
`eˣ`	Convex	exp(E[X]) ≤ E[exp(X)]	MGF bound / risk premium
`ln(x)`	Concave	ln(E[X]) ≥ E[ln(X)]	Entropy maximization

SECTION 6

Worked Example: Non-Negativity of KL Divergence

One of the most celebrated applications of Jensen's inequality is proving that the Kullback–Leibler divergence D_KL(P ‖ Q) is always non-negative, a result known as Gibbs' inequality. Let P and Q be two probability distributions on the same measurable space with P absolutely continuous with respect to Q.

Proving D_KL(P ‖ Q) ≥ 0 via Jensen's Inequality

Step 1 — Write the KL Divergence DefinitionThe Kullback–Leibler divergence from P to Q is defined as D_KL(P ‖ Q) = E_P[ln(dP/dQ)] = ∫ ln(p(x)/q(x)) p(x) dx, where p and q are the densities of P and Q respectively.

D_KL(P ‖ Q) = E_P[ln(p(X)/q(X))]

Step 2 — Rewrite Using −lnWe can write D_KL(P ‖ Q) = −E_P[ln(q(X)/p(X))]. Now define the random variable Y = q(X)/p(X). Notice that E_P[Y] = ∫ (q(x)/p(x)) p(x) dx = ∫ q(x) dx = 1.

D_KL = −E_P[ln(Y)] where E_P[Y] = 1

Step 3 — Apply Jensen's InequalitySince φ(x) = ln(x) is a concave function, Jensen's inequality for concave functions gives us ln(E_P[Y]) ≥ E_P[ln(Y)]. Substituting E_P[Y] = 1, we get ln(1) ≥ E_P[ln(Y)], i.e., 0 ≥ E_P[ln(Y)].

E_P[ln(q(X)/p(X))] ≤ 0

Step 4 — Conclude Non-NegativitySince D_KL(P ‖ Q) = −E_P[ln(q(X)/p(X))] and we showed E_P[ln(q(X)/p(X))] ≤ 0, it follows immediately that D_KL(P ‖ Q) ≥ 0. Moreover, by the equality condition of Jensen's inequality, D_KL(P ‖ Q) = 0 if and only if q(X)/p(X) = 1 almost surely under P, i.e., P = Q.

D_KL(P ‖ Q) ≥ 0, with equality iff P = Q

SECTION 7

Strengths, Limitations, and Common Pitfalls

Jensen's inequality is a remarkably versatile tool, but like any mathematical instrument, its power comes with boundary conditions. Understanding when the inequality provides a tight bound versus a loose one, and recognizing situations where it does not apply, is essential for effective use in proofs and applications.

Strengths and limitations of Jensen's inequality

Strengths	Limitations / Pitfalls
Applies to any convex or concave function — no smoothness or differentiability is required.	The bound can be very loose if the distribution has high variance or φ has mild curvature in the relevant region.
Works in infinite-dimensional settings (conditional expectations, Bochner integrals) with minimal modifications.	Requires integrability: both E[\|X\|] and E[\|φ(X)\|] must be finite. Fails for heavy-tailed distributions lacking moments.
Subsumes many classical inequalities (AM–GM, Cauchy–Schwarz, Lyapunov) as special cases.	Only provides a one-sided bound. For two-sided bounds, one needs complementary tools (e.g., the Hermite–Hadamard inequality).
The equality condition (X constant a.s. or φ affine on support) is fully characterized, enabling tight analysis.	Common error: applying the inequality in the wrong direction (forgetting to check convexity vs. concavity of φ).
The conditional version provides the key link between convexity and martingale theory.	Does not quantify the gap E[φ(X)] − φ(E[X]). Refined versions (e.g., using Var(X) and φ″) are needed for quantitative bounds.

✦ KEY TAKEAWAY

Jensen's inequality is like a Swiss Army knife in your mathematical toolkit: it handles an enormous range of tasks, but for any specific task, a purpose-built tool (Cauchy–Schwarz, Markov's inequality, etc.) may give you a sharper result. The true power of Jensen lies in its generality — whenever you encounter a convex function composed with an expectation, Jensen immediately gives you a direction of inequality, even when a more refined bound requires additional work.

SECTION 8

Connection to Advanced Theory

Jensen's inequality sits at a crossroads of modern probability and analysis. Its conditional version is the gateway to martingale theory, and quantitative refinements of the inequality lead to concentration inequalities that are indispensable in high-dimensional statistics and machine learning. The following table compares Jensen's basic inequality with some of its more advanced relatives.

Jensen's inequality in the landscape of advanced results

Concept	Relationship to Jensen	Additional Assumptions
Submartingale property	If (Mₙ) is a martingale and φ is convex, then φ(Mₙ) is a submartingale — a direct consequence of conditional Jensen.	E[\|φ(Mₙ)\|] < ∞ for all n.
Hermite–Hadamard inequality	Provides both an upper and lower bound for ∫φ over an interval, refining Jensen for uniform distributions.	X ~ Uniform[a, b]; φ convex.
Efron–Stein inequality	Bounds the variance of a function of independent r.v.'s using a leave-one-out approach. Uses Jensen internally in its proof.	Independence of components.
EM algorithm (ELBO)	The E-step of EM uses Jensen to construct a lower bound (ELBO) on the log-likelihood, exploiting concavity of log.	Latent variable model structure.
Chernoff bound	Applies Markov's inequality to exp(tX), then optimizes over t. Jensen on φ(x) = eˣ provides the initial exponential bound.	Moment generating function exists.

🔭 Looking Ahead

In a graduate probability course, the conditional Jensen's inequality becomes the primary form. It is the key ingredient in proving that convex functions of martingales are submartingales — a result that underpins the optional stopping theorem, Doob's upcrossing inequality, and ultimately the martingale convergence theorem. Mastering Jensen in its full measure-theoretic generality is a prerequisite for these deeper results.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain why Jensen's inequality implies that Var(X) ≥ 0 for any random variable X with finite second moment. Identify the convex function φ, state what the inequality says for this choice, and show how it yields the non-negativity of variance.

PROBLEM 2 — BASIC CALCULATION

Let X be a discrete random variable with P(X = 1) = 1/3, P(X = 4) = 2/3. Verify Jensen's inequality explicitly for the convex function φ(x) = x². Compute both φ(E[X]) and E[φ(X)] and confirm that φ(E[X]) ≤ E[φ(X)].

PROBLEM 3 — INTERMEDIATE

Let X > 0 be a positive random variable with E[X] = μ. Use Jensen's inequality with an appropriate choice of φ to prove that E[1/X] ≥ 1/μ. State clearly which version of Jensen's inequality you are using and verify the convexity of your chosen φ.

PROBLEM 4 — APPLIED

In the EM algorithm, the E-step constructs a lower bound on the log-likelihood ℓ(θ) = ln p(x | θ). Given that p(x | θ) = Σ_z p(x, z | θ) and an auxiliary distribution q(z), use Jensen's inequality to derive the Evidence Lower Bound (ELBO): ℓ(θ) ≥ Σ_z q(z) ln[p(x, z | θ)/q(z)]. State which function's concavity is being exploited.

PROBLEM 5 — CRITICAL THINKING

Let (Mₙ)_{n≥0} be a martingale with respect to a filtration (ℱₙ). Prove that if φ : ℝ → ℝ is convex and E[|φ(Mₙ)|] < ∞ for all n, then (φ(Mₙ))_{n≥0} is a submartingale. Use the conditional form of Jensen's inequality and verify all required conditions.

SUMMARY

Summary

Jensen's inequality states that for any convex function φ and integrable random variable X, φ(E[X]) ≤ E[φ(X)], with the inequality reversing for concave functions. The proof relies on the supporting hyperplane property of convex functions: at any point, there exists a tangent line that lies entirely below the function, and taking expectations of this affine bound yields the result. Equality holds if and only if X is constant almost surely or φ is affine on the support of X.

The inequality's importance extends far beyond a single theorem. By choosing specific convex functions, one recovers the AM–GM inequality, non-negativity of KL divergence, and the variance lower bound. The conditional version connects convexity to martingale theory, establishing that convex transforms of martingales are submartingales — a foundational result in stochastic analysis. In applications, Jensen's inequality appears in the EM algorithm, risk analysis in finance, entropy bounds in information theory, and the derivation of concentration inequalities.