Jensen's Inequality - Statistics Graduate Level

1. Let probability measure $P$ be absolutely continuous with respect to $Q$, with likelihood ratio $L=dP/dQ$. For a statistic $S$, let $P_S$ and $Q_S$ denote the induced distributions of $S$. Assume the relevant divergences are finite. Which statement correctly applies conditional Jensen's inequality to $x\mapsto x\log x$?
   Answer: $D(P_S\|Q_S)\leq D(P\|Q)$, with equality exactly when $L$ is measurable with respect to $S$ up to null sets. Whenever you see KL divergence and a statistic $S$, think about the data-processing inequality: processing data can only destroy information, never create it. The rigorous derivation uses conditional Jensen's inequality applied to the convex function $\phi(x) = x\log x$. Here's the core argument. We have $D(P\|Q) = E_Q[L \log L]$. The induced likelihood ratio is $L_S = dP_S/dQ_S = E_Q[L \mid S]$ (the conditional expectation of $L$ given $S$, under $Q$). Since $\phi(x) = x\log x$ is convex, Jensen's inequality applied conditionally gives: $E_Q[\phi(L) \mid S] \geq \phi(E_Q[L \mid S]) = \phi(L_S)$ Taking expectations over $S$: $D(P\|Q) = E_Q[L\log L] \geq E_{Q_S}[L_S \log L_S] = D(P_S\|Q_S)$, confirming $D(P_S\|Q_S) \leq D(P\|Q)$. Equality in Jensen's holds if and only if $L$ is already $\sigma(S)$-measurable (i.e., $L = E_Q[L\mid S]$ a.s.), meaning $L$ is measurable with respect to $S$. This is exactly the definition of $S$ being sufficient for $P$ versus $Q$. So D is correct. A gets the inequality direction right but misidentifies the equality condition. Independence of $S$ and $L$ under $Q$ would collapse $L_S$ to a constant, a much stronger and incorrect condition. B flips the inequality entirely — divergence cannot increase under a statistic, period. C claims equality always holds, which contradicts the strict inequality whenever $L$ is not $S$-measurable. As a study tip: whenever equality in Jensen's is discussed, always ask "when is the random variable already constant given the conditioning?" — that directly identifies sufficiency conditions.
2. A random variable $X$ is supported on $[-2,-1]$, has mean $\mathbb{E}[X]=-3/2$, and has positive variance. What does Jensen's inequality imply about its third moment?
   Answer: $\mathbb{E}[X^3]< -\frac{27}{8}$ because the cubic function is strictly concave on the support. Whenever you see Jensen's inequality applied to a specific support, your first task is to determine the local convexity or concavity of the function on that support — not globally. The cubic function $f(x) = x^3$ has second derivative $f''(x) = 6x$. On the support $[-2, -1]$, we have $x < 0$, so $f''(x) = 6x < 0$ throughout. This means $f$ is strictly concave on $[-2, -1]$. Jensen's inequality for a strictly concave function states that $\mathbb{E}[f(X)] < f(\mathbb{E}[X])$ whenever $X$ has positive variance. Applying this: $\mathbb{E}[X^3] < \left(\mathbb{E}[X]\right)^3 = \left(-\frac{3}{2}\right)^3 = -\frac{27}{8}$ This confirms C is correct. Choice A is wrong on two counts: it claims the inequality goes the other way ($\geq$), and the phrase "odd powers preserve the Jensen direction" is meaningless — direction is determined by local concavity/convexity, not the parity of the exponent. Choice B makes the opposite convexity error. $x^3$ is strictly convex for $x > 0$, but on $[-2,-1]$ it is strictly concave. Confusing global behavior with local behavior is a classic trap. Choice D is tempting but wrong. Jensen's inequality requires only knowledge of $\mathbb{E}[X]$, the sign of variance, and the local curvature — all of which are given. No further distributional information is needed. Study tip: Always evaluate $f''(x)$ over the actual support before applying Jensen's — the sign of the second derivative on the support determines everything.
3. An integrable random variable $X$ satisfies $\mathbb{E}[X]=-1$ and $\mathbb{E}|X|=1$. Which conclusion is forced by the equality case of Jensen's inequality for $x\mapsto |x|$?
   Answer: $X\leq 0$ almost surely, although its variance need not be zero. When you see conditions on both $\mathbb{E}[X]$ and $\mathbb{E}|X|$, immediately think about Jensen's inequality applied to the convex function $\varphi(x) = |x|$. Jensen's inequality states $|\mathbb{E}[X]| \leq \mathbb{E}|X|$, with equality holding if and only if $X$ is almost surely confined to a region where $\varphi$ is linear — that is, where $|x|$ does not "bend." Here, $|\mathbb{E}[X]| = |-1| = 1 = \mathbb{E}|X|$, so equality holds exactly. The function $|x|$ is linear on $(-\infty, 0]$ (where it equals $-x$) and on $[0, \infty)$ (where it equals $x$). Equality in Jensen's is achieved when $X$ stays within one of these linear branches almost surely — meaning $X \leq 0$ a.s. or $X \geq 0$ a.s. Since $\mathbb{E}[X] = -1 < 0$, the branch $X \geq 0$ a.s. is ruled out (it would force a non-negative mean). Therefore, $X \leq 0$ almost surely. This is answer B. Notice that $X$ could still be spread across many negative values, so its variance is not forced to zero. Answer A is too strong: $X = -1$ a.s. would additionally require $\operatorname{Var}(X) = 0$, which is not implied. Answer C is a fabrication — nothing about the equality condition requires symmetry around zero or equal probabilities on each side. Answer D correctly identifies zero variance as sufficient but not necessary; the equality condition constrains the sign of $X$, not its spread. Your study tip: memorize that equality in Jensen's for $|x|$ forces the random variable onto a single linear branch of the absolute value — this pins down the sign, not the exact value.
4. Let $X_1,\ldots,X_n$ be positive, identically distributed random variables with finite common mean $\mu$. They are not assumed independent. Define $\overline{X}=n^{-1}\sum_{i=1}^n X_i$ and assume $\mathbb{E}|\log \overline{X}|<\infty$. Which is the sharp Jensen conclusion?
   Answer: $\mathbb{E}[\log \overline{X}]\leq \log\mu$, with equality exactly when $\overline{X}=\mu$ almost surely. Whenever you see a question invoking Jensen's inequality, your first move should be to identify the function's concavity and what random variable it's being applied to. Here, $f(x) = \log x$ is concave on the positive reals. Jensen's inequality for a concave function states that $\mathbb{E}[f(Y)] \leq f(\mathbb{E}[Y])$ for any integrable random variable $Y$. Apply this directly to $Y = \overline{X}$: since $\mathbb{E}[\overline{X}] = \mu$, you immediately get $\mathbb{E}[\log \overline{X}] \leq \log \mathbb{E}[\overline{X}] = \log \mu$. Notice that the entire argument hinges only on $\overline{X}$ itself — no assumptions about independence or the joint structure of the $X_i$ are needed. The equality condition for Jensen with a strictly concave function is that the argument must be almost surely constant — that is, $\overline{X} = c$ a.s. for some constant, and since $\mathbb{E}[\overline{X}] = \mu$, that constant must be $\mu$. This confirms A is correct. B is wrong because it over-specifies the equality condition. Requiring each $X_i = \mu$ a.s. is sufficient but not necessary — you only need $\overline{X} = \mu$ a.s., which can happen in non-trivial ways (e.g., dependent $X_i$ that average to a constant without each being constant). C reverses the inequality direction, confusing concavity with convexity. D is a classic distractor that injects independence into a result that requires no such assumption — Jensen's inequality is purely a statement about the distribution of a single random variable. Your study tip: always ask "what random variable am I applying Jensen to?" The equality condition belongs to that variable being a.s. constant — not to any underlying structure that produces it.
5. Let $W$, $X$, and $Y$ be random variables on the same probability space, with $0\leq W\leq 1$ almost surely. Let $\phi$ be convex, and assume all displayed expectations are finite. Which inequality is valid even when $W$ is dependent on both $X$ and $Y$?
   Answer: $\mathbb{E}\phi(WX+(1-W)Y)\leq\mathbb{E}[W\phi(X)+(1-W)\phi(Y)]$ When you see a question mixing convexity with a random weight, your first instinct should be Jensen's inequality — but the key is identifying which version applies and whether independence is required. The core tool here is the conditional Jensen's inequality: for convex $\phi$ and any random weight $W \in [0,1]$, the convexity of $\phi$ guarantees that pointwise, for every realization of $(W, X, Y)$, $\phi(WX + (1-W)Y) \leq W\phi(X) + (1-W)\phi(Y).$ This holds because convexity is a deterministic, sample-path property — it doesn't care about the joint distribution of $W$, $X$, and $Y$. Taking expectations of both sides of a pointwise inequality is always valid (by monotonicity of expectation), so $\mathbb{E}\phi(WX+(1-W)Y) \leq \mathbb{E}[W\phi(X)+(1-W)\phi(Y)],$ which is exactly B — and it requires no independence assumption whatsoever. A is wrong because factoring $\mathbb{E}[W\phi(X)]$ into $\mathbb{E}[W]\cdot\mathbb{E}[\phi(X)]$ requires $W \perp X$ (and similarly $W \perp Y$), which fails when dependence is present. C reverses the inequality in B, contradicting Jensen's inequality for convex $\phi$. D applies Jensen in the wrong direction: for convex $\phi$, Jensen gives $\phi(\mathbb{E}[Z]) \leq \mathbb{E}[\phi(Z)]$, so $\phi(\mathbb{E}[\cdot])$ is the smaller side, not the larger one. Study tip: Always distinguish between pointwise inequalities (which survive taking expectations freely) and results that require independence to factor joint expectations. Dependence only breaks factorizations, not pointwise convexity arguments.
6. Let $T$ be an integrable estimator of a parameter $\theta$, and let $S$ be a statistic. Define the Rao–Blackwellized estimator $T^*=\mathbb{E}[T\mid S]$. Under absolute-error loss, which statement is guaranteed without assuming that $S$ is sufficient or that $T$ is unbiased?
   Answer: $\mathbb{E}|T^*-\theta|\leq \mathbb{E}|T-\theta|$, by the convexity of $x\mapsto|x-\theta|$ and Jensen's inequality applied conditionally. When you see a question about Rao–Blackwellization, your first instinct should be to reach for Jensen's inequality — not the sufficiency condition. Many students conflate the classical Rao–Blackwell theorem (which requires sufficiency to guarantee the conditional expectation is a valid, parameter-free estimator) with the purely probabilistic risk comparison, which holds more generally. Here's the core argument for A: For any fixed $\theta$, the function $x \mapsto |x - \theta|$ is convex. By the conditional form of Jensen's inequality, $|\mathbb{E}[T \mid S] - \theta| \leq \mathbb{E}[|T - \theta| \mid S]$ almost surely. Taking expectations of both sides gives $\mathbb{E}|T^* - \theta| \leq \mathbb{E}|T - \theta|$. This requires only integrability of $T$ — no sufficiency, no unbiasedness. A is therefore always guaranteed. B is tempting but wrong. Absolute value $|x - \theta|$ is not strictly convex everywhere — it has a "kink" at $x = \theta$ and is linear on each side. Strict convexity arguments that yield strict inequality (like those valid for $x^2$) fail here; you cannot guarantee strict improvement under absolute-error loss even when $\operatorname{Var}(T \mid S) > 0$. C is simply backwards. Conditioning smooths out variation via Jensen's, which weakly reduces risk, not inflates it. D conflates two separate issues. Sufficiency ensures $T^*$ doesn't depend on $\theta$ (making it a legitimate estimator), but the risk inequality follows from Jensen alone and needs no sufficiency assumption. Study tip: Always separate the probabilistic question (does Jensen apply?) from the statistical question (is the result a valid estimator?). Sufficiency matters for the latter, not the former.
7. A positive random variable $X$ is supported on $[1,3]$ and satisfies $\mathbb{E}[X]=2$ and $\operatorname{Var}(X)=1/4$. Using a variance-refined form of Jensen's inequality and the curvature of $f(x)=-\log x$, which lower bound is valid?
   Answer: $\mathbb{E}[-\log X]\geq -\log 2+\frac{1}{72}$ When you see a question combining Jensen's inequality with variance information, think "variance-refined Jensen's bound." The standard Jensen's inequality gives $\mathbb{E}[f(X)] \geq f(\mathbb{E}[X])$ for convex $f$, but a tighter version incorporates curvature and variance. The variance-refined lower bound states: for a convex function $f$, $\mathbb{E}[f(X)] \geq f(\mu) + \frac{1}{2}f''(\xi)\operatorname{Var}(X)$ where $\xi$ lies in the support. For $f(x) = -\log x$, compute $f''(x) = 1/x^2$. To find a valid lower bound, you want the minimum curvature over the support $[1,3]$, which occurs at the largest $x$: $f''(3) = 1/9$. Using $\mu = 2$ and $\operatorname{Var}(X) = 1/4$: $\mathbb{E}[-\log X] \geq -\log 2 + \frac{1}{2} \cdot \frac{1}{9} \cdot \frac{1}{4} = -\log 2 + \frac{1}{72}$ This confirms answer A. Answer B uses $f''(2) = 1/4$ (evaluating curvature at the mean rather than at the support endpoint), giving $\frac{1}{2} \cdot \frac{1}{4} \cdot \frac{1}{4} = \frac{1}{32}$, which is not $\frac{1}{36}$ — and more importantly, using the mean's curvature doesn't guarantee a valid lower bound without additional argument. Answer C ignores that convexity plus variance information always yields a strict improvement when $\operatorname{Var}(X) > 0$. Answer D is negative, but since $-\log x$ is convex, the variance correction must push the bound upward, not downward. Your strategy: for variance-refined Jensen bounds, always use the minimum value of $f''$ over the support to ensure the inequality direction is preserved — this is where most errors occur.
8. Let $\mathcal{G}\subseteq\mathcal{H}$ be two sub-$\sigma$-fields, let $X$ be integrable, and let $\phi$ be convex with all relevant expectations finite. Which ordering of expected convex transforms is guaranteed?
   Answer: $\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{G}])\leq\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{H}])\leq\mathbb{E}\phi(X)$, by two applications of Jensen's inequality and the tower property. Whenever you see conditional expectations composed with a convex function, your instinct should be to reach for Jensen's inequality combined with the tower property (also called the law of iterated expectations). Here's the core reasoning for why C is correct. Since $\mathcal{G}\subseteq\mathcal{H}$, the tower property tells you $\mathbb{E}[X\mid\mathcal{G}] = \mathbb{E}[\mathbb{E}[X\mid\mathcal{H}]\mid\mathcal{G}]$. Now apply Jensen's inequality to the convex function $\phi$: conditioning on $\mathcal{G}$ and then applying $\phi$ gives $\mathbb{E}[\phi(\mathbb{E}[X\mid\mathcal{H}])\mid\mathcal{G}] \geq \phi(\mathbb{E}[\mathbb{E}[X\mid\mathcal{H}]\mid\mathcal{G}]) = \phi(\mathbb{E}[X\mid\mathcal{G}])$. Taking full expectations yields $\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{H}])\geq\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{G}])$. A second application of Jensen directly gives $\mathbb{E}\phi(X)\geq\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{H}])$. Chaining these produces the inequality in C: more information → larger expected convex transform, bounded above by $\mathbb{E}\phi(X)$. A reverses the correct direction by claiming less information yields a smaller expected convex transform — exactly backwards. B correctly identifies that $\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{G}])\leq\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{H}])$, but then wrongly claims both exceed $\mathbb{E}\phi(X)$, flipping Jensen's inequality entirely. D gets the lower bound $\phi(\mathbb{E}[X])\leq\mathbb{E}\phi(\mathbb{E}[X\mid\mathcal{G}])$ right but wrongly claims no upper bound by $\mathbb{E}\phi(X)$ exists — Jensen guarantees exactly that bound. Study tip: Memorize the mantra — *conditioning contracts, $\phi$ expands* — meaning finer $\sigma$-fields push $\mathbb{E}\phi(\mathbb{E}[X\mid\cdot])$ upward toward $\mathbb{E}\phi(X)$. Tower property + Jensen, applied twice, is the engine behind nearly every such ordering problem.
9. Suppose $X$ is nondegenerate, $\mathbb{E}[X]=0$, and its moment-generating function $M_X(t)=\mathbb{E}[e^{tX}]$ is finite for every real $t$. Let $K_X(t)=\log M_X(t)$. Which statement follows directly from the strict Jensen's inequality applied to $x\mapsto e^{tx}$?
   Answer: $K_X(t)>0$ for every $t\neq 0$, regardless of the sign of $t$. When you see a question involving the cumulant generating function $K_X(t) = \log M_X(t)$ and Jensen's inequality, your instinct should be to apply strict convexity of $x \mapsto e^{tx}$ (for $t \neq 0$) to a nondegenerate random variable. Here's the core reasoning: since $e^{tx}$ is strictly convex and $X$ is nondegenerate (not a point mass), strict Jensen's inequality gives $\mathbb{E}[e^{tX}] > e^{t\mathbb{E}[X]}$. Because $\mathbb{E}[X] = 0$, this becomes $M_X(t) > e^0 = 1$ for every $t \neq 0$. Taking logarithms (a strictly increasing function) preserves the inequality: $K_X(t) = \log M_X(t) > \log 1 = 0$. This holds for all $t \neq 0$, confirming A is correct. The key insight is that strict convexity plus nondegeneracy forces $M_X(t) > 1$ symmetrically — both positive and negative $t$ yield the same conclusion. B is wrong because it claims asymmetry. The argument above is symmetric in $t$: for negative $t$, $e^{tX}$ is still strictly convex, so $M_X(t) > 1$ still holds, giving $K_X(t) > 0$. C is wrong because it weakens the conclusion to $\geq 0$ and suggests equality might occur at nonzero $t$. But strict Jensen's inequality (nondegeneracy is essential here!) guarantees strict inequality. D is a distractor involving a Gaussian-style upper bound on $K_X(t)$. This would be an upper bound direction and doesn't follow from Jensen applied here — it conflates Jensen with a Taylor/moment bound. Study tip: Memorize the chain — nondegeneracy + strict convexity → strict Jensen → $M_X(t) > 1$ → $K_X(t) > 0$. This chain is symmetric in $t$, so never assume sign-dependence without cause.
10. Two predictive models assign strictly positive densities $p_1$ and $p_2$ to an outcome $Y$ whose true distribution is $Q$. For a fixed $0<\lambda<1$, define the mixture density $p_\lambda=\lambda p_1+(1-\lambda)p_2$. Assume all expected log losses are finite. Which comparison follows from Jensen's inequality?
    Answer: $\mathbb{E}_Q[-\log p_\lambda(Y)]\leq\lambda\mathbb{E}_Q[-\log p_1(Y)]+(1-\lambda)\mathbb{E}_Q[-\log p_2(Y)]$ Whenever you encounter a question involving a log of a mixture (or any concave function applied to a convex combination), Jensen's inequality is your primary tool. Recall that $f$ is concave if $f(\lambda a + (1-\lambda)b) \geq \lambda f(a) + (1-\lambda)f(b)$. Since $\log$ is concave, applying it to a mixture satisfies this direction. Here's the key move. Because $p_\lambda = \lambda p_1 + (1-\lambda)p_2$, Jensen's inequality for the concave function $\log$ gives: $\log(\lambda p_1(y) + (1-\lambda)p_2(y)) \geq \lambda \log p_1(y) + (1-\lambda)\log p_2(y).$ Multiplying through by $-1$ flips the inequality (since $-\log$ is convex): $-\log p_\lambda(y) \leq -\lambda \log p_1(y) - (1-\lambda)\log p_2(y).$ Taking expectations under $Q$ preserves the direction, yielding exactly D: the expected log loss of the mixture is no greater than the convex combination of the individual log losses. This is the key log-pooling result — mixing predictions never hurts in terms of worst-case log loss. A claims equality, which only holds when $\log$ is linear — it never is, so A is wrong. B reverses the inequality, contradicting Jensen's direction for a concave function. C applies Jensen in the wrong place — it bounds $\mathbb{E}_Q[-\log p_\lambda]$ using $-\log$ of an expectation of $p$, which conflates two separate applications of Jensen and yields a weaker, less relevant bound. As a study habit: always identify whether your function is convex or concave first, then determine which direction Jensen pulls the inequality — getting that direction right is the entire question.

Start this quiz