Fisher Information & Cramér-Rao Bound - Statistics Graduate Level

1. Independent observations have exponential density $f(x;\theta)=\theta e^{-\theta x}$ for $x>0$, where $\theta>0$ is the rate. The model is reparameterized by the mean $\eta=1/\theta$. What are the Fisher information for $\eta$ in a sample of size $n$ and the corresponding Cramér–Rao bound for unbiased estimation of $\eta$?
   Answer: $I_n(\eta)=n/\eta^2$ and the bound is $\eta^2/n$ Reparameterization questions test your ability to transform Fisher information under a change of parameters using the delta method identity: if $\eta = g(\theta)$, then $I_n(\eta) = I_n(\theta) / [g'(\theta)]^2$. Start by computing Fisher information in the original parameterization. For the exponential with rate $\theta$, the log-likelihood is $\ell(\theta) = n\log\theta - \theta\sum x_i$, giving $\partial^2\ell/\partial\theta^2 = -n/\theta^2$, so $I_n(\theta) = n/\theta^2$. Now reparameterize: $\eta = 1/\theta$, so $\theta = 1/\eta$ and $d\theta/d\eta = -1/\eta^2$. Applying the transformation rule: $I_n(\eta) = I_n(\theta)\left(\frac{d\theta}{d\eta}\right)^2 = \frac{n}{\theta^2}\cdot\frac{1}{\eta^4} = n\eta^2 \cdot \frac{1}{\eta^4} = \frac{n}{\eta^2}$ The Cramér–Rao lower bound for any unbiased estimator of $\eta$ is $1/I_n(\eta) = \eta^2/n$. This confirms C. Choice A inverts the information itself ($n\eta^2$) rather than correctly applying the chain rule squared, and the bound $1/(n\eta^2)$ follows from that flawed information. Choice B gives $n/\eta^4$, which corresponds to using $d\eta/d\theta = -1/\theta^2$ in the wrong direction without accounting for the correct substitution — essentially applying the inverse transform incorrectly. Choice D suggests $I_n(\eta) = n$, which would only hold if $\eta$ and $\theta$ had a linear relationship (unit Jacobian), ignoring the nonlinear reparameterization entirely. When you see a reparameterization problem, always write out the derivative of the old parameter with respect to the new one, square it, and multiply by the original Fisher information — direction matters.
2. Suppose $X_1,\ldots,X_n$ are independent Bernoulli random variables with success probability $p$, where $0<p<1$. An unbiased estimator is sought for the log-odds parameter $g(p)=\log\{p/(1-p)\}$. Assuming the regular scalar Cramér–Rao inequality applies, what is the resulting lower bound on its variance?
   Answer: $\frac{1}{n p(1-p)}$ Whenever you see a question asking for a Cramér–Rao lower bound (CRLB) on an estimator of a transformed parameter, your strategy is clear: compute the Fisher information for the original model, then apply the delta method to adjust for the transformation. For $X_i \sim \text{Bernoulli}(p)$, the Fisher information in a single observation is $I_1(p) = \frac{1}{p(1-p)}$, so for $n$ independent observations, $I_n(p) = \frac{n}{p(1-p)}$. The CRLB for estimating $g(p)$ is: $\text{CRLB} = \frac{[g'(p)]^2}{I_n(p)}$ Now compute the derivative of $g(p) = \log\!\left(\frac{p}{1-p}\right)$. Using the chain rule: $g'(p) = \frac{1}{p} + \frac{1}{1-p} = \frac{1}{p(1-p)}$ Plugging in: $\text{CRLB} = \frac{\left[\frac{1}{p(1-p)}\right]^2}{\frac{n}{p(1-p)}} = \frac{1}{[p(1-p)]^2} \cdot \frac{p(1-p)}{n} = \frac{1}{n\, p(1-p)}$ This confirms B is correct. Now for the distractors: A, $\frac{p(1-p)}{n}$, is the CRLB for estimating $p$ itself (where $g'(p)=1$), completely ignoring the log-odds transformation. C, $\frac{1}{n[p(1-p)]^2}$, uses $[g'(p)]^2$ in the numerator but forgets to divide by Fisher information correctly — it omits cancellation of one power of $p(1-p)$. D, $\frac{(1-2p)^2}{np(1-p)}$, uses the derivative of $p(1-p)$ rather than $\log\{p/(1-p)\}$, a classic mix-up between transformations. Your study tip: always differentiate $g(p)$ carefully before applying the CRLB formula — most distractors on this type of question exploit an incorrect $g'(p)$.
3. A regular two-parameter model has parameter vector $(\theta,\lambda)$ and per-observation Fisher information matrix $I(\theta,\lambda)=\begin{pmatrix}4&2\\2&3\end{pmatrix}.$ For a sample of $n$ independent observations, $\lambda$ is unknown and is treated as a nuisance parameter. What is the Cramér–Rao lower bound for the variance of an unbiased estimator of $\theta$?
   Answer: $\frac{3}{8n}$, obtained from the corresponding entry of the inverse matrix When nuisance parameters are present, the Cramér–Rao lower bound (CRLB) for a parameter of interest is not simply the reciprocal of that parameter's diagonal entry in the Fisher information matrix. Instead, you must work with the full inverse of the information matrix, because nuisance parameters introduce correlations that inflate the bound. For a two-parameter model with Fisher information $I(\theta,\lambda) = \begin{pmatrix}4&2\\2&3\end{pmatrix}$, the CRLB for $\theta$ when $\lambda$ is unknown is the $(1,1)$ entry of $[nI(\theta,\lambda)]^{-1}$. First, invert the per-observation matrix: $\det(I) = (4)(3)-(2)(2) = 8$, so $I^{-1} = \frac{1}{8}\begin{pmatrix}3&-2\\-2&4\end{pmatrix}$. The $(1,1)$ entry is $\frac{3}{8}$, giving a CRLB of $\frac{3}{8n}$. This confirms answer B. Answer A is the most common trap: inverting only the $\theta\theta$ entry gives $\frac{1}{4n}$, which would be correct only if $\lambda$ were known. Treating a nuisance parameter as known ignores the estimation uncertainty it introduces, artificially deflating the bound. Answer C, $\frac{1}{2n}$, doesn't correspond to any standard formula — it appears to subtract cross-information once rather than applying the full matrix inversion, which has no theoretical justification. Answer D, $\frac{3}{4n}$, comes from inverting diagonal entries separately (e.g., $\frac{1}{4} \cdot \frac{1}{3}$), ignoring off-diagonal terms entirely — another incorrect shortcut. Key strategy: Whenever a nuisance parameter is present, always compute the full matrix inverse first, then read off the relevant diagonal entry. The phrase "nuisance parameter" on an exam is your signal to invert the complete Fisher information matrix.
4. Let $X_1,\ldots,X_n$ be independent $N(\theta,\sigma^2)$ observations, where $\sigma^2$ is known. Consider the class of estimators $T$ satisfying $E_\theta(T)=\rho\theta$ for every $\theta$, where $\rho$ is a fixed constant. What lower bound on $E_\theta[(T-\theta)^2]$ follows from the biased Cramér–Rao inequality?
   Answer: $\frac{\rho^2\sigma^2}{n}+(\rho-1)^2\theta^2$ When you see a question invoking the Cramér–Rao lower bound for a biased estimator, your first move should be to recall the biased CRB formula: for an estimator with $E_\theta(T) = \psi(\theta)$, the MSE satisfies $E_\theta[(T-\theta)^2] \geq \frac{[\psi'(\theta)]^2}{I_n(\theta)} + [\psi(\theta)-\theta]^2,$ where the second term is the squared bias and the first term bounds the variance via the Fisher information. Here, $\psi(\theta) = \rho\theta$, so $\psi'(\theta) = \rho$. For $n$ i.i.d. $N(\theta,\sigma^2)$ observations with known $\sigma^2$, the total Fisher information is $I_n(\theta) = n/\sigma^2$. The squared bias is $(\rho\theta - \theta)^2 = (\rho-1)^2\theta^2$. Plugging in: $E_\theta[(T-\theta)^2] \geq \frac{\rho^2}{n/\sigma^2} + (\rho-1)^2\theta^2 = \frac{\rho^2\sigma^2}{n} + (\rho-1)^2\theta^2.$ This confirms B is correct. Now for the distractors: A uses $\rho^2 = 1$ in the variance term, effectively ignoring the $\rho$ scaling from $\psi'(\theta)$ — a common error of treating the estimator as unbiased in the CRB numerator. C swaps the roles of $\rho$ and $(\rho-1)$, placing $(\rho-1)^2$ in the variance term and $\rho^2$ in the bias term — a clean reversal of the correct formula. D takes a square root of the bias term prematurely, writing $|\rho-1|\theta^2$ instead of $(\rho-1)^2\theta^2$, and also drops the $\rho^2$ from the variance. Study tip: Always write out $\psi(\theta)$ and $\psi'(\theta)$ explicitly before applying the biased CRB — the derivative goes in the variance bound, and the function itself determines the bias term.
5. An experiment compares two independent treatment means. Each response in treatment 1 is $N(\mu_1,1)$, and each response in treatment 2 is $N(\mu_2,4)$. With a fixed total of $N$ observations, let $n_1+n_2=N$ and suppose both sample sizes may be treated as continuous for design purposes. Which allocation minimizes the Cramér–Rao bound for unbiased estimation of $\mu_1-\mu_2$, and what is the minimized bound?
   Answer: $n_1:n_2=1:2$, with minimized bound $9/N$ When comparing two independent normal means with unequal variances, the Cramér–Rao bound for any unbiased estimator of $\mu_1 - \mu_2$ equals the variance of the best linear unbiased estimator, which is $\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}$. Here $\sigma_1^2 = 1$ and $\sigma_2^2 = 4$, so you want to minimize $f(n_1) = \frac{1}{n_1} + \frac{4}{N - n_1}$ subject to $n_1 + n_2 = N$. Taking the derivative and setting it to zero: $-\frac{1}{n_1^2} + \frac{4}{(N-n_1)^2} = 0$, which gives $n_2/n_1 = 2$, or $n_1:n_2 = 1:2$. This confirms answer A is correct. With $n_1 = N/3$ and $n_2 = 2N/3$, the minimized bound is $\frac{1}{N/3} + \frac{4}{2N/3} = \frac{3}{N} + \frac{6}{N} = \frac{9}{N}$. The key insight is that the optimal allocation is proportional to the standard deviations, not the variances: $n_i \propto \sigma_i$, so $n_1:n_2 = 1:2$. B uses equal allocation (1:1), which gives $\frac{5}{N/2} + \frac{5}{N/2}$ — wait, more precisely $\frac{2}{N} + \frac{8}{N} = \frac{10}{N}$, which is suboptimal. C uses ratio 1:4 (proportional to variances, a common trap), yielding $\frac{5}{N} + \frac{5}{N} = \frac{10}{N}$, also suboptimal. D uses 1:3, giving $\frac{4}{N} + \frac{6}{N} = \frac{10}{N}$ — not $16/N$ as stated, making it doubly wrong. Remember: optimal allocation in unequal-variance designs scales with $\sigma_i$, not $\sigma_i^2$. Confusing standard deviations with variances is the most common trap on design-of-experiments questions like this.
6. Suppose $X_1,\ldots,X_n$ are independent and $X_i\sim N(\theta,\theta^2)$, where $\theta>0$. Both the mean and variance therefore depend on the same parameter. What are the sample Fisher information for $\theta$ and the Cramér–Rao lower bound for an unbiased estimator of $\theta$?
   Answer: $I_n(\theta)=3n/\theta^2$ and the bound is $\theta^2/(3n)$ When the same parameter controls both the mean and variance, you must account for both sources of information when computing Fisher information — a detail that trips up many students. For $X_i \sim N(\theta, \theta^2)$, the log-likelihood for a single observation is: $\ell(\theta; x) = -\log\theta - \frac{(x-\theta)^2}{2\theta^2}$ To find $I_1(\theta) = -E\left[\frac{\partial^2 \ell}{\partial \theta^2}\right]$, differentiate $\ell$ with respect to $\theta$. The score function is: $\frac{\partial \ell}{\partial \theta} = -\frac{1}{\theta} + \frac{(x-\theta)}{\theta^2} + \frac{(x-\theta)^2}{\theta^3}$ After computing the second derivative and taking the expectation (using $E[(X-\theta)^2] = \theta^2$ and $E[(X-\theta)] = 0$), you obtain: $I_1(\theta) = \frac{1}{\theta^2} + \frac{2}{\theta^2} = \frac{3}{\theta^2}$ The "1" comes from the mean carrying information, and the "2" comes from the variance carrying information (standard result for a normal with only variance unknown). Together they sum to 3. For $n$ observations, $I_n(\theta) = 3n/\theta^2$, and the Cramér–Rao lower bound is $1/I_n(\theta) = \theta^2/(3n)$, confirming C. Choice A corresponds to ignoring both the mean's and variance's joint contribution — it matches a case where only a scale parameter is estimated with one unit of information. Choice B captures only the variance contribution (the "2"), forgetting the mean contributes an additional unit. Choice D overcounts, perhaps by doubling the variance term. Study tip: Whenever $\mu$ and $\sigma^2$ are tied to the same parameter, always differentiate the full log-likelihood — don't treat them separately. The information adds across both dependencies.
7. Let $X_1,\ldots,X_n$ be independent $N(\theta,\sigma^2)$ observations with known $\sigma^2$. The estimator $T=\bar X^2-\sigma^2/n$ is unbiased for $\theta^2$. Which statement correctly compares its variance with the Cramér–Rao lower bound for unbiased estimation of $\theta^2$?
   Answer: It exceeds the bound by $2\sigma^4/n^2$, and no unbiased estimator attains the bound for every $\theta$. When comparing an estimator's variance to the Cramér–Rao lower bound (CRLB), your first move should be to compute the actual variance and the CRLB separately, then compare. **Computing the actual variance of $T = \bar{X}^2 - \sigma^2/n$:** Since $\bar{X} \sim N(\theta, \sigma^2/n)$, we know $\bar{X}^2$ follows a scaled noncentral chi-squared distribution. Using the formula $\text{Var}(\bar{X}^2) = 4\theta^2\sigma^2/n + 2\sigma^4/n^2$, and since $\sigma^2/n$ is a constant, $\text{Var}(T) = 4\theta^2\sigma^2/n + 2\sigma^4/n^2$. **Computing the CRLB for estimating $g(\theta) = \theta^2$:** The Fisher information for $\theta$ from $n$ normal observations is $I(\theta) = n/\sigma^2$. The CRLB is $[g'(\theta)]^2/I(\theta) = (2\theta)^2 \cdot \sigma^2/n = 4\theta^2\sigma^2/n$. Subtracting: $\text{Var}(T) - \text{CRLB} = 2\sigma^4/n^2 > 0$ for all $\theta$. This confirms D — the variance exceeds the bound by exactly $2\sigma^4/n^2$, and no unbiased estimator can attain it everywhere. A is tempting because $T$ is indeed a function of the complete sufficient statistic $\bar{X}$, making it the UMVUE. But being UMVUE means minimum variance among unbiased estimators — it doesn't guarantee the CRLB is attained. B incorrectly identifies the excess as $4\theta^2\sigma^2/n$, confusing the full variance with the gap. C is backwards: the CRLB vanishes at $\theta=0$ (since $g'(0)=0$), but the estimator's variance there equals $2\sigma^4/n^2 > 0$, so the bound is exceeded most severely at $\theta=0$, not beaten. Remember: UMVUE ≠ CRLB-achieving. Always compute both sides explicitly before drawing conclusions.
8. For each of $n$ subjects, let $X_i\sim\operatorname{Bernoulli}(p)$. Independently of $X_i$, an indicator $R_i\sim\operatorname{Bernoulli}(q)$ determines whether $X_i$ is observed; the value of every $R_i$ is recorded, and $q$ is known. What is the expected Fisher information for $p$ in the observed data, and what scalar Cramér–Rao bound does it imply?
   Answer: $I(p)=\frac{nq}{p(1-p)}$ and the bound is $\frac{p(1-p)}{nq}$ When you encounter missing data problems in Fisher information, the key is carefully identifying what your observed data actually consists of — not just the complete-case observations, but the full observed structure including the missingness indicators. Here, for each subject $i$, you observe the pair $(R_i, R_i X_i)$: you always see whether the subject responded, and you see $X_i$ only when $R_i = 1$. The log-likelihood contribution from subject $i$ comes from two cases. When $R_i = 0$ (probability $1-q$), you learn nothing about $p$, contributing zero Fisher information. When $R_i = 1$ (probability $q$), you observe a Bernoulli$(p)$ draw, which contributes $\frac{1}{p(1-p)}$ to Fisher information — the standard single-observation Fisher information for a Bernoulli. Weighting by the probability of being observed, each subject contributes $q \cdot \frac{1}{p(1-p)}$, and summing over $n$ independent subjects gives $I(p) = \frac{nq}{p(1-p)}$. The Cramér–Rao bound is the reciprocal: $\frac{p(1-p)}{nq}$. This confirms D. Choice A ignores missingness entirely — it's the information you'd have with complete data, as if every subject were always observed. Choice B divides by $q$ instead of multiplying, which would imply more information under higher missingness — the exact opposite of reality. Choice C uses $q^2$, which has no statistical basis here; squaring $q$ would arise only under a very different data structure. The key study tip: always ask "what is the full observed data?" in missing-data problems. The missingness indicators themselves carry structure, and Fisher information must reflect the actual observation mechanism — never assume complete data unless the problem guarantees it.
9. Let $X_1,\ldots,X_n$ be independent with density $f(x;\theta)=1/\theta$ for $0<x<\theta$, where $\theta>0$ and $n\ge 2$. The estimator $T=(n+1)X_{(n)}/n$ is unbiased for $\theta$ and has variance $\theta^2/[n(n+2)]$. Squaring the formal score $-n/\theta$ would instead suggest information $n^2/\theta^2$ and a lower bound $\theta^2/n^2$, which is larger than $\operatorname{Var}_\theta(T)$. Which statement best resolves this apparent contradiction?
   Answer: The support depends on $\theta$, so differentiation under the integral fails and the usual Cramér–Rao argument is invalid. Whenever you encounter a Cramér–Rao lower bound (CRLB) problem, your first instinct should be to check whether the regularity conditions are satisfied — before doing any algebra with the score function. The CRLB derivation relies on differentiating under an integral sign: $\frac{d}{d\theta}\int f(x;\theta)\,dx = \int \frac{\partial}{\partial\theta}f(x;\theta)\,dx$. This interchange is only valid when the support of $f$ does not depend on the parameter. Here, $f(x;\theta) = 1/\theta$ on $(0,\theta)$, so the upper limit of integration moves with $\theta$. The differentiation-under-the-integral step breaks down entirely, making the standard CRLB machinery inapplicable. That is why B is correct: the "information" $n^2/\theta^2$ computed from the score is a meaningless artifact of applying a formula outside its domain of validity. The fact that $\text{Var}(T) = \theta^2/[n(n+2)]$ is smaller than the spurious "bound" $\theta^2/n^2$ is not a contradiction — the bound simply never applied. Choice A is wrong because the CRLB, when valid, applies to any unbiased estimator regardless of whether it uses order statistics or the sample mean. Choice C is wrong because $T = (n+1)X_{(n)}/n$ is exactly unbiased in finite samples, not just asymptotically — you can verify this from the known distribution of $X_{(n)}$. Choice D is wrong because there is no corrected "true" Fisher information that salvages the standard CRLB here; the entire framework collapses when support depends on $\theta$. Your study tip: always list the CRLB regularity conditions before applying the bound. "Support depends on $\theta$" is the single most common condition that fails on exam problems — flag it immediately when you see truncated or bounded distributions like Uniform$(0,\theta)$, Exponential shifted by $\theta$, or similar families.
10. In a regular scalar-parameter model, let $U(X;\theta)$ denote the full-data score and let $T=T(X)$ be a statistic. At a specified parameter value, suppose $\operatorname{Var}_\theta[U(X;\theta)]=10$ and $E_\theta\{\operatorname{Var}_\theta[U(X;\theta)\mid T]\}=3$. What is the Fisher information about $\theta$ contained in the distribution of $T$?
    Answer: $7$, because the statistic's score is the conditional mean of the full score Whenever you see a question linking full-data Fisher information to a statistic's information, reach for the variance decomposition of the score. The law of total variance tells you that for any random variable and conditioning statistic: $\operatorname{Var}[U] = \operatorname{Var}[E(U \mid T)] + E[\operatorname{Var}(U \mid T)]$ The key insight is that the **Fisher information in $T$** equals $\operatorname{Var}_\theta[E_\theta(U \mid T)]$, because the score of the marginal likelihood for $T$ is precisely $E_\theta[U(X;\theta) \mid T]$. This is a fundamental result: conditioning the full score on a statistic yields that statistic's score. Plugging in the given values: $\operatorname{Var}[E(U \mid T)] = \operatorname{Var}[U] - E[\operatorname{Var}(U \mid T)] = 10 - 3 = 7$. So the Fisher information in $T$ is 7, confirming choice A is correct. Choice B (answer 3) confuses the two components — $E[\operatorname{Var}(U \mid T)] = 3$ represents the **within-$T$ score variation**, which is the information lost by reducing to $T$, not the information retained. Choice C (answer 10) would only be correct if $T$ were sufficient, preserving all information; a general statistic loses some. Choice D (answer 13) incorrectly adds the two variance components rather than recognizing they partition the total variance. Study tip: Memorize the identity $I_T(\theta) = I(\theta) - E[\operatorname{Var}(U \mid T)]$. On exam questions about partial sufficiency or information loss, the "leftover" conditional variance is always the missing information, not the retained information.

Start this quiz