Sufficient Statistics - Statistics Graduate Level

Suppose $X_1,\ldots,X_n$ are independent gamma random variables with density $f(x\mid\alpha,\beta)=\beta^\alpha x^{\alpha-1}e^{-\beta x}/\Gamma(\alpha)$ for $x>0$ , where both $\alpha$ and $\beta$ are unknown. Which statistic is jointly sufficient for $\left(\alpha,\beta\right)$ ?
Answer: $\left(\sum_{i=1}^nX_i,\,\sum_{i=1}^n\log X_i\right)$ Whenever you see a question about sufficiency for a multi-parameter family, your first move should be the Fisher-Neyman Factorization Theorem: a statistic $T(\mathbf{X})$ is sufficient if and only if the joint likelihood factors as $g(T(\mathbf{X}), \alpha, \beta) \cdot h(\mathbf{X})$ , where $h$ doesn't involve the parameters. Write out the joint likelihood for the gamma family: $L(\alpha,\beta \mid \mathbf{x}) = \beta^{n\alpha} \cdot e^{-\beta \sum x_i} \cdot \left(\prod x_i\right)^{\alpha-1} \cdot \left[\Gamma(\alpha)^n\right]^{-1}$ Rewriting $\left(\prod x_i\right)^{\alpha-1} = e^{(\alpha-1)\sum \log x_i}$ , the likelihood becomes: $\beta^{n\alpha} \cdot e^{-\beta \sum x_i} \cdot e^{(\alpha-1)\sum \log x_i} \cdot \left[\Gamma(\alpha)^n\right]^{-1}$ The parameters $\alpha$ and $\beta$ enter only through $\sum x_i$ and $\sum \log x_i$ . By the Factorization Theorem, C — $\left(\sum X_i,\, \sum \log X_i\right)$ — is jointly sufficient. A is wrong because $(\sum \log X_i)^2$ is not a natural sufficient statistic; the likelihood depends on $\sum \log x_i$ linearly, not its squared sum. B is wrong because $\sum X_i^2$ does not appear anywhere in the likelihood — quadratic terms arise in normal, not gamma, families. D is wrong because while $\bar{X}$ captures $\sum X_i$ , replacing $\sum \log X_i$ with the sample variance discards essential information about $\alpha$ . Study tip: For exponential family distributions, the sufficient statistics are always the natural statistics multiplying the natural parameters in the exponent — identify those terms directly from the log-likelihood.
In the fixed-design normal linear model $Y\sim N(X\beta,\sigma^2I_n)$ , the design matrix $X$ is known and has full column rank, while both $\beta$ and $\sigma^2$ are unknown. Which statistic is jointly sufficient for $\left(\beta,\sigma^2\right)$ ?
Answer: $\left(X^{\mathsf T}Y,\,Y^{\mathsf T}Y\right)$ When dealing with sufficiency in parametric families, your go-to tool is the Fisher-Neyman Factorization Theorem: a statistic $T(Y)$ is sufficient for $\theta$ if and only if the likelihood factors as $f(Y|\theta) = g(T(Y),\theta)\cdot h(Y)$ . Here, your parameter is the pair $(\beta, \sigma^2)$ , so you need a statistic that captures all information about both. Write out the normal log-likelihood for $Y \sim N(X\beta, \sigma^2 I_n)$ : $\ell(\beta,\sigma^2) = -\frac{n}{2}\log\sigma^2 - \frac{1}{2\sigma^2}(Y - X\beta)^\mathsf{T}(Y - X\beta).$ Expanding the quadratic: $(Y-X\beta)^\mathsf{T}(Y-X\beta) = Y^\mathsf{T}Y - 2\beta^\mathsf{T}X^\mathsf{T}Y + \beta^\mathsf{T}X^\mathsf{T}X\beta.$ Since $X$ is fixed and known, the term $\beta^\mathsf{T}X^\mathsf{T}X\beta$ is a function of $\beta$ alone — no data needed. Therefore the likelihood depends on the data only through $X^\mathsf{T}Y$ and $Y^\mathsf{T}Y$ , confirming B is jointly sufficient for $(\beta, \sigma^2)$ . Choice A replaces $Y^\mathsf{T}Y$ with $\mathbf{1}^\mathsf{T}Y$ (the sample sum), which cannot recover the residual sum of squares needed to estimate $\sigma^2$ . Choice C includes $X^\mathsf{T}X$ , which is a fixed constant containing no sample information — it's not a statistic in the data. Choice D discards magnitude information by keeping only the signs of residuals, losing everything needed to estimate $\sigma^2$ and $\beta$ . Study tip: Always expand the exponent of the normal likelihood fully — sufficiency almost always reveals itself through which data summaries survive after you factor out known constants.
Suppose $X_1,\ldots,X_n$ are independent and each has distribution $N(\theta,\theta)$ , where the second argument is the variance and $\theta>0$ . Which statistic is sufficient for $\theta$ ?
Answer: $\sum_{i=1}^n X_i^2$ When both the mean and variance of a normal distribution depend on the same parameter, your first instinct should be the Fisher-Neyman Factorization Theorem: a statistic $T$ is sufficient for $\theta$ if and only if the joint likelihood factors as $g(T(\mathbf{x}), \theta) \cdot h(\mathbf{x})$ , where $h$ doesn't involve $\theta$ . Write out the joint density of $X_1, \ldots, X_n \sim N(\theta, \theta)$ : $L(\theta) = (2\pi\theta)^{-n/2} \exp\!\left(-\frac{1}{2\theta}\sum_{i=1}^n (X_i - \theta)^2\right)$ Expand the exponent: $\sum(X_i - \theta)^2 = \sum X_i^2 - 2\theta\sum X_i + n\theta^2$ . Substituting back: $L(\theta) = (2\pi\theta)^{-n/2} \exp\!\left(-\frac{\sum X_i^2}{2\theta} + \sum X_i - \frac{n\theta}{2}\right)$ The entire expression depends on the data only through $\sum X_i^2$ , making A the sufficient statistic. Notice that after expanding, $\sum X_i$ appears as a term with coefficient 1 (free of $\theta$ ), so it gets absorbed into a constant — it is not separately needed. Choice B, $\sum X_i$ , is insufficient alone because it ignores the $\sum X_i^2$ term that carries information about $\theta$ . Choice C, the sample variance term $\sum(X_i - \bar{X})^2$ , loses the information contained in $\bar{X}$ and is incomplete. Choice D, $\left(\sum X_i\right)^2$ , is a nonlinear compression that discards even more information than B. Study tip: When $\mu = \sigma^2 = \theta$ , always expand $(X_i - \theta)^2$ fully — the cross-term collapse is what reveals $\sum X_i^2$ as the single sufficient statistic.
Independent event counts satisfy $X_i\sim\operatorname{Poisson}(\lambda a_i)$ for $i=1,\ldots,n$ , where the exposures $a_i>0$ are known and unequal and $\lambda>0$ is unknown. Which statistic is sufficient for $\lambda$ ?
Answer: $\sum_{i=1}^n X_i$ When you see a question about sufficiency, your first instinct should be the Fisher-Neyman Factorization Theorem: a statistic $T(\mathbf{X})$ is sufficient for $\lambda$ if and only if the joint likelihood factors into a part depending on the data only through $T$ and a part free of $\lambda$ . For independent Poisson variables, the joint PMF is: $L(\lambda) = \prod_{i=1}^n \frac{e^{-\lambda a_i}(\lambda a_i)^{x_i}}{x_i!} = e^{-\lambda \sum a_i} \cdot \lambda^{\sum x_i} \cdot \prod_i \frac{a_i^{x_i}}{x_i!}$ Notice the likelihood depends on the data only through $\sum_{i=1}^n X_i$ . The factor $e^{-\lambda \sum a_i} \cdot \lambda^{\sum x_i}$ involves $\lambda$ and $\sum X_i$ , while $\prod a_i^{x_i}/x_i!$ is free of $\lambda$ . By the factorization theorem, A — $\sum_{i=1}^n X_i$ — is sufficient. Choice B, $\sum X_i/a_i$ , is a weighted sum that would arise in moment estimation but doesn't appear naturally in the likelihood factorization. Choice C, $\sum a_i X_i$ , up-weights large-exposure observations — again, no such term emerges from the likelihood. Choice D, $\sum X_i^2/a_i$ , introduces squared counts, which never appear in Poisson likelihoods since the exponent on $\lambda$ is always $\sum x_i$ , linear in the data. A key study tip: for exponential family distributions, the sufficient statistic is always the natural statistic that multiplies the natural parameter in the log-likelihood. For Poisson $(\lambda a_i)$ , that's $\sum x_i$ — the exposures $a_i$ fold into the normalizing constant, not the sufficient statistic itself.
Let $X_1$ and $X_2$ be independent exponential random variables with unknown rate $\lambda$ , and define $T=X_1+X_2$ . Which statement most directly verifies the sufficiency of $T$ using the conditional-distribution characterization?
Answer: Given $T=t$ , the conditional density of $X_1$ is $1/t$ on $(0,t)$ , which is uniform and free of $\lambda$ , so the full sample distribution given $T$ carries no further information about $\lambda$ . Whenever you encounter a question about sufficiency, anchor yourself to the conditional distribution characterization: a statistic $T$ is sufficient for $\lambda$ if and only if the conditional distribution of the data given $T$ is free of $\lambda$ . This is the Fisherian definition made operational. For $X_1, X_2 \sim \text{Exp}(\lambda)$ with $T = X_1 + X_2$ , you can derive the conditional density of $X_1$ given $T = t$ by taking the ratio of the joint density to the marginal of $T$ . The joint density factors as $\lambda^2 e^{-\lambda(x_1 + x_2)}$ , and the marginal of $T$ is the Gamma(2, $\lambda$ ) density $\lambda^2 t e^{-\lambda t}$ . Dividing gives $f(x_1 \mid T=t) = 1/t$ for $x_1 \in (0,t)$ — a uniform distribution with no $\lambda$ in sight. This is precisely the conditional-distribution characterization of sufficiency, making C the correct answer. A is tempting but subtly wrong. While $X_1/T \sim \text{Uniform}(0,1)$ is true and related to sufficiency, the claim that any statistic expressible through $X_1/T$ is also sufficient is false — ancillary statistics like $X_1/T$ actually contain no information about $\lambda$ , not all information. B correctly states the marginal density of $T$ , but showing that the marginal depends on $\lambda$ does not verify sufficiency — it merely confirms $T$ is informative. Sufficiency requires the conditional distribution to be parameter-free. D is a red herring. Covariance measures linear association, not information content. A nonzero $\text{Cov}(X_1, T)$ says nothing rigorous about sufficiency. Your study tip: on sufficiency questions, always ask "does the conditional distribution of the data given the statistic depend on the parameter?" If no, sufficiency is confirmed — that's your direct verification tool.
A time-homogeneous Markov chain on states $\{0,1\}$ is observed for a fixed number of transitions. Its initial state is fixed and known, and all four transition probabilities are unknown subject to each row summing to one. Which statistic is sufficient for the transition matrix?
Answer: The four transition counts $\left(N_{00},N_{01},N_{10},N_{11}\right)$ When you encounter sufficiency questions for Markov chains, your instinct should be to write down the likelihood function and identify exactly which quantities it depends on. That's the core strategy here. For a two-state time-homogeneous Markov chain, the likelihood of an observed path factors as: $L(P) = p_{00}^{N_{00}}\, p_{01}^{N_{01}}\, p_{10}^{N_{10}}\, p_{11}^{N_{11}}$ where $N_{ij}$ is the number of observed transitions from state $i$ to state $j$ . By the Fisher–Neyman factorization theorem, a statistic is sufficient if and only if the likelihood can be written entirely as a function of that statistic (times something free of the parameters). Here, the likelihood depends on the parameter matrix $P$ only through all four counts $(N_{00}, N_{01}, N_{10}, N_{11})$ , making B the sufficient statistic. Choice A gives only the row totals — how many transitions departed from each state. These tell you nothing about how those transitions were distributed between staying and switching, so you lose the information needed to estimate $p_{00}$ versus $p_{01}$ separately. Choice C collapses the counts into a single switch count plus the terminal state; this discards the separate information about transitions out of state 0 versus state 1, which are governed by different rows of $P$ . Choice D aggregates all stays together and all switches together, again conflating the two rows and destroying the row-specific structure that separate parameters $p_{01}$ and $p_{10}$ require. The key study tip: always write the likelihood first. Whatever the likelihood depends on — and nothing more — is your sufficient statistic. If a proposed answer collapses or aggregates counts in ways the likelihood does not, it's insufficient.
Let $X_1,\ldots,X_n$ , where $n\ge 3$ , be independent with density $f_\theta(x)=\mathbf{1}\{\theta\le x\le \theta+1\}$ . Which statistic is sufficient for $\theta$ by the factorization theorem?
Answer: $\left(X_{(1)},X_{(n)}\right)$ When you see a uniform distribution shifted by an unknown parameter $\theta$ , your first instinct should be the Factorization Theorem: write the joint density and identify which statistics "carry" all the $\theta$ -dependence. The joint density of $X_1, \ldots, X_n$ is: $f_\theta(\mathbf{x}) = \prod_{i=1}^n \mathbf{1}\{\theta \le x_i \le \theta+1\} = \mathbf{1}\{x_{(1)} \ge \theta\} \cdot \mathbf{1}\{x_{(n)} \le \theta+1\}$ This factors into a function of $\theta$ and the data only through $x_{(1)}$ (the minimum) and $x_{(n)}$ (the maximum). By the Factorization Theorem, $\left(X_{(1)}, X_{(n)}\right)$ is sufficient for $\theta$ — that's answer A. Now let's see why the others fail. Answer D, using only $X_{(1)}$ , discards the upper-constraint information embedded in $X_{(n)}$ : you need both inequalities to fully capture $\theta$ 's feasible range, so $X_{(1)}$ alone is insufficient. Answer B, the range $X_{(n)} - X_{(1)}$ , is actually an ancillary statistic — its distribution doesn't depend on $\theta$ at all, so it carries zero information about $\theta$ . A statistic that ignores $\theta$ entirely cannot be sufficient. Answer C adds $\sum X_i$ to the range, but since the range is ancillary and $\sum X_i$ provides no additional information beyond what $\left(X_{(1)}, X_{(n)}\right)$ already contains, this is unnecessarily bloated and still grounded in an ancillary component. Study tip: For location-family uniforms $[\theta, \theta+1]$ , always check the joint density's indicator constraints — they immediately point you to the order statistics $X_{(1)}$ and $X_{(n)}$ as the sufficient pair.
Bernoulli trials with success probability $p$ are observed sequentially until the $r$ th success occurs, where $r$ is fixed. The complete ordered sequence is recorded, and $N$ denotes the stopping time. Which scalar statistic captures all likelihood dependence on $p$ ?
Answer: $N$ Whenever you see a question about sequential experiments and sufficient statistics, your instinct should be to write down the likelihood function and ask: which statistic carries all the information about the unknown parameter? For negative binomial sampling — Bernoulli trials run until the $r$ th success — the likelihood of the complete ordered sequence is determined entirely by how many trials were needed. If the $r$ th success occurs on trial $N = n$ , then the sequence contains exactly $r$ successes and $n - r$ failures, and the likelihood is: $L(p) = \binom{n-1}{r-1} p^r (1-p)^{n-r}$ Notice that once you know $n$ (and the fixed $r$ ), you know the entire functional form in $p$ . By the Fisher–Neyman factorization theorem, a statistic $T$ is sufficient if and only if the likelihood factors as $g(T, p) \cdot h(\mathbf{x})$ , where $h$ does not depend on $p$ . Here, $T = N$ plays exactly that role — the binomial coefficient $\binom{n-1}{r-1}$ is absorbed into $h$ , and the remaining factor depends on the data only through $N$ . So D is correct. Choice A (longest run of failures) discards information about total trial count and is not sufficient. Choice B (position of the first success) uses only a small piece of the sequence and ignores everything after trial 1. Choice C (number of success-failure transitions) is a feature of the sequence's pattern, but two sequences with the same transition count can have different $N$ values, yielding different likelihoods. As a study tip: always write out $L(p)$ explicitly and apply the factorization theorem — if the statistic determines every term involving $p$ , it's sufficient.
For an independent sample from the uniform distribution on $[0,\theta]$ , let $M=X_{(n)}$ . It is known that $M$ is sufficient for $\theta>0$ . Which statistic is also sufficient for every sample size solely because of its relationship to $M$ ?
Answer: $M^2$ When a statistic is sufficient, any one-to-one function of it is also sufficient — this is the key principle being tested here. Sufficiency captures all the information about $\theta$ contained in the data, and a bijective transformation preserves that information completely, since you can always recover the original sufficient statistic from its image. For the uniform distribution on $[0, \theta]$ , $M = X_{(n)}$ is sufficient. You should ask: which answer choice is a one-to-one function of $M$ ? Answer D, $M^2$ , fits perfectly. Since $M > 0$ (as a maximum of positive observations), the map $m \mapsto m^2$ is strictly increasing and therefore invertible on the positive reals — knowing $M^2$ is equivalent to knowing $M$ , so $M^2$ inherits sufficiency for every sample size. Now consider why the other choices fail. A, $X_{(n)} - X_{(1)}$ , involves $X_{(1)}$ as well, making it a function of the full order statistics rather than of $M$ alone — it is not determined by $M$ and actually discards information about the scale. B, $\lfloor M \rfloor$ , is a many-to-one function: different values of $M$ (e.g., 2.3 and 2.7) yield the same floor value, so you cannot recover $M$ from it, meaning information is lost. C, $\mathbf{1}\{M > 1\}$ , is even coarser — it collapses all of $M$ 's variation into a binary indicator, clearly not sufficient. Study tip: On sufficiency questions, always check whether the proposed statistic is a bijection of a known sufficient statistic. Many-to-one functions lose information and destroy sufficiency; one-to-one functions preserve it exactly.
Let $X_1,\ldots,X_n$ be independent with density $f_\theta(x)=e^{-(x-\theta)}\mathbf{1}\{x\ge\theta\}$ , where $\theta\in\mathbb{R}$ . Which statistic is minimal sufficient for $\theta$ ?
Answer: $X_{(1)}$ When you encounter a minimal sufficiency question, your first move should be the Factorization/Ratio criterion: two samples share the same sufficient statistic if and only if the ratio of their likelihoods is free of $\theta$ . For this shifted exponential, the joint density is: $L(\theta; \mathbf{x}) = e^{-\sum(x_i - \theta)}\mathbf{1}\{x_{(1)} \ge \theta\} = e^{n\theta}e^{-\sum x_i}\mathbf{1}\{x_{(1)} \ge \theta\}$ The likelihood depends on the data only through $x_{(1)}$ (the minimum) and $\sum x_i$ . But notice the support constraint $\mathbf{1}\{x_{(1)} \ge \theta\}$ is the binding factor — it creates a hard boundary that encodes everything about where $\theta$ can be. Forming the ratio $L(\theta;\mathbf{x})/L(\theta;\mathbf{y})$ , the exponential terms cancel when $\sum x_i = \sum y_i$ , but the ratio is free of $\theta$ if and only if $x_{(1)} = y_{(1)}$ . Therefore ** $X_{(1)}$ alone is minimal sufficient — confirming answer A**. Choice B, $X_{(n)}$ , is the maximum and carries no information about the lower bound $\theta$ ; the support constraint involves $X_{(1)}$ , not $X_{(n)}$ . Choice C, $\sum X_i$ , is sufficient for the scale/rate in a standard exponential but ignores the location shift — it fails to capture the support boundary. Choice D, $X_{(n)} - X_{(1)}$ , is the range, which is ancillary (its distribution is free of $\theta$ ), making it essentially useless for estimation. Study tip: For location-family problems, always check whether the support depends on $\theta$ . When it does, the order statistic defining that boundary almost always drives minimal sufficiency.

Start this quiz