Exponential Families - Statistics Graduate Level

1. Which statement correctly identifies the nonidentifiability in this representation?
   Answer: Two parameter vectors are equivalent when their changes satisfy $\Delta\eta_1+2\Delta\eta_2=0$. When a statistic vector has linearly dependent components, the exponential family representation becomes nonidentifiable — multiple parameter vectors produce the identical distribution. Your job is to find which parameter changes leave the inner product $\eta^{\mathsf T} T(X)$ unchanged for all values of $X$. The key quantity is the linear combination in the exponent: $\eta^{\mathsf T} T(X) = \eta_1 \cdot X + \eta_2 \cdot (2X+1)$. Expanding gives $(\eta_1 + 2\eta_2)X + \eta_2$. Two parameter vectors $\eta$ and $\eta + \Delta\eta$ yield the same distribution if and only if their exponents agree for all $X$, meaning both the coefficient of $X$ and the constant term are unchanged. The coefficient of $X$ changes by $\Delta\eta_1 + 2\Delta\eta_2$, so equivalence requires $\Delta\eta_1 + 2\Delta\eta_2 = 0$. This confirms A is correct. B reverses the coefficients — it would be correct if $T(X) = (2X, X+1)^{\mathsf T}$ or some other arrangement, but it misreads which component multiplies which parameter. C is a classic trap: the two components of $T(X)$ are numerically different for any given $X$, but that's irrelevant — what matters is whether they are linearly independent as functions, and $(X)$ and $(2X+1)$ are not (they span only a one-dimensional function space). D would be the answer only if $T(X)$ had linearly independent components; since it doesn't, a whole line of parameter shifts produces the same distribution. Study tip: Whenever you see an exponential family, immediately check whether the components of $T(X)$ are linearly independent as functions of $X$. If not, set $\Delta\eta^{\mathsf T} T(X) = 0$ and solve — the resulting constraint equation directly reveals the nonidentifiability structure.
2. What is the Fisher information in one observation when the model is parameterized by $m$?
   Answer: $I_m(m)=r/m^2$ When you see Fisher information asked under a reparameterization, your first instinct should be the reparameterization formula: if you change from the natural parameter $\eta$ to a new parameter $m$, the Fisher information transforms as $I_m(m) = I_\eta(\eta)\left(\frac{d\eta}{dm}\right)^2$. Start by finding $I_\eta(\eta)$. In a natural exponential family, the Fisher information in the natural parameter equals the variance, which equals $A''(\eta)$. Differentiating $A(\eta) = -r\log(-\eta)$ twice: $A'(\eta) = -r/\eta$ and $A''(\eta) = r/\eta^2$. So $I_\eta(\eta) = r/\eta^2$. Next, find $d\eta/dm$. From $m = -r/\eta$, solve to get $\eta = -r/m$, so $d\eta/dm = r/m^2$. Applying the chain rule formula: $I_m(m) = \frac{r}{\eta^2}\cdot\left(\frac{r}{m^2}\right)^2$. But substitute $\eta = -r/m$, giving $\eta^2 = r^2/m^2$, so $I_\eta = r/(r^2/m^2) = m^2/r$. Then $I_m(m) = \frac{m^2}{r}\cdot\frac{r^2}{m^4} = \frac{r}{m^2}$. This confirms answer A. Answer B, $m^2/r$, is actually $I_\eta$ expressed in terms of $m$ — it forgets to apply the Jacobian factor. Answer C, $r^2/m^2$, likely arises from squaring $r/m$ without correctly assembling the transformation. Answer D, $1/(rm^2)$, flips the role of $r$ in the numerator and denominator. Study tip: Always track which parameterization you're working in and apply the squared-derivative Jacobian explicitly — confusing $I_\eta$ with $I_m$ is one of the most common errors on reparameterization problems.
3. Which natural parameter and log-partition function produce an equivalent representation using $T^*$?
   Answer: $\eta^*=M^{-\mathsf T}\eta$ and $A^*(\eta^*)=A(\eta)+\eta^{\mathsf T}M^{-1}b$ When reparametrizing an exponential family, your goal is to rewrite $\eta^{\mathsf T}T(x)$ in terms of the new sufficient statistic $T^*(x) = MT(x) + b$, preserving the density exactly. This requires finding $\eta^*$ such that $(\eta^*)^{\mathsf T}T^*(x)$ reproduces the original inner product, then adjusting the log-partition function to absorb any leftover constants. Start by solving: you need $(\eta^*)^{\mathsf T}T^*(x) = \eta^{\mathsf T}T(x)$. Substituting $T^*(x) = MT(x) + b$, expand the left side as $(\eta^*)^{\mathsf T}MT(x) + (\eta^*)^{\mathsf T}b$. For the $T(x)$ terms to match, you need $(\eta^*)^{\mathsf T}M = \eta^{\mathsf T}$, which gives $\eta^* = M^{-\mathsf T}\eta$. This confirms D is correct. The residual constant term $(\eta^*)^{\mathsf T}b = \eta^{\mathsf T}M^{-1}b$ must be absorbed into the new log-partition function: $A^*(\eta^*) = A(\eta) + \eta^{\mathsf T}M^{-1}b$, exactly as in D. A uses $M^{\mathsf T}\eta$ instead of $M^{-\mathsf T}\eta$—a common transpose/inverse mix-up—and gets the sign on the correction term wrong. B applies $M$ directly to $\eta$ without transposing or inverting, ignoring the structure of the inner product algebra entirely. C inverts without transposing ($M^{-1}\eta$ instead of $M^{-\mathsf T}\eta$) and replaces the correction term with the unrelated quantity $b^{\mathsf T}b$, which has no basis in the derivation. A useful rule of thumb: when you see a linear transformation of a sufficient statistic, always derive $\eta^*$ by matching inner products term-by-term, and track every leftover scalar into $A^*$. Confusing transpose with inverse is the most common trap here.
4. For one observation from the uniform model $f(x\mid\theta)=\theta^{-1}I\{0<x<\theta\}$ with $\theta>0$, a researcher proposes treating $-\log\theta$ as a natural parameter. Which assessment is most accurate?
   Answer: The model is not a regular fixed-support exponential family because the support depends on $\theta$. When classifying a statistical model as an exponential family, the most critical requirement is that the support of the distribution must not depend on the parameter. This is the fixed-carrier condition, and it's the first thing you should check before attempting any reparameterization. For the uniform model $f(x\mid\theta) = \theta^{-1}I\{0 < x < \theta\}$, the support $\{x : 0 < x < \theta\}$ moves with $\theta$. This immediately disqualifies the model from being any form of regular exponential family, because the exponential family form $h(x)\exp\{\eta(\theta)T(x) - A(\theta)\}$ requires $h(x)$ to be free of $\theta$—but the indicator function $I\{0 < x < \theta\}$ cannot be factored into a $\theta$-free piece times a $\theta$-only piece. That's why B is correct. Answer A is tempting because $\log f = -\log\theta$ does appear in the log density, making it look like the log density has the right shape. But this reasoning ignores the indicator function entirely—the support problem persists regardless of how you write the density's main term. Answer D proposes the reparameterization $\eta = 1/\theta$. While valid reparameterizations can simplify exponential family representations, they cannot fix a varying-support problem. The support still depends on $\theta$, and no algebraic trick removes that dependence. Answer C misuses the term "curved exponential family." Curved exponential families are subfamilies of regular exponential families with constraints on the natural parameter space—they don't arise from indicator-function complications. Study tip: Before attempting any exponential family classification, always check the support first. If the support depends on $\theta$, the conversation ends there—no reparameterization can rescue it.
5. Which equation characterizes the posterior mode $\widehat\eta$?
   Answer: $A'(\widehat\eta)=(\xi+\sum_iT(X_i))/(\nu+n)$ When working with exponential families and conjugate priors, your instinct should be to write out the log-posterior and differentiate. The posterior combines the conjugate prior kernel with the likelihood, giving: $\log \pi(\eta \mid \mathbf{x}) \propto \xi\eta - \nu A(\eta) + \eta\sum_i T(x_i) - nA(\eta) = \left(\xi + \sum_i T(x_i)\right)\eta - (\nu + n)A(\eta)$ Taking the derivative with respect to $\eta$ and setting it to zero at the mode $\widehat{\eta}$: $\left(\xi + \sum_i T(x_i)\right) - (\nu + n)A'(\widehat{\eta}) = 0$ Solving directly yields $A'(\widehat{\eta}) = \dfrac{\xi + \sum_i T(x_i)}{\nu + n}$, which is D. This makes intuitive sense: in an exponential family, $A'(\eta) = \mathbb{E}_\eta[T(X)]$, so the posterior mode sets the mean-value parameter equal to a precision-weighted average of the prior pseudo-observations and data sufficient statistics. A is wrong because it involves $A''(\widehat{\eta})$, the second derivative (which governs curvature, not the mode condition). Confusing $A'$ with $A''$ is a common careless error. B flips the roles of $\xi$ and $\nu$: $\xi$ counts pseudo-observations on the sufficient statistic side, while $\nu$ acts as a pseudo-sample-size scaling $A(\eta)$, so swapping them mixes up the prior's two hyperparameters. C inverts the entire fraction, which would arise from accidentally dividing by the wrong term. As a study tip: always distinguish the two roles of the conjugate prior hyperparameters — one multiplies $\eta$ (sufficient statistic side) and one multiplies $A(\eta)$ (log-normalizer side). Keeping that structure clear prevents both B-type and C-type errors.
6. Using $\eta=\log\lambda$, which pair gives the canonical statistic and log-partition function for the joint model, up to terms independent of $\eta$?
   Answer: $T=\sum_iY_i$ and $A(\eta)=e^\eta\sum_ie_i$ When you encounter exponential family questions, your job is to factor the joint log-likelihood into the canonical form $\ell(\eta) = \eta \cdot T(\mathbf{Y}) - A(\eta) + C(\mathbf{Y})$, identifying the sufficient statistic $T$ and log-partition function $A(\eta)$ by inspection. Start with the joint log-likelihood. Since the $Y_i$ are independent Poisson$(e_i\lambda)$, the log-likelihood is: $\ell(\lambda) = \sum_i \left[ Y_i \log(e_i\lambda) - e_i\lambda - \log(Y_i!) \right]$ Substituting $\lambda = e^\eta$: $\ell(\eta) = \sum_i Y_i(\log e_i + \eta) - e^\eta\sum_i e_i - \sum_i\log(Y_i!)$ Collecting terms involving $\eta$: $\ell(\eta) = \eta\underbrace{\sum_i Y_i}_{T} - \underbrace{e^\eta \sum_i e_i}_{A(\eta)} + \text{(terms free of } \eta\text{)}$ This confirms Answer A: $T = \sum_i Y_i$ and $A(\eta) = e^\eta \sum_i e_i$. Answer B uses $T = \sum_i e_i Y_i$, which would arise if $e_i$ multiplied $Y_i$ inside the canonical term — but the log-likelihood shows $\eta$ pairs with $\sum Y_i$, not $\sum e_i Y_i$. The exposures belong in the log-partition function, not the sufficient statistic. Answer C incorrectly multiplies $n$ and $\sum e_i$ together in $A(\eta)$, confusing the structural role of the exposures. Answer D divides $Y_i$ by $e_i$, which has no basis in the log-likelihood factorization. A reliable strategy: always write out the full joint log-likelihood first, then isolate every term containing your natural parameter. Whatever multiplies $\eta$ is your canonical statistic $T$; whatever you subtract is $A(\eta)$.
7. Suppose the model is restricted by $p_1p_3=p_2^2$. Which description of the restricted model is correct?
   Answer: It is a one-parameter full exponential family satisfying $\eta_1=2\eta_2$. When a multinomial model is restricted, your first move should always be to translate the constraint into the natural parameter space — that's where exponential family structure becomes transparent. The natural parameters here are $\eta_1 = \log(p_1/p_3)$ and $\eta_2 = \log(p_2/p_3)$. Now apply the constraint $p_1 p_3 = p_2^2$. Taking logarithms of both sides gives $\log p_1 + \log p_3 = 2\log p_2$. Rewriting in terms of the natural parameters: $\log(p_1/p_3) = 2\log(p_2/p_3)$, which is exactly $\eta_1 = 2\eta_2$. This is a linear constraint on the natural parameters, so the restricted model is a one-parameter exponential family obtained by substituting $\eta_1 = 2\eta_2$ and letting $\eta_2 = \theta$ vary freely. Because the natural parameter space is an open interval and the parameterization is linear, this is a full exponential family — confirming answer A is correct. Answer B claims $\eta_1 \eta_2 = 2$, which would be a nonlinear (curved) constraint in the natural parameter space, making it a curved exponential family. But the log transformation turns the multiplicative constraint into an additive one, so this is the wrong algebraic form entirely. Answer C gets the direction backwards: it says $2\eta_1 = \eta_2$, but the derivation clearly gives $\eta_1 = 2\eta_2$, not its reciprocal. Answer D is a conceptual trap — nonlinearity in the probabilities does not disqualify a model from being an exponential family; what matters is linearity in the natural parameters after the log transformation. Your takeaway: always convert restrictions to log-odds (natural parameter) form before classifying the family. A constraint that looks curved in probability space can be perfectly linear — and thus "full" — in natural parameter space.
8. A normal model has mean $\mu>0$ and variance constrained by $\sigma^2=\mu^2$. Viewed as a subfamily of the two-parameter normal exponential family with natural parameters $\eta_1=\mu/\sigma^2$ and $\eta_2=-1/(2\sigma^2)$, which statement is correct?
   Answer: It is a curved one-parameter family with $\eta_2=-\eta_1^2/2$ and natural parameter space $\eta_1>0$. When a question asks you to embed a constrained subfamily into an exponential family, your first task is to express the natural parameters purely in terms of the free parameter, then classify the resulting curve in natural parameter space. Here, the two-parameter normal exponential family has natural parameters $\eta_1 = \mu/\sigma^2$ and $\eta_2 = -1/(2\sigma^2)$. The constraint $\sigma^2 = \mu^2$ (with $\mu > 0$) means $\sigma = \mu$. Substituting: $\eta_1 = \mu/\mu^2 = 1/\mu$ and $\eta_2 = -1/(2\mu^2)$. Since $\eta_1 = 1/\mu$, we get $\mu = 1/\eta_1$, so $\eta_2 = -1/(2(1/\eta_1)^2) = -\eta_1^2/2$. Because $\mu > 0$, we have $\eta_1 > 0$. This relationship $\eta_2 = -\eta_1^2/2$ traces a curved (nonlinear) path through the two-dimensional natural parameter space — the hallmark of a curved exponential family. This confirms B is correct. A is wrong because it claims $\eta_2 = -\eta_1/2$, a linear relationship, which would make this a full (not curved) subfamily. The algebra simply doesn't support linearity here. C has the right functional form $\eta_2 = -1/(2\eta_1^2)$, but this expression equals $-\mu^2/2$, not $-\eta_1^2/2$. It also incorrectly states $\eta_1 < 0$, when $\mu > 0$ forces $\eta_1 > 0$. D is a conceptual error. Variation of both natural parameters with a single underlying parameter doesn't make a family "full" — fullness requires the natural parameter space to contain an open set in $\mathbb{R}^2$, which a one-dimensional curve never does. Remember: curved = nonlinear constraint in natural parameter space; full = open set in natural parameter space. Always substitute and simplify before classifying.
9. Viewed inside the full exponential family of independent Bernoulli distributions, how should this probit regression model generally be classified?
   Answer: It is a curved family with coordinates $\eta_i=\logit\{\Phi(x_i^{\mathsf T}\beta)\}$. When working with exponential families, a crucial distinction is between the full family (where the natural parameter vector ranges freely over an open set with dimension equal to the sufficient statistic) and a curved exponential family (where the parameter of interest lives on a lower-dimensional submanifold). This question asks you to place probit regression within that taxonomy. The full Bernoulli exponential family has natural parameters $\eta_i = \log\frac{p_i}{1-p_i}$ for each observation, living freely in $\mathbb{R}^n$. In probit regression, however, these $n$ natural parameters are constrained: $\eta_i = \text{logit}\{\Phi(x_i^\mathsf{T}\beta)\}$, which traces out a $p$-dimensional curved submanifold inside $\mathbb{R}^n$ as $\beta$ varies. Because the model is parameterized by $\beta \in \mathbb{R}^p$ with $p < n$, it sits on a curved, lower-dimensional surface within the full family — making C the correct classification. Answer A is wrong because membership in an exponential family doesn't require the canonical link. Probit regression is still a valid statistical model; the question is about its position within the full Bernoulli family, not whether it "qualifies" as exponential. Answer B fails because $X\beta$ is not the natural parameter vector — $\text{logit}\{\Phi(x_i^\mathsf{T}\beta)\}$ is, and the model does not fill the full $n$-dimensional natural parameter space. Answer D confuses the range of individual components with the dimension of the parameter manifold; even though each $\eta_i$ can range over all reals, the $n$ components are jointly constrained to a $p$-dimensional surface. A useful rule of thumb: whenever a model imposes $p < n$ constraints on an $n$-dimensional natural parameter, think curved exponential family.
10. What is $\operatorname{Cov}(\log X,X)$ under this model?
    Answer: $1/\beta$ Whenever you see a question about covariances of sufficient statistics in an exponential family, reach for the information matrix identity: the covariance matrix of the canonical sufficient statistics equals the matrix of second-order partial derivatives of the log-partition function $A(\boldsymbol{\eta})$ with respect to the natural parameters. For the gamma family with natural parameters $\eta_1 = \alpha - 1$ and $\eta_2 = -\beta$, the log-partition function is: $A(\eta_1, \eta_2) = \log\Gamma(\eta_1+1) - (\eta_1+1)\log(-\eta_2)$ The cross-covariance is given by the mixed partial derivative: $\operatorname{Cov}(T_1, T_2) = \frac{\partial^2 A}{\partial \eta_1 \,\partial \eta_2}$ Taking $\partial A/\partial \eta_1 = \psi(\eta_1+1) - \log(-\eta_2)$, where $\psi$ is the digamma function, and then differentiating with respect to $\eta_2$: $\frac{\partial^2 A}{\partial \eta_1\,\partial \eta_2} = -\frac{1}{\eta_2} = \frac{1}{\beta}$ So $\operatorname{Cov}(\log X, X) = 1/\beta$, confirming C. Choice A ($\alpha/\beta$) corresponds to $\operatorname{Var}(X) = \alpha/\beta^2$ confused with the covariance — it mixes up variance of $X$ and scales incorrectly. Choice B ($-1/\beta$) is a sign error: students often forget that $\eta_2 = -\beta < 0$, so $-1/\eta_2 = +1/\beta$, not $-1/\beta$. Choice D ($1/(\alpha\beta)$) has no grounding in the derivatives and likely reflects a dimensional-analysis guess. Study tip: Memorize that in any exponential family, $\operatorname{Cov}(T_i, T_j) = \partial^2 A/\partial\eta_i\,\partial\eta_j$. This single identity replaces lengthy moment calculations and is a recurring tool on graduate-level theory exams.

Start this quiz