M-Estimators - Statistics Graduate Level

In the scalar regression model $Y=X\beta_0+\varepsilon$ , consider the M-estimating equation $\sum_{i=1}^n X_i\psi(Y_i-X_i\beta)=0$ for a nonlinear score $\psi$ . Which condition most directly guarantees that the population estimating equation is unbiased at $\beta_0$ for arbitrary distributions of $X$ ?
Answer: $E\{\psi(\varepsilon)\mid X\}=0$ almost surely, together with finite required moments. When analyzing M-estimating equations, your goal is to ensure the population version of the equation equals zero at the true parameter $\beta_0$ . The population estimating equation is $E\{X\psi(Y - X\beta_0)\} = E\{X\psi(\varepsilon)\} = 0$ , and you need this to hold for arbitrary distributions of $X$ . The cleanest way to verify this uses the tower property (iterated expectations): $E\{X\psi(\varepsilon)\} = E\{E(X\psi(\varepsilon)\mid X)\} = E\{X \cdot E(\psi(\varepsilon)\mid X)\}$ . If $E\{\psi(\varepsilon)\mid X\} = 0$ almost surely, then each inner expectation is zero regardless of the distribution of $X$ , making the full expectation zero. This is exactly option C, and it works universally because the condition eliminates the inner term before $X$ 's distribution even matters. Option A gives you only the marginal condition $E\{\psi(\varepsilon)\} = 0$ , which is insufficient when $\varepsilon$ and $X$ are dependent. You can construct cases where $E\{\psi(\varepsilon)\} = 0$ but $E\{X\psi(\varepsilon)\} \neq 0$ due to covariance between $X$ and $\psi(\varepsilon)$ . Option B requiring $E(X) = 0$ is a condition on $X$ alone and doesn't control $E\{X\psi(\varepsilon)\}$ — a zero-mean $X$ can still covary with $\psi(\varepsilon)$ . Option D is the classical unbiasedness condition for the linear score (OLS), where $\psi(\varepsilon) = \varepsilon$ . For a nonlinear $\psi$ , knowing $E(\varepsilon \mid X) = 0$ tells you nothing about $E\{\psi(\varepsilon) \mid X\}$ in general. Study tip: Whenever a score function is nonlinear, immediately shift your thinking from marginal to conditional moment conditions — the conditional version is almost always the operative requirement, and the tower property is your primary tool for verifying it.
Consider the criterion $Q_n(\theta)=\theta^4/4-\theta^2/2-\theta\overline X_n$ , where $E(X)=0$ and $\overline X_n\to0$ in probability. Its estimating equation is $\theta^3-\theta-\overline X_n=0$ . Which conclusion about consistency is most accurate?
Answer: A global minimizer need not converge to one unique value because the population criterion has equal minima at $-1$ and $1$ . When analyzing M-estimators or extremum estimators, you need to distinguish between the sample criterion and its population (limiting) counterpart. Consistency arguments depend critically on the structure of the limiting criterion, not just the sample version. Here, as $\overline{X}_n \to 0$ , the sample criterion $Q_n(\theta) = \theta^4/4 - \theta^2/2 - \theta\overline{X}_n$ converges to the population criterion $Q(\theta) = \theta^4/4 - \theta^2/2$ . Taking the derivative of this limit gives $\theta^3 - \theta = \theta(\theta-1)(\theta+1) = 0$ , which has roots at $\theta \in \{-1, 0, 1\}$ . Checking second derivatives, $Q''(\theta) = 3\theta^2 - 1$ , reveals that $\theta = \pm 1$ are local (and in fact global) minima with equal criterion values $Q(\pm 1) = -1/4$ , while $\theta = 0$ is a local maximum. Because the limiting criterion achieves its minimum at two distinct points, a global minimizer of $Q_n$ can converge to either $-1$ or $1$ depending on the realized path of $\overline{X}_n$ , making the global minimizer inconsistent for any single value. Answer A correctly captures this non-uniqueness. Answer B is wrong because $\theta = 0$ solves the limiting estimating equation but is a maximizer, not a minimizer — a critical error in reasoning. Answer C is wrong because $\overline{X}_n \to 0$ does not collapse the two minima into one; the symmetry-breaking is insufficient when both minima are exactly equal in the limit. Answer D is wrong because multiple roots do not imply divergence; they imply non-uniqueness of the limit point. As a study tip: always verify whether candidate roots of the limiting estimating equation are minima or maxima, and check whether the minimum is uniquely attained — both conditions are necessary for a consistent global minimizer.
For the estimating equation $U(\theta)=\sum_{i=1}^n(X_i-\theta)^3=0$ , suppose that at a starting value $\theta^{(0)}$ the residuals satisfy $\sum_i(X_i-\theta^{(0)})^3=24$ and $\sum_i(X_i-\theta^{(0)})^2=12$ . What is the first Newton–Raphson iterate?
Answer: $\theta^{(1)}=\theta^{(0)}+\tfrac{2}{3}$ When you encounter a question about Newton–Raphson iteration, your first instinct should be to write down the update formula precisely: $\theta^{(1)} = \theta^{(0)} - \frac{U(\theta^{(0)})}{U'(\theta^{(0)})}$ , where $U'(\theta)$ is the derivative of the estimating equation with respect to $\theta$ . Here, $U(\theta) = \sum_i (X_i - \theta)^3$ . Differentiating with respect to $\theta$ gives $U'(\theta) = -3\sum_i (X_i - \theta)^2$ . At $\theta^{(0)}$ , you're told $U(\theta^{(0)}) = 24$ and $\sum_i (X_i - \theta^{(0)})^2 = 12$ , so $U'(\theta^{(0)}) = -3(12) = -36$ . Plugging in: $\theta^{(1)} = \theta^{(0)} - \frac{24}{-36} = \theta^{(0)} + \frac{2}{3}$ , confirming answer D. Each wrong answer reflects a specific error. A results from forgetting the factor of $-3$ in the derivative entirely and mishandling the sign, producing $-2$ . B drops the negative sign on $U'$ but keeps the magnitude, yielding $-\tfrac{2}{3}$ — a sign error that's easy to make if you forget the chain rule contributes a $-1$ when differentiating $(X_i - \theta)$ . C uses $+2$ , which could come from incorrectly computing the derivative as $-3(12) = -36$ but then dividing $24$ by $12$ alone (ignoring the 3), giving the wrong magnitude with the right sign direction. As a study tip, always differentiate $U(\theta)$ carefully before substituting — the negative from the chain rule on $(X_i - \theta)$ is the most common source of sign errors in Newton–Raphson problems on graduate exams.
An estimator solves $\sum_{i=1}^n\psi(X_i,\theta)=0$ . A researcher replaces this equation by $\sum_{i=1}^n c(\theta)\psi(X_i,\theta)=0$ , where $c(\theta)$ is continuously differentiable and nonzero near the population root $\theta_0$ . What is the effect of this replacement?
Answer: It preserves the local roots and also preserves the correctly computed sandwich asymptotic variance. When analyzing M-estimators, the key question is: what determines the local roots and the asymptotic variance? Both depend on the estimating equation only through its zero-crossings and its local behavior near $\theta_0$ . Because $c(\theta)$ is nonzero near $\theta_0$ , the equation $\sum c(\theta)\psi(X_i,\theta)=0$ has exactly the same local roots as $\sum\psi(X_i,\theta)=0$ . Multiplying by a nonzero scalar function cannot create or destroy zeros. Now consider the sandwich variance. For an M-estimator the asymptotic variance is $V = A^{-1}BA^{-T}$ , where $A = E[\partial_\theta \psi]$ and $B = E[\psi\psi^T]$ . When you replace $\psi$ by $c(\theta)\psi$ , both $A$ and $B$ are scaled: the new "bread" becomes $c(\theta_0)A$ and the new "meat" becomes $c(\theta_0)^2 B$ . The sandwich then gives $(c(\theta_0)A)^{-1}(c(\theta_0)^2 B)(c(\theta_0)A)^{-T} = c(\theta_0)^{-1}c(\theta_0)^2 c(\theta_0)^{-1}A^{-1}BA^{-T} = V$ . The $c(\theta_0)$ factors cancel exactly, so the correctly computed sandwich variance is unchanged. This confirms A is correct. Choice B is the classic trap: it forgets that $c(\theta_0)$ appears in both the bread and the meat, so the cancellation is exact. Choice C is wrong because multiplying by a parameter-dependent but nonzero function never changes the roots — the zeros of a product are only where factors are zero. Choice D conflates sign-changes with variance changes; even if $c(\theta_0)<0$ , the roots and variance are preserved (the sign of the estimating equation is irrelevant to its zeros). Study tip: Whenever a question asks about rescaling an estimating equation, immediately check whether the scaling factor cancels in both numerator and denominator of the sandwich — it almost always does if the factor is evaluated at the same $\theta_0$ .
Let $P(X=0)=0.3$ , $P(X=2)=0.5$ , and $P(X=5)=0.2$ . An estimator minimizes the empirical quantile loss $\sum_i\rho_{0.6}(X_i-\theta)$ , where $\rho_\tau(u)=u\{\tau-I(u<0)\}$ . Which statement correctly describes its population target?
Answer: The target is $2$ even though the ordinary derivative-based score need not equal zero there. Whenever you see a question involving quantile loss minimization, your first instinct should be to identify the population quantile being targeted — specifically, the $\tau$ -th quantile, which minimizes $E[\rho_\tau(X - \theta)]$ . For $\tau = 0.6$ , the population minimizer is the 0.6-quantile, defined as $Q(0.6) = \inf\{\theta : F(\theta) \geq 0.6\}$ . Computing the CDF: $F(0) = 0.3$ , $F(2) = 0.8$ , $F(5) = 1.0$ . Since $F(2) = 0.8 \geq 0.6$ and $F(0) = 0.3 < 0.6$ , the infimum of values where $F \geq 0.6$ is exactly $\theta = 2$ . So B is correct — the population target is $2$ . The subtle but important point B makes is that the "score equation" $E[0.6 - I(X < \theta)] = 0$ need not hold at a discrete mass point. At $\theta = 2$ : $E[0.6 - I(X < 2)] = 0.6 - P(X < 2) = 0.6 - 0.3 = 0.3 \neq 0$ . The subgradient at $\theta = 2$ contains zero, which is the correct optimality condition for non-smooth losses — not the ordinary derivative. A is wrong because $\theta = 0$ is the largest value below cumulative probability 0.6, but that makes it the 0.3-quantile, not the 0.6-quantile. C is wrong in its reasoning: the quantile is the point where $F$ first reaches 0.6, which is $2$ , not $5$ . D is wrong because the absence of an exact zero subgradient doesn't mean no minimizer exists — subgradient optimality is the right criterion here. Your key takeaway: for discrete distributions, the quantile minimizes the check loss via a subgradient condition, not a classical score equation. Always use $\inf\{F(\theta) \geq \tau\}$ , not a derivative argument.
Let the distribution of $X$ be continuous, with finite mean $\mu$ and unique median $m$ , where $\mu\ne m$ . For a fixed constant $\lambda>0$ , define $\widehat\theta_n$ to minimize $n^{-1}\sum_{i=1}^n\{(X_i-\theta)^2+\lambda|X_i-\theta|\}$ . Under standard uniform convergence and uniqueness conditions, to what population value does $\widehat\theta_n$ converge?
Answer: It converges to the unique solution of $2(\theta-\mu)+\lambda E\{\operatorname{sign}(\theta-X)\}=0$ , which reflects contributions from both the quadratic and absolute-deviation components. When you encounter a loss function that combines multiple terms, your first instinct should be to differentiate the expected loss with respect to $\theta$ and set it to zero — each term contributes its own first-order condition, and neither can simply be ignored. Here, the population objective is $Q(\theta) = E\{(X-\theta)^2 + \lambda|X-\theta|\}$ . Differentiating with respect to $\theta$ : the quadratic term gives $\frac{d}{d\theta}E(X-\theta)^2 = -2E(X-\theta) = 2(\theta - \mu)$ , and the absolute-value term gives $\frac{d}{d\theta}E(\lambda|X-\theta|) = \lambda E\{\text{sign}(\theta - X)\}$ , since $\frac{d}{d\theta}|X - \theta| = \text{sign}(\theta - X)$ . Setting the sum to zero yields $2(\theta - \mu) + \lambda E\{\text{sign}(\theta - X)\} = 0$ , which is exactly what C states. By the law of large numbers and standard M-estimation theory, $\widehat\theta_n$ converges to the unique root of this equation. A is wrong because it claims the quadratic term dominates and discards the absolute-deviation contribution entirely — both terms remain present in the first-order condition for any fixed $\lambda > 0$ . B makes the symmetric error: it discards the quadratic term, which would only be appropriate if the loss were pure LAD (least absolute deviations), not a combined criterion. D has the sign wrong. Differentiating $|X - \theta|$ with respect to $\theta$ yields $+\text{sign}(\theta - X)$ , not $-\text{sign}(\theta - X)$ . The chain rule introduces a factor of $-1$ from $\frac{\partial(X-\theta)}{\partial\theta}$ , but that negative is already absorbed into $\text{sign}(\theta - X)$ rather than $\text{sign}(X - \theta)$ . Study tip: For any combined loss, always differentiate term by term and keep every contribution — "dominance" arguments only apply in limiting regimes (e.g., $\lambda \to 0$ or $\lambda \to \infty$ ), not for fixed $\lambda$ .
For a location M-estimator defined by $\sum_i\psi(X_i-\theta)=0$ , suppose $A=E\{\psi'(X-\theta_0)\}$ exists and is finite and nonzero. Which statement about its influence function is correct?
Answer: The influence function is bounded whenever $\psi$ is bounded; the nonzero sensitivity constant $A$ only rescales it without affecting boundedness. When analyzing influence functions for M-estimators, your central focus should be the explicit formula connecting the influence function to the defining equation. For a location M-estimator solving $\sum_i \psi(X_i - \theta) = 0$ , the influence function at a point $x$ is derived via the implicit function theorem and equals: $\text{IF}(x) = \frac{\psi(x - \theta_0)}{A}, \quad A = E\{\psi'(X - \theta_0)\}$ This formula makes the key insight immediate: the influence function is simply $\psi$ rescaled by the constant $A$ . Since $A$ is assumed finite and nonzero, it acts purely as a scaling factor. Therefore, **the influence function is bounded if and only if $\psi$ itself is bounded** — confirming that B is correct. The constant $A$ rescales the range of $\psi$ but cannot introduce or remove boundedness. Choice A is wrong because convexity and symmetry of the loss $\rho$ do not guarantee that $\psi = \rho'$ is bounded. For example, the squared loss is convex and symmetric, yet its score $\psi(x) = x$ is unbounded. Choice C confuses two distinct properties. Fisher consistency means $E_{\theta_0}[\psi(X - \theta_0)] = 0$ , which is a centering condition ensuring the estimating equation is unbiased. It says nothing about whether $\psi$ takes large values at outlying points — boundedness of $\psi$ is a separate requirement. Choice D is incorrect because strict monotonicity and global differentiability of $\psi$ are sufficient for uniqueness and regularity of the estimator, not for boundedness of the influence function. A strictly increasing $\psi(x) = x$ yields an unbounded influence function. A useful study tip: whenever you see an influence function question for M-estimators, write down $\text{IF}(x) = \psi(x-\theta_0)/A$ immediately — then every robustness property of the influence function reduces directly to a property of $\psi$ .
For observations $-4,0,1,9$ , a location M-estimator minimizes $\sum_{i=1}^4 \rho(X_i-\theta)$ , where $\rho$ is the Huber loss with threshold $2$ . Thus its score is $\psi(u)=\max\{-2,\min(u,2)\}$ . Which value solves the corresponding estimating equation?
Answer: $\widehat\theta=\tfrac{1}{2}$ When working with M-estimators, the key insight is that you don't minimize the loss directly — you solve the estimating equation $\sum_{i=1}^4 \psi(X_i - \theta) = 0$ , where $\psi$ is the derivative of $\rho$ . For Huber loss with threshold $c=2$ , this means $\psi(u) = \max\{-2, \min(u, 2)\}$ : residuals within $[-2, 2]$ contribute their actual value, while residuals beyond that are winsorized to $\pm 2$ . To find the solution, evaluate $\sum \psi(X_i - \theta)$ for each candidate. Try $\theta = \tfrac{1}{2}$ (answer B): the residuals are $-4 - 0.5 = -4.5$ , $0 - 0.5 = -0.5$ , $1 - 0.5 = 0.5$ , $9 - 0.5 = 8.5$ . Applying $\psi$ : the extreme residuals $-4.5$ and $8.5$ get clipped to $-2$ and $2$ , while the middle residuals pass through. The sum is $(-2) + (-0.5) + (0.5) + (2) = 0$ . ✓ This confirms B is correct. For A ( $\theta=0$ ): residuals are $-4, 0, 1, 9$ , giving $\psi$ -values $-2, 0, 1, 2$ , which sum to $1 \neq 0$ . For C ( $\theta=1$ ): residuals $-5, -1, 0, 8$ yield $-2, -1, 0, 2$ , summing to $-1 \neq 0$ . For D ( $\theta=\tfrac{3}{2}$ ): residuals $-5.5, -1.5, -0.5, 7.5$ yield $-2, -1.5, -0.5, 2$ , summing to $-2 \neq 0$ . Your strategy: always plug candidates into the estimating equation rather than trying to solve algebraically. Huber M-estimators won't have closed-form solutions in general, so checking the score equation directly is the reliable approach.
Suppose $X_1,\ldots,X_n$ are independent observations from a normal distribution with mean $\theta_0$ and variance $1$ . An M-estimator is the consistent root near $\theta_0$ of $\sum_{i=1}^n(X_i-\theta)^3=0$ . What is the asymptotic variance of $\sqrt n(\widehat\theta_n-\theta_0)$ ?
Answer: $\tfrac{5}{3}$ When you see an M-estimator question, your default tool is the sandwich variance formula: for the estimating equation $\sum \psi(X_i, \theta) = 0$ , the asymptotic variance of $\sqrt{n}(\hat{\theta}_n - \theta_0)$ is $\frac{E[\psi^2]}{(E[\psi'])^2}$ , where $\psi' = \partial\psi/\partial\theta$ . Here, $\psi(X,\theta) = (X-\theta)^3$ , so $\psi'(X,\theta) = -3(X-\theta)^2$ . Evaluated at $\theta_0$ , with $Z = X - \theta_0 \sim N(0,1)$ : Denominator: $E[\psi'] = -3E[Z^2] = -3(1) = -3$ , so $(E[\psi'])^2 = 9$ . Numerator: $E[\psi^2] = E[Z^6]$ . For a standard normal, $E[Z^{2k}] = (2k-1)!!$ , so $E[Z^6] = 5!! = 15$ . The asymptotic variance is therefore $\frac{15}{9} = \frac{5}{3}$ , confirming answer C. Now for the distractors. A ( $\tfrac{1}{3}$ ) likely comes from inverting only the denominator term $E[\psi'] = -3$ without squaring it and ignoring the numerator entirely — a misremembering of the formula. B ( $\tfrac{3}{5}$ ) is the reciprocal of the correct answer; students who flip numerator and denominator in the sandwich formula land here. D ( $15$ ) forgets to divide by $(E[\psi'])^2 = 9$ at all — it's just the raw sixth moment. Study tip: Always memorize the standard normal moments — $E[Z^2]=1$ , $E[Z^4]=3$ , $E[Z^6]=15$ — and write the sandwich formula clearly before plugging in. Swapping numerator and denominator is the most common computational error on M-estimator problems.
Suppose an M-estimator satisfies $\sqrt n(\widehat\theta_n-\theta_0)\mathrel{\Rightarrow}N(0,V)$ , where $\theta_0>0$ . The parameter is reexpressed as $\eta=\theta^2$ , and the estimator is $\widehat\eta_n=\widehat\theta_n^2$ . What is the limiting distribution of $\sqrt n(\widehat\eta_n-\eta_0)$ ?
Answer: $N(0,4\theta_0^2V)$ , because the transformed variance uses the squared derivative. When you see a question involving a transformation of an asymptotically normal estimator, your first instinct should be the Delta Method: if $\sqrt{n}(\hat{\theta}_n - \theta_0) \Rightarrow N(0, V)$ and $g$ is differentiable at $\theta_0$ , then $\sqrt{n}(g(\hat{\theta}_n) - g(\theta_0)) \Rightarrow N(0, [g'(\theta_0)]^2 V)$ . Here, $g(\theta) = \theta^2$ , so $g'(\theta) = 2\theta$ , giving $g'(\theta_0) = 2\theta_0$ . The asymptotic variance of the transformed estimator is $(2\theta_0)^2 \cdot V = 4\theta_0^2 V$ , confirming that D is correct: $\sqrt{n}(\hat{\eta}_n - \eta_0) \Rightarrow N(0, 4\theta_0^2 V)$ . A is wrong because reparameterization absolutely changes the variance — the estimating-equation root is invariant (that's the invariance property of M-estimators), but the asymptotic distribution of the transformed estimator still picks up the Jacobian factor from the Delta Method. B confuses squaring the derivative with just multiplying by it. The variance transformation uses $[g'(\theta_0)]^2$ , not $g'(\theta_0)$ alone. Multiplying by $2\theta_0$ rather than $4\theta_0^2$ is a common algebraic slip. C arrives at $\theta_0^4 V$ , which would correspond to $g'(\theta_0) = \theta_0^2$ — the derivative of $\theta^3/3$ , not $\theta^2$ . This reflects a failure to actually differentiate $g$ and instead substituting the function value itself. Strategy tip: Whenever a transformation is involved, immediately write down $g(\theta)$ , compute $g'(\theta_0)$ , and apply $[g'(\theta_0)]^2 V$ . The squaring of the derivative is the step most students miss under pressure.

Start this quiz