Identifiability & Parameterization — Identifiability and parameterization issues (conceptual) - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The question of whether observed data can uniquely determine the parameters of a statistical model is as old as the discipline of mathematical statistics itself. Early pioneers in probability theory and estimation recognized that certain models admit multiple parameter configurations that produce identical observable distributions, rendering point estimation fundamentally ambiguous. This problem, known as non-identifiability, has surfaced repeatedly across the history of statistics — from the earliest mixture models to modern latent variable analyses. Without identifiability, the machinery of maximum likelihood estimation, Bayesian inference, and hypothesis testing can break down in subtle and dangerous ways, producing apparently precise answers that are in fact artifacts of an ill-posed model.

1894 • Pearson's Mixture Problem

Karl Pearson fitted a mixture of two normal distributions to crab morphometric data, discovering that different parameter combinations could fit the data nearly equally well — one of the earliest documented encounters with non-identifiability in mixture models.

1950s • Koopmans & Econometric Identification

Tjalling Koopmans and the Cowles Commission formalized the concept of identification in simultaneous equation models, establishing rank and order conditions that became foundational in econometrics and structural modeling.

1970s • Rothenberg's Formal Framework

Thomas Rothenberg provided rigorous mathematical conditions for local and global identifiability using the Fisher information matrix, linking the rank of the information matrix to the ability to distinguish nearby parameter values.

1990s–2000s • Latent Variable & Bayesian Resurgence

The explosion of latent variable models (factor analysis, structural equation models, topic models) and MCMC-based Bayesian inference brought identifiability and parameterization issues to the forefront again, as practitioners encountered label switching, ridge-like likelihoods, and slow mixing due to non-identified or weakly identified parameters.

2010s–Present • Deep Generative Models

Modern deep learning architectures such as variational autoencoders and normalizing flows have revived interest in identifiability theory, particularly regarding disentangled representations, where researchers seek conditions under which latent factors can be uniquely recovered from high-dimensional data.

The central question that this lesson addresses is deceptively simple: given a family of probability distributions indexed by a parameter vector θ, can we recover the true θ from data? When the answer is no, we must either reparameterize the model, impose constraints, or accept that certain parameter functions — rather than individual parameters — are the proper objects of inference. Understanding these issues is essential before deploying any statistical model in practice.

SECTION 2

Core Principles & Definitions

Before examining the mathematical machinery, we need to establish several conceptual pillars. At its heart, identifiability is about the injectivity of the mapping from parameters to distributions: different parameter values must produce genuinely different observable behavior. When this mapping fails to be injective, no amount of data can distinguish between the observationally equivalent parameter values, and the model is said to be non-identifiable. Closely related is the choice of parameterization — the specific coordinate system we place on the model manifold — which influences not only identifiability but also the geometry of the likelihood surface, the behavior of optimization algorithms, and the interpretability of the results.

Identifiability (Global)

A parametric model P = {P_θ : θ ∈ Θ} is globally identifiable if the map θ ↦ P_θ is injective: θ₁ ≠ θ₂ implies P_θ₁ ≠ P_θ₂. This is the strongest form of identifiability.

Local Identifiability

A model is locally identifiable at θ₀ if there exists a neighborhood U of θ₀ such that no other θ ∈ U yields the same distribution. This is weaker than global identifiability — the map is locally injective but may fail globally.

Parameterization

Parameterization refers to the choice of coordinate representation for a model family. Two parameterizations of the same model class may differ in identifiability status, computational convenience, and interpretability. Reparameterization can sometimes cure non-identifiability or improve numerical stability.

Observational Equivalence

Two parameter values θ₁ and θ₂ are observationally equivalent if they generate the same distribution over observables. The equivalence classes under this relation characterize the fundamental ambiguity in a non-identified model.

Identifiable Functions

Even when a full parameter vector is not identifiable, certain functions g(θ) may be. A function g : Θ → ℝ is identifiable if it is constant on every equivalence class of observationally equivalent parameters. Identifying which functions are identifiable helps focus inference on what the data can actually reveal.

✦

KEY TAKEAWAY

Think of identifiability like trying to reconstruct a 3D object from its shadow. If two different 3D shapes cast the exact same shadow (the observed data), you cannot distinguish them from the shadow alone — the shapes are observationally equivalent. A model is identifiable when every distinct parameter configuration casts a unique shadow. Parameterization is the choice of how you describe those 3D shapes — Cartesian coordinates vs. spherical coordinates, for example — and a clever choice of coordinates can sometimes resolve or expose ambiguities.

SECTION 3

Visual Explanation — The Identifiability Map

The following diagram illustrates the fundamental structure of the identifiability problem. On the left, we have the parameter space Θ, representing all possible parameter configurations of the model. On the right, we have the model class 𝒫, the set of probability distributions that the model can generate. The map θ ↦ P_θ sends each parameter to a distribution. Identifiability is precisely the question of whether this map is injective. The diagram contrasts an identifiable model — where distinct parameters map to distinct distributions — with a non-identifiable model, where an equivalence class of parameters collapses onto a single distribution.

Left panel: an identifiable model where the map θ ↦ P_θ is injective (one-to-one). Right panel: a non-identifiable model where multiple parameter values (θ₁, θ₂, θ₃) map to the same distribution P*. The bottom box shows that identifiable functions remain constant across the equivalence class, making them the legitimate targets of inference.

In the diagram above, observe how the arrows in the identifiable case form a strict one-to-one correspondence between parameters and distributions, whereas in the non-identifiable case, three distinct parameter vectors θ₁, θ₂, θ₃ all point to the same distribution P*. This many-to-one collapse is precisely the structural defect that makes consistent estimation of individual parameters impossible. The equivalence class {θ : P_θ = P*} forms a surface or manifold in parameter space along which the likelihood is completely flat, meaning the data provide zero information for distinguishing among its members. Any function g(θ) that one hopes to estimate must be constant on this surface — otherwise the estimand itself is ill-defined.

SECTION 4

Mathematical Framework

We now formalize the concepts introduced above. Consider a parametric family of probability distributions {P_θ : θ ∈ Θ ⊆ ℝ^k} defined on a common measurable space (𝒳, ℬ), where each P_θ has density f(x | θ) with respect to some dominating measure μ. The mathematical definitions of identifiability follow naturally from the injectivity requirement on the map θ ↦ f(· | θ).

GLOBAL IDENTIFIABILITY

θ₁ ≠ θ₂ ⟹ f(x | θ₁) ≠ f(x | θ₂) for some x ∈ 𝒳

Equivalently, f(x | θ₁) = f(x | θ₂) for μ-almost all x implies θ₁ = θ₂. The model is globally identifiable if this condition holds for all pairs (θ₁, θ₂) in Θ.

LOCAL IDENTIFIABILITY (ROTHENBERG'S CONDITION)

rank I(θ₀) = k ⟹ θ₀ is locally identifiable

Here I(θ₀) denotes the Fisher information matrix evaluated at θ₀ ∈ Θ, with entries I_ij(θ) = −E[∂² log f(X | θ) / ∂θ_i ∂θ_j], and k = dim(θ). Full rank of I(θ₀) means the likelihood surface has non-degenerate curvature at θ₀ in every direction, precluding flat ridges.

EQUIVALENCE CLASS OF OBSERVATIONALLY EQUIVALENT PARAMETERS

[θ] = {θ' ∈ Θ : f(x | θ') = f(x | θ) for μ-a.e. x}

The set [θ] is the equivalence class of θ. The model is identifiable if and only if [θ] = {θ} for every θ ∈ Θ. A function g : Θ → ℝ is identifiable if and only if g is constant on every equivalence class.

A reparameterization can be viewed as a smooth bijection φ : Θ → Θ' that replaces θ with η = φ(θ). The model {f(x | φ⁻¹(η)) : η ∈ Θ'} is identifiable in η if and only if the original model is identifiable in θ, since bijections preserve injectivity. However, if the model is over-parameterized (dim θ > number of functionally independent parameters), reducing dimensionality via reparameterization can restore identifiability. This is not merely a cosmetic change — it alters the topology of the parameter space and the geometric properties of the likelihood.

REPARAMETERIZATION PRINCIPLE

η = φ(θ), dim(η) = rank of the identifiable parameter space

When the original parameter θ ∈ ℝ^k is non-identifiable but the equivalence classes form a smooth (k − r)-dimensional manifold, the identifiable parameter η lives in ℝ^r where r < k. The reduction from k to r dimensions eliminates the flat directions in the likelihood.

SECTION 5

Types of Non-Identifiability & Parameterization Pitfalls

Non-identifiability comes in several distinct flavors, each with different implications for inference and different remedies. Understanding the taxonomy of identifiability failures is crucial for diagnosis and repair. The following diagram classifies the major types and their relationships, and the subsequent table provides concrete examples for each category.

A hierarchical taxonomy of identifiability and parameterization issues. Global non-identifiability (e.g., label switching, scaling ambiguity) creates discrete or continuous equivalence classes. Local non-identifiability manifests as near-singularity of the Fisher information. Parameterization issues can exist even in identifiable models and affect computational and inferential performance.

Type	Example Model	Source of Non-Identifiability	Common Remedy
Label Switching	K-component Gaussian mixture	Permuting component indices (μ₁, σ₁, π₁) ↔ (μ₂, σ₂, π₂) yields same density	Order constraint (μ₁ < μ₂ < … < μ_K) or relabeling algorithms
Scaling/Rotation	Factor analysis: X = Λf + ε	For any invertible matrix Q, Λf = (ΛQ)(Q⁻¹f), so (Λ, f) and (ΛQ, Q⁻¹f) are equivalent	Fix Λ to be lower triangular with positive diagonal, or fix Cov(f) = I
Over-parameterization	Logistic regression: P(Y=1\|x) = σ(α + βx)	Adding a redundant intercept (e.g., separate intercepts per class in multinomial logit) creates a flat ridge	Set one reference category intercept to zero (corner constraint)
Weak Identification	Instrumental variables regression with weak instruments	Correlation between instrument and endogenous variable is near zero; I(θ) is near-singular	Stronger instruments, limited-information maximum likelihood, or robust confidence sets
Boundary	Random intercept model: Y_ij = μ + u_i + ε_ij	When σ²_u = 0, the random effect vanishes and its distribution is unidentifiable	Penalized estimation, restricted likelihood (REML), or informative priors on variance components

Common identifiability failures with concrete model examples and remedies

SECTION 6

Worked Example — Gaussian Mixture Identifiability

Consider a two-component Gaussian mixture model, which is one of the most instructive settings for understanding identifiability. We will demonstrate why the model is non-identifiable in its natural parameterization, identify the equivalence class structure, and show how a constraint restores identifiability.

Identifiability Analysis of a Two-Component Gaussian Mixture

1
Step 1 — Write the Model

Let X₁, …, X_n be i.i.d. from the density f(x | θ) = π φ(x; μ₁, σ₁²) + (1 − π) φ(x; μ₂, σ₂²), where φ(·; μ, σ²) is the normal density with mean μ and variance σ², π ∈ (0, 1) is the mixing weight, and θ = (π, μ₁, σ₁², μ₂, σ₂²) ∈ Θ ⊆ ℝ⁵.

θ = (π, μ₁, σ₁², μ₂, σ₂²), a 5-dimensional parameter vector
2
Step 2 — Identify the Symmetry

Observe that swapping the component labels produces the same density: f(x | π, μ₁, σ₁², μ₂, σ₂²) = f(x | 1 − π, μ₂, σ₂², μ₁, σ₁²). Define θ* = (1 − π, μ₂, σ₂², μ₁, σ₁²). Then P_θ = P_θ* for all x, yet θ ≠ θ* whenever (μ₁, σ₁²) ≠ (μ₂, σ₂²). This is label switching.

Equivalence class: [θ] = {θ, θ*} — each parameter has exactly one observationally equivalent partner
3
Step 3 — Check the Fisher Information

At any point where μ₁ = μ₂ and σ₁² = σ₂², the two components become identical. At this boundary, the mixing weight π is completely unidentifiable: changing π does not change the density. Consequently, the Fisher information matrix I(θ) is singular (rank < 5) at these points. Away from this boundary, I(θ) has full rank 5 modulo the discrete label-switching symmetry, indicating that the model is locally identifiable at generic parameter values.

rank I(θ) = 5 for generic θ (local identifiability holds), but rank < 5 when components coincide
4
Step 4 — Impose a Constraint to Restore Global Identifiability

To eliminate the label-switching ambiguity, we impose the ordering constraint μ₁ ≤ μ₂ (with strict inequality when the components are distinct). Under this constraint, the mapping θ ↦ f(· | θ) becomes injective on the restricted parameter space Θ' = {θ ∈ Θ : μ₁ ≤ μ₂}, and the model is globally identifiable.

With μ₁ ≤ μ₂, the two-component Gaussian mixture is globally identifiable (up to the equal-means boundary)
5
Step 5 — Identify Identifiable Functions

Even without the ordering constraint, certain functions of θ are identifiable because they are invariant under label switching. For example, the overall mean E[X] = πμ₁ + (1 − π)μ₂ and the overall variance Var(X) = πσ₁² + (1 − π)σ₂² + π(1 − π)(μ₁ − μ₂)² are both identifiable. These symmetric functions can be consistently estimated regardless of the labeling convention.

Identifiable functions include E[X], Var(X), and any symmetric function of the component parameters

SECTION 7

Strengths & Limitations of Identifiability Strategies

Addressing identifiability and parameterization problems requires a toolbox of strategies, each with trade-offs. The choice depends on whether the non-identifiability is global or local, discrete or continuous, and on the inferential framework (frequentist or Bayesian). The following table compares the major strategies along several dimensions.

Strategy	Strengths	Limitations
Ordering Constraints	Simple to implement; resolves discrete label-switching; preserves likelihood shape within each equivalence class	Introduces boundary effects (constrained MLE may lie on boundary); does not address continuous non-identifiability; may not correspond to scientifically meaningful orderings
Reparameterization	Can reduce dimension to match identifiable degrees of freedom; often improves numerical conditioning; may yield orthogonal or approximately orthogonal parameters	Requires analytic understanding of the equivalence class structure; reparameterized parameters may lack scientific interpretability; non-trivial for complex models
Bayesian Priors	Can regularize non-identified parameters; posterior exists even when MLE does not; allows incorporation of substantive knowledge; handles both discrete and continuous ambiguity	Posterior is heavily prior-dependent for non-identified parameters; may give false confidence; can mask model misspecification; MCMC mixing may be poor (label switching)
Focus on Identifiable Functions	Philosophically clean — only estimates what data can determine; avoids arbitrary constraints; valid inference guarantees	May not answer the substantive question (e.g., if you need individual component means); requires characterizing the identifiable function space, which can be non-trivial
Structural/Exclusion Restrictions	Establishes identification from theoretical/economic/scientific reasoning; standard in econometrics and causal inference; can be tested (over-identifying restrictions)	Restrictions may be untestable or scientifically dubious; wrong restrictions lead to biased inference; requires domain expertise

Comparison of strategies for addressing identifiability issues

✦

KEY TAKEAWAY

No single strategy dominates. In practice, the best approach combines structural reasoning about why non-identifiability arises with a pragmatic choice of remedy tailored to the inferential goal. Much like fixing a coordinate system in physics — the laws of nature don't change, but some coordinates make the math tractable — choosing a good parameterization doesn't alter the model's predictive content but can make the difference between a well-behaved and a pathological inference procedure.

SECTION 8

Connection to Information Geometry & Advanced Theory

Identifiability and parameterization are not merely practical nuisances — they sit at the intersection of several deep theoretical frameworks. Information geometry views a parametric family as a manifold equipped with the Fisher-Rao metric, and identifiability corresponds to the embedding being an immersion (no self-intersections). Non-identifiability creates singularities on this manifold — points where the metric degenerates and standard asymptotic theory fails. The study of these singularities connects to algebraic statistics and singular learning theory (Watanabe, 2009), where the Bayesian evidence and generalization error near singular points follow different scaling laws than those predicted by classical BIC.

Concept	Classical (Regular) Setting	Non-Identifiable / Singular Setting
Fisher Information I(θ₀)	Full rank (k × k positive definite)	Rank-deficient; null space corresponds to non-identified directions
MLE Asymptotics	√n-consistent, asymptotically normal: √n(θ̂ − θ₀) →_d N(0, I(θ₀)⁻¹)	May converge at slower rates (e.g., n¹ᐟ⁴ for boundary parameters); non-normal limits; standard errors invalid
Likelihood Ratio Test	−2 log Λ →_d χ² under H₀	Non-standard null distribution; often a mixture of χ² distributions (e.g., testing σ² = 0 in random effects gives ½χ²₀ + ½χ²₁)
Model Selection (BIC)	BIC = −2 log L + k log n	Effective dimension ≠ k; WBIC or singular BIC needed, using the real log canonical threshold (RLCT) from algebraic geometry
Bayesian Posterior	Bernstein–von Mises: posterior concentrates around MLE at rate 1/√n	Bernstein–von Mises may fail; posterior shape is prior-dependent in non-identified directions; concentration rates differ

How non-identifiability disrupts standard statistical results

Looking forward, the intersection of identifiability theory with causal inference is particularly active. In structural causal models, identifiability of causal effects from observational data depends on the graphical structure (the do-calculus and ID algorithm), and many apparently simple causal questions reduce to subtle identifiability problems. Similarly, in deep generative modeling, recent work by Khemakhem et al. (2020) establishes conditions under which nonlinear ICA models become identifiable, connecting classical statistical theory with modern representation learning. Understanding the conceptual foundations laid out in this lesson is the essential prerequisite for engaging with these advanced topics.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Consider the model f(x | a, b) = exp(−(a + b)x) for x > 0, with parameters a > 0 and b > 0. Is this model identifiable? If not, characterize the equivalence classes and identify a function of (a, b) that is identifiable.

Show answer

The model is not identifiable because f(x | a, b) depends on a and b only through their sum λ = a + b. Any two parameter pairs (a₁, b₁) and (a₂, b₂) satisfying a₁ + b₁ = a₂ + b₂ yield the same exponential distribution. The equivalence class of (a, b) is the line segment {(a', b') : a' + b' = a + b, a' > 0, b' > 0}. The function g(a, b) = a + b is identifiable — it is constant on each equivalence class — and one can reparameterize to the single parameter λ = a + b to obtain an identifiable model.

Question 2: BASIC CALCULATION

Let X ~ N(μ, σ²) with θ = (μ, σ²). Compute the Fisher information matrix I(θ) and verify that it has full rank, confirming local identifiability of the normal location-scale family.

Show answer

The log-likelihood for a single observation is ℓ(θ; x) = −½ log(2πσ²) − (x − μ)² / (2σ²). Computing the second derivatives: ∂²ℓ/∂μ² = −1/σ², ∂²ℓ/(∂μ ∂σ²) = −(x − μ)/σ⁴, and ∂²ℓ/∂(σ²)² = 1/(2σ⁴) − (x − μ)²/σ⁶. Taking expectations: I₁₁ = 1/σ², I₁₂ = 0, I₂₂ = 1/(2σ⁴). Thus I(θ) = diag(1/σ², 1/(2σ⁴)). Since both diagonal entries are strictly positive for σ² > 0, the matrix has full rank 2 = dim(θ). By Rothenberg's theorem, the normal location-scale model is locally (and in fact globally) identifiable.

Question 3: INTERMEDIATE

In a factor analysis model X = Λf + ε with X ∈ ℝ^p, f ∈ ℝ^m, Λ is p × m, f ~ N(0, I_m), and ε ~ N(0, Ψ) with Ψ diagonal, explain why the loading matrix Λ is not identifiable. How many constraints are needed to achieve identifiability, and what is a standard constraint set?

Show answer

The covariance of X is Σ = ΛΛ^T + Ψ. For any m × m orthogonal matrix Q (i.e., QQ^T = I), we have (ΛQ)(ΛQ)^T = ΛQQ^TΛ^T = ΛΛ^T, so Λ and ΛQ produce identical covariance structures and are observationally equivalent. The group of orthogonal rotations O(m) has m(m − 1)/2 free parameters, so m(m − 1)/2 constraints are needed. A standard approach is to require that the upper m × m block of Λ be lower triangular with positive diagonal entries, which imposes exactly m(m − 1)/2 zero constraints plus m sign constraints. This pins down a unique representative from each equivalence class, making Λ identifiable.

Question 4: APPLIED

A researcher fits a Bayesian hierarchical model for clinical trial data: Y_ij | μ_i ~ N(μ_i, σ²), μ_i ~ N(γ, τ²), with priors on (γ, σ², τ²). The MCMC sampler shows very high autocorrelation and bimodal trace plots for τ². Diagnose the likely identifiability or parameterization issue and suggest a remedy.

Show answer

The symptoms — high autocorrelation and bimodality in τ² — are classic signs of a near-boundary identifiability issue combined with a poor parameterization. When the between-study heterogeneity τ² is small, the model approaches a non-hierarchical model (τ² = 0) where the individual μ_i collapse to γ, creating a boundary singularity. Additionally, the (γ, τ²) parameterization often induces strong posterior dependence between γ and the μ_i's. A common remedy is the non-centered parameterization: write μ_i = γ + τη_i where η_i ~ N(0, 1). This decouples τ from the latent effects and dramatically improves MCMC mixing. Additionally, a weakly informative half-Cauchy or half-normal prior on τ keeps the sampler away from the degenerate τ = 0 boundary.

Question 5: CRITICAL THINKING

Prove or disprove: if a parametric model {P_θ : θ ∈ Θ} is locally identifiable at every θ ∈ Θ, then it is globally identifiable. If the statement is false, provide a counterexample and explain the conceptual gap between local and global identifiability.

Show answer

The statement is false. Local identifiability everywhere does not imply global identifiability. A counterexample: consider the family of distributions on [0, 2π) with density f(x | θ) = (1 + cos(x − θ)) / (2π) for θ ∈ [0, 2π). This model is locally identifiable at every θ (the Fisher information is strictly positive). However, f(x | θ) = f(x | θ + 2π), so if Θ = ℝ rather than [0, 2π), the model is not globally identifiable — the map θ ↦ P_θ is periodic. More subtly, even on compact parameter spaces, a smooth injective local map can fail to be globally injective (the map can 'wrap around'). The conceptual gap is that local identifiability checks the rank of I(θ) at each point (a differential condition), whereas global identifiability is a topological condition about the entire map. Rothenberg's theorem guarantees local → global only when Θ is convex and additional regularity conditions hold.

SUMMARY

Lesson Summary

Summary

Identifiability is the requirement that the map from parameters to distributions be injective: distinct parameter values must produce observationally distinct distributions. When this fails, the model suffers from non-identifiability, and the equivalence classes of observationally equivalent parameters define the fundamental ambiguity. Local identifiability is diagnosed via the rank of the Fisher information matrix (Rothenberg's condition), while global identifiability requires verifying injectivity across the entire parameter space — a stronger, often more difficult condition.

Closely intertwined is the choice of parameterization, which affects computational stability, MCMC mixing, and interpretability even when the model is identifiable. Common sources of non-identifiability include label switching in mixture models, rotational invariance in factor models, and over-parameterization in regression settings. Remedies range from ordering constraints and reparameterization to informative priors and focusing inference on identifiable functions. These concepts connect forward to information geometry, singular learning theory, and modern challenges in causal inference and deep generative modeling.