p-Values & Test Inversion — p-values and test inversion to confidence sets (conceptual) - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The quest to quantify statistical evidence and to construct regions of plausible parameter values has shaped modern inference for over a century. The p-value emerged as a measure of the compatibility between observed data and a null hypothesis, while confidence sets arose from the desire to make statements about unknown parameters that carry frequency-based guarantees. Although these two ideas developed along parallel tracks, the deep connection between them — through the mechanism of test inversion — unifies hypothesis testing and interval estimation into a single, coherent framework.

1900–1925 • Karl Pearson & R. A. Fisher

Pearson introduces the χ² goodness-of-fit test. Fisher formalizes the concept of a p-value as a continuous measure of evidence against a null hypothesis, embedding it in his significance testing paradigm.

1928–1937 • Neyman–Pearson Framework

Jerzy Neyman and Egon Pearson develop the theory of fixed-level tests with controlled Type I and Type II error rates, shifting focus from single p-values to decision-theoretic procedures.

1934–1937 • Confidence Intervals via Inversion

Neyman introduces the concept of confidence intervals (1934) and explicitly shows they can be constructed by collecting all parameter values whose hypothesis test does not reject — the test inversion principle.

1950s–1970s • Generalization to Confidence Sets

Lehmann, Scheffé, and others extend the inversion principle to multiparameter settings, producing confidence regions, simultaneous confidence bands, and connections to union-intersection and intersection-union tests.

1990s–present • Modern Refinements

Bootstrap methods, resampling-based confidence sets, and computational inversion techniques make the test-inversion duality practical for complex models far beyond the classical location family.

The central question that motivates this lesson is both elegant and practical: if you already know how to test hypotheses, can you leverage those tests to build confidence sets — and vice versa? The answer is a resounding yes, and the bridge is the test-inversion correspondence. Understanding this duality not only deepens conceptual insight but provides a constructive recipe for deriving confidence regions in settings where direct pivotal arguments are unavailable.

SECTION 2

Core Principles & Definitions

Before exploring the inversion mechanism, we must establish precise definitions. A p-value for a hypothesis H₀ : θ = θ₀ is a statistic p(X) satisfying $P_{θ₀}$ (p(X) ≤ α) ≤ α for all α ∈ (0, 1). Intuitively, smaller p-values indicate that the observed data are less compatible with the null hypothesis. A level-α test rejects H₀ when p(X) ≤ α, and a confidence set C(X) is a random subset of the parameter space with the guarantee P_θ(θ ∈ C(X)) ≥ 1 − α for every θ.

p-Value as a Random Variable

Under the null, p(X) has a distribution that is stochastically greater than or equal to Uniform(0,1). For continuous test statistics, this distribution is exactly Uniform(0,1), making p a valid probability integral transform of the evidence.

Acceptance Region of a Test

For each θ₀, the acceptance region A(θ₀) = {x : the level-α test does not reject H₀ : θ = θ₀}. Equivalently, A(θ₀) = {x :

p_{θ₀}

(x) > α}, the set of sample outcomes compatible with θ₀.

Test Inversion

A confidence set by inversion is C(x) = {θ₀ : x ∈ A(θ₀)} — the collection of all parameter values whose null hypothesis is not rejected by the data x. This swaps the roles of parameter and data.

Duality Guarantee

If every individual test has level α, the inverted confidence set has coverage at least 1 − α. Conversely, any 1 − α confidence set can be used to define a level-α test by rejecting when θ₀ ∉ C(X). The two constructions are logically equivalent.

✦

KEY TAKEAWAY

Think of test inversion like trying on shoes. For each candidate shoe size (θ₀), you perform a 'fit test': does this size work for your foot (data)? Your confidence set is the range of sizes that passed the test. A well-calibrated test ensures that the true size is among the accepted ones at least 1 − α of the time — exactly the coverage guarantee of a confidence interval.

SECTION 3

Visual Explanation — The Inversion Duality

Each horizontal band is the acceptance region A(θ) for testing H₀ : θ = θᵢ. The vertical dashed green line represents the observed data x_obs. The confidence set C(x_obs) is the collection of parameter values whose acceptance region contains x_obs (filled circles). Here θ₀ is excluded because x_obs falls outside A(θ₀).

The diagram above captures the fundamental geometry of test inversion. Each parameter value θ in the parameter space Θ gives rise to its own hypothesis H₀ : θ = θᵢ, and the corresponding level-α test has an acceptance region A(θᵢ) in the sample space. When we observe data x_obs, we scan vertically through the parameter space and collect every θ for which x_obs lies inside A(θ). That collection is precisely the 1 − α confidence set. The duality is visible: looking horizontally at a fixed θ gives the acceptance region (a subset of the sample space), while looking vertically at a fixed x gives the confidence set (a subset of the parameter space).

SECTION 4

Mathematical Framework

Formal Definition of a p-Value

VALID p-VALUE

P_{θ₀}( p(X) ≤ α ) ≤ α for all α ∈ (0, 1)

Here p(X) is the p-value function computed from data X, and θ₀ is the parameter value under the null. When the test statistic T(X) has a continuous distribution under θ₀, we have p(X) = $P_{θ₀}$ (T(X) ≥ T(x)) for a one-sided test, and p(X) ~ Uniform(0, 1) under H₀.

From Test to Confidence Set

TEST INVERSION FORMULA

C(x) = { θ₀ ∈ Θ : p_{θ₀}(x) > α }

Equivalently, C(x) = { θ₀ ∈ Θ : x ∈ A(θ₀) }, where A(θ₀) is the acceptance region of the level-α test for H₀ : θ = θ₀. The set C(x) contains every parameter value not rejected by the data.

Coverage Proof Sketch

Fix the true parameter θ*. We need $P_{θ*}$ (θ* ∈ C(X)) ≥ 1 − α. By definition, θ* ∈ C(X) if and only if X ∈ A(θ*), which happens if and only if the test of H₀ : θ = θ* does not reject. Since the test has level α, $P_{θ*}$ (reject H₀ : θ = θ*) ≤ α, so $P_{θ*}$ (X ∈ A(θ*)) ≥ 1 − α. This completes the argument. Notice that the proof works for every θ* simultaneously, which is precisely the frequentist coverage requirement.

COVERAGE GUARANTEE

P_{θ*}(θ* ∈ C(X)) = P_{θ*}(X ∈ A(θ*)) = 1 − P_{θ*}(reject H₀ : θ = θ*) ≥ 1 − α

This holds for every θ* ∈ Θ. The inequality becomes an equality when the test is exactly level α (size α).

From Confidence Set to Test (The Reverse Direction)

REVERSE INVERSION

Reject H₀ : θ = θ₀ ⟺ θ₀ ∉ C(X)

Given any 1 − α confidence set C(X), define a test that rejects H₀ : θ = θ₀ precisely when θ₀ falls outside C(X). This test has level α because $P_{θ₀}$ (θ₀ ∉ C(X)) ≤ α by the coverage property.

SECTION 5

The p-Value Function and Confidence Set Geometry

A powerful way to visualize the test-inversion correspondence is through the p-value function (sometimes called the confidence curve or significance function). For a given dataset x, define the function θ₀ ↦ $p_{θ₀}$ (x), which maps each candidate parameter value to the p-value obtained when testing H₀ : θ = θ₀. This function encapsulates the entire inference story: the 1 − α confidence set is simply the level set {θ₀ : $p_{θ₀}$ (x) > α}. As α varies, you sweep out nested confidence sets — smaller α yields larger, more conservative intervals.

The p-value function (gold curve) plots

p_{θ₀}

(x) against θ₀ for fixed observed data. A horizontal slice at α = 0.05 (red dashed line) intersects the curve at θ_L and θ_U, yielding the 95% confidence interval [θ_L, θ_U]. The peak at the MLE θ̂ has p-value 1, meaning the data are maximally compatible with that parameter value.

Several important features are visible in this diagram. First, the p-value function peaks at the maximum likelihood estimate θ̂, where p_{θ̂}(x) = 1 for typical tests based on likelihood ratios. Second, the function decreases monotonically on either side of θ̂ for one-parameter exponential families, ensuring that the resulting confidence set is always an interval rather than a disjoint union of regions. Third, changing the significance level α simply moves the horizontal slice up or down: a more stringent α = 0.01 yields a wider 99% confidence interval, while a more relaxed α = 0.10 gives a narrower 90% interval. This graphical device, the p-value function, encodes all possible confidence intervals simultaneously.

⚠

When Inversion Gives Non-Intervals

If the p-value function is not unimodal — for example, when testing a variance parameter with a two-sided χ² test — the level set can be a disjoint union, producing a disconnected confidence set. This is not a pathology but a genuine reflection of the test's geometry. In multiparameter problems, the confidence set is typically a region in ℝᵏ (e.g., an ellipse for bivariate normal means), not a simple interval.

SECTION 6

Worked Example — Normal Mean with Known Variance

Consider the canonical setting: X₁, …, Xₙ are i.i.d. Normal(μ, σ²) with σ² known. We will construct a 95% confidence interval for μ by inverting the family of two-sided z-tests at level α = 0.05.

Test Inversion for a Normal Mean (σ² known)

1
Step 1 — Set Up the Family of Tests

For each candidate value μ₀, consider H₀ : μ = μ₀ versus H₁ : μ ≠ μ₀. The test statistic is Z(μ₀) = (X̄ − μ₀) / (σ / √n). The level-α test rejects when |Z(μ₀)| > $z_{α/2}$ . For α = 0.05, $z_{0.025}$ = 1.96.
2
Step 2 — Determine the Acceptance Region

The test does not reject H₀ : μ = μ₀ when |Z(μ₀)| ≤ 1.96, which is equivalent to |X̄ − μ₀| ≤ 1.96 × σ/√n. In terms of data x̄, the acceptance region is A(μ₀) = {x̄ : |x̄ − μ₀| ≤ 1.96σ/√n}.
3
Step 3 — Invert: Collect Non-Rejected Values

The confidence set is C(x̄) = {μ₀ : x̄ ∈ A(μ₀)} = {μ₀ : |x̄ − μ₀| ≤ 1.96σ/√n}. Rewriting this inequality as −1.96σ/√n ≤ x̄ − μ₀ ≤ 1.96σ/√n and solving for μ₀:

C(x̄) = [ x̄ − 1.96σ/√n , x̄ + 1.96σ/√n ]
4
Step 4 — Plug In Numbers

Suppose n = 25, σ = 10, and x̄ = 52. Then σ/√n = 10/5 = 2. The margin of error is 1.96 × 2 = 3.92.

C(52) = [52 − 3.92, 52 + 3.92] = [48.08, 55.92]
5
Step 5 — Interpret the p-Value Function

The p-value at any candidate μ₀ is $p_{μ₀}$ (x̄) = 2 × [1 − Φ(|x̄ − μ₀| / (σ/√n))]. For instance, testing μ₀ = 50 gives Z = (52 − 50)/2 = 1.0, so p = 2 × [1 − Φ(1.0)] = 2 × 0.1587 ≈ 0.317. Since 0.317 > 0.05, μ₀ = 50 is inside the confidence interval, consistent with our result above.

$p_{50}$ (52) ≈ 0.317 > 0.05 → 50 ∈ [48.08, 55.92] ✓

✦

FAMILIAR FORMULA, DEEPER ORIGIN

The classic formula x̄ ±

z_{α/2}

× σ/√n is not just a memorized recipe — it is the natural consequence of inverting the two-sided z-test. Recognizing this derivation makes it straightforward to generalize: whenever you have a valid test, you have a valid confidence set, regardless of whether a neat closed-form formula exists.

SECTION 7

Strengths, Limitations & Alternative Approaches

Aspect	Test Inversion	Pivotal Quantity Method
Generality	Works for any valid test — does not require a pivot. Applicable to discrete, multiparameter, and nuisance-parameter settings.	Requires a pivotal quantity whose distribution is independent of all parameters. Limited when no exact pivot exists.
Ease of Computation	Can be computationally intensive: must solve for all θ₀ satisfying $p_{θ₀}$ (x) > α, potentially via numerical root-finding or grid search.	Usually yields a closed-form interval after simple algebraic manipulation of the pivot.
Optimality	Inherits the optimality of the underlying test: inverting a UMP test produces a uniformly most accurate (UMA) confidence set.	May or may not be optimal depending on the choice of pivot and manipulation.
Interval Shape	May produce non-interval (disconnected) confidence sets if the p-value function is not unimodal.	Typically produces a single interval via simple rearrangement.
Connection to Testing	Built-in by construction: every CI maps directly to a coherent family of tests.	Connection exists but must be made explicitly — not automatic from the construction.

Comparison of test inversion and the pivotal quantity method for constructing confidence sets.

✦

WHY IT MATTERS

Test inversion is the universal construction — whenever a pivotal method does not exist or is infeasible, you can always fall back on inverting a test. In modern statistics, bootstrap-calibrated tests, permutation tests, and likelihood-ratio tests for complex models are routinely inverted to produce confidence regions that would be impossible to derive by hand.

SECTION 8

Connections to Advanced Theory

The test-inversion principle extends naturally to several advanced topics in mathematical statistics. Understanding these connections positions you to tackle graduate-level problems in asymptotic theory, Bayesian inference, and multiple testing.

Advanced Topic	Connection to Test Inversion
Likelihood Ratio Confidence Regions	Invert the likelihood ratio test: C(x) = {θ₀ : −2 log Λ(θ₀) ≤ χ²_{k,1−α}}. By Wilks' theorem, this yields asymptotically valid confidence regions as ellipsoids in ℝᵏ.
Bayesian Credible vs. Confidence Sets	Bayesian highest posterior density (HPD) regions do not generally arise from test inversion and may not have frequentist coverage. The inversion framework highlights where Bayesian and frequentist inference diverge.
Simultaneous Confidence Bands	For function-valued parameters (e.g., a regression function), inverting a global test (F-test, sup-norm test) produces Scheffé-type simultaneous confidence bands that maintain coverage over all linear combinations.
Bootstrap Confidence Sets	The bootstrap can estimate the p-value function $p_{θ₀}$ (x) by simulation, then invert: C(x) = {θ₀ : p̂_{θ₀}(x) > α}. This is the conceptual basis of the bootstrap-t and bootstrap percentile-t methods.
UMP Tests & UMA Confidence Sets	Inverting a uniformly most powerful (UMP) test produces a uniformly most accurate (UMA) confidence set — the narrowest possible at every true parameter value. This is the gold standard when it exists (e.g., one-sided exponential family problems).

Advanced topics linked through the test-inversion duality.

Looking ahead, the duality between tests and confidence sets extends to decision theory through the concept of duality between minimax testing and minimax estimation. In high-dimensional statistics, the ability to test — even approximately — is closely linked to the ability to construct honest confidence sets, and impossibility results in one domain imply impossibility in the other. This deep structural connection remains an active area of research.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain in your own words why the set C(x) = {θ₀ :

p_{θ₀}

(x) > α} has coverage at least 1 − α. Your explanation should reference the definition of a valid p-value and the level of the underlying test.

Show answer

Fix the true parameter θ*. A valid p-value satisfies

P_{θ*}

(

p_{θ*}

(X) ≤ α) ≤ α. The event θ* ∉ C(X) is exactly the event

p_{θ*}

(X) ≤ α (we reject H₀ : θ = θ*). Therefore

P_{θ*}

(θ* ∉ C(X)) =

P_{θ*}

(

p_{θ*}

(X) ≤ α) ≤ α, which gives

P_{θ*}

(θ* ∈ C(X)) ≥ 1 − α. The key insight is that the coverage guarantee follows directly from the level condition of the test — no additional assumptions are needed beyond the validity of the p-value.

Question 2: BASIC CALCULATION

Let X₁, …, X₁₆ be i.i.d. Normal(μ, 16). The observed sample mean is x̄ = 10. By inverting the two-sided z-test at level α = 0.05, construct a 95% confidence interval for μ. Verify that μ₀ = 8 belongs to the interval by computing

p_{8}

(10).

Show answer

Here σ = 4, n = 16, so σ/√n = 4/4 = 1. The 95% CI by inversion is x̄ ±

z_{0.025}

× 1 = 10 ± 1.96 × 1 = [8.04, 11.96]. For μ₀ = 8: Z = (10 − 8)/1 = 2.0, so p = 2 × [1 − Φ(2.0)] = 2 × 0.0228 = 0.0456. Since 0.0456 < 0.05, μ₀ = 8 is rejected and indeed 8 < 8.04, confirming it falls just outside the interval. If we check μ₀ = 8.1: Z = 1.9, p = 2 × [1 − Φ(1.9)] ≈ 0.0574 > 0.05, so 8.1 would be inside — consistent with the boundary at 8.04.

Question 3: INTERMEDIATE

Suppose you have a UMP test for H₀ : μ ≤ μ₀ versus H₁ : μ > μ₀ in a one-parameter exponential family. You invert this one-sided test at level α. What form does the resulting confidence set take — is it a two-sided interval, a one-sided interval, or something else? Explain why this confidence set is called uniformly most accurate (UMA) unbiased when the test is UMP unbiased.

Show answer

Inverting a one-sided test of H₀ : μ ≤ μ₀ at level α produces a one-sided confidence set of the form C(x) = [μ_L(x), ∞), i.e., a lower confidence bound. This is because we reject when the test statistic is large (evidence for μ > μ₀), so we fail to reject for small μ₀ values and reject for large μ₀. The resulting set collects all μ₀ below some threshold — hence a one-sided interval. A UMA (uniformly most accurate) confidence set minimizes P_θ(θ₀ ∈ C(X)) for every θ₀ ≠ θ, meaning the set is as small as possible at every true parameter. This property is inherited directly from the UMP property: the UMP test maximizes power at every alternative, which translates to minimizing the probability that a false parameter value is covered.

Question 4: APPLIED

A clinical trial yields a test statistic for comparing a treatment mean to placebo. The p-value function

p_{δ}

(x) (where δ = μ_treatment − μ_placebo) is computed numerically via a permutation test for each candidate δ₀ on a grid from −5 to 15. The function exceeds 0.05 for δ₀ ∈ [1.3, 9.8]. Describe how this information produces a 95% confidence interval, how you would refine the endpoints, and what the interval tells you about clinical significance if the minimal clinically important difference is δ = 2.

Show answer

The 95% confidence interval by test inversion is [1.3, 9.8] — the set of treatment effects for which the permutation test does not reject at α = 0.05. To refine the endpoints, one would use a finer grid or bisection search near 1.3 and 9.8 to find the exact δ₀ values where

p_{δ₀}

(x) = 0.05. Clinically, the interval includes values below 2 (the MCID), so the lower bound does not rule out a treatment effect that is statistically significant but clinically negligible. A clinician would note that while the treatment is statistically significant (0 ∉ CI), the evidence for clinical meaningfulness is ambiguous because the lower bound 1.3 < 2. This illustrates how test inversion via a permutation test — which requires no parametric assumptions — still produces interpretable confidence intervals.

Question 5: CRITICAL THINKING

Consider testing H₀ : θ = θ₀ in a discrete model (e.g., Binomial(n, θ)). The p-value is conservative:

P_{θ₀}

(p(X) ≤ α) < α strictly, for most α. When you invert this family of tests, the resulting confidence set has coverage strictly above 1 − α. Discuss: (a) Is this a feature or a limitation? (b) How does randomization of the test affect the confidence set? (c) How might you use the 'mid-p-value' to obtain less conservative intervals, and what coverage guarantee, if any, is lost?

Show answer

(a) It is both: conservatism ensures the coverage guarantee is never violated (desirable for reliability), but the resulting intervals are wider than necessary, reducing informativeness. In small-sample discrete problems, this over-coverage can be substantial. (b) A randomized test achieves exact size α by rejecting with some probability γ when the test statistic falls on the boundary. Inverting a randomized test produces a randomized confidence set — one whose membership depends on an auxiliary uniform random variable. This set has exact 1 − α coverage but is controversial because two analysts with the same data could obtain different intervals. (c) The mid-p-value, defined as

P_{θ₀}

(T > t_obs) + ½

P_{θ₀}

(T = t_obs), is less conservative and often closer to the nominal level on average. Inverting the mid-p produces narrower confidence intervals. However, the strict guarantee

P_{θ₀}

(p(X) ≤ α) ≤ α no longer holds for all α, so the confidence set may have coverage slightly below 1 − α at some θ values. This represents a trade-off between exact coverage and interval width that is central to modern discussions of discrete inference.

SUMMARY

Lesson Summary

Summary

This lesson established the deep duality between hypothesis testing and confidence set construction through the mechanism of test inversion. A p-value measures the compatibility of observed data with a null hypothesis and is formally valid when $P_{θ₀}$ (p(X) ≤ α) ≤ α. By scanning over all candidate parameter values θ₀ and collecting those not rejected at level α — equivalently, those with $p_{θ₀}$ (x) > α — we construct a 1 − α confidence set with guaranteed frequentist coverage.

The p-value function θ₀ ↦ $p_{θ₀}$ (x) provides a complete graphical summary: horizontal slices at any significance level produce the corresponding confidence interval. The familiar formula x̄ ± $z_{α/2}$ σ/√n is a special case — obtained by inverting the two-sided z-test. In more complex settings (permutation tests, bootstrap, likelihood ratios), the inversion principle remains the universal recipe for deriving confidence regions, and inverting optimal tests yields optimal confidence sets — a powerful guiding principle for inference.