Algorithmic Bias & Fairness — I can evaluate algorithmic...

SECTION 1

Historical Context & Motivation

The question of whether technologies can be politically or morally neutral is not new, but the rapid deployment of algorithmic decision-making systems in hiring, criminal sentencing, credit scoring, and healthcare has given it unprecedented urgency. Philosophers of technology have long argued that artifacts embed values—Langdon Winner's 1980 essay "Do Artifacts Have Politics?" examined how the physical design of bridges and power plants could reinforce social hierarchies. As machine-learning models began to replace or augment human judgment in high-stakes domains during the 2010s, researchers discovered that these systems could reproduce and even amplify patterns of racial, gender, and socioeconomic discrimination present in their training data. The result is a rich, ongoing conversation that sits at the intersection of applied ethics, political philosophy, computer science, and law.

1980

Winner's "Do Artifacts Have Politics?"

Langdon Winner argues that technological artifacts can embody political arrangements, establishing a philosophical foundation for questioning the neutrality of designed systems.

2014

ProPublica Investigates COMPAS

Investigative journalists begin analyzing the COMPAS recidivism algorithm, later publishing findings in 2016 showing significant racial disparities in risk scores assigned to Black and white defendants.

2016

Impossibility Theorems Emerge

Researchers including Chouldechova and Kleinberg et al. formally prove that certain fairness criteria are mathematically incompatible, demonstrating that fairness is not merely a technical problem but a normative one.

2018

Gender Shades Study

Joy Buolamwini and Timnit Gebru reveal that commercial facial recognition systems have significantly higher error rates for darker-skinned women, sparking widespread concern about intersectional bias in AI.

2023–2024

EU AI Act & Global Regulation

The European Union adopts the AI Act, the first comprehensive regulatory framework classifying AI systems by risk level and mandating fairness audits for high-risk applications.

These developments raise a central question for applied philosophy: By what ethical and political standards should we evaluate the fairness of algorithmic systems? The challenge is not merely identifying when an algorithm produces disparate outcomes, but articulating what fairness requires in contexts where different values—accuracy, equality, liberty, desert—may conflict with one another. This lesson equips you to engage that challenge by connecting formal definitions of fairness to the philosophical traditions that give them meaning.

SECTION 2

Core Ethical & Political Principles

Evaluating algorithmic fairness requires drawing on multiple ethical and political frameworks, each of which foregrounds different values and diagnoses bias differently. No single framework captures all dimensions of the problem, which is precisely why philosophical analysis is indispensable. Below are the foundational concepts that structure contemporary debates about algorithmic justice.

Distributive Justice

Concerned with how benefits and burdens are allocated across a population. Rawlsian approaches ask whether an algorithm's outcomes could be endorsed from behind a veil of ignorance, while utilitarian approaches evaluate whether total welfare is maximized.

Procedural Fairness

Focuses on whether the process by which decisions are made is just, regardless of outcomes. A procedurally fair algorithm should treat relevantly similar cases similarly and exclude morally irrelevant features such as race or gender from its decision logic.

Non-Domination

Drawing on republican political theory, this principle holds that no agent—including an algorithm—should exercise arbitrary power over others. Algorithmic systems that are opaque and unaccountable may dominate even when their outputs appear fair.

Epistemic Justice

Introduced by Miranda Fricker, this concept highlights harms inflicted when a person's testimony or knowledge is discounted. Algorithms trained on historically biased data may perpetuate testimonial injustice by systematically undervaluing the needs or inputs of marginalized groups.

Consequentialism vs. Deontology

Consequentialist evaluations ask whether an algorithm produces the best overall outcomes, while deontological evaluations ask whether it respects individual rights and duties, regardless of aggregate consequences. These frameworks often yield conflicting verdicts on the same system.

✦ KEY TAKEAWAY

Think of fairness as a prism rather than a lens. When white light passes through a prism, it separates into distinct colors—just as a single algorithmic outcome, when examined through different ethical frameworks, reveals distinct dimensions of fairness that may conflict with one another. A system that is "fair" under a utilitarian criterion (maximizing aggregate accuracy) may be deeply unfair under a Rawlsian criterion (protecting the least advantaged group). Recognizing this irreducible pluralism is the first step toward responsible evaluation.

SECTION 3

Mapping the Ethical Landscape

The following diagram maps the principal ethical and political frameworks onto the dimensions of algorithmic evaluation they prioritize. The horizontal axis represents the unit of analysis—whether the framework evaluates individuals or collectives—while the vertical axis represents whether the framework emphasizes procedures (how decisions are made) or outcomes (what results are produced). Understanding where each tradition sits in this conceptual space clarifies why different analysts reach different conclusions about the same algorithm.

The diagram positions six ethical and political frameworks along two axes. Deontology and Epistemic Justice foreground the individual, while Utilitarianism and Procedural Fairness attend to collective or systemic dimensions. Frameworks near the top prioritize how decisions are made; those near the bottom prioritize the consequences of decisions.

Notice that no single framework occupies the center of this space. This spatial separation reflects genuine philosophical tension: an algorithm optimized for aggregate accuracy (lower-right quadrant) may violate individual rights (upper-left quadrant). The COMPAS case is instructive—when ProPublica argued the system was biased against Black defendants, they emphasized its unequal false positive rates across racial groups, a criterion closely linked to procedural and deontological concerns. Northpointe (the system's developer) countered that the algorithm satisfied predictive parity, a criterion more aligned with consequentialist reasoning about calibration. Both sides were correct within their chosen framework, which is precisely why philosophical analysis—not merely statistical analysis—is essential.

SECTION 4

How Algorithmic Bias Operates

Understanding how bias enters and propagates through algorithmic systems requires tracing the pipeline from data collection to deployment. At each stage, normative choices—often invisible—shape who benefits and who is harmed. Philosophers distinguish between at least three sources of algorithmic bias: pre-existing bias embedded in training data, technical bias introduced through model design choices, and emergent bias that arises when a system interacts with social contexts its designers did not anticipate.

Formal Fairness Criteria

While this lesson privileges philosophical over mathematical analysis, understanding the formal definitions of fairness is necessary for evaluating the normative commitments they encode. Each definition operationalizes a different moral intuition about what "treating people fairly" means in a predictive context.

DEMOGRAPHIC PARITY

P(Ŷ = 1 | A = a) = P(Ŷ = 1 | A = b)

Where Ŷ is the predicted outcome, A is a protected attribute (e.g., race), and a and b are distinct groups. This criterion requires that the selection rate be equal across groups—an operationalization of anti-classification or formal equality.

EQUALIZED ODDS

P(Ŷ = 1 | A = a, Y = y) = P(Ŷ = 1 | A = b, Y = y) ∀ y ∈ {0, 1}

Where Y is the true outcome. This criterion requires equal true positive rates and equal false positive rates across groups. It is closely tied to deontological concerns about treating individuals on the basis of their actual conduct rather than group membership.

PREDICTIVE PARITY (CALIBRATION)

P(Y = 1 | Ŷ = 1, A = a) = P(Y = 1 | Ŷ = 1, A = b)

This criterion requires that among those the algorithm classifies as positive, the proportion who truly are positive should be equal across groups. It reflects a consequentialist commitment to accuracy and the decision-maker's interest in reliable predictions.

⚠️ The Impossibility Result

Chouldechova (2017) proved that when base rates differ between groups—when the actual prevalence of the predicted outcome is unequal—demographic parity, equalized odds, and predictive parity cannot all be satisfied simultaneously. This mathematical impossibility means that choosing a fairness criterion is inescapably a normative act—a philosophical decision about which values to prioritize, not a technical optimization problem that admits a single correct answer.

SECTION 5

A Taxonomy of Algorithmic Bias

Bias can enter algorithmic systems at multiple points in the development lifecycle. The following diagram traces the pipeline from data collection through model deployment, identifying the characteristic forms of bias that arise at each stage and the philosophical concerns they raise. Recognizing where bias originates is essential for assigning moral responsibility and designing effective interventions.

This pipeline diagram illustrates how pre-existing bias in data, technical bias in model design, and emergent bias at deployment each connect to different philosophical concerns. The feedback loop at the top shows how biased outputs can become future training data, compounding injustice over time.

Sources of algorithmic bias and their ethical dimensions

Bias Type	Example	Relevant Ethical Framework
Pre-existing	A hiring algorithm trained on historical data where women were systematically underpromoted learns to penalize female applicants.	Distributive justice, epistemic justice
Technical	A credit scoring model uses zip code as a feature, which serves as a proxy for race due to residential segregation, even though race is not explicitly included.	Procedural fairness, deontological ethics
Emergent	A predictive policing algorithm directs officers to neighborhoods that are already over-policed, generating more arrest data from those areas and reinforcing the original pattern.	Non-domination, structural injustice

SECTION 6

Worked Example: Evaluating the COMPAS Algorithm

The COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm is used in several U.S. jurisdictions to predict the likelihood that a criminal defendant will reoffend. ProPublica's 2016 analysis revealed that the system's false positive rate—the rate at which it incorrectly labeled someone as high risk—was significantly higher for Black defendants than for white defendants. The following worked example demonstrates how to apply multiple ethical frameworks to evaluate this system.

Ethical Evaluation of COMPAS Recidivism Scores

Step 1 — Identify the Empirical FactsProPublica found that COMPAS had a false positive rate of approximately 44.9% for Black defendants versus 23.5% for white defendants. That is, Black defendants who did not go on to reoffend were nearly twice as likely to be incorrectly flagged as high risk. Meanwhile, Northpointe responded that the algorithm satisfied predictive parity: among those scored as high risk, roughly equal proportions of Black and white defendants actually reoffended.

Two fairness metrics yield conflicting assessments of the same system.

Step 2 — Apply a Deontological FrameworkFrom a Kantian perspective, each defendant has a right to be treated as an individual rather than as a member of a statistical group. The disparate false positive rate means that Black defendants who will not reoffend are disproportionately subjected to harsher treatment—longer sentences, higher bail, denied parole. This violates the categorical imperative's demand to treat persons as ends in themselves, never merely as means to systemic efficiency.

Deontological verdict: COMPAS is unjust because it imposes unequal burdens on innocent individuals based on group membership.

Step 3 — Apply a Utilitarian FrameworkA utilitarian might argue that predictive parity is the relevant criterion because it ensures that when the system assigns a "high risk" score, that score has the same meaning regardless of race—it carries the same probability of actual reoffending. If judges calibrate their decisions on this basis, the overall accuracy of the system is preserved, and aggregate social welfare (including public safety) is maximized. Under this view, disparate false positive rates are an unfortunate but tolerable consequence of differing base rates of recidivism.

Utilitarian verdict: COMPAS may be defensible if predictive parity maximizes overall decision quality.

Step 4 — Apply a Rawlsian FrameworkRawls's difference principle requires that social inequalities benefit the least advantaged members of society. If Black defendants are already structurally disadvantaged by the criminal justice system, then a system that imposes additional, disproportionate risk on them fails the Rawlsian test. Moreover, behind the veil of ignorance—not knowing one's race—rational agents would not consent to a system with such disparate error rates, because they could not be certain of belonging to the group that benefits.

Rawlsian verdict: COMPAS is unjust because it worsens the position of those who are already worst off.

Step 5 — Synthesize and Reflect on the Impossibility ResultThe conflicting verdicts from Steps 2–4 are not the result of sloppy reasoning; they are a philosophical expression of the mathematical impossibility theorem. Because base rates of recidivism differ between racial groups (a fact itself shaped by structural racism), no algorithm can simultaneously satisfy equalized odds and predictive parity. This means that choosing a fairness criterion is always a political act—a decision about whose interests to prioritize and which values to uphold. A responsible evaluation must therefore make its normative commitments explicit rather than hiding them behind technical language.

Synthesis: The choice of fairness metric is irreducibly normative; philosophical reasoning is required to justify it.

SECTION 7

Strengths & Limitations of Fairness Frameworks

Each ethical framework illuminates certain dimensions of algorithmic fairness while leaving others in shadow. A responsible evaluator must understand these trade-offs in order to justify their choice of framework and acknowledge its limitations. The table below summarizes the strengths and weaknesses of the major approaches when applied to algorithmic decision-making.

Comparative strengths and limitations of ethical frameworks applied to algorithmic fairness

Framework	Strengths	Limitations
Utilitarianism	Provides clear optimization targets; aligns with empirical measurement; can aggregate diverse impacts into a single metric for comparison.	May justify sacrificing minority interests for majority benefit; difficulty measuring well-being; insensitive to the distribution of harms.
Deontology	Protects individual rights; offers clear prohibitions (e.g., do not use race as a basis for unequal treatment); respects human dignity.	May be too rigid in complex contexts; prohibiting consideration of protected attributes can sometimes produce worse outcomes for the protected group itself.
Rawlsian Justice	Prioritizes the worst-off; provides a powerful thought experiment (veil of ignorance) for stress-testing institutional design.	Defining the "least advantaged" group is contested; the veil of ignorance abstracts away real social identities that may be ethically relevant.
Procedural Fairness	Focuses on transparency and accountability; encourages due process protections; can be operationalized through auditing.	A formally fair process can still produce substantively unjust outcomes, especially under conditions of structural inequality.
Non-Domination	Attends to power asymmetries; demands contestability and accountability for algorithmic governance.	Less developed as a formal framework for computational systems; can be difficult to operationalize in specific metrics.

✦ KEY TAKEAWAY

Choosing a fairness framework is analogous to choosing a map projection: every projection preserves some properties (area, shape, distance) while distorting others, and no single projection can preserve all properties simultaneously. Similarly, no single ethical framework captures every dimension of fairness. The Mercator projection is excellent for navigation but dramatically distorts the size of landmasses near the poles; utilitarianism is excellent for aggregate welfare analysis but dramatically distorts the claims of minorities. A responsible practitioner selects the framework whose distortions are most tolerable given the context—and explains that choice transparently.

SECTION 8

Connections to Structural Injustice & Critical Theory

While the frameworks discussed so far evaluate individual algorithmic systems, a growing body of philosophical work argues that the more fundamental problem is structural injustice—the ways in which social, economic, and political institutions interact to produce systematic disadvantage. On this view, associated most closely with Iris Marion Young's political philosophy, bias in an algorithm is not merely a technical defect to be corrected but a symptom of deeper structural conditions that the algorithm inherits, encodes, and reproduces. Critical race theory, feminist epistemology, and postcolonial thought all contribute additional analytical tools for understanding how power relations shape the categories, data, and design choices that algorithms rely upon.

Individual fairness vs. structural justice approaches

Dimension	Individual Fairness Approach	Structural Justice Approach
Unit of analysis	The algorithm and its immediate inputs/outputs	The broader sociotechnical system including institutions, markets, and cultural norms
Source of bias	Bad data, flawed model design, or misapplication	Historical and ongoing patterns of domination, exclusion, and exploitation
Remedy	Debiasing data, adjusting thresholds, enforcing fairness constraints	Institutional reform, redistribution, reparative justice, democratic governance of technology
Responsibility model	Designers, developers, and deployers of the specific system	Shared political responsibility across all agents who participate in unjust structures (Young's social connection model)

The structural approach does not invalidate individual-level fairness analysis; rather, it contextualizes it. Debiasing a hiring algorithm is valuable, but it cannot address the educational inequities, wealth gaps, and professional networks that shape the applicant pool in the first place. Students of applied philosophy should be prepared to move between these levels of analysis—using individual fairness criteria to evaluate specific systems while remaining attentive to the structural conditions that determine what counts as "fair" data, who designs the algorithms, and whose voices are included in governance decisions. Emerging work on participatory design and algorithmic reparations represents efforts to bridge these levels, incorporating the perspectives of affected communities into the design and governance of algorithmic systems from the outset.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain the distinction between procedural fairness and distributive justice as applied to algorithmic decision-making. Can an algorithm be procedurally fair but distributively unjust? Provide a concrete example.

PROBLEM 2 — BASIC APPLICATION

A university uses an algorithm to rank applicants for scholarship funding. The algorithm satisfies predictive parity: among applicants the algorithm identifies as "high potential," equal proportions from affluent and low-income backgrounds go on to graduate with honors. However, the false positive rate for low-income students is much higher—many low-income students predicted to fail actually succeed. Identify which fairness criterion is violated and explain why a Rawlsian philosopher would object to this system.

PROBLEM 3 — INTERMEDIATE

A city deploys a predictive policing algorithm that directs patrol officers to neighborhoods with historically high crime rates. Over time, arrest rates in those neighborhoods increase while remaining stable elsewhere, and the algorithm interprets this as confirmation that its predictions were correct. Analyze this scenario using the concepts of emergent bias, feedback loops, and the non-domination principle. What philosophical remedy might you propose?

PROBLEM 4 — APPLIED

A healthcare system uses an algorithm to identify patients who would benefit from enrollment in a care management program. Researchers discover that the algorithm uses healthcare spending as a proxy for health need, but because Black patients historically have less access to healthcare, their spending is lower even when their health conditions are equally or more severe. As a policy advisor, you must recommend a revision. Drawing on at least two ethical frameworks, argue for specific changes to the algorithm and explain the normative trade-offs involved.

PROBLEM 5 — CRITICAL THINKING

Some scholars argue that the entire framework of "debiasing" individual algorithms is inadequate because it treats algorithmic fairness as a technical problem rather than a political one. Iris Marion Young's social connection model of responsibility holds that all members of a society who participate in structures that produce injustice share responsibility for addressing it—not just the algorithm's designers. Critically evaluate this position. Is the structural approach to algorithmic injustice more philosophically defensible than the individual fairness approach, or do both approaches have indispensable roles? Defend your position with reference to specific cases and at least three philosophical concepts from this lesson.

SUMMARY

Lesson Summary

This lesson has demonstrated that evaluating algorithmic bias and fairness is inescapably a philosophical undertaking, not merely a technical one. We traced the historical development of concerns about algorithmic decision-making from Winner's political artifacts through the COMPAS controversy to contemporary regulation. We identified five core ethical and political frameworks—distributive justice, procedural fairness, non-domination, epistemic justice, and the consequentialist–deontological divide—and showed how each yields distinct evaluations of the same algorithm. The impossibility theorem proved that formal fairness criteria such as demographic parity, equalized odds, and predictive parity cannot all be satisfied simultaneously when base rates differ, making the choice of criterion an irreducibly normative decision.

We classified algorithmic bias into three types—pre-existing, technical, and emergent—and traced how feedback loops can compound injustice over time. The worked example on COMPAS illustrated how to apply multiple frameworks to a single case and synthesize their verdicts. Finally, we connected individual-level fairness analysis to the broader tradition of structural injustice theory, arguing that responsible evaluation requires attention to both algorithmic design and the institutional contexts in which algorithms operate. The central lesson is this: every technical choice in algorithm design encodes a philosophical commitment, and making those commitments explicit is the first step toward genuinely accountable artificial intelligence.