Algorithmic Bias & Fairness — I can evaluate algorithmic bias and fairness using ethical and political concepts at my level. - Philosophy

SECTION 1

Historical Context & Motivation

The question of whether technologies can be politically or morally neutral is not new, but the rapid deployment of algorithmic decision-making systems in hiring, criminal sentencing, credit scoring, and healthcare has given it unprecedented urgency. Philosophers of technology have long argued that artifacts embed values—Langdon Winner's 1980 essay "Do Artifacts Have Politics?" examined how the physical design of bridges and power plants could reinforce social hierarchies. As machine-learning models began to replace or augment human judgment in high-stakes domains during the 2010s, researchers discovered that these systems could reproduce and even amplify patterns of racial, gender, and socioeconomic discrimination present in their training data. The result is a rich, ongoing conversation that sits at the intersection of applied ethics, political philosophy, computer science, and law.

1980 • Winner's "Do Artifacts Have Politics?"

Langdon Winner argues that technological artifacts can embody political arrangements, establishing a philosophical foundation for questioning the neutrality of designed systems.

2014 • ProPublica Investigates COMPAS

Investigative journalists begin analyzing the COMPAS recidivism algorithm, later publishing findings in 2016 showing significant racial disparities in risk scores assigned to Black and white defendants.

2016 • Impossibility Theorems Emerge

Researchers including Chouldechova and Kleinberg et al. formally prove that certain fairness criteria are mathematically incompatible, demonstrating that fairness is not merely a technical problem but a normative one.

2018 • Gender Shades Study

Joy Buolamwini and Timnit Gebru reveal that commercial facial recognition systems have significantly higher error rates for darker-skinned women, sparking widespread concern about intersectional bias in AI.

2023–2024 • EU AI Act & Global Regulation

The European Union adopts the AI Act, the first comprehensive regulatory framework classifying AI systems by risk level and mandating fairness audits for high-risk applications.

These developments raise a central question for applied philosophy: By what ethical and political standards should we evaluate the fairness of algorithmic systems? The challenge is not merely identifying when an algorithm produces disparate outcomes, but articulating what fairness requires in contexts where different values—accuracy, equality, liberty, desert—may conflict with one another. This lesson equips you to engage that challenge by connecting formal definitions of fairness to the philosophical traditions that give them meaning.

SECTION 2

Core Ethical & Political Principles

Evaluating algorithmic fairness requires drawing on multiple ethical and political frameworks, each of which foregrounds different values and diagnoses bias differently. No single framework captures all dimensions of the problem, which is precisely why philosophical analysis is indispensable. Below are the foundational concepts that structure contemporary debates about algorithmic justice.

Distributive Justice

Concerned with how benefits and burdens are allocated across a population. Rawlsian approaches ask whether an algorithm's outcomes could be endorsed from behind a veil of ignorance, while utilitarian approaches evaluate whether total welfare is maximized.

Procedural Fairness

Focuses on whether the process by which decisions are made is just, regardless of outcomes. A procedurally fair algorithm should treat relevantly similar cases similarly and exclude morally irrelevant features such as race or gender from its decision logic.

Non-Domination

Drawing on republican political theory, this principle holds that no agent—including an algorithm—should exercise arbitrary power over others. Algorithmic systems that are opaque and unaccountable may dominate even when their outputs appear fair.

Epistemic Justice

Introduced by Miranda Fricker, this concept highlights harms inflicted when a person's testimony or knowledge is discounted. Algorithms trained on historically biased data may perpetuate testimonial injustice by systematically undervaluing the needs or inputs of marginalized groups.

Consequentialism vs. Deontology

Consequentialist evaluations ask whether an algorithm produces the best overall outcomes, while deontological evaluations ask whether it respects individual rights and duties, regardless of aggregate consequences. These frameworks often yield conflicting verdicts on the same system.

✦

KEY TAKEAWAY

Think of fairness as a prism rather than a lens. When white light passes through a prism, it separates into distinct colors—just as a single algorithmic outcome, when examined through different ethical frameworks, reveals distinct dimensions of fairness that may conflict with one another. A system that is "fair" under a utilitarian criterion (maximizing aggregate accuracy) may be deeply unfair under a Rawlsian criterion (protecting the least advantaged group). Recognizing this irreducible pluralism is the first step toward responsible evaluation.

SECTION 3

Mapping the Ethical Landscape

The following diagram maps the principal ethical and political frameworks onto the dimensions of algorithmic evaluation they prioritize. The horizontal axis represents the unit of analysis—whether the framework evaluates individuals or collectives—while the vertical axis represents whether the framework emphasizes procedures (how decisions are made) or outcomes (what results are produced). Understanding where each tradition sits in this conceptual space clarifies why different analysts reach different conclusions about the same algorithm.

The diagram positions six ethical and political frameworks along two axes. Deontology and Epistemic Justice foreground the individual, while Utilitarianism and Procedural Fairness attend to collective or systemic dimensions. Frameworks near the top prioritize how decisions are made; those near the bottom prioritize the consequences of decisions.

Notice that no single framework occupies the center of this space. This spatial separation reflects genuine philosophical tension: an algorithm optimized for aggregate accuracy (lower-right quadrant) may violate individual rights (upper-left quadrant). The COMPAS case is instructive—when ProPublica argued the system was biased against Black defendants, they emphasized its unequal false positive rates across racial groups, a criterion closely linked to procedural and deontological concerns. Northpointe (the system's developer) countered that the algorithm satisfied predictive parity, a criterion more aligned with consequentialist reasoning about calibration. Both sides were correct within their chosen framework, which is precisely why philosophical analysis—not merely statistical analysis—is essential.

SECTION 4

How Algorithmic Bias Operates

Understanding how bias enters and propagates through algorithmic systems requires tracing the pipeline from data collection to deployment. At each stage, normative choices—often invisible—shape who benefits and who is harmed. Philosophers distinguish between at least three sources of algorithmic bias: pre-existing bias embedded in training data, technical bias introduced through model design choices, and emergent bias that arises when a system interacts with social contexts its designers did not anticipate.

Formal Fairness Criteria

While this lesson privileges philosophical over mathematical analysis, understanding the formal definitions of fairness is necessary for evaluating the normative commitments they encode. Each definition operationalizes a different moral intuition about what "treating people fairly" means in a predictive context.

DEMOGRAPHIC PARITY

P(Ŷ = 1 | A = a) = P(Ŷ = 1 | A = b)

Where Ŷ is the predicted outcome, A is a protected attribute (e.g., race), and a and b are distinct groups. This criterion requires that the selection rate be equal across groups—an operationalization of anti-classification or formal equality.

EQUALIZED ODDS

P(Ŷ = 1 | A = a, Y = y) = P(Ŷ = 1 | A = b, Y = y) ∀ y ∈ {0, 1}

Where Y is the true outcome. This criterion requires equal true positive rates and equal false positive rates across groups. It is closely tied to deontological concerns about treating individuals on the basis of their actual conduct rather than group membership.

PREDICTIVE PARITY (CALIBRATION)

P(Y = 1 | Ŷ = 1, A = a) = P(Y = 1 | Ŷ = 1, A = b)

This criterion requires that among those the algorithm classifies as positive, the proportion who truly are positive should be equal across groups. It reflects a consequentialist commitment to accuracy and the decision-maker's interest in reliable predictions.

⚠

The Impossibility Result

Chouldechova (2017) proved that when base rates differ between groups—when the actual prevalence of the predicted outcome is unequal—demographic parity, equalized odds, and predictive parity cannot all be satisfied simultaneously. This mathematical impossibility means that choosing a fairness criterion is inescapably a normative act—a philosophical decision about which values to prioritize, not a technical optimization problem that admits a single correct answer.

SECTION 5

A Taxonomy of Algorithmic Bias

Bias can enter algorithmic systems at multiple points in the development lifecycle. The following diagram traces the pipeline from data collection through model deployment, identifying the characteristic forms of bias that arise at each stage and the philosophical concerns they raise. Recognizing where bias originates is essential for assigning moral responsibility and designing effective interventions.

This pipeline diagram illustrates how pre-existing bias in data, technical bias in model design, and emergent bias at deployment each connect to different philosophical concerns. The feedback loop at the top shows how biased outputs can become future training data, compounding injustice over time.

Bias Type	Example	Relevant Ethical Framework
Pre-existing	A hiring algorithm trained on historical data where women were systematically underpromoted learns to penalize female applicants.	Distributive justice, epistemic justice
Technical	A credit scoring model uses zip code as a feature, which serves as a proxy for race due to residential segregation, even though race is not explicitly included.	Procedural fairness, deontological ethics
Emergent	A predictive policing algorithm directs officers to neighborhoods that are already over-policed, generating more arrest data from those areas and reinforcing the original pattern.	Non-domination, structural injustice

Sources of algorithmic bias and their ethical dimensions

SECTION 6

Worked Example: Evaluating the COMPAS Algorithm

The COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm is used in several U.S. jurisdictions to predict the likelihood that a criminal defendant will reoffend. ProPublica's 2016 analysis revealed that the system's false positive rate—the rate at which it incorrectly labeled someone as high risk—was significantly higher for Black defendants than for white defendants. The following worked example demonstrates how to apply multiple ethical frameworks to evaluate this system.

Ethical Evaluation of COMPAS Recidivism Scores

1
Step 1 — Identify the Empirical Facts

ProPublica found that COMPAS had a false positive rate of approximately 44.9% for Black defendants versus 23.5% for white defendants. That is, Black defendants who did not go on to reoffend were nearly twice as likely to be incorrectly flagged as high risk. Meanwhile, Northpointe responded that the algorithm satisfied predictive parity: among those scored as high risk, roughly equal proportions of Black and white defendants actually reoffended.

Two fairness metrics yield conflicting assessments of the same system.
2
Step 2 — Apply a Deontological Framework

From a Kantian perspective, each defendant has a right to be treated as an individual rather than as a member of a statistical group. The disparate false positive rate means that Black defendants who will not reoffend are disproportionately subjected to harsher treatment—longer sentences, higher bail, denied parole. This violates the categorical imperative's demand to treat persons as ends in themselves, never merely as means to systemic efficiency.

Deontological verdict: COMPAS is unjust because it imposes unequal burdens on innocent individuals based on group membership.
3
Step 3 — Apply a Utilitarian Framework

A utilitarian might argue that predictive parity is the relevant criterion because it ensures that when the system assigns a "high risk" score, that score has the same meaning regardless of race—it carries the same probability of actual reoffending. If judges calibrate their decisions on this basis, the overall accuracy of the system is preserved, and aggregate social welfare (including public safety) is maximized. Under this view, disparate false positive rates are an unfortunate but tolerable consequence of differing base rates of recidivism.

Utilitarian verdict: COMPAS may be defensible if predictive parity maximizes overall decision quality.
4
Step 4 — Apply a Rawlsian Framework

Rawls's difference principle requires that social inequalities benefit the least advantaged members of society. If Black defendants are already structurally disadvantaged by the criminal justice system, then a system that imposes additional, disproportionate risk on them fails the Rawlsian test. Moreover, behind the veil of ignorance—not knowing one's race—rational agents would not consent to a system with such disparate error rates, because they could not be certain of belonging to the group that benefits.

Rawlsian verdict: COMPAS is unjust because it worsens the position of those who are already worst off.
5
Step 5 — Synthesize and Reflect on the Impossibility Result

The conflicting verdicts from Steps 2–4 are not the result of sloppy reasoning; they are a philosophical expression of the mathematical impossibility theorem. Because base rates of recidivism differ between racial groups (a fact itself shaped by structural racism), no algorithm can simultaneously satisfy equalized odds and predictive parity. This means that choosing a fairness criterion is always a political act—a decision about whose interests to prioritize and which values to uphold. A responsible evaluation must therefore make its normative commitments explicit rather than hiding them behind technical language.

Synthesis: The choice of fairness metric is irreducibly normative; philosophical reasoning is required to justify it.

SECTION 7

Strengths & Limitations of Fairness Frameworks

Each ethical framework illuminates certain dimensions of algorithmic fairness while leaving others in shadow. A responsible evaluator must understand these trade-offs in order to justify their choice of framework and acknowledge its limitations. The table below summarizes the strengths and weaknesses of the major approaches when applied to algorithmic decision-making.

Framework	Strengths	Limitations
Utilitarianism	Provides clear optimization targets; aligns with empirical measurement; can aggregate diverse impacts into a single metric for comparison.	May justify sacrificing minority interests for majority benefit; difficulty measuring well-being; insensitive to the distribution of harms.
Deontology	Protects individual rights; offers clear prohibitions (e.g., do not use race as a basis for unequal treatment); respects human dignity.	May be too rigid in complex contexts; prohibiting consideration of protected attributes can sometimes produce worse outcomes for the protected group itself.
Rawlsian Justice	Prioritizes the worst-off; provides a powerful thought experiment (veil of ignorance) for stress-testing institutional design.	Defining the "least advantaged" group is contested; the veil of ignorance abstracts away real social identities that may be ethically relevant.
Procedural Fairness	Focuses on transparency and accountability; encourages due process protections; can be operationalized through auditing.	A formally fair process can still produce substantively unjust outcomes, especially under conditions of structural inequality.
Non-Domination	Attends to power asymmetries; demands contestability and accountability for algorithmic governance.	Less developed as a formal framework for computational systems; can be difficult to operationalize in specific metrics.

Comparative strengths and limitations of ethical frameworks applied to algorithmic fairness

✦

KEY TAKEAWAY

Choosing a fairness framework is analogous to choosing a map projection: every projection preserves some properties (area, shape, distance) while distorting others, and no single projection can preserve all properties simultaneously. Similarly, no single ethical framework captures every dimension of fairness. The Mercator projection is excellent for navigation but dramatically distorts the size of landmasses near the poles; utilitarianism is excellent for aggregate welfare analysis but dramatically distorts the claims of minorities. A responsible practitioner selects the framework whose distortions are most tolerable given the context—and explains that choice transparently.

SECTION 8

Connections to Structural Injustice & Critical Theory

While the frameworks discussed so far evaluate individual algorithmic systems, a growing body of philosophical work argues that the more fundamental problem is structural injustice—the ways in which social, economic, and political institutions interact to produce systematic disadvantage. On this view, associated most closely with Iris Marion Young's political philosophy, bias in an algorithm is not merely a technical defect to be corrected but a symptom of deeper structural conditions that the algorithm inherits, encodes, and reproduces. Critical race theory, feminist epistemology, and postcolonial thought all contribute additional analytical tools for understanding how power relations shape the categories, data, and design choices that algorithms rely upon.

Dimension	Individual Fairness Approach	Structural Justice Approach
Unit of analysis	The algorithm and its immediate inputs/outputs	The broader sociotechnical system including institutions, markets, and cultural norms
Source of bias	Bad data, flawed model design, or misapplication	Historical and ongoing patterns of domination, exclusion, and exploitation
Remedy	Debiasing data, adjusting thresholds, enforcing fairness constraints	Institutional reform, redistribution, reparative justice, democratic governance of technology
Responsibility model	Designers, developers, and deployers of the specific system	Shared political responsibility across all agents who participate in unjust structures (Young's social connection model)

Individual fairness vs. structural justice approaches

The structural approach does not invalidate individual-level fairness analysis; rather, it contextualizes it. Debiasing a hiring algorithm is valuable, but it cannot address the educational inequities, wealth gaps, and professional networks that shape the applicant pool in the first place. Students of applied philosophy should be prepared to move between these levels of analysis—using individual fairness criteria to evaluate specific systems while remaining attentive to the structural conditions that determine what counts as "fair" data, who designs the algorithms, and whose voices are included in governance decisions. Emerging work on participatory design and algorithmic reparations represents efforts to bridge these levels, incorporating the perspectives of affected communities into the design and governance of algorithmic systems from the outset.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain the distinction between procedural fairness and distributive justice as applied to algorithmic decision-making. Can an algorithm be procedurally fair but distributively unjust? Provide a concrete example.

Show answer

Procedural fairness concerns whether the process by which decisions are made is just—for instance, whether the algorithm uses only morally relevant features and applies consistent rules across cases. Distributive justice concerns whether the outcomes of those decisions allocate benefits and burdens equitably. An algorithm can satisfy one without the other. For example, an algorithm that evaluates mortgage applications using only income, credit score, and debt-to-income ratio—without using race or gender—is procedurally fair in the anti-classification sense. However, because historical redlining and employment discrimination have produced stark racial wealth gaps, this procedurally neutral algorithm may systematically deny mortgages to Black applicants at higher rates, producing distributively unjust outcomes. The procedural neutrality of the inputs does not guarantee substantive equality in the results.

Question 2: BASIC APPLICATION

A university uses an algorithm to rank applicants for scholarship funding. The algorithm satisfies predictive parity: among applicants the algorithm identifies as "high potential," equal proportions from affluent and low-income backgrounds go on to graduate with honors. However, the false positive rate for low-income students is much higher—many low-income students predicted to fail actually succeed. Identify which fairness criterion is violated and explain why a Rawlsian philosopher would object to this system.

Show answer

The criterion violated is equalized odds (specifically, equal false positive rates). Although predictive parity holds—meaning the algorithm's positive predictions are equally reliable across groups—low-income students are disproportionately screened out by false negatives or, equivalently, the system imposes a higher burden of mistaken denial on them. A Rawlsian philosopher would invoke the difference principle: since low-income students are already the least advantaged group in terms of access to educational resources, a system that imposes additional, unequal costs on them is unjust. Behind the veil of ignorance, a rational agent would not consent to a scholarship algorithm that systematically disadvantages the group most in need of financial support, because they could not know whether they would belong to that group.

Question 3: INTERMEDIATE

A city deploys a predictive policing algorithm that directs patrol officers to neighborhoods with historically high crime rates. Over time, arrest rates in those neighborhoods increase while remaining stable elsewhere, and the algorithm interprets this as confirmation that its predictions were correct. Analyze this scenario using the concepts of emergent bias, feedback loops, and the non-domination principle. What philosophical remedy might you propose?

Show answer

This scenario illustrates emergent bias through a pernicious feedback loop. The algorithm was not necessarily biased at the moment of deployment, but its interaction with policing practices creates a self-fulfilling prophecy: more officers in a neighborhood means more observed (and therefore more recorded) crime, which feeds back into the algorithm and reinforces the prediction that the neighborhood is high-crime. The non-domination principle is violated because residents of these neighborhoods are subjected to a form of arbitrary power—they are surveilled and policed at heightened levels by a system they did not consent to, cannot contest, and that is insensitive to the structural causes of crime such as poverty and disinvestment. A philosophical remedy grounded in republican political theory would require, at minimum, mechanisms of contestability: affected communities should have the ability to challenge and override algorithmic recommendations through democratic deliberation, and the system should be subject to independent auditing that accounts for feedback effects rather than treating arrest data as ground truth.

Question 4: APPLIED

A healthcare system uses an algorithm to identify patients who would benefit from enrollment in a care management program. Researchers discover that the algorithm uses healthcare spending as a proxy for health need, but because Black patients historically have less access to healthcare, their spending is lower even when their health conditions are equally or more severe. As a policy advisor, you must recommend a revision. Drawing on at least two ethical frameworks, argue for specific changes to the algorithm and explain the normative trade-offs involved.

Show answer

This real-world case (documented by Obermeyer et al., 2019) exemplifies pre-existing bias: the algorithm uses a technically measurable variable (healthcare spending) that serves as a biased proxy for the morally relevant variable (health need). From a deontological perspective, the system violates the principle of equal treatment because it denies equally sick Black patients the same access to care management as white patients; the morally irrelevant factor of historical access to healthcare is effectively functioning as a basis for discrimination. From a distributive justice perspective—whether Rawlsian or capabilities-based—the algorithm worsens the position of those who are already disadvantaged, violating the difference principle and undermining the capability of sick individuals to achieve basic health functioning. A recommended revision would replace healthcare spending with a direct measure of health need, such as the number and severity of active diagnoses, lab results, or patient-reported symptoms. The trade-off is that direct health measures may be more expensive to collect and may themselves reflect biases (e.g., if Black patients receive fewer diagnostic tests), requiring ongoing monitoring. A utilitarian might also note that treating sicker patients earlier reduces overall system costs, so the revision is arguably welfare-enhancing across the board. The normative trade-off lies in the cost of data infrastructure versus the justice of the outcomes: investing in better data collection is costly, but failing to do so perpetuates a system that compounds historical injustice.

Question 5: CRITICAL THINKING

Some scholars argue that the entire framework of "debiasing" individual algorithms is inadequate because it treats algorithmic fairness as a technical problem rather than a political one. Iris Marion Young's social connection model of responsibility holds that all members of a society who participate in structures that produce injustice share responsibility for addressing it—not just the algorithm's designers. Critically evaluate this position. Is the structural approach to algorithmic injustice more philosophically defensible than the individual fairness approach, or do both approaches have indispensable roles? Defend your position with reference to specific cases and at least three philosophical concepts from this lesson.

Show answer

A strong response to this question recognizes that both approaches are necessary and that the tension between them is productive rather than resolvable. The structural approach is philosophically compelling because it correctly identifies that algorithms do not operate in a vacuum: the COMPAS algorithm's disparate error rates are partly a consequence of differential base rates of recidivism, which are themselves products of centuries of structural racism in housing, education, and employment. Young's social connection model appropriately broadens the circle of moral responsibility beyond algorithm designers to include legislators, employers, educators, and citizens who sustain unjust institutions. However, the structural approach alone is insufficient for practical ethics. It risks a form of moral paralysis: if everyone is responsible, accountability can diffuse to the point where no one takes corrective action. Individual fairness criteria—despite their limitations and mutual incompatibility—provide concrete, actionable standards that can be implemented, audited, and legally enforced. The impossibility theorem does not invalidate these criteria; it simply requires that their selection be justified through philosophical argument rather than presented as value-neutral technical optimization. Procedural fairness mechanisms such as transparency, contestability, and participatory design can bridge the gap: they address structural concerns about power and voice while also producing tangible improvements in specific systems. The most philosophically defensible position, therefore, integrates both levels: use individual fairness criteria as tools for evaluating specific algorithms while embedding those evaluations within a broader program of structural reform that addresses the root causes of biased data and unequal access to technological governance.

SUMMARY

Lesson Summary

Summary

This lesson has demonstrated that evaluating algorithmic bias and fairness is inescapably a philosophical undertaking, not merely a technical one. We traced the historical development of concerns about algorithmic decision-making from Winner's political artifacts through the COMPAS controversy to contemporary regulation. We identified five core ethical and political frameworks—distributive justice, procedural fairness, non-domination, epistemic justice, and the consequentialist–deontological divide—and showed how each yields distinct evaluations of the same algorithm. The impossibility theorem proved that formal fairness criteria such as demographic parity, equalized odds, and predictive parity cannot all be satisfied simultaneously when base rates differ, making the choice of criterion an irreducibly normative decision.

We classified algorithmic bias into three types—pre-existing, technical, and emergent—and traced how feedback loops can compound injustice over time. The worked example on COMPAS illustrated how to apply multiple frameworks to a single case and synthesize their verdicts. Finally, we connected individual-level fairness analysis to the broader tradition of structural injustice theory, arguing that responsible evaluation requires attention to both algorithmic design and the institutional contexts in which algorithms operate. The central lesson is this: every technical choice in algorithm design encodes a philosophical commitment, and making those commitments explicit is the first step toward genuinely accountable artificial intelligence.