ALGORITHMIC BIAS AND DISCRIMINATION

INTRODUCTION

Algorithmic systems increasingly influence decisions in employment, finance, healthcare, criminal justice, and beyond. These systems were designed to capture the knowledge and thought processes of legal experts and provide automated decision-making tools. While they promise efficiency and impartiality, their reliance on historical data and the biases embedded within these datasets often result in discriminatory outcomes. For instance, facial recognition technology has been criticized for higher error rates when identifying individuals of certain ethnic groups, and predictive policing algorithms may disproportionately target minority communities. The rapid integration of artificial intelligence (AI) and machine learning into decision-making processes has revolutionized industries. However, these systems are not immune to bias and discrimination, which can have significant social and legal implications.

The integration of Artificial Intelligence into the legal system has brought about both opportunities for enhanced efficiency and impartiality as well as new challenges, particularly in the form of algorithm and discrimination biases. Algorithm bias stems from inherent errors in AI model creation, training, or implementation, often mirroring the existing societal inequities found in training data or decision-making frameworks. Conversely, discrimination bias leads to the unfair treatment of individuals or groups due to these algorithmic biases, resulting in unjust or prejudiced outcomes within legal contexts. In the realm of legal processes, biases can surface in tools employed for predictive policing, sentence suggestions, and risk evaluations, potentially reinforcing societal inequalities.

THE FUTURE OF ARTIFICIAL INTELLIGENCE

Artificial Intelligence (AI) is no longer a futuristic abstraction. It is now embedded in the everyday decision making of governments, corporations, and institutions. From recruitment algorithms and predictive policing to credit scoring and welfare distribution, AI-powered systems are rapidly transforming how rights are exercised, resources are allocated, and power is exercised. While these technologies promise greater efficiency, objectivity, and scale, they also carry a significant risk: the reproduction and amplification of existing social biases. 

Besides, India stands at a critical crossroads. With initiatives like Aadhaar, Digital India, and the National AI Mission, the country is embracing AI in governance and service delivery. Yet, despite the Constitution’s robust guarantees of equality (Articles 14 and 15) and dignity (Article 21), there exists a legal vacuum regarding algorithmic harms. Unlike the European Union, which has introduced a comprehensive Artificial Intelligence Act, or the United States, which is considering targeted accountability legislation, India lacks a statutory framework to regulate bias in Artificial Intelligence.

ALGORITHMIC BIAS

Algorithmic bias refers to systematic and repeatable errors in automated systems that result in unfair or discriminatory outcomes often privileging certain groups while disadvantaging others. Despite the common perception of algorithms as objective and data-driven, they are in fact socio-technical constructs, shaped by human choices, institutional norms, and historical data. Bias in AI does not emerge in a vacuum; it is often a reflection of the real-world inequalities embedded in the data on which these systems are trained, or in the design assumptions of developers. There are three major sources of algorithmic bias:

1. Data bias - This occurs when datasets reflect historical prejudices, stereotypes or underrepresentation of certain communities. For instance, if past hiring data shows a preference for male candidates, an AI model trained on such data may learn to replicate and reinforce such gender biases.
2. Design Bias - It emerges from the unconscious assumptions or values embedded in the structure or logic of an algorithm. This may happen when developers fail to account for social diversity, or overlook intersectional vulnerabilities during system design and testing.
3. Feedback Loops -  It arises when biased outcomes from algorithms are fed back into the system as new data, reinforcing and amplifying the original distortions. This is especially common in predictive policing or credit scoring, where past biased decisions influence future risk assessments.

CASE STUDIES OF ALGORITHMIC BIAS

COMPAS (USA) - The Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), a risk assessment tool used in the U.S. criminal justice system, was found to disproportionately rate Black defendants as high-risk for reoffending compared to white defendants with similar profiles. A 2016 investigation by ProPublica revealed significant racial disparities, sparking widespread concern about the use of opaque AI in sentencing and bail decisions.

AMAZON'S HIRING TOOL - In 2018, Amazon discontinued an experimental AI recruiting tool after discovering that it systematically downgraded resumes containing the word “women’s” (e.g., “women’s chess club”), as well as candidates from women’s colleges. This occurred because the model had been trained on historical hiring data that reflected the company’s male-dominated workforce, thus internalizing and reproducing existing gender imbalances. 

Aadhaar Welfare Exclusions (India) - India's Aadhaar biometric authentication system, though designed to streamline welfare delivery, has faced criticism for excluding large numbers of beneficiaries. Technical errors in fingerprint or iris scans disproportionately affect manual laborer, the elderly, and persons with disabilities populations often belonging to marginalized communities. Reports and field studies have highlighted how these exclusions, rooted in technological failure, can amount to a denial of social and economic rights.

INDIA’S PURVIEW

India’s constitutional architecture strongly supports the principles of equality, nondiscrimination, and due process: Article 14 of the Constitution guarantees equality before the law and equal protection of the laws. Article 15 prohibits discrimination on grounds such as religion, race, caste, sex, or place of birth.  Article 21, as interpreted in Justice K.S. Puttaswamy v. Union of India (2017), recognizes the right to privacy as a fundamental right, encompassing informational autonomy and dignity. However, these constitutional guarantees have not been extended to specifically address harms caused by algorithmic capacity, bias, or exclusion. As of now, Indian jurisprudence has not developed doctrines that apply constitutional scrutiny to automated decision-making by the state or private actors. The Digital Personal Data Protection Act, 2023 marks a step toward regulating data use and privacy. It introduces obligations regarding purpose limitation, data minimization, and consent. However, the law does not address algorithmic fairness, explainability, or human oversight, nor does it mandate algorithmic audits or impact assessments. Moreover, the Act focuses predominantly on personal data protection, leaving significant gaps in governing AI systems used in public governance, criminal justice, or private employment. India has no comprehensive or sector-specific AI legislation, and no legally mandated standards currently exist for testing algorithmic bias, redressing harm, or enforcing accountability in AI systems. This regulatory vacuum poses serious risks to the constitutional commitment to social justice and equal treatment. 

LIABILITY IN ALGORITHMIC DISCRIMINATION

One of the most pressing and complex legal issues in regulating AI is the attribution of liability for harms caused by algorithmic discrimination. Traditional legal frameworks rooted in tort law, contract law, and statutory obligations are often ill-suited to capture the unique characteristics of AI systems. Central to this challenge is the opacity of decision-making processes (the "black box" problem), compounded by the diffuse nature of responsibility across various stakeholders such as developers, data scientists, platform providers, and end-users. 

SOME OF THE MAJOR LEGAL CHALLENGES

1. Causation - Proving a direct link between a discriminatory output and a specific action or input is difficult due to the dynamic and adaptive nature of algorithms. 
2. Mens rea or Intent - Most legal systems require some element of intent or knowledge in establishing liability, which is often absent in automated decision-making systems. 
3. Multiplicity of Actors - AI systems involve multiple layers of actors such as designers, data providers, and algorithm trainers, making it difficult to pinpoint a single responsible party. 

A WAY FORWARD MITIGATING STRATEGIES

Upon recognizing the elements that generate and influence Algorithmic Bias, it is clear that global laws and regulations must evolve accordingly in their continuous endeavors to govern AI operations comprehensively. Several organizations worldwide have undertaken significant initiatives in this regard. One of the notable steps taken by European Union is the General Data Protection Regulation. The GDPR was enforced in May 2018. It does not inherently use the term “algorithm bias” directly but there are some provisions which indirectly applies to algorithm bias. European Union (EU) data protection authorities have the power to investigate and fine companies if the algorithm causes harm or discrimination based on gender, sex, place of birth, and so on.

ALGORITHMIC IMPACT ASSESSMENTS AND DATA AUDITS GOVERNMENTS AND THE COMMUNITY MEMBERS

They serve lack the necessary information to assess how algorithms are working and how they affect individuals, families and communities. Requiring companies and public agencies to conduct “algorithmic impact assessments” can help solve this problem. An impact assessment would require public agencies and other algorithmic operators to evaluate their automated decision systems and their impacts on fairness, justice, bias and other community concerns, and to consult with affected communities. Data audits involve having third parties examine algorithms and the underlying data for bias and to see if the algorithm is transparent, fair and its decisions are explainable. Data audits help developers ensure their algorithm works and that it complies with applicable laws and ethical norms. Audits and impact assessments can help stop companies from taking shortcuts in developing and implementing their algorithms and ensure that they confront the implications of their tool before it goes live. These practices can also build trust and public confidence that the algorithm is fair and works as intended.

CONCLUSION 

Algorithms and automated decisions are powerful, pervasive and often unfair, inaccurate and discriminatory. This push for legislative action presents an opportunity to not only develop policies that minimize unfair algorithmic discrimination but also to create a system where decision-makers optimize algorithms for equity and inclusion, and design them in ways that drive investments to the most vulnerable communities and use them to build a better and more equal society. Develop laws that integrate equity metrics into public decision-making algorithms, particularly for those that govern access to community investment and resource allocation. Engage and fund equity mapping and community-cantered data gathering efforts to track the cumulative impacts of exposure to health disparities, economic inequality and pollution as a way to inform equitable policy implementation. India’s constitutional promise of equality under Articles 14 and 15, and the evolving right to informational privacy under Article 21, remain underutilized in algorithmic contexts. Without a robust and specific legal regime, the potential of AI to discriminate will go unchecked, undermining democratic values and human dignity. Therefore, legal accountability for AI is not merely a technical or regulatory issue, it is a constitutional imperative.

💬Law Notes

Author: Mr. Hany Sabri

5th Year Law Student - B.COM-LLB (HONS)

Marwadi University - India