The Ethics of AI: Bias, Fairness and Accountability

An AI system trained on historical lending data denied loans to Black applicants at higher rates than white applicants with identical financial profiles. The company did not programme it to discriminate. But the historical data encoded decades of discriminatory lending practices, and the model learned from it. This is the core problem of AI ethics — not malice, but amplification of existing harm at machine scale.

What Is AI Ethics?

AI ethics is the study of what makes AI systems fair, transparent, accountable, and safe. It sits awkwardly between philosophy, law, and computer science — and has become urgent as AI moves from research labs into hiring decisions, medical diagnoses, credit scoring, and criminal sentencing.

When I started looking into this field seriously, the term “ethics” felt abstract. What is actually meant is practical: who gets harmed by an AI system, why does it happen, and who is responsible? Those are engineering and legal questions as much as moral ones.

UK organisations are now being asked these questions by regulators. The ICO, FCA, and NHS have all issued guidance on algorithmic decision-making. The question is not whether your company uses AI — it is whether you have checked it for harm.

Where AI Bias Comes From

Bias in AI does not usually come from intent. It comes from data. Three sources dominate.

Historical data is the first problem. AI systems learn from the past. If historical hiring data shows that senior roles were mostly held by men, a model trained on it will predict that male candidates are better fits for senior roles. The algorithm did not invent sexism. It encoded it.

Labelling is the second. Many AI systems rely on humans to flag images, categorise text, or mark decisions as correct or incorrect. When the labellers share similar backgrounds, those perspectives get baked into the model. Amazon famously built a CV-screening tool that penalised CVs from all-women’s colleges because its labellers preferred candidates that matched historical hires. The tool was scrapped.

Underrepresentation causes the third category of problems. When certain groups appear rarely in training data, AI performs badly for those groups. Facial recognition systems trained mostly on lighter-skinned faces have error rates 10 to 34 times higher for darker-skinned faces, according to a 2019 MIT Media Lab study. The technology is not fundamentally wrong — it is undertrained on people outside the dominant demographic.

Real-World AI Bias: Four Cases That Shocked the Industry

Abstract principles become clearer through concrete examples. These four cases illustrate how AI bias moves from theory to real-world harm.

Apple Card in 2019 gave men significantly higher credit limits than their wives, even when the wife had a better credit score. Neither Goldman Sachs nor Apple had programmed this in. When investigated, neither company could explain the algorithm’s reasoning — which created additional regulatory problems on top of the discrimination concern.

The COMPAS recidivism tool, used in US courts to predict criminal reoffending, was found by ProPublica to be twice as likely to incorrectly flag Black defendants as future criminals compared to white defendants with similar profiles. It was deployed in courts across multiple states before this was discovered through investigative journalism.

UK A-levels in 2020 saw a government algorithm systematically downgrade students from state schools relative to those from private schools. Around 40% of teacher-predicted grades were lowered by the algorithm. After mass protests, the results were scrapped and teacher assessments reinstated. The political and personal damage had already been done.

A widely studied healthcare algorithm used in US hospitals gave lower risk scores to Black patients with identical symptoms to white patients, because it used healthcare spending as a proxy for health need. Black patients historically spent less due to lower access to care — so the model underestimated their medical needs, resulting in fewer referrals for serious conditions.

Fairness: A Surprisingly Hard Definition

You might think fairness is obvious — treat everyone equally. It is not. There are mathematically incompatible definitions of fairness, and you cannot satisfy all of them simultaneously.

Demographic parity means equal outcomes across groups. If 20% of loan applicants are approved overall, exactly 20% from every demographic group should be approved. Equalised odds means equal accuracy across groups — the model should be equally good at correctly predicting outcomes for all groups. Individual fairness means similar individuals get similar treatment.

The problem: achieving demographic parity often requires different accuracy across groups. A 2016 paper from Cornell and IBM proved mathematically that you cannot achieve all three definitions simultaneously in most real-world scenarios. This is not a bug — it is a fundamental constraint. Choosing between them requires a value judgement.

Most companies have not made this choice explicitly. They have shipped the product without deciding which definition of fairness they are optimising for. That is not a technical failure — it is a governance failure.

Accountability: Who Is Responsible When AI Gets It Wrong?

Responsibility for AI harm is murky in UK law right now. The Equality Act 2010 makes discrimination illegal regardless of whether a human or algorithm makes the decision. But proving discrimination by an algorithm is hard. You need access to the system’s decision data, and companies often refuse disclosure on grounds of commercial sensitivity.

The EU AI Act introduces a liability framework for high-risk AI systems — covering healthcare, employment, credit, education, and law enforcement. UK companies selling into Europe face these requirements from 2026. The Act requires documentation, bias testing, transparency, and human oversight for covered systems.

The UK government’s approach, as of mid-2026, is pro-innovation and sector-specific rather than a single AI act. The FCA regulates AI in financial services. The ICO covers data-driven decisions. The CQC is developing healthcare AI frameworks. Critics argue this fragmentation creates compliance complexity and coverage gaps.

In practice, accountability often falls on the organisation deploying the AI, not the company that built it. If a UK employer uses an AI recruitment tool that discriminates, the employer faces the Equality Act claim — even if the tool was built and sold by a US technology company. Vendor contracts rarely adequately address this risk.

What Good AI Governance Actually Looks Like

Organisations doing AI ethics well tend to share seven practices. They document what data was used to train AI systems and audit it for representation gaps. They test AI outputs across demographic groups before deployment — not after. They create a human review process for AI decisions that significantly affect people’s lives.

They also make it straightforward for affected individuals to contest AI decisions. They publish their methodology or at minimum allow independent audits. They have a named senior person accountable for AI decision-making. And they review systems regularly — deployment is not the end of the process.

Companies doing this well include Anthropic, which publishes model cards detailing capabilities and limitations, and Google DeepMind, which maintains an external research ethics board. The majority of companies deploying AI commercially do none of this systematically.

Your Rights Under UK GDPR and AI Decisions

UK GDPR gives individuals specific rights related to automated decision-making. If a decision is made about you solely through automated processing — no human involved — and it produces a legal or similarly significant effect, you have the right to request human review. You also have the right to meaningful information about the logic behind the decision.

Significant decisions include credit applications, insurance pricing, job shortlisting, and benefit eligibility assessments. If you have been refused credit or a job and suspect an algorithm was involved, you can ask the company whether automated processing was used. They are legally required to tell you. If it was, you can request a human review.

The ICO can investigate complaints and fine organisations that fail to comply. Fines under UK GDPR can reach £17.5 million or 4% of global annual turnover, whichever is higher. Awareness of these rights is the first line of defence.

What This Means for You

AI ethics is not an abstract academic debate. If you have applied for a mortgage, a job, an insurance policy, or benefits in the past five years, there is a reasonable chance an algorithm contributed to the decision. You have the right to know when automated decision-making was involved and the right to request human review of significant decisions.

For businesses using AI tools, the practical message is direct: find out what your AI vendor can tell you about bias testing and fairness methodology. If they cannot answer those questions, that is your answer. In a regulatory environment that is tightening, understanding your AI systems now is far cheaper than a discrimination investigation later.

This article is for educational purposes only and does not constitute financial advice.