Data Governance

Bias in AI: Algorithms with Machine Minds and Machine Hearts

By Matthew Teale, Data Analyst, Marketing Metrix

Only a brave person would rationally accuse algorithm-based Artificial Intelligence (AI) of lacking a heart.

Unfortunately, there have been many examples of AI producing thoroughly heartless outcomes; be that in the context of race, sex and various other aspects of society. This begs the question: if we passively accept AI injustice can we claim to have anything other than a machine heart? 

The issue is clothed in complexity. Often these algorithms are not overtly making decisions based on societal variables. For example, an algorithm that decides to grant more credit to a man than a woman is not necessarily using gender as a factor in its decision-making process.

The discrimination is a manifestation of wider societal bias which is a feature of the data the algorithm uses – the data used to ‘train’ the machine. In this case, the well reported income disparity between the sexes is the decisive factor in credit being withheld from women.

While AI can undoubtedly be a force for good, a fatalistic acceptance of bias in machine learning will only reinforce inequalities that have enduring historical roots. The time has come to stop hiding behind algorithms; there is a need to engage with the wider implications for society of unfettered AI. 

A history of AI bias

One of the earliest instances of algorithmic bias occurred in the late ’70s. Dr Geoffrey Franglen was frustrated by the time-consuming nature of the admissions process for St George’s Hospital Medical School.

His solution was to write an algorithm that closely replicated the actions of human assessors. This was thought to mark a triumph in fairness since the element of human error would be eradicated and each applicant would have a dispassionate non-bias machine deciding their fate.

The outcome of the algorithm diverted St. George’s far from the path of fairness. In the years following automation of admissions, staff became increasingly concerned about the lack of diversity among students. The UK Commission for Racial Equality launched an investigation which uncovered flagrant prejudice inherent in Franglen’s algorithm.

The Commission found that candidates would be classified as either “Caucasian” or “Non-Caucasian” based on their name and birthplace. Having a non-European name would knock 15 points off an applicant’s score.

This would potentially be compounded for females who were on average docked three points by virtue of their sex. Franglen had unintentionally created a racist and sexist monster.

Despite the widespread media attention of the St George’s case, instances of this kind of bias continually occur. The Commission ruled that race and ethnicity were inadmissible as factors for future algorithms to consider, but there are innumerable examples of algorithms using variables that act as a proxy for things like ethnicity or sex which produce the same unfair outcomes.

Recently, Apple co-founder Steve Wozniak commented how shocked he was to find out that he was offered a credit limit that was 10 times larger than that of his wife, despite sharing assets and accounts.

Meanwhile, Pro Publica, an investigative journalism outfit, found that software to predict future criminals had a heavy bias against African Americans. Last year a research paper from MIT illustrated how Amazon’s facial recognition software had a dramatically higher error rate for dark skinned women compared to light skinned males.

The history of AI is littered with examples of clear prejudice against the most marginalised groups in society. It is far from enlightening that the discussion around transparent AI has been lying dormant over several decades.

Why is there bias in AI algorithms?

The instances above are instructive as to the reasons why AI can produce such marked unequal treatment. Clearly in the case of St George’s, the fault lies with the algorithm using features like name and birthplace, which produced a discriminatory outcome.

The failing of the Apple Card algorithms is more subtle. It is unlikely the algorithms were explicitly using sex as a variable in the decision-making process, but since various studies have shown that the difference in median incomes between men and women is favourable for men, this being reflected in the training data meant men tend to be offered a higher credit limit.

Fair representation of ethnic minorities is a crucial ingredient missing from most algorithms’ training data. It is hardly surprising that Amazon’s AI software had a 0.8% error rate for light-skinned men compared to a 20% error rate for dark-skinned women, since the training data was 77% male and 83% white. 

Until there is full transparency regarding the inner workings of algorithms and data collection which is weighted more accurately to reflect the general population, the statistical community is doomed to repeat the failures of the past.

Why AI bias is a societal problem

The St George’s case of AI bias perfectly demonstrates the point that AI decisions invariably reflect the patterns embedded in society. In the years succeeding the introduction of Dr Franglen’s algorithm, student applications were double tested, comparing the results from the computer and what the human assessors’ decisions would have been.

The algorithm was in agreement with the human selection panel 90% to 95% of the time. What the algorithm served to do was make human discrimination more systematic and scientific, ensuring that all individuals were subjected to inherent prejudicial biases.

The fact this was done by a computer served to undermine any dissenting voices about the negative consequences of this admissions process, because how could you accuse a computer of human prejudice?

The sobering takeaway is that AI has untold potential to cement an inequitable status quo, since it can legitimise a discriminatory outcome, which would be unpalatable if the same result was generated from human actions. What is unacceptable in the form of a face, is deemed justifiable from a computer because there is a perception it is unscientific to argue against it.

A secondary and more fundamental problem with algorithms that produce a prejudicial result is the outcome becomes further woven into the fabric of society. There will be a negative reinforcement of the existing societal inequalities. If minority groups are discriminated against in financial markets, education and housing then there will be a persistent lack of opportunity presented to them.

There is a strong moral argument that the hallmark of a progressive society is the ability to nurture and develop the most disadvantaged groups. If not swayed by this, then there is also an argument on efficiency grounds. A society that holds back certain groups is clearly not efficiently utilising its resources.

Research the by OECD found that the single biggest impact on economic growth is income inequality and the main propagation mechanism is the lack of educational opportunities afforded to low-income groups. A care-free approach to the long-term impacts of AI decisions may be a threat to general prosperity. 

Computers are rightly seen as an advancement on human capability and even a humanising force for good. The crucial caveat being that if the algorithms are fed data points collected from a historical record of discrimination against women, people of colour and other minority groups, invariably the computer will assume it is desirable to reproduce this reality. The bleak consequence is the legitimation and reinforcement of prevailing societal injustices.

What can be done?

Demanding greater transparency regarding the mechanics and composition of training data seem like an obvious place to start. Amazon’s facial recognition error rate bias could be solved by a greater representation of both women and non-whites.

Unfortunately, the majority of algorithms are trained on historical data which serve as a record of the existence of a societal bias. As with the Apple Card bias, it is impossible to make an adjustment to financial history data – which echoes the financial disparity between men and women – without outright data tampering.

If it is impossible to reweight the data, it follows the key to the solution is greater transparency in the workings of the algorithms. This is where advocates of ‘white box’ machine learning come in.

The concept is in direct opposition to ‘black box’ or unsupervised learning, which essentially lets algorithms loose on the available information and affords the statistician a far more limited role in deciding which input features algorithms can use to achieve their goal.

The starting point is to ensure that there is an audit process whereby the variables that drive the algorithm’s decision making are not used in a way that can perpetuate bias.

A fairer future?

Positive steps have been made towards better AI scrutiny. In 2016, Google, Facebook and Apple formed a partnership with the intention of increasing research into AI ethics. Companies like IBM have even designed bias detection algorithms to counteract this problem.

The plethora of recent examples of AI bias points to the conclusion that there is undeniably more work to be done in this area. AI bias is the clearest indictment of a society which has hitherto allowed the growth of inequitable outcomes.

Dr. Rumman Chowdhury, Accenture’s lead for responsible AI, summed up the predicament aptly: “With societal bias, you can have perfect data and a perfect model, but we have an imperfect world.”

A fairer future begins with the acceptance that privilege and power is a greater driving force in distributing resources than merit and inherent ability. It hardly strains credulity that an algorithm that selects the best people for a job selects the same type of people that have historically occupied that job. Therefore, while there is of course a role for AI in society, our excitement must be tempered by foresight of its limitations and potential for regressive results.

About the author

Matthew Teale is head of analytics at data science consultancy Marketing Metrix, and leverages years of experience in model building and implementation of algorithms for specific client requirements.

Having worked on a variety of projects spanning different industries, e.g. finance, charity and government, Matt has gained an understanding of how to use different methodological tools in different contexts. He is also interested in the history and provenance of ideas and enjoys writing about developments within the analytics industry.