Algorithms play a growing role in modern society. They are used to make all kinds of decisions, from recommending products online to helping determine who qualifies for a loan or a job. These algorithms can make our lives more efficient, but they also carry a serious risk: algorithmic bias. This phenomenon can unintentionally lead to inequality and discrimination, with potentially far-reaching consequences for society. But what is bias, exactly? How does it end up in algorithms, and what are the consequences?AI algorithms are trained on data that contains bias.
What is bias in humans?
##
At its core, bias is a systematic deviation from a norm or standard that can lead to a wrong or unfair decision. When we talk about bias in humans, we usually mean prejudices or unconscious preferences that shape our thinking, our behavior, and with that our choices. Bias can be based on race, gender, age, or social status, for example. A well-known example of human bias is "shooter bias": research showed that American police officers were unconsciously quicker to shoot at Black suspects than at white suspects. It shows that even unconscious prejudice can do real harm.
Bias also shows up in cognitive processes, such as "confirmation bias". This means people tend to look only for information that confirms their own beliefs, while ignoring information that contradicts them. Another example of human bias is the "status quo bias", where people tend to stick with the current situation even when change might serve them better.
Although bias is usually seen as negative, it can be useful in some cases. It can help people make decisions faster, for example. Picture a doctor who has to decide quickly on the best treatment in an emergency. Drawing on earlier knowledge and experience (which can sometimes be a form of bias), the doctor can reach a conclusion faster without weighing every option from scratch. In situations like these, bias helps people act quickly and effectively when time is critical.
From humans to algorithms: how does bias get into AI?
Many people assume that algorithms are neutral. After all, they are executed by machines, which, unlike humans, have no emotions or prejudices. Yet algorithms turn out to be susceptible to bias too. That bias often comes from the data they are trained on. If the training data is biased, the algorithm picks up those patterns.
One of the best-known examples of algorithmic bias is the COMPAS algorithm, which was used in the United States to predict whether a suspect would commit another crime. The algorithm turned out to systematically flag Black suspects as future criminals more often than white suspects, even when the facts did not justify it. This bias came from the historical data the algorithm was trained on. That data reflected earlier arrests and convictions, which were themselves already shaped by existing social prejudice and discriminatory policing practices. Because Black communities had historically been checked and arrested more often, the data contained a disproportionately high number of Black suspects. The algorithm was trained on this distorted reality and therefore wrongly flagged Black suspects as higher risk more often. This is how bias arises: the data fed into the algorithm already contains built-in prejudice, so the algorithm draws the wrong conclusions.
Facial recognition software, widely used by governments and companies, shows bias as well. The technology is deployed for all kinds of purposes, such as airport security, unlocking smartphones, and even tracking down criminals. Yet studies show that facial recognition software is often less accurate for people with darker skin, and especially for women. A 2018 study by MIT Media Lab found that the error rate of commercial facial recognition systems for correctly identifying white men was just 0.8%, while it rose to 34.7% for Black women. These systems were trained on datasets in which between 77% and 86% of the images showed light-skinned men. As a result, the technology performed worse on Black people and women. So once again, bias arises because the data the algorithms are trained on reflects these inequalities.
The many forms of bias in AI
##
Bias in algorithms can take many forms. Beyond race and gender, there can also be cultural and political bias. In some countries, social media platforms are said to have a liberal bias. This means conservative voices are reportedly less visible or even suppressed. One example is the claim by conservative groups in the United States that their posts are disproportionately censored or shown less often in social media news feeds. This is still being debated, but the example illustrates how algorithms can influence political preferences.
Another major problem is the influence of algorithmic bias on how we consume information. Platforms like YouTube and Facebook use recommendation systems that mainly show users content matching their earlier interests. This creates a so-called filter bubble in which people are exposed to one-sided information that reinforces their existing beliefs. That has serious consequences for society, because it can lead to social polarization and the rise of “echo chambers”, in which people only hear like-minded
voices while other perspectives are ignored. That can make it harder to have nuanced discussions or to understand other points of view.
What are the consequences of bias in AI for society?
The consequences of algorithmic bias can be substantial and reach into many parts of daily life. A well-known example is the Amazon scandal. In an attempt to automate its hiring process, Amazon developed an algorithm that would score job applications automatically. The algorithm turned out to discriminate against women. This happened because it was trained on historical applicant data in which men had been hired more often. As a result, the algorithm became biased against women, scoring female applicants lower than male ones, even when they were equally or even better qualified.
In healthcare, bias in AI can do real damage. One study showed that an algorithm used to determine which patients needed extra medical care selected white patients over Black patients more often, even when the Black patients needed more care. This happened because the algorithm was trained on data reflecting past healthcare spending. Black patients generally spent less on care, not because they needed less care, but because they had less access to quality healthcare. The algorithm, however, interpreted this lower spending as a lower need for care, unfairly disadvantaging Black patients. As a result, they did not get the care they needed, with serious consequences for their health.
How can we tackle bias in AI?
Fortunately, there are ways to limit and counter algorithmic bias. One of the most obvious steps is making sure the data used to train algorithms represents the diversity of society. If the data only represents a specific group of people or characteristics, the algorithm will make biased decisions too. With more diverse data, the algorithm can learn to make fairer and more inclusive decisions.
Unfortunately, this is not always easy. A big problem is that some characteristics are so closely tied to other information that it is hard to eliminate bias completely. A good example is the use of zip codes in the United States. Even if an algorithm does not explicitly consider race, using zip codes in the data can still lead to biased outcomes. Zip codes in the US are often strongly linked to ethnic and economic differences. If an algorithm decides not to include race as a factor but does use an individual's zip code, it can still introduce racial bias indirectly through the correlation between zip code and race.
Besides changing the data, there is another way to counter bias. Making the developers of algorithms more aware of the risks and challenges bias brings can help fix bias in algorithms. Diversity in the teams responsible for developing these algorithms helps here. People from underrepresented groups are often better at spotting and addressing potential prejudice in systems, because they experience its impact firsthand.
Finally, there are technical solutions that can help reduce bias. One example is training algorithms to ignore certain social characteristics, such as race, gender, or age. This principle is known as "fairness through unawareness". In theory it sounds like an effective solution, but in practice it turns out to be difficult. Because certain characteristics are, as mentioned earlier, intertwined with other data, the algorithm can still end up making biased decisions unintentionally. That is why it is important to look not only at the data, but also at how algorithms make decisions and which factors play a role in them.
Conclusion
Bias in algorithms is a complex problem with major consequences for society. Algorithms are not prejudiced in themselves, but through the data they are trained on they can still pick up prejudices and use them in decisions. This can lead to unequal treatment of certain groups and can even contribute to social problems such as polarization.
It is essential that we stay aware of the risks of algorithmic bias and actively work on solutions. By using more diverse data, raising awareness among developers, and building technical solutions, we can make algorithms fairer and more inclusive. Still, we need to stay alert and keep following developments in algorithmic decision-making. Only through constant attention and careful oversight can we make sure algorithms have a positive and fair influence on society.