By Alexis Schofield
@redlineproject
AI disclosure: Perplexity.ai was used to help research this paper. ChatGPT was used to generate an outline and write the summary below. Midjourney was used to generate photos throughout the story, and Google’s Notebook LM created the FAQ section at the end.
—-
Summary: This paper explores the significant yet often overlooked ways artificial intelligence (AI) reinforces societal biases. While many AI systems aim for neutrality, they often perpetuate discrimination through linguistic or dialectical biases embedded in training data. These hidden biases reflect human tendencies toward social prejudice and favoritism, exemplified by flawed hiring algorithms, faulty voice recognition, and discriminatory outcomes against individuals who present certain language-based patterns. It also examines ongoing efforts to mitigate these biases and the challenges of ensuring fair and unbiased AI systems in real-world scenarios.
The dangers of artificial intelligence (AI) are often discussed in the context of dystopian futures, extreme deepfakes, widespread chaos and other existential, deeply unsettling scenarios. While all of these concerns are highly valid, a more subtle and immediate risk may lie in our future, with AI being increasingly used in workplaces, corporations, and other industries to make decisions that directly impact people’s lives.
These AI-driven systems, which are often automated and integrated into many of the decision-making processes of companies we interact with daily, raise a critical question: What happens when AI models are trained on inherently biased data?
According to MIT’s AI Risk Repository, one of the current main risks associated with AI is unfair discrimination and misrepresentation driven by biased training data, which only reinforces biases toward certain marginalized groups of people. Despite progress in being able to filter out clear discriminatory or prejudiced content, AI continues to show linguistic and dialect-based biases based solely on the ways people speak and write.
This loss of human oversight is a major concern, as AI-driven systems are rapidly being introduced into many of the decision-making processes across various industries — from hiring and interviewing to customer service and educational tools — yet subtle biases still seem to persist due to the data it learns from. Without careful regulation or attention to ethical concerns, AI could have the potential to quietly reinforce and perpetuate societal biases and prejudices in ways that are difficult to predict or control.
This paper explores the hidden biases in AI and their discriminatory impact when used in automated decision-making processes. In particular, by looking at ways AI reinforces subtle biases based on linguistic, dialectal, or language-based patterns, we can better understand the risks these systems pose when used in real-life situations. This paper will also highlight the ongoing efforts to mitigate these biases and find potential solutions for creating more equitable AI-assisted models — although achieving a true level of fairness in AI may be more challenging than researchers initially anticipated.
How AI Perpetuates Linguistic and Dialect Bias
One of the primary ways AI perpetuates linguistic and dialect biases is through the use of large language models (LLMs), which are AI models that can both comprehend and generate human-like text. While these tools can be powerful and beneficial in assisting with a variety of tasks, they may also inadvertently reinforce human-made biases that are present in their training data.
A study published in “Nature Computational Science” in Dec. 2024 and reported by New York University revealed that LLMs exhibit social identity biases that are remarkably similar to those observed in humans. These models showed both ‘ingroup favoritism’ and ‘outgroup derogation,’ which are real-life tendencies where individuals are more likely to favor members of their own group while viewing those of outside groups in a more negative light.
In AI systems, this can result in preferential, biased treatment of language patterns that are associated with dominant social groups while devaluing or depreciating those linked to marginalized communities. As a result, these LLMs can unintentionally produce more favorable, helpful content for some groups and less helpful, negative content for others.
For instance, a study published in “Nature” in Aug. 2024 and reported by the University of Chicago found that when explicitly prompted to describe African Americans, AI models consistently gave positive traits such as being “passionate” and “intelligent.” However, when asked about speakers of African American English using their dialect and not explicitly mentioning race, these same AI models generated harmful and racist stereotypes.
This reveals a critical issue in how AI processed dialectal differences, as models appear to have learned to respond with politically correct answers when directly asked about certain racial groups — but then often revert to stereotypical biases when encountering the common language patterns linked to these same groups.
Moreover, linguistic bias in AI can also take shape in the form of gender discrimination. According to a report from Reuters, Amazon had to get rid of their AI-based job hiring and recruitment algorithm because it favored certain words that men were more likely to use on their resume. Although the model was not explicitly designed to exclude women from the hiring process, the algorithm was trained over a 10-year period on resumes submitted by mostly men. As a result, it frequently penalized resumes that contained words like “women’s” (e.g., women’s chess club champion) and downgraded graduates from all-women colleges.
Real-World Consequences of Linguistic Discrimination in AI
As seen with Amazon’s AI-based hiring algorithm above, linguistic and dialect-based discrimination becomes especially problematic when used in decision-making contexts such as employment. As AI becomes more integrated into many facets of our daily lives, it’s important to recognize how these implicit biases could have the potential to perpetuate existing social divisions and even create new forms of inequality.
Regarding hiring and employment, the study from earlier reported by the University of Chicago demonstrated how linguistic bias can lead to problematic decision-making. It found that AI models consistently assigned AAE speakers to less-educated careers, even when their qualifications and skills were equivalent to standard American English speakers. This has major negative implications for companies who use AI in their hiring process, as AI-assisted hiring tools could systematically exclude non-standard dialect speakers from various job opportunities.
Similar biases also already exist in automated customer service tools and AI-assisted interviews, as models continually misunderstand a wide range of accents and dialects, ultimately leading to frustrating experiences and putting certain groups at a serious disadvantage. A survey of 3,000 Americans reported by CBS News found that AI voice recognition software even struggles to differentiate between American accents in many customer service interactions, with the Southern accent being the most difficult for AI to understand, followed by accents from New York, New Jersey, Texas, and Boston.
Additionally, a study published by Stanford University found that AI voice recognition is more likely to misunderstand Black Americans’ speech compared to White Americans’. Using voice recognition software from five leading companies (Amazon, IBM, Google, Microsoft and Apple), the research showed significant disparities in both comprehension and accuracy rates, with error being nearly twice as high for Black American speakers — even when they were matched by corresponding age and gender while speaking the exact same words.
If AI already has trouble distinguishing between American dialects, how is it doing with detecting international accents? The Stanford researchers also suggested that similar problems can arise for those who speak with non-native English accents as well, which may create barriers globally for non-native speakers who are using AI technologies.
This is especially relevant in professional environments, in which non-native English speakers often utilize AI tools to improve their grammar, correct spelling, and translate text to meet the proper English language requirements. This is commonly seen in many academic or workplace settings where the English language is often required.
However, a study published by UC Berkeley found that AI detectors actually misclassified 19% of non-native English speakers’ writing as being produced by AI due to their more rigid and constrained use of the language. This misclassification can lead to false accusations of academic dishonesty and quick judgment calls on people’s character, which becomes even more problematic when it targets certain groups on non-native English speakers.
Furthermore, research from Berkeley’s AI lab found that ChatGPT shows biases against non-standard English varieties through the use of harmful stereotypes and generating condescending response styles. These models also tend to default to American ways of thinking and conventional western perspectives in ways that easily frustrate or confuse non-American users.
This bias only exacerbates the issue of linguistic discrimination in AI, as these models often fail to offer the same level of support and resources to those of non-standard English varieties as they do those using ‘standard’ English. As a result, this disparity further reinforces existing social divisions by limiting the accessibility and utilization of AI tools for many groups and communities. After all, if non-native English speakers cannot get AI to understand them correctly, it becomes much more difficult for them to use these tools effectively in everyday life.
Efforts to Reduce Linguistic Bias in AI
To address these hidden linguistic biases in AI, researchers are actively exploring new ways to reduce biased and discriminatory response styles. For instance, researchers at Dartmouth are working on identifying the specific parts of AI models that learn and perpetuate social biases. They discovered that by removing or adjusting the worst of these “attention heads,” which are parts of AI models that focus on specific aspects of language, they could significantly reduce the number of stereotypical responses without negatively affecting the model’s overall linguistic abilities or performance level.
This approach recognizes that different uses of AI may require different inputs regarding specific linguistic information. For instance, an AI-assisted medical diagnosis system may need to detect age or gender-specific language to provide adequate care, while an AI-driven hiring algorithm should ideally be blind to such factors to maintain the highest level of fairness in the evaluation of potential job candidates.
However, according to an article published in Harvard Business Review in March 2023, this seemingly intuitive process of removing demographic data may make discrimination worse in certain contexts. Although removing demographic information has proven to be successful in human decision-making, it can backfire when it’s completely removed from AI models and potentially lead to biased outputs — which is particularly evident when a significant power imbalance between two groups already exists.
Furthermore, according to the Harvard Business Review’s article, researchers in a study published in 2022 found that excluding demographic information such as gender from the input can lead to unintentional discrimination or bias that cannot be easily controlled, corrected, or accounted for beforehand.
Without this demographic data, the machine learning model greatly loses its predictive power, potentially resulting in over-predictions for women (or other marginalized groups), while remaining accurate for men (or other privileged groups). This loss of predictive accuracy has the potential to introduce a new set of unfair biases that the system cannot adequately address. Consequently, the AI model is forced to make assumptions and rely on other variables as substitutes for the excluded demographic information — which doesn’t always yield the best outcomes.
Building on this, AI researchers in a study published by MIT News in 2024 asked the question of whether it’s possible to train AI models to be both truthful and politically unbiased. The study revealed that even when models were designed to be neutral, some still showed deeply embedded political and social biases. As a result, identifying the source of this biased training data has become a key focus for future research and efforts to reduce discrimination in AI’s output.
As AI continues to be implemented into real-world decision-making processes across a range of industries, ensuring these systems do not reinforce existing social inequalities is more critical than ever. Without careful oversight, AI models have the potential to unintentionally perpetuate biases through linguistic, dialectal, or other language-based patterns, leading to unfair outcomes to a variety of groups. Addressing these biases is not just about improving AI’s accuracy — it’s about ensuring these technologies are fair, inclusive, and truly beneficial for everyone.
Frequently Asked Questions About Bias in AI
1.What is the primary concern regarding the increasing use of AI in decision-making processes?
While extreme and dystopian AI scenarios are valid concerns, a more immediate risk lies in the potential for AI systems used in workplaces and other industries to make biased decisions that negatively impact individuals’ lives. This bias often stems from the data these AI models are trained on.
2. How does biased training data lead to unfair discrimination in AI?
AI models learn from the data they are fed. If this data reflects existing societal biases against certain marginalized groups, the AI will likely perpetuate and even amplify these biases. This can manifest as unfair discrimination and misrepresentation, even when explicit discriminatory content is filtered out, as subtle linguistic and dialect-based biases can still be present.
3. How do Large Language Models (LLMs) contribute to linguistic and dialect bias in AI?
LLMs, which understand and generate human-like text, can inadvertently reinforce biases present in their training data. Studies have shown that LLMs exhibit “ingroup favoritism” and “outgroup derogation,” leading to preferential treatment of language patterns associated with dominant social groups and devaluation of those linked to marginalized communities. This can result in AI generating more favorable content for some groups and less favorable or even stereotypical content for others, particularly when dialect is involved.
4. Can you provide examples of how linguistic bias in AI manifests in real-world scenarios?
Several examples illustrate this issue. Amazon had to discard an AI hiring tool that favored male-typical language. Research showed AI models associate speakers of African American English (AAE) with less educated careers, even with equivalent qualifications. Voice recognition software struggles with various American accents, particularly Southern accents, and has significantly higher error rates for Black Americans’ speech. AI writing detectors have also been shown to misclassify writing by non-native English speakers as AI-generated.
5. How does linguistic bias in AI impact hiring and employment?
AI-powered hiring tools trained on biased data can systematically exclude individuals based on their language or dialect. This can lead to non-standard dialect speakers being assigned to less desirable roles, even if their skills and qualifications are equal to those who use standard English. This perpetuates existing inequalities in the job market.
6. What are the consequences of linguistic bias in AI for non-native English speakers?
Non-native English speakers face multiple challenges due to linguistic bias in AI. Voice recognition may struggle to understand their accents. AI writing detectors can falsely flag their writing as AI-generated due to their potentially more constrained language use. Furthermore, AI models like ChatGPT have been shown to exhibit biases against non-standard English, offering less support and defaulting to Western perspectives, making these tools less effective and accessible for this group.
7. What efforts are being made to reduce linguistic bias in AI?
Researchers are exploring various methods to mitigate bias. One approach involves identifying and adjusting specific parts of AI models (“attention heads”) that contribute to stereotypical responses. However, the seemingly intuitive approach of removing demographic data entirely has been shown to sometimes worsen discrimination in AI. Research is also focusing on understanding the sources of bias in training data and developing methods to train AI to be both truthful and politically unbiased, although this remains a significant challenge.
8. Why is addressing linguistic bias in AI considered crucial?
Addressing linguistic bias in AI is essential not just for improving the accuracy of these systems but for ensuring they are fair, inclusive, and beneficial for everyone. As AI becomes increasingly integrated into critical decision-making processes, failing to address these biases could lead to the unintentional perpetuation and even creation of new forms of social inequality, unfairly disadvantaging various groups based on their language and dialect.