Overview

<aside> 💡 AI bias is a real concern in generative AI systems. The source of the bias is the training data itself, and even from the reinforcement learning as well.

Even before the launch of chatGPT, in 2016, Microsoft’s Tay chatbot was a cautionary tale. Gemini’s image generator misfire in early 2024 showed how even well-intentioned ‘tuners’ of AI could make a situation worse.

And, there’s the Waluigi Effect. It’s a well-known principle amongst LLM designers that says, if an AI has been trained on some form of ethical/ discourse, it is more likely for it to be able to do the opposite; i.e., training it on the ‘right thing’ makes it more possible for it to do the ‘wrong thing’.

</aside>

Known Dangers

• Discrimination and Inequity
  • Impact on Hiring Practices: AI tools used for resume scanning and candidate selection can inadvertently perpetuate biases present in the training data. For instance, if an AI system is trained on resumes predominantly from a particular demographic, it may favor candidates from that demographic while disadvantaging others. This can lead to discriminatory hiring practices and reinforce existing inequalities in the workplace.
    • Example: Amazon had to scrap its AI recruiting tool after discovering it was biased against women. The system was trained on resumes submitted over a ten-year period, which were predominantly from men, leading the AI to downgrade resumes that included the word "women's" or were from all-women's colleges[1].
• Healthcare Disparities
  • Impact on Medical Diagnostics: AI systems used in healthcare can exhibit biases if the training data does not adequately represent all demographic groups. This can result in less accurate diagnoses and treatment recommendations for underrepresented groups, exacerbating healthcare disparities.
    • Example: Studies have shown that some AI diagnostic tools perform worse on minority populations. For instance, a computer-aided diagnosis (CAD) system was found to return lower accuracy results for African-American patients compared to white patients[1].
• Financial Inequities
  • Impact on Credit Scoring: AI systems used for credit scoring and lending decisions can perpetuate biases if the training data reflects historical inequalities. This can result in unfair lending practices, where certain demographic groups are systematically disadvantaged.
    • Example: There have been instances where AI-driven credit scoring systems have shown bias against minority applicants, leading to higher rejection rates or less favorable loan terms for these groups. This not only affects individual financial opportunities but also perpetuates broader economic disparities[1].
• Reputational Damage and Legal Risks
  • Impact on Brand Trust: When AI systems produce biased or discriminatory outputs, it can lead to significant reputational damage and legal consequences for businesses. This can erode customer trust and result in financial losses.
    • Example: See Tay and Gemini examples above.
• Amplification of Stereotypes
  • Impact on Content Generation: Generative AI models can inadvertently reinforce harmful stereotypes if they are trained on biased data. This can perpetuate societal prejudices and contribute to the spread of misinformation.
    • Example: Language translation systems have been found to associate certain professions with specific genders, reinforcing gender stereotypes. For instance, translating the Turkish gender-neutral word "o" to English often results in "he" for professions like "doctor" and "she" for professions like "nurse"[2].

Mitigation Strategies

To address these dangers, several strategies can be employed:

• Better Training Data: Ensuring that training data is diverse and representative of all demographic groups can help mitigate biases.
• Bias Detection Tools: Implementing tools to continuously monitor and detect bias in AI systems.
• Human-in-the-Loop: Incorporating human oversight to review and approve AI-generated outputs.
• Transparency and Accountability: Developing transparent AI systems with clear accountability mechanisms to address biases when they occur.
• Ethical AI Governance: Establishing robust AI governance frameworks to guide the responsible development and deployment of AI technologies.

Sources: