Can Healthcare AI Be Fair and Transparent? A Perspective by AgileIntel

Medical professionals generate vast amounts of data daily, from electronic health records to imaging scans and lab results. These datasets power AI systems that promise faster diagnoses and more personalised care. However, AI can perpetuate biases, undermine fairness, and compromise patient privacy without careful design and oversight. Such risks can lead to unequal treatment outcomes and erode trust in healthcare institutions. 

Addressing these challenges requires a concerted effort to ensure that AI systems are developed and deployed responsibly. This involves implementing strategies to detect and mitigate bias, promoting fairness in AI-driven decisions, safeguarding patient privacy, and ensuring transparency and accountability in AI systems. By adopting these practices, healthcare providers can harness the benefits of AI while minimising potential harms.  

At AgileIntel, we design AI systems for health tech firms, and we believe ethical guardrails must be integral, not optional.  

Detecting and Mitigating Bias in AI Systems  

Bias in AI systems can arise from various sources, including skewed training data, flawed algorithms, and unintended consequences during deployment. Detecting and mitigating these biases is crucial to ensuring equitable healthcare delivery.  

Key Strategies:  

• Diverse Data Collection: Ensure that training datasets are representative of all patient demographics, including age, gender, ethnicity, and socioeconomic status. This helps in building models that generalise well across diverse populations.  

  
  

• Bias Audits: Regularly conduct audits to identify and address biases in AI models. This includes testing models across different demographic groups to ensure consistent performance.  

  
  

• Algorithmic Transparency: Develop interpretable algorithms and allow for the identification of decision-making processes. This transparency aids in understanding and rectifying biased outcomes.  

Case Example: Optum’s Risk Prediction Algorithm  

Optum, a UnitedHealth Group subsidiary, developed a tool to flag patients for high-risk care programs. The algorithm predicted future healthcare costs as a proxy for medical need. A science study revealed this created racial bias: at the same risk score, Black patients had more uncontrolled conditions than white patients. Only 17.7% of Black patients were flagged for extra care compared to 46.5% when the model was corrected.  

Following this, New York regulators pressed Optum to prove the system did not discriminate. This case shows how proxy variables like cost can embed structural inequities and why regular audits are vital.  

Promoting Fairness in AI-Driven Decisions  

Fairness in AI refers to the equitable treatment of all patients, ensuring that AI systems do not favour one group over another.  

Key Strategies:  

• Fairness Constraints: Incorporate fairness constraints into AI models during training to ensure that predictions do not disproportionately benefit or harm any specific group.  

  
  

• Continuous Monitoring: Implement systems to continuously monitor AI decisions and outcomes, allowing for the detection and correction of any emerging fairness issues.  

  
  

• Stakeholder Engagement: Involve diverse stakeholders, including patients, healthcare providers, and ethicists, in the development and evaluation of AI systems to ensure that multiple perspectives are considered.  

Case Example: IBM Watson for Oncology at MD Anderson  

IBM Watson for Oncology was designed to recommend cancer treatments by analysing vast clinical data. The project faced criticism at MD Anderson Cancer Centre when the system produced recommendations that sometimes conflicted with established medical guidelines. Doctors found the tool had limited training on diverse, real-world patient populations, raising fairness concerns.  

The hospital eventually halted the project after spending over US$60 million. This case highlights why fairness requires advanced algorithms, validation against varied patient data, and continuous clinical oversight.  

Safeguarding Patient Privacy  

Protecting patient privacy is paramount when integrating AI into healthcare systems. AI systems often require access to sensitive personal health information, making robust privacy measures essential.  

Key Strategies:  

• Data Anonymisation: Employ techniques to anonymise patient data, ensuring that individual identities are not exposed during AI model training and deployment.  

  
  

• Secure Data Storage: Implement strong encryption protocols for storing and transmitting patient data to prevent unauthorised access.  

  
  

• Compliance with Regulations: Adhere to relevant data protection regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States, to ensure legal compliance and protect patient rights.  

Case Example: DeepMind and the NHS

In 2015, Google DeepMind worked with the Royal Free London NHS Trust on Streams, an app to detect kidney injury. The deal gave access to 1.6 million patient records without explicit patient consent. In 2017, the UK’s Information Commissioner’s Office ruled that data sharing was unlawful due to a lack of transparency. Although revised later, the controversy damaged trust.  

The case shows the importance of purpose-limited use, informed consent, and oversight when private firms handle patient data.  

Ensuring Transparency and Accountability  

Transparency and accountability are critical for maintaining trust in AI systems. Stakeholders must understand how AI decisions are made and have mechanisms to challenge and rectify errors.  

Key Strategies:  

• Explainable AI: Develop AI models that clearly explain their decisions, enabling healthcare providers and patients to understand the rationale behind recommendations.  

  
  

• Accountability Frameworks: Establish clear accountability structures to determine who is responsible for AI-driven decisions and outcomes.  

  
  

• Feedback Mechanisms: Implement systems that allow users to provide feedback on AI decisions, facilitating continuous improvement and correction of errors.  

Case Example: Babylon Health’s AI Chatbot

Babylon Health, a UK-based telehealth startup, created an AI chatbot to provide medical triage advice. In 2020, UK doctors raised concerns after the system delivered unsafe recommendations, including underestimating heart attack symptoms. Critics pointed to the lack of transparency in how the chatbot generated its advice. The case illustrates how insufficient explainability undermines trust and highlights the need for rigorous clinical validation before deploying AI-driven diagnostic tools.  

Building Ethical AI Governance Structures  

 

Establishing robust governance structures is essential to overseeing the ethical deployment of AI in healthcare.  

Key Strategies:  

• Ethics Committees: Form multidisciplinary ethics committees to review and guide AI projects, ensuring alignment with ethical standards.  

  
  

• Policy Development: Develop and enforce policies that set clear guidelines for the ethical use of AI, including considerations of bias, fairness, privacy, and accountability.  

  
  

• Training and Education: Provide ongoing training for healthcare professionals on the ethical implications of AI and best practices for its use.  

Case Example: Mayo Clinic’s AI Governance Approach  

Mayo Clinic has developed a structured AI governance framework that includes ethics reviews, clinical validation, and patient engagement before AI deployment. Every AI tool undergoes evaluation for fairness, privacy, and clinical accuracy. The organisation also trains clinicians on the limitations of AI to ensure responsible use. This proactive governance model positions Mayo Clinic as a leader in setting safe and ethical healthcare AI standards.  

Conclusion: Building a Roadmap for Responsible AI 

Integrating AI into healthcare offers significant opportunities to enhance patient care and operational efficiency. However, these technologies can perpetuate existing disparities and introduce new risks without a strong ethical foundation. By proactively addressing issues of bias, fairness, privacy, transparency, and governance, healthcare organisations can develop AI systems that are not only effective but also ethical and equitable.  

This commitment to ethical AI will foster trust among patients and providers, ensuring that technological advancements translate into meaningful improvements in healthcare delivery.