Jul, 144 min to read
AI Products
News · OpenAI and Penda Health test an alerting copilot across 40,000 clinic visits in Nairobi
Jul, 84 min to read
AI ProductsOpenAI and Penda Health test an alerting copilot across 40,000 clinic visits in Nairobi
A background LLM safety net cut documented diagnostic and treatment errors — but the study shows the deployment work mattered as much as the model.
An alerting layer, not an autonomous agent
Penda Health built AI Consult as a copilot that runs in the background of the electronic health record its clinicians already use. As a clinician documents a visit, de-identified notes are sent to the OpenAI API at key points, and the system returns one of three signals: a green checkmark for no concern, a yellow ringing bell the clinician may choose to open, and a red pop-up they must view before continuing.
The design choice worth noting is what the system does not do. It identifies potential errors for a human to verify rather than taking any action on a record. The worked example in the announcement is concrete: a full haemogram showing microcytic anemia (HGB 9.90, MCV 58.30) with a diagnosis of only bacterial tonsillitis. AI Consult flagged the unaddressed anemia, and the clinician added iron deficiency anemia to the diagnoses.
The model behind this was GPT-4o from August 2024. OpenAI is explicit that model capability was not the limiting factor — a notable framing given the prompts were also tailored with Kenyan epidemiological context, local clinical guidelines, and Penda's own standard procedures.
The 'left in red' rate is the real story
Across 39,849 visits split between clinicians with and without the tool, physician reviewers found the AI group had 16% fewer diagnostic errors and 13% fewer treatment errors, with history-taking errors down 32%. Effects were larger on visits that would have triggered a red alert: 31% fewer diagnostic errors, 18% fewer treatment errors.
But an earlier version of the copilot, which required clinicians to actively request a second opinion, saw limited uptake because it interrupted the patient interaction. And even the redesigned version underperformed at first. Penda tracked a metric it calls the 'left in red' rate — the share of visits with unresolved red alerts. During the induction period it sat at 35–40% in both groups, meaning clinicians with the tool were often ignoring red alerts entirely.
That number only dropped to 20% after Penda invested in active deployment: peer champions explaining the tool, personalized coaching driven by usage tracking, and recognition for clinics that used it well. The technology was constant across both periods. What changed was the human program around it.
What the outcome data does and doesn't show
The error reductions are graded by physician review of documentation. When Penda looked at patients themselves — via eight-day follow-up calls asking whether they felt better — the difference was 3.8% not improved in the AI group versus 4.3% without, which the announcement states was not statistically significant. Rates of seeking care elsewhere were also similar.
OpenAI is careful here rather than overclaiming. Of 12 patient-safety reports (7 in the AI group, 5 without), none involved harm caused by AI Consult recommendations. Penda is now running a separate randomized controlled trial with PATH specifically to measure patient outcomes, an implicit acknowledgment that fewer documented errors is a proxy, not proof of better health.
One secondary finding cuts against the idea that clinicians grow dependent: red-alert triggers fell from 45% of visits at the study's start to 35% at the end, suggesting clinicians began avoiding common mistakes before the tool flagged them.
The implication: the model-implementation gap is now an operations problem
The clearest takeaway from Penda's work is that a capable frontier model was necessary but far from sufficient. The gap OpenAI names — between what models can do and how they are used — was closed here by workflow integration, local prompt tailoring, ethics approvals from four Kenyan bodies, and a sustained coaching effort that halved the left-in-red rate.
For teams deploying clinical or high-stakes copilots, the reproducible lesson is not the 16% figure. It is that measuring how often people act on alerts, and then investing in the change management to raise that number, is where the observed benefit actually came from. A tool that fires red alerts nobody heeds produces the same outcomes as no tool at all.
Penda calls AI Consult an early archetype rather than a final form, gesturing toward voice-first documentation and confirmed agent actions in the record. Those are model-side improvements. This study suggests the harder, less glamorous work will remain on the deployment side.
Find this story relevant to you?
Contact us to find a unique solution
Need an AI engineering partner that can actually build?
We help businesses integrate AI, build AI-powered products, automate high-value workflows, and modernize the software systems behind them.
Related reading
More analysis around product delivery, operational AI, and the systems work that makes deployment hold up in reality.