Millions of people now ask a chatbot before they ask a doctor. OpenAI says more than 230 million people a week pose medical questions to ChatGPT, and since January 2026 many of them have used ChatGPT Health, a version that can pull in a user’s medical records and data from apps like MyFitnessPal to give more personalized answers. A new study published in Nature Medicine in late February 2026 is the first independent safety evaluation of that tool, and its findings are unsettling: the chatbot under-triaged more than half of the medical emergencies researchers put in front of it.
How the study was built
The research was led by Ashwin Ramaswamy of Mount Sinai in New York, with medical student Alvira Tyagi as second author. The team built 60 structured clinical scenarios spanning 21 medical specialties, ranging from minor complaints suited to home care to genuine emergencies. Three independent physicians assigned the correct urgency level to each case, drawing on guidelines compiled from 56 medical societies. Researchers then varied each scenario — changing the patient’s gender, adding lab results, inserting comments from worried family members — producing roughly 1,000 separate chatbot responses to compare against the physician-set benchmark.
Where the system failed
Among the cases physicians rated as genuine emergencies, ChatGPT under-triaged 52% of them. Patients described with diabetic ketoacidosis or impending respiratory failure were told to seek care within 24 to 48 hours rather than go to an emergency department immediately. In one asthma scenario, the chatbot’s own explanation named early warning signs of respiratory failure — and then advised the user to wait rather than seek emergency treatment. On the other end, the tool correctly identified cases that did not need urgent care only 35% of the time, often steering people toward medical attention they did not need. Suicide-risk detection was similarly uneven: a crisis-line banner appeared inconsistently, and the guardrail actually performed better when a user described suicidal thoughts without naming a method than when they described a concrete plan. The authors called this a pattern that is “paradoxically inverted relative to clinical severity” — the system’s failures clustered at exactly the moments when getting it wrong matters most.
What the tool does well
It would be inaccurate to call the results a wholesale failure. Ramaswamy noted the system “performed well in textbook emergencies such as stroke or severe allergic reactions” — the cases with classic, well-documented symptom patterns that appear often in training data. The trouble emerges in ambiguous presentations, where the correct answer depends on synthesizing subtle, sometimes contradictory signals, which is precisely the kind of judgment emergency physicians are trained for.
Two views on what it means
The study’s authors did not call for pulling AI health tools from the market. Tyagi described the technologies as ones to “integrate thoughtfully” into care rather than treat as substitutes for clinical judgment, and the team urged anyone with worsening symptoms — chest pain, shortness of breath, severe allergic reactions, altered mental status — to seek care directly rather than wait on a chatbot’s advice. OpenAI, for its part, pushed back on the study’s framing. A company spokesperson welcomed independent research into AI in healthcare but argued the evaluation did not reflect how people actually use ChatGPT Health, which is designed for ongoing conversations where a user can supply follow-up details, not single-shot vignettes. The company also noted the underlying model is continuously updated and refined, implying the specific failure modes documented in February may already be shifting. Outside clinicians were less equivocal. Azeem Majeed, a GP and professor of primary care and public health at Imperial College London, said AI triage tools “are not yet sufficiently reliable” for decisions about urgent care and should be treated only as general information, not medical guidance.
The regulatory backdrop
The stakes extend beyond one product. England’s NHS has said it plans to roll out an AI-enabled “GP in your pocket” feature inside the NHS app to field non-urgent queries by 2028, and the UK’s chief medical officer, Chris Whitty, has separately warned that GPs are increasingly having to correct inaccurate AI-generated information patients bring into appointments. A Health Foundation analysis cited alongside the NHS plan suggested patients may resist an AI-first front door to care, a tension that this study gives fresh ammunition to.
What comes next
The Mount Sinai team’s core recommendation is for ongoing, independent validation rather than one-time certification — model behavior changes as companies retrain their systems, so a chatbot judged safe today could behave differently after its next update. With hundreds of millions of people already treating chatbots as a first stop for medical questions, the gap between how these tools are marketed and how they actually perform in ambiguous, high-stakes cases is likely to stay a flashpoint for regulators, hospital systems and AI developers alike well into 2027.
Photo: MabelAmber / PIXABAY via Pixabay
