Winners and Losers in AI’s ‘Shadow Medical System’

Earlier this summer, I wrote a piece titled, “How AI Will Replace Your PCP — If It Hasn’t Already.” It highlights how health care AI’s march upmarket demonstrates many characteristics of Disruptive Innovations, as compared to incumbent PCPs. In brief, no, AI won’t replace all functions of a PCP in the near future, but it is already performing some aspects of the job just fine.

Then, in August 2026, I read the STAT First Opinion piece, “AI Has Created a Shadow Medical System,” and it raised a number of questions in my mind. The piece is written by an MD/PhD candidate (Rao) and a physician-professor-inventor (Succi) who also published a JAMA article earlier this year. The JAMA analysis concluded that 21 LLMs named the correct diagnosis more than 90% of the time when given the complete case. However, when the AI was given just what a physician would gather at the beginning of a visit, the AI failed to produce a comprehensive differential more than 80% of the time.

That’s quite a difference in outcomes.

In the STAT piece, the authors argue that AI models have set up “a shadow system that borrows the authority of medicine while avoiding its responsibilities.“ Naturally, after reading this, my mind immediately went to Disruptive Innovation. The debate Rao and Succi put forth begs the question, “When it comes to innovation, how far down the ‘not as good as’ ladder should entrepreneurs legally and ethically be allowed to go?”

Given Meta’s landmark $18B settlement in August 2026, perhaps the answer is “not as far as we’ve allowed in the past.” While I won’t pretend to have the full answer, below I lay out the questions leaders and regulators need to ask and answer in order to create an AI-enabled system that improves health, instead of harming it by avoiding responsibility.

Disruptive Innovation, in Brief

Disruptive Innovations usually start at the low end of the market, targeting those who can’t afford or access prevailing incumbent solutions or those who are overserved by them. So, low-end consumers, previous nonconsumers, or overserved consumers settle for something that’s not as good as the prevailing solution, since that solution is inaccessible to them or overshoots their needs. But incumbents’ best customers don’t usually use Disruptive Innovations (at first) because they’re so inferior based on traditional measures of performance.

Disruptive Innovations succeed because they offer worse performance than the mainstream customer demands, but they outperform incumbents on convenience and cost, and then they improve over time.

Regulation’s Role

Innovation often precedes regulation, so in our current moment, a critical question for health care leaders, inventors, and regulators to address is, “How inferior to clinicians can AI companies allow their medical chatbot products (i.e., ChatGPT Health, or even just ChatGPT or Claude) to perform?”

This isn’t an easy question. A few issues come to mind:

1. How do we define “inferior to clinicians”? Which clinicians (i.e., physicians, APPs, nurses, etc.) are we using as the comparator, and what’s the definition of “inferior”?
2. Once we determine what “inferior” means, what level of inferiority is acceptable? What’s the floor for “not as good as” a clinician?a) Note: In the health insurance industry, a health insurance offering on the open market must provide the ten essential benefits. As I’ve written about before, that floor stifles innovation and effectively makes low-end disruption illegal. Health insurance is in need of low-end disruption to reduce health care costs, so that leaves us in a bind. How can we avoid the same issue with AI increasing access to medical knowledge?
3. What is the product of comparison? Disruption is relative, so are AI chatbots competing with clinicians’ answers, their thought processes to arrive at a diagnosis, their recommendations for treatment, relationships with patients, something else, or all of the above?

These are big questions that will take time, patience, and trial and error to answer effectively.

Avoiding the Meta Mistake

We also have the opportunity to learn from the past. The recent Meta settlement highlights how a lack of regulation allows disruptions to grab hold of market openings and scale; it also shows what happens when those disruptions cause more harm than benefit.

When Facebook first came to market, it offered a way to connect with either:

1. New connections (when offered to select college campuses) or
2. Weak ties (when rolled out to the larger population)

On college campuses, Facebook served as a low-end way to engage with new contacts online, which college students would have previously done in real life. This could be considered a low-end disruption in that it offered lower cost (from a time/energy perspective) and a more convenient way to engage with new friends, as compared to meeting up in real life.

Connecting with people online also served nonconsumers of weak tie engagement as Facebook spread beyond college campuses (i.e., “connecting” with an old high school friend you’d lost touch with and wouldn’t otherwise make an effort to see in real life).

Over time, as Facebook, Instagram and other social media platforms moved upmarket, they displaced more and more in-person engagement and, as many argue, caused significant psychological and social distress, and ultimately, harm.

How do we stop AI that people increasingly use for medical questions from following a similar story arc? A critical step is answering the questions laid out here, along with many more.