Every ML team eventually argues about batch versus streaming, and the argument usually ends up being about latency and cost. Batch is cheaper, simpler to operate, easier to backfill. Streaming costs more and buys you freshness. For background, see our recent coverage.
There is a case where the distinction is not about freshness at all. It is about whether the information you need still exists by the time you run inference. US healthcare has been running the batch version of that pipeline for years, and federal auditors have now published what it produces.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
The pipeline
Medicare Advantage pays insurers more to cover sicker patients. How sick a patient is gets derived from diagnosis codes extracted from clinical notes, which are free text written by a clinician in the last two minutes of an appointment.
The standard architecture is batch. Notes accumulate. Weeks or months later, a model runs over the corpus and surfaces diagnoses that should have been coded and were not. Output is a ranked list of candidates with confidence scores, reviewed by humans, submitted.
The failure mode is not model quality. It is that the batch job runs against a lossy artifact. The clinician had the full clinical picture. What reached the model was a compressed, time-pressured summary, and no amount of inference recovers signal that was never written down.
What that costs, measured
The US Office of Inspector General published its detection method in December 2023: a toolkit with the selection logic and the database queries it uses to flag diagnosis codes at high risk of lacking support. Then it ran.
Across its audit work, roughly 70 percent of those codes were not supported by the associated medical records, some categories above 90 percent. One national audit sampled 97 records carrying an acute stroke diagnosis. None were supported as acute strokes. In 68 of those cases, the patient had genuinely had a stroke, documented in history, and the submitted code described it as active.
That last detail is an interesting one for anyone who builds these systems. The error was not hallucinated data. It was a temporal attribute lost in compression, then reconstructed wrong downstream.
The structural problem with ranking
Now the part that has no amount of retraining addresses.
A model producing a ranked candidate list with confidence scores has a defined output space: items, ordered, with probabilities. There is no element in that space that means “this existing record is wrong, remove it.” You can threshold a score low. You cannot emit a negative determination, because the output type does not include one.
So the pipeline is directionally locked. It proposes additions. It cannot propose removals. Run it a hundred times and the record grows monotonically, regardless of what anybody using it intends.
The regulator noticed the automated version of this. Its compliance guidance names prompts “generated by artificial intelligence algorithms” that push clinicians toward adding diagnoses among conduct its investigations have identified as potentially abusive. Read as a systems critique rather than a legal one, the objection is that the pipeline has one direction and no inverse.
What changes if you move inference into the encounter
Run inference while the clinician is still in the encounter and three things change.
The domain term for this is prospective hcc coding, and the architectural shift underneath it is the one worth noting: from post-hoc extraction over a compressed artifact to in-workflow inference at the point the artifact is created.
The design question that generalises
Two things worth checking in any system you build that assesses records rather than creates them.
First, does your output space contain a negative? If the model can only rank candidates, it can only ever add, and no policy layer above it changes that. A system that holds explicit criteria can fail a test, which is what makes a removal expressible.
Second, how lossy is the artifact you run against, and was the information you need present when it was created? Batch inference over a compressed representation is a reasonable design when the compression preserves what you need. When it does not, moving inference upstream is not an optimisation. It is the only thing that works.
Related articles
- Meta Disputes Muse AI Read 187,000 Messages [2026]
- OpenAI, Meta Agents Test Public Trust, Meta Up 11% [2026]
- Google Launches Gemini 4 Argon, Blocks Public Access [2026]
- Stable Diffusion WebUI Setup: 13 Steps, 4GB VRAM [2026]
