Inference has always been expensive
There’s been a lot of discussion lately about what it costs to actually use AI. The Wall Street Journal is covering corporate token budgets, and Becker’s is covering the same problem in healthcare. CIOs are trying to forecast spending, companies are trying to rein it in, and we’ve acquired the word “tokenomics.”
These are reasonable things to worry about. But inference has always been expensive. The compute is just starting to demand a more noticeable share of the budget.
For the ML pedants (hi!), I’m talking about the cost of putting an inference to work, not just calling model.predict().
TypeSafe’s recently introduced Jev is an interesting example of the push toward cheaper inference. The company describes a model that takes unstructured inputs and returns typed, probabilistic outputs rather than generating free-form text. Its advertised pricing charges for input tokens, with no charge for outputs. It’s easy to like: outputs that software can consume without paying for an essay along the way.
Some of this should feel familiar. We’ve had computationally cheap models with constrained outputs for a long time. Our old friend logistic regression could return a probability without first telling us that we’d asked an excellent question.
This isn’t to suggest that the models are interchangeable or that there’s nothing new in Jev’s underlying technology. It’s that we have considerable experience with cheap predictions, and in medicine, computing the prediction was often not the difficult or expensive part.
A good chunk of my healthcare AI lifecycle series is about everything else that needs to happen. The model needs access to the right data, connections to clinical systems, prospective validation, workflow integration, monitoring, and maintenance. Making the prediction cheaper doesn’t make those responsibilities disappear. Neither does guaranteeing the output’s type. A perfectly valid Boolean can still be clinically wrong.
This isn’t a uniquely medical observation. Sculley and colleagues were writing about the substantial ongoing maintenance costs of ML systems in 2015. We’ve had plenty of time to learn that the model is only part of the system.
Then there’s the work the model creates.
Consider a simple model that fires an alert. A physician then opens a chart, checks information they might have already reviewed, decides whether anything needs to happen, and perhaps documents why nothing needs to happen. The computation might be trivial. The workflow now includes a new interruption, another chart review, and an additional little task for someone who presumably had nothing better to do.
There’s a long history of studying this across clinical decision-support systems. A primary-care study found that more reminders per encounter and more repeated reminders were associated with lower reminder acceptance. Our own study across 24 hospitals found that daily sepsis alerts increased substantially early in the pandemic, even as hospital census fell. These weren’t cost studies, but they illustrate why the volume and behavior of alerts deserve attention beyond how efficiently they’re computed.
Now consider the opposite: a model that costs more to run but actually takes work off someone’s plate.
What is the cost of a resident spending another half hour at home finishing notes? Or staying late to make sure a discharge summary is written properly? Assuming their compensation doesn’t change, the additional work generates no additional payroll expense. The resident still loses the half hour, but fortunately that doesn’t trouble the IT budget.
Suppose an AI tool helps complete the task, including the necessary review, in less time. The resident’s salary hasn’t changed, but the health system now has a variable expense. Each use generates a charge. The tool can reduce the work required and still look like a financial loss.
By that accounting, the business case was better when we just took the resident’s evening.
In LLM-parlance: The new bill is real. But so was the work it replaced.
To be clear, saving a salaried clinician thirty minutes does not automatically produce thirty minutes of cash savings. Nobody gets to claim a reduction in payroll while paying the same salary. But I don’t think we should have to turn every recovered minute into another billable activity to justify giving it back. Time for teaching, time to think, or simply time with loved ones at home are outcomes I’d be willing to pay for.
Of course, the tool has to actually help. If the generated note takes as long to check and correct as it would have taken to write, we haven’t accomplished much. Calling that a “human-in-the-loop” workflow doesn’t make the human’s work disappear. The review, corrections, duplicate work, and new interruptions all belong in the calculation.
This is why implemented AI should be funded partly through the operations budget, rather than treated exclusively as an IT expense. The people responsible for the workflow should help decide whether a tool is useful, pay for it, and assess what happens after it’s introduced. IT shouldn’t have to justify the whole investment using savings that might show up somewhere else. The Becker’s piece describes health systems already wrestling with that mismatch.
But moving the bill between departments only gets us so far. We still need to count benefits that don’t conveniently reduce a budget line, including reducing the amount of work that follows people home.
I’m all for cheaper inference. But counting every token while treating a physician’s evening as free is a pretty lousy way to decide what’s expensive.
Cheers,
Erkin
Go ÖN Home
