NB: this series is still a work in progress.
This post builds off of our previous discussions on healthcare AI infrastructure. If you are unfamiliar with that infrastructure, it may be helpful to review the posts that cover the general lay of healthcare IT land, development infrastructure, and implementation infrastructure.
Overview
In this post we discuss the technical side of a project called the Michigan Critical Care Utilization and Risk Evaluation System (M-CURES). We developed MCURES as an in-hospital deterioration prediction system for patients admitted to the hospital for acute respiratory failure during the initial onset of the COVID-19 pandemic. In the early days of the pandemic we where very worried about being able to quickly triage patients between different levels of care. We expected to see a massive influx of patients and wanted to being able to place them in the correct care setting (e.g. home, field hospital, regular hospital, ICU). To meet this institutional need we needed to get the M-CURES model developed and implemented as fast a possible.
To speed the process up as much as possible we started talking about implementation extremely early in the process. Within the first week we had settled on the doing internal integration using Epic’s tools. Although it was our first time using Epic’s tooling we felt like it would give us the best shot at the fastest integration process. After we made that decision we started doing technical integration work in parallel with model development.
The development of the project as well as its external validation is discussed in a paper published by the bmj.
Epic’s Internal Integration Approaches
At the time we started the M-CURES project Epic offered two options for internal integration:
- Epic Cognitive Computing Platform (ECCP) and
- Predictive Model Markup Language (PMML).
Epic’s PMML approach is interesting because you essentially specify the model via configuration (using the PMML standard) and Epic builds a copy of the model based on their implementations of different model architectures. I have not built anything using this approach; however, based on my research at the time it seemed fairly limited, as it supported a small handful of simple model architectures.
Because of the model architecture limitations of PMML we decided to go with ECCP for MCURES. ECCP enables you to run models in you’ve developed in Python using a proprietary model serving infrastructure. This model serving infrastructure is essentially a sandboxed Python AI environment hosted using Microsoft Azure. At a high level data are passed from Chronicles to this special Azure instance, the model produces predictions, which are then passed back to Chronicles. ECCP takes care of the data transitions and AI developers primarily need to worry about their AI code.
Model Development
You can find a lot of the model development and validation details in our bmj paper. The short version is that model development was primarily driven by the used of Michigan Medicine’s research infrastructure, which is where we got all of our training and internal validation data. However, discussions around implementation quickly started shaping model development.
Difference in data pipelines led to a shift in how we built the model. The research data pipeline we were familiar with for model development gave us a lot of control in terms of pulling a wide array of features per patient. However, this control came at the cost of accessing very low level data and meant the had to put a good deal of development effort getting the data in the right representational state. For example, we could easily pull all of the meds and vitals for a given patient encounter. But then it was up to us to figure out the right way to filter and aggregate this data prior to ingestion into the model.
The reporting infrastructure that Epic uses for ECCP can be thought of as higher level, with data being slightly restricting with the benefit of automatic filtering and aggregation. So we could easily specify that we would like the last set of vitals or if the patient had received a beta-blocker. What’s great is that these are data elements that are shared across the health systems’s Epic reporting infrastructure, so you just need to build the column/feature once.
On the whole, this is a great benefit. However, it does limit the palette
e.g., we had all the vitals recorded over an encoutn) that we had to do a lot of processing on
Although RDW enabled us to build models using an incredibly wide number of features per patient this
As mentioned in the development infrastructure post, Epic provides tooling to facilitate the internal technical integration. Epic provides a Python environment with a load of standard Python AI/ML libraries (…list) They also provide a couple custom Python libraries that help you interact with the data interfaces.
- You can receive tabular data in a JSON structure that is parsed by their libraries You can then pre-process the data and pass it to your model
- Once you have your predictions you packaged up the data using another set of Epic’s Python calls.
Although the development environment is sandboxed, you are not overly constrained in terms of the code you want to include. You can include additional python and data files in the model package Additionally you can call external APIs from within this environment if they are whitelisted. This means that you could include information from other sources or do data processing via another service.
You can take an existing model that was developed in and as long as you use epic approved list of libraries, you can use epic bundling tools to then convert it into a package that can be run on their ECCP instance the way that the model receives, data is through a reporting workmen report so you’ll work with your epic analyst to set up a report essentially is the tabular data you want your models received so you specify all the columns and have this done you’ll also have an epic analyst.
In this workflow you assess and convert a model that you made with your own infrastructure into a package that can be run on Epic’s implementation infrastructure. What’s crucial about the workflow depicted above is that there’s a data report that comes directly out of Chronicles (not Clarity) that you use as a part of this packaging workflow. This is report is often a small extract representing current patients in the health system. Thus, despite being small it is a very good representation of what the data will look like prospectively, as its generated by the prospective infrastructure.
Model input data is passed out of chronicles using reporting workbench. Reporting workbench is designed for different types of EMR reports. You can configure special versions of these reports that would pull the necessary data for patients that could be used for an AI model. Data are in a tabular structure, where rows represent patients or encounters, and columns represent attributes like age, current heart rate, etc.. I won’t go into a ton of details here, but this is the place where you can run into significant limitations, because the underlying data in Chronicles isn’t actually tabular, and the best representation of longitudinal health data is often also not tabular as well so there’s lots of engineering that needs to be done in order to get a good representation of the patients.
Data will then be passed in a secure manner to the model, which is running on the special Azure instance. We talked a little bit about model packaging so we won’t go into that here. But there is some configuration that is needed when running the model in real time, in addition to the model we need a couple items:
- input data report, and
- model run information.
We need to explicitly connect the reporting workbench model discussed above to our configuration. Additionally, we need to instantiate the logic that controls the frequency at which the model runs. For this one creates a special Epic batch job that will run with a specified frequency. This job runs the reporting workbench reports and passes that data to the model process that then calculated predictions.
The predictions computed by the model are then passed back to Chronicles. These end up in special in a special part of the database that’s designed to store predictive model results The kind of information that you can pass back are a little bit limited because the database is expecting certain types of information.
When the data is back in Chronicles you serve it to users in many different ways. For example, you could use it to fire best practice alerts or have it be highlighted as an additional column in a list of patients stratify patients based on a risk score. This is all fairly easy to do because you’ve already been working with your epic analysts to get the data directly into the status structure, and then they can work with their colleagues to set up the best practice alert, or column display.
Despite a couple technical limitations, the entire flow data from Chronicles to ECP and back to Chronicles controlled, unless you have pretty good guarantees about Safety and reliability.
One thing major limitation of this integration approach is that a significant amount of the model run configuration is controlled by health system analysts as opposed to model developers. This is fine if there is really good communication between the two parties, but there’s often a big disconnect, because analysts sort of sit in a siloed place inside of health system IT And developers tend to be outside of direct health IT and structure. Usually this ends up devolving into a big game of telephone, as these parties that don’t normally talk to one another or have good relationships. So, as always, we need to work on this so part of our sociotechnical system.
This decision to do technical integration simultaneously with model development turned out to be fairly important. The learnings from technical integration directly impacted our choices for model development. For example, we realized that building the reporting workbench report was a relatively laborious process. Each column in the report took a good amount of time to build and validate. These columns corresponded to a variable (also known as a feature) that the model took as input. So the integration effort scaled linearly with the number of features we wanted to include in the model.
During early parts of development we were exploring models with thousands of features, as we had access to the features from RDW and had code to easily manage these features. However, once we learned more about integration effort we decided to cap the number of features being used to a fairly small number (around 10). We felt comfortable with this decision because we felt like we hit a good balance between performance and implementation time. Our early experiments indicated that we wouldn’t lose a ton of performance going from thousands of features to ten (something on the order of less than 10% relative decrease in AUROC) and we were fairly sure that we could implement and test the report with the allocated Epic analyst built time.
Cheers,
Erkin
Go ÖN Home
Generally, for this setup you have to a model package and some additional configuration. The model package contains your model and the code necessary to package your model in a manner that can be run on the hosting service and that you have additional configuration that determines the data passed to the model
