A Guide to Developing AI systems in Medical Imaging

 

Are you interested in developing cutting-edge Artificial Intelligence (AI) systems tailored for medical applications? If you're considering embarking on this exciting journey, there are crucial factors to keep in mind right from the outset. Building AI products for healthcare presents unique challenges distinct from those in other fields. In this blog, lets go in detail about the journey of developing and deploying AI products in healthcare domain.

1. Clinical Problem Statement

First and foremost thing is identifying a problem statement with a true clinical need. While this may seem like an obvious starting point, it's essential to recognize that not all machine learning problem statements in clinical applications align with real-world clinical requirements. 

Imagine this scenario: There exists a non-invasive, cost-effective solution for a particular medical condition. Now, as an AI enthusiast, you're intrigued by the prospect of developing a new AI-driven product to tackle the same ailment. However, your product uses ionizing radiation. Though it might be an interesting problem as ML scientist at heart, clinical adaptability pose challenges unless it brings truly exceptional benefits compared to existing technologies.

That is why it is important to understand the existing technologies and also perform risk benefit analysis even before you attempt to develop a product. As per regulatory standards as well, it is recommended to perform risk benefit analysis in initial stages of product lifecycle. Thus, investing time and effort in comprehensive risk-benefit analysis becomes an invaluable safeguard, potentially saving resources and steering your innovation towards transformative impact.

2. Data Curation 

Once you decided to embark on the problem statement involving development of AI models, it is important to understand data. Unlike data in non-healthcare domains, healthcare data is often of 

• Low volumes: Data collection in clinics/hospitals mandates different approvals and often involves paying for data collection (either to patient or hospital). Consequently, researchers frequently grapple with the challenge of small datasets, particularly in the realm of medical imaging.
• Low variety: Medical conditions depend a lot on the population cohorts. Since due to the challenges in data collections, the data might be a collected in only certain locations, leading to low variety in data collected.
•  Low velocity: In medical settings, the rate of patient visits is typically low. If we are talking about breast cancer screening, on a normal day, the average number of walk-ins per day in a hospital could be at most 20 (~ 600 per month).

Apart from this, it is important that one should carefully check at the veracity of data labels. In medical imaging, most screening/diagnostic modalities are not 100% accurate. Further, human labeling is subjective. So, one should look for labels obtained from multiple imaging modalities or from multiple human interpreters. Other important aspects for data curation is anticipating potential biases eg: imbalance in datasets (there are lots of negative sample compared to positive samples), inclusion criterions in the dataset (Is the data collected from certain population cohorts?) or data sources (Is the data collected from a specific machine?)

3. AI Development

The next step is developing the AI models with the curated data. You can either choose to train off-the-shelf models or custom models. However, it is important one should integrate model interpretability and explainability into the AI development. If your AI models simply say 'Yes' or 'No' without proper explanation, clinical adaptability would be a major challenge as the doctors might not be able to rely on the generated outcome. Doctors, tasked with making life-altering decisions, require not just answers but insights. They need to understand the reasoning behind those answers, to trust the technology as a valuable ally.  Figure out ways you can explain the AI results. One can opt for Activation maps like GradCAMs, so that doctors can visualize these regions with high activations and take an informed decision.

As the training dataset are off low variety, make the model generalizable by employing techniques like data augmentation, image standardization etc. Always test on independent datasets and ensure no data leakage between train, valid and test sets.

Finally, it is important to know utility and limitations of your model. In some scenarios, it depends on the training data distribution as well. For instance, if your training data primarily comprises images from a symptomatic population, the model might yield elevated risk assessments when faced with asymptomatic cases. This is because the incidence of disease on symptomatic population is generally higher. Acknowledging these nuances is key to harnessing your AI model's potential effectively. 

4. Clinical Trial and Regulatory

This particular step sets healthcare applications apart from many non-healthcare endeavors and it stands as a crucial stage in bringing your product into clinical practice. Valid clinical studies and regulatory approvals from the respective country agencies are required for commercialization of your product. It's crucial to emphasize that planning your clinical studies and regulatory pathway should be integral to your product's lifecycle from the very outset. Both of these processes demand substantial investments of time, effort, and financial resources. Let us see some common mistakes during clinical trials related to study objectives:

Unclear or Ambitious objectives - It is important to define your objectives very clearly. Do not go for ambitious objectives as failure to meet these objectives could lead to a unsuccessful clinical trial. 

 

Objectives are not aligned with the Intended Use of the product -  The primary goal of conducting these trials is to generate the evidence necessary to support claims about the safety and efficacy of a medical product or intervention. However, if the objectives of the clinical trial are not closely aligned with the intended use that you plan to claim for your product, it can lead to significant issues including regulatory hurdles.

Neglecting Pilot Studies - Pilot studies can become crucial to identify any complications you might foresee during the actual clinical trial. It might allow you to refine the study protocol, assess the feasibility, assess the data quality etc.

5. Clinical Integration and Real World Surveillance

Many developers think that their job is done once their product is developed and regulatory cleared. A product is of no use if it is not usable. Medical professionals, including doctors, nurses, and other healthcare practitioners, may have varying levels of tech-savviness and familiarity with AI systems. It's crucial to engage with these end-users to understand their needs, challenges, and preferences. Few points to consider for better clinical adaptability can include:

• Conduct user testing and gather feedback during the development phase.
• Prioritize user-friendly interfaces and workflows.
• Offer training and support to ensure healthcare professionals can effectively use the AI product.

Other important aspect is the necessity of continuous model evolution. As there are large variants of a medical illness, it is important to continuously evolve your AI models.  Two potential challenges you foresee include. 

Data drift: It reflects the ever-changing nature of medical data, which can evolve over time due to various factors. It's crucial to monitor and adapt to these shifts, ensuring that your model remains accurate and dependable.

Concept drift: As medical knowledge advances, the concepts and relationships within healthcare data may evolve. Your AI model should be designed for continuous learning, with mechanisms in place to adapt to changing medical practices and insights.

Do post your questions in the comment section!