Medical Image Analysis

Medical image analysis is the use of computer vision (CV) and artificial intelligence (AI) to extract clinically useful information from medical images: detecting findings such as lung nodules or fractures, outlining organs and tumors, measuring them over time, and classifying disease. It works on X-ray, CT, MRI, ultrasound, PET, retinal photographs, endoscopy video, and digitized pathology slides. It is the interpretation step that follows medical imaging, the scanners and reconstruction methods that produce the images. Clinical analysis tools are regulated as medical devices, and the FDA's public list of AI-enabled medical devices holds well over a thousand authorizations, most of them in radiology.

How Medical Image Analysis Works#

Medical images arrive in the DICOM format, which stores pixel data together with metadata such as patient identifiers, scanner settings, and slice spacing. Preprocessing depends on the modality. CT pixel values are calibrated Hounsfield units, so a lung or bone "window" maps the relevant intensity range to the model's input, while MRI intensities have no fixed scale and are normalized per scan. CT and MRI produce 3D volumes of hundreds of slices that models process slice by slice or as volumes, and digitized pathology slides reach billions of pixels, so they are cut into tiles and handled with multiple instance learning when only slide-level labels exist.

The core tasks are standard vision tasks applied to anatomy:

• Image classification: Assigns a label to a whole image, such as whether a retinal photograph shows diabetic retinopathy that needs referral.
• Object detection: Draws boxes around findings such as nodules, fractures, or colon polyps, guiding the clinician's eye.
• Image segmentation: Labels every pixel or voxel to outline organs and tumors for volume measurement and radiotherapy planning.
• Image registration: Aligns scans of the same patient taken at different times or with different modalities, so changes can be measured.

The U-Net architecture was designed for biomedical segmentation in 2015, nnU-Net configures U-Net pipelines automatically for new datasets, and open-source frameworks such as MONAI package these tasks with medical-specific data loaders and transforms.

Fine-tuning a detector on labeled scans follows the same workflow as any other domain: the Ultralytics brain tumor dataset pairs MRI slices with tumor boxes, and training Ultralytics YOLO26 on it produces a model that marks tumor regions on new slices:

from ultralytics import YOLO

# Fine-tune a COCO-pretrained YOLO26 detector on labeled brain MRI slices
model = YOLO("yolo26n.pt")
model.train(data="brain-tumor.yaml", epochs=100, imgsz=640)

# Detect tumor regions on a new MRI slice
results = model("https://ultralytics.com/assets/brain-tumor-sample.jpg")
result = results[0]

# Save the slice with predicted boxes and confidence scores
result.save(filename="brain_tumor_result.jpg")

Applications in Healthcare#

A widely deployed pattern is triage. Software that flags a suspected brain bleed or large-vessel stroke on CT moves the study to the top of the radiologist's worklist, and the radiologist's final read doubles as a check on the model:

Other established uses span screening, treatment planning, and procedures:

• Screening: In Sweden's randomized MASAI trial, AI-supported mammography screening found 29% more cancers than standard double reading without more false positives and cut radiologists' screen-reading workload by 44%. In 2018, IDx-DR became the first FDA-authorized device to give a screening decision, for diabetic retinopathy, without a clinician interpreting the image.
• Oncology: Segmentation models outline tumors and organs at risk for radiotherapy planning and measure tumor size across follow-up scans.
• Pathology: Models scan whole-slide images for cancer, and in 2021 Paige Prostate became the first AI software the FDA authorized for finding cancer in prostate biopsy slides.
• Endoscopy: Real-time detectors highlight polyps during colonoscopy, and GI Genius became the first such AI device the FDA authorized, in 2021.

Challenges and Considerations#

• Expert labels: Annotation requires radiologists or pathologists, and experts disagree on difficult cases, so labeled datasets are small and expensive.
• Domain shift and shortcut learning: Scanners, protocols, and patient populations differ between hospitals. A 2018 study found that pneumonia classifiers trained on chest X-rays could identify which hospital an image came from and performed worse on X-rays from hospitals outside their training data.
• Bias in AI: Models trained on unrepresentative patient populations underperform for the groups they underrepresent.
• Explainability: Heatmaps and other explanations help clinicians check what a model relied on before they trust its output.
• Privacy and federated learning: Patient data falls under HIPAA in the US and the GDPR in Europe, DICOM headers and text burned into images must be de-identified, and federated learning trains models across hospitals without pooling patient data.
• Regulation and updates: Significant changes to an authorized model normally require a new regulatory submission. The FDA's final guidance of December 2024 on predetermined change control plans lets manufacturers specify planned model updates, and how they will validate them, in the original submission.