That sounds impossible at first. AI assistants typically need access to the data they are helping you process.

In this tutorial, you'll build an AI agent that doesn’t inspect sensitive medical images. Instead, it orchestrates a local de-identification pipeline through carefully designed tools, keeping the patient data entirely on your machine.

The technology that makes this possible is the Model Context Protocol (MCP), an open standard that lets AI models call external tools instead of relying only on their built-in capabilities.

In my previous article, How to Build an AI-Powered Medical Image De-Identification Pipeline for Clinical Research, we already saw how to build the de-identification tool - Aegis, an open-source tool built using a MONAI (PyTorch) pipeline that removes PHI from both DICOM metadata and image pixels using OCR and NER.

I have since extended it with local MCP (Model Context Protocol) server support, and in this article we'll build that server from scratch with FastMCP. Then we'll connect it to Claude Desktop, turning Claude into an AI agent that can run, monitor, and audit de-identification jobs through natural conversation.

One note before we start: while Aegis is the example throughout, the pattern in this tutorial applies to any Python tool you want to give an AI agent access to. If you have your own pipeline, CLI, or library, you can follow along and wrap that instead.

Table of Contents

• What You'll Build

• Prerequisites

• What Aegis Does

• How to Set Up Aegis

• What Is MCP, and Why Use It?

• How the Architecture Works

• Step 1: Design the Tool Surface

• Step 2: Build the MCP Server with FastMCP

• Step 3: Test with MCP Inspector — Before Any AI Is Involved

• Step 4: Connect Claude Desktop

• Step 5: Talk to Your Agent

• Verification

• Does the AI Ever See Patient Data?

• Security Considerations

• A Note on the Word "De-identification"

• Where This Fits, and What's Next

• Conclusion

• References

What You'll Build

By the end of this tutorial, you'll have:

• A local MCP server that exposes the Aegis de-identification pipeline as six tools.

• Claude Desktop connected to that server, with human-in-the-loop approval for each action.

• An agent you can talk to in plain English: "De-identify this folder and tell me if anything needs manual review."

• A verifiable audit trail on disk that you can check against everything the agent reports.

Prerequisites

To follow this tutorial, you should have:

• Intermediate Python experience

• The Aegis repository (or your own Python tool to wrap): https://github.com/lakshmi-mahabaleshwara/aegis

• Python 3.10 or later

• Claude Desktop installed (macOS or Windows); a free Claude account works for local MCP servers

• Node.js (only used for a testing tool, not for the server itself)

We'll use:

• The MCP Python SDK (mcp)

• FastMCP (included in the SDK)

• MCP Inspector for testing

• Claude Desktop as the MCP host

If you haven't read the previous article, you don't need to rebuild the pipeline from scratch. Cloning the Aegis repository is enough, but the earlier article explains what the pipeline is actually doing under the hood.

What Aegis Does

Aegis is a medical image de-identification pipeline that removes PHI from DICOM metadata and image pixels, records every action in audit reports, and routes uncertain cases for manual review.

How to Set Up Aegis

Before we build the server, get Aegis installed. The server we write in the next step imports this package, so this has to be in place first.

# Get the code
git clone https://github.com/lakshmi-mahabaleshwara/aegis.git
cd aegis

# Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate
# On Windows: venv\Scripts\activate

# Install Aegis (editable) plus the MCP server in one step.
# The [mcp] pulls in the MCP SDK; it also installs the
# `aegis-mcp` console command you'll point Claude Desktop at.
pip install -e ".[mcp]"

# One-time: download the OCR and NER model weights
python scripts/prefetch_models.py

The editable install (pip install -e) makes the monai_aegis package importable from any directory and puts the aegis-mcp console command on your PATH (inside the venv). The MCP server depends on this, because Claude Desktop launches it from its own working directory, not from the repository root.

With the package installed, we can start building the server that exposes it.

What Is MCP, and Why Use It?

The Model Context Protocol (MCP) is an open standard that lets AI applications call external tools. Instead of trying to solve everything from the information already in its context, an AI model can invoke functions exposed by an external program.

Three roles are involved:

• Host – the AI application (Claude Desktop in our case)

• Server – a small program you write that exposes tools

• Tools – Python functions with names, parameters, and descriptions that the AI can call

When Claude Desktop starts, it launches your MCP server and discovers the tools it exposes. It sees only each tool’s name, parameter schema, and docstring, not your implementation code. In practice, your docstrings become the prompt that helps Claude decide when to call a tool.

Why use MCP at all? If you’re comfortable with Python, you could call the Aegis library directly from your own scripts. MCP becomes valuable when you want an AI assistant to operate that pipeline through natural conversation. Instead of writing scripts or remembering command-line options, you can simply ask:

• “De-identify this folder.”

• “Is the batch finished?”

• “Which files need manual review?”

• “Summarize today’s run.”

The underlying pipeline never changes. MCP simply provides a safe interface between the AI and your software, allowing the model to orchestrate the workflow while the actual processing remains in your local Python application.

We’ll use Claude Desktop as the host because it supports MCP natively. There are no bridges, extra services, or network ports to configure the server communicates with Claude Desktop over standard input/output as a local subprocess, and the entire setup is configured through a single JSON file.

How the Architecture Works

Claude Desktop communicates with the MCP server, which invokes the Aegis pipeline locally. The pipeline processes the medical images on your machine, while only summaries such as counts, decisions, and file paths are returned to Claude.

Step 1: Design the Tool Surface

Before writing code, decide what the agent can do. Our server exposes six tools:

We designed these tools around three simple principles:

• Return summaries instead of sensitive data.

• Use background jobs for long-running tasks.

• Keep the tool surface small and focused.

Step 2: Build the MCP Server with FastMCP

Now let’s turn that design into a working MCP server. We will build it one piece at a time so you can reuse the same pattern for your own Python tools. The complete implementation lives in src/monai_aegis/mcp_server.py; the sections below show how it comes together.

1. The Server Instance

FastMCP (bundled with the MCP Python SDK) turns a decorated Python function into a tool.

from mcp.server.fastmcp import FastMCP

# creates a FastMCP server instance
mcp = FastMCP("aegis-mcp")

The string "aegis-mcp" is just the server's name. It's what Claude Desktop shows in its tool list. Every tool we add from here is a function decorated with @mcp.tool().

2. Your First Tool

A tool is a decorator, a typed signature, and a docstring. Here's the one that does the real work, de-identifying a single file:

@mcp.tool()
def deidentify_file(input_path: str, output_dir: str = "") -> dict:
    """De-identify a single medical image (DICOM, JPEG, or PNG).

    Scrubs DICOM header PHI and redacts burned-in pixel PHI using
    OCR and NER. Returns summary statistics and the output location
    only, never the redacted text or any pixel data.
    """
    ...
    return {
        "status": "success",
        "source_file": src.name,
        "output_dir": str(out),
        "pixel_regions_detected": len(pixel_rows),
        "pixel_decisions": decisions,   # e.g. {"redacted": 4, "safelisted": 10}
        "header_tags_scrubbed": tags_scrubbed,
        "needs_manual_review": decisions.get("low_confidence", 0) > 0,
    }

Notice that Claude only sees the docstring, while the tool returns summary statistics rather than image data or extracted text.

3. Never Print to stdout

Do not use print() in an MCP server because stdout is reserved for JSON-RPC. Send logs to stderr instead.

4. Long Jobs Need the Async Pattern

Processing an entire directory can take several minutes, which is longer than an MCP tool call should block. Instead of waiting synchronously, start_batch_job creates a background thread, immediately returns a job_id, and lets Claude poll the progress using get_job_status().

import threading, uuid

_jobs = {}

@mcp.tool()
def start_batch_job(input_dir: str, output_dir: str = "", mode: str = "auto") -> dict:
    """Start a background job that de-identifies all DICOM/image files
    in a directory. Returns immediately with a job_id. Use get_job_status
    to check progress, do not wait synchronously.
    """
    job_id = uuid.uuid4().hex[:8]
    _jobs[job_id] = {"job_id": job_id, "state": "queued",
                     "processed": 0, "total": None,
                     "decisions": {}, "errors": []}
    threading.Thread(
        target=_batch_worker, args=(job_id, input_dir, output_dir),
        daemon=True,
    ).start()
    return {"status": "started", "job_id": job_id,
            "next_step": f"Call get_job_status with job_id '{job_id}'."}

The worker thread updates _jobs[job_id] as it processes each file, and the polling tool just reads it back:

@mcp.tool()
def get_job_status(job_id: str) -> dict:
    """Return the current state, progress, and decision counts for a job."""
    return _jobs.get(job_id, {"status": "unknown", "job_id": job_id})

Since the MCP server is a long-running process, it can keep the job registry in memory throughout the conversation. If the server restarts, active job IDs are lost, but the de-identified files and audit reports remain safely on disk.

5. Heavy Models Need a Warm-Up

Aegis loads EasyOCR and a Stanford NER model, which takes time. The server builds the pipeline lazily and caches it, so it pays that cost once, and exposes a warm_up tool so the first real call doesn't run into a timeout while models load:

@mcp.tool()
def warm_up() -> dict:
    """Preload the OCR and NER models so the first real call is fast."""
    _get_pipeline()   # builds and caches the pipeline on first use
    return {"status": "ready"}

6. The Audit Tools Read the Records, Not the Images

The remaining tools read the reports and review folders that Aegis already produces. Because they summarize existing audit records rather than reprocessing images, Claude can answer questions about completed runs without accessing the underlying medical images.

@mcp.tool()
def summarize_run(run_dir: str) -> dict:
    """Audit a run from its CSV reports. Returns counts only — never text or tag values."""
    run = Path(run_dir).expanduser().resolve()
    pixels = _read_csv_rows(run / "aegis_pixel_detections.csv")
    tags = _read_csv_rows(run / "aegis_tag_actions.csv")
    return {
        "pixel_decisions": Counter(r["decision"] for r in pixels),  # redacted / safelisted / low_confidence
        "tag_actions": Counter(r["action"] for r in tags),          # REMOVE / REMAP / ZERO / DUMMY / ATTEST

@mcp.tool()
def list_review_queue() -> dict:
    """List files quarantined for manual review — names only, never contents."""
    names = sorted(f for _, _, fs in os.walk(REVIEW_DIR) for f in fs if not f.startswith("."))
    return {"count": len(names), "files": names[:50]}

Step 3: Test with MCP Inspector — Before Any AI Is Involved

If you are skeptical about all of this (I was), this step is for you. MCP Inspector is a debug UI that connects to your server and lets you click the tools manually.

npx @modelcontextprotocol/inspector /<ABSOLUTE PATH>/aegis/venv/bin/aegis-mcp

The Inspector opens on the Servers screen with your aegis-mcp server listed. Click the toggle to connect, it turns green when the server is running.

Next, open the Tools tab. You’ll see the six tools exposed by your MCP server. Run warm_up first, and watch the terminal where you started the Inspector to see the OCR and NER models load.

Next, run deidentify_file with the path to a test image. The Results panel shows the tool’s JSON response, while Messages shows the request and response exchanged with the server.

Step 4: Connect Claude Desktop

Open Claude Desktop and go to Settings → Developer → Edit Config. This opens (or creates) the MCP configuration file. On macOS it’s located at ~/Library/Application Support/Claude/claude_desktop_config.json, and on Windows at %APPDATA%\Claude\claude_desktop_config.json.

Add the following aegis entry under mcpServers:

{
  "mcpServers": {
    "aegis": {
      "command": "/<PATH TO AEGIS>/aegis/venv/bin/aegis-mcp",
      "args": [],
      "env": {
        "AEGIS_OUTPUT_DIR": "/<PATH TO AEGIS>/aegis/staging_output",
        "AEGIS_REVIEW_DIR": "/<PATH TO AEGIS>/aegis/staging_not_processed",
        "AEGIS_DEVICE": "mps"
      }
    }
  }
}

Configuration for Claude Desktop to connect to the Aegis MCP server.

A few things to check before saving:

• Use absolute paths for command, AEGIS_OUTPUT_DIR, and AEGIS_REVIEW_DIR.

• Point command to the aegis-mcp executable inside your virtual environment so Claude uses the correct Python installation and dependencies.

• Set AEGIS_DEVICE to mps for Apple Silicon, cpu for Intel Macs or Linux, or cuda if you have an NVIDIA GPU.

Save the file, fully quit and restart Claude Desktop (closing the window isn’t enough). In a new chat, open the Search & Tools menu, you should now see the Aegis MCP server with its six available tools.

Step 5: Talk to Your Agent

Time for the first conversation. Send:

Warm up the Aegis de-identification server.

Claude will ask for permission before calling the tool. This prompt is a feature, every capability you have given the agent requires your explicit approval, and you can grant it per call or per tool.

Once approved, the models load and Claude reports back with the elapsed time.

De-identify a single file:

De-identify the file </test_ultrasound.dcm.>

Claude calls deidentify_file and answers with the counts: how many text regions were detected, how many were redacted versus safe listed as clinical text, how many DICOM tags were scrubbed, and whether anything needs review. It narrates all of this without ever having seen the image.

Next, a batch:

Now start a batch de-identification job for </test_ultrasound>

The tool returns instantly with a job ID, and the processing continues in the background. Ask about it a moment later:

How's that job going?

Claude remembers the job ID across turns and polls get_job_status, reporting files processed, running decision counts, and any errors.

Once the job finishes, you can ask Claude to summarize the run, list files requiring manual review, or generate a report for your review meeting, all from the pipeline’s audit records.

Summarize the completed run. Does anything need manual review?

What exactly was done to test_ultrasound.dcm, which header tags were touched?

Draft a short de-identification summary for this run, suitable for a review meeting. Include totals, the decision breakdown, and files pending human review.

Everything in the below summary came from tool results, counts and decisions read from the pipeline's own records.

Verification

Compare Claude’s summary with the CSV reports and job_summary_<job_id>.json. The numbers should match, giving you a simple way to verify everything the agent reports.

Does the AI Ever See Patient Data?

No. The tools never return image pixels, OCR-extracted text, or DICOM tag values, so Claude cannot access the underlying PHI.

For example, if you ask:

Show me the patient name detected in that file.

Claude cannot answer because that information is never exposed by the MCP tools.

However, conversation metadata does reach the model. File names, directory paths, counts, and tool outputs are processed by Claude just like any other chat. Keep these points in mind:

• Avoid PHI in file names. Rename files if they contain patient names or identifiers.

• Keep error messages clean. Don’t include sensitive DICOM values in exceptions returned by your tools.

• Use synthetic data while developing. This tutorial uses fake PHI. Before working with real patient data, follow your organization’s security and compliance requirements.

Security Considerations

If you adapt this pattern for your own tools, consider these best practices:

• Restrict file access. Limit tools to approved directories instead of allowing any readable path.

• Review tool permissions. It’s reasonable to always allow read-only tools like get_job_status, but keep approval prompts for tools that modify files.

• Return structured results. Prefer counts and categories over raw text to reduce the chance of exposing sensitive information.

• Pin model versions. Using fixed OCR and NER model versions makes your pipeline more reproducible and predictable.

A Note on the Word "De-identification"

In regulations such as HIPAA, de-identification has a specific legal meaning with defined requirements. This tutorial shows how to build an AI agent interface for a de-identification pipeline, not how to certify regulatory compliance. The pipeline intentionally routes uncertain cases for human review, and any real-world deployment should be validated against your organization’s policies and applicable regulations.

Where This Fits, and What's Next

MCP doesn’t change how well Aegis detects PHI, it changes how people interact with it. Instead of using the command line, users can run jobs, review results, and ask questions in natural language.

For automated workflows such as nightly batch processing, the CLI is still the better choice. MCP is best for interactive, human-in-the-loop tasks, while the CLI remains ideal for scheduled jobs. In both cases, the files and audit reports on disk remain the source of truth.

Here are a few directions you can explore next:

• Run everything locally. Pair the same MCP server with Ollama and Open WebUI so both the pipeline and the AI model stay on your machine.

• Strengthen security. Restrict tools to approved directories before deploying in a shared environment.

• Build dataset preparation workflows. Use the agent to de-identify data, summarize results, and prepare datasets for machine learning.

• Add downstream analysis. Run vision models on the de-identified outputs instead of the original images.

• Reuse the pattern elsewhere. The same architecture can orchestrate pipelines that remove sensitive information from legal documents, logs, financial records, or other confidential data.

Conclusion

We turned an existing Python pipeline into an AI-accessible tool using MCP. The key design principle is the model orchestrates the workflow but never sees the sensitive data.

This pattern extends beyond medical imaging. Any pipeline that handles sensitive information: such as legal documents, financial records, or personal photos can expose safe, structured tools while keeping the underlying data private.

You can find the complete implementation in the Aegis repository: https://github.com/lakshmi-mahabaleshwara/aegis. If you found this tutorial useful, consider starring the repository to help others discover it.

References

• Model Context Protocol documentation

• MCP Python SDK

• Getting started with local MCP servers on Claude Desktop

• How to Build an AI-Powered Medical Image De-Identification Pipeline for Clinical Research

• Aegis on GitHub

In this tutorial we’ll profile the dataset first, then let those observations drive every design decision in a MONAI segmentation pipeline.

What We'll Cover:

• Who is This For?

• About the Dataset

• What is MONAI, and Why Use it?

• What is Dice?

• Part 1 — Data Profile Before Modeling

  • Class Balance Drives the Loss

  • Patient Counts Drive the Split

• Part 2 — Building the Pipeline

  • A Single Config Object

  • The Patient-grouped Split

  • Transforms, Chosen by the Snapshot

  • Model, Loss, and Metric

• Reading the Results

• Prediction Visualization

• The Failure Modes Matter More Than the Average

• Where to Go Next

• Takeaway

• Reference

Who is This For?

This walkthrough assumes you have some comfort with Python and the basics of training a neural network. It explains the MONAI-specific pieces (dictionary transforms, DiceCELoss, DiceMetric) and the medical-imaging terms (BI-RADS, hypoechoic, patient-grouped folds) as they come up. No prior ultrasound experience is needed.

About the Dataset

The dataset is BUS-BRA, a public collection of breast ultrasound images with biopsy-proven labels and tumor segmentation masks.

Each image carries a benign/malignant label, a BI-RADS (Breast Imaging Reporting and Data System) category (a radiologist's suspicion score from 2 to 5), a histology string, and a binary tumor mask. The CSV that ships with it also includes predefined cross-validation folds.

The task is binary: separate tumor from background. BUS-BRA contains 1,875 B-mode breast ultrasound images from 1,064 patients, acquired on four scanners at a cancer institute in Brazil.

What is MONAI, and Why Use it?

MONAI (Medical Open Network for AI) is an open-source PyTorch framework built specifically for medical imaging. It's a domain-specific layer that sits on top of PyTorch: you still write standard PyTorch training loops, but MONAI provides the medical imaging-specific components so you don't have to build them yourself.

It gives you:

• Transforms for medical data, loading formats like DICOM and NIfTI, normalizing intensities, resizing, and augmenting, all in a dictionary-based pipeline that keeps an image and its mask in sync.

• Network architectures common in medical segmentation (U-Net, UNETR, SegResNet, and others) ready to instantiate.

• Loss functions and metrics designed for segmentation, including Dice-based losses and the Dice metric.

The result is less boilerplate and fewer chances for an image and its mask to drift out of alignment.

What is Dice?

Dice (the Dice similarity coefficient) measures how much two regions overlap. In segmentation, it compares the model's predicted mask against the ground-truth mask and returns a score from 0 to 1: 0 means no overlap at all, 1 means a perfect match.

The formula is:

Dice = 2 × (overlap) / (predicted area + true area)

The "2 ×" in the numerator is what keeps the score in the 0-to-1 range even though the denominator counts the overlapping pixels on both sides.

Two roles it plays in this tutorial:

• As a metric, Dice is how the run is scored. A validation Dice of 0.876 means the predicted tumor masks overlap the true masks by about 88% on average.

• As a loss (DiceCELoss), a Dice-based term is what the model trains against. This is the part that matters for the class-imbalance problem: because Dice measures overlap rather than per-pixel correctness, a model can't score well by labeling everything as background. A small tumor counts as much as a large one, so the model is pushed to actually find the tumor region.

Part 1 — Data Profile Before Modeling

This first pass is data profiling. It reads every image and mask once and answers a short list of questions whose answers determine how the pipeline must be built. Running these checks takes a few seconds and saves a lot of guesswork later.

The snapshot below summarizes the properties that directly influenced the pipeline design. We’ll let these observations determine each step of the workflow.

Two of these deserve a closer look because they shape the two most important decisions.

Class Balance Drives the Loss

Tumors are small. Across the dataset, background pixels outnumber tumor pixels by more than ten to one.

A model trained with ordinary binary cross-entropy can score around 91% pixel accuracy by labeling everything as background. This high number reflects the imbalance rather than any ability to find the tumor.

The fix is a loss that rewards overlap with the actual tumor region, which points directly at Dice.

Patient Counts Drive the Split

There are fewer patients than images because many patients contribute both a left-side and a right-side scan. If a random split puts one patient's left scan in training and their right scan in validation, the validation score is inflated by leakage.

The dataset authors already solved this: the CSV ships a K5P column: a 5-fold split where P stands for patient-grouped, meaning every image from a given patient lands in the same fold. Reusing it is safer than rebuilding the same grouping by hand.

With those answers in hand, the pipeline has a specification to build now.

Part 2 — Building the Pipeline

Everything below uses MONAI for the segmentation-specific work:
transforms, dataset wrapping, the network, the loss, and the metric.

A Single Config Object

The pipeline reads all its knobs from one dataclass. Nothing downstream hard-codes a constant, so re-running an experiment with a different fold or image size is a single edit.

from dataclasses import dataclass
from typing import Tuple, Optional
from pathlib import Path

@dataclass
class TrainConfig:
    data_root: Optional[Path] = None
     fold_column: str = "K5P"          # patient-grouped 5-fold (dev set)
    val_fold: int = 1                 # which K5P fold is validation
    test_column: str = "HOP"          # patient-grouped hold-out partition
    test_group: int = 1               # HOP value reserved as the test set

    image_size: Tuple[int, int] = (256, 256)
    batch_size: int = 16
    lr: float = 1e-3
    epochs: int = 30
    use_amp: bool = True              # mixed precision
    ckpt_path: str = "best_model.pt"

cfg = TrainConfig()

The code above defines a TrainConfig dataclass holding every setting the pipeline needs: the fold column and which fold to validate on, the target image size, batch size, learning rate, epoch count, a mixed-precision switch, and where to save the best model. Creating cfg once gives every later step a single place to read its settings from.

The Patient-grouped Split

The split uses two predefined columns. HOP (Hold-Out Partition) reserves a patient-disjoint slice as the test set, untouched until the very end. Within the remaining development set, one K5P fold becomes validation and the other four are training. Short assertions confirm no patient appears in more than one split.

dev_df   = manifest[manifest[cfg.test_column] != cfg.test_group]
test_df  = manifest[manifest[cfg.test_column] == cfg.test_group]

train_df = dev_df[dev_df[cfg.fold_column] != cfg.val_fold]
val_df   = dev_df[dev_df[cfg.fold_column] == cfg.val_fold]

# no patient may appear in more than one split
for a, b in [(train_df, val_df), (train_df, test_df), (val_df, test_df)]:
    assert not (set(a["Case"]) & set(b["Case"])), "patient leakage"

The above code first splits off the HOP test set, then divides the remaining development rows into validation (the chosen K5P fold) and training (the rest). It then checks that every pair of splits shares no patient Case. If any does, the assertion fails immediately.

Transforms, Chosen by the Snapshot

MONAI's dictionary transforms operate on records keyed by name ("image" and "label") and apply matched operations to both. Each step here answers a Part 1 data profile finding.

from monai.transforms import (
    Compose, LoadImaged, EnsureChannelFirstd, ScaleIntensityd,
    AsDiscreted, Resized, RandFlipd, EnsureTyped,
)
import torch

base = [
    LoadImaged(keys=["image", "label"], reader="PILReader", image_only=True),
    EnsureChannelFirstd(keys=["image", "label"]),
    ScaleIntensityd(keys="image"),                       # brightness spread
    AsDiscreted(keys="label", threshold=0.5),            # clean {0, 1} mask
    Resized(keys=["image", "label"],                     # hundreds of sizes
            spatial_size=cfg.image_size,
            mode=("bilinear", "nearest")),
]

train_transforms = Compose(base + [
    RandFlipd(keys=["image", "label"], prob=0.5, spatial_axis=1),  # horizontal
    EnsureTyped(keys=["image", "label"], dtype=torch.float32),
])
val_transforms = Compose(base + [
    EnsureTyped(keys=["image", "label"], dtype=torch.float32),
])

The above code builds a shared list of base steps, loads the PNG, moves the channel to the front, scales the image to [0, 1], binarizes the mask, and resizes both to 256×256. It then wraps that list in two pipelines. The training pipeline adds a random horizontal flip, and the validation pipeline does not, so evaluation always sees the image as-is.

Horizontal flips are a simple augmentation that preserve anatomical plausibility in this dataset. More aggressive augmentations, such as large rotations or elastic deformations, should be validated carefully because they may distort clinically meaningful structures.

Images use bilinear interpolation to preserve intensity gradients, while masks use nearest-neighbor interpolation so class labels remain strictly 0 or 1. Bilinear interpolation on masks would create artificial label values along object boundaries.

Model, Loss, and Metric

The network is a MONAI UNet with one input channel (grayscale) and one output channel (the tumor logit). The loss is the one the class-balance finding pointed at.

U-Net consists of an encoder that captures context at progressively coarser resolutions and a decoder that reconstructs fine spatial detail. Skip connections transfer high-resolution features directly from encoder to decoder, making U-Net especially effective for medical segmentation where boundaries matter.

from monai.networks.nets import UNet
from monai.losses import DiceCELoss
from monai.metrics import DiceMetric
from monai.transforms import Activations, AsDiscrete

model = UNet(
    spatial_dims=2, in_channels=1, out_channels=1,
    channels=(16, 32, 64, 128, 256), strides=(2, 2, 2, 2),
    num_res_units=2,
).to(device)

loss_fn = DiceCELoss(sigmoid=True)       # Dice handles the imbalance; CE smooths the gradient
metric  = DiceMetric(include_background=True, reduction="mean")
post_pred = Compose([Activations(sigmoid=True), AsDiscrete(threshold=0.5)])
   

The above code creates the U-Net (five resolution levels, one input and one output channel) and moves it to the GPU. It then defines the three pieces that surround it: the loss, the validation metric, and a post_pred step that turns raw model outputs into a clean 0/1 mask by applying a sigmoid and thresholding at 0.5.

DiceCELoss combines two terms. The Dice part is scale-invariant in the foreground area, so a small tumor counts as much as a large one and the model can't win by ignoring tumors. The cross-entropy part adds a smoother gradient where Dice is flat. The sigmoid=True flag tells the loss to apply the activation itself, so the model outputs raw logits and the post_pred step handles the sigmoid-and-threshold at evaluation time. This U-Net comes out to about 1.6 million parameters.

The training loop itself is mostly standard PyTorch. MONAI stays out of the optimization logic, the only segmentation-specific pieces are the loss, transforms, and evaluation metric.

for epoch in range(1, cfg.epochs + 1):
    model.train()
    for batch in train_loader:
        img, lab = batch["image"].to(device), batch["label"].to(device)
        optimizer.zero_grad(set_to_none=True)
        with torch.amp.autocast("cuda", enabled=cfg.use_amp):
            loss = loss_fn(model(img), lab)
        scaler.scale(loss).backward()
        scaler.step(optimizer); scaler.update()

    model.eval(); metric.reset()
    with torch.no_grad():
        for batch in val_loader:
            img, lab = batch["image"].to(device), batch["label"].to(device)
            pred = post_pred(model(img))
            metric(y_pred=pred, y=lab)
    val_dice = metric.aggregate().item()
    if val_dice > best_dice:
        best_dice = val_dice
        torch.save(model.state_dict(), cfg.ckpt_path)

In the above code, each epoch runs two passes. The training pass moves every batch to the GPU, computes the loss under mixed precision, and updates the weights through the gradient scaler. The validation pass then runs with gradients turned off, converts predictions with post_pred, and accumulates Dice across the fold. Whenever the epoch's Dice beats the best seen so far, the model weights are saved to disk.

Reading the Results

Two curves summarize the run. Training loss falls steadily and flattens near 0.12. Validation Dice climbs from about 0.57 to a plateau, with a best of 0.866 reached at epoch 28.

A few things are worth reading off these curves:

• The loss decreasing monotonically means the model is learning. The gradient signal is real.

• The loss flattening above zero rather than reaching it is expected. DiceCELoss has a floor, because the cross-entropy term never fully vanishes on ambiguous boundary pixels. A loss that reached zero would be a warning sign, not a triumph.

• Validation Dice plateauing above ~0.85 while training loss keeps falling is the mild-overfitting signature. Extra epochs mostly lower train loss without moving val Dice. It's not severe here, so the 30-epoch budget is fine, but a patience-based early-stopping rule would be a reasonable add.

A validation Dice of 0.866 sits in a reasonable range for a plain 2D U-Net on this dataset. But validation Dice measures a checkpoint chosen using that same set, so it runs a little optimistic.

The final, untouched check is the HOP test set, scored exactly once, after all training and model selection are done. It comes in at 0.864, essentially matching the 0.866 validation figure. The model generalizes to patients it never saw during training or selection, and the validation number wasn't hiding leakage.

Prediction Visualization

Metrics summarize overall performance, but they don’t show how the model is segmenting individual tumors.

The figure below presents a representative validation example. From left to right are the input ultrasound image, the ground-truth mask, the model’s predicted mask, and the prediction overlaid on the original image.

The close agreement between the prediction and the ground-truth annotation illustrates how the model localizes both the position and the boundary of the lesion.

The Failure Modes Matter More Than the Average

An average Dice of 0.866 can hide very different behaviors. It could mean every case is mediocre, or most cases are excellent and a few fail badly.

To distinguish between those possibilities, sort the validation set by per-image Dice and inspect the lowest-scoring predictions.

On this fold, only 4 of 299 validation cases scored below 0.5, about 1%. Looking at those four overlays surfaces a clear pattern. Three of the four worst predictions are fragmented: the model outputs several disconnected blobs where the ground truth is a single region. The fourth confuses a dark acoustic shadow, a common ultrasound artifact, for tumor tissue.

That fragmentation pattern connects straight back to a snapshot finding: the data-quality pass measured that every ground-truth mask in BUS-BRA is a single connected component. So a multi-blob prediction is wrong by a property of the dataset, which points at keeping only the largest connected component as a post-processing step:

from monai.transforms import KeepLargestConnectedComponent

post_pred = Compose([
    Activations(sigmoid=True),
    AsDiscrete(threshold=0.5),
    KeepLargestConnectedComponent(applied_labels=[1]),
])

This code rebuilds the post_pred pipeline with one extra step at the end. After the sigmoid and threshold produce a binary mask, KeepLargestConnectedComponent discards every predicted region except the largest one, so a prediction split into several blobs collapses to its single biggest piece. This matches the dataset's one-region-per-mask property.

I measured this on the validation set, and the honest result is more nuanced than "free accuracy." It recovers a few of the fragmented cases, but the net change in mean Dice is marginal and can even go slightly negative. When a real lesion is predicted as two touching pieces, discarding the smaller one throws away true-positive area. So it's a targeted lever for a specific failure mode, not a free boost: worth exploring, not adopting blindly.

The shadow-confusion case is harder still, telling a hypoechoic tumor from a dark shadow region sometimes needs context a small grayscale crop doesn't carry. This points toward higher resolution or a wider receptive field as directions for later experiments.

Where to Go Next

Once you have a reliable baseline, the next experiments become much more meaningful. Rather than randomly trying larger models, start from the failure modes you observed:

• Replace the 2D U-Net with Attention U-Net or DynUNet.

• Train at higher resolution to better capture small lesions.

• Apply connected-component analysis selectively during inference.

• Explore test-time augmentation.

• Compare DiceCE with Focal Tversky loss for highly imbalanced lesions.

Takeaway

The through-line is profile the data, then let what you find make the decisions.

The resize came from a resolution check. The loss came from a class-balance check. The split came from a patient count. The most useful post-processing idea came from a mask-component check run before training started. None of these were guesses, and none of them needed a sweep to discover.

A model is easy to build. A model whose every choice has a reason behind it is easier to trust, easier to debug, and easier to explain to the person who reads it next.

Reference

The complete, runnable code for this walkthrough is available as a MONAI notebook: busbra_segmentation_monai.ipynb. It runs top to bottom on Kaggle or Colab, and auto-downloads the dataset if it's not already attached.

The Problem

Imagine you’re building an AI system that removes patient information from medical images.

The model needs thousands of examples showing where Protected Health Information (PHI) appears and what it looks like. The more examples it sees, the better it becomes at finding and removing sensitive information.

But there is a problem:

The data you need to train the model is the same data you’re not allowed to share freely.

Healthcare organizations must protect patient privacy. Regulations like HIPAA require that patient identifiers are removed before medical images can be shared for research, AI development, or external collaboration.

This creates an interesting engineering challenge: How do you build and test de-identification systems when the data needed to train those systems can't be easily used?

One practical solution is Synthetic PHI.

In this article, I’ll show why synthetic PHI is valuable, explain the hidden PHI problem inside medical images, and walk through a pipeline my team built that generates realistic ultrasound datasets with fully controlled synthetic patient information.

What You'll Learn in This Tutorial

By the end of this tutorial, you'll understand:

• The hidden PHI challenges in medical imaging data.

• Why synthetic PHI is useful for building and testing healthcare AI systems.

• How to generate realistic synthetic patient identities using Python and Faker.

• How to inject PHI into both image pixels and DICOM metadata.

• How to create ground-truth labels for AI model training and evaluation.

• How to validate synthetic medical imaging datasets before using them in downstream workflows.

What We'll Cover:

• Source Images: OpenPOCUS

• The Iceberg Problem: Most PHI Is Hidden

• Why Synthetic PHI Matters

  • Challenge 1: Privacy Regulations

  • Challenge 2: Annotation at Scale

  • Challenge 3: Validation

• Synthetic PHI Solves All Three Problems

• Building a Synthetic PHI Pipeline

• Pipeline Architecture

• Safety Checks Before Burning

  • Step 1: Generate Synthetic Patient Identities

  • Step 2: Burn PHI into Image Pixels

  • Step 3: Add PHI to DICOM Headers

  • Step 4: Identity Mapping: The De-Identified PatientID

  • Step 5: Ground Truth: Structured CSV Output

• Three-Tier DICOM Validation

• A Surprising Bug: MONAI vs PIL

• Final Thoughts

Source Images: OpenPOCUS

The synthetic PHI generation uses lung point-of-care ultrasound (POCUS) frames from OpenPOCUS, an openly licensed collection of real ultrasound images contributed by the POCUS community.

These images carry no real PHI. OpenPOCUS provides clinically authentic ultrasound images while avoiding patient privacy concerns. This makes it an ideal foundation for synthetic PHI generation because we can focus entirely on creating and tracking identifiers without risking exposure of real patient information.

The Iceberg Problem: Most PHI Is Hidden

When people think about PHI in medical images, they usually think about visible text overlays.

These include:

Patient name
Medical Record Number (MRN)
Date of birth
Study date

These identifiers are often burned directly into image pixels by ultrasound, X-ray, CT, and MRI systems.

But visible text is only the tip of the iceberg. Much of the remaining PHI lives inside the DICOM header, a collection of metadata fields that describe the image and the study. These fields contains identifiers such as PatientName, PatientID, StudyDate, institution names, and other sensitive information.

Unlike burned-in text, header PHI isn't visible when looking at the image itself, but it travels with the file and must also be removed during de-identification.

A de-identification system must handle both.

Removing visible text while leaving PHI inside DICOM metadata still creates a privacy risk. Likewise, stripping metadata while leaving patient names burned into image pixels is equally problematic.

This hidden PHI challenge makes testing de-identification software much harder than it first appears.

Why Synthetic PHI Matters

At first glance, it seems hospitals already have plenty of real-world data available. So why not simply use that?

The answer comes down to three challenges.

Challenge 1: Privacy Regulations

Medical images often contain patient identifiers.

Sharing those images outside secure clinical environments introduces significant legal and compliance risk.

The more institutions involved, the more difficult governance becomes.

Challenge 2: Annotation at Scale

Modern AI systems require labeled examples.

Someone must identify:

• Where PHI appears

• What type of PHI is it

• Which DICOM tags contain PHI

Creating these annotations manually is expensive and time-consuming.

Challenge 3: Validation

Suppose you’re evaluating a de-identification tool. How do you know whether it successfully removed every identifier?

With real patient data, you often don’t know exactly where every piece of PHI exists. Without ground truth, measuring accuracy becomes difficult.

Synthetic PHI Solves All Three Problems

Instead of starting with real patient identifiers, we can generate realistic fake identities and intentionally inject them into medical images.

Because the pipeline creates the PHI itself, we know:

• Every identifier value

• Every pixel location

• Every DICOM tag

• Every expected output

This gives us perfect ground truth.

Now, a de-identification system can be evaluated objectively. If a patient name remains after processing, we know it failed. If clinical content is accidentally removed, we know that too.

Synthetic PHI creates a privacy-safe dataset that can be used for:

• Training AI models

• Benchmarking de-identification software

• Regression testing

• Validation before deployment

Building a Synthetic PHI Pipeline

To explore this problem, my team built a pipeline that generates synthetic PHI for lung Point-of-Care Ultrasound (POCUS) images.

The goal was to:

1. Start with ultrasound images containing no patient information.

2. Generate realistic synthetic patient identities.

3. Burn PHI into image pixels.

4. Insert matching PHI into DICOM metadata.

5. Automatically generate ground truth labels.

6. Validate the resulting DICOM files.

The output looks realistic from the perspective of a de-identification system while containing no real patient information.

Pipeline Architecture

The workflow looks like this (we'll go over each step in detail below):

Each stage produces artifacts consumed by the next stage. Failures are quarantined rather than silently ignored.

Safety Checks Before Burning

Before writing synthetic PHI onto an image, the pipeline performs a safety check to ensure that the selected region to insert PHI lies outside the ultrasound fan.

The top-left corner of a lung POCUS image is usually outside the imaging fan, a dark border, safe to burn PHI onto without obscuring clinical content.

To make sure this region holds good for every image, the pipeline runs two checks per image:

• Brightness check: If the average intensity of the configured burn region exceeds a threshold, the region likely overlaps the ultrasound fan rather than the dark border.

• Boundary check: The pipeline verifies that the configured burn region fits entirely within the image. Images that are smaller than the expected burn area are quarantined.

In either case, the image is quarantined with the reason recorded into the manifest. There are no partial burns, no overwritten clinical content, and no silent corruption of test data.

This prevents synthetic identifiers from accidentally obscuring anatomy.

def burn_region_is_safe(arr):
    """Check the burn region is dark enough to be outside the fan."""
    h, w = arr.shape
    y2 = min(BURN_REGION_Y + BURN_REGION_H, h)
    x2 = min(BURN_REGION_X + BURN_REGION_W, w)
    region = arr[BURN_REGION_Y:y2, BURN_REGION_X:x2]
    if region.size == 0:
        return False, float("nan")
    mean = float(region.mean())
    return mean <= BRIGHTNESS_SKIP_THRESHOLD, mean

The function extracts the configured burn region and computes its average brightness. If the region is too bright, it likely overlaps the ultrasound fan rather than the border.

Step 1: Generate Synthetic Patient Identities

The synthetic identity is produced by Faker and seeded per case, so the same image always yields the same fake patient.

Determinism matters because:

• Reproducing a test result requires reproducing the test data.

• Debugging downstream tools is easier when the input doesn't change between runs.

• Comparing two de-identification tools fairly requires both to see the same planted PHI.

def case_seed(global_seed: int, source_id: str) -> int:
    """Per-image deterministic seed derived from global seed and source path."""
    h = hashlib.sha256(f"{global_seed}|{source_id}".encode()).hexdigest()
    return int(h[:8], 16)


def generate_phi(seed: int) -> dict:
    fake = Faker()
    Faker.seed(seed)
    rng = random.Random(seed)

    last = fake.last_name()
    first = fake.first_name()
    middle = fake.random_letter().upper()
    mrn = f"{rng.randint(1000000, 9999999)}"
    dob = fake.date_of_birth(minimum_age=18, maximum_age=95)
    study_date = fake.date_time_this_decade()
    institution = rng.choice(INSTITUTION_POOL)

    return {
        "case_uuid": f"SYNTH-{uuid.UUID(int=rng.getrandbits(128))}",
        "patient_name_display": f"{last}, {first} {middle}.",
        "patient_name_dicom": f"{last}^{first}^{middle}",   # DICOM PN VR format
        "patient_id": mrn,
        "dob": dob,
        "study_date": study_date,
        "institution_name": institution,
    }

The case_seed() function generates a deterministic seed from the source image path. That seed is then used by Faker to create a synthetic identity.

Because the seed is repeatable, the same input image always receives the same synthetic patient information. This makes debugging and benchmarking reproducible.

Step 2: Burn PHI into Image Pixels

Rendering text onto an image is comparatively expensive. For a single zone containing 30+ frames, repeating that work per frame is wasteful.

The pipeline instead renders the PHI overlay onto a transparent canvas one time per zone. This mirrors how many ultrasound systems operate in practice, where patient information remains fixed while the underlying image content changes from frame to frame.

def make_phi_overlay(shape, phi):
    """Render PHI ONCE onto a canvas. Returns (overlay_array, overlays_meta)."""
    h, w = shape
    canvas = Image.new("L", (w, h), 0)  # blank canvas
    draw = ImageDraw.Draw(canvas)

    overlays, x, y = [], BURN_REGION_X, BURN_REGION_Y
    for entry in _phi_text_block(phi):
        x0, y0, x1, y1 = draw.textbbox((x, y), entry["line"], font=FONT)
        tw, th = x1 - x0, y1 - y0

        if x + tw > w or y + th > h:
            raise ValueError(
                f"rendered PHI overflows image: '{entry['line']}' "
                f"at ({x},{y}) size ({tw}x{th}), image {w}x{h}"
            )

        draw.text((x, y), entry["line"], font=FONT, fill=TEXT_COLOR)
        overlays.append({
            "phi_category": entry["phi_category"],
            "rendered_text": entry["line"],
            "phi_value": entry["value"],
            "bbox": [x, y, tw, th],
            "dicom_tag": entry["dicom_tag"],
        })
        y += th + LINE_GAP
    return np.array(canvas), overlays

The make_phi_overlay() function creates a blank canvas and renders each PHI line onto it. At the same time, it records metadata such as the rendered text, bounding box coordinates, and corresponding DICOM tag.

The function returns both the image overlay and the annotation metadata, ensuring that the ground truth always matches the pixels that were actually drawn.

Rendering once and reusing the overlay provides several advantages:

• Faster processing

• Consistent PHI placement across frames

• Simplified ground-truth generation

• Behavior that more closely matches real ultrasound devices

An additional benefit is that the pipeline automatically records the location of every burned identifier.

Step 3: Add PHI to DICOM Headers

The DICOM standard supports two ways to represent a cine ultrasound loop: as a sequence of single-frame DICOMs that share a series UID, or as one multi-frame DICOM where the pixel data holds every frame stacked together.

The pipeline uses the multi-frame approach because:

• It matches how real ultrasound devices write cine loops.

• One header serves all frames — no duplication of patient metadata.

• Storage and transfer are more efficient.

ds.PatientName = phi["patient_name_dicom"]
ds.PatientID = deid_patient_id
ds.PatientBirthDate = phi["dob"].strftime("%Y%m%d")

ds.StudyInstanceUID = study_uid
ds.StudyDate = phi["study_date"].strftime("%Y%m%d")
ds.InstitutionName = phi["institution_name"]

These fields populate the DICOM header with the same synthetic identity used in the image overlay. This ensures that visible PHI and hidden metadata remain consistent, producing realistic test data.

A few details that the DICOM standard enforces but the spec doesn't make obvious:

• StudyID is required and must be a short string, distinct from StudyInstanceUID. It's easy to forget.

• ImageType must be present. ["DERIVED", "SECONDARY"] is the honest value for synthetic data because it wasn't acquired by a device.

• Manufacturer is part of the General Equipment IOD module and is required even though the data is synthetic. Setting it to a clearly synthetic value (SYNTHETIC-DEID-TUTORIAL) makes the origin unambiguous.

Step 4: Identity Mapping: The De-Identified PatientID

To support downstream evaluation, every source patient receives a stable identifier such as DEID-0001. A mapping file links source patients, synthetic studies, and generated DICOM objects. This allows evaluators to compare a de-identification tool’s output against the original ground truth.

source_patient,deid_patient_id,study_instance_uid
patient_001,DEID-0001,1.2.826.0.1.3680043.8.498.1234...
patient_002,DEID-0002,1.2.826.0.1.3680043.8.498.5678...

Step 5: Ground Truth: Structured CSV Output

One major advantage of synthetic PHI is automatic label generation. Because the pipeline creates every identifier, it already knows the text value, bounding box coordinates, and corresponding DICOM tag.

These annotations are exported as structured CSV files and become the ground truth used for training and evaluation.

def build_overlay_rows(*, case_uuid, sop_instance_uid, source_id, source_relpath, output_dicom_relpath, overlays,
                      image_shape):
    h, w = image_shape
    rows = []
    for ov in overlays:
        x, y, ow, oh = ov["bbox"]
        rows.append({
            "case_uuid": case_uuid,
            "sop_instance_uid": sop_instance_uid,
            "source_id": source_id,
            "source_relpath": source_relpath,
            "output_dicom_relpath": output_dicom_relpath,
            "image_h": h,
            "image_w": w,
            "region": "top_left_banner",
            "phi_category": ov["phi_category"],
            "phi_value": ov["phi_value"],
            "rendered_text": ov["rendered_text"],
            "bbox_x": x, "bbox_y": y,
            "bbox_w": ow, "bbox_h": oh,
            "dicom_tag": ov["dicom_tag"],
            "seed": SEED,
            "pipeline_version": PIPELINE_VERSION,
            "run_id": RUN_ID,
        })
    return rows

build_overlay_rows function converts each overlay into a row of structured metadata. Along with the text and bounding box coordinates, it records identifiers and reproducibility information such as the pipeline version and random seed.

These CSV files become the ground truth used for training and evaluating de-identification systems.

At the end of the run, the accumulated rows are grouped by de-identified patient ID and written into per-patient CSV files. Each patient folder receives its own phi_overlays.csv covering all of that patient's zones, alongside a run_manifest.csv summarizing zone-level status (processed, quarantined, failed) and paths.

Three-Tier DICOM Validation

A synthetic DICOM file is only useful if it actually conforms to the DICOM standard. Otherwise, downstream tools that consume it will fail or worse silently mis-handle it.

The pipeline uses a three-tier validation chain that gracefully degrades depending on what's available in the environment:

1. dciodvfy from dicom3tools: the most rigorous standards-conformance validator, written by David Clunie. It's not pip-installable. It checks against the full DICOM IOD definitions. If it's available on PATH, this is the preferred check.

2. dicom-validator CLI: this is pip-installable. It downloads the DICOM standard definitions on first run, then validates IOD compliance. it's used when dciodvfy isn't available.

3. pydicom re-read: the minimal fallback. It confirms that every file can be re-opened, decoded, and that pixel data round-trips correctly. It doesn't check standards compliance, but catches gross corruption.

A Surprising Bug: MONAI vs PIL

Originally, I planned to use MONAI for image loading because it's widely used in medical imaging workflows.

During testing, I discovered an issue: MONAI’s image loading conventions caused non-square images to appear rotated when downstream code assumed traditional image layouts.

At the same time, many ultrasound images contained EXIF orientation metadata that required correction.

Switching to PIL solved both issues.

from PIL import Image, ImageOps

img = Image.open(path)
img = ImageOps.exif_transpose(img)

Final Thoughts

Synthetic PHI does not replace real-world testing, but it provides something healthcare AI teams rarely have: a safe, shareable, and fully labeled dataset with known answers.

By generating realistic identifiers and embedding them into both image pixels and DICOM metadata, we can build reproducible benchmarks for de-identification systems without exposing real patient data.

As AI systems become increasingly responsible for handling sensitive medical information, synthetic PHI may become one of the most important tools for building trustworthy healthcare AI workflows.

The complete implementation is available as a Jupyter notebook in the MONAI Ultrasound Working Group repository. You can explore the notebook and experiment with the pipeline yourself.

Sometimes the safest way to test whether a system can remove PHI is to create the PHI yourself.

The pipeline included:

• proper subject-grouped train/validation splits

• robust preprocessing

• carefully decoded segmentation masks

• sensible loss functions

• consistent evaluation

And the model still struggled to learn.

The interesting part isn't that the model underperformed. What mattered was the diagnosis: a series of simple checks that traced the problem back to the dataset, not the model.

Those checks are useful far beyond medical imaging. They apply to almost any machine learning project.

If you're new to ML, this is a lesson worth carrying into every project: understand your data before you tune your model.

I set out to build a medical image segmentation tutorial. I ended up learning a more valuable lesson: no amount of careful engineering can rescue a model from a dataset that can't support the task.

By the end of this article, you'll understand:

• How to evaluate whether a dataset can actually support your task

• Why "the model isn't learning" is often a data problem

• How to rule out engineering bugs before blaming the data

• Practical diagnostics you can run in minutes

• Why synthetic training data often struggles in real-world deployment

• When to stop tuning and walk away from a dataset

This is not a beginner introduction to deep learning – it assumes familiarity with concepts like UNet architectures and training loops. But the data-quality lessons apply broadly to many ML projects.

What We'll Cover:

• The Dataset

• Step 1: Rule Out the Pipeline Before Blaming the Data

  • Subject-grouped splits

  • Decoding masks correctly

  • Loss design and class weighting

• Step 2: The Model Still Struggled

• Step 3: Interrogating the Dataset

  • Diagnostic 1: What Does the Dataset Actually Contain?

  • Diagnostic 2: Do Synthetic and Real Images Look Similar?

  • Diagnostic 3: Can the gap be fixed by adding real data?

• Step 4: Knowing When to Stop

• A Practical Dataset Evaluation Checklist

• What I Would Try Next

• The Bigger Lesson

The Dataset

I used the US Simulation & Segmentation dataset, a public collection of abdominal ultrasound images with organ segmentation labels from Kaggle.

It contains:

• 926 synthetic ultrasound images — generated by a ray-casting simulator from CT scans, with full organ annotations

• 617 real ultrasound images — from an actual ultrasound scanner

• Labels for 8 organs — liver, kidney, gallbladder, pancreas, spleen, bones, vessels, and adrenals

At first glance, the dataset looked ideal:

• thousands of images

• multiple organ classes

• both synthetic and real ultrasound data

Whether it actually supported the task was a different question.

Step 1: Rule Out the Pipeline Before Blaming the Data

Ground rule: you should always rule out the pipeline before blaming the data. A model failing on buggy code looks exactly like a model failing on bad data. The engineering needs to be trustworthy.

Subject-Grouped Splits

A common mistake in medical imaging is randomly splitting images into train and test sets.

That approach is problematic because many frames come from the same patient. Those frames share anatomy, scanner settings, and noise patterns.

If frames from the same patient appear in both the train and test sets, the model can partially memorize patient-specific patterns. Test scores look artificially good, even though the model may fail on truly unseen patients.

This is called subject leakage.

The fix is to split by patient instead of by image:

from sklearn.model_selection import GroupShuffleSplit

def assign_splits(manifest, val_fraction=0.15, seed=42):
    train_data = manifest[manifest["orig_split"] == "train"]
    groups = train_data["subject_id"].values

    gss = GroupShuffleSplit(n_splits=1, test_size=val_fraction, random_state=seed)
    train_idx, val_idx = next(gss.split(X=train_data, y=None, groups=groups))

    train_subjects = set(train_data.iloc[train_idx]["subject_id"].unique())
    val_subjects = set(train_data.iloc[val_idx]["subject_id"].unique())

    # Crash loudly if leakage ever sneaks in
    assert train_subjects.isdisjoint(val_subjects), "Subject leak detected!"
    return train_subjects, val_subjects

That assertion matters. If the split logic ever breaks, the pipeline fails loudly instead of silently producing misleading metrics.

Decoding Masks Correctly

The dataset stores labels as color-coded masks. Each organ corresponds to a different RGB color.

Training requires converting those colors into integer class labels.

A naïve implementation uses exact color matching, but resizing operations can slightly alter colors at mask boundaries.

A more robust approach maps each pixel to its nearest palette color:

import numpy as np

PALETTE = np.array([
    [0, 0, 0],
    [100, 0, 100],
    [255, 255, 255],
    [0, 255, 0],
    [255, 255, 0],
    [0, 0, 255],
    [255, 0, 0],
    [255, 0, 255],
    [0, 255, 255],
], dtype=np.int32)

def decode_mask(mask_rgb):
    h, w = mask_rgb.shape[:2]
    flat = mask_rgb.reshape(-1, 3).astype(np.int32)
    d2 = (
        (flat[:, None, :] - PALETTE[None, :, :]) ** 2
    ).sum(-1)
    classes = d2.argmin(axis=1).astype(np.uint8)
    return classes.reshape(h, w)

Before training, it’s worth visually checking a few decoded masks against the original images. This catches issues like incorrect palettes, RGB/BGR channel swaps, or resizing artifacts that silently corrupt labels.

These bugs rarely throw errors. Instead, the model simply learns poorly. And “trained on wrong labels” looks exactly like “the model can’t learn the data.”

Verifying masks early removes that uncertainty.

Loss Design and Class Weighting

For training, I usd standard MONAI segmentation losses. The goal wasn’t to aggressively maximize performance, but to establish a stable and trustworthy baseline.

The training curves below show that the model optimized normally: the loss decreased consistently, and the validation dice stabilized rather than diverging. This helped rule out optimization instability as the primary cause of poor final performance.

Three choices were deliberate:

• Dice + Cross-Entropy combined: Cross-entropy keeps learning stable early on – Dice directly rewards good region overlap. Together they balance each other.

• include_background=False for binary segmentation: In a single-organ task, background can be 85–90% of the pixels. Counting it in the loss drowns out the signal for the organ you actually care about, so it's better left out.

• Class weighting for multi-class segmentation: With organs of very different sizes, an unweighted loss lets the model ignore the small, rare ones and still score well. Weighting rare-class mistakes more heavily pushes back against that.

Step 2: The Model Still Struggled

The first experiment focused on liver segmentation — the simplest single-organ task in the dataset.

Dice scores range from 0 (no overlap) to 1 (perfect overlap).

Qualitatively, the predictions often captured rough liver regions but failed at boundaries and consistency across real scans.

Especially important:

• the model struggled even on synthetic in-domain data

• performance dropped further on real ultrasound images

At this point, two explanations were possible:

1. the model or pipeline was flawed

2. the dataset itself was limiting performance

Because the engineering had been carefully validated, the second possibility became worth investigating seriously.

That's where the real lesson began.

Step 3: Interrogating the Dataset

Rather than endlessly tuning the model, the productive move is to turn the diagnostic lens on the dataset.

Three simple checks revealed the real problem. None required retraining or expensive experiments.

Diagnostic 1: What Does the Dataset Actually Contain?

The first step was simply plotting the dataset composition.

• 926 labeled synthetic images (the bulk of training data)

• Only 60 labeled real images — less than 4% of the dataset

• 557 unlabeled real images — real data exists, but without labels it can't be used for supervised training

This immediately changed the interpretation of the dataset.

Although the dataset contains many real ultrasound scans, almost all labeled training data is synthetic.

The model is effectively trained on synthetic ultrasound and expected to generalize to real ultrasound.

That's a difficult transfer problem from the start.

The limitation is simple: the real images mostly don't have labels, so supervised training has very little real-world data to learn from.

Lesson: Before training anything, chart the dataset composition. A headline image count can be misleading. "1,500 images" sounds large until you discover that only a tiny fraction are labeled examples from the target domain.

Diagnostic 2: Do Synthetic and Real Images Look Similar?

The next question was whether the synthetic and real ultrasound images actually followed similar visual distributions.

Plotting intensity histograms showed a clear mismatch.

• synthetic images clustered heavily near darker intensities

• real ultrasound images had broader mid-range intensity distributions

The synthetic simulator captured anatomical geometry reasonably well, but it didn't reproduce the texture and noise characteristics of real ultrasound:

• speckle patterns

• intensity falloff

• scanner-specific artifacts

This is the classic synthetic-to-real domain gap.

The model learned features tuned to synthetic images and then encountered a substantially different distribution during evaluation. Poor transfer performance became expected rather than surprising.

Lesson: Whenever training and deployment happen on different domains — synthetic → real, scanner A → scanner B, hospital A → hospital B — measure the distribution shift directly. Simple histogram comparisons can reveal major problems in minutes.

Diagnostic 3: Can the gap be fixed by adding real data?

The obvious next idea was: why not include some real labeled data during training?

But before implementing that approach, it's worth checking how many distinct patients actually had labels.

Labeled real images: 60
Distinct subjects (labeled real): 4

Frames per subject:
  subject h: 26
  subject a: 16
  subject g: 10
  subject b: 8

Only four patients.

That result fundamentally changed the situation.

Proper medical imaging evaluation requires subject-grouped train/test splits. But with only four patients, any evaluation becomes statistically unstable.

Training on two or three patients and testing on one or two patients would produce highly unreliable metrics that depend heavily on which patient happened to be held out.

At that point, the dataset simply couldn't support trustworthy real-world evaluation.

Lesson: In medical imaging, count subjects, not images. The true size of a dataset is bounded by the number of independent patients, not the number of files.

Step 4: Knowing When to Stop

At this point, additional tuning no longer made sense.

The bottleneck was not the architecture, optimizer, or learning rate. The bottleneck was the dataset itself.

The pipeline was still valuable and reusable. But this particular dataset couldn't reliably support the intended segmentation task.

That distinction matters: sometimes a problem is difficult but solvable, and sometimes the data simply can't support the conclusion you want to draw.

Learning to recognize the difference is an important ML skill.

A Practical Dataset Evaluation Checklist

Before committing weeks to model development, these checks are worth running on any dataset:

1. Chart the dataset composition — labeled vs unlabeled, class distribution, domain distribution

2. Count subjects, not images — independent patients matter more than frame count

3. Check class balance — rare classes are often ignored without weighting or sampling strategies

4. Compare train and deployment distributions — especially for cross-domain problems

5. Verify labels visually — catch preprocessing or annotation errors early

6. Look for published baselines — low published performance may indicate dataset limitations

These checks take minutes and can save weeks of unnecessary tuning.

What I Would Try Next

Improving results would likely require better data rather than a larger model. The next steps I'd prioritize:

• collecting more labeled real ultrasound scans, from more distinct patients

• improving annotation consistency

• semi-supervised learning to make use of the unlabeled real images

• domain adaptation between synthetic and real ultrasound

All of these target the actual bottleneck: data quality and data diversity.

The Bigger Lesson

In machine learning, it's easy to focus most of our attention on architectures, hyperparameters, optimization tricks, and newer models.

But the dataset quietly defines the ceiling.

A sophisticated model on weak data often disappoints, while a simpler model on strong data performs surprisingly well.

That was the real lesson from this project.

The most valuable skill wasn't building the pipeline. It was diagnosing why the model couldn't succeed and being willing to trust what the data was saying.

The workflow — checking dataset composition, counting subjects, comparing distributions, ruling out engineering bugs, and deciding when to stop — transfers to almost any ML project.

In many projects, better judgment about the data matters more than a better model.

The pipeline code and diagnostic notebooks are available at the MONAI Ultrasound Working Group repository. Questions, corrections, and improvements are always welcome.