<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
    <channel>
        
        <title>
            <![CDATA[ dataengineering - freeCodeCamp.org ]]>
        </title>
        <description>
            <![CDATA[ Browse thousands of programming tutorials written by experts. Learn Web Development, Data Science, DevOps, Security, and get developer career advice. ]]>
        </description>
        <link>https://www.freecodecamp.org/news/</link>
        <image>
            <url>https://cdn.freecodecamp.org/universal/favicons/favicon.png</url>
            <title>
                <![CDATA[ dataengineering - freeCodeCamp.org ]]>
            </title>
            <link>https://www.freecodecamp.org/news/</link>
        </image>
        <generator>Eleventy</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:34:03 +0000</lastBuildDate>
        <atom:link href="https://www.freecodecamp.org/news/tag/dataengineering/rss.xml" rel="self" type="application/rss+xml" />
        <ttl>60</ttl>
        
            <item>
                <title>
                    <![CDATA[ How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial ]]>
                </title>
                <description>
                    <![CDATA[ Healthcare generates huge amounts of data every day, from patient visits and lab results to diagnoses, medications, and medical imaging. For developers and data engineers building healthcare applicati ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial/</link>
                <guid isPermaLink="false">6aff718ab7bbd995ee693880</guid>
                
                    <category>
                        <![CDATA[ SyntheticHealthcareData ]]>
                    </category>
                
                    <category>
                        <![CDATA[ healthcaredata ]]>
                    </category>
                
                    <category>
                        <![CDATA[ dataengineering ]]>
                    </category>
                
                    <category>
                        <![CDATA[ HealthcareAI ]]>
                    </category>
                
                    <category>
                        <![CDATA[ healthtech ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Lakshmi Mahabaleshwara ]]>
                </dc:creator>
                <pubDate>Thu, 19 Nov 2026 22:00:42 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/17f495db-a70b-427b-a69d-4b200d573081.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Healthcare generates huge amounts of data every day, from patient visits and lab results to diagnoses, medications, and medical imaging.</p>
<p>For developers and data engineers building healthcare applications, access to realistic patient data is essential. Whether you are building a healthcare analytics pipeline, testing an Electronic Health Record (EHR) system, or experimenting with machine learning, you need data that reflects how healthcare systems actually work.</p>
<p>But there is a challenge: real patient data is sensitive, highly regulated, and often difficult to access. So how can we develop and test healthcare applications without exposing real patient information? One solution is to generate synthetic healthcare data.</p>
<p>In this tutorial, you will learn how to use Synthea, an open-source synthetic patient generator, to create realistic patient records. We'll generate a population of approximately 100 patients, explore the resulting healthcare datasets using Python, and reconstruct a patient's clinical history.</p>
<p>By the end, you will have your own synthetic healthcare dataset that you can use for learning, testing, and prototyping.</p>
<p>I have also included a <a href="https://github.com/lakshmi-mahabaleshwara/healthtech-playground/blob/main/explore_synthea_healthcare_data.ipynb?utm_source=chatgpt.com">companion Jupyter notebook</a> that you can use to explore the generated data with Python, visualize clinical events, and understand a synthetic patient's healthcare journey.</p>
<h3 id="heading-what-well-cover">What We'll Cover:</h3>
<ul>
<li><a href="#heading-1-what-is-synthetic-healthcare-data">1. What Is Synthetic Healthcare Data?</a><ul>
<li><a href="#heading-synthetic-data-vs-de-identified-data">Synthetic data vs. de-identified data</a></li>
<li><a href="#heading-why-is-synthetic-healthcare-data-useful">Why is synthetic healthcare data useful?</a></li>
</ul>
</li>
<li><a href="#heading-2-what-is-synthea">2. What Is Synthea?</a><ul>
<li><a href="#heading-what-data-can-synthea-generate">What data can Synthea generate?</a></li>
</ul>
</li>
<li><a href="#heading-3-prerequisites">3. Prerequisites</a><ul>
<li><a href="#heading-check-your-java-installation">Check your Java installation</a></li>
</ul>
</li>
<li><a href="#heading-4-how-to-install-synthea">4. How to Install Synthea</a><ul>
<li><a href="#heading-step-1-clone-the-repository">Step 1: Clone the repository</a></li>
<li><a href="#heading-step-2-build-the-project">Step 2: Build the project</a></li>
<li><a href="#heading-step-3-check-the-available-commands">Step 3: Check the available commands</a></li>
</ul>
</li>
<li><a href="#heading-5-how-to-generate-100-synthetic-patients">5. How to Generate 100 Synthetic Patients</a><ul>
<li><a href="#heading-why-use-a-random-seed">Why use a random seed?</a></li>
<li><a href="#heading-where-are-the-generated-files">Where are the generated files?</a></li>
<li><a href="#heading-why-did-synthea-generate-more-than-100-patients">Why did Synthea generate more than 100 patients?</a></li>
</ul>
</li>
<li><a href="#heading-6-understanding-the-generated-healthcare-dataset">6. Understanding the Generated Healthcare Dataset</a><ul>
<li><a href="#heading-understanding-the-csv-relationships">Understanding the CSV relationships</a></li>
<li><a href="#heading-how-are-these-files-connected">How are these files connected?</a></li>
</ul>
</li>
<li><a href="#heading-7-how-to-explore-synthetic-healthcare-data-with-python">7. How to Explore Synthetic Healthcare Data with Python</a><ul>
<li><a href="#heading-step-1-install-pandas">Step 1: Install Pandas</a></li>
<li><a href="#heading-step-2-load-the-datasets">Step 2: Load the datasets</a></li>
<li><a href="#heading-step-3-examine-patient-records">Step 3: Examine patient records</a></li>
<li><a href="#heading-step-4-explore-encounters">Step 4: Explore encounters</a></li>
<li><a href="#heading-step-5-explore-clinical-observations">Step 5: Explore clinical observations</a></li>
</ul>
</li>
<li><a href="#heading-8-reconstructing-a-patients-clinical-journey">8. Reconstructing a Patient's Clinical Journey</a><ul>
<li><a href="#heading-step-1-select-a-patient">Step 1: Select a patient</a></li>
<li><a href="#heading-step-2-examine-the-patients-demographics">Step 2: Examine the patient's demographics</a></li>
<li><a href="#heading-step-3-find-the-patients-encounters">Step 3: Find the patient's encounters</a></li>
<li><a href="#heading-step-4-examine-their-conditions">Step 4: Examine their conditions</a></li>
<li><a href="#heading-step-5-examine-their-observations">Step 5: Examine their observations</a></li>
</ul>
</li>
<li><a href="#heading-9-how-to-validate-your-synthetic-healthcare-dataset">9. How to Validate Your Synthetic Healthcare Dataset</a><ul>
<li><a href="#heading-check-for-duplicate-patient-identifiers">Check for duplicate patient identifiers</a></li>
<li><a href="#heading-check-whether-encounters-reference-valid-patients">Check whether encounters reference valid patients</a></li>
<li><a href="#heading-examine-missing-values">Examine missing values</a></li>
</ul>
</li>
<li><a href="#heading-10-limitations-of-synthetic-healthcare-data">10. Limitations of Synthetic Healthcare Data</a><ul>
<li><a href="#heading-1-synthetic-data-isnt-real-clinical-evidence">1. Synthetic data isn't real clinical evidence</a></li>
<li><a href="#heading-2-simulated-patterns-may-differ-from-real-world-patterns">2. Simulated patterns may differ from real-world patterns</a></li>
<li><a href="#heading-3-synthetic-data-isnt-automatically-ready-for-machine-learning">3. Synthetic data isn't automatically ready for machine learning</a></li>
<li><a href="#heading-4-the-generated-population-depends-on-configuration">4. The generated population depends on configuration</a></li>
</ul>
</li>
<li><a href="#heading-11-conclusion">11. Conclusion</a></li>
</ul>
<h2 id="heading-1-what-is-synthetic-healthcare-data">1. What Is Synthetic Healthcare Data?</h2>
<p>Synthetic healthcare data is artificially generated information designed to resemble real healthcare records without representing actual patients.</p>
<p>For example, a synthetic patient might have:</p>
<ul>
<li>Demographic information such as age and gender</li>
<li>A history of primary care visits</li>
<li>Diagnosed medical conditions</li>
<li>Lab test results</li>
<li>Prescribed medications</li>
<li>Procedures and hospital visits</li>
</ul>
<p>These records are generated using simulation models and predefined rules and not copied from real patient histories.</p>
<h3 id="heading-synthetic-data-vs-de-identified-data">Synthetic Data vs. De-identified Data</h3>
<p>It's important to understand the difference between these two types of data.</p>
<p><strong>De-identified data</strong> starts with real patient records and removes or transforms identifying information, such as names, addresses, and other personal identifiers.</p>
<p><strong>Synthetic data</strong>, on the other hand, is generated artificially. It doesn't directly represent an actual patient's medical history.</p>
<p>Synthetic data offers another way to create development and testing datasets without relying on actual patient records. But synthetic data isn't automatically statistically representative, clinically accurate, or free from bias.</p>
<h3 id="heading-why-is-synthetic-healthcare-data-useful">Why is Synthetic Healthcare Data Useful?</h3>
<p>Developers and researchers can use synthetic data in a variety of ways. It lets them develop and test healthcare applications, build data ingestion and transformation pipelines, and experiment with healthcare data models</p>
<p>They can also use it to test interoperability between healthcare systems, prototype analytics and machine learning workflows, and learn more about certain healthcare concepts and data relationships.</p>
<p>For this tutorial, we'll focus on generating structured healthcare data and understanding how the records relate to one another.</p>
<h2 id="heading-2-what-is-synthea">2. What Is Synthea?</h2>
<p><a href="https://github.com/synthetichealth/synthea">Synthea</a> is an open-source synthetic patient population simulator developed by MITRE.</p>
<p>Instead of generating random rows of patient information, Synthea simulates patient lifecycles. It models healthcare events over time, including medical conditions, clinical encounters, medications, procedures, and laboratory observations.</p>
<p>For example, a simulated patient might develop a medical condition, visit a healthcare provider, undergo testing, and receive medication. These events become part of that patient's longitudinal health record.</p>
<p>Synthea uses configurable clinical modules and population characteristics to generate these histories.</p>
<h3 id="heading-what-data-can-synthea-generate">What Data Can Synthea Generate?</h3>
<p>Synthea can generate many types of data, including:</p>
<table>
<thead>
<tr>
<th>Healthcare data</th>
<th>Description</th>
</tr>
</thead>
<tbody><tr>
<td>Patients</td>
<td>Demographics and patient identifiers</td>
</tr>
<tr>
<td>Encounters</td>
<td>Hospital visits, outpatient consultations, and emergency visits</td>
</tr>
<tr>
<td>Conditions</td>
<td>Diagnoses and clinical findings</td>
</tr>
<tr>
<td>Observations</td>
<td>Laboratory results, vital signs, and clinical measurements</td>
</tr>
<tr>
<td>Medications</td>
<td>Medication prescriptions and history</td>
</tr>
<tr>
<td>Procedures</td>
<td>Clinical procedures and interventions</td>
</tr>
<tr>
<td>Allergies</td>
<td>Recorded allergies</td>
</tr>
<tr>
<td>Immunizations</td>
<td>Vaccination records</td>
</tr>
<tr>
<td>Imaging studies</td>
<td>Medical imaging study metadata</td>
</tr>
<tr>
<td>Claims</td>
<td>Healthcare billing and insurance data</td>
</tr>
</tbody></table>
<p>Synthea supports several export formats, including <strong>CSV</strong> (which is useful for data analysis, SQL databases, and data engineering), <strong>HL7 FHIR</strong> (a standard for exchanging healthcare information between systems, commonly represented as JSON resources), and <strong>C-CDA</strong> (an XML-based format for exchanging clinical documents).</p>
<p>We'll use CSV in this tutorial because it's easy to explore with Python and can be imported into relational databases.</p>
<h2 id="heading-3-prerequisites">3. Prerequisites</h2>
<p>Before getting started, you'll need:</p>
<ul>
<li>Java Development Kit (JDK) 17 or newer</li>
<li>Git</li>
<li>A terminal or command-line environment</li>
<li>Python 3, for the exploration section</li>
<li>Basic familiarity with CSV files and Python</li>
</ul>
<p>Synthea recommends supported long-term-support Java versions, such as Java 17 or 25.</p>
<p>I'll demonstrate the commands using macOS, but the same general workflow works on Linux and Windows.</p>
<h3 id="heading-check-your-java-installation">Check Your Java Installation</h3>
<p>Open Terminal and run:</p>
<pre><code class="language-bash">java -version
</code></pre>
<p>You should see an installed Java version, for example:</p>
<pre><code class="language-shell">openjdk version "17.x.x"
</code></pre>
<p>Make sure you're using a JDK rather than only a Java Runtime Environment.</p>
<p>If Java isn't installed, download a compatible JDK before proceeding.</p>
<p>You can also check Git:</p>
<pre><code class="language-bash">git --version
</code></pre>
<p>And Python:</p>
<pre><code class="language-bash">python3 --version
</code></pre>
<h2 id="heading-4-how-to-install-synthea">4. How to Install Synthea</h2>
<p>Let's download the Synthea source code from its official GitHub repository.</p>
<h3 id="heading-step-1-clone-the-repository">Step 1: Clone the Repository</h3>
<p>Open your terminal and execute:</p>
<pre><code class="language-bash">git clone https://github.com/synthetichealth/synthea.git
</code></pre>
<p>This downloads the Synthea project into a local directory.</p>
<p>Navigate into it:</p>
<pre><code class="language-bash">cd synthea
</code></pre>
<h3 id="heading-step-2-build-the-project">Step 2: Build the Project</h3>
<p>Synthea is written in Java and uses Gradle for building the application.</p>
<p>Run:</p>
<pre><code class="language-bash">./gradlew build
</code></pre>
<p>The included Gradle wrapper handles the required Gradle version, so you don't need to install Gradle separately.</p>
<p>The first build may take some time because Gradle needs to download dependencies. When the build completes successfully, you're ready to generate patients.</p>
<p>On Windows, you can use <code>gradlew.bat</code> instead of <code>./gradlew</code>.</p>
<h3 id="heading-step-3-check-the-available-commands">Step 3: Check the Available Commands</h3>
<p>Run:</p>
<pre><code class="language-bash">./run_synthea -h
</code></pre>
<p>This displays the available command-line options.</p>
<p>Some important options are:</p>
<table>
<thead>
<tr>
<th>Argument</th>
<th>Purpose</th>
</tr>
</thead>
<tbody><tr>
<td><code>-p</code></td>
<td>Target population size</td>
</tr>
<tr>
<td><code>-s</code></td>
<td>Random seed</td>
</tr>
<tr>
<td><code>-r</code></td>
<td>Simulation reference date</td>
</tr>
<tr>
<td><code>-a</code></td>
<td>Patient age range</td>
</tr>
<tr>
<td><code>-g</code></td>
<td>Gender filter</td>
</tr>
<tr>
<td><code>-o</code></td>
<td>Population overflow setting</td>
</tr>
</tbody></table>
<p>You can also modify configuration properties directly through command-line arguments.</p>
<p>We'll use these options in the next step.</p>
<h2 id="heading-5-how-to-generate-100-synthetic-patients">5. How to Generate 100 Synthetic Patients</h2>
<p>Now we'll generate our first synthetic healthcare population.</p>
<p>From your Synthea directory, execute:</p>
<pre><code class="language-bash">./run_synthea -s 12345 -p 100 \
  --exporter.csv.export=true
</code></pre>
<p>Let's understand what the command does.</p>
<ul>
<li><code>-s 12345</code> sets the random seed used during data generation. Using a fixed seed helps make experiments reproducible.</li>
<li><code>-p 100</code> requests a population size of 100 patients.</li>
<li><code>--exporter.csv.export=true</code> enables the CSV exporter.</li>
</ul>
<p>This last option is important. Synthea's default configuration doesn't enable CSV export, so running it without this setting may leave you with FHIR files and metadata but no <code>csv</code> directory.</p>
<h3 id="heading-why-use-a-random-seed">Why Use a Random Seed?</h3>
<p>Suppose you're building a data engineering pipeline and repeatedly testing your transformations.</p>
<p>You want your input dataset to remain stable between runs. Using a fixed seed helps achieve this. For stronger reproducibility, also pin the Synthea version and fix the simulation's reference date.</p>
<p>For example, you can add:</p>
<pre><code class="language-bash">-r 20261001
</code></pre>
<p>This sets the simulation reference date to October 1, 2026.</p>
<p>Reproducibility also depends on the generator version and configuration, and not every generated field is necessarily invariant across execution environments.</p>
<h3 id="heading-where-are-the-generated-files">Where Are the Generated Files?</h3>
<p>After Synthea finishes, you should see an output directory:</p>
<pre><code class="language-text">synthea/
│
├── output/
│   ├── csv/
│   ├── fhir/
│   └── metadata/
│
├── src/
├── gradlew
└── run_synthea
</code></pre>
<p>The files of interest are inside:</p>
<pre><code class="language-text">output/csv/
</code></pre>
<p>You can list them using:</p>
<pre><code class="language-bash">ls output/csv
</code></pre>
<p>Depending on your version and configuration, you'll find files such as:</p>
<pre><code class="language-text">patients.csv
encounters.csv
conditions.csv
observations.csv
medications.csv
procedures.csv
allergies.csv
immunizations.csv
careplans.csv
imaging_studies.csv
claims.csv
providers.csv
organizations.csv
</code></pre>
<p>Each CSV represents a different part of the simulated healthcare ecosystem.</p>
<h3 id="heading-why-did-synthea-generate-more-than-100-patients">Why Did Synthea Generate More Than 100 Patients?</h3>
<p>Here's something interesting I noticed during my experiment. I requested a population of 100, but Synthea generated <strong>115 patient records</strong>.</p>
<p>This happens because Synthea normally uses population overflow when generating its target population. The simulation may generate additional people when patients die before the reference date, so the exported records can include more than the requested number.</p>
<p>If you need exactly 100 generated patient records, you can disable overflow:</p>
<pre><code class="language-bash">./run_synthea -s 12345 -p 100 \
  -o false \
  --exporter.csv.export=true
</code></pre>
<p>But this changes how the target population is constructed. In particular, the demographics of the surviving population might no longer match the original population targets as closely.</p>
<p>For this tutorial, we'll keep the default behavior because it gives us a more representative example of how Synthea models a population.</p>
<h2 id="heading-6-understanding-the-generated-healthcare-dataset">6. Understanding the Generated Healthcare Dataset</h2>
<p>Before analyzing anything, let's see how much data we've generated.</p>
<p>Run:</p>
<pre><code class="language-bash">wc -l output/csv/*.csv
</code></pre>
<p>This counts the number of lines in each CSV file.</p>
<p>Here's a selection from my generated dataset:</p>
<table>
<thead>
<tr>
<th>File</th>
<th align="right">Records</th>
</tr>
</thead>
<tbody><tr>
<td>patients.csv</td>
<td align="right">115</td>
</tr>
<tr>
<td>encounters.csv</td>
<td align="right">5,985</td>
</tr>
<tr>
<td>conditions.csv</td>
<td align="right">4,070</td>
</tr>
<tr>
<td>observations.csv</td>
<td align="right">75,004</td>
</tr>
<tr>
<td>medications.csv</td>
<td align="right">4,454</td>
</tr>
<tr>
<td>procedures.csv</td>
<td align="right">16,777</td>
</tr>
<tr>
<td>imaging_studies.csv</td>
<td align="right">504</td>
</tr>
</tbody></table>
<p>These figures come from one simulation run. Your results may differ depending on the version, seed, reference date, and configuration.</p>
<p>Notice something important: we generated a relatively small population, but it produced thousands of clinical events.</p>
<p>That's because a patient may have multiple healthcare encounters over their lifetime, and each encounter can generate several conditions, observations, medications, or procedures.</p>
<h3 id="heading-understanding-the-csv-relationships">Understanding the CSV Relationships</h3>
<p>Let's look at six important files.</p>
<h4 id="heading-1-patientscsv">1. patients.csv</h4>
<p>Contains demographic and identifying information for each synthetic patient.</p>
<p>Some columns include:</p>
<pre><code class="language-text">Id
BIRTHDATE
DEATHDATE
GENDER
RACE
ETHNICITY
CITY
STATE
</code></pre>
<p>The <code>Id</code> column uniquely identifies a synthetic patient.</p>
<h4 id="heading-2-encounterscsv">2. encounters.csv</h4>
<p>Contains information about healthcare visits.</p>
<p>Important columns include:</p>
<pre><code class="language-text">Id
START
STOP
PATIENT
ENCOUNTERCLASS
DESCRIPTION
</code></pre>
<p>An encounter represents an interaction between a patient and the healthcare system. Examples include outpatient visits, emergency encounters, and hospital admissions.</p>
<p>The <code>PATIENT</code> column links each encounter back to a patient in <code>patients.csv</code>.</p>
<h4 id="heading-3-conditionscsv">3. conditions.csv</h4>
<p>Contains clinical conditions and diagnoses.</p>
<p>For example:</p>
<pre><code class="language-text">START
STOP
PATIENT
ENCOUNTER
SYSTEM
CODE
DESCRIPTION
</code></pre>
<p>The <code>SYSTEM</code> and <code>CODE</code> columns identify the terminology system and clinical concept.</p>
<h4 id="heading-4-observationscsv">4. observations.csv</h4>
<p>Contains clinical measurements, laboratory results, and other recorded observations.</p>
<p>Some columns include:</p>
<pre><code class="language-text">DATE
PATIENT
ENCOUNTER
CATEGORY
CODE
DESCRIPTION
VALUE
UNITS
TYPE
</code></pre>
<p>Observations might include blood pressure, heart rate, glucose levels, or laboratory test results.</p>
<p>An important detail is that not every observation value is numeric. Some observations contain text.</p>
<h4 id="heading-5-medicationscsv">5. medications.csv</h4>
<p>Contains medication-related records associated with patients and encounters.</p>
<h4 id="heading-6-procedurescsv">6. procedures.csv</h4>
<p>Contains records of procedures performed during clinical care.</p>
<h3 id="heading-how-are-these-files-connected">How Are These Files Connected?</h3>
<p>Here's the simplified view.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69fd77e89f93a850a46d376f/6e648bdb-f11b-4d2a-97a9-c39e2b423585.png" alt="Simplified healthcare data relationship diagram showing how patient records connect to encounters, conditions, observations, medications, and procedures through PATIENT and ENCOUNTER identifiers." style="display: block;" width="1238" height="483" loading="lazy">

<p>The two key identifiers you'll see frequently are:</p>
<ul>
<li><code>PATIENT</code>: Links a record to a patient.</li>
<li><code>ENCOUNTER</code>: Links a clinical event to a healthcare encounter.</li>
</ul>
<p>These relationships let us reconstruct a longitudinal patient history from multiple tables.</p>
<p>If you want to understand every column, the official <a href="https://github.com/synthetichealth/synthea/wiki/CSV-File-Data-Dictionary">Synthea CSV Data Dictionary</a> is a useful reference.</p>
<h2 id="heading-7-how-to-explore-synthetic-healthcare-data-with-python">7. How to Explore Synthetic Healthcare Data with Python</h2>
<p>Now that we have generated the dataset, let's explore it. We'll use Pandas because it provides a straightforward way to work with structured healthcare data.</p>
<h3 id="heading-step-1-install-pandas">Step 1: Install Pandas</h3>
<p>From the Synthea directory, create a Python virtual environment:</p>
<pre><code class="language-bash">python3 -m venv .venv
source .venv/bin/activate
</code></pre>
<p>On Windows, activate the virtual environment using the appropriate script for your shell.</p>
<p>Then install Pandas:</p>
<pre><code class="language-bash">python -m pip install pandas
</code></pre>
<h3 id="heading-step-2-load-the-datasets">Step 2: Load the Datasets</h3>
<p>Create a file called:</p>
<pre><code class="language-text">explore_synthea.py
</code></pre>
<p>Add the following Python code:</p>
<pre><code class="language-python">from pathlib import Path
import pandas as pd

csv_dir = Path("output/csv")

files = [
    "patients",
    "encounters",
    "conditions",
    "observations",
    "medications",
    "procedures",
]

tables = {
    name: pd.read_csv(
        csv_dir / f"{name}.csv",
        low_memory=False
    )
    for name in files
}

for name, df in tables.items():
    print(f"{name}: {len(df):,} records")
</code></pre>
<p>Run the script:</p>
<pre><code class="language-python">python explore_synthea.py
</code></pre>
<p>You should see the number of records in each dataset.</p>
<p>We're counting data rows here, so unlike <code>wc -l</code>, the header isn't included.</p>
<h3 id="heading-step-3-examine-patient-records">Step 3: Examine Patient Records</h3>
<p>Let's look at the patient table:</p>
<pre><code class="language-python">patients = tables["patients"]

print(patients.shape)
print(patients.columns.tolist())
</code></pre>
<p>We can select a smaller set of columns:</p>
<pre><code class="language-python">patient_view = patients[
    ["Id", "BIRTHDATE", "GENDER", "CITY", "STATE"]
]

print(patient_view.head())
</code></pre>
<p>Although the original data is synthetic, I prefer not to display unnecessary fields such as simulated Social Security numbers, passport numbers, or full addresses. <strong>Only use the fields needed for a particular purpose.</strong></p>
<h3 id="heading-step-4-explore-encounters">Step 4: Explore Encounters</h3>
<p>Let's examine the encounters table:</p>
<pre><code class="language-python">encounters = tables["encounters"]

print(
    encounters[
        ["Id", "PATIENT", "START", "ENCOUNTERCLASS"]
    ].head()
)
</code></pre>
<p>We can also examine encounter categories:</p>
<pre><code class="language-python">print(
    encounters["ENCOUNTERCLASS"].value_counts()
)
</code></pre>
<p>This helps us understand the different types of healthcare visits represented in our dataset.</p>
<h3 id="heading-step-5-explore-clinical-observations">Step 5: Explore Clinical Observations</h3>
<p>Let's examine the observations table:</p>
<pre><code class="language-python">observations = tables["observations"]

print(
    observations[
        ["DESCRIPTION", "VALUE", "UNITS"]
    ].head(10)
)
</code></pre>
<p>To see the most frequently recorded observations:</p>
<pre><code class="language-python">print(
    observations["DESCRIPTION"]
    .value_counts()
    .head(10)
)
</code></pre>
<p>This is where healthcare data becomes interesting.</p>
<p>The contains demographic information, but it also contains measurements and clinical events recorded over time.</p>
<p>For healthcare analytics and AI, understanding what each measurement represents is as important as understanding its numerical value.</p>
<h2 id="heading-8-reconstructing-a-patients-clinical-journey">8. Reconstructing a Patient's Clinical Journey</h2>
<p>We've explored the tables independently. But what happens when we want to understand one patient's healthcare history?</p>
<p>A patient might visit a physician several times, receive multiple diagnoses, undergo laboratory tests, and receive medication.</p>
<p>That information is distributed across different CSV files. Let's bring some of it together.</p>
<h3 id="heading-step-1-select-a-patient">Step 1: Select a Patient</h3>
<p>For our example, let's choose the patient with the most recorded encounters.</p>
<p>Add this code to the script:</p>
<pre><code class="language-plaintext">patient_id = (
    encounters["PATIENT"]
    .value_counts()
    .idxmax()
)

print("Selected patient:", patient_id)
</code></pre>
<p>We're selecting this patient because a larger number of visits gives us more events to explore. It doesn't necessarily mean the patient has more severe health conditions.</p>
<h3 id="heading-step-2-examine-the-patients-demographics">Step 2: Examine the Patient's Demographics</h3>
<pre><code class="language-plaintext">patient = patients[
    patients["Id"] == patient_id
]

print(
    patient[
        ["BIRTHDATE", "GENDER", "CITY", "STATE"]
    ]
)
</code></pre>
<h3 id="heading-step-3-find-the-patients-encounters">Step 3: Find the Patient's Encounters</h3>
<pre><code class="language-plaintext">patient_encounters = encounters[
    encounters["PATIENT"] == patient_id
].copy()

patient_encounters["START"] = pd.to_datetime(
    patient_encounters["START"],
    utc=True
)

patient_encounters = (
    patient_encounters
    .sort_values("START")
)

print(
    patient_encounters[
        ["START", "ENCOUNTERCLASS", "DESCRIPTION"]
    ].head(15)
)
</code></pre>
<p>Now we can see the patient's visits in chronological order.</p>
<h3 id="heading-step-4-examine-their-conditions">Step 4: Examine Their Conditions</h3>
<pre><code class="language-plaintext">conditions = tables["conditions"]

patient_conditions = conditions[
    conditions["PATIENT"] == patient_id
]

print(
    patient_conditions[
        ["START", "DESCRIPTION"]
    ].sort_values("START").head(15)
)
</code></pre>
<h3 id="heading-step-5-examine-their-observations">Step 5: Examine Their Observations</h3>
<pre><code class="language-plaintext">patient_observations = observations[
    observations["PATIENT"] == patient_id
]

print(
    patient_observations[
        ["DATE", "DESCRIPTION", "VALUE", "UNITS"]
    ].sort_values("DATE").tail(15)
)
</code></pre>
<p>We have now explored several pieces of one synthetic patient's medical history.</p>
<p>A patient's healthcare journey rarely exists in one table. Understanding it requires connecting different clinical events across time.</p>
<p>A single encounter may have several observations and multiple conditions, so joining both tables without considering their relationships can multiply records unexpectedly.</p>
<p>For this introductory example, examining each event type separately is easier to follow.</p>
<h2 id="heading-9-how-to-validate-your-synthetic-healthcare-dataset">9. How to Validate Your Synthetic Healthcare Dataset</h2>
<p>Before using any generated dataset, it's good practice to understand its structure and check important assumptions.</p>
<h3 id="heading-check-for-duplicate-patient-identifiers">Check for Duplicate Patient Identifiers</h3>
<pre><code class="language-plaintext">duplicate_ids = (
    patients["Id"].duplicated().sum()
)

print("Duplicate patient IDs:", duplicate_ids)
</code></pre>
<p>Patient identifiers should be unique within <code>patients.csv</code>.</p>
<h3 id="heading-check-whether-encounters-reference-valid-patients">Check Whether Encounters Reference Valid Patients</h3>
<pre><code class="language-plaintext">invalid_encounters = encounters[
    ~encounters["PATIENT"].isin(patients["Id"])
]

print(
    "Encounters referencing unknown patients:",
    len(invalid_encounters)
)
</code></pre>
<p>This verifies a basic relational integrity constraint.</p>
<h3 id="heading-examine-missing-values">Examine Missing Values</h3>
<pre><code class="language-plaintext">print(
    observations.isna().sum()
    .sort_values(ascending=False)
    .head(10)
)
</code></pre>
<p>Missing values don't always mean bad data. For example, Synthea can generate quality-of-life observations that aren't associated with a particular encounter. These records may legitimately have an empty <code>ENCOUNTER</code> field.</p>
<p>We should understand why something is missing before deciding whether to remove or replace it. This is especially important when working with healthcare data.</p>
<h2 id="heading-10-limitations-of-synthetic-healthcare-data">10. Limitations of Synthetic Healthcare Data</h2>
<p>Synthetic healthcare data is extremely useful, but it's important to understand what it can and can't represent.</p>
<h3 id="heading-1-synthetic-data-isnt-real-clinical-evidence">1. Synthetic Data Isn't Real Clinical Evidence</h3>
<p>Synthea simulates healthcare events using predefined models and rules.</p>
<p>The resulting data can look realistic, but it shouldn't be assumed to reproduce the full complexity of actual patient populations.</p>
<h3 id="heading-2-simulated-patterns-may-differ-from-real-world-patterns">2. Simulated Patterns May Differ from Real-world Patterns</h3>
<p>Real healthcare data can contain inconsistencies introduced by clinical workflows, patient behavior, billing practices, interoperability failures, and documentation differences.</p>
<p>Synthetic datasets may not capture all these complexities.</p>
<h3 id="heading-3-synthetic-data-isnt-automatically-ready-for-machine-learning">3. Synthetic Data Isn't Automatically Ready for Machine Learning</h3>
<p>Generating data is only the first step. Before developing a machine learning model, you may still need to:</p>
<ul>
<li>Validate clinical concepts and coding systems</li>
<li>Handle missing and inconsistent data</li>
<li>Integrate related datasets</li>
<li>Define the prediction target</li>
<li>Engineer meaningful features</li>
<li>Evaluate data representativeness and potential bias</li>
</ul>
<p>A model that performs well on synthetic data may not generalize to real clinical settings.</p>
<h3 id="heading-4-the-generated-population-depends-on-configuration">4. The Generated Population Depends on Configuration</h3>
<p>Synthea's clinical modules, population characteristics, reference date, and export settings can affect the generated records.</p>
<p>For reproducible experiments, document these settings along with the Synthea version and seed.</p>
<h2 id="heading-11-conclusion">11. Conclusion</h2>
<p>In this tutorial, we generated a synthetic healthcare population using Synthea and explored the resulting patient records.</p>
<p>You learned how to:</p>
<ul>
<li>Install and configure Synthea</li>
<li>Generate a synthetic patient population</li>
<li>Export healthcare records as CSV files</li>
<li>Understand patients, encounters, conditions, observations, and medications</li>
<li>Explore the data using Python and Pandas</li>
<li>Reconstruct a synthetic patient's clinical history</li>
<li>Perform basic dataset validation</li>
</ul>
<p>Tools like Synthea give developers a practical way to explore relationships in healthcare data without relying on access to real patient records.</p>
<p>The dataset we have generated can now serve as a starting point for building healthcare data pipelines, testing database designs, experimenting with interoperability standards, and exploring healthcare analytics.</p>
<p>Additional resources</p>
<ul>
<li><a href="https://synthetichealth.github.io/synthea/">Synthea Official Website</a></li>
<li><a href="https://github.com/synthetichealth/synthea">Synthea GitHub Repository</a></li>
<li><a href="https://github.com/synthetichealth/synthea/wiki/CSV-File-Data-Dictionary">Synthea CSV Data Dictionary</a></li>
<li><a href="https://github.com/synthetichealth/synthea/blob/master/src/main/resources/synthea.properties">Synthea Configuration</a></li>
</ul>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Encrypt PII in Data Pipelines While Keeping It Searchable ]]>
                </title>
                <description>
                    <![CDATA[ At Intuit, my team built a data pipeline that processed TurboTax e-filing data. This data needed to be made available for dashboards, analytics, and other downstream use cases. One of the biggest chal ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-encrypt-pii-in-data-pipelines-while-keeping-it-searchable/</link>
                <guid isPermaLink="false">6ac4810141f9b6b6ff5bf1cd</guid>
                
                    <category>
                        <![CDATA[ dataengineering ]]>
                    </category>
                
                    <category>
                        <![CDATA[ datasecurity ]]>
                    </category>
                
                    <category>
                        <![CDATA[ encryption ]]>
                    </category>
                
                    <category>
                        <![CDATA[ #DataPipelines ]]>
                    </category>
                
                    <category>
                        <![CDATA[ PII ]]>
                    </category>
                
                    <category>
                        <![CDATA[ hmac ]]>
                    </category>
                
                    <category>
                        <![CDATA[ KMS ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Lakshmi Mahabaleshwara ]]>
                </dc:creator>
                <pubDate>Tue, 06 Oct 2026 05:02:57 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/b9b99f84-c0c8-46a0-b217-0b2e598c8ac1.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>At Intuit, my team built a data pipeline that processed TurboTax e-filing data. This data needed to be made available for dashboards, analytics, and other downstream use cases.</p>
<p>One of the biggest challenges was protecting <strong>Personally Identifiable Information (PII)</strong> such as Social Security Numbers (SSNs).</p>
<p>SSNs are highly sensitive and must be protected at every stage of the data pipeline. At the same time, analytics teams sometimes need to perform legitimate business operations such as identifying records associated with a particular customer or performing exact-match lookups.</p>
<p>So how do you protect sensitive data while still allowing controlled equality searches without exposing the plaintext?</p>
<p>In this article, I'll explain the encryption concepts behind this problem, including:</p>
<ul>
<li><p>Probabilistic vs. deterministic encryption</p>
</li>
<li><p>Equality searches on encrypted data</p>
</li>
<li><p>Base keys and derived keys</p>
</li>
<li><p>Key separation</p>
</li>
<li><p>Local vs. remote cryptographic operations</p>
</li>
<li><p>Tokenization</p>
</li>
<li><p>The security trade-offs involved in making encrypted data searchable</p>
</li>
</ul>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-why-encrypt-pii">Why Encrypt PII?</a></p>
</li>
<li><p><a href="#heading-aes-advanced-encryption-standard">AES: Advanced Encryption Standard</a></p>
</li>
<li><p><a href="#heading-probabilistic-encryption">Probabilistic Encryption</a></p>
</li>
<li><p><a href="#heading-deterministic-encryption">Deterministic Encryption</a></p>
<ul>
<li><a href="#heading-the-security-trade-off">The Security Trade-off</a></li>
</ul>
</li>
<li><p><a href="#heading-base-keys-and-derived-keys">Base Keys and Derived Keys</a></p>
<ul>
<li><p><a href="#heading-key-derivation-with-hkdf">Key Derivation with HKDF</a></p>
</li>
<li><p><a href="#heading-where-does-the-master-key-live">Where Does the Master Key Live?</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-local-cryptographic-operations">Local Cryptographic Operations</a></p>
</li>
<li><p><a href="#heading-remote-cryptographic-operations">Remote Cryptographic Operations</a></p>
</li>
<li><p><a href="#heading-deterministic-encryption-vs-hmac">Deterministic Encryption vs HMAC</a></p>
</li>
<li><p><a href="#heading-tokenization">Tokenization</a></p>
</li>
<li><p><a href="#heading-comparing-the-approaches">Comparing the Approaches</a></p>
</li>
<li><p><a href="#heading-access-control-still-matters">Access Control Still Matters</a></p>
</li>
<li><p><a href="#heading-what-we-learned-from-the-data-pipeline">What We Learned from the Data Pipeline</a></p>
</li>
<li><p><a href="#heading-key-takeaways">Key Takeaways</a></p>
</li>
</ul>
<h2 id="heading-why-encrypt-pii">Why Encrypt PII?</h2>
<p>PII such as SSNs, email addresses, phone numbers, and financial information is highly sensitive.</p>
<p>If you store plaintext PII directly in a data lake or database, a storage-layer compromise could expose large amounts of sensitive information.</p>
<p>Encryption provides protection by transforming plaintext into ciphertext:</p>
<pre><code class="language-plaintext">Plaintext
   │
   ▼
Encryption + Key
   │
   ▼
Ciphertext
</code></pre>
<pre><code class="language-plaintext">SSN:        111-22-3333

Encrypted:  8f92a7c1...
</code></pre>
<p>Without the appropriate cryptographic key, the ciphertext shouldn't reveal the original value.</p>
<p>For a data platform, though, encryption introduces another requirement.</p>
<p>Suppose an analytics application needs to find records for:</p>
<pre><code class="language-plaintext">SSN = 111-22-3333
</code></pre>
<p>If the SSN is encrypted, we don't want the application to decrypt the entire dataset just to perform an equality lookup.</p>
<p>This is where <strong>deterministic encryption</strong> becomes useful.</p>
<h2 id="heading-aes-advanced-encryption-standard">AES: Advanced Encryption Standard</h2>
<p>Before looking at probabilistic and deterministic encryption, let's briefly understand <strong>AES (Advanced Encryption Standard)</strong>.</p>
<p>AES (Advanced Encryption Standard) is a widely used symmetric encryption algorithm for protecting sensitive data. It uses a secret key to transform plaintext into ciphertext, and the same key is used to decrypt the ciphertext back into the original plaintext.</p>
<p>The diagram below illustrates this basic process: the plaintext is encrypted using AES and a secret key to produce ciphertext. The ciphertext can then be decrypted using the same secret key to recover the original plaintext.</p>
<p>In real-world systems, AES is used with different encryption modes and constructions depending on the security and application requirements. The way randomness, initialization vectors, or nonces are handled affects properties such as whether repeated encryption of the same plaintext produces the same or different ciphertext.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69fd77e89f93a850a46d376f/6d5b1313-1f73-460c-ac13-5e47f9661c6a.png" alt="Diagram showing symmetric AES encryption: plaintext is encrypted with a secret key to produce ciphertext, which can then be decrypted using the same key to recover the original plaintext." style="display: block;" width="1774" height="887" loading="lazy">

<p>For example:</p>
<pre><code class="language-plaintext">Plaintext:   111-22-3333
Key:         key1


        ↓ AES encryption


Ciphertext:  8f92a7c1...
</code></pre>
<p>AES is widely used to protect sensitive data such as PII. But how encryption behaves depends on how the encryption is constructed and used. One important distinction is whether randomness is used during encryption.</p>
<p>This brings us to <strong>probabilistic encryption</strong>.</p>
<h2 id="heading-probabilistic-encryption">Probabilistic Encryption</h2>
<p>Probabilistic encryption introduces randomness during encryption.</p>
<p>This means that even when we encrypt the <strong>same plaintext with the same key</strong>, the resulting ciphertext can differ each time.</p>
<pre><code class="language-plaintext">Encrypt("ABC", key1) → qwoeoewowe
Encrypt("ABC", key1) → cXcslslsd
Encrypt("ABC", key1) → fjkdfdfd
</code></pre>
<p>Although the input is the same:</p>
<pre><code class="language-plaintext">ABC
</code></pre>
<p>the encrypted values are different.</p>
<p>This randomness is intentional. It prevents someone looking at encrypted data from easily determining that two ciphertexts represent the same underlying value.</p>
<p>For example:</p>
<pre><code class="language-plaintext">Record 1 → qwoeoewowe
Record 2 → cXcslslsd
Record 3 → fjkdfdfd
</code></pre>
<p>An observer can't simply compare the ciphertexts and conclude that the records contain the same plaintext.</p>
<p>This makes probabilistic encryption a strong choice when <strong>confidentiality is the primary requirement</strong>.</p>
<h3 id="heading-advantages">Advantages</h3>
<ul>
<li><p>Provides strong protection against equality-pattern analysis</p>
</li>
<li><p>Makes repeated plaintext values look different after encryption</p>
</li>
<li><p>Suitable when encrypted values don't need to be directly compared</p>
</li>
</ul>
<h3 id="heading-limitation">Limitation</h3>
<p>The randomness that improves security also creates a challenge for analytics.</p>
<p>Suppose we want to find all records containing:</p>
<pre><code class="language-plaintext">SSN = 111-22-3333
</code></pre>
<p>If the same SSN was encrypted multiple times, we could have:</p>
<pre><code class="language-plaintext">111-22-3333 → X8a91...
111-22-3333 → P72k4...
111-22-3333 → M91q2...
</code></pre>
<p>The encrypted values are different, even though the underlying SSN is the same.</p>
<p>Therefore, a simple equality query such as:</p>
<pre><code class="language-plaintext">WHERE encrypted_ssn = encrypted_search_value
</code></pre>
<p>wouldn't work reliably.</p>
<p>This creates an important trade-off for data platforms: randomness provides stronger protection against pattern leakage, but it makes equality-based searching more difficult.</p>
<p>When analytics requires exact-match searches on sensitive fields, we need a different approach: <strong>deterministic encryption</strong>.</p>
<h2 id="heading-deterministic-encryption">Deterministic Encryption</h2>
<p><strong>Deterministic encryption</strong> is designed so that the same plaintext, encrypted under the same key and encryption context, produces the same ciphertext.</p>
<p>For Example:</p>
<pre><code class="language-plaintext">Encrypt("ABC", key)
    → adsfffdfd

Encrypt("ABC", key)
    → adsfffdfd

Encrypt("ABC", key)
    → adsfffdfd
</code></pre>
<p>The important property is:</p>
<pre><code class="language-plaintext">Same plaintext
      ↓
Same key + context
      ↓
Same ciphertext
</code></pre>
<p>This allows equality matching.</p>
<p>For example:</p>
<pre><code class="language-plaintext">SELECT *
FROM customer_data
WHERE encrypted_ssn = EncryptDeterministically(
    '111-22-3333',
    encryption_key
);
</code></pre>
<p>The application can generate the same ciphertext for the search value and compare it against the stored ciphertext.</p>
<h3 id="heading-the-security-trade-off">The Security Trade-off</h3>
<p>Deterministic encryption provides searchability, but that searchability comes at a cost.</p>
<p>Consider this dataset:</p>
<pre><code class="language-plaintext">Ciphertext
-----------
A9F82...
A9F82...
B72AC...
A9F82...
C81DE...
</code></pre>
<p>An attacker may not know that:</p>
<pre><code class="language-plaintext">A9F82... = 111-22-3333
</code></pre>
<p>But they can determine that the same plaintext occurs three times.</p>
<p>In other words, deterministic encryption leaks <strong>equality patterns</strong>.</p>
<p>If an attacker has additional information about the underlying dataset, they may be able to use those patterns to infer plaintext values.</p>
<p>This is particularly important for fields with a small number of possible values, such as:</p>
<ul>
<li><p>State codes</p>
</li>
<li><p>Boolean values</p>
</li>
<li><p>Gender categories</p>
</li>
<li><p>Small categorical fields</p>
</li>
<li><p>Other low-entropy attributes</p>
</li>
</ul>
<p>So you should use deterministic encryption deliberately and only when the equality-search requirement justifies the additional leakage.</p>
<h2 id="heading-base-keys-and-derived-keys">Base Keys and Derived Keys</h2>
<p>Another important part of a secure encryption architecture is <strong>key management</strong>.</p>
<p>A common design uses a highly protected root or master key and derives separate keys for specific purposes. Instead of using one key everywhere.</p>
<p>The diagram below illustrates <strong>key separation</strong>. Instead of using the same key for every type of data, a highly protected master key can serve as the root of a key hierarchy. Separate keys can then be created for different datasets, cryptographic purposes, or environments.</p>
<p>For example, a dataset key could be used for a particular data domain, while a purpose-specific key could be dedicated to encrypting SSNs. This limits each key's scope and reduces the impact if one key is compromised.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69fd77e89f93a850a46d376f/c7c36e66-2c1d-4d43-8218-17f35ab52af6.png" alt="Diagram showing a master key securely stored in KMS or HSM and used to derive separate keys for different datasets, purposes, or environments, illustrating key separation." style="display: block;" width="1536" height="1024" loading="lazy">

<p>The idea with key separation is that different cryptographic purposes should use different keys or cryptographic contexts.</p>
<p>An organization might derive a key for:</p>
<pre><code class="language-text">Production + PII + SSN encryption
</code></pre>
<p>and another for:</p>
<pre><code class="language-plaintext">Production + PII + Email encryption
</code></pre>
<p>The exact hierarchy depends on the application's security requirements.</p>
<h3 id="heading-key-derivation-with-hkdf">Key Derivation with HKDF</h3>
<p>A common standard for deriving cryptographic keys is <strong>HKDF</strong>, or HMAC-based Key Derivation Function.</p>
<blockquote>
<p><strong>HKDF (HMAC-based Key Derivation Function):</strong> A standard method to derive multiple keys from a single master key.</p>
</blockquote>
<p>The diagram below shows how <strong>HKDF derives separate keys from a common master secret</strong>. HKDF takes the master secret as input, along with context information that identifies the intended purpose. For example, the context <code>"SSN encryption"</code> produces one derived key, while <code>"Email encryption"</code> produces another.</p>
<p>Because the context is different, the resulting keys are cryptographically separated even though they originate from the same master secret. You can use the same approach for other purposes, such as token generation.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69fd77e89f93a850a46d376f/1063ad45-f8b1-46ef-91fb-496d73e73551.png" alt="Diagram showing a master secret passed through HKDF to derive separate cryptographic keys using different contexts, including SSN encryption, email encryption, and token generation." style="display: block;" width="1536" height="1024" loading="lazy">

<p>The important point is that the application doesn't need to maintain a completely independent master secret for every purpose. Instead, you can use a securely managed root secret as the starting point for deriving purpose-specific keys.</p>
<p>So context is a key part of the key-derivation design.</p>
<p>For example:</p>
<pre><code class="language-python">DerivedKey =
    HKDF(
        master_secret,
        context = "production:ssn"
    )
</code></pre>
<p>Another context produces a different derived key:</p>
<pre><code class="language-python">DerivedKey =
    HKDF(
        master_secret,
        context = "production:email"
    )
</code></pre>
<p>This provides <strong>key separation</strong>.</p>
<p>A compromise of one derived key should not automatically expose data protected using independently derived keys.</p>
<p>But key derivation does not magically make compromised ciphertext safe. If a derived key is compromised, all data protected with that particular key may still be at risk.</p>
<h3 id="heading-where-does-the-master-key-live">Where Does the Master Key Live?</h3>
<p>The root key should not be stored inside application source code or configuration files.</p>
<p>Instead, organizations typically use a managed key-management system or a Hardware Security Module (HSM).</p>
<blockquote>
<p><strong>HSM (Hardware Security Module):</strong> A physical or cloud-based device that safely stores digital keys and performs encryption.</p>
</blockquote>
<p>Examples include:</p>
<ul>
<li><p>AWS Key Management Service (KMS)</p>
</li>
<li><p>Google Cloud KMS</p>
</li>
<li><p>Azure Key Vault</p>
</li>
<li><p>Dedicated HSM infrastructure</p>
</li>
</ul>
<p>The key-management system provides controlled access, auditing, rotation capabilities, and integration with identity and access-management systems.</p>
<p>The application should receive only the cryptographic material or cryptographic operation that it actually needs.</p>
<h2 id="heading-local-cryptographic-operations">Local Cryptographic Operations</h2>
<p>In a local encryption workflow, the application performs the actual data encryption.</p>
<p>In this workflow, the <strong>KMS protects the root or key-encryption key</strong>, while the application performs the actual encryption locally. The application first requests a data-encryption key from the KMS. The KMS returns the data key along with a protected copy of that key. The application can then use the data key to encrypt data without making a KMS request for every individual value.</p>
<p>For a high-volume data pipeline, this approach scales better because the application can encrypt large amounts of data locally while the KMS protects the higher-level key. The data key should be treated as sensitive and kept in application memory only for as long as necessary.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69fd77e89f93a850a46d376f/ed7c7f1a-f306-49c3-a5a7-dfb3e6496ccc.png" alt="Application requests a data key from KMS. KMS provides a data key along with a protected copy of the key. The application uses the data key to encrypt data locally and stores the encrypted data together with the encrypted data key." style="display: block;" width="936" height="924" loading="lazy">

<p>The application uses the appropriate data-encryption key or derived cryptographic key to perform encryption.</p>
<h3 id="heading-advantages">Advantages</h3>
<ul>
<li><p>High throughput</p>
</li>
<li><p>Fewer network calls for large datasets</p>
</li>
<li><p>Suitable for batch processing and data pipelines</p>
</li>
<li><p>Lower latency for individual encryption operations</p>
</li>
</ul>
<h3 id="heading-consideration">Consideration</h3>
<p>Cryptographic material used by the application exists temporarily in application memory.</p>
<p>Therefore, application security becomes an important part of the overall key-protection strategy.</p>
<h2 id="heading-remote-cryptographic-operations">Remote Cryptographic Operations</h2>
<p>In a remote encryption workflow, the application sends an encryption request to the KMS rather than performing the cryptographic operation itself. The KMS performs the operation using a protected key and returns the resulting ciphertext to the application. The underlying key material remains within the KMS boundary.</p>
<p>This provides centralized control and auditing of cryptographic operations, but every encryption request introduces a network interaction. For high-volume data processing, this can increase latency and may make the approach less suitable for encrypting individual values at very large scale.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69fd77e89f93a850a46d376f/7689eeda-27bb-4238-b364-a03f5834403d.png" alt="Application sends an encryption request to KMS. KMS performs the encryption using its protected key and returns the ciphertext to the application, which stores the encrypted data. The encryption key remains within KMS." style="display: block;" width="924" height="919" loading="lazy">

<p>The application doesn't receive the underlying key material. This can reduce direct key exposure and centralize cryptographic operations and auditing.</p>
<h3 id="heading-advantages">Advantages</h3>
<ul>
<li><p>Stronger centralization of key access</p>
</li>
<li><p>Keys can remain within the managed cryptographic boundary</p>
</li>
<li><p>Centralized authorization and auditing</p>
</li>
<li><p>Useful for operations where remote cryptographic APIs are appropriate</p>
</li>
</ul>
<h3 id="heading-trade-offs">Trade-offs</h3>
<ul>
<li><p>Network latency</p>
</li>
<li><p>API throughput limits</p>
</li>
<li><p>Additional operational dependencies</p>
</li>
<li><p>Potential cost for large numbers of cryptographic operations</p>
</li>
</ul>
<p>For high-volume data pipelines, calling a remote KMS for every field or row may not be practical.</p>
<h2 id="heading-deterministic-encryption-vs-hmac">Deterministic Encryption vs HMAC</h2>
<p>Another option is worth considering when the only requirement is equality matching.</p>
<p>Suppose the analytics team doesn't need to recover the original SSN from the stored value.</p>
<p>They only need to answer:</p>
<blockquote>
<p>"Does this record have the same SSN as the search value?"</p>
</blockquote>
<p>In that case, reversible encryption may not be necessary.</p>
<p>A <strong>keyed cryptographic hash</strong>, such as HMAC, can sometimes be a better design.</p>
<pre><code class="language-text">HMAC(secret_key, normalized_SSN)
</code></pre>
<p>For example:</p>
<pre><code class="language-plaintext">111-22-3333
      │
      ▼
HMAC(secret_key, SSN)
      │
      ▼
X7a91f...
</code></pre>
<p>The same SSN produces the same HMAC:</p>
<pre><code class="language-plaintext">111-22-3333 → X7a91f...
111-22-3333 → X7a91f...
222-33-4444 → P3k21...
</code></pre>
<p>This allows equality comparisons:</p>
<pre><code class="language-plaintext">WHERE ssn_hmac = HMAC(secret_key, '111-22-3333')
</code></pre>
<p>But unlike encryption, an HMAC isn't intended to be reversible.</p>
<p>This can be a useful security property when the original value doesn't need to be recovered from the analytics dataset.</p>
<p>The choice between deterministic encryption and HMAC depends on the requirements, threat model, key-management architecture, and whether reversibility is required.</p>
<h2 id="heading-tokenization">Tokenization</h2>
<p>Another common approach is <strong>tokenization</strong>.</p>
<p>Instead of storing:</p>
<pre><code class="language-plaintext">111-22-3333
</code></pre>
<p>the system stores something like:</p>
<pre><code class="language-plaintext">TOKEN-8F72A1
</code></pre>
<p>The mapping between the original value and the token is maintained by a secure tokenization service or vault.</p>
<p>The data pipeline and analytics systems can work with the token instead of the original PII.</p>
<p>The diagram below shows how <strong>tokenization separates the sensitive value from the systems that consume the data</strong>. The original SSN is sent to a secure token vault, which creates or retrieves a token representing that value. The vault securely maintains the relationship between the original SSN and its token.</p>
<p>The data lake receives only the token, rather than the original SSN. Downstream systems can use the token as an identifier for matching or joining records without directly storing the underlying PII. If an authorized system ever needs the original SSN, it can interact with the token vault to resolve the token, subject to appropriate access controls.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69fd77e89f93a850a46d376f/dd76572a-7c40-411b-b51e-b6c3fa92acb3.png" alt="Diagram showing an SSN sent to a secure token vault, which maps the original SSN to a token. The token is returned to the application and stored in the data lake instead of the original PII." style="display: block;" width="1774" height="887" loading="lazy">

<p>Tokenization can be particularly useful when many downstream systems need to work with an identifier without having access to the underlying sensitive value.</p>
<h2 id="heading-comparing-the-approaches">Comparing the Approaches</h2>
<p>The right choice depends on what the application actually needs.</p>
<table>
<thead>
<tr>
<th>Approach</th>
<th>Equality Search</th>
<th>Reversible</th>
<th>Equality Leakage</th>
<th>Typical Use</th>
</tr>
</thead>
<tbody><tr>
<td>Randomized encryption</td>
<td>❌</td>
<td>✅</td>
<td>Low</td>
<td>General encrypted data</td>
</tr>
<tr>
<td>Deterministic encryption</td>
<td>✅</td>
<td>✅</td>
<td>Yes</td>
<td>Equality lookups</td>
</tr>
<tr>
<td>HMAC</td>
<td>✅</td>
<td>❌</td>
<td>Yes</td>
<td>Equality matching without recovery</td>
</tr>
<tr>
<td>Tokenization</td>
<td>✅</td>
<td>Usually through a vault</td>
<td>Depends on design</td>
<td>Shared identifiers across systems</td>
</tr>
</tbody></table>
<p>There is no universally "best" approach.</p>
<p>The important question is <strong>what operations do downstream systems actually need to perform on the sensitive data?</strong></p>
<h2 id="heading-access-control-still-matters">Access Control Still Matters</h2>
<p>Encryption is only one part of protecting PII. Even when data is encrypted, you still need:</p>
<ul>
<li><p>Strong IAM policies</p>
</li>
<li><p>Least-privilege access</p>
</li>
<li><p>Key-access controls</p>
</li>
<li><p>Encryption-key rotation policies</p>
</li>
<li><p>Audit logging</p>
</li>
<li><p>Data classification</p>
</li>
<li><p>Network controls</p>
</li>
<li><p>Secure secret management</p>
</li>
<li><p>Data retention and deletion policies</p>
</li>
<li><p>Monitoring and alerting</p>
</li>
</ul>
<p>For example, an analytics user might be allowed to perform an equality lookup without being allowed to decrypt the entire SSN column.</p>
<p>This is an important distinction: <strong>the ability to query an encrypted identifier doesn't have to imply the ability to decrypt it.</strong></p>
<h2 id="heading-what-we-learned-from-the-data-pipeline">What We Learned from the Data Pipeline</h2>
<p>For our data pipeline, the key challenge wasn't simply:</p>
<blockquote>
<p>"How do we encrypt the SSN?"</p>
</blockquote>
<p>The more important question was:</p>
<blockquote>
<p>"How do we protect the SSN while still supporting legitimate analytics requirements?"</p>
</blockquote>
<p>That distinction changes the architecture.</p>
<p>A practical design needs to consider:</p>
<ol>
<li><p>What data is sensitive?</p>
</li>
<li><p>Who needs access to it?</p>
</li>
<li><p>Does the data need to be reversible?</p>
</li>
<li><p>Does the application need equality matching?</p>
</li>
<li><p>Does it need range or other query types?</p>
</li>
<li><p>What information can be safely exposed to downstream systems?</p>
</li>
<li><p>Where should cryptographic keys live?</p>
</li>
<li><p>How should keys be separated and rotated?</p>
</li>
<li><p>How much cryptographic processing can the pipeline perform locally?</p>
</li>
<li><p>What needs to happen inside a managed KMS or HSM?</p>
</li>
</ol>
<p>Once these questions are answered, the encryption mechanism becomes an architectural decision rather than simply a library choice.</p>
<h2 id="heading-key-takeaways">Key Takeaways</h2>
<p>When designing a data pipeline containing sensitive PII, there are some key takeaways to keep in mind.</p>
<p>First, AES is a cryptographic primitive. The encryption mode determines important properties such as nonce/randomness behavior.</p>
<p>Second, randomized encryption protects against equality-pattern leakage but doesn't directly support deterministic equality searches.</p>
<p>Deterministic encryption enables equality matching but leaks information about repeated values. And deterministic encryption doesn't automatically support range, prefix, or arbitrary searches.</p>
<p>Next, you can use HKDF to derive cryptographically separated keys from a root secret. Key separation limits the impact of compromising one cryptographic context.</p>
<p>KMS and HSMs provide secure key management and cryptographic boundaries.</p>
<p>HMAC can be a better choice than reversible encryption when you only need equality matching.</p>
<p>Tokenization can isolate sensitive identifiers from downstream systems.</p>
<p>And don't forget that encryption must be combined with IAM, auditing, monitoring, and least-privilege access.</p>
<p>Most importantly: the goal isn't simply to encrypt sensitive data. The goal is to design a system where sensitive data remains protected while legitimate business operations can still be performed safely.</p>
<p>That's the real challenge when building secure, large-scale data pipelines.</p>
 ]]>
                </content:encoded>
            </item>
        
    </channel>
</rss>
