<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
    <channel>
        
        <title>
            <![CDATA[ pydantic - freeCodeCamp.org ]]>
        </title>
        <description>
            <![CDATA[ Browse thousands of programming tutorials written by experts. Learn Web Development, Data Science, DevOps, Security, and get developer career advice. ]]>
        </description>
        <link>https://www.freecodecamp.org/news/</link>
        <image>
            <url>https://cdn.freecodecamp.org/universal/favicons/favicon.png</url>
            <title>
                <![CDATA[ pydantic - freeCodeCamp.org ]]>
            </title>
            <link>https://www.freecodecamp.org/news/</link>
        </image>
        <generator>Eleventy</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:32:54 +0000</lastBuildDate>
        <atom:link href="https://www.freecodecamp.org/news/tag/pydantic/rss.xml" rel="self" type="application/rss+xml" />
        <ttl>60</ttl>
        
            <item>
                <title>
                    <![CDATA[ Schema Drift vs. Semantic Drift in Data Pipelines ]]>
                </title>
                <description>
                    <![CDATA[ When a column is removed from a source database or dataset, the next pipeline run usually fails, making the problem easy to detect. But when a field keeps the same name and data type but its meaning c ]]>
                </description>
                <link>https://www.freecodecamp.org/news/schema-drift-vs-semantic-drift-in-data-pipelines/</link>
                <guid isPermaLink="false">6af6f326f94aa3564f351d31</guid>
                
                    <category>
                        <![CDATA[ Data pipelines ]]>
                    </category>
                
                    <category>
                        <![CDATA[ pydantic ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ data-engineering ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Yemi Ojedapo ]]>
                </dc:creator>
                <pubDate>Fri, 13 Nov 2026 11:23:18 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/ae47b7ae-2c13-49b9-bad7-a20a28edbecb.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>When a column is removed from a source database or dataset, the next pipeline run usually fails, making the problem easy to detect. But when a field keeps the same name and data type but its meaning changes (known as semantic drift), the pipeline can continue running normally, allowing the problem to go unnoticed.</p>
<p>This happens because a schema describes the structure of data, not what the values actually represent. It tells us which fields exist, their data types, and whether they can contain null values, but not whether a timestamp uses UTC, a durability score should range from 10 to 100, or an empty string means something different from null.</p>
<p>As a result, most pipeline checks are designed to catch obvious errors, such as missing columns or incorrect data types, but often fail to detect less obvious problems where the data looks correct but its meaning or values have changed.</p>
<p>I wanted to see how far this limitation reaches in practice, so I built a small lab where I introduced nine types of pipeline failures and tested four detectors against each one. This article looks at what each detector caught and what happened when changes went unnoticed and corrupted the data.</p>
<p>You can find the code <a href="https://github.com/hyemiie/game_inventory-schema_drift_lab-">here</a> and reproduce the experiment with a few commands.</p>
<h3 id="heading-what-well-cover">What We'll Cover:</h3>
<ul>
<li><a href="#heading-how-the-lab-works">How the Lab Works</a></li>
<li><a href="#heading-understanding-data-shape-and-meaning">Understanding Data Shape and Meaning</a></li>
<li><a href="#heading-the-results-matrix">The Results Matrix</a></li>
<li><a href="#heading-detecting-changes-in-data-shape">Detecting Changes in Data Shape</a></li>
<li><a href="#heading-semantic-changes-what-the-wrong-numbers-looked-like">Semantic Changes: What the Wrong Numbers Looked Like</a></li>
<li><a href="#heading-damage-vs-visibility">Damage vs. Visibility</a></li>
<li><a href="#heading-detecting-changes-in-data-meaning">Detecting Changes in Data Meaning</a></li>
<li><a href="#heading-what-this-experiment-suggests">What this Experiment Suggests</a></li>
<li><a href="#heading-limitations">Limitations</a></li>
</ul>
<h3 id="heading-prerequisites">Prerequisites</h3>
<p>To follow along and run the lab yourself, you’ll need:</p>
<ul>
<li><strong>Python 3.10 or newer</strong>, and basic comfort reading Python code</li>
<li><strong>Pydantic v2</strong>, installed with <code>pip install pydantic</code></li>
<li><strong>Git</strong>, to clone the repository</li>
<li><strong>A terminal</strong>, to run the scripts</li>
<li><strong>Basic SQL</strong>, such as <code>GROUP BY</code> and <code>WHERE ... IS NULL</code>. SQLite comes with Python, so there is nothing extra to install.</li>
<li><strong>A general idea of what a data pipeline does</strong>: reading data from a source, transforming it, and loading it somewhere else</li>
</ul>
<h2 id="heading-how-the-lab-works">How the Lab Works</h2>
<p>I intentionally kept the pipeline simple. It reads 1,000 game-inventory records from a JSONL file, flattens the nested <code>stats</code> and <code>attributes</code> objects, loads the data into SQLite, and calculates six metrics: total number of records, unequipped items, item counts by rarity, average attack per category, items acquired per day, and average durability. I also left out defensive checks so that each change could affect the pipeline in the same way it might affect a real pipeline when someone assumes the source data is reliable.</p>
<p>Each drift applies a deterministic function that takes one record and returns a changed one, so every result is repeatable. For instance, the timezone drift shifts each UTC timestamp to UTC+4 wall time and drops the offset:</p>
<pre><code class="language-python">def local_time_without_offset(row):
    timestamp = row["acquired_at"]
    if not timestamp.endswith("Z"):
        raise ValueError(f"Expected UTC timestamp ending in Z, got: {timestamp!r}")

    utc_time = datetime.fromisoformat(timestamp.removesuffix("Z") + "+00:00")
    local_time = utc_time + timedelta(hours=4)
    row["acquired_at"] = local_time.replace(tzinfo=None).isoformat()
    return row
</code></pre>
<p>After applying each drift, I ran the pipeline and saved its output in a separate SQLite database. I then compared it with the database generated from the original data, checking each row and column for differences. This step helped identify whether the pipeline had produced correct results, rather than simply checking whether it ran successfully.</p>
<p>The comparison placed each change into one of four categories:</p>
<ul>
<li>failed loudly (the error names a field the drift touched)</li>
<li>failed confusingly (the pipeline crashed but the error doesn't mention the cause)</li>
<li>passed with corruption (it finished, but values differ from the baseline)</li>
<li>or passed clean (the database matches the baseline exactly).</li>
</ul>
<p>I tested each drifted file with four detectors. Each detector follows the same structure: it takes the baseline file and the incoming file as input, checks for changes, and returns whether it found a problem along with a message. This shared interface keeps the setup simple and makes it easy to add more detectors.</p>
<p>The four detectors are:</p>
<ol>
<li><strong>Schema snapshot diff:</strong> Compares the fields and data types in the incoming file against the baseline to identify schema changes.</li>
<li><strong>Loose Pydantic model:</strong> Validates the data against a basic set of rules and restricts <code>rarity</code> to the five known values.</li>
<li><strong>Strict Pydantic model:</strong> Applies additional validation rules. It also restricts <code>rarity</code> to the five known values, requires timezone-aware timestamps, limits <code>equipped_slot</code> to five known values, and rejects unknown top-level fields. This refers to the validation rules in the model, not Pydantic's <code>strict=True</code> mode.</li>
<li><strong>Statistical profile:</strong> Compares the statistical profile of the incoming data with the baseline. It checks each field's mean, category shares, and null rate to identify changes that the other detectors might miss</li>
</ol>
<h2 id="heading-understanding-data-shape-and-meaning">Understanding Data Shape and Meaning</h2>
<p>The rest of this article uses two terms: <strong>data shape</strong> and <strong>data meaning</strong>. Data shape refers to the fields in a dataset, their paths, and the types of values they contain. The schema detector captures this information by checking each field and the types found in it.</p>
<pre><code class="language-python">def walk(obj, prefix, schema):
    for key, value in obj.items():
        field = f"{prefix}{key}"
        if isinstance(value, dict):
            walk(value, field + ".", schema)
        else:
            schema.setdefault(field, set()).add(type(value).__name__)
</code></pre>
<p>For the baseline, this returns entries like <code>acquired_at: {'str'}</code>, <code>equipped_slot: {'NoneType', 'str'}</code>, and <code>attributes.durability: {'int'}</code>.<br><br>Data meaning, however, includes everything the dictionary leaves out including the unit and time zone of a value, its expected range, the set of valid strings, and what a null value represents. This reveals a limitation of any detector that only looks at data shape. Since the schema detector bases its results on the dictionary of field paths and data types, it can't distinguish between datasets that produce the same dictionary but contain values with different meanings.</p>
<p>Three of the nine drifts left this dictionary unchanged: the timezone shift, the new enum value, and the durability scaling. As a result, the schema detector didn't flag any of them.</p>
<p>Detectors can check data at different levels:</p>
<ol>
<li>Shape: Checks which fields exist and the types of values they contain.</li>
<li>Value constraints: Checks whether individual values follow specific rules, such as requiring timezone information or limiting a field to a known set of values.</li>
<li>Distribution: Compares statistics, such as averages, category shares, and null rates, against a baseline. This can reveal changes in the data without defining exactly what each value should mean.</li>
<li>Semantics: Checks what the data actually represents, including units, timezone conventions, and the meaning of null values. These rules require knowledge of the data and must come from the people or systems that produce and use it.</li>
</ol>
<p>The four detectors cover the first three levels. The fourth level goes beyond what these detectors can establish because a dataset can pass checks on its structure, individual values, and statistical distributions while its meaning has changed. To know what the data means, we need context that can't be gotten from its values alone.</p>
<h2 id="heading-the-results-matrix">The Results Matrix</h2>
<p>The results matrix summarizes how each of the four detectors responded to the nine drift scenarios.</p>
<pre><code class="language-python">for drift in ALL_DRIFTS:
    dst = ROOT / "data" / "drifted" / f"{drift.name}.jsonl"
    drifted_db = DB_DIR / f"{drift.name}.db"

    apply_drift(drift, BASELINE, dst)
    result = run_pipeline(dst, db_path=str(drifted_db))
    outcome = classify(result, baseline_db, drifted_db, drift.fields)

    detected = {
        det.name: det.check(str(BASELINE), str(dst))["detected"]
        for det in DETECTORS
    }

    rows.append((drift, outcome, detected, result.get("error")))
</code></pre>
<p>It shows which changes they caught, which they missed, and whether the pipeline failed or continued with incorrect data.</p>
<table>
<thead>
<tr>
<th>Drift</th>
<th>What changed</th>
<th>Pipeline outcome</th>
<th>Rows corrupted</th>
<th>Schema diff</th>
<th>Pydantic (loose)</th>
<th>Pydantic (strict)</th>
<th>Statistical profile</th>
</tr>
</thead>
<tbody><tr>
<td><code>rename_rarity</code></td>
<td><code>rarity</code> renamed</td>
<td>Failed loudly</td>
<td>n/a</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td><code>drop_equipped_slot</code></td>
<td><code>equipped_slot</code> removed</td>
<td>Failed loudly</td>
<td>n/a</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td><code>stats_become_list</code></td>
<td><code>stats</code> object became an array</td>
<td>Failed confusingly</td>
<td>n/a</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td><code>add_column</code></td>
<td>New <code>guild</code> field</td>
<td>Passed clean</td>
<td>0</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td><code>type_change_level</code></td>
<td><code>level</code> int became string</td>
<td>Passed clean</td>
<td>0</td>
<td>Yes (only)</td>
<td>No</td>
<td>No</td>
<td>No</td>
</tr>
<tr>
<td><code>empty_equipped_slot</code></td>
<td>Null slots became <code>""</code></td>
<td>Passed with corruption</td>
<td>205</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td><code>new_enum_value</code></td>
<td>20 rows relabeled <code>mythic</code></td>
<td>Passed with corruption</td>
<td>20</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td><code>local_time_without_offset</code></td>
<td>Timestamps shifted to UTC+4, offset dropped</td>
<td>Passed with corruption</td>
<td>1,000</td>
<td>No</td>
<td>No</td>
<td>Yes (only)</td>
<td>No</td>
</tr>
<tr>
<td><code>durability_scaled</code></td>
<td>Durability multiplied by 10</td>
<td>Passed with corruption</td>
<td>1,000</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>Yes (only)</td>
</tr>
<tr>
<td><strong>Total caught</strong></td>
<td></td>
<td></td>
<td></td>
<td><strong>6 / 9</strong></td>
<td><strong>4 / 9</strong></td>
<td><strong>7 / 9</strong></td>
<td><strong>2 / 9</strong></td>
</tr>
</tbody></table>
<p>"Yes (only)” means that detector was the only one to catch the change. “n/a” means the pipeline crashed, so no rows were loaded.</p>
<h3 id="heading-pipeline-errors">Pipeline Errors</h3>
<table>
<thead>
<tr>
<th>Drift</th>
<th>Label</th>
<th>Error</th>
</tr>
</thead>
<tbody><tr>
<td><code>rename_rarity</code></td>
<td>Failed loudly</td>
<td><code>KeyError: 'rarity'</code></td>
</tr>
<tr>
<td><code>drop_equipped_slot</code></td>
<td>Failed loudly</td>
<td><code>KeyError: 'equipped_slot'</code></td>
</tr>
<tr>
<td><code>stats_become_list</code></td>
<td>Failed confusingly</td>
<td><code>TypeError: list indices must be integers or slices, not str</code></td>
</tr>
</tbody></table>
<p>The strict model detected the most changes (7 out of 9), followed by the schema diff (6), the loose model (4), and the statistical profile (2).</p>
<p>But each detector missed some changes. Three drifts were detected by only one detector: the strict model caught the timezone shift, the schema diff caught the <code>level</code> change, and the statistical profile caught the durability scaling.</p>
<h2 id="heading-detecting-changes-in-data-shape">Detecting Changes in Data Shape</h2>
<p>Five of the nine drifts changed the data structure. These included renaming <code>rarity</code>, removing <code>equipped_slot</code>, adding a <code>guild</code> field, replacing the <code>stats</code> object with an array, and changing <code>level</code> from an integer to a string.</p>
<p>The schema diff detected all five changes. The strict model detected four, and the loose model detected three. This shows that the schema diff was the most effective at detecting structural changes.</p>
<p>The pipeline failed on three of the drifts, but some error messages were more useful than others. Renaming <code>rarity</code> caused a <code>KeyError: 'rarity'</code>, which tells us exactly which field is missing. Replacing <code>stats</code> with <code>[attack, defense, speed]</code> caused a <code>TypeError: list indices must be integers or slices, not str</code>. This message explains the type of error but doesn't identify the field involved. To find the cause, the reader has to trace the error back through the transformation code.</p>
<p>The classifier labels the first error loud and the second <a href="http://confusing.It">confusing.It</a> makes this distinction by checking whether the error message mentions a field affected by the drift.</p>
<pre><code class="language-python">if result["status"] == "error":
    error_text = result["error"].lower()
    names_cause = any(f.lower() in error_text for f in drift_fields)
    return {
        "label": "failed_loudly" if names_cause else "failed_confusingly",
        "rows_affected": None,
        "loss_cells": 0,
        "corrupt_cells": 0,
        "extra_rows": 0,
    }
</code></pre>
<p>The Pydantic detectors ran on the same file and reported <code>stats: Input should be a valid dictionary</code>, directly identifying the field with the problem.</p>
<p>The <code>level</code> change reveals another way a structural change can go undetected. Both Pydantic models accepted it because Pydantic's default lax mode converts the string <code>"42"</code> into the integer <code>42</code>. The pipeline also accepted it because SQLite's INTEGER affinity converts text that looks like a number when inserting it into an INTEGER column. As a result, the database matched the baseline, and the classifier labeled the run <code>passed_clean</code>.</p>
<p>Only the schema diff detected the change because it checks the type of the raw value before any conversion takes place. This shows the limitation of checking data after coercion: once a value has been converted, the original type change is no longer visible. In this case, the simplest detector caught a change that both Pydantic models accepted.</p>
<p>The experiment also showed that not every detected change causes a problem. Adding <code>guild</code> and changing <code>level</code> from an integer to a string both triggered alerts. The schema diff detected both changes, while the strict model detected only the addition of <code>guild</code>. But neither change corrupted any rows because the load step reads only the fields it expects.</p>
<p>This highlights the need to distinguish between changes that should stop a pipeline and those that only need to be logged. If detectors flag too many harmless changes, users may start to ignore their warnings.</p>
<h2 id="heading-semantic-changes-what-the-wrong-numbers-looked-like">Semantic Changes: What the Wrong Numbers Looked Like</h2>
<p>The remaining drifts didn't change the fields or data types. They changed what the values represented. These changes were harder to detect but had real effects on the results. The table below shows the impact of each drift and whether it was visible in the pipeline's output.</p>
<table>
<thead>
<tr>
<th>Drift</th>
<th>Rows corrupted</th>
<th>Metric that moved</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody><tr>
<td>Timestamps shifted to UTC+4, offset dropped</td>
<td>1,000</td>
<td>items acquired per day</td>
<td>Jan 2: 9, Jan 3: 3</td>
<td>Jan 2: 6, Jan 3: 6</td>
</tr>
<tr>
<td>Null slots replaced with <code>""</code></td>
<td>205</td>
<td>unequipped item count</td>
<td>205</td>
<td>0</td>
</tr>
<tr>
<td>New <code>mythic</code> rarity on 20 rows</td>
<td>20</td>
<td>item count per rarity</td>
<td>common: 619</td>
<td>common: 609, mythic: 20</td>
</tr>
<tr>
<td>Durability multiplied by 10</td>
<td>1,000</td>
<td>average durability</td>
<td>54.78</td>
<td>547.85</td>
</tr>
</tbody></table>
<p>A major thing I noticed is that most of the numbers still look normal. For example, moving timestamps forward by four hours shifts some items onto the next calendar day. Since four hours is one-sixth of a day, roughly one in six items moves to a different day. Each day loses some items to the next day but also gains items from the previous day.</p>
<p>The total number of items remains 1,000, and the daily counts change very little overall. With an average of about 5.5 items per day, the counts already vary by two or three items. So a drop from 9 items to 6 can easily look like typical variation rather than a data problem.</p>
<p>The pipeline's total-row-count check still passed because no items were lost. Any check that only compares totals would also miss the problem. The total is correct, but some items are now counted under the wrong day.</p>
<p>I also looked at why the pipeline didn't report a problem. The original timestamp, <code>2026-01-02T22:30:00+00:00</code>, changed to <code>2026-01-03T02:30:00</code> after the drift. The new timestamp has no timezone offset, so <code>datetime.fromisoformat</code> reads it as a naïve datetime. The <code>isoformat()</code> method then writes it back without an offset. SQLite's <code>DATE()</code> treats this offset-free timestamp as UTC, placing the item on January 3 instead of January 2.</p>
<p>The pipeline processed the timestamp without errors but assigned the item to the wrong day. If the producer had written <code>2026-01-03T02:30:00+04:00</code> instead, the timestamp would have represented the same instant as the original. SQLite would have converted it to UTC, and the item would still have been counted on January 2.</p>
<p>The problem was the missing timezone offset. Without it, the pipeline had no way to know which timezone the timestamp represented, so it treated the time as UTC.</p>
<p>The empty-string drift caused a different problem. Replacing <code>null</code> with <code>""</code> kept the field as a string, so checks that only verify the data type still passed.</p>
<p>The schema diff caught the change because the type set for <code>equipped_slot</code> changed from <code>{'NoneType', 'str'}</code> to <code>{'str'}</code>. The pipeline raised no errors, but the query <code>WHERE equipped_slot IS NULL</code> returned no rows. The dashboard would show 0 unequipped items instead of 205, making it look as though every player had equipped their gear.</p>
<p>The statistical profile showed what had happened: <code>equipped_slot: null rate 20% -&gt; 0%; equipped_slot='': share 0% -&gt; 20%</code>. The null values had been replaced with empty strings, not real values.</p>
<p>The new <code>mythic</code> rarity went through the pipeline without an error for a similar reason. The drift changed 20 records to <code>mythic</code>: 10 from <code>common</code> (619 to 609), 9 from <code>uncommon</code> (236 to 227), and 1 from <code>rare</code> (98 to 97). The query <code>GROUP BY rarity</code> accepted the new value without any error. It simply created a new row for <code>mythic</code> and reduced the counts for the other rarities. A dashboard that only knows the five original rarities might hide <code>mythic</code> or group it under "other", depending on how it handles unknown values.</p>
<p>The durability change shows most clearly how data can keep the same structure but mean something different. In the original data, durability ranges from 10 to 100, so an average of 547.85 is clearly wrong. But the field kept the same name and integer type, so the schema diff didn't detect the change.</p>
<p>The pipeline also accepted the values because nothing checked them against the expected range. Only a check that knows durability should be between 10 and 100 can catch this problem. To test this, I added a range constraint to the strict Pydantic model:</p>
<pre><code class="language-python">class RangedAttributes(Attributes):
    durability: int = Field(ge=10, le=100)


class StrictWithRange(StrictItem):
    attributes: RangedAttributes


flagged = 0
for row in drifted:
    try:
        StrictWithRange(**row)
    except ValidationError:
        flagged += 1
</code></pre>
<p>This caught 985 of the 1,000 drifted records. The remaining 15 had an original durability value of 10, which became 100 after scaling by 10. Since 100 still falls within the allowed range, those records passed validation.</p>
<h2 id="heading-damage-vs-visibility">Damage vs. Visibility</h2>
<p>The four drifts corrupted 1,000, 205, 20, and 1,000 rows, but each one changed only one of the six metrics the pipeline calculates. The other five metrics stayed the same. Anyone monitoring the pipeline could miss the damage unless they checked the affected metric and knew its expected value.</p>
<p>The main finding is that data can change across many rows without changing most of the results people monitor. People only notice the problem if they check the metric that the drift affects.</p>
<h2 id="heading-detecting-changes-in-data-meaning">Detecting Changes in Data Meaning</h2>
<p>To catch changes beyond the data structure, we need checks that examine the values themselves.</p>
<p>The strict model checks whether values follow specific rules. It caught the timezone drift because <code>AwareDatetime</code> rejects timestamps without a timezone offset (<code>acquired_at: Input should have timezone info</code>). It also caught the empty-string slot because its <code>Literal</code> rule only accepts five known slot names and rejects <code>""</code>.</p>
<p>The loose model has fewer restrictions. It accepts any string for <code>acquired_at</code>, any string or <code>None</code> for <code>equipped_slot</code>, and unknown fields. As a result, it missed the timezone shift, the empty-string slot, and the new column.</p>
<p>The strict model also missed the durability change because I hadn't defined an allowed range for that field. In a separate run, I added <code>Field(ge=10, le=100)</code> to a copy of the model. This caught 985 of the 1,000 changed rows. It missed the remaining 15 because their original durability was 10, which became exactly 100 after scaling and still passed the check.</p>
<p>The statistical profile compares data distributions with the baseline. It was the only detector to catch the durability scaling because it doesn't need a predefined range. It simply detected that the average durability rose from 54.8 to 547.9.</p>
<p>But it may also flag normal changes in the data. To test this, I scaled durability by factors from 1.02 to 1.3 and tried four mean-change thresholds. I also tested the detector on 200 clean files generated with different seeds to see how often it raised false alarms. I excluded the baseline seed because it would produce identical data.</p>
<table>
<thead>
<tr>
<th>Mean threshold</th>
<th>False alarms, durability only</th>
<th>False alarms, any numeric field</th>
<th>Shifts caught</th>
</tr>
</thead>
<tbody><tr>
<td>25%</td>
<td>0 / 200</td>
<td>0 / 200</td>
<td>1.3x only</td>
</tr>
<tr>
<td>10%</td>
<td>0 / 200</td>
<td>0 / 200</td>
<td>1.1x and above</td>
</tr>
<tr>
<td>5%</td>
<td>0 / 200</td>
<td>23 / 200 (11.5%)</td>
<td>1.05x and above</td>
</tr>
<tr>
<td>2%</td>
<td>37 / 200 (18.5%)</td>
<td>188 / 200 (94%)</td>
<td>every shift tested</td>
</tr>
</tbody></table>
<p>The 2% threshold shows how easily the detector can flag normal variation. Durability is roughly uniform from 10 to 100, so a 1,000-row sample has a standard deviation of about 26. That means the sample mean typically lands within about 0.8 of the true mean, which is roughly 1.5% of its value. A 2% threshold is only a little bigger than that, so it should fire on roughly 18% of clean files. It fired on 37 of 200 (18.5%). This shows a tradeoff that lowering the threshold helps catch small changes but also increases false alarms.</p>
<p>The results also show why one threshold doesn't work for every numeric field. <code>stats.attack</code>, <code>level</code>, and <code>player_id</code> vary more than durability relative to their averages. At a 2% threshold, at least one of these fields triggered an alarm in 188 of 200 clean files. At 5%, they caused all 23 false alarms, while durability caused none. <code>stats.attack</code> caused 15 of those alarms. Even <code>player_id</code> triggered alarms, although its average has no useful meaning.</p>
<p>Using a single threshold treats every numeric field the same, even though some vary more than others and some shouldn't be checked by their average at all.</p>
<p>The category-share check uses a fixed 5-percentage-point threshold, so changing the mean threshold doesn't affect its results. It raised 26 alarms across 22 of the 200 clean files (11%). Seventeen alarms involved <code>equipped_slot</code>, and 16 of those involved <code>chest</code> or <code>off_hand</code>.</p>
<p>There are two reasons for these alarms. First, the check calculates each slot's share using only the 795 non-null rows, not all 1,000. This gives each share a standard error of about 1.4 percentage points.</p>
<p>Second, my baseline doesn't match the intended shares exactly. <code>chest</code> makes up 17.2% of the baseline instead of the intended 20%. As a result, a clean file only needs to reach 22.2% to trigger an alarm. This happens in roughly 6% of files. <code>off_hand</code> has the opposite problem: its baseline share is higher than intended, at 22.3%. This shows that a baseline can contain its own sampling errors, which then affect every comparison.</p>
<p>One more limitation comes from the detector design, not from a measured result. I didn't test this case, but the code shows why it would go undetected. The strict model caught the timezone drift because the timestamp lost its offset. But a producer could shift the clock and keep the <code>Z</code> suffix. Every detector would then accept the data. The strict model checks only whether a timestamp has an offset, not whether the time is correct. The statistical profile skips timestamp strings because they have too many distinct values to treat as categories and aren't numeric.</p>
<p>This is a limitation of the fourth layer: the timestamp still looks valid, but it represents the wrong time. None of the detectors checks whether the timestamp matches the actual time it is supposed to represent.</p>
<h2 id="heading-what-this-experiment-suggests">What this Experiment Suggests</h2>
<p>The results suggest that combining all four detectors provides better coverage because each one catches changes the others miss.</p>
<p>The schema diff is simple and cheap to run on every load. It caught six of the nine drifts, including the <code>level</code> change that both Pydantic models missed because they converted the value before checking it. It also identified the field that caused an error when the pipeline's own error message did not. But it caught only one of the four drifts that corrupted data.</p>
<p>The strict model and statistical profile caught all four harmful drifts between them. I'd use the strict model for fields with known rules, such as timezone-aware timestamps, allowed values, and numeric ranges. I'd use statistical checks for important metrics that have no clear validation rules, setting a separate threshold for each field based on past data.</p>
<p>The lab showed why one threshold doesn't work for every field. A global 5% mean threshold raised 23 false alarms across 200 clean files, all from fields other than durability. I'd only apply statistical checks to fields where the average is meaningful and set thresholds based on how much each field normally varies.</p>
<p>The fourth layer requires the producer and consumers to agree on what the data means. If the producer changes a field's unit, time zone, or allowed values without telling anyone, consumers can't interpret it correctly. Both sides should agree on clear rules, such as the unit of measurement, the time zone, and the allowed values.</p>
<p>This is a recommendation based on what my detectors couldn't catch. I haven't tested this approach in the lab yet.</p>
<h2 id="heading-limitations">Limitations</h2>
<p>The data is synthetic and independent, so it's much simpler than real data, which often has seasonal patterns and natural growth. Real-world false-alarm rates may therefore be higher than the ones I measured. I also used one fixed baseline, so any sampling error in that sample affects every comparison. A production system would likely use a rolling baseline instead.</p>
<p>I used the same thresholds across fields. Setting a separate threshold based on how much each field normally varies might work better, but I didn't test this. I also tested only four detectors from a much larger set of available tools. These results show where the detectors have blind spots, not which tools are best.</p>
<p>Finally, I chose to group changes into "shape" and "meaning," but some changes, such as replacing <code>null</code> with an empty string, could fit into either group.</p>
<p>My first threshold test also contained a bug. I used the wrong variable name (<code>SEED</code> instead of <code>seed</code>) in the data generator, which caused all 20 clean files to match the baseline exactly. This produced zero false alarms. A result that good at a 2% threshold seemed unlikely, so I checked the code and found the mistake. I corrected the bug and excluded seed 42, the baseline's own seed, because it would generate identical data and could never trigger an alarm. All results in this article use the corrected run.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>The experiment showed that schema changes are not the only source of pipeline failures. The schema diff caught all five structural changes but missed semantic changes that left field names and types untouched. For example, timestamps shifted to the wrong day, null values became empty strings, and durability values increased tenfold. In each case, the pipeline ran successfully but produced incorrect results.</p>
<p>The four detectors caught different types of changes, and using them together improved detection coverage. However, detecting changes is only part of the problem. Pipeline checks also need to validate whether values fall within expected ranges and whether they still represent what downstream calculations assume.</p>
<p>A pipeline completing successfully doesn't necessarily mean the data is correct. It’s just as important to check that the data retains its intended meaning and that downstream calculations still produce reliable results.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Build Production-Grade Agents with Pydantic AI  ]]>
                </title>
                <description>
                    <![CDATA[ Building AI agents with raw LLM SDKs works fine for prototypes until you need structured outputs, testable code, and production reliability. The gap shows up in a predictable way. Your notebook code w ]]>
                </description>
                <link>https://www.freecodecamp.org/news/building-agents-with-pydantic-ai/</link>
                <guid isPermaLink="false">6a7e49f74e7ef3bb34dc60ba</guid>
                
                    <category>
                        <![CDATA[ pydantic ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AI ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Jay Mehta ]]>
                </dc:creator>
                <pubDate>Thu, 13 Aug 2026 21:30:00 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/39375594-36fd-4ef4-9b5f-c2fcd3fd5197.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Building AI agents with raw LLM SDKs works fine for prototypes until you need structured outputs, testable code, and production reliability.</p>
<p>The gap shows up in a predictable way. Your notebook code works, so you move it toward production and start patching: a try/except around json.loads, a helper to strip markdown fences, a few if statements to check field types, a retry loop, a dispatch function mapping tool names to callables. None of these are hard on their own. Together, they become the majority of your codebase, and the actual agent logic disappears under the glue.</p>
<p>This article walks through six of those problems in the order you'd hit them, and shows how Pydantic AI solves each one, with code:</p>
<ol>
<li><p>Unstructured outputs require brittle parsing — your output schema lives in an English prompt string, disconnected from the dict your code expects.</p>
</li>
<li><p>Tool definitions are boilerplate-heavy — ~70 lines of hand-written JSON schema and dispatch code for three tools, with nothing keeping the schema in sync with your function signatures.</p>
</li>
<li><p>No clean way to pass runtime context — once the framework calls your tools, you can't hand them a database connection or a user ID without reaching for globals or closures.</p>
</li>
<li><p>Testing requires real LLM calls — every test costs money, takes seconds, needs network access, and flakes.</p>
</li>
<li><p>Retry and validation logic is hand-rolled — you rewrite the same validate/re-prompt/retry pattern in every agent you build.</p>
</li>
<li><p>Switching models means rewriting integration code — each provider has a different SDK shape, tool format, and response structure.</p>
</li>
</ol>
<p>We'll use one running example throughout: a receipt analysis agent. It takes raw receipt text (what you'd get from a photo-to-text scan), calls tools to look up merchant categories and exchange rates, and returns a typed summary — merchant, spending category, itemized breakdown, and a confidence score — that a budgeting dashboard or expense tool can consume directly. Structured input, tool calls for lookups, typed output for downstream systems. It's a common enough shape that the problems it surfaces will look familiar.</p>
<p>By the end you'll have a working agent with typed outputs, dependency-injected tools, business-rule validation with automatic retry, and a test suite that runs in milliseconds without an API key.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-a-quick-word-on-pydantic-ai">A Quick Word on Pydantic AI</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-the-problem-building-agents-without-a-framework">The Problem: Building Agents Without a Framework</a></p>
</li>
<li><p><a href="#heading-problem-1-unstructured-outputs-require-brittle-parsing">Problem 1: Unstructured Outputs Require Brittle Parsing</a></p>
</li>
<li><p><a href="#heading-problem-2-tool-definitions-are-boilerplate-heavy">Problem 2: Tool Definitions are Boilerplate-Heavy</a></p>
</li>
<li><p><a href="#heading-problem-3-no-clean-way-to-pass-runtime-context">Problem 3: No Clean Way to Pass Runtime Context</a></p>
</li>
<li><p><a href="#heading-problem-4-testing-requires-real-llm-calls">Problem 4: Testing Requires Real LLM Calls</a></p>
</li>
<li><p><a href="#heading-problem-5-retry-and-validation-logic-is-hand-rolled">Problem 5: Retry and Validation Logic is Hand-rolled</a></p>
</li>
<li><p><a href="#heading-problem-6-switching-models-means-rewriting-integration-code">Problem 6: Switching Models Means Rewriting Integration Code</a></p>
</li>
<li><p><a href="#heading-wrapping-up">Wrapping Up</a></p>
</li>
</ul>
<h2 id="heading-a-quick-word-on-pydantic-ai">A Quick Word on Pydantic AI</h2>
<p>Pydantic is Python based data validation library. You define a schema as a normal Python class with type hints, and Pydantic enforces it at runtime — coercing types where sensible, rejecting what doesn't fit, and raising precise errors that name the offending field</p>
<pre><code class="language-plaintext">  from pydantic import BaseModel, Field

  class Item(BaseModel):
      name: str
      amount: float = Field(gt=0)

  Item(name="Espresso", amount="2.50")  # → amount=2.5, coerced
  Item(name="Espresso", amount=-1)      # → ValidationError: amount must be &gt; 0
</code></pre>
<p>Pydantic AI is an agent framework that handles the LLM boundary — turning your models into provider-native schema requests, parsing and validating what comes back, generating tool definitions from function signatures, injecting dependencies, and retrying on validation failure. Your agent logic stays Python; the framework handles the translation in both directions.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>This article assumes you're comfortable with:</p>
<ul>
<li><p><strong>Python 3.10+</strong> — type hints, dataclasses, async/await</p>
</li>
<li><p><strong>LLM API basics</strong> — you've made at least a few calls to OpenAI, Anthropic, or similar SDKs</p>
</li>
<li><p><strong>Agent concepts</strong> — you understand what an AI agent is (LLM + tools + reasoning loop). If not, start with <a href="https://jay-g-mehta.github.io/ai-agents">AI Agents — A Builder's Guide</a></p>
</li>
</ul>
<p>You don't need prior experience with Pydantic AI. We'll build up from scratch.</p>
<h2 id="heading-the-problem-building-agents-without-a-framework">The Problem: Building Agents Without a Framework</h2>
<p>To expose and explain these problems better, we'll use a running example: a receipt analysis agent. It takes raw receipt text (what you'd get from a photo-to-text scan), categorizes spending, looks up merchant info, and returns a structured summary. This gives you the merchant name, spending category, itemized breakdown, and a confidence score. Downstream systems (a budgeting dashboard, an expense report tool) consume this structured output directly.</p>
<p>This is a common real-world pattern: structured input, tool calls for lookups, and typed output for downstream systems. Let's see what building it looks like with raw OpenAI SDK calls.</p>
<h2 id="heading-problem-1-unstructured-outputs-require-brittle-parsing">Problem 1: Unstructured Outputs Require Brittle Parsing</h2>
<p>At its core, every interaction with an LLM is just text in, text out. Your entire contract with the model (the input data, the goal, and the desired output format) is crammed into a single text prompt. There's no schema, type system, or compiler enforcing correctness. You describe what you want in English and hope the model complies.</p>
<p>Here's the straightforward implementation using the OpenAI SDK. Notice how the system prompt has to encode the input context, the task instruction, <em>and</em> the output schema all in one blob of text:</p>
<pre><code class="language-python">import json
from openai import OpenAI

client = OpenAI()

def analyze_receipt(receipt_text: str) -&gt; dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": """Analyze this receipt and return JSON:
{
    "merchant": "string",
    "category": "one of: food, transport, utilities, entertainment, shopping, other",
    "total": float,
    "currency": "string",
    "items": [{"name": "string", "amount": float}],
    "is_business_expense": bool,
    "confidence": float between 0 and 1
}"""},
            {"role": "user", "content": receipt_text}
        ]
    )

    raw = response.choices[0].message.content

    # Parse the response
    try:
        if raw.startswith("```"):
            raw = raw.split("\n", 1)[1].rsplit("```", 1)[0]
        result = json.loads(raw)
    except json.JSONDecodeError:
        raise ValueError(f"LLM returned invalid JSON: {raw[:200]}")

    # Validate fields manually
    allowed_categories = {"food", "transport", "utilities", "entertainment", "shopping", "other"}
    if result.get("category") not in allowed_categories:
        result["category"] = "other"

    return result
</code></pre>
<p>This code is clean and readable, and it works in your notebook. But the fundamental issue is that your entire contract with the LLM (the input, goal, and output format) lives in an unstructured string. There's nothing enforcing that contract on either side.</p>
<p>If you push it toward production, the problems start to surface:</p>
<ul>
<li><p><strong>The prompt <em>is</em> the schema, and it's just English:</strong> The system prompt describes the output format in natural language. There's nothing connecting that description to the <code>dict</code> your code actually expects. Add a field to the prompt and forget to handle it downstream, and you don't get an error until production.</p>
</li>
<li><p><strong>The LLM doesn't always return clean JSON:</strong> It wraps output in <code>```json ```</code> fences, adds explanatory text before/after, includes trailing commas, or returns a partial response on timeout. Your parsing code handles one case (fences) but not the others.</p>
</li>
<li><p><strong>There's no real validation:</strong> Is <code>total</code> actually a number, or did the LLM return the string <code>"$45.99"</code>? Is <code>confidence</code> between 0 and 1, or did it return <code>95</code> (percent)? Does the <code>items</code> list contain dicts with the right keys? You'd have to check all of this manually.</p>
</li>
<li><p><strong>Failures are silent or catastrophic:</strong> The category fallback (<code>result["category"] = "other"</code>) hides a problem that should trigger a retry. The <code>json.loads</code> failure raises an exception with no path to recovery.</p>
</li>
</ul>
<p>What we actually need is a way to define the output schema once, in code, as a typed data structure, not an English description in a prompt string. The schema should be the single source of truth for both the LLM and the consuming code.</p>
<p>We also need the framework to enforce the schema automatically and validate the LLM's response against the type definitions, with proper errors on mismatch.</p>
<p>And finally, we need retry on validation failure without manual logic. If the output doesn't match the schema, re-prompt the LLM with the validation error so it can self-correct.</p>
<p>In short: the output format should be a <em>contract</em> expressed in the type system, not a <em>suggestion</em> expressed in English.</p>
<h3 id="heading-how-pydantic-ai-solves-this">How Pydantic AI Solves This</h3>
<p>Pydantic AI lets you define the output as a Pydantic model. The framework handles schema generation, prompt injection, JSON parsing, validation, and retry. And this is all derived from that single model definition:</p>
<pre><code class="language-python">from pydantic import BaseModel, Field
from pydantic_ai import Agent
from enum import Enum


class SpendingCategory(str, Enum):
    FOOD = "food"
    TRANSPORT = "transport"
    UTILITIES = "utilities"
    ENTERTAINMENT = "entertainment"
    SHOPPING = "shopping"
    OTHER = "other"


class LineItem(BaseModel):
    name: str
    amount: float


class ReceiptAnalysis(BaseModel):
    merchant: str
    category: SpendingCategory
    total: float = Field(gt=0)
    currency: str = Field(min_length=3, max_length=3)
    items: list[LineItem]
    is_business_expense: bool
    confidence: float = Field(ge=0, le=1)


receipt_agent = Agent(
    "openai:gpt-4o",
    output_type=ReceiptAnalysis,
    system_prompt="Analyze the provided receipt and extract structured details.",
)

result = receipt_agent.run_sync("CAFE PARIS\n€12.50\nCroissant x2 €5.00\nEspresso €2.50\nCroque Monsieur €5.00")
print(result.output)
# merchant='CAFE PARIS' category=&lt;SpendingCategory.FOOD: 'food'&gt; total=12.5 ...
</code></pre>
<p>So what's different here?</p>
<p>First, the schema is the model. <code>ReceiptAnalysis</code> defines the fields, types, and constraints. Pydantic AI converts this into the appropriate JSON schema for the LLM and validates the response against it. One definition, used everywhere.</p>
<p>Second, there's no parsing of code. You don't strip markdown fences, call <code>json.loads</code>, or catch <code>JSONDecodeError</code>. The framework handles all of that.</p>
<p>Also, validation is real. <code>Field(ge=0, le=1)</code> on <code>confidence</code> means a value of <code>95</code> is rejected, not silently accepted. <code>SpendingCategory</code> as an Enum means only valid categories are allowed with no fallback masking.</p>
<p>Finally, retry is automatic. If the LLM returns output that fails validation, Pydantic AI sends the validation error back to the model and asks it to correct itself. There's no hand-rolled retry loop.</p>
<p>The function returns a <code>ReceiptAnalysis</code> object: typed, validated, and IDE-autocomplete-friendly. Not a <code>dict</code> you hope has the right keys.</p>
<h4 id="heading-what-happens-beneath-the-surface">What happens beneath the surface</h4>
<p>When you define <code>output_type=ReceiptAnalysis</code>, Pydantic AI does a few key things on each agent run.</p>
<p>On the way in, it generates a JSON schema from your Pydantic model and injects it into the LLM request. Depending on the model provider, this uses the native structured output / tool-call mechanism (OpenAI's <code>response_format</code>, Anthropic's tool-use, and so on) so the LLM knows <em>exactly</em> what structure to produce.</p>
<p>On the way back, it takes the LLM's raw response, parses it against the Pydantic model, and runs full validation (type coercion, field constraints, and enum membership). If validation fails, it feeds the error message back into the conversation and asks the LLM to correct its output (automatically, up to a configurable retry limit).</p>
<pre><code class="language-markdown">┌──────────────────────────────────────────────────────────────────────┐
│                    Pydantic AI — Structured Output Flow              │
│                                                                      │
│  ┌────────────────┐         ┌──────────────────────────────────┐     │
│  │  Your Code     │         │  Pydantic AI Framework           │     │
│  │                │         │                                  │     │
│  │  output_type = │────────&gt;│  1. Generate JSON schema from    │     │
│  │  ReceiptAnalysis         │     ReceiptAnalysis model        │     │
│  │                │         │                                  │     │
│  └────────────────┘         │  2. Inject schema into LLM      │     │
│                             │     request (provider-native     │     │
│                             │     format: response_format,     │     │
│                             │     tool_call, etc.)             │     │
│                             │              │                   │     │
│                             └──────────────┼───────────────────┘     │
│                                            ▼                         │
│                             ┌──────────────────────────────────┐     │
│                             │           LLM                    │     │
│                             │  Sees schema → produces JSON     │     │
│                             └──────────────┬───────────────────┘     │
│                                            │                         │
│                                            ▼                         │
│                             ┌──────────────────────────────────┐     │
│                             │  Pydantic AI Framework           │     │
│                             │                                  │     │
│                             │  3. Parse raw LLM response       │     │
│                             │  4. Validate against model:      │     │
│                             │     - Type checks                │     │
│                             │     - Field constraints (ge, le) │     │
│                             │     - Enum membership            │     │
│                             │              │                   │     │
│                             │         ┌────┴────┐              │     │
│                             │         │         │              │     │
│                             │      PASS ✓    FAIL ✗            │     │
│                             │         │         │              │     │
│                             │         ▼         ▼              │     │
│                             │  Return typed  Send validation   │     │
│                             │  object        error back to LLM │     │
│                             │                for self-correct   │     │
│                             │                (auto-retry)       │     │
│                             └──────────────────────────────────┘     │
│                                            │                         │
│                                            ▼                         │
│                             ┌──────────────────────────────────┐     │
│                             │  Your Code receives:             │     │
│                             │  result.output → ReceiptAnalysis │     │
│                             │  (typed, validated, ready to use)│     │
│                             └──────────────────────────────────┘     │
└──────────────────────────────────────────────────────────────────────┘
</code></pre>
<p>You define the contract once as a Python class. The framework handles both sides of the LLM boundary, telling the model what to produce and verifying that it did.</p>
<h2 id="heading-problem-2-tool-definitions-are-boilerplate-heavy">Problem 2: Tool Definitions are Boilerplate-Heavy</h2>
<p>Your receipt agent needs tools that let it look up merchant categories, check exchange rates, and query spending history.</p>
<p>Here's what that would look like with raw function calling:</p>
<pre><code class="language-python">tools = [
    {
        "type": "function",
        "function": {
            "name": "lookup_merchant_category",
            "description": "Look up the spending category for a merchant name",
            "parameters": {
                "type": "object",
                "properties": {
                    "merchant_name": {
                        "type": "string",
                        "description": "The merchant name from the receipt"
                    }
                },
                "required": ["merchant_name"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_exchange_rate",
            "description": "Get current exchange rate between two currencies",
            "parameters": {
                "type": "object",
                "properties": {
                    "from_currency": {
                        "type": "string",
                        "description": "Source currency code (e.g., EUR)"
                    },
                    "to_currency": {
                        "type": "string",
                        "description": "Target currency code (e.g., USD)"
                    }
                },
                "required": ["from_currency", "to_currency"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_spending_history",
            "description": "Get spending totals by category for a date range",
            "parameters": {
                "type": "object",
                "properties": {
                    "category": {
                        "type": "string",
                        "description": "Spending category"
                    },
                    "days": {
                        "type": "integer",
                        "description": "Number of past days to query"
                    }
                },
                "required": ["category", "days"]
            }
        }
    }
]


# Then you ALSO need to write the dispatch logic:
def handle_tool_call(tool_call):
    name = tool_call.function.name
    args = json.loads(tool_call.function.arguments)

    if name == "lookup_merchant_category":
        return lookup_merchant_category(args["merchant_name"])
    elif name == "get_exchange_rate":
        return get_exchange_rate(args["from_currency"], args["to_currency"])
    elif name == "get_spending_history":
        return get_spending_history(args["category"], args["days"])
    else:
        raise ValueError(f"Unknown tool: {name}")
</code></pre>
<p>For three tools, you've written ~70 lines of JSON schema plus dispatch code. The schema is disconnected from the actual function signatures: change a parameter name in the function and forget to update the schema, and it silently breaks at runtime.</p>
<h3 id="heading-what-the-solution-should-look-like">What the Solution Should Look Like</h3>
<p>Instead, the tool definition should be derived from the function itself. The function's name, docstring, and type hints already describe what the tool does and what arguments it takes. That should be enough.</p>
<p>It should also be automatically kept in sync. If you rename a parameter or change its type, the schema sent to the LLM should update without you touching a second file.</p>
<p>And it should be dispatch-free. The framework should call the right function directly. No manual if/elif chain mapping string names to callables.</p>
<h3 id="heading-how-pydantic-ai-solves-this">How Pydantic AI Solves This</h3>
<p>In Pydantic AI, a tool is just a function with a decorator. The framework generates the JSON schema from the function's signature and docstring, and handles dispatch automatically:</p>
<pre><code class="language-python">from pydantic_ai import Agent, RunContext

receipt_agent = Agent(
    "openai:gpt-4o",
    output_type=ReceiptAnalysis,
    system_prompt="Analyze the provided receipt and extract structured details.",
)


@receipt_agent.tool_plain
def lookup_merchant_category(merchant_name: str) -&gt; str:
    """Look up the spending category for a merchant name."""
    # Your actual implementation
    categories_db = {"CAFE PARIS": "food", "UBER": "transport", "NETFLIX": "entertainment"}
    return categories_db.get(merchant_name.upper(), "other")


@receipt_agent.tool_plain
def get_exchange_rate(from_currency: str, to_currency: str) -&gt; float:
    """Get current exchange rate between two currencies."""
    # Your actual implementation — call an API, hit a cache, etc.
    rates = {"EUR_USD": 1.08, "GBP_USD": 1.27}
    return rates.get(f"{from_currency}_{to_currency}", 1.0)


@receipt_agent.tool_plain
def get_spending_history(category: str, days: int) -&gt; dict:
    """Get spending totals by category for a date range."""
    # Your actual implementation
    return {"category": category, "total": 142.50, "transaction_count": 12}
</code></pre>
<p>That's it. No JSON schema dictionaries. No dispatch function. The same three tools in ~25 lines instead of ~70.</p>
<p>What the framework does for you:</p>
<ul>
<li><p><strong>Schema generation from type hints:</strong> <code>merchant_name: str</code> becomes <code>{"type": "string"}</code> in the JSON schema. The docstring becomes the tool's <code>description</code>. Parameter names become property names. It's all derived from what you already wrote.</p>
</li>
<li><p><strong>Automatic dispatch:</strong> When the LLM calls <code>get_exchange_rate</code>, the framework routes to the decorated function directly. No string matching and no manual mapping.</p>
</li>
<li><p><strong>Sync guaranteed:</strong> Rename <code>from_currency</code> to <code>source_currency</code> in the function signature and the schema updates automatically on the next run. There's no second place to forget.</p>
</li>
</ul>
<h2 id="heading-problem-3-no-clean-way-to-pass-runtime-context">Problem 3: No Clean Way to Pass Runtime Context</h2>
<p>With automatic dispatch (Problem 2), the framework calls your tool functions, not you. You no longer control the call site, so you can't just pass <code>db</code> or <code>user_id</code> as extra arguments.</p>
<p>And those aren't things the LLM should provide either. You need a side-channel to deliver runtime dependencies into tools that the framework invokes on your behalf.</p>
<p>Without that mechanism, you end up with something like this:</p>
<pre><code class="language-python"># Option A: Global state (untestable, unsafe)
db = get_database_connection()
current_user = None  # Set somewhere else... hopefully before tools run

def get_spending_history(category: str, days: int) -&gt; dict:
    # Uses global `db` and `current_user` — how do you test this?
    # How do you run two users concurrently?
    return db.query(
        "SELECT sum(amount) FROM transactions WHERE user_id = ? AND category = ? AND date &gt; ?",
        current_user.id, category, days_ago(days)
    )


# Option B: Closure-based (awkward, deeply nested)
def make_tools(db, user):
    def get_spending_history(category: str, days: int) -&gt; dict:
        return db.query(...)  # Captures db and user from enclosing scope

    def lookup_merchant_category(merchant_name: str) -&gt; str:
        return db.query(...)  # Same closure trick

    return [get_spending_history, lookup_merchant_category]

# Every time you add a dependency, you restructure the closure nesting
</code></pre>
<p>Both approaches make testing painful. You can't easily swap in a mock database or a test user without restructuring the code.</p>
<h3 id="heading-what-the-solution-should-look-like">What the Solution Should Look Like</h3>
<p>For a better solution, you should declare what your tools need. Express dependencies (DB, HTTP client, user session) as typed requirements that are separate from tool arguments the LLM provides.</p>
<p>You should also inject at runtime, not definition time. Pass the concrete instances when you run the agent, not when you define the tools. This keeps tool definitions pure and reusable.</p>
<p>And swap dependencies for testing. Substitute a real database with an in-memory mock, or a real user with a test fixture, without changing tool code.</p>
<h3 id="heading-how-pydantic-ai-solves-this">How Pydantic AI Solves This</h3>
<p>Pydantic AI has a first-class dependency injection system. You define a <code>deps_type</code> on the agent, and tools receive those dependencies via a typed <code>RunContext</code>, with no globals or closures:</p>
<pre><code class="language-python">from dataclasses import dataclass
from pydantic_ai import Agent, RunContext


@dataclass
class ReceiptDeps:
    db: DatabaseClient
    user_id: str
    http_client: HttpClient


receipt_agent = Agent(
    "openai:gpt-4o",
    output_type=ReceiptAnalysis,
    deps_type=ReceiptDeps,
    system_prompt="Analyze the provided receipt and extract structured details.",
)


@receipt_agent.tool
def get_spending_history(ctx: RunContext[ReceiptDeps], category: str, days: int) -&gt; dict:
    """Get spending totals by category for a date range."""
    return ctx.deps.db.query(
        "SELECT sum(amount), count(*) FROM transactions WHERE user_id = ? AND category = ? AND date &gt; ?",
        ctx.deps.user_id, category, days_ago(days)
    )


@receipt_agent.tool
def get_exchange_rate(ctx: RunContext[ReceiptDeps], from_currency: str, to_currency: str) -&gt; float:
    """Get current exchange rate between two currencies."""
    response = ctx.deps.http_client.get(f"/rates/{from_currency}/{to_currency}")
    return response.json()["rate"]


# At runtime — pass real dependencies
result = receipt_agent.run_sync(
    "CAFE PARIS\n€12.50\nCroissant x2",
    deps=ReceiptDeps(
        db=get_database_connection(),
        user_id="user_123",
        http_client=HttpClient(base_url="https://api.exchangerate.host"),
    ),
)

# In tests — swap with mocks, no code changes to tools
result = receipt_agent.run_sync(
    "CAFE PARIS\n€12.50\nCroissant x2",
    deps=ReceiptDeps(
        db=InMemoryDb(fake_transactions),
        user_id="test_user",
        http_client=MockHttpClient(fixed_rate=1.08),
    ),
)
</code></pre>
<p>What this gives you:</p>
<ul>
<li><p><strong>Tools declare what they need, not how to get it:</strong> <code>ctx.deps.db</code> is typed. Your IDE autocompletes methods on it, and a type checker catches misuse. The tool doesn't know or care whether it's a real Postgres connection or a test mock.</p>
</li>
<li><p><strong>No globals, no closures:</strong> Dependencies flow in explicitly at <code>run_sync()</code> time. Two concurrent users get two separate <code>ReceiptDeps</code> instances with no shared mutable state.</p>
</li>
<li><p><strong>Testing is trivial:</strong> Swap <code>DatabaseClient</code> for <code>InMemoryDb</code>, and swap <code>HttpClient</code> for <code>MockHttpClient</code>. The tool code is unchanged. There's no monkeypatching, dependency injection frameworks, or test fixtures reaching into module-level state.</p>
</li>
<li><p><strong>The LLM never sees dependencies:</strong> <code>RunContext</code> isn't exposed as a tool parameter. The LLM only sees <code>category</code> and <code>days</code>. The framework strips it out of the schema automatically.</p>
</li>
</ul>
<h2 id="heading-problem-4-testing-requires-real-llm-calls">Problem 4: Testing Requires Real LLM Calls</h2>
<p>You want to verify that your agent handles edge cases: receipts in foreign currencies, missing merchant names, or ambiguous categories. But every test hits the real API:</p>
<pre><code class="language-python">def test_foreign_currency_receipt():
    # This test:
    # - Costs money (API call)
    # - Takes 2-5 seconds
    # - Is non-deterministic (might pass today, fail tomorrow)
    # - Requires network access (breaks in CI without secrets)
    result = analyze_receipt("CAFÉ PARIS\n€12.50\nCroissant x2")
    assert result["currency"] == "EUR"
    assert result["category"] == "food"  # Might return "dining" instead — flaky!
</code></pre>
<p>You can't run this in CI reliably. You can't run 50 edge case tests without burning through your API budget. You end up with either no tests or integration tests that flake.</p>
<h3 id="heading-what-the-solution-should-look-like">What the Solution Should Look Like</h3>
<p>First, swap the LLM for a deterministic stand-in — something that returns predictable, controlled responses so tests are fast, free, and repeatable.</p>
<p>Next, keep the agent logic intact. The test should exercise the real tool dispatch, validation, and output parsing. Only the model is faked.</p>
<p>Finally, assert on behavior, not LLM wording. Verify that the right tools were called with the right arguments, and that the output matches the expected structure.</p>
<h3 id="heading-how-pydantic-ai-solves-this">How Pydantic AI Solves This</h3>
<p>Pydantic AI provides <code>TestModel</code> and <code>FunctionModel</code>. These are drop-in model replacements that let you control exactly what the "LLM" returns, without network calls:</p>
<pre><code class="language-python">from pydantic_ai import Agent
from pydantic_ai.models.test import TestModel
from pydantic_ai.models.function import FunctionModel


# TestModel — returns a predictable, schema-valid response automatically
def test_receipt_analysis_structure():
    """Test that the agent returns a valid ReceiptAnalysis object."""
    with receipt_agent.override(model=TestModel()):
        result = receipt_agent.run_sync(
            "CAFE PARIS\n€12.50\nCroissant x2",
            deps=ReceiptDeps(
                db=InMemoryDb(fake_transactions),
                user_id="test_user",
                http_client=MockHttpClient(fixed_rate=1.08),
            ),
        )
        # TestModel fills fields with valid dummy data matching the schema
        assert isinstance(result.output, ReceiptAnalysis)
        assert 0 &lt;= result.output.confidence &lt;= 1


# FunctionModel — you control the exact response for specific scenarios
def test_foreign_currency_triggers_exchange_rate_tool():
    """Test that a EUR receipt causes the agent to call get_exchange_rate."""

    def mock_model(messages, info):
        # Simulate the LLM deciding to call the exchange rate tool
        return ModelResponse(
            tool_calls=[ToolCall(name="get_exchange_rate", args={"from_currency": "EUR", "to_currency": "USD"})]
        )

    with receipt_agent.override(model=FunctionModel(mock_model)):
        result = receipt_agent.run_sync(
            "CAFE PARIS\n€12.50\nCroissant x2",
            deps=ReceiptDeps(
                db=InMemoryDb(fake_transactions),
                user_id="test_user",
                http_client=MockHttpClient(fixed_rate=1.08),
            ),
        )
        # Assert the exchange rate tool was actually invoked
        tool_calls = [msg for msg in result.all_messages() if hasattr(msg, "tool_name")]
        assert any(tc.tool_name == "get_exchange_rate" for tc in tool_calls)
</code></pre>
<p>This is helpful, because it's fast and free: there's no API calls, network, or tokens burned. Tests run in milliseconds.</p>
<p>It's also deterministic, meaning for the same input, you get the same output, every time. There are no flaky tests due to LLM temperature or wording changes.</p>
<p>The real agent logic also still runs. Tool dispatch, dependency injection, and output validation are all exercised. Only the model is swapped.</p>
<p>It's also CI-friendly. You don't need any API keys in your CI environment, and there are no secrets to manage or rate limits to hit.</p>
<p>You also get two levels of control. You have <code>TestModel</code> for "does the plumbing work?" tests. And you have <code>FunctionModel</code> for "does the agent make the right decisions?" tests where you script specific LLM behaviors.</p>
<h2 id="heading-problem-5-retry-and-validation-logic-is-hand-rolled">Problem 5: Retry and Validation Logic is Hand-rolled</h2>
<p>When the LLM returns a bad response, you need to retry. But the retry logic gets complex fast:</p>
<pre><code class="language-python">def analyze_receipt_with_retry(receipt_text: str, max_retries: int = 3) -&gt; dict:
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(...)
            raw = response.choices[0].message.content
            result = json.loads(strip_markdown(raw))

            # Validate
            if not isinstance(result.get("total"), (int, float)):
                raise ValueError("total must be numeric")
            if result.get("confidence", 0) &gt; 1 or result.get("confidence", 0) &lt; 0:
                raise ValueError("confidence must be 0-1")
            if result.get("category") not in ALLOWED_CATEGORIES:
                raise ValueError(f"invalid category: {result.get('category')}")

            return result

        except (json.JSONDecodeError, ValueError, KeyError) as e:
            if attempt == max_retries - 1:
                raise
            # Should we feed the error back to the LLM? Modify the prompt?
            # How do we track which attempts failed and why?
            continue

    raise RuntimeError("Should not reach here")
</code></pre>
<p>Every agent you build needs this same retry/validate/re-prompt pattern, and you rewrite it each time. The logic for feeding validation errors back to the LLM (so it can self-correct) adds another layer of complexity.</p>
<h3 id="heading-what-the-solution-should-look-like">What the Solution Should Look Like</h3>
<p>First, validation should be declarative: that is, defined by the output schema, not by hand-written if-statements scattered through your code.</p>
<p>Second, retry should be automatic. If the output fails validation, the framework should re-prompt the LLM with the error message so it can self-correct.</p>
<p>And custom validation should plug in cleanly. For business rules beyond type checks (for example, "if category is 'other', confidence must be below 0.8"), you should be able to add validators without rewriting the retry loop.</p>
<h3 id="heading-how-pydantic-ai-solves-this">How Pydantic AI Solves This</h3>
<p>Schema-level validation is already handled by the Pydantic model (as shown in Problem 1). But for business logic validation, Pydantic AI provides <code>result_validator</code>. This is a decorator that runs after parsing and can trigger an automatic retry:</p>
<pre><code class="language-python">from pydantic_ai import Agent, RunContext, ModelRetry


receipt_agent = Agent(
    "openai:gpt-4o",
    output_type=ReceiptAnalysis,
    deps_type=ReceiptDeps,
    system_prompt="Analyze the provided receipt and extract structured details.",
    retries=3,  # Max retry attempts on validation failure
)


@receipt_agent.result_validator
def validate_receipt_analysis(ctx: RunContext[ReceiptDeps], result: ReceiptAnalysis) -&gt; ReceiptAnalysis:
    """Business logic validation — runs after schema validation passes."""

    # Rule: if total doesn't match sum of items, ask LLM to fix it
    items_sum = sum(item.amount for item in result.items)
    if abs(result.total - items_sum) &gt; 0.01:
        raise ModelRetry(
            f"Total ({result.total}) doesn't match sum of items ({items_sum}). "
            f"Please recheck the receipt and correct either the total or the item amounts."
        )

    # Rule: low confidence + "other" category likely means the LLM gave up — retry
    if result.category == SpendingCategory.OTHER and result.confidence &lt; 0.5:
        raise ModelRetry(
            "Category is 'other' with low confidence. Look more carefully at the "
            "merchant name and items to determine a more specific category."
        )

    return result
</code></pre>
<p>Here's what happens when validation fails:</p>
<pre><code class="language-plaintext">┌─────────────────────────────────────────────────────────────┐
│  Automatic Retry Flow                                       │
│                                                             │
│  LLM response                                               │
│       │                                                     │
│       ▼                                                     │
│  Schema validation (Pydantic model)                         │
│       │                                                     │
│       ├── FAIL → error message sent back to LLM → retry     │
│       │                                                     │
│       ▼                                                     │
│  result_validator (your business rules)                     │
│       │                                                     │
│       ├── ModelRetry raised → message sent to LLM → retry   │
│       │                                                     │
│       ▼                                                     │
│  PASS → return typed result                                 │
└─────────────────────────────────────────────────────────────┘
</code></pre>
<ul>
<li><p>Schema violations (wrong type, missing field, enum mismatch) is caught by Pydantic automatically. The validation error is sent back to the LLM as context so it knows <em>what</em> to fix.</p>
</li>
<li><p>Business rule violations are caught by your <code>result_validator</code>. <code>ModelRetry</code> sends your custom message to the LLM, guiding it toward a correct response.</p>
</li>
<li><p>There's no retry loop in your code. The <code>retries=3</code> parameter controls max attempts. The framework handles the loop, the re-prompting, and the error message formatting.</p>
</li>
</ul>
<h2 id="heading-problem-6-switching-models-means-rewriting-integration-code">Problem 6: Switching Models Means Rewriting Integration Code</h2>
<p>Your agent works with OpenAI. Now you want to try Anthropic (cheaper for your use case) or run locally with Ollama (for data privacy). Each provider has a different SDK, tool-calling format, and response structure:</p>
<pre><code class="language-python"># OpenAI
response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools  # OpenAI tool format
)
tool_calls = response.choices[0].message.tool_calls

# Anthropic — completely different API shape
response = anthropic_client.messages.create(
    model="claude-sonnet-4-20250514",
    messages=messages,
    tools=anthropic_tools  # Different format than OpenAI!
)
tool_use_blocks = [b for b in response.content if b.type == "tool_use"]

# Google — yet another shape
response = genai_client.generate_content(
    contents=messages,
    tools=google_tools  # Yet another format!
)
function_calls = response.candidates[0].content.parts
</code></pre>
<p>You end up with provider-specific code paths, adapter layers, and the agent logic gets buried under integration glue.</p>
<h3 id="heading-what-the-solution-should-look-like">What the Solution Should Look Like</h3>
<p>Start by defining the agent logic once. Your tools, output types, system prompts, and validation should be model-independent.</p>
<p>You should also be able to switch models by changing a string, not by rewriting SDK calls, tool schemas, or response parsing.</p>
<p>And you should keep provider-specific details hidden. The framework should translate your universal agent definition into whatever format each provider expects.</p>
<h3 id="heading-how-pydantic-ai-solves-this">How Pydantic AI Solves This</h3>
<p>The agent definition is entirely model-agnostic. The model is just a string identifier: change it and everything else stays the same:</p>
<pre><code class="language-python"># Your agent definition — tools, output type, deps, validators — all unchanged
receipt_agent = Agent(
    "openai:gpt-4o",  # ← this is the only line that changes
    output_type=ReceiptAnalysis,
    deps_type=ReceiptDeps,
    system_prompt="Analyze the provided receipt and extract structured details.",
)

# Switch to Anthropic — same agent, same tools, same output type
receipt_agent = Agent(
    "anthropic:claude-sonnet-4-20250514",
    output_type=ReceiptAnalysis,
    deps_type=ReceiptDeps,
    system_prompt="Analyze the provided receipt and extract structured details.",
)

# Switch to a local model via Ollama
receipt_agent = Agent(
    "ollama:llama3.1",
    output_type=ReceiptAnalysis,
    deps_type=ReceiptDeps,
    system_prompt="Analyze the provided receipt and extract structured details.",
)

# Or make it configurable at runtime
import os

receipt_agent = Agent(
    os.getenv("RECEIPT_AGENT_MODEL", "openai:gpt-4o"),
    output_type=ReceiptAnalysis,
    deps_type=ReceiptDeps,
    system_prompt="Analyze the provided receipt and extract structured details.",
)
</code></pre>
<p>What the framework handles behind the scenes:</p>
<ul>
<li><p><strong>Tool schema translation:</strong> Your <code>@receipt_agent.tool</code> functions are converted to OpenAI's <code>tools</code> format, Anthropic's <code>tools</code> format, or Google's <code>function_declarations</code> — whichever the chosen provider expects. You never see the difference.</p>
</li>
<li><p><strong>Response normalization:</strong> Whether the model returns <code>choices[0].message.tool_calls</code> (OpenAI), <code>content[].type == "tool_use"</code> (Anthropic), or <code>candidates[0].content.parts</code> (Google), the framework normalizes it into a consistent internal representation.</p>
</li>
<li><p><strong>Provider-specific features handled transparently:</strong> Each provider implements structured output mode, streaming, and token counting differently. The framework adapts without exposing the differences to your code.</p>
</li>
</ul>
<p>One agent definition, any model. Swap via config or environment variable.</p>
<h2 id="heading-wrapping-up">Wrapping Up</h2>
<p>Every problem in this article comes from the same place: an LLM call is text in, text out, while the code on either side of it is typed. The raw-SDK approach bridges that gap with hand-written glue — a fence stripper, a chain of if checks, a dispatch table, a retry loop. Each piece is easy. Together they outgrow the agent logic they surround, and you own all of them.</p>
<p>Pydantic AI closes the gap by making the boundary a declared contract. Here's what that bought us, section by section:</p>
<table>
<thead>
<tr>
<th>Problem</th>
<th>Raw SDK</th>
<th>Pydantic AI</th>
</tr>
</thead>
<tbody><tr>
<td>Structured output</td>
<td>Schema described in English, parsed by hand</td>
<td><code>output_type=ReceiptAnalysis</code> — schema generated, response validated</td>
</tr>
<tr>
<td>Tool definitions</td>
<td>~70 lines of JSON schema + dispatch</td>
<td><code>@agent.tool_plain</code> on a typed function</td>
</tr>
<tr>
<td>Runtime context</td>
<td>Globals or nested closures</td>
<td><code>deps_type</code> + <code>RunContext</code>, injected per run</td>
</tr>
<tr>
<td>Testing</td>
<td>Real API calls: slow, paid, flaky</td>
<td><code>TestModel</code> / <code>FunctionModel</code>, no network</td>
</tr>
<tr>
<td>Retry &amp; validation</td>
<td>Hand-rolled loop, error discarded</td>
<td>Field constraints + <code>ModelRetry</code>, error fed back to the model</td>
</tr>
<tr>
<td>Model switching</td>
<td>Per-provider SDK and parsing code</td>
<td>One string: <code>"openai:gpt-4o"</code> → <code>"anthropic:claude-sonnet-4-20250514"</code></td>
</tr>
</tbody></table>
<p>The receipt agent we ended up with is a Pydantic model, a handful of typed functions, a deps dataclass, and one validator. No parsing, no dispatch, no retry loop.</p>
<p>The <a href="https://ai.pydantic.dev/">Pydantic AI docs</a> are short and worth reading end to end; everything here maps onto their API reference.</p>
 ]]>
                </content:encoded>
            </item>
        
    </channel>
</rss>
