You wait. On day 30, the effect is still there. But you spent 16 days running a feature you already knew worked with 95% confidence, delaying the next experiment and burning opportunity cost.

The statistician is technically right, if you're running a classical fixed-sample test. The p-value in a standard t-test is valid only when you commit to a sample size in advance and look at the results exactly once. Look earlier and stop when p < 0.05, and your false positive rate climbs toward 30%.

The p-value was designed for a single pre-committed look: it was built for a static experiment with a fixed endpoint. Applying it to a live stream where you can check at any point requires a different mathematical object entirely.

Sequential testing was designed for exactly this situation. The mixture Sequential Probability Ratio Test (mSPRT) (Johari et al.) produces always-valid inference using a mathematical object called an e-value: you can check results every day, stop when the evidence is strong enough, and your false positive rate stays at 5%.

Netflix has documented the production use of always-valid sequential testing frameworks (Lindon et al.), and the underlying ideas trace back to Wald's 1945 work on sequential analysis and Ville's 1939 inequality.

This tutorial makes the connection explicit. You'll simulate the peeking problem to see the inflated error rate directly, implement a working mSPRT from scratch in Python, apply it to the shared synthetic LLM product dataset, and understand exactly when sequential testing fails.

Companion notebook: every code block in this article runs end-to-end in msprt_demo.ipynb in the companion repo.

Table of Contents

• Why Optional Stopping Breaks Classical Tests

• What a Sequential Test Actually Does

• Identification Assumptions

• Prerequisites

• Setting Up the Working Example

  • Step 1: Simulate the peeking problem

  • Step 2: Implement the mSPRT e-value

  • Step 3: Apply mSPRT to the real dataset

  • Step 4: Compare power against a fixed-sample test

  • Validate against ground truth

  • Step 5: Bootstrap confidence intervals

• When mSPRT Fails

• What to Do Next

Why Optional Stopping Breaks Classical Tests

Peeking at running p-values inflates your false positive rate toward 30%. That's the number that should give you pause, and you'll reproduce it in Step 1 below.

The p-value in a classical hypothesis test answers a specific question: given the null is true, what's the probability of seeing data this extreme when you run the experiment exactly as planned with the sample size you committed to upfront?

The "exactly as planned" clause is the problem. When you check results on day 5, day 10, day 14, and stop on day 14 because p < 0.05, you haven't run the experiment you planned. You've run 14 different experiments, looked at the results of each, and stopped at the one that passed your threshold. The p-value formula doesn't know that.

Here's the intuition. Under the null hypothesis (no effect), your p-value bounces around randomly between 0 and 1. It doesn't stay parked at 0.5. Over a 30-day run, a null experiment will dip below 0.05 at some point with high probability. If you're watching every day and ready to stop the moment you see p < 0.05, you'll almost always catch one of those dips. You'll declare a winner. But the effect isn't real.

Looking less often just delays the same problem. You need to look often: products move fast, and running an experiment 16 days longer than necessary costs real money, delays launches, and burns opportunity cost. You need a test statistic that stays valid regardless of when you stop.

What a Sequential Test Actually Does

Sequential tests are designed for optional stopping by replacing the p-value with an alternative statistic called an e-value.

Unlike a p-value, an e-value is nonnegative, and the process formed by e-values over time satisfies a supermartingale property under the null: conditional on the history, the expected next e-value is at most the current one.

This path-level supermartingale condition is what makes optional stopping safe. Having a marginal mean below 1 at each step is necessary but not sufficient: the supermartingale condition is strictly stronger, holding the bound uniformly across all stopping times.

Here's why. If the e-value process is a nonneg supermartingale with E[e_t] ≤ 1 under H0, then a classical result called Ville's inequality gives: the probability that the running maximum of the process ever exceeds 1/α is at most α. With α = 0.05 and stopping threshold 1/α = 20, the probability that a null e-value process ever reaches 20 is at most 5%.

That Type I error bound holds no matter when you stop or how many times you check. The guarantee is time-uniform: it covers all possible stopping times simultaneously.

A classical p-value's guarantee applies only at the pre-committed sample size. Check repeatedly and the bound dissolves. There is no time-uniform analog.

The mSPRT computes the e-value as a Bayes factor: the ratio of the likelihood of the observed data under the alternative to that under the null.

The "mixture" part means you don't specify a single effect size under H1. You average the likelihood ratio over a prior distribution on effect sizes.

For Bernoulli outcomes (did the task complete: yes or no), placing a Beta(1,1) prior on each arm's completion rate makes the Bayes factor tractable in closed form using the log-beta function. The math is less intimidating than it looks: the entire computation reduces to four calls to betaln, as Step 2 shows.

The practical consequence is concrete: accumulate data, compute the running e-value each day, and stop when it crosses 20. When it remains below 20 across your maximum sample size, you fail to reject the null. Check every day, every hour, or every minute. The Type I error rate holds at 5%.

Identification Assumptions

mSPRT's always-valid guarantee rests on four conditions. Each can break, and the failure modes section below maps each failure mode to the condition it violates.

1. Nonneg supermartingale property under H0. The e-value process must satisfy E[e_{t+1} | e_1, ..., e_t] ≤ e_t under H0. For the Beta-Binomial Bayes factor used here, this holds as long as the prior is proper (Beta(1,1) qualifies) and the observations are i.i.d. within each arm.

2. Stationarity. The data-generating process must be stationary across the experiment window. If the underlying completion rate shifts mid-experiment due to an unrelated change (a model update, a cohort shift from a marketing campaign, or a day-of-week effect), the e-value picks up noise that your experiment can't separate from the treatment effect.

3. Independent observations within each arm. Each user's outcome must be independent of other users'. Network effects, shared workspaces, or spillover from recommendation systems can violate this.

4. Prior specification. The Beta(1,1) prior is a modeling assumption. The mSPRT's power depends on whether the prior places reasonable mass on the true effect size. A badly misspecified prior won't break the Type I error guarantee, but it can make the e-value grow so slowly that you exhaust your sample budget without crossing the threshold.

Prerequisites

• Python 3.11+

• pandas 2.x (pip install pandas)

• numpy 1.26+ (pip install numpy)

• scipy 1.12+ (pip install scipy)

• matplotlib 3.8+ (pip install matplotlib)

Clone the companion repo to get the synthetic dataset:

git clone https://github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm.git
cd product-experimentation-causal-inference-genai-llm
python data/generate_data.py --seed 42 --n-users 50000 --out data/synthetic_llm_logs.csv

Here's what's happening: this clones the repo that contains all 13 companion notebooks for this series, generates the shared 50,000-user synthetic dataset, and saves it to data/synthetic_llm_logs.csv. Every article in the series runs against this same CSV so the methods are directly comparable. The data generator bakes in a +5 percentage-point causal effect on task completion for wave 1 users.

Setting Up the Working Example

The synthetic dataset simulates a SaaS AI assistant product with 50,000 users. The task_completed column records whether the AI successfully completed the user's task (1) or not (0). The wave column assigns users to groups: wave 1 receives the new AI feature, wave 2 is the holdout control.

Figure 1: conceptual e-value trajectories. The blue path (real effect) rises and crosses the stopping threshold at the green dashed line. The purple path (weaker effect) grows but doesn't cross in 30 days. The grey path (null) meanders near 1 throughout. The red dashed line is the stopping boundary at 1/α = 20. Compare this to Figure 2 below, which shows the actual e-value trajectory on the real dataset.

import pandas as pd
import numpy as np

df = pd.read_csv("data/synthetic_llm_logs.csv")

treated = df[df["wave"] == 1]["task_completed"].values
control = df[df["wave"] == 2]["task_completed"].values

print(f"Treated: n={len(treated):,}, mean={treated.mean():.4f}")
print(f"Control: n={len(control):,}, mean={control.mean():.4f}")
print(f"Observed lift: {treated.mean() - control.mean():.4f}")

Expected output:

Treated: n=24,937, mean=0.6202
Control: n=25,063, mean=0.5718
Observed lift: 0.0485

Here's what's happening: you load the 50,000-row dataset and split by wave. Wave 1 has 24,937 treated users with a 62.0% task completion rate. Wave 2 has 25,063 control users with a 57.2% task completion rate. The observed 4.85 percentage-point lift is close to the ground-truth 5pp baked into the data generator, with the small gap due to sampling noise. These arrays feed the sequential test one observation at a time, as outlined in the steps below.

Step 1: Simulate the Peeking Problem

The peeking problem is real and measurable: 30 days of daily monitoring inflates your false positive rate from 4.2% to 30.2%, confirmed by the simulation below.

This simulation runs 1,000 null experiments (in which the treatment has zero effect) and checks every day whether the running p-value has dropped below 0.05. The scenario uses 60 users per arm per day across a 30-day experiment: 1,800 total observations per arm, a realistic scale for a mid-sized SaaS product.

from scipy import stats
import numpy as np

np.random.seed(42)

N_SIMS = 1000
N_DAYS = 30
USERS_PER_ARM_PER_DAY = 60
NULL_RATE = 0.60

false_positives_peeking = 0
false_positives_single_look = 0

for _ in range(N_SIMS):
    control_outcomes = []
    treated_outcomes = []
    stopped_early = False

    for day in range(N_DAYS):
        control_outcomes.extend(np.random.binomial(1, NULL_RATE, USERS_PER_ARM_PER_DAY))
        treated_outcomes.extend(np.random.binomial(1, NULL_RATE, USERS_PER_ARM_PER_DAY))

        # The peeking problem: checking the test every single day
        if len(control_outcomes) >= 10:
            _, p = stats.ttest_ind(treated_outcomes, control_outcomes)
            if p < 0.05 and not stopped_early:
                false_positives_peeking += 1
                stopped_early = True

    # The fixed-sample approach: checking only once at the very end
    _, p_final = stats.ttest_ind(treated_outcomes, control_outcomes)
    if p_final < 0.05:
        false_positives_single_look += 1

print(f"False positive rate (peeking daily):  {false_positives_peeking / N_SIMS:.1%}")
print(f"False positive rate (single look):    {false_positives_single_look / N_SIMS:.1%}")

Expected output:

False positive rate (peeking daily):  30.2%
False positive rate (single look):    4.2%

Here's what's happening: each simulation generates null data, with both arms drawn from the same 60% completion rate, so any detected effect is pure noise. The inner loop adds 60 observations per arm per day and runs a t-test on the accumulated data for that day.

When the p-value falls below 0.05 for the first time, the simulation flags a false positive and stops (mimicking a team that ships when it detects significance).

The single-look check at day 30 is the honest fixed-sample test. One look gives 4.2% false positives, close to nominal. Daily peeking reaches 30.2%, meaning more than one in four "significant" experiments is detecting noise.

Step 2: Implement the mSPRT e-value

The mSPRT computes a Bayes factor at each time step: how much more likely are the observed data under a mixture of alternatives than under the null? For binary outcomes with a Beta(1,1) prior on each arm's completion rate, the running Bayes factor has a closed form using the log-beta function.

from scipy.special import betaln

def compute_evalue_running(outcomes_treated, outcomes_control,
                           alpha_prior=1.0, beta_prior=1.0):
    """
    Compute the running mSPRT e-value for two Bernoulli arms.

    Parameters
    ----------
    outcomes_treated : array-like of 0/1
    outcomes_control : array-like of 0/1
    alpha_prior, beta_prior : Beta prior hyperparameters (default: uniform)

    Returns
    -------
    e_values : np.ndarray of shape (n,), one e-value per observation
    """
    outcomes_treated = np.asarray(outcomes_treated, dtype=float)
    outcomes_control = np.asarray(outcomes_control, dtype=float)
    n = min(len(outcomes_treated), len(outcomes_control))

    cum_t = np.cumsum(outcomes_treated[:n])
    cum_c = np.cumsum(outcomes_control[:n])
    t_arr = np.arange(1, n + 1, dtype=float)

    # Alternative hypothesis: each arm has its own independent Beta prior on completion rate
    log_ml_t = (betaln(alpha_prior + cum_t, beta_prior + t_arr - cum_t)
                - betaln(alpha_prior, beta_prior))
    log_ml_c = (betaln(alpha_prior + cum_c, beta_prior + t_arr - cum_c)
                - betaln(alpha_prior, beta_prior))

    # Null hypothesis: both arms share a single pooled Beta prior on the common rate
    pooled_successes = cum_t + cum_c
    pooled_n = 2 * t_arr
    log_ml_h0 = (betaln(alpha_prior + pooled_successes,
                        beta_prior + pooled_n - pooled_successes)
                 - betaln(alpha_prior, beta_prior))

    # Log Bayes factor is the difference in log marginal likelihoods
    log_bf = log_ml_t + log_ml_c - log_ml_h0

    return np.exp(log_bf)

Here's what's happening: the function takes two arrays of 0/1 outcomes arriving in temporal order. For each time step t, it computes the cumulative number of successes and trials for each arm.

betaln gives the log of the beta function, which is the normalizing constant for the Beta-Binomial marginal likelihood. H1 integrates over independent Beta priors on each arm's rate;.H0 integrates over a single shared-rate prior.

The log Bayes factor is the difference. Exponentiating gives the e-value. When the treatment has a real effect, the e-value grows over time. With no effect, it bounces near 1 and is a non-negative supermartingale under H0.

A quick sanity check on null data confirms the expected behavior:

np.random.seed(0)
null_t = np.random.binomial(1, 0.60, 500)
null_c = np.random.binomial(1, 0.60, 500)
ev_null = compute_evalue_running(null_t, null_c)
print(f"E-value at end under null (should be near 1): {ev_null[-1]:.3f}")
print(f"Max e-value under null: {ev_null.max():.3f}")

Expected output:

E-value at end under null (should be near 1): 0.078
Max e-value under null: 2.188

Here's what's happening: under the null, the final e-value ends near 1 (0.078 here, due to sampling variation), and the maximum over 500 observations stays well below the stopping threshold of 20. By Ville's inequality, the probability that a valid null e-value process ever reaches 20 is at most 5%, consistent with a 5% Type I error rate. In this single 500-observation run, the max is 2.188, which is expected behavior.

Step 3: Apply mSPRT to the Real Dataset

Now apply the test to the synthetic data where a real treatment effect exists. You'll compute the running e-value day by day and find the first day it crosses the stopping threshold.

import matplotlib.pyplot as plt

np.random.seed(42)
treated_shuffled = treated.copy()
control_shuffled = control.copy()
np.random.shuffle(treated_shuffled)
np.random.shuffle(control_shuffled)

USERS_PER_ARM_PER_DAY = 60
N_DAYS_RUN = 30
n_per_arm = USERS_PER_ARM_PER_DAY * N_DAYS_RUN  # 1,800

treated_seq = treated_shuffled[:n_per_arm]
control_seq = control_shuffled[:n_per_arm]

e_values = compute_evalue_running(treated_seq, control_seq)

ALPHA = 0.05
THRESHOLD = 1 / ALPHA  # = 20

days = np.arange(1, len(e_values) + 1) / USERS_PER_ARM_PER_DAY
cross_indices = np.where(e_values >= THRESHOLD)[0]
if len(cross_indices) > 0:
    stopping_day = days[cross_indices[0]]
    print(f"mSPRT stopping day: {stopping_day:.1f}")
    print(f"E-value at stopping: {e_values[cross_indices[0]]:.1f}")
else:
    stopping_day = None
    print("mSPRT did not cross threshold in this window")

print(f"Final e-value on day {N_DAYS_RUN}: {e_values[-1]:.2f}")

Expected output:

mSPRT stopping day: 25.9
E-value at stopping: 20.9
Final e-value on day 30: 75.64

Here's what's happening: you shuffle the treatment and control arrays to simulate random daily arrival of users (real experiments don't deliver users in any particular order), then feed the first 1,800 per arm into compute_evalue_running one observation at a time. The e-value crosses the threshold of 20 on day 25.9, meaning you could have called the experiment 4 days early with a fully valid inference guarantee. By day 30, the e-value has climbed to 75.64, far above the threshold.

Figure 2: actual mSPRT e-value trajectory on the real 50,000-user synthetic dataset (wave 1 treatment vs. wave 2 control). The blue line is the running e-value on a log scale. The red dashed line is the stopping threshold at 1/α = 20.

The dotted green vertical line marks day 25.9, when the e-value first crosses the threshold. The bottom panel shows cumulative task completion rates per arm converging as data accumulates. Unlike the schematic in Figure 1, these are real data from the shared dataset, with a true 4.85 pp lift.

Step 4: Compare Power Against a Fixed-Sample Test

The mSPRT carries a real cost. When the effect is active, it lets you stop earlier than the scheduled end time. When the effect is smaller than your prior expects, or when you're working with modest sample sizes, the power penalty is substantial. This simulation quantifies the trade-off honestly.

from scipy.stats import ttest_ind

np.random.seed(42)

N_SIMS = 1000
TRUE_EFFECT = 0.05
BASE_RATE = 0.60
N_PER_ARM = 1800          # 30 days x 60 users/arm/day
DAILY_BATCH = 60
THRESHOLD = 20

msprt_stopping_days = []
msprt_detected = 0
ttest_detected = 0

for sim in range(N_SIMS):
    t_obs = np.random.binomial(1, BASE_RATE + TRUE_EFFECT, N_PER_ARM)
    c_obs = np.random.binomial(1, BASE_RATE, N_PER_ARM)

    e_vals = compute_evalue_running(t_obs, c_obs)
    days = np.arange(1, N_PER_ARM + 1) / DAILY_BATCH
    crosses = np.where(e_vals >= THRESHOLD)[0]
    if len(crosses) > 0:
        msprt_detected += 1
        msprt_stopping_days.append(days[crosses[0]])
    else:
        msprt_stopping_days.append(30.0)

    _, p = ttest_ind(t_obs, c_obs)
    if p < 0.05:
        ttest_detected += 1

msprt_power = msprt_detected / N_SIMS
ttest_power = ttest_detected / N_SIMS
median_stop = np.median(msprt_stopping_days)
pct_stopped_early = np.mean(np.array(msprt_stopping_days) < 30.0)

print(f"mSPRT power:               {msprt_power:.1%}")
print(f"Fixed-sample t-test power: {ttest_power:.1%}")
print(f"Median mSPRT stop day:     {median_stop:.1f} / 30")
print(f"Fraction stopping early:   {pct_stopped_early:.1%}")

Expected output:

mSPRT power:               49.3%
Fixed-sample t-test power: 88.7%
Median mSPRT stop day:     30.0 / 30
Fraction stopping early:   49.3%

Here's what's happening: you run 1,000 simulations with a true 5pp lift. For mSPRT, the running e-value is computed, and the first crossing of 20 is recorded.

For the fixed-sample test, you look once at the end of day 30. The results show a meaningful power gap: mSPRT detects the effect in 49.3% of experiments, whereas the fixed-sample test detects it in 88.7%. With a 5pp lift and 1,800 observations per arm, the mSPRT requires roughly twice as many observations to match the fixed-sample test's power.

That's the price of the always-valid guarantee. What you gain is the Type I error control when you check daily: a fixed-sample test peeked at daily inflates to 30.2% false positives. mSPRT stays at 5% regardless of when you stop.

The right choice depends on which is more expensive for your team: running experiments longer, or shipping false positives. Most teams underestimate the cost of power until they run this simulation themselves.

Validate Against Ground Truth

The synthetic dataset incorporates a known 5pp lift, so you can check whether mSPRT correctly identifies the effect when given more data beyond the 30-day window.

np.random.seed(0)
t_full = treated_shuffled
c_full = control_shuffled[:len(t_full)]

e_full = compute_evalue_running(t_full, c_full)
days_full = np.arange(1, len(e_full) + 1) / USERS_PER_ARM_PER_DAY

cross_full = np.where(e_full >= THRESHOLD)[0]
if len(cross_full) > 0:
    print(f"mSPRT correctly detected the effect.")
    print(f"Could have stopped on day {days_full[cross_full[0]]:.1f}")
    print(f"True effect in data: {treated.mean() - control.mean():.4f}")
    print(f"E-value at stopping point: {e_full[cross_full[0]]:.1f}")
else:
    print("mSPRT did not cross threshold with this data slice.")

Expected output:

mSPRT correctly detected the effect.
Could have stopped on day 27.1
True effect in data: 0.0485
E-value at stopping point: 22.2

Here's what's happening: running mSPRT on the full shuffled arrays (24,937 treated, 25,063 control), the e-value crosses the threshold at day 27.1. The true causal effect in the data, 4.85 pp, is close to the generator's ground truth of 5 pp and is correctly detected.

A fixed-sample test designed for 30 days holds you to day 30 even when the evidence has already accumulated. With 60 users per arm per day, mSPRT would have let you ship on day 27.1, saving almost 3 days on a feature that was always going to ship.

Step 5: Bootstrap Confidence Intervals

A stopping day tells you when to call the experiment, but it doesn't tell you how large the effect is or how precisely it's estimated. Bootstrap confidence intervals give you both.

rng = np.random.default_rng(7)
point_est = treated.mean() - control.mean()

boot_diffs = np.array([
    rng.choice(treated, size=len(treated), replace=True).mean() -
    rng.choice(control, size=len(control), replace=True).mean()
    for _ in range(500)
])

lower = float(np.percentile(boot_diffs, 2.5))
upper = float(np.percentile(boot_diffs, 97.5))

print(f"Point estimate (treated - control): {point_est:.4f} ({point_est*100:.2f}pp)")
print(f"95% bootstrap CI: [{lower:.4f}, {upper:.4f}]  "
      f"([{lower*100:.2f}pp, {upper*100:.2f}pp])")
print(f"Ground-truth 5pp is {'inside' if lower <= 0.05 <= upper else 'outside'} the CI.")

Expected output:

Point estimate (treated - control): 0.0485 (4.85pp)
95% bootstrap CI: [0.0407, 0.0581]  ([4.07pp, 5.81pp])
Ground-truth 5pp is inside the CI.

Here's what's happening: you resample the treated and control arrays independently with replacement 500 times, computing the difference in means each time. The 2.5th and 97.5th percentiles of the 500 differences form the confidence interval. The CI runs from 4.07pp to 5.81pp, covering the ground-truth 5pp and excluding zero, confirming the effect is real. The interval is reasonably tight given 25k users per arm, giving you both the "did it work" answer (yes) and the "how much" answer (between 4.07 and 5.81 percentage points) in a single step.

When mSPRT Fails

Sequential tests still demand experimental rigor. Four situations either break the guarantee or make the method practically useless.

Badly Misspecified Prior

The mSPRT assumes a Beta(1,1) prior on each arm's completion rate, a modeling choice with real consequences. This violates the prior specification assumption when your true effect is far outside the range the prior expects.

A uniform Beta(1,1) prior performs reasonably well for moderate effects in the 3–10 pp range at base rates around 60%. If your true effect is a 0.3pp lift, a realistic outcome for a marginal AI feature change, the e-value grows extremely slowly. You'll exhaust your sample budget before crossing the threshold.

Calibrate the prior against historical A/B test data from your product: fit Beta hyperparameters to the distribution of past effect sizes using maximum likelihood, and verify that the resulting prior puts meaningful mass near your minimum detectable effect.

Non-Stationary Outcomes

The guarantee requires the e-value process to be a non-negative supermartingale under the null, which requires the data-generating process to be stationary. If your AI model updates mid-experiment, if the user population shifts (a marketing campaign brings in a different cohort on day 12), or if there's a day-of-week effect in task difficulty, the e-value absorbs environment noise that your experiment can't separate from the treatment effect.

Diagnose non-stationarity by running your e-value implementation on holdout A/A experiments: if the null e-value process trends upward when it should stay near 1, your environment isn't stationary enough for the method to be reliable.

Multiple Metrics Without Multiplicity Correction

mSPRT controls Type I error for a single comparison. The method itself doesn't fail when you test 20 metrics, so each individual e-value remains valid. What fails is your familywise error rate: running mSPRT on 20 metrics simultaneously and stopping when any one crosses 20 inflates the probability of at least one false positive well above 5%.

Apply a Bonferroni correction by raising the threshold to 1/(α/m) = 400 for m=20 metrics at α=0.05, or use a Benjamini-Hochberg procedure on the final e-values when the experiment ends.

The multiplicity problem is identical to the one you'd face with fixed-sample tests. mSPRT doesn't make it worse, and it doesn't solve it either. This is a common misconception worth naming explicitly.

Minimum Runtime is Still Real

Because the always-valid guarantee applies regardless of when you check, it's tempting to start monitoring immediately. Don't. The guarantee holds whenever you check, but low power means the test rarely rejects even when the effect is real.

The Step 4 simulation shows this directly: with 1,800 observations per arm and a 5 pp lift, mSPRT has only 49.3% power. Before starting an mSPRT-monitored experiment, compute the minimum sample size for 80% power at your expected effect size using a standard power calculator, and set that as your floor before you start monitoring. Don't check the e-value until you've reached that floor.

What to Do Next

Apply mSPRT to your primary metric, with a minimum runtime floor set to the sample size required for 80% power at your expected effect size.

Run A/A tests on historical holdout data first: the calibration check costs you nothing and catches non-stationary environments before they corrupt a real experiment. Teams that skip the A/A test discover calibration failures during live experiments. That's an expensive way to learn about non-stationary data.

For the full implementation including bootstrap confidence intervals, see 07_sequential_msprt/ in the companion repo.

You're ready to test it in production. Then your platform engineer raises a structural problem: you can't randomize at the user level.

This issue is rooted in causal inference and runs deeper than a technical constraint. Every user draws from a centralized pool of premium model capacity. A standard A/B test creates an uneven playing field in this environment. When the routing AI is active for the treatment group, those users consume premium resources first, leaving the control group with degraded availability.

The routing AI does more than alter the treatment group's experience. It fundamentally shifts the resource environment for everyone else. You're not isolating the AI's impact. You're measuring the combined effect of the routing AI and the artificial scarcity your experimental design imposed on the control group. That's a confounded measurement, not a clean experiment.

Switchback experiments are the standard fix for LLM-based platforms and for any shared-resource product where user-level randomization would break the comparison. You stop randomizing users and randomize time slots instead.

The full platform runs with AI routing on for a 30-minute slot, then off for the next 30 minutes. You repeat the cycle, accumulate enough slots, and estimate the average treatment effect from the contrast between AI-on and AI-off slots.

This tutorial walks through the full switchback pipeline in Python: building the time series from session logs, diagnosing carryover contamination, estimating the direct effect with and without carryover adjustment, applying HAC standard errors for time-series data, computing bootstrap confidence intervals, and validating all estimates against a known ground truth.

By the end, you'll know how to run this analysis on your own LLM platform data and how to spot the four conditions that break it.

Table of Contents

• Why User-Level A/B Testing Fails on Shared LLM Infrastructure

• How Switchback Design Restores a Clean Comparison

  • Identification Assumptions

  • Prerequisites

  • Step 1: Build the Switchback Time Series

  • Step 2: Naïve Estimate (Ignoring Time Structure)

  • Step 3: Carryover-Adjusted OLS Regression

  • Step 4: HAC Standard Errors for Time-series Data

  • Step 5: Bootstrap Confidence Intervals

• Validating Against the Ground Truth

• When Switchback Fails

• When to Use Switchback vs. Cluster Randomization

• What to Do Next

Why User-Level A/B Testing Fails on Shared LLM Infrastructure

Standard A/B testing buys you causal inference through randomization. When you flip a coin to assign each user to treatment or control, both groups share identical distributions of every confounder on average. Differences in outcomes trace back to the treatment. The logic holds when users act independently of each other.

Shared LLM infrastructure breaks that independence. Consider the query-routing scenario. If 50% of users are assigned to AI routing, they receive priority access to the premium model, enabling them to complete tasks faster and at higher rates. The remaining 50% operate in a degraded environment, where premium-model queues are longer because treatment-group sessions occupy capacity. Control-group users experience worse availability not because the AI routing feature fails them, but because your experiment design created artificial scarcity for them.

Interference is the structural problem here: the Stable Unit Treatment Value Assumption, known as SUTVA, holds that a unit's outcome depends solely on that unit's treatment assignment.

SUTVA fails on shared LLM infrastructure. A treated user's session claims capacity that determines whether a control user gets routed to the premium model or the degraded standard model. The control group is no longer a clean counterfactual.

The estimated treatment effect under user-level randomization is:

Naive ATE = E[outcome | AI-on user] - E[outcome | AI-off user, degraded capacity]

The counterfactual you actually need is what AI-off users would have experienced if no users had AI routing, with no capacity degradation. You never observe that counterfactual in a 50/50 user-level split. Your estimate conflates the routing AI's direct effect with the capacity-degradation penalty, and separating them requires knowing the full capacity-utilization function, which you almost never have.

Other shared-resource LLM platform patterns produce the same failure: a caching layer that speeds retrieval for treated users but drains shared cache space for control users, and a fine-tuned model version that consumes GPU memory, leaving standard inference slower for the control group, or a batch-processing scheduler that prioritizes AI-routed requests and creates queuing delays for everything else. Anything touching a shared resource pool contaminates the control group.

How Switchback Design Restores a Clean Comparison

Because standard randomization can poison the control group through shared resources, a switchback design changes what you randomize. You stop randomizing users. You randomize time slots.

The entire platform operates under a single treatment condition at any given time: AI routing is either on or off for all users.

The treatment indicator switches between slots on a predetermined schedule, cycling through alternating blocks across the experiment. At the end of the run, you have a time series of slots, each with a treatment indicator and an aggregate outcome, such as the mean task completion rate or the mean cost per session. You regress the outcome on the treatment indicator, and the coefficient is your average treatment effect estimate.

Figure 1: Conceptual schematic of the 3-slot switchback design. Blue regions are AI-routing-on blocks, while orange marks the first AI-off slot of each cycle where carryover from the prior on-block artificially elevates outcomes.
The green band shows the true 6 pp direct effect. A naïve comparison of all-on vs. all-off slots inflates the estimated effect because it can't disentangle the direct contribution from within-block carryover.

A clean comparison is restored because the platform operates under a single condition for any given slot. Every user within a slot sees the same treatment. The AI-off slots function as a reliable counterfactual for the AI-on slots, provided that demand conditions remain comparable across slots.

The key complication is carryover. If AI routing effects persist into a subsequent AI-off slot due to factors such as warm routing caches, in-flight sessions that began under AI routing and complete after the switch, or changed user behavior that persists across the slot boundary, then AI-off slot outcomes are artificially elevated by residual AI effects.

The naïve comparison conflates this inherited elevation with the direct treatment effect, biasing the estimate upward. Estimating and removing carryover is the core analytical challenge in switchback experiments: it's where most of the real work lives, and most of what this tutorial covers.

Identification Assumptions

Switchback estimates have a causal interpretation only when four conditions hold.

1. Zero or bounded carryover between slots.

AI routing effects from one slot don't persist far enough into later slots to bias the comparison. The carryover model in this tutorial captures first-order persistence (one lag). If effects persist for multiple periods, you need more lag terms in the regression.

2. Demand stationarity across the treatment schedule.

AI-on and AI-off slots face similar underlying demand conditions. If Monday morning slots are always AI-on and Sunday afternoon slots are always AI-off, demand differences contaminate the treatment comparison in ways no lag correction can fix.

3. No ramp-up effects at block boundaries.

The system reaches steady-state behavior within each slot. If the first slot of each AI-on block performs worse than subsequent slots because the routing model's cache is cold, that ramp-up period produces a downward-biased estimate of the steady-state direct effect.

4. Residual autocorrelation is addressed.

Slot residuals may be correlated over time due to demand cycles, capacity events, and platform-level shocks spanning multiple periods. HAC standard errors or bootstrap CIs correct for this (as plain OLS standard errors aren't sufficient).

The "When switchback fails" section maps each failure mode to the specific assumption it violates.

All code in this tutorial runs end-to-end in the companion notebook at 06_switchback/switchback_demo.ipynb.

Prerequisites

• Python 3.11+

• pandas 2.x (pip install pandas)

• numpy 1.26+ (pip install numpy)

• statsmodels 0.14+ (pip install statsmodels)

• matplotlib 3.8+ (pip install matplotlib)

Clone the companion repo to get the synthetic dataset:

git clone https://github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm
cd product-experimentation-causal-inference-genai-llm
python data/generate_data.py

The generate script writes data/synthetic_llm_logs.csv, a 50,000-row file of synthetic SaaS LLM product telemetry. Key columns are user_id, task_completed (binary outcome), cost_usd, and session_minutes.

After slot assignment in Step 1, each of the 48 time slots contains approximately 1,042 sessions. The dataset represents realistic LLM platform traffic: query arrival rates, model cost distributions, and session lengths are drawn from distributions calibrated to production patterns.

Step 1: Build the Switchback Time Series

Switchback experiments are run with a live treatment-assignment controller that flips the routing AI on or off at the slot boundary in production.

For this tutorial, you construct the time series from the session log by mapping each row to a synthetic hour slot, then aggregating to the slot level.

import pandas as pd
import numpy as np

df = pd.read_csv("data/synthetic_llm_logs.csv")
print(f"Dataset shape: {df.shape}")
print(df[["user_id", "task_completed", "cost_usd", "session_minutes"]].head(3).round(3))

# Shuffle to eliminate row-ordering bias before slot assignment
df = df.sample(frac=1, random_state=42).reset_index(drop=True)

# Assign hour slots: 48 slots, each containing ~1,042 sessions
df['hour_slot'] = df.index % 48

# Treatment schedule: 3-slot blocks (on, on, on, off, off, off, ...)
# 3-slot blocks give the platform time to settle into each state and break
# the perfect collinearity between ai_on and its one-period lag.
ai_on_schedule = np.tile([1, 1, 1, 0, 0, 0], 8)   # 48 slots, 8 full cycles
df['ai_on'] = ai_on_schedule[df['hour_slot']]

# Aggregate to slot level: mean outcome, mean cost, treatment indicator, session count
slots = df.groupby('hour_slot').agg(
    mean_task_completed = ('task_completed', 'mean'),
    mean_cost           = ('cost_usd',       'mean'),
    ai_on               = ('ai_on',          'first'),
    n_obs               = ('user_id',         'count')
).reset_index()

print(f"\nSlot-level data: {len(slots)} slots")
print(slots[['hour_slot', 'ai_on', 'mean_task_completed', 'mean_cost', 'n_obs']].head(8).round(4))
print(f"\nAI-on slots: {slots['ai_on'].sum()},  AI-off slots: {(1 - slots['ai_on']).sum()}")

Expected output:

Dataset shape: (50000, 16)
   user_id  task_completed  cost_usd  session_minutes
0        0               0     0.022             7.03
1        1               1     0.008             4.07
2        2               1     0.040             8.34

Slot-level data: 48 slots
   hour_slot  ai_on  mean_task_completed  mean_cost  n_obs
0          0      1               0.5950     0.0222   1042
1          1      1               0.5806     0.0223   1042
2          2      1               0.5950     0.0224   1042
3          3      0               0.6353     0.0218   1042
4          4      0               0.6017     0.0222   1042
5          5      0               0.6094     0.0218   1042
6          6      1               0.5912     0.0218   1042
7          7      1               0.5931     0.0219   1042

AI-on slots: 24,  AI-off slots: 24

The process begins by shuffling the dataset before slot assignment to eliminate any row-ordering artifacts from data generation. Each of the 50,000 rows is assigned to one of 48 synthetic hour slots using modulo arithmetic, and the treatment schedule alternates in 3-slot blocks, completing eight full cycles.

The 3-slot block structure serves two purposes: it gives the platform time to settle into each treatment state, and it breaks the perfect collinearity between the current treatment indicator and its one-period lag, which would otherwise make carryover estimation impossible under a purely alternating schedule. After aggregation, each slot contains approximately 1,042 sessions.

Notice that before injection, the slot-level means don't yet separate clearly by treatment. Slots 3, 4, and 5 (AI-off) show slightly higher completion rates than slots 0, 1, and 2 (AI-on) in the raw data. That's expected: before injection, the treatment assignment is arbitrary, and outcomes carry no true signal. The injection step below bakes in the ground truth.

# Known ground truth baked into the simulation
TRUE_EFFECT = 0.060   # AI routing raises task completion by 6 percentage points
CARRYOVER   = 0.030   # Residual routing effect persists into the following slot

# Replace slot means with synthetic balanced base rates.
# Slot noise std matches the CLT variance of aggregating ~1,042 Bernoulli sessions,
# simulating realistic slot-to-slot demand variation without treatment-group imbalance.
BASE_RATE = df['task_completed'].mean()
slot_noise_std = np.sqrt(BASE_RATE * (1 - BASE_RATE) / slots['n_obs'].iloc[0])
rng = np.random.default_rng(42)
slots['mean_task_completed'] = BASE_RATE + rng.normal(0, slot_noise_std, size=len(slots))

# Lag the treatment indicator: did the previous slot have AI routing on?
slots['ai_on_lag1'] = slots['ai_on'].shift(1).fillna(0).astype(int)

# Observed outcome = base outcome + treatment effect + carryover from prior slot
slots['mean_task_completed'] = (
    slots['mean_task_completed']
    + TRUE_EFFECT * slots['ai_on']
    + CARRYOVER   * slots['ai_on_lag1']
)

print("Post-injection slot data:")
print(slots[['hour_slot', 'ai_on', 'ai_on_lag1', 'mean_task_completed']].head(8).round(4))

Expected output:

Post-injection slot data:
   hour_slot  ai_on  ai_on_lag1  mean_task_completed
0          0      1           0               0.6606
1          1      1           1               0.6701
2          2      1           1               0.6973
3          3      0           1               0.6402
4          4      0           0               0.5663
5          5      0           0               0.5761
6          6      1           0               0.6579
7          7      1           1               0.6811

The injection substitutes raw slot means with noise calibrated to the variance of 1,042 Bernoulli trials, producing slot-to-slot fluctuation that mirrors production demand variability without artificial treatment-group imbalance.

The lag of ai_on identifies which slots immediately follow an AI-on period. The injection formula then adds TRUE_EFFECT (0.060) to every AI-on slot and CARRYOVER (0.030) to every slot that follows an AI-on slot, regardless of its own treatment status.

Look at slot 3: ai_on=0 but ai_on_lag1=1, so its outcome receives the +0.030 carryover boost even though AI routing is off. That's the carryover contamination a naïve model can't see.

The first AI-off slot of each cycle reflects a genuine off period, but its outcome is elevated by residual routing state from the previous block. A naïve comparison of all AI-on vs. all AI-off slots treats that elevated outcome as part of the AI-off baseline, distorting the true direct effect.

Figure 2: Left: the 48-slot time series from the synthetic dataset after injecting a 6 pp treatment effect and 3 pp carryover. Orange dots mark the first AI-off slot of each cycle (ai_on=0, ai_on_lag1=1), where outcomes remain elevated from the prior AI-on block.
Right: naïve OLS (red) overshoots the true 6 pp effect by 0.9 pp because it conflates direct and inherited carryover. The carryover-adjusted OLS (blue) recovers the true effect. Both 95% bootstrap CIs include the green dashed true-effect line.

Step 2: Naive Estimate (Ignoring Time Structure)

Before adding any sophistication, compute the obvious estimate: regress mean task completion on the binary AI-on indicator, ignoring the time structure entirely.

import statsmodels.api as sm

# Naive OLS: outcome ~ constant + ai_on
# No lag term, no time controls
X_naive = sm.add_constant(slots['ai_on'])
naive_model = sm.OLS(slots['mean_task_completed'], X_naive).fit()

naive_ate = naive_model.params['ai_on']
naive_se  = naive_model.bse['ai_on']

print("=== Naive estimate (no carryover control) ===")
print(f"  ATE estimate : {naive_ate:.4f}")
print(f"  Std error    : {naive_se:.4f}")
print(f"  95% CI       : [{naive_ate - 1.96*naive_se:.4f},  {naive_ate + 1.96*naive_se:.4f}]")
print(f"\n  True effect  : {TRUE_EFFECT}")
print(f"  Bias         : {naive_ate - TRUE_EFFECT:+.4f}")

Expected output:

=== Naive estimate (no carryover control) ===
  ATE estimate : 0.0688
  Std error    : 0.0048
  95% CI       : [0.0595,  0.0782]

  True effect  : 0.06
  Bias         : +0.0088

The naïve OLS regresses mean task completion on the binary AI-on indicator alone, treating the 48 slots as 48 independent observations with no time structure. It returns an ATE of 0.0688 against a true direct effect of 0.060, a bias of +0.0088, nearly a full percentage point of artificial lift.

The bias stems from how carryover is distributed between the two groups. In a 3-slot-on / 3-slot-off design, slots 1 and 2 of every AI-on block receive both the direct treatment effect (+0.060) and the carryover effect (+0.030) from the previous on-slot, pushing their outcomes to base + 0.090.

The naïve model can't separate these two contributions: it sees a high outcome in an AI-on slot and attributes it entirely to the direct treatment. Across 24 AI-on slots, 16 receive this compound injection, pulling the group average well above the true direct effect.

On the AI-off side, the first off-slot of each block receives +0.030 carryover, which raises the AI-off group's baseline. That partially offsets the AI-on group inflation, but 16 slots of compound AI-on inflation outweigh 8 slots of AI-off carryover. The net result is a positive bias of roughly +0.009 percentage points.

A team acting on 0.0688, when the true effect is 0.060, will declare a larger effect than exists and over-prioritize the routing feature relative to other initiatives.

Step 3: Carryover-Adjusted OLS Regression

The fix is to add the lagged treatment indicator to the regression. The coefficient on ai_on then measures the direct effect of the current period's treatment, holding the prior period's treatment constant. That's the quantity you want.

# Carryover-adjusted OLS: outcome ~ constant + ai_on + ai_on_lag1
X_adj = sm.add_constant(slots[['ai_on', 'ai_on_lag1']])
adj_model = sm.OLS(slots['mean_task_completed'], X_adj).fit()

adj_ate      = adj_model.params['ai_on']
adj_carryover = adj_model.params['ai_on_lag1']
adj_se        = adj_model.bse['ai_on']

print("=== Carryover-adjusted estimate ===")
print(adj_model.summary().tables[1])

print(f"\n  Direct ATE estimate  : {adj_ate:.4f}  (true: {TRUE_EFFECT})")
print(f"  Carryover estimate   : {adj_carryover:.4f}  (true: {CARRYOVER})")
print(f"  Residual bias        : {adj_ate - TRUE_EFFECT:+.4f}")

# How much did we remove?
removed = naive_ate - adj_ate
print(f"\n  Bias removed vs naive: {removed:.4f}")

Expected output:

=== Carryover-adjusted estimate ===
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const          0.5996      0.003    222.975      0.000       0.594       0.605
ai_on          0.0607      0.004     16.830      0.000       0.053       0.068
ai_on_lag1     0.0244      0.004      6.754      0.000       0.017       0.032
==============================================================================

  Direct ATE estimate  : 0.0607  (true: 0.06)
  Carryover estimate   : 0.0244  (true: 0.03)
  Residual bias        : +0.0007

  Bias removed vs naive: 0.0081

The adjusted regression includes both ai_on (current slot treatment) and ai_on_lag1 (previous slot treatment) as regressors.

The model now decomposes the drivers of elevated outcomes in each slot: some elevation comes from the current period's AI routing, and some from the previous period's residual. The coefficient on ai_on isolates only the current-period direct effect.

The direct ATE estimate drops from 0.0688 to 0.0607, recovering the true value of 0.060 to within 0.0007, with a residual bias smaller than the standard error.

The carryover estimate is 0.0244, compared with a true carryover of 0.030. Some underestimation is expected: the 3-slot block structure creates slots where both ai_on and ai_on_lag1 equal 1, introducing mild collinearity that slightly attenuates the carryover coefficient. Adding ai_on_lag1 removed 0.0081 of the 0.0088 naïve bias, recovering roughly 92% of the upward distortion.

The two-coefficient interpretation matters for product decisions. The ai_on coefficient (0.0607) is the direct effect: what AI routing adds in the current slot, independent of what happened in the prior slot. The ai_on_lag1 coefficient (0.0244) is the carryover effect: the residual impact that persists into the next slot after routing is switched off. In a real LLM platform, carryover might reflect session-level state, warm inference caches, or shifts in user behavior that span the slot boundary.

If ai_on_lag2 and ai_on_lag3 still improve model fit as measured by decreasing AIC, your slot length is shorter than the system's memory, and you need more lag terms. Add lags until AIC stops improving, and use domain knowledge to set a ceiling on plausible persistence given your platform's architecture.

Step 4: HAC Standard Errors for Time-series Data

The adjusted OLS model gives you the right point estimate. But the standard errors it reports assume residuals are uncorrelated across time.

Slot residuals inherit any systematic variation not captured by the treatment indicators: demand cycles, capacity events, model-version deployments, and user behavior patterns that span multiple periods. That autocorrelation makes OLS standard errors too small, which inflates your t-statistics and makes the effect look more precisely measured than it is.

The correction is Heteroskedasticity- and Autocorrelation-Consistent (HAC) standard errors, also called Newey-West standard errors. They correct for serial correlation in residuals using a bandwidth parameter equal to the number of lags you expect to matter.

from statsmodels.stats.sandwich_covariance import cov_hac
from statsmodels.stats.stattools import durbin_watson

# First check for autocorrelation in the residuals
dw_stat = durbin_watson(adj_model.resid)
print(f"Durbin-Watson statistic: {dw_stat:.4f}")
print("  DW near 2.0 = little autocorrelation in residuals.")
print("  DW < 1.5 = positive serial correlation.")
print("  DW > 2.5 = negative serial correlation.")
print("  Apply HAC standard errors regardless -- DW only tests AR(1) structure.")

# Apply HAC correction (Newey-West), 3 lags
hac_cov = cov_hac(adj_model, nlags=3)
hac_se  = np.sqrt(np.diag(hac_cov))

print("\n=== Standard error comparison ===")
print(f"  OLS SE on ai_on  : {adj_model.bse['ai_on']:.4f}")
print(f"  HAC SE on ai_on  : {hac_se[1]:.4f}")
print(f"  OLS t-stat       : {adj_model.tvalues['ai_on']:.2f}")
print(f"  HAC t-stat       : {adj_ate / hac_se[1]:.2f}")

# Construct HAC-based confidence interval manually
hac_ci_lower = adj_ate - 1.96 * hac_se[1]
hac_ci_upper = adj_ate + 1.96 * hac_se[1]
print(f"\n  HAC 95% CI: [{hac_ci_lower:.4f},  {hac_ci_upper:.4f}]")
print(f"  True effect {TRUE_EFFECT} inside CI: {hac_ci_lower < TRUE_EFFECT < hac_ci_upper}")

Expected output:

Durbin-Watson statistic: 1.9628
  DW near 2.0 = little autocorrelation in residuals.
  DW < 1.5 = positive serial correlation.
  DW > 2.5 = negative serial correlation.
  Apply HAC standard errors regardless -- DW only tests AR(1) structure.

=== Standard error comparison ===
  OLS SE on ai_on  : 0.0036
  HAC SE on ai_on  : 0.0037
  OLS t-stat       : 16.83
  HAC t-stat       : 16.41

  HAC 95% CI: [0.0535,  0.0680]
  True effect 0.06 inside CI: True

The Durbin-Watson statistic near 2.0 (1.9628) indicates very little AR(1) autocorrelation in the residuals on this synthetic dataset, so the HAC and OLS standard errors are nearly identical. The HAC 95% CI [0.0535, 0.0680] contains the true effect of 0.060, confirming the adjusted estimate is valid.

In production LLM platforms where demand correlates across consecutive hours (morning surges, lunchtime dips, evening peaks), positive serial correlation causes OLS standard errors to understate uncertainty. I've seen teams skip this step and report t-statistics of 20+ on effects that don't hold up.

HAC corrections in those settings bring those numbers down to realistic levels and occasionally flip a "significant" result to inconclusive. The flip to inconclusive is the method working correctly. Apply HAC by default in any time-series regression: it costs nothing when autocorrelation is absent, and it provides real protection when it's present.

The nlags parameter deserves deliberate choice. A reasonable default is the number of slots you'd expect your largest demand cycle to span. If your platform shows strong hour-of-day patterns and you're using 30-minute slots, set nlags=4 or nlags=6 to cover the two-to-three-hour neighborhood. If you use two-hour slots, nlags=2 or nlags=3 usually covers the relevant range.

Step 5: Bootstrap Confidence Intervals

HAC standard errors correct for autocorrelation under the assumption that the autocorrelation structure follows a specific parametric form. Bootstrap CIs make no such assumption. They quantify estimation uncertainty by resampling slots with replacement and recomputing the estimator each time.

def bootstrap_ci(slots, B=500, seed=7):
    """Bootstrap CIs treating each slot as an independent observation.
  
    Each slot's ai_on_lag1 value is fixed from the original treatment schedule.
    Resampling slots with replacement while keeping their original lag values
    correctly quantifies estimation uncertainty without destroying the lag structure.
    """
    rng  = np.random.default_rng(seed)
    n    = len(slots)
    naive_ates, adj_ates, carryover_ests = [], [], []

    for _ in range(B):
        idx = rng.integers(0, n, size=n)
        s   = slots.iloc[idx]  # ai_on_lag1 stays as the original slot's value

        X_n = sm.add_constant(s['ai_on'])
        naive_ates.append(sm.OLS(s['mean_task_completed'], X_n).fit().params['ai_on'])

        X_a = sm.add_constant(s[['ai_on', 'ai_on_lag1']])
        m   = sm.OLS(s['mean_task_completed'], X_a).fit()
        adj_ates.append(m.params['ai_on'])
        carryover_ests.append(m.params['ai_on_lag1'])

    naive_ci     = np.percentile(naive_ates,     [2.5, 97.5])
    adj_ci       = np.percentile(adj_ates,       [2.5, 97.5])
    carryover_ci = np.percentile(carryover_ests, [2.5, 97.5])

    print(f"\n=== Bootstrap 95% confidence intervals (B={B}, seed={seed}) ===")
    print(f"  Naive ATE        : [{naive_ci[0]:.4f},  {naive_ci[1]:.4f}]  "
          f"(covers {TRUE_EFFECT}: {naive_ci[0] < TRUE_EFFECT < naive_ci[1]})")
    print(f"  Adjusted ATE     : [{adj_ci[0]:.4f},  {adj_ci[1]:.4f}]  "
          f"(covers {TRUE_EFFECT}: {adj_ci[0] < TRUE_EFFECT < adj_ci[1]})")
    print(f"  Carryover effect : [{carryover_ci[0]:.4f},  {carryover_ci[1]:.4f}]  "
          f"(covers {CARRYOVER}: {carryover_ci[0] < CARRYOVER < carryover_ci[1]})")

    return naive_ci, adj_ci, carryover_ci

naive_ci, adj_ci, carryover_ci = bootstrap_ci(slots)

Expected output:

=== Bootstrap 95% confidence intervals (B=500, seed=7) ===
  Naive ATE        : [0.0596,  0.0783]  (covers 0.06: True)
  Adjusted ATE     : [0.0541,  0.0683]  (covers 0.06: True)
  Carryover effect : [0.0175,  0.0320]  (covers 0.03: True)

Each bootstrap iteration resamples 48 slots with replacement, refits both the naive and adjusted OLS models, and records the key estimates. The 2.5th and 97.5th percentiles of those 500 replications give the bootstrap CIs.

Each slot brings its own ai_on_lag1 value from the original treatment schedule, so the lag structure is preserved within each bootstrap draw. The resampling captures estimation uncertainty without fabricating temporal relationships that didn't exist.

All three 95% CIs cover their respective ground truths. The naive ATE CI [0.0596, 0.0783] covers the true effect (0.060) but is shifted upward, consistent with the +0.009 positive bias. The adjusted ATE CI [0.0541, 0.0683] is centered closer to the true effect and is narrower. The carryover CI [0.0175, 0.0320] covers the true carryover of 0.030 and excludes zero, confirming that the carryover is statistically distinguishable from no persistence.

The excluded-zero result matters for the product decision: if the carryover CI included zero, you couldn't rule out that all the elevated AI-off outcomes were sampling noise rather than genuine persistence.

Validating Against the Ground Truth

Pull together the three point estimates against their known ground truths:

print("=" * 52)
print(f"{'Estimator':<30} {'Estimate':>8}  {'True':>6}  {'Bias':>7}")
print("-" * 52)
print(f"{'Naive OLS (no lag)':<30} {naive_ate:>8.4f}  {TRUE_EFFECT:>6.4f}  {naive_ate - TRUE_EFFECT:>+7.4f}")
print(f"{'Carryover-adjusted OLS':<30} {adj_ate:>8.4f}  {TRUE_EFFECT:>6.4f}  {adj_ate - TRUE_EFFECT:>+7.4f}")
print(f"{'Carryover coefficient':<30} {adj_carryover:>8.4f}  {CARRYOVER:>6.4f}  {adj_carryover - CARRYOVER:>+7.4f}")
print("=" * 52)

Expected output:

====================================================
Estimator                      Estimate    True     Bias
----------------------------------------------------
Naive OLS (no lag)               0.0688  0.0600  +0.0088
Carryover-adjusted OLS           0.0607  0.0600  +0.0007
Carryover coefficient            0.0244  0.0300  -0.0056
====================================================

The comparison table shows exactly what each estimator recovers against the known ground truth.

The naïve OLS overshoots by 0.0088 percentage points because it can't separate the direct AI routing effect from the carryover that inflates AI-on and adjacent AI-off slots. The adjusted OLS recovers the true effect to within 0.0007, well inside the width of any reasonable confidence interval. The carryover coefficient is 0.0244, compared with a true value of 0.030.

That's a systematic underestimate: the collinearity between ai_on and ai_on_lag1 in the 3-slot block structure produces this attenuation across all designs of this type.

The practical implication runs beyond this synthetic example. In a real LLM platform, carryover can be larger than the treatment effect. If the AI routing system fundamentally reshapes how the inference cluster allocates warm-cache slots across users, the next period will inherit a compute distribution shaped by AI routing, even after the routing AI is off.

Under those conditions, the naïve estimate could substantially overstate the effect you'd observe from a full always-on rollout, where no switching exists, and no carryover asymmetry accumulates.

Always estimate the carryover coefficient. If it's statistically significant and greater than 20% of your direct ATE estimate, the naïve estimate is unreliable for rollout decisions.

When Switchback Fails

Switchback solves marketplace interference under four conditions, and breaks under four others.

1. Carryover period longer than the slot length.

Violated assumption: (1) zero or bounded carryover.

If AI routing changes how the inference cluster pre-warms caches across multi-hour periods, the carryover half-life might exceed 60 or 90 minutes. A 30-minute slot length is shorter than the system's memory, and adding a single lag term won't capture the full persistence. You'll underestimate carryover and your direct effect estimate will remain biased.

The diagnostic: add progressively more lags and watch whether AIC keeps improving. If ai_on_lag3 and ai_on_lag4 still improve fit, your slot length is too short relative to system memory. Lengthening slots and adding more lag terms trade the same resource: fewer effective observations and wider confidence intervals.

2. Non-stationary demand confounding slots.

Violated assumption: (2) demand stationarity across the treatment schedule.

Weekday morning traffic surges, weekend evening spikes, and post-deployment adoption curves produce fundamentally different platform load conditions. If your treatment schedule places AI-on slots disproportionately in high-traffic windows and AI-off slots in low-traffic windows, the treatment coefficient absorbs demand differences as well as the routing AI's effect.

Randomizing the schedule within each day addresses this, as does including time-of-day fixed effects in the regression: a set of indicators for morning, afternoon, evening, and overnight absorbs within-day demand variation that would otherwise contaminate the treatment estimate.

3. Ramp-up effects at the first slot of each on-period.

Violated assumption: (3) no ramp-up at block boundaries.

In a real LLM platform, the first AI-on slot often underperforms subsequent slots. The routing model's cache is cold. The demand-prediction layer hasn't observed the current day's query distribution.

Including the cold-start slot alongside steady-state AI-on slots averages a low-performing initialization period with a high-performing equilibrium period, and the ATE estimate understates the steady-state effect you'd observe at full rollout. Standard practice is to drop the first slot of each on-period as a burn-in window and estimate the ATE from slots 2 and 3 of each block.

4. Period autocorrelation producing overconfident p-values.

Violated assumption: (4) residual autocorrelation addressed.

The Durbin-Watson diagnostic is a first check, but it only detects AR(1) autocorrelation. Real LLM platform time series often have daily seasonality, intraday autocorrelation at specific hours, and structural breaks after model version deployments.

Plot the full ACF of the model residuals: spikes at lags corresponding to meaningful demand cycles signal that your nlags parameter in cov_hac needs to increase, or you should switch to bootstrap CIs that don't assume any particular autocorrelation structure.

Failing to correct for autocorrelation is the most common source of false positives in switchback analyses at LLM platforms.

Two additional design-level failure modes are worth tracking.

Slot lengths under 15 minutes mean the platform hasn't cleared between switches: queue depth, in-flight session count, and cache state all carry over from the prior period, amplifying contamination and making AI-off periods non-representative of steady-state operations.

Slot lengths longer than 4 hours reduce the number of treatment-control pairs, shrinking the effective sample size and widening confidence intervals to the point where you can't detect plausible-sized effects.

The practical sweet spot for most LLM platform experiments is 30 minutes to 2 hours per slot, with final calibration determined by the carryover half-life estimated from early pilot data.

When to Use Switchback vs. Cluster Randomization

Switchback and cluster randomization solve the same interference problem through different mechanisms.

Cluster randomization partitions users into non-overlapping segments by geographic region, tenant ID, or organizational account, and assigns segments to treatment and control simultaneously. Switchback assigns the full population to treatment and control at different times.

Cluster randomization works well when you have enough separable segments and between-segment spillover is negligible. For an LLM SaaS platform with enterprise tenants on dedicated compute slices, cluster randomization by tenant is feasible: one tenant's routing decisions don't exhaust capacity for another's sessions.

For a consumer LLM platform where all users share the same inference fleet, capacity spillover crosses any user-segment boundary you draw, and cluster randomization can't isolate it.

Switchback is appropriate when spillover crosses segment boundaries or when you don't have enough separable clusters to run a properly powered cluster experiment.

Most large platforms use both: switchback for platform-wide infrastructure changes where no clean segment boundary exists, cluster randomization for features that can be scoped to a tenant or geographic region.

The choice comes down to where you can plausibly break the interference. Time is a natural boundary when the system clears faster than the slot length, so the platform fully processes the effects of one condition before switching to the next. Segment identity is a natural boundary when resource pools genuinely don't overlap. Where neither boundary holds, you're in causal estimation territory: synthetic control methods, difference-in-differences with matched controls, or structural models of the interference mechanism.

What to Do Next

If your switchback analysis shows a significant positive direct effect with a well-identified carryover term, the next hard question is whether the effect size justifies full rollout given the cost of the AI routing infrastructure. The premium model costs more per query than the standard model. Whether a 6 pp completion-rate lift covers that incremental inference cost depends on your product's monetization mechanics.

The carryover estimate shapes that decision too.

A large carryover coefficient means that some of the measured lift is dissipated once you switch to always-on routing, and the switching asymmetry disappears. The causal cost-benefit calculation requires the direct ATE, not the naïve estimate you'd get without the lag adjustment: revenue impact of the completion-rate gain, incremental inference cost at full traffic, and the confidence interval around each estimate before committing to an infrastructure investment.

If the routing AI shows heterogeneous effects across query types or user segments, the next analytical step is uplift modeling: building a model that predicts which queries benefit most from premium routing, so you route selectively and capture most of the task-completion gain at a fraction of the cost.

The causal identification work you've done here, including the switchback design, carryover adjustment, and HAC correction, gives you the unbiased population ATE you need as the ground-truth anchor for calibrating that uplift model.

The full companion code is at 06_switchback/, including the notebook with all five steps, the figure-generation scripts, and the dataset-generation code.

Four months later, a four-agent LangChain loop ran for eleven days and cost 47,000 USD. Nobody noticed until the invoice arrived. The pipeline worked correctly in testing, and the agents were doing exactly what they were told. Same pattern.

This tutorial is about that missing instruction.

You'll build five small Python primitives that catch most agent loop failures before they ship:

• A spec writer that forces you to define done before the loop starts

• A circuit breaker that kills the loop when it exceeds hard limits

• A ledger that records every turn in an append-only SQLite audit trail

• An agent loop that ties all three together

• A review surface that forces human attestation before downstream systems receive anything

By the end you'll have a working repo you can drop into any agent project. The full code is at github.com/dannwaneri/production-safe-agent-loop.

Table of Contents

1. Why This Keeps Happening

2. Prerequisites

3. Phase 1: Define Done Before You Build

4. Phase 2: Enforce Done at Runtime

5. Phase 3: Record Everything

6. Phase 4: The Loop That Respects Its Boundaries

7. Phase 5: The Review Surface

8. Phase 6: A Real Example, SEO Audit Agent

9. Pluggable LLM Client

10. Running the Tests

11. What You've Built

12. Next Steps

Why This Keeps Happening

The math that got companies into trouble was simple. A chatbot costs roughly 0.04 USD per interaction. An orchestrated multi-agent workflow costs 1.20 USD. That's a 30x multiplier — and production benchmarks show it can reach 70x on complex tasks.

The problem isn't that agents are expensive. The problem is that most teams budgeted for chatbot costs and deployed agent architectures. Gartner found the token consumption gap between pilot chatbots and production agent workflows sits at 5-30x. The FinOps Foundation's 2026 State of FinOps report found 73% of enterprises say AI costs exceeded original projections.

The mechanism is straightforward once you see it. When an agent fails a task and retries, it doesn't start fresh. It re-reads the entire context window — every prior failed attempt — before trying again. Iteration one costs 100 tokens. Iteration two costs 200. Iteration ten costs thousands. You're paying for every failure, over and over, in milliseconds.

# This is the entire problem in three lines
while True:
    result = agent.run(task)
    # done when...?

That question mark is where the money goes.

The other thing making it worse: agents don't fail loudly. Traditional code hits an undefined state and crashes. An LLM hits ambiguity and tries to be helpful. It retries. It reformats the tool call. It spins up a verification agent. The verification agent finds something. A correction agent fires. Nobody defined what "correct" means. The loop looks beautiful on every dashboard you have — activity, tool calls, completion rate — while quietly burning through your budget.

Gartner predicts that 40% of agentic projects will be scrapped by 2027 due to economic failure. Most of that failure is preventable. Not with better models, but with exit conditions.

Prerequisites

• Python 3.10+

• An Anthropic API key (or any provider — more on that later)

• Basic familiarity with Python classes and SQLite

git clone https://github.com/dannwaneri/production-safe-agent-loop
cd production-safe-agent-loop
pip install -r requirements.txt
export ANTHROPIC_API_KEY=sk-...

Phase 1: Define Done Before You Build

The most expensive mistake in agent development isn't a bad model choice or a missing retry limit. It's starting the build before you can answer one question in one sentence:

What does done look like?

Most teams can't answer it. Not because they're careless, but because nothing forces them to before they open the terminal. The spec writer is that forcing function.

# spec_writer.py
from spec_writer import SpecWriter

spec = SpecWriter(db_path="spec.db").run()

When you call .run(), it won't return until you've answered three questions:

1. What does this do?

2. What does this NOT do?

3. What does done look like in one sentence?

The third question is the one that matters. It's also the hardest. "The agent audits the site" is not an answer. "The agent crawls the target URL, extracts all <title> and <meta description> tags, flags any missing or over-length, and stops" is an answer. One of those gives the circuit breaker something to enforce.

The spec stores to SQLite and returns a SpecResult dataclass with a session_id. That ID becomes the thread connecting your spec, your ledger rows, and your loop result. One session, traceable end to end.

@dataclass(frozen=True)
class SpecResult:
    what_it_does: str
    what_it_does_not: str
    done_looks_like: str
    session_id: str

frozen=True matters. The spec is a commitment, not a draft. Once it's written, the loop runs against it. No mid-run revisions.

For testing, SpecWriter accepts injectable input_fn and output_fn callables. No stdin monkey-patching required. See tests/test_spec_writer.py for working examples — the suite uses a small scripted_input helper that returns answers from a generator, and writes to a per-test SQLite file via pytest's tmp_path fixture. SQLite's :memory: isn't safe here, because SpecWriter opens a fresh connection per method and each :memory: connection is its own isolated database.

Phase 2: Enforce Done at Runtime

Defining the exit condition upstream is discipline. The circuit breaker is enforcement.

# circuit_breaker.py
from circuit_breaker import CircuitBreaker, CircuitBreakerError

breaker = CircuitBreaker(turn_limit=5, token_limit=15000)
breaker.check(turn_count, accumulated_tokens)  # raises on breach

Two ceilings. Both hard.

turn_limit caps how many times the loop can call the LLM. token_limit caps total token consumption across all turns. Either one tripping raises CircuitBreakerError immediately.

The boundary is strict: turn_count == turn_limit is allowed. turn_count == turn_limit + 1 trips. No grace periods or warnings. A hard stop forces a human checkpoint.

from dataclasses import dataclass


@dataclass
class CircuitBreakerError(Exception):
    reason: str          # "turn_ceiling" or "token_ceiling"
    turn_count: int
    accumulated_tokens: int

    def __post_init__(self) -> None:
        super().__init__(
            f"circuit breaker tripped: {self.reason} "
            f"(turn={self.turn_count}, tokens={self.accumulated_tokens})"
        )


class CircuitBreaker:
    def __init__(self, turn_limit: int = 5, token_limit: int = 15000) -> None:
        self.turn_limit = turn_limit
        self.token_limit = token_limit

    def check(self, turn_count: int, accumulated_tokens: int) -> None:
        if turn_count > self.turn_limit:
            self._trip("turn_ceiling", turn_count, accumulated_tokens)
        if accumulated_tokens > self.token_limit:
            self._trip("token_ceiling", turn_count, accumulated_tokens)

    def _trip(self, reason: str, turn_count: int, accumulated_tokens: int) -> None:
        print(
            "\n=== CIRCUIT BREAKER CHECKPOINT ===\n"
            f"reason         : {reason}\n"
            f"turn_count     : {turn_count} / limit {self.turn_limit}\n"
            f"tokens_used    : {accumulated_tokens} / limit {self.token_limit}\n"
            "action         : halt loop, surface to human reviewer\n"
            "=================================="
        )
        raise CircuitBreakerError(
            reason=reason,
            turn_count=turn_count,
            accumulated_tokens=accumulated_tokens,
        )

CircuitBreakerError is an exception, not a return code. That's intentional. A return code can be ignored. An uncaught exception can't. Silent breach is impossible. The human-readable checkpoint banner is printed to stdout by _trip() before the exception is raised, so even if a caller swallows the exception the operator still sees state.

The critical rule: call .check() before every LLM call, not after. Post-flight checking means you've already burned the tokens before you knew the limit was exceeded.

# Wrong — post-flight
result = client.messages.create(...)
breaker.check(turn_count, accumulated_tokens)  # too late

# Right — pre-flight
breaker.check(turn_count, accumulated_tokens)  # raises before any spend
result = client.messages.create(...)

The defaults (5 turns, 15,000 tokens) match a tight tutorial demo. Your production budget is different. Tune at instantiation:

# Production example — tighter token budget, more turns
breaker = CircuitBreaker(turn_limit=10, token_limit=50000)

Phase 3: Record Everything

The circuit breaker protects your bank account. The ledger protects your understanding of what happened.

Most teams log for debugging — they want to know what went wrong after it went wrong. The ledger has a different purpose. It's governance. Every row is proof that the loop stayed within its boundaries, or didn't, and exactly when.

# ledger.py
from ledger import Ledger

ledger = Ledger(db_path="ledger.db")
ledger.write(
    session_id=spec.session_id,
    turn_count=1,
    state_origin="llm",
    input_str=task,
    token_delta=523,
    execution_time_ms=1240,
    pass_fail=True,
)

One row per turn. Append-only, no updates, and no deletes. The immutability is the point: a ledger you can edit isn't a ledger, it's a notebook.

The schema:

CREATE TABLE IF NOT EXISTS ledger (
    id                 INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id         TEXT    NOT NULL,
    turn_count         INTEGER NOT NULL,
    state_origin       TEXT    NOT NULL,
    input_hash         TEXT    NOT NULL,
    token_delta        INTEGER NOT NULL,
    execution_time_ms  INTEGER NOT NULL,
    pass_fail          INTEGER NOT NULL,  -- 1=pass, 0=fail
    breach_reason      TEXT,              -- NULL unless circuit breaker fired
    created_at         TEXT    NOT NULL   -- ISO 8601, UTC
);
CREATE INDEX IF NOT EXISTS idx_ledger_session ON ledger(session_id);

The index makes get_session(session_id) — the primary read path — a constant-time lookup as the ledger grows.

Three decisions worth explaining:

1. input_hash not input_text. The raw input string never persists. Only its SHA-256 hash does. There are two benefits to this: identical inputs across runs are detectable, and PII never enters the audit trail.

2. pass_fail as INTEGER not BOOLEAN. SQLite has no boolean type. 1 and 0 are canonical. Clean Python ergonomics at the API edge, correct SQL types on disk.

3. created_at as datetime.now(timezone.utc).isoformat(). datetime.utcnow() was deprecated in Python 3.12. Timezone-aware timestamps avoid the footgun in any system that crosses timezones.

Retrieve by session:

rows = ledger.get_session(spec.session_id)
for row in rows:
    print(f"Turn {row.turn_count}: {'PASS' if row.pass_fail else 'FAIL'} "
          f"| {row.token_delta} tokens | {row.execution_time_ms}ms")

Phase 4: The Loop That Respects Its Boundaries

The agent loop wires the three primitives together. It's the only component that calls the LLM. Everything else is local.

# agent_loop.py
from agent_loop import AgentLoop

loop = AgentLoop(spec, breaker, ledger, client)
result = loop.run(task)
# LoopResult(success, turns, total_tokens, session_id, breach_reason)

The anatomy of a turn, in order:

1. circuit_breaker.check(turn_count, accumulated_tokens) — raises if either ceiling is exceeded

2. client.messages.create(...) — the actual LLM call

3. ledger.write(...) — one row, append-only

4. If stop_reason == "end_turn", return. Otherwise loop.

Pre-flight checking before every LLM call, with no exceptions.

def run(self, task: str) -> LoopResult:
    session_id = self.spec.session_id
    messages: list[dict] = [{"role": "user", "content": task}]
    turn = 0
    total_tokens = 0

    try:
        while True:
            turn += 1
            self.circuit_breaker.check(turn, total_tokens)

            started = time.perf_counter()
            response = self.client.messages.create(
                model=self.model,
                max_tokens=self.max_tokens,
                system=self._system_prompt(),
                messages=messages,
            )
            elapsed_ms = int((time.perf_counter() - started) * 1000)

            turn_tokens = (
                getattr(response.usage, "input_tokens", 0)
                + getattr(response.usage, "output_tokens", 0)
            )
            total_tokens += turn_tokens

            text = self._text_from(response)
            messages.append({"role": "assistant", "content": text})

            self.ledger.write(
                session_id=session_id,
                turn_count=turn,
                state_origin="llm",
                input_str=task,
                token_delta=turn_tokens,
                execution_time_ms=elapsed_ms,
                pass_fail=True,
            )

            if getattr(response, "stop_reason", "end_turn") == "end_turn":
                return LoopResult(
                    success=True,
                    turns=turn,
                    total_tokens=total_tokens,
                    session_id=session_id,
                )

            messages.append({"role": "user", "content": "continue"})

    except CircuitBreakerError as err:
        self.ledger.write(
            session_id=session_id,
            turn_count=turn,
            state_origin="circuit_breaker",
            input_str=task,
            token_delta=0,
            execution_time_ms=0,
            pass_fail=False,
            breach_reason=err.reason,
        )
        return LoopResult(
            success=False,
            turns=turn,
            total_tokens=total_tokens,
            session_id=session_id,
            breach_reason=err.reason,
        )

def _system_prompt(self) -> str:
    return (
        "You are an agent working on a tightly-scoped task.\n\n"
        f"What this does: {self.spec.what_it_does}\n"
        f"What this does NOT do: {self.spec.what_it_does_not}\n"
        f"Done looks like: {self.spec.done_looks_like}\n"
    )

@staticmethod
def _text_from(response) -> str:
    content = getattr(response, "content", None)
    if not content:
        return ""
    block = content[0]
    return getattr(block, "text", "") or ""

A few choices worth calling out in this body:

• The whole while True: is wrapped in one try/except CircuitBreakerError. The check happens at the top of every turn, so a breach is caught the same way whether it fires on turn 1 or turn 6.

• input_str=task on every ledger row — the original task, not the last assistant message. The input_hash column then groups rows that share the same starting input across the run.

• pass_fail=True for every LLM turn that returns, False only on breach. The pass/fail flag tracks whether the loop reached the row legitimately, not whether the model's output was good. Quality scoring is a separate concern.

• _system_prompt() uses all three spec fields, not just done_looks_like. The model needs the negative scope (what_it_does_not) at least as much as the positive scope.

• time.perf_counter() not time.time() — monotonic, immune to wall-clock adjustments mid-run.

LoopResult.session_id is inherited from spec.session_id. The ledger rows tie back to the spec without a join. One session ID, one traceable run, start to finish.

Phase 5: The Review Surface

The circuit breaker protects your bank account. The ledger records what happened. But neither tells you whether what happened matched what you promised.

That gap is where bad loops get approved. Polished output, green dashboard, missed commitment. A reviewer sees the artifact, decides it looks acceptable, and signs off. Nobody asked whether the original promise was kept.

The review surface closes that gap. It reads the session from SQLite, assembles the five-element frame, and forces a comparison before anything downstream receives the output.

from review_surface import ReviewSurface

rs = ReviewSurface(spec_db_path="spec.db", ledger_db_path="ledger.db")
print(rs.render(session_id))

Here's the five-element frame, in order:

1. Original promise — pulled from the spec table: what it does, what it doesn't do, what done looks like

2. Acceptance criteria — the done_looks_like field rendered as the explicit benchmark

3. Diff — first turn input vs final turn output, turns completed, total tokens, whether the loop breached

4. Evidence — all ledger rows for the session: turn-by-turn pass/fail, token delta, execution time

5. Unresolved assumptions — derived from breach rows and failed turns. Empty when clean.

When the reviewer is satisfied, they attest:

attestation = rs.attest(
    session_id=result.session_id,
    reviewer="daniel",
    notes="Output matches spec. Approved."
)
print(attestation.frame_hash)

.attest() writes to the attestations table in ledger.db. The frame_hash is a SHA-256 of the canonical frame data — deterministic across reviewers attesting the same session. It's the audit receipt. It proves the reviewer saw the exact frame as rendered, not a summary or a paraphrase.

Approval confirms the process ran. Attestation confirms the reviewer compared output to commitment. When the loop touches something regulated, those are different legal documents.

@dataclass(frozen=True)
class ReviewFrame:
    session_id: str
    original_promise: SpecResult
    acceptance_criteria: str
    diff: DiffResult
    evidence: tuple  # tuple[LedgerRow, ...]
    unresolved_assumptions: tuple  # tuple[str, ...]
    created_at: str

ReviewFrame is frozen for the same reason SpecResult is — the frame is evidence, not a draft. evidence and unresolved_assumptions are tuples because lists aren't hashable and frozen dataclasses need hashable fields.

The full end-to-end flow with the review surface lives in examples/review_example.py in the repo. Run it after any completed session: it renders the five-element frame, prompts for attestation, and writes the receipt if you approve.

The loop runs to you. Downstream systems get nothing until someone signs.

Phase 6: A Real Example — SEO Audit Agent

The pattern only makes sense against a real problem. This is the same agent architecture behind my seo-agent project.

SEO audits have a natural cadence: crawl, surface what's broken, fix, wait for reindex. Running the agent continuously doesn't change that cadence. It just burns tokens in the empty space between the moments that matter. A cron job wired to the loop is the honest architecture.

# examples/seo_audit_example.py
import requests
from bs4 import BeautifulSoup
import anthropic
from spec_writer import SpecWriter
from circuit_breaker import CircuitBreaker
from ledger import Ledger
from agent_loop import AgentLoop

def crawl_url(url: str) -> str:
    response = requests.get(url, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")
    title = soup.find("title")
    meta_desc = soup.find("meta", attrs={"name": "description"})
    h1_tags = soup.find_all("h1")
    return (
        f"URL: {url}\n"
        f"Title: {title.text if title else 'MISSING'}\n"
        f"Meta description: "
        f"{meta_desc['content'] if meta_desc else 'MISSING'}\n"
        f"H1 count: {len(h1_tags)}\n"
        f"H1 tags: {[h.text[:50] for h in h1_tags]}"
    )

def run_seo_audit(url: str) -> None:
    # Step 1: Define done before the loop starts
    spec = SpecWriter(db_path="spec.db").run()

    # Step 2: Initialise circuit breaker and ledger
    breaker = CircuitBreaker(turn_limit=5, token_limit=15000)
    ledger = Ledger(db_path="ledger.db")
    client = anthropic.Anthropic()

    # Step 3: Crawl the URL
    site_data = crawl_url(url)

    # Step 4: Run the loop
    # AgentLoop catches CircuitBreakerError internally and returns
    # LoopResult(success=False, breach_reason=...). Branch on the
    # result — do NOT wrap loop.run() in try/except CircuitBreakerError.
    loop = AgentLoop(spec, breaker, ledger, client)
    result = loop.run(
        f"Audit this page for SEO issues:\n\n{site_data}"
    )

    # Step 5: Print the ledger
    print(f"\nResult: {'SUCCESS' if result.success else 'BREACH'}")
    if not result.success:
        print(f"Breach reason: {result.breach_reason}")
    print(f"Turns: {result.turns} | Tokens: {result.total_tokens}")
    print("\nAudit trail:")
    for row in ledger.get_session(result.session_id):
        status = "PASS" if row.pass_fail else "FAIL"
        print(f"  Turn {row.turn_count}: {status} | "
              f"{row.token_delta} tokens | {row.execution_time_ms}ms")

if __name__ == "__main__":
    import sys
    run_seo_audit(sys.argv[1] if len(sys.argv) > 1 else "https://example.com")

Run it:

python examples/seo_audit_example.py https://yourdomain.com

The spec writer prompts you. The loop runs, the circuit breaker fires if the limits are exceeded, and the ledger records every turn. The output lands in front of you and you decide what to fix.

The loop runs to you, not into a void.

Pluggable LLM Client

The loop works with any client that satisfies the LLMClient protocol (Anthropic by default). Bring your own via a ~20-line adapter.

# agent_loop.py
from typing import Protocol, runtime_checkable


@runtime_checkable
class MessagesEndpoint(Protocol):
    def create(self, *, model: str, max_tokens: int,
               system: str, messages: list) -> object: ...


@runtime_checkable
class LLMClient(Protocol):
    messages: MessagesEndpoint

messages is an instance attribute (not a nested class) because that's how the real Anthropic SDK exposes it — anthropic.Anthropic().messages.create(...). Modeling it as a nested class would mean the real client wouldn't satisfy the Protocol. The @runtime_checkable decorator lets you sanity-check conformance with isinstance(client, LLMClient), and the repo's test suite uses exactly that assertion against the FakeClient test double.

Here's an OpenAI adapter example (This is illustrative. A production adapter would also map streaming, tool-use, and error shapes.):

# openai_adapter.py — illustrative pseudocode, not production-ready.
from openai import OpenAI as _OpenAI


class _MessagesAdapter:
    def __init__(self, client):
        self._client = client

    def create(self, *, model, max_tokens, system, messages):
        completion = self._client.chat.completions.create(
            model=model,
            max_tokens=max_tokens,
            messages=[{"role": "system", "content": system}] + messages,
        )
        # Reshape OpenAI's response into the Anthropic-shaped surface
        # AgentLoop reads: response.usage.{input,output}_tokens,
        # response.content[0].text, response.stop_reason.
        return _adapt_response(completion)


class OpenAIAdapter:
    def __init__(self, api_key: str):
        self._client = _OpenAI(api_key=api_key)
        self.messages = _MessagesAdapter(self._client)  # instance attr, not a nested class

The adapter pattern is worth teaching explicitly. Provider APIs don't share a shape. Anthropic puts system at the top level. OpenAI puts it inside the messages array. An adapter shim is ~20 lines and makes the loop provider-agnostic without rewriting anything. Note that self.messages is assigned in __init__ so it's a real attribute on each adapter instance, the same shape as the actual SDK.

Running the Tests

python -m pytest tests/

With coverage:

python -m coverage run --source=circuit_breaker,ledger,spec_writer,agent_loop,review_surface -m pytest tests/
python -m coverage report -m

80 tests, 100% coverage on all five core modules. The loop is exercised against a FakeClient test double defined inline in tests/test_agent_loop.py. It satisfies the LLMClient protocol via duck typing: messages is set to self, so client.messages.create(...) routes back to the same object and ships with scripted responses for each test scenario. Clone the repo and run pytest to see all 80 tests pass without touching the network or needing an API key.

circuit_breaker.py has 100% coverage — no untested paths. It's the financial safety component. Every path through it is exercised.

What You've Built

In this tutorial, you've build five small primitives, each independently usable.

The pattern: upstream discipline defines the boundaries. Downstream enforcement breaks the circuit. Neither trusts the model to police itself.

A loop that runs without an exit condition isn't autonomous. It's a billing event waiting to happen.

Define what done looks like before you start. That's the job, and always has been.

Next Steps

The repo is at github.com/dannwaneri/production-safe-agent-loop.

There are three natural extensions if you want to go further:

1. Graduation to Distributed Systems

The SQLite ledger works for isolated sequential loops. The moment you run multiple agents against shared state, you need serializable isolation — concurrent writes to flat JSON corrupt silently. The README documents the three tipping points where a flat ledger needs to graduate.

2. Cryptographic Signing

For compliance-scale systems where the auditor wasn't present when the loop ran, SQLite rows aren't enough. A database admin can run an UPDATE query. Ed25519 signing wraps each ledger row in a receipt that proves the log wasn't altered after execution. But that's a different tutorial.

Wiring a Cron Job

The honest architecture for the SEO audit agent isn't 24/7 autonomous operation. It's a cron job that runs on schedule, surfaces what's broken, and stops. 0 3 * * 2 python examples/seo_audit_example.py https://yourdomain.com is the whole thing. The loop runs to you, not into a void.

If you need this architecture built for your own stack (circuit breakers, audit trails, production-safe agent loops), I do freelance work. dannwaneri.com/ai-agents/

Each new generation seemed to confirm a simple belief: bigger models lead to better capabilities. Yet there was one area where progress appeared frustratingly limited. When problems required multiple steps of reasoning, language models often struggled in ways that were difficult to ignore.

A math word problem, a common sense question, or a symbolic puzzle could expose a surprising gap between fluent language generation and genuine problem solving. Models could frequently produce confident answers, but confidence alone wasn't enough. The challenge was whether they could reason through a problem before arriving at an answer.

Against this backdrop, the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models introduced an idea that was both simple and unexpected. Rather than asking a model to produce an answer immediately, the authors encouraged it to work through intermediate reasoning steps first.

What followed was one of the most influential discoveries in modern AI research: many reasoning abilities that appeared absent in large language models weren't necessarily missing. In many cases, they simply hadn't been elicited in the right way.

This paper went on to reshape how researchers think about prompting, reasoning, and the capabilities of large language models. More importantly, it laid the intellectual foundation for many of the reasoning-oriented techniques and systems that emerged in the years that followed.

Paper Overview

In this article, we'll explore the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, published by researchers at Google Research in 2022.

This paper introduced one of the most influential ideas in modern AI: Chain-of-Thought (CoT) Prompting. At a time when researchers were focused on scaling language models to ever-larger sizes, this study revealed that performance improvements were not always about building bigger models. Sometimes, the key was changing how we communicate with them.

The paper investigates a simple but powerful question: what happens if a language model is encouraged to show its reasoning process before giving an answer? Instead of responding directly, the model is guided to generate intermediate reasoning steps that lead to the final solution.

What makes this paper historically important is that it changed how researchers think about reasoning in large language models. The authors demonstrated that many reasoning capabilities can be unlocked through prompting alone, without additional training, fine-tuning, or architectural modifications.

The impact of this idea quickly extended beyond arithmetic reasoning. It influenced a new generation of research on reasoning, including Self-Consistency, Process Supervision, Verification-based methods, and the reasoning-oriented models that followed in subsequent years.

In many ways, this paper marked a shift from asking language models what the answer is to asking them how they arrived at the answer.

Here's the original paper if you'd like to explore it directly:

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

And here's a quick infographic of what we'll cover throughout this review.

Table of Contents:

• Abstract

• Introduction

• Chain-of-Thought Prompting

• Arithmetic Reasoning

• Results

• Ablation Study

• Robustness of Chain-of-Thought Prompting

• Common Sense Reasoning

• Symbolic Reasoning

• Discussion

• Related Work

• Conclusion

• Resources

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with a few foundational ideas and the evolution of large language models that led to Chain-of-Thought prompting.

Reading the previous reviews in this series will be especially helpful:

• AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1)

• AI Paper Review: Language Models are Unsupervised Multitask Learners (GPT-2)

• AI Paper Review: Language Models are Few-Shot Learners (GPT-3)

• AI Paper Review: Training Language Models to Follow Instructions with Human Feedback (InstructGPT)

The GPT-3 review is particularly important because the Chain-of-Thought paper builds directly on one of GPT-3's most surprising capabilities: in-context learning. Rather than changing the model architecture or retraining the model, the authors discovered that reasoning performance could be dramatically improved simply by changing how examples were presented in the prompt.

It also helps to have:

• A general understanding of natural language processing (NLP) and large language models

• A basic understanding of Transformer-based autoregressive models

• Familiarity with prompting, few-shot learning, and in-context learning

• A high-level understanding of how language models generate text token by token

• General machine learning concepts such as training, inference, scaling laws, and model evaluation

• Some exposure to reasoning tasks, logic problems, and mathematical word problems

• A basic understanding of benchmark datasets and model performance evaluation

You don't need a deep background in mathematics or machine learning research to follow this article.

I'll keep the explanations intuitive and practical, focusing on why Chain-of-Thought prompting became one of the most influential reasoning techniques in modern AI and how a simple prompting strategy changed the way researchers think about language model reasoning.

Abstract

One of the long-standing challenges for large language models has been reasoning. While these models can generate fluent text and answer a wide variety of questions, they often struggle when a task requires multiple logical steps.

This paper introduces a remarkably simple idea to address that limitation: instead of prompting a model with only questions and answers, you should provide examples that also include the intermediate reasoning steps leading to the solution.

The authors call this approach Chain-of-Thought (CoT) Prompting. By showing a model a few demonstrations of step-by-step reasoning, they find that sufficiently large language models can generate their own reasoning chains and solve complex problems more effectively. Importantly, this improvement doesn't require additional training or fine-tuning, only a different style of prompting.

Through experiments on arithmetic, common sense, and symbolic reasoning tasks, the paper demonstrates that chain-of-thought prompting consistently improves performance. The gains become especially pronounced at larger model scales, suggesting that reasoning abilities emerge naturally as models grow and are given the right prompting strategy.

The paper's most striking result comes from the GSM8K math benchmark, where PaLM 540B, using only eight chain-of-thought examples, achieved state-of-the-art performance and even surpassed a fine-tuned GPT-3 system equipped with a verifier. This finding revealed that prompting alone could unlock reasoning capabilities that standard prompting often fails to expose.

The figure below compares standard prompting with Chain-of-Thought (CoT) prompting using a simple arithmetic example.

Source: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

In standard prompting, the model is shown question–answer pairs and is expected to produce an answer directly, which can lead to mistakes on multi-step problems.

In Chain-of-Thought prompting, the examples include intermediate reasoning steps before the final answer. When faced with a new problem, the model follows a similar step-by-step process, arriving at the correct solution.

This paper shows that providing reasoning demonstrations can substantially improve performance on arithmetic, common sense, and symbolic reasoning tasks, particularly in large language models.

Introduction

By 2022, large language models had already transformed natural language processing. Models such as GPT-3 demonstrated that scaling model size could unlock impressive capabilities, from text generation to few-shot learning.

But there was an important limitation: larger models weren't necessarily better at reasoning. Tasks that required multi-step arithmetic, common sense inference, or symbolic manipulation remained surprisingly difficult, even for some of the largest models available.

The authors begin by observing two promising research directions. The first comes from prior work showing that reasoning tasks can benefit from natural language explanations or intermediate solution steps. Instead of jumping directly to an answer, a model can generate a rationale that mirrors how a human might solve the problem.

The second direction is few-shot prompting, where a model learns a task from a handful of examples provided in the prompt, eliminating the need for task-specific fine-tuning.

Still, both approaches have drawbacks. Training models on large collections of human-written rationales is expensive and time-consuming, while standard few-shot prompting often struggles on tasks that require genuine reasoning.

The key insight of this paper was to combine the strengths of both ideas. Rather than providing only input-output examples, the prompt includes an additional component: the reasoning process itself. Each example follows the structure of input → chain of thought → output.

This simple modification led to Chain-of-Thought Prompting. By exposing intermediate reasoning steps, the model is encouraged to break complex problems into smaller, more manageable stages before arriving at a final answer.

To evaluate the idea, the authors tested chain-of-thought prompting across arithmetic, common sense, and symbolic reasoning benchmarks. The results showed substantial improvements over standard prompting, with some gains being remarkably large.

Chain-of-Thought Prompting

At the heart of this paper is a simple observation about how humans solve difficult problems. When faced with a multi-step reasoning task, we rarely jump directly to the answer. Instead, we break the problem into smaller pieces, solve each intermediate step, and gradually work toward a conclusion. The authors argued that large language models could benefit from a similar process.

This idea led to Chain-of-Thought (CoT) Prompting, where examples in the prompt included not only the question and answer, but also the reasoning steps connecting them. By seeing a few demonstrations of this reasoning process, sufficiently large language models learned to generate their own chains of thought before producing a final answer.

The significance of this approach extends beyond improving accuracy. First, it allows complex problems to be decomposed into manageable intermediate steps, making multi-step reasoning easier to perform.

Second, the generated reasoning process offers a degree of interpretability, giving researchers and users a glimpse into how the model arrived at its answer. While these reasoning traces don't fully reveal the model's internal computations, they can help identify where mistakes occur.

Another important aspect of chain-of-thought prompting is its generality. The authors proposed it not as a solution for a single benchmark, but as a broad reasoning framework that can be applied to arithmetic problems, common sense reasoning tasks, symbolic manipulation, and potentially many other challenges that require sequential reasoning.

Perhaps most importantly, this capability can be elicited from existing language models through prompting alone, without additional training or architectural modifications.

This section establishes the paper's central claim: reasoning abilities don't necessarily require new model architectures or specialized fine-tuning. In sufficiently large language models, these capabilities can emerge when the model is guided to generate intermediate reasoning steps rather than being asked to produce an answer immediately.

Arithmetic Reasoning

The authors begin their empirical evaluation with arithmetic reasoning, a domain that had long exposed a weakness of large language models.

Although solving math word problems is relatively straightforward for humans, it often requires a sequence of intermediate calculations and logical deductions.

Previous research had shown that even large language models struggled with these tasks, making arithmetic reasoning an ideal setting for testing whether chain-of-thought prompting could genuinely improve reasoning ability.

To evaluate their approach, the authors selected five established benchmarks covering a variety of math word problems. These datasets differ in style and difficulty, ranging from straightforward arithmetic questions to more complex problems that require multiple reasoning steps before arriving at a solution. Together, they provide a broad picture of how well language models handle mathematical reasoning.

The experiments compare two prompting strategies. The first is standard few-shot prompting, where the model is shown examples consisting only of questions and their corresponding answers. This was the dominant prompting approach at the time and serves as the baseline throughout the paper.

The second is chain-of-thought prompting, where each example is expanded to include the intermediate reasoning steps that connect the question to the final answer.

To ensure a fair comparison, the authors manually created a small set of eight reasoning demonstrations and reused them across the arithmetic benchmarks. Importantly, these examples weren't heavily optimized or engineered for specific datasets. Instead, they were intended to test whether a modest number of natural reasoning demonstrations could reliably encourage models to reason through new problems on their own.

The study also evaluates a diverse collection of language models, including GPT-3, LaMDA, PaLM, UL2, and Codex, spanning model sizes from hundreds of millions to hundreds of billions of parameters. This broad range allowed the authors to examine not only whether chain-of-thought prompting works, but also how its effectiveness changes as models become larger.

With this experimental framework in place, the paper investigated a central question: can providing a few examples of step-by-step reasoning enable large language models to solve mathematical problems that standard prompting struggles to handle?

Results

The arithmetic reasoning experiments revealed that the success of chain-of-thought prompting depends heavily on model scale.

One of the clearest patterns across the benchmarks was that smaller models gained little benefit from generating reasoning steps. In some cases, their performance even deteriorated because the models produced explanations that sounded plausible but were logically flawed.

The advantages of chain-of-thought prompting only became apparent once the models reached very large scales, suggesting that the ability to effectively use intermediate reasoning steps is itself an emergent capability.

Another important observation was that the benefits of chain-of-thought prompting grew as problems became more challenging. On simpler tasks that required only a single reasoning step, standard prompting was already sufficient and the additional reasoning process provided little value.

But as the complexity of the problems increased, the gap between standard prompting and chain-of-thought prompting widened substantially. The GSM8K benchmark provides the strongest example of this trend, where the largest GPT and PaLM models more than doubled their performance when allowed to reason step by step.

Perhaps the most significant result is that chain-of-thought prompting enabled large language models to compete with, and in some cases surpass, specialized systems trained directly for these tasks.

Using only a handful of reasoning demonstrations, PaLM 540B established new state-of-the-art results on several arithmetic benchmarks, despite relying solely on prompting rather than task-specific fine-tuning. This outcome challenged the prevailing assumption that strong performance on reasoning tasks necessarily required dedicated training datasets and specialized models.

To better understand these improvements, the authors manually inspected the reasoning traces generated by the models. When the model arrived at the correct answer, the reasoning process was usually correct as well, indicating that the model was often following a coherent sequence of logical steps rather than guessing the final answer.

Even among incorrect predictions, many reasoning chains were largely accurate and failed only because of small mistakes such as arithmetic slips, incorrect symbol mappings, or a missing intermediate step. More serious failures tended to arise from misunderstanding the problem itself or producing incoherent reasoning.

The error analysis also offered an explanation for why larger models benefited more from chain-of-thought prompting. Comparing PaLM 62B with PaLM 540B showed that increasing scale reduced many of the semantic misunderstandings and incomplete reasoning patterns that appeared in smaller models.

In other words, larger models were not merely generating longer explanations. They were producing reasoning chains that were more logically complete and more faithful to the underlying problem.

Ablation Study

Before diving into this section, it's worth briefly explaining what an ablation study is. In machine learning research, an ablation study systematically removes or modifies parts of a method to determine which components are actually responsible for its performance. Rather than asking whether a method works, an ablation study asks why it works.

In this paper, the authors use ablation experiments to identify which aspects of Chain-of-Thought prompting contribute most to its reasoning improvements.

After demonstrating that chain-of-thought prompting improved reasoning performance, the authors turned to a more fundamental question: why does it work? Simply observing higher accuracy isn't enough. To understand the source of these gains, they designed a series of ablation experiments that isolated different aspects of the prompting strategy.

One possible explanation is that chain-of-thought prompting helps because it encourages the model to generate mathematical equations before producing an answer. If this were true, then the natural language reasoning itself might not be necessary.

To test this idea, the authors replaced the reasoning steps with equations alone. The results showed that this approach provides only limited benefits on complex benchmarks such as GSM8K. While equations can help with simpler problems, they are often insufficient for tasks that require understanding the meaning of the question before translating it into mathematical operations. This suggests that the value of chain-of-thought prompting comes from more than symbolic calculation.

The authors then examined another hypothesis: perhaps chain-of-thought prompting succeeds simply because it allows the model to generate more tokens and therefore spend more computation on difficult problems.

To isolate this factor, they created a prompt that produces additional tokens without any meaningful reasoning content. Performance remained close to the standard prompting baseline, indicating that extra computation alone doesn't explain the observed improvements. What mattered wasn't the number of intermediate tokens, but the reasoning expressed within them.

A third possibility was that chain-of-thought prompts merely activated relevant knowledge already stored in the model. If that were the case, the reasoning steps wouldn't need to appear before the answer.

The authors tested this by moving the reasoning process to after the final answer. Once again, performance largely fell back to the baseline. This result suggested that the sequence of reasoning steps plays an active role in helping the model arrive at the correct solution rather than simply serving as an explanation after the fact.

Taken together, these experiments strengthen the paper's central argument. The success of chain-of-thought prompting can't be explained by equation generation, additional computation, or easier access to stored knowledge alone.

Instead, the evidence points toward the reasoning process itself as the critical ingredient. The intermediate steps aren't merely decorative explanations. They appear to guide the model through a sequence of decisions that makes complex problem solving more effective.

Robustness of Chain-of-Thought Prompting

One of the long-standing concerns with prompting methods is their sensitivity to the examples included in the prompt. Small changes in wording, example selection, or even the order of examples can sometimes produce noticeably different results.

Once they established that chain-of-thought prompting improves reasoning performance, the authors investigated whether these gains were robust or whether they depended on a particular set of carefully crafted demonstrations.

To answer this question, the researchers asked multiple authors of the paper to independently write reasoning traces for the same examples. They also experimented with a more concise writing style and tested prompts built from entirely different sets of examples.

The goal was to determine whether chain-of-thought prompting was succeeding because of a specific wording choice or because the underlying reasoning structure was genuinely useful.

The results provided reassuring evidence that the technique isn't tied to a particular author, writing style, or collection of exemplars. While some variation in performance naturally appeared across different prompts, every version of chain-of-thought prompting consistently outperformed standard prompting by a substantial margin. Whether the reasoning steps were detailed or concise, manually written or drawn from an independent dataset, the overall pattern remained remarkably stable.

The authors further broadened their analysis by varying the order and number of exemplars used in the prompt. Once again, the central finding persisted: although prompt design still influenced performance to some degree, the effectiveness of chain-of-thought prompting didn't depend on a single carefully engineered prompt.

This robustness analysis strengthens one of the paper's most important claims that the success of chain-of-thought prompting isn't an artifact of a particular phrasing or annotation style. Instead, the benefits appear to arise from exposing the model to a reasoning process itself, suggesting that the method captures a more general principle rather than a prompt-specific trick.

Common Sense Reasoning

Up to this point, the paper focused primarily on mathematical reasoning. While the results are impressive, they leave an important question unanswered: is chain-of-thought prompting useful only for arithmetic problems, or can it improve reasoning more broadly?

To investigate this, the authors turned to common sense reasoning tasks. Unlike math problems, these tasks often require background knowledge about the world, an understanding of human behavior, or the ability to connect multiple pieces of information before arriving at a conclusion. In many cases, the challenge isn't performing calculations but reasoning through situations that humans find intuitive.

The evaluation spanned a diverse collection of benchmarks, including common sense question answering, multi-hop reasoning, date understanding, sports-related reasoning, and even tasks that involved converting natural language instructions into robot actions.

Despite their differences, these tasks share a common requirement: solving them often involves a sequence of intermediate inferences rather than an immediate answer.

The results showed that the benefits of chain-of-thought prompting extend well beyond mathematics. Across most benchmarks, models consistently performed better when encouraged to generate intermediate reasoning steps before producing a final answer.

The improvements became particularly noticeable for larger models, suggesting that the same pattern observed in arithmetic reasoning also applies to common sense reasoning.

Some of the strongest gains appeared on tasks that required multi-step inference. On StrategyQA, for example, chain-of-thought prompting enabled PaLM 540B to surpass the previous state of the art. Similarly, on the Sports Understanding benchmark, the model achieved performance that exceeded that of an unaided human sports enthusiast.

These results suggest that the reasoning process encouraged by chain-of-thought prompting can help models connect facts, evaluate plausibility, and navigate more complex decision-making scenarios.

At the same time, the improvements weren't uniform across every dataset. The gains on CommonsenseQA were relatively modest, indicating that not all reasoning tasks benefit equally from explicit reasoning traces. This serves as an early reminder that chain-of-thought prompting isn't a universal solution, even though it consistently proves valuable across a wide range of settings.

More broadly, this section strengthens the paper's central argument by showing that chain-of-thought prompting isn't merely a technique for solving math word problems. Its effectiveness across diverse common sense tasks suggests that the method taps into a more general reasoning capability that emerges in sufficiently large language models.

Symbolic Reasoning

The final evaluation moves away from mathematics and real-world knowledge altogether. Instead, the authors focus on symbolic reasoning tasks, where success depends on following abstract rules rather than recalling facts or performing calculations. These tasks are simple for humans, yet they provide a useful way to test whether language models can consistently apply a sequence of reasoning steps.

To explore this question, the authors designed two controlled tasks. The first required the model to extract and concatenate the last letters of words in a name. The second asked the model to track the state of a coin after a sequence of flips and non-flips.

Although these tasks may appear simple, they required the model to perform precise symbolic manipulations without relying on memorized knowledge about the world.

What made these experiments particularly interesting was the introduction of an out-of-distribution setting. During prompting, the model only saw examples involving short reasoning chains. At evaluation time, it was asked to solve versions of the same tasks that required more steps than any example it had previously encountered.

This setup allowed the authors to test not only whether the model could follow a reasoning procedure, but also whether it could extend that procedure to longer and unfamiliar cases.

The results revealed a familiar pattern. Large models benefitted substantially from chain-of-thought prompting, while smaller models struggled even when the required reasoning process was straightforward.

On the in-domain tasks, where the evaluation closely matched the examples provided in the prompt, the largest models achieved near-perfect performance when guided by chain-of-thought reasoning. This indicated that they could successfully learn and apply the underlying procedure demonstrated in the prompt.

The more revealing results come from the out-of-distribution evaluations. Standard prompting largely fails when the reasoning chain becomes longer than those seen in the examples. In contrast, chain-of-thought prompting enabled performance to improve as model size increased, demonstrating an ability to extend learned reasoning patterns beyond the exact situations shown during prompting.

Although accuracy declines compared to the in-domain setting, the models were still able to generalize in ways that standard prompting couldn't.

This section provided some of the strongest evidence that chain-of-thought prompting is doing more than improving benchmark performance. By helping models apply reasoning procedures to longer and previously unseen inputs, it suggests that the generated reasoning steps serve as a scaffold for systematic problem solving rather than merely a mechanism for producing better answers on familiar examples.

Discussion

The most important contribution of this paper wasn't a new model architecture, a new training objective, or a larger dataset. Instead, it demonstrated that a simple change in prompting could unlock capabilities that standard prompting often failed to reveal.

Across arithmetic, common sense, and symbolic reasoning tasks, chain-of-thought prompting consistently allowed large language models to solve problems that were previously difficult or inaccessible.

A recurring theme throughout the paper was the relationship between reasoning and scale. The authors repeatedly observed that chain-of-thought prompting became effective only once models reached a sufficient size. Smaller models generated fluent reasoning traces, but those traces were often logically inconsistent.

Larger models, in contrast, were able to use intermediate reasoning steps in a way that genuinely improved problem-solving performance.

This finding reinforced a broader lesson emerging from language model research at the time: some capabilities don't appear gradually, but emerge once a model crosses a certain scale threshold.

Perhaps the most intriguing implication was that standard prompting may significantly underestimate what large language models are capable of doing.

Before this work, many reasoning tasks appeared to have reached a performance ceiling. Chain-of-thought prompting revealed that the limitation wasn't always the model itself, but sometimes the way the model was being asked to solve the problem. In that sense, the paper shifted attention from building more capable models to discovering better ways of interacting with the capabilities that already exist within them.

At the same time, the authors were careful not to overstate their conclusions. Although chain-of-thought outputs can resemble human reasoning, the paper doesn't prove that language models reason in the same way humans do. The generated reasoning traces may reflect genuine problem-solving processes, post-hoc rationalizations, or something in between. Determining the relationship between generated reasoning and internal model computation remains an open research question.

The authors also acknowledged several practical limitations. Constructing high-quality reasoning demonstrations can require additional effort, particularly if the approach is extended beyond few-shot prompting.

Also, generating a chain of thought doesn't guarantee that the reasoning itself is correct. Models can still produce convincing but flawed reasoning paths, leading to incorrect answers.

Finally, the strongest benefits appear only in very large models, raising questions about computational cost and whether similar reasoning abilities can be induced in smaller systems.

Viewed from a historical perspective, this paper marked a turning point in research on language model reasoning. Rather than treating reasoning as something that must be explicitly trained into a model, it suggested that reasoning abilities could be elicited through the right prompting strategy.

Many influential ideas that followed, including self-consistency, reasoning supervision, process supervision, and the reasoning-focused models that emerged in later years, can trace part of their intellectual foundation back to the simple insight introduced here: sometimes a model performs better when it's encouraged to show its work.

Related Work

The ideas behind Chain-of-Thought prompting didn't emerge in isolation. Instead, the paper sits at the intersection of two research directions that had been evolving independently for several years.

The first direction focused on helping models solve complex problems through intermediate reasoning steps. Earlier work had already shown that tasks such as mathematical reasoning become easier when a model generates natural language rationales rather than producing an answer directly. Researchers explored methods that trained models to generate explanations, reasoning traces, or intermediate computations before arriving at a final solution.

Other approaches relied on formal symbolic representations, translating problems into structured equations or logical forms. Despite their differences, these efforts shared a common intuition: difficult reasoning tasks are often easier to solve when they're decomposed into smaller steps.

Chain-of-thought prompting inherits this intuition but introduces an important shift. Earlier methods typically required dedicated training procedures, specialized datasets, or task-specific fine-tuning.

In contrast, this paper demonstrated that reasoning traces could be elicited through prompting alone. Rather than teaching a model to reason through additional training, the authors showed that providing a handful of reasoning examples may be enough to unlock capabilities that already exist within sufficiently large language models.

The second research direction concerns prompting itself. Following the success of GPT-3 and few-shot learning, a growing body of work explored how prompts could be used to improve model performance without retraining.

Researchers experimented with prompt engineering, prompt tuning, and natural language instructions to better communicate tasks to language models. Most of these techniques focused on improving the input side of the interaction by changing how a task was described to the model.

Chain-of-thought prompting takes a different approach. Instead of modifying the instructions that precede a task, it augments the examples that follow them by exposing the reasoning process that connects inputs and outputs. This distinction may seem subtle, but it represents one of the paper's key insights: the contribution goes beyond a better prompt template. It focuses on the realization that demonstrating how to reason can be just as important as describing what task should be solved.

Viewed in this broader context, the paper acts as a bridge between research on reasoning traces and research on prompting. It combines the strengths of both traditions and, in doing so, lays the foundation for many later advances in language model reasoning, including self-consistency, STaR, process supervision, and the reasoning-oriented systems that followed in subsequent years.

Conclusion

Chain-of-Thought Prompting introduced a simple idea that changed how researchers think about reasoning in large language models. Rather than modifying model architectures or relying on additional training, the authors showed that reasoning abilities could often be unlocked by encouraging models to generate intermediate reasoning steps before producing an answer.

Across arithmetic, common sense, and symbolic reasoning tasks, the results demonstrated that large language models become significantly more capable when allowed to work through a problem step by step. More importantly, the paper revealed that many of these improvements emerge at larger scales, suggesting that reasoning isn't simply a product of prompting but a capability that becomes increasingly accessible as models grow more powerful.

What made this work particularly influential wasn't the complexity of the method, but the insight behind it. A model may possess the knowledge required to solve a problem, yet still fail to use that knowledge effectively when asked for an immediate answer. By exposing the reasoning process, Chain-of-Thought prompting showed that how a model arrives at an answer can be just as important as the answer itself.

This idea helped shift the focus of AI research beyond what language models know toward how they reason, plan, and solve problems. Many of the techniques that followed (including Self-Consistency, process supervision, verification-based methods, and modern reasoning-focused systems) build upon the foundation established by this paper.

Viewed in retrospect, Chain-of-Thought Prompting was more than a prompting technique. It marked a turning point in the study of language model reasoning, demonstrating that some capabilities aren't absent from a model but simply require the right conditions to emerge.

The infographic below highlights some of the most influential papers and milestones that shaped modern AI, from the introduction of GPT-1 and the scaling era of GPT-2 and GPT-3, to instruction tuning, Chain-of-Thought reasoning, Self-Consistency, process supervision, and the latest generation of reasoning-focused models. Together, these works reveal how the field evolved from teaching models to predict language toward helping them reason, verify, and solve increasingly complex problems.

Resources

• Pytorch Projects for GPT series

• Attention Is All You Need

• Improving Language Understanding by Generative Pre-Training (GPT-1)

• Language Models are Unsupervised Multitask Learners (GPT-2)

• Language Models are Few-Shot Learners (GPT-3)

• Scaling Laws for Neural Language Models

• Training Language Models to Follow Instructions with Human Feedback (InstructGPT)

• Finetuned Language Models are Zero-Shot Learners (FLAN)

• LaMDA: Language Models for Dialog Applications

• PaLM: Scaling Language Modeling with Pathways

• Program Induction by Rationale Generation: Learning to Solve and Explain Algebra Word Problems

• Training Verifiers to Solve Math Word Problems

• Show Your Work: Scratchpads for Intermediate Computation with Language Models

• Self-Consistency Improves Chain of Thought Reasoning in Language Models

• STaR: Bootstrapping Reasoning with Reasoning

• Emergent Abilities of Large Language Models

• Sparks of Artificial General Intelligence: Early Experiments with GPT-4

• Let's Verify Step by Step

• Learning to Reason with LLMs

• GPT-4 Technical Report

Contact Me

• Github

• X

• Linkedin

Naturally, my first instinct was to look at cloud providers. While services like ElevenLabs offer incredible voice quality, I quickly ran the numbers. Between the API pricing, token consumption for lengthy tutoring sessions, and the sheer volume of users I anticipated, the math got ugly very quickly. Relying on a paid API for every single sentence spoken within the app simply wasn't sustainable for an independent developer.

If you’re about to ask, "How far did you get with QuizRope?", well honestly, I straight-up gave up on the project back then because I couldn't find a sane, affordable solution for the TTS feature.

Beyond the prohibitive cost, there was the latency. Waiting for a server to process a prompt, generate the audio, and stream it back down to a mobile device completely breaks the conversational illusion. And worst of all, it meant every question a student asked would be beamed to a third-party server.

That frustration became the catalyst for my search to find a reliable, offline, and completely zero-cost solution.

In this article, we’re going to build a React Native application that performs high-fidelity Text-to-Speech (TTS) completely offline using your device's own hardware.

If you haven't set up your environment or need a refresher on local inference fundamentals, I highly recommend reading my previous article, How to Run a Local LLM Offline in React Native with QVAC, where I cover project initialization, prebuilding, and native hardware dependencies.

This guide assumes you already have a project with the QVAC SDK configured and ready to run on a physical device.

Table of Contents

• Prerequisites

• What is QVAC?

• The Architecture Supported by QVAC

• The Inference Pipeline

• Environment and Dependency Config

• The Audio Utility Packaging

• Complete Implementation

• Codebase Breakdown

• Conclusion

• Resources and Further Reading

Prerequisites

To get the most out of this article, you should have a solid foundation in modern web and mobile development:

• JavaScript/TypeScript & React: Familiarity with React concepts and hooks, especially useState, useEffect, and useRef.

• React Native & Expo: Basic understanding of layout structures (such as View, ScrollView, TextInput) and styling conventions.

• Asynchronous JavaScript & Binary Buffers: Experience with async/await, Promises, and basic manipulation of arrays like Int16Array or Buffer.

• Development Build Environment: Familiarity with running local development compilation commands, specifically npx expo prebuild to build native iOS and Android modules.

• Physical Mobile Device: Because local machine learning models leverage device-specific hardware acceleration and native optimizations, the QVAC SDK doesn't support simulator environments. You must have a physical iOS or Android testing device with Developer Mode enabled.

What is QVAC?

To help you follow along more effectively, let’s establish what QVAC is and why it exists.

Developed by Tether, QVAC is a local-first AI SDK designed for building cross-platform, peer-to-peer (P2P) applications and systems.

Many mobile applications that utilize Large Language Models (LLMs) or Text-to-Speech (TTS) engines rely on network requests to cloud-hosted APIs (such as OpenAI or ElevenLabs). While convenient, this model introduces dependencies on network connectivity, recurring API usage fees, and transmission of user data to third-party servers.

QVAC provides an alternative by executing AI models directly on the client device. This local-first architecture offers several practical advantages:

• Local-first execution: Runs inference directly on the client hardware, eliminating the need for external APIs or active internet connections.

• Peer-to-peer (P2P) support: Allows distributing inference tasks across local networks, helping coordinate workloads without centralized servers.

• Cross-platform compatibility: Provides a single JavaScript/TypeScript interface that works consistently across different hardware and runtime environments.

• Unified capabilities: Exposes text generation, transcription, image generation, and speech synthesis within a single package.

Key Concepts for On-Device Inference

To understand how QVAC runs on a mobile device, we must keep a few key concepts in mind:

• On-Device Inference: Running model calculations locally. Rather than relying on a single engine, QVAC supports multiple specialized local inference backends depending on the task (such as llama.cpp for text, whisper.cpp for transcription, or custom diffusion backends for image generation). Under the hood, these engines memory-map quantized model weights directly into the device's RAM and run calculations using native GPU hardware acceleration.

• Quantization (GGUF format): A mathematical optimization technique that compresses the model's weights (for example, from a standard 16-bit floating-point precision down to 4-bit or 8-bit integers). This makes it possible for models to fit into the memory constraints of consumer mobile hardware while keeping output quality high.

• KV (Key-Value) Cache: A memory area that stores calculated states of previous tokens so the model doesn't have to re-evaluate the entire context window with every word or token it generates.

The Architecture Supported by QVAC

Before writing code, it's crucial to understand what's actually happening under the hood. To handle local execution without melting your device, the QVAC SDK manages the hardware binding and model lifecycle while hooking into optimized, community-maintained GGML inference backends.

Instead of a one-size-fits-all approach, the QVAC SDK supports two distinctly different neural architectures for speech synthesis. Depending on your application's needs — whether you want instant voice cloning or ultra-high-fidelity pre-trained voices — you'll choose between Chatterbox and Supertonic.

1. The Chatterbox Engine

Chatterbox is built on a transformer-based language model architecture. It treats audio generation similarly to how an LLM predicts the next word in a sentence, but instead, it predicts discrete acoustic tokens.

Because of this architecture, Chatterbox excels at zero-shot voice cloning. Instead of relying purely on pre-baked voices, you can pass an optional referenceAudioSrc (a short WAV file of someone speaking) alongside your text. The transformer analyzes the reference audio's acoustic properties and generates a cloned voice based on those features.

2. The Supertonic Engine

Supertonic takes a completely different approach, utilizing diffusion-based latent denoising — the same fundamental architecture used by AI image generators like Stable Diffusion, but applied to audio.

It starts with pure digital noise and iteratively refines it into a 44.1 kHz high-fidelity speech waveform based on the text prompt. Supertonic uses a single, unified GGUF file rather than a split model. Instead of dynamic voice cloning, it relies on highly optimized, pre-trained voice styles (for example, voice: "F1" or voice: "M1") baked directly into the model. This makes it incredibly efficient for generating crystal-clear, studio-quality speech when you don't need dynamic cloning capabilities.

For this tutorial, we'll use Supertonic. It yields fantastic results out of the box and avoids the complexity of loading multiple companion files.

The Inference Pipeline

To visualize how we interact with these engines in our codebase, think of local TTS (Text to Speech) as running a virtual recording studio right in your phone's memory:

1. Hiring the actor (loading the model): We map the compressed GGUF file directly into the device's RAM or GPU VRAM.

2. Handing over the script (text input): We pass plain text to the loaded engine.

3. The performance (inference): The engine reads the text and mathematically predicts the sound waves. Crucially, the AI doesn't emit a finished audio file. Instead, it outputs raw digital sound waves known as PCM samples.

4. Packaging the audio: Because a raw list of numbers can't be played by standard media players, we must manually wrap the PCM data in a standard WAV header.

5. Closing the studio (unloading): Because speech synthesis is memory-intensive and maintains a persistent state, the model is cleared from RAM to free up resources and flush its context.

Environment and Dependency Config

Before we jump into the codebase, there's a crucial dependency setup to keep in mind if your project uses the pnpm package manager.

Because QVAC plugins rely on transitive native peer dependencies, strict package managers like pnpm will lock these dependencies down inside hidden .pnpm subfolders.

To ensure the QVAC native bundler (bare-pack) can resolve your worker plugins correctly at build time, create a .npmrc file in the root of your project:

shamefully-hoist=true

IMPORTANT: After creating this file, you must run a clean dependency install (pnpm install). This ensures a flat layout in your root node_modules so that all QVAC-specific helper packages are resolved properly during your local npx expo prebuild compilation step.

The Audio Utility Packaging

Because QVAC outputs raw PCM arrays, we need to construct a valid WAV file in memory and write it to the device's storage before the native audio player can play it.

To achieve this, let's create a utility module inside src/lib/utils.ts to build the required WAV header, convert raw audio samples into a binary buffer, and write it to local storage.

import { Buffer } from "buffer";
import * as FileSystem from "expo-file-system/legacy";

/**
 * Creates a WAV header for 16-bit PCM audio
 */
export function createWavHeader(
  dataLength: number,
  sampleRate: number,
): Buffer {
  const buffer = Buffer.alloc(44);
  const channels = 1; // Mono
  const byteRate = sampleRate * channels * 2; // 16-bit audio
  const blockAlign = channels * 2;

  buffer.write("RIFF", 0);
  buffer.writeUInt32LE(36 + dataLength, 4);
  buffer.write("WAVE", 8);
  buffer.write("fmt ", 12);
  buffer.writeUInt32LE(16, 16); // Subchunk1Size
  buffer.writeUInt16LE(1, 20); // AudioFormat (PCM)
  buffer.writeUInt16LE(channels, 22);
  buffer.writeUInt32LE(sampleRate, 24);
  buffer.writeUInt32LE(byteRate, 28);
  buffer.writeUInt16LE(blockAlign, 32);
  buffer.writeUInt16LE(16, 34); // BitsPerSample
  buffer.write("data", 36);
  buffer.writeUInt32LE(dataLength, 40);

  return buffer;
}

/**
 * Converts the raw Int16Array samples from QVAC to a binary Buffer
 */
export function int16ArrayToBuffer(int16Array: Int16Array): Buffer {
  const buffer = Buffer.alloc(int16Array.length * 2);
  for (let i = 0; i < int16Array.length; i++) {
    buffer.writeInt16LE(int16Array[i] ?? 0, i * 2);
  }
  return buffer;
}

/**
 * Main function to package and save the file to local mobile storage
 */
export async function saveAudioToDevice(
  audioBuffer: Int16Array,
  sampleRate: number,
): Promise<string> {
  try {
    const audioData = int16ArrayToBuffer(audioBuffer);
    const wavHeader = createWavHeader(audioData.length, sampleRate);
    const finalWavBuffer = Buffer.concat([wavHeader, audioData]);
    const base64Data = finalWavBuffer.toString("base64");

    const filename = `tts-speech-${Date.now()}.wav`;
    const fileUri = `\({FileSystem.documentDirectory}\){filename}`;

    await FileSystem.writeAsStringAsync(fileUri, base64Data, {
      encoding: FileSystem.EncodingType.Base64,
    });

    console.log(`✅ File saved locally at: ${fileUri}`);
    return fileUri;
  } catch (error) {
    console.error("❌ Failed to save audio file locally:", error);
    throw error;
  }
}

Complete Implementation

Let's bring it all together. We'll implement an interface that takes user input, manages download and loading states for the Supertonic engine, packages generated raw waves into a playable local file, and renders an interactive visual waveform player.

Replace your entry app file src/app/index.tsx with the following implementation:

import { useState, useEffect } from "react";
import {
  TextInput,
  KeyboardAvoidingView,
  Platform,
  ScrollView,
} from "react-native";
import {
  loadModel,
  unloadModel,
  textToSpeech,
  downloadAsset,
  TTS_EN_SUPERTONIC_Q8_0,
  getModelInfo,
  type ModelProgressUpdate,
} from "@qvac/sdk";
import { saveAudioToDevice } from "@/lib/utils";
import { TtsModelLoader } from "@/components/tts-model-loader";
import { AudioPlayer } from "@/components/audio-player";
import {
  Card,
  CardContent,
  CardDescription,
  CardHeader,
  CardTitle,
} from "@/components/ui/card";
import { Button } from "@/components/ui/button";
import { Text } from "@/components/ui/text";

const SUPERTONIC_SAMPLE_RATE = 44100;

// Global reference for our model ID
let globalModelId: string | null = null;

type TtsStatus =
  | { phase: "idle" }
  | { phase: "synthesizing" }
  | { phase: "done"; audioUri: string }
  | { phase: "error"; message: string };

export default function TextToVoiceScreen() {
  const [text, setText] = useState("");
  const [status, setStatus] = useState<TtsStatus>({ phase: "idle" });

  const [isModelLoaded, setIsModelLoaded] = useState(!!globalModelId);
  const [isDownloading, setIsDownloading] = useState(false);
  const [downloadProgress, setDownloadProgress] = useState(0);

  const isBusy = status.phase === "synthesizing";

  useEffect(() => {
    async function checkAndAutoLoad() {
      if (globalModelId) return;
      try {
        const info = await getModelInfo({ name: TTS_EN_SUPERTONIC_Q8_0.name });
        if (info.isCached) {
          setIsDownloading(true);
          setDownloadProgress(1);

          globalModelId = await loadModel({
            modelSrc: TTS_EN_SUPERTONIC_Q8_0,
            modelConfig: {
              ttsEngine: "supertonic",
              language: "en",
              voice: "F1",
              ttsSpeed: 1.05,
              ttsNumInferenceSteps: 5,
            },
          });

          setIsModelLoaded(true);
          setIsDownloading(false);
        }
      } catch (err: unknown) {
        console.warn("Failed to auto-load cached model on mount:", err);
        setIsDownloading(false);
      }
    }
    checkAndAutoLoad();
  }, []);

  const handleDownloadModel = async () => {
    if (isDownloading || isModelLoaded) return;

    try {
      setIsDownloading(true);
      setDownloadProgress(0);

      await downloadAsset({
        assetSrc: TTS_EN_SUPERTONIC_Q8_0,
        onProgress: (p: ModelProgressUpdate) => {
          setDownloadProgress(p.percentage / 100);
        },
      });

      setDownloadProgress(1);

      globalModelId = await loadModel({
        modelSrc: TTS_EN_SUPERTONIC_Q8_0,
        modelConfig: {
          ttsEngine: "supertonic",
          language: "en",
          voice: "F1",
          ttsSpeed: 1.05,
          ttsNumInferenceSteps: 5,
        },
      });

      setIsModelLoaded(true);
      setIsDownloading(false);
    } catch (err: unknown) {
      console.error("Failed to download or load model:", err);
      setIsDownloading(false);
      setStatus({
        phase: "error",
        message: err instanceof Error ? err.message : String(err),
      });
      setIsModelLoaded(false);
    }
  };

  const handleSubmit = async () => {
    if (!text.trim() || isBusy || !globalModelId) return;

    try {
      setStatus({ phase: "synthesizing" });

      // 1. Unload and reload the model to reset its state and clear the KV cache.
      if (globalModelId) {
        await unloadModel({ modelId: globalModelId });
      }
      globalModelId = await loadModel({
        modelSrc: TTS_EN_SUPERTONIC_Q8_0,
        modelConfig: {
          ttsEngine: "supertonic",
          language: "en",
          voice: "F1",
          ttsSpeed: 1.05,
          ttsNumInferenceSteps: 5,
        },
      });

      // 2. Synthesize text to raw PCM samples
      const result = textToSpeech({
        modelId: globalModelId,
        text: text.trim(),
        inputType: "text",
        stream: false,
      });

      const audioBuffer = await result.buffer;

      // 3. Package and save WAV file using our local util
      const samplesInt16 = new Int16Array(audioBuffer);
      const wavUri = await saveAudioToDevice(
        samplesInt16,
        SUPERTONIC_SAMPLE_RATE,
      );

      // 4. Show player
      setStatus({ phase: "done", audioUri: wavUri });
    } catch (err: unknown) {
      console.error("TTS error:", err);
      const msg = err instanceof Error ? err.message : String(err);
      setStatus({ phase: "error", message: msg });
    }
  };

  const buttonLabel =
    status.phase === "synthesizing" ? "Synthesizing…" : "Synthesize Speech";

  if (!isModelLoaded) {
    return (
      <TtsModelLoader
        onDownload={handleDownloadModel}
        isDownloading={isDownloading}
        progress={downloadProgress}
      />
    );
  }

  return (
    <KeyboardAvoidingView
      behavior={Platform.OS === "ios" ? "padding" : "height"}
      className="flex-1 bg-black"
    >
      <ScrollView contentContainerClassName="flex-grow p-6  justify-center">
        <Card className="border border-border bg-card max-w-md w-full mx-auto">
          <CardHeader>
            <CardTitle variant="h3" className="text-white text-center">
              Text to Voice
            </CardTitle>
            <CardDescription className="text-center mt-1">
              Type or paste your content to synthesize speech
            </CardDescription>
          </CardHeader>

          <CardContent className="gap-6">
            <TextInput
              className="bg-muted text-white border border-border rounded-lg p-4 h-48 text-base leading-6"
              multiline
              numberOfLines={8}
              placeholder="Type your message here..."
              placeholderTextColor="#666"
              value={text}
              onChangeText={setText}
              style={{ textAlignVertical: "top" }}
              editable={!isBusy}
            />

            {status.phase === "error" && (
              <Text className="text-destructive text-sm text-center">
                {status.message}
              </Text>
            )}

            {status.phase === "done" && <AudioPlayer uri={status.audioUri} />}

            <Button
              onPress={handleSubmit}
              className="w-full h-12 rounded-xl"
              disabled={!text.trim() || isBusy}
            >
              <Text className="font-semibold text-lg">{buttonLabel}</Text>
            </Button>
          </CardContent>
        </Card>
      </ScrollView>
    </KeyboardAvoidingView>
  );
}

Codebase Breakdown

Let’s lift the hood on how this local Text-to-Speech implementation manages native model lifecycles and processes raw audio arrays.

1. Managing the Native Lifecycle

Loading neural network weights for speech synthesis is computationally expensive. When the QVAC runtime initializes a model, it must read parameters from the local disk and copy the active weights into device RAM.

To handle this efficiently, we declared the reference variable outside the component scope:

let globalModelId: string | null = null;

If globalModelId were tracked inside component states, navigating away from the text-to-speech screen would clean up the state, causing the app to unnecessarily drop the reference. Storing the ID globally ensures we hold onto it across layout transitions.

2. Flushing the KV Cache: Unload and Reload

One of the most important aspects of offline generation using GGML engines is state management:

// 1. Unload and reload the model to reset its state and clear the KV cache.
if (globalModelId) {
  await unloadModel({ modelId: globalModelId });
}

globalModelId = await loadModel({ ... });

WARNING about acoustic hallucinations: If you continuously synthesize sentences on a single TTS model instance without resetting it, the model's Key-Value (KV) cache fills up. It begins treating your new sentence as a continuation of the previous one, leading to heavy robotic distortion, echoing, and repeated voices.

By explicitly destroying the model via unloadModel and immediately booting a fresh instance with loadModel, we're forcing a pristine, empty context window. Since the model is already downloaded and memory-mapped, reloading the model directly from local flash storage is extremely fast, typically completing in a fraction of a second on modern mobile hardware to ensure a seamless user experience while guaranteeing artifact-free audio.

3. Demystifying the WAV Header Structure

Operating systems and built-in mobile media decoders are unable to parse raw, naked PCM (Pulse Code Modulation) sound waves directly. A raw PCM buffer is simply a stream of numerical coordinates representing audio wave amplitudes.

We resolve this by prepending-formatting our PCM buffer with a standard 44-byte RIFF/WAVE header.

This header acts as a passport, defining:

• AudioFormat (1): Signals uncompressed linear PCM.

• NumChannels (1): Mono audio.

• SampleRate (44100): The clock frequency required for Supertonic playback.

• BitsPerSample (16): 16-bit word length (2 bytes per sample).

Additionally, writing the file is handled via Base64 encoding to safely cross React Native's JavaScript-to-Native bridge without dropping binary data:

const base64Data = finalWavBuffer.toString("base64");
await FileSystem.writeAsStringAsync(fileUri, base64Data, {
  encoding: FileSystem.EncodingType.Base64,
});

4. Visual Waveform Player

Rather than using a basic headless native audio player that fires immediately in the background, we pass the local WAV file path to a custom <AudioPlayer> component powered by @simform_solutions/react-native-audio-waveform.

This module analyzes our newly written WAV file and draws a sleek, WhatsApp-inspired interactive visual waveform, giving the user full control over playback, dynamic speed adjustments (1x, 1.5x, 2x), and seeking. It's a vast UX improvement that makes the final result feel premium and polished.

Conclusion

Transitioning Text-to-Speech from the cloud to on-device hardware offers a practical approach for mobile application developers. Running model inference locally eliminates reliance on remote internet connectivity, removes recurring API usage costs, and ensures that user text inputs never leave the physical device.

Integrating local speech synthesis can be highly beneficial for interactive, educational, or conversational apps. For example, in voice-guided systems, on-device TTS allows applications to function in private or offline environments. As edge processors gain dedicated hardware acceleration cores and open-source models decrease in memory size through quantization research, local-first architectures present a compelling alternative for developers prioritizing privacy, offline resilience, and predictable cost structures.

Resources and Further Reading

To dive deeper into local Text-to-Speech inference, inspect the source code, or explore advanced configurations for your mobile applications, check out the following resources:

• QVAC Expo Integration Docs: Learn more about configuring custom local models in Expo.

• react-native-audio-waveform: Learn more about interactive React Native audio visualizations.

• GGUF Model Hub on Hugging Face: Browse compatible quantized open-source models.

• Latent Denoising Deep Dive: Technical deep dive into Diffusion-based acoustic generation.

• https://github.com/DjibrilM/QVAC-TTS-Expo-Implementation: Full implementation code.

In this article, you’ll learn how to prepare a real-world medical imaging dataset for machine learning, from initial data validation to a complete preprocessing pipeline.

We’ll use the Chest X-Ray Pneumonia dataset as our running example, but the lessons apply broadly to healthcare imaging data, including ultrasound, MRI, CT, and dermatology images.

What You'll Learn in This Article

By the end of this article, you'll know how to:

• Approach healthcare data preprocessing differently from preprocessing structured data, and recognize where standard techniques fall short

• Validate a medical imaging dataset before training to catch corrupted files, mislabels, and data leakage between train and test

• Apply six core preprocessing techniques for medical images

• Build a complete preprocessing pipeline for chest X-rays using Python with OpenCV.

What We'll Cover:

• Why Preprocessing Data Matters More in Healthcare

• The Dataset

• Before Preprocessing: Validate the Dataset

• The Six Pillars of Healthcare Imaging Preprocessing

• Pillar 1: Scaling — Making the Numbers Play Fair

• Pillar 2: Normalization — Centering the Data

• Pillar 3: Guiding the Model's Attention

• Pillar 4: Handling Missing Data

• Pillar 5: Resizing & Resampling — Fitting Everything in the Same Frame

• Pillar 6: Denoising & Artifact Handling — Cleaning the Window

• Putting it All together: A Complete Pipeline

• Try it Yourself

• Conclusion

Why Preprocessing Data Matters More in Healthcare

Imagine handing a toddler a jigsaw puzzle with missing pieces, warped edges, and pieces from three different puzzles mixed together. The toddler can't solve it, but that isn't really the toddler's fault.

The same thing happens when raw, messy data gets fed into a machine learning model. A bad prediction on a clinical image can mean a missed diagnosis.

Healthcare data tends to be messier than what most ML practitioners are used to:

• Images come from different machines, hospitals, and acquisition protocols

• Labels are inconsistent, sometimes missing, sometimes wrong

• Patient data is incomplete

• Image sizes, contrast levels, and orientations vary across sources

Poor preprocessing often leads to models that perform well on benchmark datasets but struggle to generalize to data collected from different hospitals or imaging devices.

The Dataset

This guide uses the Chest X-Ray Pneumonia dataset by Paul Mooney on Kaggle. It's a strong choice for learning preprocessing because:

• It contains around 5,800 pediatric chest X-rays

• It has two clear classes — Normal and Pneumonia

• It's already organized into train, validation, and test folders

• The images are recognizable without specialized medical training

• It exhibits almost every preprocessing challenge worth learning

The dataset is available at Kaggle: Chest X-Ray Pneumonia.

Folder Structure

After downloading, the dataset is organized like this:

chest_xray/
├── train/
│   ├── NORMAL/
│   └── PNEUMONIA/
├── val/
│   ├── NORMAL/
│   └── PNEUMONIA/
└── test/
    ├── NORMAL/
    └── PNEUMONIA/

Side-by-side comparison — Normal vs Pneumonia chest X-ray:

A quick first look at one of the images:

import os
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
import cv2

DATA_DIR = "chest_xray"
TRAIN_DIR = os.path.join(DATA_DIR, "train")

# Peek at a sample image
sample_path = os.path.join(TRAIN_DIR, "NORMAL", os.listdir(os.path.join(TRAIN_DIR, "NORMAL"))[0])
sample_image = cv2.imread(sample_path, cv2.IMREAD_GRAYSCALE)

print(f"Image shape: {sample_image.shape}")
print(f"Pixel range: {sample_image.min()} to {sample_image.max()}")
print(f"Data type: {sample_image.dtype}")

The output reveals a few useful things right away: most images are large (often around 1500×2000 pixels), pixel values fall in the 0–255 range, and image sizes vary across the dataset. Each of these observations will inform a preprocessing step.

Before Preprocessing: Validate the Dataset

Before applying any transformations, it's worth checking that the data itself is intact. This step alone catches issues that would otherwise cause training to fail silently or produce misleading results.

A simple validation function:

def validate_dataset(data_dir):
    """Scan a dataset folder and flag common data quality issues."""
    corrupted = []
    too_small = []
    nearly_black = []
    total = 0
    
    for class_name in os.listdir(data_dir):
        class_path = os.path.join(data_dir, class_name)
        if not os.path.isdir(class_path):
            continue
        for fname in os.listdir(class_path):
            fpath = os.path.join(class_path, fname)
            total += 1
            try:
                img = cv2.imread(fpath, cv2.IMREAD_GRAYSCALE)
                if img is None:
                    corrupted.append(fpath)
                    continue
                if img.shape[0] < 100 or img.shape[1] < 100:
                    too_small.append(fpath)
                if img.mean() < 5:
                    nearly_black.append(fpath)
            except Exception:
                corrupted.append(fpath)
    
    print(f"Total files scanned: {total}")
    print(f"Corrupted: {len(corrupted)}")
    print(f"Too small: {len(too_small)}")
    print(f"Nearly black: {len(nearly_black)}")
    return corrupted, too_small, nearly_black

validate_dataset(TRAIN_DIR)

Common issues this catches:

• Corrupted files — files that won't open at all

• Empty or nearly-black images — failed acquisitions or saved-as-blank files

• Wrong dimensions — thumbnails or partial downloads mixed in

• Duplicate images — the same scan appearing in both train and test (this causes data leakage)

• Mislabeled images — a normal X-ray placed in the pneumonia folder

⚠️ This step is critical, One corrupted file can crash a training loop hours into a run. One duplicate between train and test can inflate accuracy scores by several percentage points without anyone noticing.

The Six Pillars of Healthcare Imaging Preprocessing

Preprocessing for medical images can be organized around six core concerns. Two of them carry over directly from preprocessing structured data. Two need to be adapted because the mechanics change when the input is an image. And two are entirely new, they only exist once the data becomes pictures of human bodies.

Pillar 1: Scaling — Making the Numbers Play Fair

Imagine two children comparing their collections. One has 3 seashells. The other has 3,000 stickers. Asking who has more makes the answer seem obvious, but the scales are completely different. Comparing them meaningfully means putting both collections on the same measuring system.

In medical images, pixels usually range from 0 to 255 in 8-bit images, or 0 to 65,535 in some 16-bit medical DICOM images. Neural networks tend to train faster and more reliably when input values are small numbers close to zero.

The fix: Divide every pixel by its maximum possible value, bringing everything into the 0-to-1 range.

image = cv2.imread(sample_path, cv2.IMREAD_GRAYSCALE)

# Scale to [0, 1]
image_scaled = image.astype(np.float32) / 255.0

print(f"Before scaling: {image.min()} to {image.max()}")
print(f"After scaling:  {image_scaled.min():.3f} to {image_scaled.max():.3f}")

Takeaway: Pixel scaling follows the same principle as scaling any numerical feature. The values simply happen to be arranged as an image rather than a column.

Pillar 2: Normalization — Centering the Data

Imagine a teacher asks a class to rate a movie from 1 to 10. One child always gives 9s and 10s. Another spreads ratings evenly from 1 to 10. Comparing their opinions fairly requires adjusting each child's score relative to their own average.

In medical imaging even after scaling to 0–1, the overall brightness of images can vary. Some X-rays are taken with stronger exposure than others. Normalization shifts and rescales each image (or each channel) so the values are centered around zero with a standard deviation of one.

The fix: Subtract the mean, divide by the standard deviation.

# Compute mean and std from the TRAINING set only — never from validation or test
def compute_train_stats(train_dir, sample_limit=1000):
    """Compute pixel mean and std across the training set."""
    pixel_values = []
    count = 0
    for class_name in os.listdir(train_dir):
        class_path = os.path.join(train_dir, class_name)
        for fname in os.listdir(class_path):
            if count >= sample_limit:
                break
            img = cv2.imread(os.path.join(class_path, fname), cv2.IMREAD_GRAYSCALE)
            if img is not None:
                pixel_values.append(img.astype(np.float32).flatten() / 255.0)
                count += 1
    pixels = np.concatenate(pixel_values)
    return pixels.mean(), pixels.std()

train_mean, train_std = compute_train_stats(TRAIN_DIR)
image_normalized = (image_scaled - train_mean) / train_std

⚠️ Avoid this common mistake: Statistics for normalization should be computed from the training set only, never from validation or test. Including those in the calculation leaks information from the evaluation data into the model. The same statistics should then be applied to validation, test, and any new data at inference time.

Takeaway: Centering and scaling each image around the dataset's statistics is the imaging equivalent of standardizing a feature column. The pixels are now comparable across images, regardless of how bright or dim each scan happened to be.

Pillar 3: Guiding the Model's Attention

Imagine a child walking into a crowded pet store. Instead of describing every animal in sight, a parent points to the features that matter: “Look at the soft fur, the fluffy tail, and the nice small size.” The child learns where to focus their attention.

Medical image preprocessing does something similar. It highlights the regions and features most relevant to the diagnostic task.

• Region-of-interest (ROI) cropping — focus on the lung field and discard the patient's arms, machine borders, and any imprinted text

• Contrast enhancement — use techniques like CLAHE (Contrast Limited Adaptive Histogram Equalization) to make subtle lung textures more visible

• Channel selection — for images stored as RGB but containing grayscale information, convert to single-channel input to reduce noise

CLAHE applied to an X-ray:

# CLAHE enhances local contrast — useful for X-rays
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
image_enhanced = clahe.apply(image)

# Visualize the difference
fig, axes = plt.subplots(1, 2, figsize=(12, 6))
axes[0].imshow(image, cmap='gray')
axes[0].set_title('Original')
axes[1].imshow(image_enhanced, cmap='gray')
axes[1].set_title('After CLAHE')
plt.show()

Takeaway: The goal of teaching the model what to look at hasn't changed. With structured data, the answer is in new columns. With images, the answer is in cropping, enhancement, and emphasizing the regions that carry diagnostic signal.

Pillar 4: Handling Missing Data

Imagine reading a storybook with a few damaged pages. You don’t throw away the entire book, you decide whether to skip the page, infer what might be missing, or mark it for review.

In medical imaging, missing data can mean corrupted files, missing labels, or incomplete studies rather than empty spreadsheet cells.

The same three strategies — drop, impute, flag — still apply, just with different mechanics:

# Strategy 1: Drop — remove unreadable or empty images
def is_valid_image(path):
    try:
        img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
        if img is None:
            return False
        if img.mean() < 5:           # nearly black
            return False
        if img.shape[0] < 50 or img.shape[1] < 50:  # too small
            return False
        return True
    except Exception:
        return False

# Strategy 2: Impute — rare for images, but possible (e.g., in painting to fill in missing patches). Generally avoided for diagnostic data.

# Strategy 3: Flag — track which patients are missing which modalities,
#   and let the model condition on availability. Common in multi-modal healthcare ML.

Takeaway: "Missing" in imaging data is rarely just a NaN. It can be a broken file, an unlabeled scan, an absent modality, or a black corner inside an image. The same three strategies still apply.

Pillar 5: Resizing & Resampling — Fitting Everything in the Same Frame

Imagine displaying children’s drawings on a classroom wall. If every drawing is a different size, they won’t fit neatly into the display. You resize them while preserving their proportions.

Medical images must often be resized to a common input size, but anatomical structures should retain their original shape.

The fix: Resize all images to a common shape. For medical data, how the resizing is done matters.

TARGET_SIZE = (224, 224)

# Simple resize (may distort aspect ratio)
image_resized = cv2.resize(image, TARGET_SIZE)

# Better: preserve aspect ratio with padding
def resize_with_padding(image, target_size):
    h, w = image.shape[:2]
    target_h, target_w = target_size
    scale = min(target_h / h, target_w / w)
    new_h, new_w = int(h * scale), int(w * scale)
    resized = cv2.resize(image, (new_w, new_h))
    
    pad_h = target_h - new_h
    pad_w = target_w - new_w
    top, bottom = pad_h // 2, pad_h - pad_h // 2
    left, right = pad_w // 2, pad_w - pad_w // 2
    padded = cv2.copyMakeBorder(resized, top, bottom, left, right,
                                 cv2.BORDER_CONSTANT, value=0)
    return padded

image_clean_resize = resize_with_padding(image, TARGET_SIZE)

⚠️ Why aspect ratio matters in healthcare: Squishing a chest X-ray horizontally makes the lungs look unnatural. Models trained on distorted anatomy often perform worse on real scans. Preserving aspect ratio is generally the safer choice.

Takeaway: Models need a consistent input size, but the geometry of the anatomy needs to be preserved. Resize, but resize carefully.

Pillar 6: Denoising & Artifact Handling — Cleaning the Window

Imagine looking through a window with dust and smudges on the glass. Cleaning the window makes the view clearer, but scrubbing too aggressively could scratch the glass.

Similarly, medical images often contain noise and acquisition artifacts that should be reduced carefully without removing clinically important details.

For chest X-rays, the most common issues are mild noise and burned-in text or markers. A gentle median or bilateral filter helps with the first, while cropping or masking helps with the second.

# Gentle denoising — careful not to blur away clinical detail
image_denoised = cv2.medianBlur(image, ksize=3)

# Bilateral filter preserves edges better than a median filter
image_bilateral = cv2.bilateralFilter(image, d=5, sigmaColor=50, sigmaSpace=50)

⚠️ A note of caution: Aggressive denoising can erase the features a model needs to detect a disease. For diagnostic ML, gentle filtering is generally preferred. A useful rule of thumb: if a radiologist can't distinguish the cleaned image from the original, the filtering has gone too far.

Takeaway: Imaging data carries noise that structured data doesn't have. The window can be cleaned, but never so aggressively that the view is wiped away with the smudges.

Putting it All Together: A Complete Pipeline

Here's how the six pillars combine into a single preprocessing function for chest X-ray images:

def preprocess_xray(image_path, target_size=(224, 224),
                    train_mean=0.482, train_std=0.236):
    """
    Full preprocessing pipeline for chest X-ray images.
    Applies all six pillars in order.
    """
    # Pillar 4: Validate first — skip corrupted files
    if not is_valid_image(image_path):
        return None
    
    image = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    
    # Pillar 5: Resize with aspect ratio preserved
    image = resize_with_padding(image, target_size)
    
    # Pillar 6: Gentle denoising
    image = cv2.medianBlur(image, 3)
    
    # Pillar 3: Enhance contrast to highlight lung texture
    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
    image = clahe.apply(image)
    
    # Pillar 1: Scale to [0, 1]
    image = image.astype(np.float32) / 255.0
    
    # Pillar 2: Normalize using training set statistics
    image = (image - train_mean) / train_std
    
    return image

Try it Yourself

Every code snippet in this article is bundled into a runnable Kaggle notebook: Chest X-Ray Preprocessing — Kaggle Notebook. Fork it, attach the dataset, and run all the cells to see each preprocessing pillar in action on real chest X-rays.

Conclusion

Here's a summary of what we've discussed in this article:

Preprocessing for structured data is about making numbers play fair so a model can see them clearly.

Preprocessing for healthcare imaging is about respecting the messy reality of how medical data is captured, stored, and labeled. Some standard techniques carry over directly. Some need to be adapted. And a few preprocessing concerns only emerge once the data becomes pictures of human bodies.

Stepping back, whether it's a child learning to organize their toy box, or a model learning to spot pneumonia in a chest X-ray, the quality of learning depends on the quality of data preparation. Get the data right.

If this was useful, you can find a related conceptual primer on preprocessing more broadly here: Data Preprocessing for Machine Learning.

Yet despite its impressive performance, GPT-3 revealed an important limitation: raw capability doesn't automatically create a useful assistant.

A language model can generate fluent text, answer questions, and solve complex tasks while still failing to follow what the user actually wants.

GPT-3 could produce responses that were inconsistent, overly confident, difficult to control, or misaligned with user instructions. It was a powerful prediction engine, but it wasn't designed to reliably act as a helpful assistant.

This challenge motivated one of the most influential papers in modern AI: Training Language Models to Follow Instructions with Human Feedback. Rather than making the model larger, the researchers focused on teaching it how to better follow human intent.

The result was InstructGPT, a system fine-tuned from GPT-3 that demonstrated how human feedback could transform a capable language model into a far more useful and aligned assistant.

This challenge became one of the most important problems in modern AI: alignment.

Researchers realized that building larger models was only part of the solution. While scaling improved capabilities, it didn't guarantee that models would reliably follow instructions or behave in ways that matched user expectations. The next stage of progress required teaching models how to respond in a more helpful, truthful, and safe manner.

This led to the development of instruction-following systems and Reinforcement Learning from Human Feedback (RLHF). Instead of optimizing models solely to predict the next word, researchers began training them to better align with human preferences and intentions.

This shift marked a major turning point in the evolution of large language models.

GPT-3 demonstrated the power of large-scale language modeling and introduced many people to prompting and few-shot learning.

InstructGPT built on that foundation by showing how human feedback could significantly improve instruction following and model behavior. ChatGPT then brought these ideas to a much broader audience by packaging aligned language models into an accessible conversational interface used by millions of people.

In many ways, language models became capable before they became aligned.

That's why the transition from GPT-3 to InstructGPT represents one of the most important milestones in the history of artificial intelligence. The focus was no longer only on making models more capable. It was also about making them more useful, reliable, and responsive to human intent.

The success of InstructGPT pioneered many of the alignment techniques that later became a core part of systems such as ChatGPT and GPT-4.

Paper Overview:

In this article, we’ll mainly focus on the paper Training Language Models to Follow Instructions with Human Feedback, published by OpenAI in 2022.

This paper introduced InstructGPT, one of the most important transitions in the history of large language models. While earlier GPT systems focused heavily on scaling model size and improving raw capabilities, this work shifted attention toward something equally important: alignment.

The paper explores how language models can be trained to better follow human instructions using reinforcement learning from human feedback (RLHF). Instead of optimizing only for next-token prediction, the model is further optimized to produce responses that humans actually prefer – responses that are more helpful, safer, and more aligned with user intent.

What makes this paper historically important is that it became the foundation for the modern ChatGPT alignment pipeline.

Many of the interaction patterns people now associate with ChatGPT (like instruction following, conversational behavior, refusal handling, and safer responses) can be traced directly back to the ideas introduced here.

Here’s the original paper again if you want to explore it directly: Training language models to follow instructions with human feedback

And here’s a quick infographic of what we’ll cover throughout this review:

Table of Contents:

• Executive Summary

• The Core Problem

• Why GPT-3 Was Not Enough

• InstructGPT: The Birth of Alignment-Centered LLMs

• RLHF Pipeline: How InstructGPT Learned to Behave Like an Assistant

  • Stage 1 — Supervised Fine-Tuning (SFT)

  • Stage 2 — Reward Model Training

  • Stage 3 — PPO Reinforcement Learning

• Helpful, Honest, Harmless

• Human Feedback as the New Scaling Factor

• Why ChatGPT Exploded Globally

• ChatGPT as an Interface Revolution

• Benchmarks and Results

• Truthfulness and Hallucinations

• Safety and Refusal Behavior

• Limitations

• Historical Importance

• Discussion: The Real Shift

• Connection to GPT-4

• GPT-3 vs InstructGPT vs ChatGPT vs GPT-4: Key Differences

• From GPT-1 to GPT-4: A Timeline of Modern AI Systems and Alignment Evolution

• Final Insight

• Resources

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with a few foundational ideas.

Reading the previous reviews in this series will be especially helpful:

• AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1)

• AI Paper Review: Language Models are Unsupervised Multitask Learners (GPT-2)

• AI Paper Review: Language Models are Few-Shot Learners (GPT-3)

Even though GPT-4 was released after InstructGPT, reading the GPT-4 review can still be helpful. It provides a broader view of how alignment techniques evolved and how they were combined with stronger reasoning and multimodal capabilities in later generations of GPT models.

• AI Paper Review: GPT-4 Technical Report (GPT-4)

It also helps to have:

• A general understanding of natural language processing (NLP) and large language models

• A high-level idea of Transformer-based autoregressive models

• Familiarity with prompting, few-shot learning, and in-context learning

• A basic understanding of reinforcement learning and human feedback systems

• General machine learning concepts like training data, fine-tuning, scaling, and inference

• Some familiarity with alignment, safety, and AI behavior control concepts

You don't need to be an AI researcher to follow this article, though.

I’ll keep the explanations practical and intuitive, focusing more on understanding how InstructGPT changed modern AI systems rather than getting lost in dense mathematical details or academic terminology.

Executive Summary

The paper Training Language Models to Follow Instructions with Human Feedback marks one of the biggest turning points in the history of modern AI systems. Instead of asking only how to make language models larger or smarter, OpenAI focused on a different question: how do we make these models actually helpful for real people?

The paper introduces InstructGPT, a version of GPT-3 fine-tuned to follow human instructions more accurately using a method called Reinforcement Learning from Human Feedback (RLHF).

The core insight of the paper is simple but extremely important:

Bigger language models don't automatically become better assistants.

Even highly capable models like GPT-3 could still:

• ignore instructions

• hallucinate facts

• generate toxic or biased outputs

• produce responses that were technically fluent but not actually useful to users

To solve this problem, OpenAI built a multi-stage alignment pipeline: humans first demonstrate ideal answers, humans then rank model outputs, and finally the model learns from those preferences using reinforcement learning.

This changed the direction of modern AI development.

The paper shows that alignment and usability can matter more than raw model size itself. One of the most surprising findings was that the 1.3B InstructGPT model was often preferred by human evaluators over the original 175B GPT-3 model, despite being dramatically smaller.

The paper also demonstrates improvements in instruction following, truthfulness, toxicity reduction, conversational behavior, and general user preference.

Historically, this paper became the foundation behind modern conversational AI systems.

GPT-3 proved that language models could learn from prompts.

GPT-4 later proved that scaling and multimodal reasoning could unlock even stronger capabilities.

But InstructGPT showed something equally important: AI systems must be aligned with human intent to become truly usable products.

In many ways, this paper represents the transition from raw language modeling to aligned assistants, capability scaling to behavior shaping, and research demos to real-world conversational AI systems.

And that transition eventually led directly to ChatGPT.

The Core Problem

One of the most important ideas in this paper is that raw language modeling is not the same thing as building a useful assistant.

Before InstructGPT, models like GPT-3 were trained mainly with a simple objective: predict the next token in a sequence.

That objective made language models extremely powerful at generating fluent text, but it also created a major limitation. The model learned how to continue internet text, not necessarily how to help humans.

This became one of the defining realizations behind modern AI alignment research.

Despite its impressive capabilities, GPT-3 often struggled to behave like a reliable assistant. The model could produce fluent text, but it was not explicitly trained to follow user intent.

Here are some examples that highlight the differences between GPT-3 and InstructGPT in how they respond to user prompts:

Source: Aligning language models to follow instructions

Source: Aligning language models to follow instructions

These examples reveal the central weakness of early GPT systems. GPT-3 often continued the pattern of the prompt rather than completing the requested task. InstructGPT, by contrast, responded directly to the user's instruction. The difference wasn't a matter of raw intelligence. It was a difference in training objectives.

GPT models were trained on massive internet-scale datasets where the goal was simply to predict what text comes next. As a result, the model optimized for plausibility, continuation, and pattern completion. Not necessarily for truthfulness, safety, helpfulness, or alignment with human goals.

This created a major gap between: language capability and useful assistant behavior.

For example, if a user asked a harmful, misleading, or nonsensical question, the model might still attempt to continue the pattern naturally instead of recognizing the issue. In many cases, the model behaved more like an internet text simulator than a reliable assistant.

The paper repeatedly emphasizes that scaling alone couldn't solve this problem.

Researchers increasingly recognized that better behavior would require more than scaling alone.

Models also needed stronger instruction following, better alignment with human intent, improved safety behavior, greater truthfulness, and optimization around real user needs.

Why GPT-3 Was Not Enough

When GPT-3 was released, it felt like a massive leap forward in AI capabilities.

The model could perform few-shot learning, answer questions, summarize text, generate code, translate languages, and even solve certain reasoning tasks: all without traditional fine-tuning. For many researchers, it was the first time a language model started to feel genuinely general-purpose.

Yet using GPT-3 in practice was often less reliable than its benchmark performance suggested.

In practice, using GPT-3 often required careful prompt engineering. Small wording changes could completely change the quality of the response. Sometimes the model followed instructions well, and other times it ignored them entirely.

Users often found themselves rewriting prompts repeatedly to obtain the response they actually wanted.

This became the core motivation behind InstructGPT.

OpenAI responded by exploring ways to make model behavior more consistent, predictable, and useful for users.

InstructGPT: The Birth of Alignment-Centered LLMs

The release of InstructGPT marked one of the biggest shifts in the history of large language models.

Before InstructGPT, most advances in language models came from scaling data, compute, and model size.

The focus shifted toward alignment: building systems that could follow instructions more reliably and behave in ways users actually preferred.

This is where InstructGPT introduced one of the most important ideas in modern AI systems: Reinforcement Learning from Human Feedback (RLHF).

Instead of optimizing models only to predict internet text, OpenAI started optimizing models based on what humans actually preferred. Human labelers ranked model outputs, and those preferences became part of the training process itself.

This fundamentally changed the objective of language models.

Rather than optimizing solely for next-token prediction, the system was increasingly optimized to produce responses that humans judged to be helpful, safe, and aligned with their intentions.

That distinction may sound subtle, but it completely changed the direction of AI development.

InstructGPT combined instruction-following training with human preference optimization, creating a model whose behavior could be shaped directly through feedback rather than solely through pretraining.

The model was no longer trained only to imitate the internet. It was trained to behave more like an assistant.

RLHF Pipeline: How InstructGPT Learned to Behave Like an Assistant

At the center of the InstructGPT paper is a training pipeline that completely changed how modern AI assistants are built.

RLHF was designed to build on traditional language-model pretraining rather than replace it.

The InstructGPT paper introduced a different idea: instead of training models only on internet text, why not train them using human preferences directly?

This led to the development of the RLHF pipeline: Reinforcement Learning from Human Feedback. This approach would later become a standard component of modern conversational AI systems.

The paper’s Figure 2 is especially important because it visualizes the entire alignment pipeline introduced by OpenAI. Rather than relying on a single training stage, the system uses multiple stages where human feedback gradually shapes model behavior.

Source: Training Language Models to Follow Instructions with Human Feedback (OpenAI, 2022).

As you can see in the image above, the process happens in three major stages.

Stage 1 — Supervised Fine-Tuning (SFT)

The first stage starts with human-written demonstrations.

Labelers are given prompts and asked to write ideal responses – the kinds of answers a helpful assistant should produce. These examples become the initial training dataset for the model.

At this stage, the model learns the basic patterns of assistant-style responses.

This is still traditional supervised learning, but the goal is different from standard language modeling. Instead of learning only from web text, the model now learns from examples of preferred assistant behavior.

This stage creates what the paper calls the Supervised Fine-Tuned model (SFT model).

And while this already improves behavior significantly, OpenAI realized something important: human preferences are more complex than simple “correct answers.”

There are often many possible responses to a prompt, but humans may strongly prefer some answers over others.

That leads to the next stage.

Stage 2 — Reward Model Training

In the second stage, humans no longer write responses directly.

Instead, the model generates multiple answers for the same prompt, and human labelers rank them from best to worst.

For a given prompt, one response may be clearer, another more accurate, and another safer or more appropriate. Human labelers rank these alternatives according to their preferences

The rankings are then used to train a separate neural network called the Reward Model (RM).

This model learns something extremely important: which outputs humans prefer.

In other words, the system converts human preferences into a trainable reward signal.

This becomes one of the biggest conceptual breakthroughs in the paper. Instead of manually programming behavior rules, OpenAI trains the model to approximate human judgment itself.

The reward model captures patterns in human preferences and turns them into a training signal.

That reward signal becomes the foundation for the final training stage.

Stage 3 — PPO Reinforcement Learning

The final stage uses reinforcement learning to optimize the language model against the reward model.

More specifically, the paper uses PPO (Proximal Policy Optimization), a reinforcement learning algorithm commonly used in policy optimization tasks.

At this stage, the model generates responses, receives scores from the reward model, and gradually updates its behavior to maximize those scores.

The model gradually shifts toward responses that receive higher scores from the reward model.

The key innovation is that optimization now occurs against a learned representation of human preferences rather than only a language-modeling objective.

According to the paper, this RLHF pipeline significantly improved instruction following and user preference ratings while also reducing toxic and unsafe behavior.

And in many ways, this pipeline became the blueprint for the modern era of conversational AI systems.

Helpful, Honest, Harmless

The authors argue that evaluating language models requires more than measuring capability alone. They should also be evaluated by how they behave around humans.

At the time, this represented a significant shift in how researchers evaluated language models.

That is why the paper repeatedly emphasizes a new alignment philosophy centered around three goals:

• Helpful

• Honest

• Harmless

These ideas became the conceptual foundation behind modern alignment research and conversational AI systems.

Helpful

The first goal is straightforward: the model should genuinely help the user accomplish what they want.

In practice, helpfulness means following instructions clearly, answering questions directly, providing relevant information, and adapting to the user's intent.

This may seem simple, but it fundamentally changes the training objective.

The model is no longer optimized only for linguistic fluency. It's optimized for usefulness.

Honest

The second goal is honesty.

One of the biggest problems with large language models is that they often produce convincing answers even when those answers are wrong. The models can hallucinate facts, invent references, or respond confidently despite uncertainty.

The paper recognizes that a useful assistant shouldn't merely sound intelligent. It should also behave truthfully and acknowledge uncertainty when necessary.

This is especially important because language models are optimized to generate plausible text, not verified truth.

As a result, earlier models sometimes prioritized sounding coherent over being accurate.

The alignment process introduced in InstructGPT attempts to reduce this behavior through human feedback and preference optimization. Human evaluators consistently prefer responses that are more accurate, transparent, and reliable, and those preferences gradually shape the model during RLHF training.

The paper doesn't claim that hallucinations disappear completely. Far from it. But it marks one of the first large-scale attempts to explicitly optimize language models for truthfulness and reliability rather than pure text generation quality.

Harmless

The third goal is harmlessness.

Large language models trained on internet data inevitably absorb toxic, biased, unsafe, or harmful patterns from that data. Without alignment, models may generate dangerous instructions, offensive content, or manipulative behavior.

The paper directly addresses this concern and treats safety as a central part of model development.

Through RLHF and human preference ranking, the model learns to refuse certain harmful requests, avoid toxic generations, produce safer responses, and behave more responsibly during interaction.

This became one of the defining characteristics of modern conversational AI systems.

Instead of maximizing unrestricted generation, the system begins balancing usefulness, safety, and alignment with human values.

But the paper is also honest about limitations.

The authors acknowledge that harmful outputs, biases, and unsafe behavior can still appear. Alignment is imperfect, and human values themselves are complex and difficult to define universally.

But historically, this paper marks the moment when safety and alignment became core engineering goals rather than secondary concerns.

Taken together, these three principles (helpful, honest, and harmless) became much more than training objectives. They became the philosophical foundation behind ChatGPT-era AI systems.

Earlier GPT papers mainly explored how to scale intelligence. But InstructGPT explored something deeper: how to make intelligence usable for humans.

Human Feedback as the New Scaling Factor

One of the most fascinating ideas behind the InstructGPT paper is that it quietly changed what “scaling” meant in modern AI.

For years, progress in language models was largely measured through scaling.

GPT-1 showed that pretraining works. GPT-2 showed that larger models develop stronger zero-shot behavior. GPT-3 pushed this idea even further by scaling to 175 billion parameters and demonstrating impressive few-shot learning abilities.

And to some extent, that was true. Larger models became better at reasoning, code generation, language understanding, translation, and generalization.

That is where human feedback became central.

Instead of relying purely on internet-scale text, OpenAI introduced a training pipeline where human preferences directly shaped model behavior. Human labelers ranked responses, evaluated quality, and guided the system toward outputs people actually preferred.

In many ways, this created a completely new scaling dimension for AI systems:

• scaling human feedback

• scaling preference learning

• scaling alignment pipelines

Historically, this shifted attention from model scale alone toward the quality of model behavior

InstructGPT focused on scaling usability. And the results were surprisingly powerful.

According to the paper, a much smaller aligned model was often preferred over the original 175B GPT-3 model by human evaluators.

That finding changed how the industry thought about progress.

The result suggested that improving behavior could sometimes matter as much as increasing scale.

This is why RLHF became one of the defining ideas of the ChatGPT era.

After InstructGPT, modern AI systems were no longer evaluated only by benchmark scores, parameter counts, or scaling curves.

They were increasingly evaluated by usefulness, conversational quality, safety, reliability, and how well they interact with humans.

And that shift fundamentally changed the future direction of large language models.

Why ChatGPT Exploded Globally

When ChatGPT launched publicly, the reaction was immediate and unlike anything the AI industry had seen before.

Millions of people started using it within days. Developers, students, writers, researchers, businesses, and everyday users suddenly felt like they were interacting with AI in a completely different way.

What made this moment so important was that advanced AI capabilities finally became accessible to ordinary users. After all, the underlying language models were already extremely capable before ChatGPT existed. GPT-3 could generate essays, answer questions, write code, summarize text, and perform impressive few-shot learning tasks. GPT-4 later pushed reasoning and multimodal abilities even further.

The challenge was no longer whether language models could perform useful tasks, but whether people could interact with them naturally.

ChatGPT combined powerful language-model capabilities with RLHF-based alignment, conversational interaction, safer behavior, and a user-friendly chat interface.

Earlier systems often required significant prompt experimentation to achieve consistent results. Users had to carefully engineer prompts, retry questions, or work around strange outputs. The models could be brilliant one moment and confusing the next.

ChatGPT changed that experience dramatically.

Thanks to the alignment techniques introduced in the InstructGPT paper, the system became far better at following instructions, maintaining conversational flow, understanding intent, and responding in a way that felt cooperative rather than purely generative.

The conversational interface itself also mattered enormously.

Before ChatGPT, interacting with advanced AI systems often required APIs, coding knowledge, prompt experimentation, or technical understanding.

ChatGPT simplified everything into a familiar chat format: you simply typed naturally, and the system responded naturally.

That design decision may sound small, but historically it was transformative. It turned large language models from research tools into consumer products.

Although imperfect, the system felt substantially more reliable than earlier language-model interfaces.

The system was designed to communicate in ways that felt more natural and cooperative.

The breakthrough was not simply that the AI became smarter. The breakthrough was that the AI became usable.

And that usability is what transformed large language models from impressive research demonstrations into globally adopted AI assistants.

ChatGPT as an Interface Revolution

One of the most important things about ChatGPT is that it changed how humans interact with computers.

Before ChatGPT, powerful AI systems mostly lived behind APIs, research demos, developer tools, and complex prompting workflows.

Using advanced language models often required technical knowledge. Developers experimented with prompt engineering, API parameters, temperature settings, and carefully structured inputs just to get reliable outputs from the model.

Even GPT-3, despite being extremely powerful, still felt like a research system for many users. You had to learn how to “talk to the model.”

And in many cases, the interaction felt fragile. Slight changes in wording could completely change the quality of the response.

ChatGPT changed that dynamic almost overnight.

Instead of making users adapt to the AI, the AI became much better at adapting to humans.

Natural conversation became the interface.

For decades, human-computer interaction depended on commands, menus, search boxes, forms, programming languages, and specialized software interfaces.

ChatGPT introduced something different: you could simply explain what you wanted in plain language. And the system would usually understand.

This made AI feel accessible to people who had never written code, used APIs, or interacted with machine learning systems before.

In many ways, ChatGPT transformed prompting into a universal interface for computing. And that single shift affected nearly every digital field.

In education, students started using conversational AI to explain difficult concepts, summarize lessons, practice languages, and receive tutoring-style help.

In coding, developers began using AI systems for debugging, code generation, documentation, and learning new frameworks.

This eventually led to the rise of AI coding assistants integrated directly into development environments.

In writing and content creation, conversational AI became a brainstorming partner capable of drafting ideas, rewriting text, organizing articles, and helping people communicate more effectively.

Search behavior also started changing. Instead of searching through lists of links, users increasingly expected direct conversational answers. This fundamentally challenged traditional search-engine interaction models.

And across productivity tools, AI systems began acting less like software features and more like collaborative assistants.

This shift was enabled by advances in conversational AI and interaction design that made dialogue feel natural and useful.

The alignment techniques introduced by InstructGPT were an important part of making these conversational experiences practical.

Historically, this may become one of the most important consequences of the GPT era: earlier software required humans to learn interfaces. ChatGPT pushed computing toward interfaces that learn humans instead.

Benchmarks and Results

We've already discussed how one of the biggest improvement didn't come from making the model larger. Instead, it came from making the model better aligned with humans.

This is one of the central findings of the entire paper, and it changed how many researchers thought about progress in large language models.

Before this work, the dominant belief was that scaling was the main path forward, with bigger models, more parameters, more compute, and more data. And GPT-3 seemed to confirm that idea. Larger models consistently showed stronger few-shot learning, reasoning, and generalization abilities.

But the InstructGPT paper introduced a different perspective. The researchers found that a relatively small 1.3B parameter InstructGPT model was often preferred by human evaluators over the original 175B GPT-3 model.

That result was extremely important. It suggested that alignment sometimes outperformed scale.

This became one of the defining insights of the ChatGPT era.

According to the paper, human evaluators consistently preferred InstructGPT responses because they were more helpful, more accurate, safer, and better aligned with what users were actually asking for.

The improvements appeared across several important areas.

One major improvement was instruction following. Earlier GPT models often ignored instructions, drifted off-topic, or generated responses that sounded fluent but failed to solve the user’s actual task. InstructGPT behaved much more like a cooperative assistant and followed prompts more reliably.

The paper also reports improvements in truthfulness. Large language models are known for hallucinating information and confidently generating false statements. Through RLHF and preference optimization, InstructGPT reduced some of these behaviors and produced answers humans judged to be more truthful and reliable.

Another important improvement involved toxicity and harmful outputs. The researchers evaluated the system on toxicity benchmarks and found that aligned models generated fewer toxic or unsafe responses compared to earlier GPT systems.

What makes these findings historically important is that they changed the industry’s understanding of what “better AI” actually meant.

Before InstructGPT, improvement was mostly measured through benchmark scores, scaling curves, and parameter counts.

After InstructGPT, researchers increasingly focused on usability, safety, alignment, conversational quality, and human preference satisfaction.

This was a major shift in AI development philosophy.

Truthfulness and Hallucinations

A major challenge for language models is that fluent responses are not always truthful.

This behavior is now commonly called hallucination.

Hallucinations can take many forms, including invented facts, fabricated references, incorrect explanations, or confident answers that lack factual support.

And because the responses are fluent and natural, the mistakes can sometimes look believable to users. The InstructGPT paper treats this as a serious issue rather than a minor flaw.

The authors note that language models are optimized for plausibility rather than verified truth. This is an important distinction: a language model can generate text that looks correct while still being inaccurate.

This is why the paper places particular emphasis on truthfulness and factual reliability.

Through RLHF and human preference optimization, InstructGPT was trained to produce answers humans judged to be more accurate and trustworthy. Human evaluators generally preferred responses that were more transparent about uncertainty and less likely to contain misleading information.

The paper also evaluates the model on truthfulness benchmarks such as TruthfulQA, where aligned models demonstrated improvements compared to earlier GPT systems.

But the paper is also careful not to overstate the results. Hallucinations didn't disappear. The aligned models could still make reasoning mistakes, generate false information, misunderstand prompts, or produce overconfident answers.

This nuance is extremely important: the paper doesn't claim that RLHF solved factuality or reasoning completely. Instead, alignment improved behavior, not perfection.

That distinction became increasingly important as ChatGPT and later GPT-4 systems reached millions of users worldwide.

The models became more useful, more truthful, and more aligned, but they still remained probabilistic language models rather than guaranteed fact engines.

In many ways, the InstructGPT paper marks the beginning of large-scale efforts to make AI systems not only intelligent, but also trustworthy enough for real-world human interaction.

Safety and Refusal Behavior

As language models became more powerful, researchers realized that safety was becoming a deployment problem.

A model that can generate human-like language at scale can also generate harmful instructions, produce toxic content, spread misinformation, or be manipulated into unsafe behavior.

The InstructGPT paper treats these risks very seriously and frames alignment as a necessary part of deploying large language models responsibly.

One of the biggest changes introduced through RLHF was safer refusal behavior.

Earlier GPT systems often attempted to answer almost anything. As a result, they often responded to unsafe prompts rather than recognizing when a refusal was appropriate.

InstructGPT begins changing that behavior.

Through human feedback and preference optimization, the model learns that some requests shouldn't be answered directly. Human labelers consistently prefer safer responses, refusals for harmful instructions, and outputs that avoid dangerous or toxic behavior.

This leads to systems that are better at refusing unsafe requests, avoiding toxic generations, and behaving more cautiously during interaction.

The paper also evaluates toxicity reduction using safety-related benchmarks and finds that aligned models generally produce fewer harmful outputs than earlier GPT systems.

Another important issue is harmful content filtering. Large language models absorb patterns from massive internet datasets, which inevitably contain biased language, misinformation, unsafe instructions, and toxic behavior.

Without alignment, models may reproduce these patterns surprisingly easily.

RLHF acts as a corrective layer on top of pretraining. Instead of only imitating internet text, the model is further optimized toward responses humans judge to be safer and more appropriate.

Of course, the paper is also realistic about limitations.

The authors acknowledge that alignment is incomplete and that unsafe outputs can still occur. Models may still be vulnerable to adversarial prompting or attempts to bypass safety behavior (what later became widely known as jailbreaks).

This is an important nuance: alignment reduces risk, but it doesn't eliminate it.

And historically, this realization became incredibly important for the future of large-scale AI deployment.

In many ways, the InstructGPT paper marks the beginning of modern AI safety engineering inside flagship language models.

InstructGPT introduced large-scale behavior alignment. Then GPT-4 expanded this even further with red teaming, adversarial testing, deployment monitoring, and much larger safety evaluation pipelines.

So this paper becomes a direct bridge between early generative language models and the much more safety-focused AI systems that followed in the GPT-4 era.

Limitations

One of the strongest aspects of the InstructGPT paper is that it doesn't present alignment as a solved problem.

Even though the results are impressive, the authors are careful and surprisingly honest about the system’s remaining weaknesses and risks.

This balance is important because the paper isn't arguing that RLHF creates perfect AI systems. The authors consistently frame alignment as a work in progress rather than a finished solution.

One major limitation is that the models still hallucinate.

The paper acknowledges that hallucinations remain a significant challenge despite alignment improvements.

RLHF improves truthfulness and instruction adherence, but it doesn't fundamentally solve the probabilistic nature of language models. The system still predicts likely text patterns rather than verifying objective truth.

Another important issue is reward hacking.

Because the model is optimized against a learned reward signal, it can sometimes discover shortcuts that maximize reward without genuinely improving reasoning or understanding. In other words, the model may learn behaviors that look aligned to evaluators while still hiding deeper problems underneath.

This is a common challenge in reinforcement learning systems more broadly.

The paper also hints at a problem that later became widely discussed in ChatGPT-era systems: over-refusal and sycophancy.

Sometimes aligned models become too cautious and refuse harmless requests unnecessarily. In other cases, models may become overly agreeable, telling users what they appear to want to hear instead of providing more balanced or truthful responses.

This creates a difficult tension between safety, helpfulness, and honesty.

Another major limitation is bias.

Since these systems are trained on massive internet datasets and further shaped through human labeling, they inevitably inherit biases from both sources. The paper explicitly acknowledges that alignment doesn't remove all harmful or biased behavior.

And perhaps most importantly, the paper emphasizes that RLHF aligns models to labeler preferences not universal human values. This is a very important nuance.

The system learns from the judgments of specific human annotators operating within specific cultural and organizational contexts. That means alignment itself is subjective and imperfect.

There is no single universally agreed definition of helpfulness, fairness, safety, or acceptable behavior.

The paper discusses these concerns carefully and recognizes that human feedback introduces its own limitations and assumptions.

The alignment itself is also fragile. Even aligned systems can sometimes be manipulated through adversarial prompting or jailbreak-style attacks that bypass safety behavior. This later became one of the defining challenges of ChatGPT and GPT-4 deployment.

And finally, there's the practical issue of scale.

RLHF requires large amounts of human labeling, ranking, evaluation, and monitoring. Building these alignment pipelines is expensive, time-consuming, and operationally complex. Unlike raw pretraining data scraped automatically from the internet, human feedback doesn't scale nearly as easily.

In many ways, the paper reveals an important truth about modern AI systems: making models intelligent is difficult. But making them reliably aligned with humans may be even harder.

Historical Importance

Looking back now, it's difficult to overstate how important the InstructGPT paper became for the entire AI industry.

Earlier GPT papers focused mostly on one central question: How do we make language models more capable?

That era was largely driven by larger datasets, larger parameter counts, scaling laws, and benchmark performance.

The models became increasingly impressive at generating text, solving tasks, and demonstrating emergent abilities. But they still behaved primarily like prediction engines trained to continue internet text.

InstructGPT changed the focus completely. For the first time, large-scale AI development began shifting from model-centric AI to interaction-centric AI.

This was a major philosophical transition: the industry realized that users didn't only care about raw intelligence, benchmark scores, or parameter counts.

They cared about usability, conversational quality, safety, trust, and whether the system could actually help them effectively.

This is why ChatGPT felt so different to the public. The underlying language model capabilities were important, but the real breakthrough came from how those capabilities were shaped into a usable human experience.

The interface became conversational. The system became more cooperative. The AI became more aligned with user intent.

That shift fundamentally changed public perception of artificial intelligence.

Before ChatGPT, most people saw AI as research software, technical demos, or specialized tools for experts.

After ChatGPT, millions of people started interacting with AI systems conversationally on a daily basis.

And that changed everything.

Earlier GPT papers focused mainly on discovering what scaling could achieve. InstructGPT introduced a different challenge: How do we safely deploy these systems in the real world?

That shift helped create entirely new areas of research and engineering, including RLHF pipelines, safety tuning, refusal behavior, red teaming, adversarial testing, policy frameworks, and large-scale human-feedback infrastructure.

In many ways, the ChatGPT era began the moment researchers realized that building powerful models was only part of the problem.

The harder challenge was making those systems reliable enough for human interaction at global scale.

It also helps explain why later systems placed much greater emphasis on safety, alignment, deployment practices, and real-world reliability.

The industry was no longer building language models only for research papers. It was building AI systems intended to operate in the real world. And the InstructGPT paper became one of the clearest turning points in that transformation.

Discussion: The Real Shift

The transition from GPT-3 to ChatGPT represents something much deeper than a simple improvement in model performance.

It changed the central question driving the entire AI industry.

During the GPT-3 era, the big question was, “Can language models learn tasks directly from prompts?”

That was the breakthrough introduced by GPT-3.

Research attention shifted toward scaling and emergent capabilities.

But the ChatGPT era introduced a completely different challenge: the question was no longer simply “Can the model perform the task?” Instead, it became, “Can humans actually trust and use these systems every day?”

That shift changed everything.

Once millions of people began interacting with AI systems directly, raw intelligence alone was no longer sufficient. Users needed systems that were understandable, reliable, safe, conversational, and aligned with human expectations.

This is exactly why the InstructGPT paper became so historically important. It introduced the idea that large language models should not only optimize for capability, but also for human interaction quality.

In many ways, the industry moved from “How smart is the model?” to “How usable is the model?”

And that transition fundamentally changed AI development.

After ChatGPT, success was no longer measured only by benchmark scores, parameter counts, or scaling curves.

It was increasingly measured by alignment, conversational quality, safety, and real-world usability.

This also explains why alignment research suddenly became central to modern AI systems.

GPT-3 showed that models could learn from prompts. ChatGPT showed that humans needed models that could cooperate.

That was the real shift.

And it may ultimately become one of the most important turning points in the history of artificial intelligence.

Connection to GPT-4

One of the most important things to understand about GPT-4 is that it didn't appear out of nowhere.

It was built on top of the alignment ideas introduced by InstructGPT and refined through the large-scale deployment experience of ChatGPT.

GPT-4 is often discussed in terms of its reasoning, multimodal abilities, and benchmark performance.

But beneath all of those improvements is something equally important: the alignment pipeline.

Without the work introduced in the InstructGPT paper, GPT-4 would likely feel far less usable as a real-world assistant.

That distinction matters enormously.

Many of GPT-4's alignment techniques can be traced back to ideas introduced by InstructGPT, including RLHF, instruction tuning, conversational alignment, safer refusal behavior, and human preference optimization.

ChatGPT then became the large-scale real-world testing ground for these ideas.

Millions of user interactions exposed weaknesses ranging from hallucinations and jailbreak attempts to broader safety and usability issues.

Those deployment lessons became incredibly valuable.

By the time GPT-4 arrived, OpenAI was no longer simply training a larger language model. It was building a large-scale aligned conversational system shaped by RLHF pipelines, human feedback, safety engineering, adversarial testing, and real-world user interaction.

This is why GPT-4 feels fundamentally different from earlier GPT models.

In many ways, GPT-4 represents the convergence of two major ideas: scaling capability and scaling alignment.

• GPT-3 proved that language models could learn tasks from prompts.

• InstructGPT proved that models could be shaped through human feedback.

• ChatGPT proved that aligned conversational AI could work at global scale.

• GPT-4 combined all of those ideas into a much more capable multimodal system.

That historical progression is important because it shows that modern AI systems aren't built through scaling alone. They're built through the combination of intelligence, alignment, interaction design, and deployment experience.

And the InstructGPT paper became one of the key foundations that made GPT-4 possible.

GPT-3 vs InstructGPT vs ChatGPT vs GPT-4: Key Differences

By this point, we've discussed GPT-3, InstructGPT, ChatGPT, and GPT-4 individually. But it can be helpful to see them side by side.

Although these systems are closely related, each one introduced a different shift in the evolution of modern AI.

GPT-3 focused on capability through scale, InstructGPT focused on alignment through human feedback, ChatGPT focused on conversational usability, and GPT-4 combined these ideas with stronger reasoning and multimodal capabilities.

The table below summarizes the main differences between them and shows how each system built on the progress of the previous generation.

From GPT-1 to GPT-4: A Timeline of Modern AI Systems and Alignment Evolution

Before we wrap up, it's worth stepping back and looking at the bigger picture.

The InstructGPT paper didn't emerge in isolation. It was part of a much larger evolution that transformed GPT models from research-focused language models into the conversational AI systems we use today.

Each generation introduced a new idea that pushed the field forward.

GPT-1 introduced large-scale pretraining, GPT-2 demonstrated zero-shot capabilities, GPT-3 popularized prompting and in-context learning, and InstructGPT introduced alignment through human feedback. ChatGPT then brought these ideas to millions of users through a conversational interface, while GPT-4 combined alignment with stronger reasoning and multimodal capabilities.

The timeline below summarizes the key transitions that shaped the modern AI era.

Final Insight

When people look back at the history of modern AI, they often focus on the moments when models became larger, more powerful, or more capable. But the story of the GPT series is not just a story about scale. It is also a story about learning how to make that intelligence useful.

GPT-1 showed that language models could learn surprisingly rich representations from large amounts of text before being adapted to specific tasks.

GPT-2 expanded that idea and revealed that scale itself could unlock new behaviors.

GPT-3 pushed the field into entirely new territory, demonstrating that a single model could perform a wide variety of tasks simply by responding to prompts and examples.

For a moment, it seemed as though scaling might be the answer to everything.

Then InstructGPT arrived and exposed a different challenge.

The problem was no longer whether a model could generate text, answer questions, or complete tasks. Models were already becoming remarkably capable.

The real question was whether people could actually rely on them. Could they follow instructions consistently? Could they respond in ways users found helpful? Could they become something more than sophisticated prediction engines?

That was the breakthrough at the heart of InstructGPT.

Rather than focusing solely on making models smarter, the paper focused on making them behave better.

Human feedback became part of the training process itself.

Alignment moved from a research concern to a core design principle. For the first time, improving the relationship between humans and AI became just as important as improving the model's raw capabilities.

The impact of that shift extended far beyond a single paper.

It laid the groundwork for ChatGPT, which introduced millions of people to conversational AI. Suddenly, interacting with advanced language models no longer required APIs, research expertise, or carefully engineered prompts. People could simply ask questions, seek advice, explore ideas, or learn something new through natural conversation.

That change transformed AI from a research breakthrough into a widely used product.

GPT-4 would later build on this foundation, combining stronger reasoning and broader capabilities with the alignment techniques that began with InstructGPT. But by then, the industry had already learned an important lesson: capability alone was not enough. Intelligence had to be usable.

In hindsight, the lasting significance of the InstructGPT paper is not that it introduced a new training pipeline. It is that it helped redefine the goal of modern AI.

The challenge was no longer just building systems that could generate language.

It was building systems that people could work with, learn from, and trust.

And that may ultimately be the transition that defined this era of artificial intelligence.

Resources:

• Pytorch Projects for GPT series

• Training Language Models to Follow Instructions with Human Feedback

• Language Models are Few-Shot Learners

• Learning to Summarize from Human Feedback

• Fine-Tuning Language Models from Human Preferences

• Deep Reinforcement Learning from Human Preferences

• Learning to Summarize with Human Feedback

• Aligning AI With Shared Human Values

• Asking for Help on Recursive Decomposition

• WebGPT: Browser-assisted Question-Answering with Human Feedback

• Constitutional AI: Harmlessness from AI Feedback

• TruthfulQA: Measuring How Models Mimic Human Falsehoods

• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

• The Power of Scale for Parameter-Efficient Prompt Tuning

• Finetuned Language Models Are Zero-Shot Learners

• Multitask Prompted Training Enables Zero-Shot Task Generalization

Contact Me

• Github

• X

• Linkedin

The pipeline included:

• proper subject-grouped train/validation splits

• robust preprocessing

• carefully decoded segmentation masks

• sensible loss functions

• consistent evaluation

And the model still struggled to learn.

The interesting part isn't that the model underperformed. What mattered was the diagnosis: a series of simple checks that traced the problem back to the dataset, not the model.

Those checks are useful far beyond medical imaging. They apply to almost any machine learning project.

If you're new to ML, this is a lesson worth carrying into every project: understand your data before you tune your model.

I set out to build a medical image segmentation tutorial. I ended up learning a more valuable lesson: no amount of careful engineering can rescue a model from a dataset that can't support the task.

By the end of this article, you'll understand:

• How to evaluate whether a dataset can actually support your task

• Why "the model isn't learning" is often a data problem

• How to rule out engineering bugs before blaming the data

• Practical diagnostics you can run in minutes

• Why synthetic training data often struggles in real-world deployment

• When to stop tuning and walk away from a dataset

This is not a beginner introduction to deep learning – it assumes familiarity with concepts like UNet architectures and training loops. But the data-quality lessons apply broadly to many ML projects.

What We'll Cover:

• The Dataset

• Step 1: Rule Out the Pipeline Before Blaming the Data

  • Subject-grouped splits

  • Decoding masks correctly

  • Loss design and class weighting

• Step 2: The Model Still Struggled

• Step 3: Interrogating the Dataset

  • Diagnostic 1: What Does the Dataset Actually Contain?

  • Diagnostic 2: Do Synthetic and Real Images Look Similar?

  • Diagnostic 3: Can the gap be fixed by adding real data?

• Step 4: Knowing When to Stop

• A Practical Dataset Evaluation Checklist

• What I Would Try Next

• The Bigger Lesson

The Dataset

I used the US Simulation & Segmentation dataset, a public collection of abdominal ultrasound images with organ segmentation labels from Kaggle.

It contains:

• 926 synthetic ultrasound images — generated by a ray-casting simulator from CT scans, with full organ annotations

• 617 real ultrasound images — from an actual ultrasound scanner

• Labels for 8 organs — liver, kidney, gallbladder, pancreas, spleen, bones, vessels, and adrenals

At first glance, the dataset looked ideal:

• thousands of images

• multiple organ classes

• both synthetic and real ultrasound data

Whether it actually supported the task was a different question.

Step 1: Rule Out the Pipeline Before Blaming the Data

Ground rule: you should always rule out the pipeline before blaming the data. A model failing on buggy code looks exactly like a model failing on bad data. The engineering needs to be trustworthy.

Subject-Grouped Splits

A common mistake in medical imaging is randomly splitting images into train and test sets.

That approach is problematic because many frames come from the same patient. Those frames share anatomy, scanner settings, and noise patterns.

If frames from the same patient appear in both the train and test sets, the model can partially memorize patient-specific patterns. Test scores look artificially good, even though the model may fail on truly unseen patients.

This is called subject leakage.

The fix is to split by patient instead of by image:

from sklearn.model_selection import GroupShuffleSplit

def assign_splits(manifest, val_fraction=0.15, seed=42):
    train_data = manifest[manifest["orig_split"] == "train"]
    groups = train_data["subject_id"].values

    gss = GroupShuffleSplit(n_splits=1, test_size=val_fraction, random_state=seed)
    train_idx, val_idx = next(gss.split(X=train_data, y=None, groups=groups))

    train_subjects = set(train_data.iloc[train_idx]["subject_id"].unique())
    val_subjects = set(train_data.iloc[val_idx]["subject_id"].unique())

    # Crash loudly if leakage ever sneaks in
    assert train_subjects.isdisjoint(val_subjects), "Subject leak detected!"
    return train_subjects, val_subjects

That assertion matters. If the split logic ever breaks, the pipeline fails loudly instead of silently producing misleading metrics.

Decoding Masks Correctly

The dataset stores labels as color-coded masks. Each organ corresponds to a different RGB color.

Training requires converting those colors into integer class labels.

A naïve implementation uses exact color matching, but resizing operations can slightly alter colors at mask boundaries.

A more robust approach maps each pixel to its nearest palette color:

import numpy as np

PALETTE = np.array([
    [0, 0, 0],
    [100, 0, 100],
    [255, 255, 255],
    [0, 255, 0],
    [255, 255, 0],
    [0, 0, 255],
    [255, 0, 0],
    [255, 0, 255],
    [0, 255, 255],
], dtype=np.int32)

def decode_mask(mask_rgb):
    h, w = mask_rgb.shape[:2]
    flat = mask_rgb.reshape(-1, 3).astype(np.int32)
    d2 = (
        (flat[:, None, :] - PALETTE[None, :, :]) ** 2
    ).sum(-1)
    classes = d2.argmin(axis=1).astype(np.uint8)
    return classes.reshape(h, w)

Before training, it’s worth visually checking a few decoded masks against the original images. This catches issues like incorrect palettes, RGB/BGR channel swaps, or resizing artifacts that silently corrupt labels.

These bugs rarely throw errors. Instead, the model simply learns poorly. And “trained on wrong labels” looks exactly like “the model can’t learn the data.”

Verifying masks early removes that uncertainty.

Loss Design and Class Weighting

For training, I usd standard MONAI segmentation losses. The goal wasn’t to aggressively maximize performance, but to establish a stable and trustworthy baseline.

The training curves below show that the model optimized normally: the loss decreased consistently, and the validation dice stabilized rather than diverging. This helped rule out optimization instability as the primary cause of poor final performance.

Three choices were deliberate:

• Dice + Cross-Entropy combined: Cross-entropy keeps learning stable early on – Dice directly rewards good region overlap. Together they balance each other.

• include_background=False for binary segmentation: In a single-organ task, background can be 85–90% of the pixels. Counting it in the loss drowns out the signal for the organ you actually care about, so it's better left out.

• Class weighting for multi-class segmentation: With organs of very different sizes, an unweighted loss lets the model ignore the small, rare ones and still score well. Weighting rare-class mistakes more heavily pushes back against that.

Step 2: The Model Still Struggled

The first experiment focused on liver segmentation — the simplest single-organ task in the dataset.

Dice scores range from 0 (no overlap) to 1 (perfect overlap).

Qualitatively, the predictions often captured rough liver regions but failed at boundaries and consistency across real scans.

Especially important:

• the model struggled even on synthetic in-domain data

• performance dropped further on real ultrasound images

At this point, two explanations were possible:

1. the model or pipeline was flawed

2. the dataset itself was limiting performance

Because the engineering had been carefully validated, the second possibility became worth investigating seriously.

That's where the real lesson began.

Step 3: Interrogating the Dataset

Rather than endlessly tuning the model, the productive move is to turn the diagnostic lens on the dataset.

Three simple checks revealed the real problem. None required retraining or expensive experiments.

Diagnostic 1: What Does the Dataset Actually Contain?

The first step was simply plotting the dataset composition.

• 926 labeled synthetic images (the bulk of training data)

• Only 60 labeled real images — less than 4% of the dataset

• 557 unlabeled real images — real data exists, but without labels it can't be used for supervised training

This immediately changed the interpretation of the dataset.

Although the dataset contains many real ultrasound scans, almost all labeled training data is synthetic.

The model is effectively trained on synthetic ultrasound and expected to generalize to real ultrasound.

That's a difficult transfer problem from the start.

The limitation is simple: the real images mostly don't have labels, so supervised training has very little real-world data to learn from.

Lesson: Before training anything, chart the dataset composition. A headline image count can be misleading. "1,500 images" sounds large until you discover that only a tiny fraction are labeled examples from the target domain.

Diagnostic 2: Do Synthetic and Real Images Look Similar?

The next question was whether the synthetic and real ultrasound images actually followed similar visual distributions.

Plotting intensity histograms showed a clear mismatch.

• synthetic images clustered heavily near darker intensities

• real ultrasound images had broader mid-range intensity distributions

The synthetic simulator captured anatomical geometry reasonably well, but it didn't reproduce the texture and noise characteristics of real ultrasound:

• speckle patterns

• intensity falloff

• scanner-specific artifacts

This is the classic synthetic-to-real domain gap.

The model learned features tuned to synthetic images and then encountered a substantially different distribution during evaluation. Poor transfer performance became expected rather than surprising.

Lesson: Whenever training and deployment happen on different domains — synthetic → real, scanner A → scanner B, hospital A → hospital B — measure the distribution shift directly. Simple histogram comparisons can reveal major problems in minutes.

Diagnostic 3: Can the gap be fixed by adding real data?

The obvious next idea was: why not include some real labeled data during training?

But before implementing that approach, it's worth checking how many distinct patients actually had labels.

Labeled real images: 60
Distinct subjects (labeled real): 4

Frames per subject:
  subject h: 26
  subject a: 16
  subject g: 10
  subject b: 8

Only four patients.

That result fundamentally changed the situation.

Proper medical imaging evaluation requires subject-grouped train/test splits. But with only four patients, any evaluation becomes statistically unstable.

Training on two or three patients and testing on one or two patients would produce highly unreliable metrics that depend heavily on which patient happened to be held out.

At that point, the dataset simply couldn't support trustworthy real-world evaluation.

Lesson: In medical imaging, count subjects, not images. The true size of a dataset is bounded by the number of independent patients, not the number of files.

Step 4: Knowing When to Stop

At this point, additional tuning no longer made sense.

The bottleneck was not the architecture, optimizer, or learning rate. The bottleneck was the dataset itself.

The pipeline was still valuable and reusable. But this particular dataset couldn't reliably support the intended segmentation task.

That distinction matters: sometimes a problem is difficult but solvable, and sometimes the data simply can't support the conclusion you want to draw.

Learning to recognize the difference is an important ML skill.

A Practical Dataset Evaluation Checklist

Before committing weeks to model development, these checks are worth running on any dataset:

1. Chart the dataset composition — labeled vs unlabeled, class distribution, domain distribution

2. Count subjects, not images — independent patients matter more than frame count

3. Check class balance — rare classes are often ignored without weighting or sampling strategies

4. Compare train and deployment distributions — especially for cross-domain problems

5. Verify labels visually — catch preprocessing or annotation errors early

6. Look for published baselines — low published performance may indicate dataset limitations

These checks take minutes and can save weeks of unnecessary tuning.

What I Would Try Next

Improving results would likely require better data rather than a larger model. The next steps I'd prioritize:

• collecting more labeled real ultrasound scans, from more distinct patients

• improving annotation consistency

• semi-supervised learning to make use of the unlabeled real images

• domain adaptation between synthetic and real ultrasound

All of these target the actual bottleneck: data quality and data diversity.

The Bigger Lesson

In machine learning, it's easy to focus most of our attention on architectures, hyperparameters, optimization tricks, and newer models.

But the dataset quietly defines the ceiling.

A sophisticated model on weak data often disappoints, while a simpler model on strong data performs surprisingly well.

That was the real lesson from this project.

The most valuable skill wasn't building the pipeline. It was diagnosing why the model couldn't succeed and being willing to trust what the data was saying.

The workflow — checking dataset composition, counting subjects, comparing distributions, ruling out engineering bugs, and deciding when to stop — transfers to almost any ML project.

In many projects, better judgment about the data matters more than a better model.

The pipeline code and diagnostic notebooks are available at the MONAI Ultrasound Working Group repository. Questions, corrections, and improvements are always welcome.

That idea eventually led to prompt engineering, AI assistants, and the first wave of large language model applications.

But GPT-4 felt different.

GPT-3 still felt like a research breakthrough: powerful, experimental, and sometimes unpredictable. GPT-4, on the other hand, felt like the beginning of a real AI platform. The focus was no longer just on scaling language models to achieve better benchmarks. Instead, the conversation shifted toward reliability, multimodal understanding, alignment, safety, and real-world deployment.

This change is visible throughout the GPT-4 Technical Report released by OpenAI.

Unlike the earlier GPT papers, OpenAI didn't publish a traditional research paper with detailed architecture diagrams, parameter counts, datasets, or training configurations. Instead, they released a more limited technical report focused primarily on capabilities, evaluations, safety work, and deployment considerations.

That decision itself reflects how much the field had changed.

By the time GPT-4 arrived, large language models were no longer just research projects used inside labs. They had become globally deployed systems used by millions of people through products like ChatGPT. Questions about misuse, hallucinations, bias, cybersecurity risks, and alignment were now just as important as raw model performance.

GPT-4 also introduced another major shift: multimodality.

Previous GPT models worked only with text. GPT-4 expanded this idea by accepting both images and text as input, allowing the model to analyze screenshots, diagrams, documents, visual jokes, and other mixed forms of information. This pushed large language models closer to more general-purpose AI systems rather than narrow text generators.

Historically, the progression becomes surprisingly clear:

• GPT-1 introduced pretraining and transfer learning

• GPT-2 introduced zero-shot multitask learning

• GPT-3 introduced few-shot prompting and in-context learning

• GPT-4 introduced the era of aligned, multimodal AI systems

In many ways, GPT-4 marks the moment when large language models stopped being viewed primarily as research experiments and started becoming foundational computing interfaces for real-world applications.

Paper Overview

In this article, we’ll review the GPT-4 Technical Report published by Open AI in 2023.

Many important technical details were intentionally omitted from this report, including:

• parameter count

• exact architecture

• training compute

• dataset composition

• hardware configuration

According to OpenAI, these limitations were introduced partly because of the competitive landscape and the growing safety implications surrounding large-scale AI systems.

That difference is historically important.

The GPT-1, GPT-2, and GPT-3 papers openly discussed architecture scaling, datasets, and training methodology in significant detail. GPT-4 marks a noticeable shift toward more restricted disclosure as language models became commercially valuable and widely deployed.

You can read the original report here:

GPT-4 Technical Report

And here’s a quick infographic of what we’ll cover throughout this review:

Table of Content:

• Executive Summary

• Goals of the Report

• Core Idea

• Predictable Scaling

• Model Architecture

• Multimodal Learning

• Fine-Tuning vs Zero-Shot vs Few-Shot vs Aligned Multimodal Learning

• RLHF and Alignment

• Benchmarks and Experiments

• Coding and Reasoning Ability

• Multilingual Capabilities

• Emergent Behavior

• Limitations

• Safety and Risks

• Discussion

• Conclusion

• Final Insight

• GPT-1 vs GPT-2 vs GPT-3 vs GPT-4: Key Differences

• PyTorch Implementations of the GPT Architecture Evolution

• Resources:

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with some of the core ideas behind modern language models.

Reading the earlier reviews in this series will be especially useful:

• AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1)

• AI Paper Review: Language Models are Unsupervised Multitask Learners (GPT-2)

• AI Paper Review: Language Models are Few-Shot Learners (GPT-3)

GPT-4 builds directly on many of the concepts introduced in those papers, especially large-scale pretraining, zero-shot and few-shot learning, and in-context prompting.

It also helps to have a general understanding of:

• Transformer architectures and self-attention

• The evolution from GPT-1 → GPT-3

• Few-shot learning and prompting

• Basic prompt engineering concepts

• Reinforcement Learning from Human Feedback (RLHF)

• Scaling laws and why larger models often develop new capabilities

You don't need deep mathematical knowledge to follow this article, though.

As with the previous reviews, I’ll focus more on explaining the ideas intuitively and practically rather than diving too deeply into heavy equations or dense academic terminology.

Executive Summary

GPT-4 is not simply a larger version of GPT-3.

That may sound obvious today, but at the time, many people initially assumed GPT-4 was just another scaling step in the same direction. But the technical report shows something more important: GPT-4 represents a shift from experimental language models toward deployable general-purpose AI systems.

According to the report, GPT-4 introduces several major advances at once.

First, as mentioned above, the model becomes multimodal. Unlike previous GPT systems that only worked with text, GPT-4 can process both images and text as input while still generating text outputs. This allows the model to analyze screenshots, diagrams, documents, photographs, visual jokes, and mixed media prompts.

Second, GPT-4 demonstrates significantly stronger reasoning and benchmark performance across a wide range of professional and academic evaluations. The report shows GPT-4 achieving near human-level results on exams including the Uniform Bar Exam, LSAT, GRE, SAT, AP tests, coding benchmarks, and advanced reasoning tasks.

The report also places heavy emphasis on alignment and factuality improvements.

Earlier GPT systems often produced unsafe, misleading, or overly confident outputs. GPT-4 still has these problems, but OpenAI invested heavily in reinforcement learning from human feedback (RLHF), adversarial testing, refusal behavior, and safety evaluation pipelines to reduce harmful behavior and improve adherence to user intent.

Another major theme throughout the report is predictable scaling.

According to the authors, OpenAI developed infrastructure and optimization methods that allowed them to accurately predict GPT-4’s final performance using much smaller training runs.

That detail matters more than it might seem.

GPT-3 demonstrated that scaling works. GPT-4 demonstrates that scaling large language models was becoming an engineering discipline with increasingly predictable behavior.

The broader implication is what makes this report historically important.

GPT-4 transforms large language models from research demonstrations into deployable AI assistants capable of reasoning across many domains, interacting through natural language, following instructions more reliably, and operating at global scale through systems like ChatGPT.

In many ways, this report marks the beginning of the modern AI deployment era.

Goals of the Report

The GPT-4 Technical Report is not only about showing a more capable language model. In many ways, the report is about demonstrating that large AI systems can be developed more reliably, more safely, and more predictably than before.

One of the main goals behind GPT-4 was improving reasoning and reliability across a broad range of tasks, which we discussed above.

Another major objective was improving alignment with user intent – investing in RLHF, safety fine-tuning, refusal training, and adversarial testing to make the model more helpful and better aligned with intended behavior.

The report also marks a significant shift beyond text-only AI systems, as GPT-4 introduces multimodal capabilities. This expands the system from being purely a language generator into something closer to a general-purpose reasoning interface capable of interpreting visual and textual information together.

Safety is another central theme throughout the report.

OpenAI repeatedly emphasizes efforts to reduce harmful outputs, improve refusal behavior, mitigate misuse risks, and build safer deployment systems around the model. The report discusses red teaming, domain expert testing, policy enforcement, and model-assisted safety pipelines designed to reduce dangerous behavior during real-world usage.

But one of the most historically important goals may actually be predictability.

According to the authors, GPT-4 was developed using infrastructure and optimization methods designed to scale in highly predictable ways. OpenAI claims they could estimate aspects of GPT-4’s final performance using models trained with thousands of times less compute.

That idea may sound technical, but it represents a major shift in how frontier AI systems were being built.

Earlier generations of language models often involved substantial uncertainty during scaling. GPT-4 suggests that large-scale AI development was becoming more systematic and engineering-driven rather than purely experimental.

In practice, the report reflects a broader transition happening across the AI industry, from research prototypes to deployable infrastructure systems designed for real-world use at massive scale.

Core Idea

One of the most surprising things about GPT-4 is that, underneath all the hype and new capabilities, the core learning objective is still fundamentally very simple.

Like GPT-1, GPT-2, and GPT-3, GPT-4 is still trained primarily as a next-token prediction model. In other words, the system learns by repeatedly predicting the next piece of text in a sequence.

The architecture also remains Transformer-based and autoregressive.

That means GPT-4 generates outputs one token at a time while using self-attention to understand relationships between words, sentences, images, and context inside the input sequence.

At a high level, the underlying principle hasn't changed very much since GPT-2:

• train on massive amounts of data

• predict the next token

• scale the model aggressively

But GPT-4 pushes this approach much further.

According to the report, the model is substantially larger, more optimized, and trained using infrastructure designed specifically for predictable large-scale behavior.

The biggest conceptual change is that GPT-4 is no longer limited to text-only input.

Another major difference is the importance of post-training alignment.

GPT-3 already demonstrated strong few-shot learning abilities, but GPT-4 places much heavier emphasis on reinforcement learning from human feedback (RLHF), safety tuning, refusal behavior, and instruction following. According to the report, these post-training processes significantly improve factuality, adherence to desired behavior, and response safety.

This leads to one of the most important ideas behind modern AI systems:

Capability doesn't emerge from scale alone.

GPT-4 suggests that powerful AI behavior comes from the combination of:

• large-scale pretraining

• scaling laws

• optimization improvements

• alignment training

• RLHF

• post-training refinement

In practice, GPT-4 feels less like a raw predictive model and more like an interactive assistant because of this additional alignment layer.

That distinction matters historically.

GPT-3 showed that scaling language models could unlock powerful emergent behavior. GPT-4 shows that scaling alone is not enough — the model also needs alignment, safety training, and deployment-focused refinement to become broadly usable in the real world.

Predictable Scaling

One of the most important ideas in the GPT-4 Technical Report is something that many people overlooked when the paper first came out: predictable scaling.

Earlier generations of large language models involved a huge amount of uncertainty.

Researchers could train larger systems and hope performance would improve, but nobody fully knew how far scaling would go or whether massive training runs would behave the way they expected.

GPT-4 changed that. According to the report, OpenAI developed infrastructure and optimization methods that allowed them to accurately predict GPT-4’s final training loss, and even some capabilities, using models trained with thousands of times less compute.

This is far more important than it first sounds. GPT-3 proved that scaling language models works.

GPT-4 suggested that scaling was starting to become predictable engineering rather than trial-and-error experimentation.

That shift introduced several major advantages:

• Better capability forecasting before training massive models

• Reduced risk of wasting millions of dollars on failed training runs

• Safer deployment planning through earlier evaluation of model behavior

• More reliable scaling from small experiments to frontier-scale systems

The report also shows that model loss followed remarkably stable power-law behavior across scales, allowing OpenAI to estimate GPT-4’s final performance long before training finished.

But the paper also makes an important point: not every capability scales smoothly. Some behaviors, especially reasoning-related tasks, can emerge unpredictably or even temporarily worsen before improving again.

Some important limitations of predictable scaling include:

• Some capabilities still emerge unpredictably at larger scales

• Benchmark performance can behave nonlinearly instead of improving smoothly

• Scaling laws may not hold forever as models continue growing

• Even with predictable training curves, reasoning failures and hallucinations can still appear unexpectedly

That tension between predictable scaling and unexpected emergence became one of the defining themes of modern frontier AI research.

Model Architecture

One of the most unusual aspects of the GPT-4 Technical Report is how little OpenAI reveals about the actual model architecture.

As discussed above, in the GPT-1, GPT-2, and GPT-3 papers, OpenAI openly discussed details like parameter counts, dataset sizes, scaling configurations, and training methodology.

As you now know, GPT-4 is very different. The report leaves out several major technical details like the exact parameter count, the precise architecture configuration, the dataset size and composition, the training compute used, and the hardware infrastructure and setup.

The report explicitly states that these omissions were motivated by both the competitive landscape and safety considerations surrounding large-scale AI systems.

That decision became one of the most discussed aspects of the release.

Historically, GPT-4 marks a transition where frontier AI research started becoming more closed and product-oriented. Earlier GPT papers felt like traditional research publications. GPT-4 feels more like a controlled systems report from a company deploying AI at global scale.

Even though many implementation details remain hidden, the report still confirms several important things:

1. GPT-4 is still fundamentally a Transformer-based model trained using autoregressive next-token prediction.

2. Like previous GPT systems, it generates outputs sequentially while using self-attention mechanisms to process context.

3. GPT-4 is multimodal, meaning it can accept both image and text inputs while producing text outputs.

This is one of the biggest architectural shifts in the GPT series because it extends the model beyond pure language understanding into combined visual and textual reasoning.

Another important component is post-training alignment, which we've already discussed a bit. In practice, it means that GPT-4 isn't just a raw pretrained language model anymore. It's a heavily refined system built through multiple stages:

• large-scale pretraining

• optimization and scaling improvements

• multimodal integration

• RLHF alignment

• safety fine-tuning

• deployment-oriented post-training

The secrecy surrounding GPT-4’s architecture is historically important because it reflects a broader change happening in AI.

As language models became commercially valuable and socially impactful, frontier AI research started moving away from full openness toward controlled disclosure, safety-focused deployment, and competitive protection.

Multimodal Learning

One of the most important breakthroughs in GPT-4 is that the model is no longer limited to text alone. GPT-4 can accept both images and text as input while generating text outputs.

That may sound simple today, but at the time, this represented a major shift in how people thought about large language models.

Earlier GPT systems worked purely with language. GPT-4 expands the idea into something much broader: a model capable of reasoning across multiple forms of information at the same time.

In practice, GPT-4 can analyze:

• screenshots

• diagrams

• photographs

• documents

• charts

• visual jokes and memes

• mixed image-and-text prompts

The report demonstrates this capability through several examples, but one became especially memorable: the famous VGA cable meme example.

In the image, a smartphone appears connected to a massive VGA monitor cable adapter – something clearly absurd in real life. GPT-4 correctly explains that the humor comes from the mismatch between outdated VGA hardware and a modern phone charging port.

What made this example important was not just object recognition. The model was interpreting contextual humor from a visual scene.

That distinction matters.

Traditional computer vision systems could often identify objects inside images, but GPT-4 demonstrated something closer to multimodal reasoning: understanding relationships, context, intent, and even jokes across combined visual and textual information.

The report also notes that many prompting techniques developed for language models (including few-shot prompting and chain-of-thought reasoning) continue working effectively in multimodal settings.

This suggests that GPT-4 is not simply attaching an image classifier onto a chatbot. Instead, the model appears to integrate visual and language understanding into a more unified reasoning system.

Historically, this was a major moment for the GPT series.

• GPT-1 focused on language pretraining

• GPT-2 expanded zero-shot capabilities

• GPT-3 introduced in-context learning

• GPT-4 publicly demonstrated practical multimodal AI

And unlike many earlier research demos, GPT-4’s multimodal abilities were not just experimental prototypes hidden inside papers. They became part of real-world products used by millions of people.

That shift made multimodal AI feel practical and deployable rather than purely theoretical.

Fine-Tuning vs Zero-Shot vs Few-Shot vs Aligned Multimodal Learning

One of the clearest ways to understand how GPT models evolved is by comparing how they learn and adapt to tasks.

Earlier NLP systems relied heavily on fine-tuning with labeled datasets, while later GPT models increasingly shifted toward zero-shot prompting, few-shot learning, and eventually aligned multimodal interaction.

The table below summarizes how these approaches differ in flexibility, training requirements, scalability, and real-world usability.

RLHF and Alignment

One of the biggest differences between GPT-4 and earlier GPT models is how much emphasis the report places on alignment and safety.

GPT-3 demonstrated impressive few-shot learning abilities, but it also exposed serious weaknesses. The model could hallucinate facts, generate harmful instructions, confidently produce false information, or fail to follow user intent reliably.

GPT-4 was designed with these problems in mind.

A major part of this improvement comes from Reinforcement Learning from Human Feedback (RLHF).

At a high level, RLHF works by collecting human feedback about model responses and then using that feedback to train the model toward preferred behavior. Instead of learning only from internet text, the system also learns from human judgments about what kinds of answers are helpful, safe, accurate, or appropriate.

According to the report, GPT-4 undergoes extensive post-training alignment designed to improve:

• factuality

• instruction following

• refusal behavior

• harmlessness

• adherence to user intent

This alignment layer is a major reason GPT-4 feels different from raw pretrained language models.

The report repeatedly emphasizes refusal behavior as an important safety capability.

Earlier versions of GPT-4 could sometimes generate dangerous instructions, including harmful chemical synthesis advice or weapon-related content during internal testing. OpenAI used adversarial testing, domain experts, RLHF training, and additional safety pipelines to reduce these behaviors significantly.

The examples shown in the report are especially revealing.

In one case, an earlier GPT-4 version provided detailed responses about creating dangerous materials. Later aligned versions instead refuse the request and redirect the conversation safely.

What makes this important is that GPT-4 is not simply being made “more restrictive.”

The report also discusses the opposite problem: models becoming too cautious. OpenAI specifically worked on reducing unnecessary refusals for harmless requests while still blocking dangerous ones.

In practice, alignment becomes a balancing act between:

• usefulness

• safety

• honesty

• flexibility

• and reliability

The paper also introduces rule-based reward models and model-assisted safety pipelines that help guide GPT-4 toward safer behavior during training.

Historically, this section of the report marks another major transition in AI development.

Earlier GPT papers focused primarily on capabilities and scaling. GPT-4 treats alignment and deployment safety as core engineering problems rather than secondary concerns.

That shift reflects a deeper realization across the industry: once AI systems become powerful enough for real-world deployment at global scale, improving intelligence alone is no longer enough. The systems also need to behave safely, follow human intent reliably, and resist harmful misuse.

Benchmarks and Experiments

One of the most striking parts of the GPT-4 Technical Report is the sheer scale of the evaluation process.

According to the report, OpenAI tested GPT-4 across a wide range of academic exams, professional certifications, reasoning tasks, coding benchmarks, and traditional NLP evaluations.

The goal was not simply to show that GPT-4 could generate fluent text. The evaluations were designed to measure whether the model could reason, solve problems, follow instructions, answer questions, and generalize across many different domains.

The human exam results attracted enormous attention when the report was released.

GPT-4 achieved particularly strong scores on several well-known exams:

• Uniform Bar Exam → around the top 10% of test takers

• LSAT → roughly 88th percentile

• SAT Reading & Writing → around 93rd percentile

• GRE Verbal → around the 99th percentile

• Strong performance across many AP exams

GPT Performance on Academic and Professional Exams

The table below summarizes GPT-4’s performance across a wide range of academic and professional exams, showing how the model compared with GPT-3.5 on tests such as the Uniform Bar Exam, LSAT, GRE, SAT, AP exams, and coding challenges.

Source: GPT-4 Technical Report (OpenAI, 2023), Table 1.

The comparison with GPT-3.5 was especially dramatic in some cases. For example, the report notes that GPT-3.5 scored near the bottom 10% on the simulated bar exam, while GPT-4 reached the top 10%.

These results helped change public perception of large language models.

Earlier systems were often viewed mainly as autocomplete engines or text generators. GPT-4 demonstrated that scaling and alignment could produce systems capable of performing competitively on many tasks originally designed for humans.

The figure below visualizes GPT-4’s percentile rankings across multiple exams, highlighting the significant improvement over GPT-3.5 in areas such as reasoning, language understanding, mathematics, and professional testing.

Source: GPT-4 Technical Report (OpenAI, 2023), Figure 4.

The report also evaluates GPT-4 on a wide collection of standard NLP benchmarks.

Some of the most important include:

• MMLU → broad academic and professional reasoning benchmark

• HellaSwag → commonsense reasoning

• HumanEval → coding and Python synthesis tasks

• GSM8K → grade-school mathematics reasoning

• ARC → science reasoning questions

• WinoGrande → pronoun and commonsense reasoning

Across most of these evaluations, GPT-4 substantially outperforms GPT-3.5 and often surpasses previous state-of-the-art language models. In several cases, it even exceeds systems that relied on benchmark-specific fine-tuning or specialized engineering pipelines.

One especially important benchmark is MMLU (Massive Multitask Language Understanding), which tests knowledge and reasoning across 57 different subjects. GPT-4 achieves remarkably strong performance on this benchmark, including multilingual variants translated into many languages.

The coding evaluations are also historically significant. On HumanEval and LeetCode-style tasks, GPT-4 demonstrates major improvements in code generation and problem solving compared to earlier GPT systems.

This capability eventually became one of the foundations behind modern AI coding assistants.

The table below compares GPT-4 with previous language models and state-of-the-art systems on major AI benchmarks such as MMLU, HellaSwag, ARC, HumanEval, and GSM-8K, demonstrating the model’s strong performance across reasoning, coding, and language understanding tasks.

Source: GPT-4 Technical Report (OpenAI, 2023), Table 2.

What makes these experiments especially important is that GPT-4 performs well across many different categories simultaneously:

• reasoning

• coding

• mathematics

• language understanding

• professional exams

• multilingual tasks

• commonsense reasoning

That breadth is part of what made GPT-4 feel qualitatively different from earlier systems.

Instead of excelling in one narrow benchmark, GPT-4 demonstrated increasingly general behavior across a wide variety of intellectual tasks.

Coding and Reasoning Ability

One of the areas where GPT-4 shows some of its most noticeable improvements over earlier models is coding and structured reasoning.

While GPT-3 was already capable of generating code, GPT-4 pushes these abilities much further. According to the report, the model demonstrates substantial gains on programming benchmarks, mathematical reasoning tasks, and multi-step problem solving.

A key benchmark highlighted in the report is HumanEval, which measures the model’s ability to generate working Python functions from natural language descriptions.

GPT-4 achieves significantly higher performance than GPT-3.5 on this benchmark, showing much stronger code synthesis and problem-solving ability.

The report also includes LeetCode-style evaluations across easy, medium, and hard programming problems.

Although GPT-4 still struggles with many difficult competitive programming tasks, it performs substantially better than GPT-3.5, especially on easier and medium-level coding challenges.

These improvements became extremely important in practice.

Around the release of GPT-4, AI coding assistants started becoming genuinely useful for real software development workflows. Systems built on GPT-4 could help developers:

• generate functions

• explain code

• debug errors

• refactor implementations

• write documentation

• solve algorithmic problems

This was one of the first moments where large language models began functioning as practical engineering tools rather than experimental demos.

The report also highlights the importance of chain-of-thought prompting for reasoning tasks.

Instead of forcing the model to produce an immediate answer, chain-of-thought prompting encourages GPT-4 to reason step by step before reaching a conclusion.

For example, on benchmarks like GSM8K (a dataset of grade-school mathematics problems), GPT-4 performs much better when allowed to generate intermediate reasoning steps.

This became another major shift in how people interacted with large language models. Earlier systems were often treated like direct answer generators. GPT-4 demonstrated that prompting the model to “think through” a problem could significantly improve performance on reasoning-heavy tasks.

Compared to GPT-3.5, GPT-4 consistently shows stronger reasoning across many domains:

• coding

• mathematics

• structured problem solving

• commonsense reasoning

• academic evaluations

Of course, the model is still far from perfect.

The report repeatedly notes that GPT-4 can still hallucinate, make logical mistakes, fail at complex reasoning chains, or confidently produce incorrect solutions.

But historically, this section of the report matters because it helped establish a new category of AI applications: large language models as interactive reasoning and coding assistants.

That idea quickly became one of the defining use cases of modern AI systems.

Multilingual Capabilities

One of the more underrated aspects of the GPT-4 Technical Report is how strongly the model performs across multiple languages.

Earlier language models were often heavily English-centric. Even when multilingual support existed, performance in lower-resource languages usually dropped significantly compared to English benchmarks.

GPT-4 shows noticeable progress in this area.

To evaluate multilingual reasoning ability, OpenAI translated the MMLU benchmark – a broad academic and professional reasoning benchmark covering 57 subjects – into many different languages using machine translation systems.

According to the report, GPT-4 performs extremely well across most tested languages and even surpasses the English-language performance of earlier models in many cases.

What makes this especially important is that the improvements are not limited to high-resource languages like French, German, or Spanish.

The report specifically highlights strong performance gains in lower-resource languages such as:

• Latvian

• Welsh

• Swahili

• Bengali

• Nepali

• Marathi

• Telugu

This suggests something important about large-scale language modeling: as models scale and training data becomes more diverse, the learned capabilities start generalizing beyond English in a much more robust way.

In other words, the scaling effects observed in GPT-3 were not purely English-language phenomena.

GPT-4 demonstrates that many reasoning and language understanding capabilities can transfer across languages, even when available training data is far more limited.

This is historically significant because it moves large language models closer to becoming globally useful systems rather than tools optimized mainly for English-speaking users.

The multilingual results also reinforce another major theme throughout the report: GPT-4 is not narrowly specialized for a single domain or benchmark. Instead, it behaves increasingly like a general-purpose reasoning system capable of adapting across:

• languages

• tasks

• modalities

• domains

• and interaction styles

Of course, multilingual performance is still uneven.

The report doesn't claim perfect fluency or equal reasoning quality across all languages. Lower-resource languages still present major challenges, and evaluation itself remains difficult in many multilingual settings.

But compared to earlier GPT systems, GPT-4 demonstrates a substantial step forward in multilingual generalization. And that became an important milestone for globally deployed AI systems.

Emergent Behavior

One of the most fascinating ideas surrounding GPT-4 is the concept of emergent behavior.

In the context of large language models, emergence refers to abilities that appear unexpectedly as models become larger and more capable. Instead of improving smoothly in every area, some skills seem to “switch on” once the model reaches a certain scale.

GPT-3 already hinted at this phenomenon through few-shot learning and in-context adaptation. GPT-4 continues that trend much more strongly.

According to the report, many capabilities improve nonlinearly as scale increases.

In simpler terms, doubling the size or compute of a model doesn't just make it slightly better at the same tasks. Sometimes, entirely new behaviors emerge that were weak or mostly absent in smaller systems.

This becomes especially visible in reasoning tasks.

GPT-4 demonstrates major improvements over GPT-3.5 in coding, mathematical reasoning, academic evaluations, instruction following, and structured problem solving.

The report also highlights how prompting strategies become more effective at larger scales.

Few-shot prompting (where the model learns from examples inside the prompt) works far more reliably in GPT-4 than in earlier systems. Similarly, chain-of-thought prompting becomes significantly more useful for reasoning-heavy tasks.

Instead of immediately generating an answer, GPT-4 can often improve performance by reasoning step by step through a problem.

What makes this important is that these abilities weren't explicitly programmed into the system. The model was still trained primarily through next-token prediction. Yet at sufficient scale, behaviors like:

• multi-step reasoning

• code synthesis

• contextual adaptation

• multilingual generalization

• instruction following

• and visual-text reasoning

began appearing much more robustly.

The report’s discussion of predictable scaling also connects directly to this idea. OpenAI explains that GPT-4’s capabilities could often be estimated from smaller training runs using scaling laws.

At the same time, some behaviors remain difficult to predict cleanly. The paper even notes cases where certain tasks improve unexpectedly or reverse earlier scaling trends as models become larger.

Historically, GPT-4 reinforces one of the biggest lessons from the GPT series: large language models don't simply become more fluent as they scale. They begin exhibiting qualitatively different behaviors.

That realization fundamentally changed AI research. Instead of treating language models as narrow NLP systems, researchers increasingly started viewing them as general-purpose learning systems whose capabilities could continue emerging with scale, alignment, and better training methods.

Limitations

Despite the impressive benchmark results and multimodal capabilities, the GPT-4 Technical Report is surprisingly direct about the model’s weaknesses.

The paper repeatedly emphasizes that GPT-4 is still not fully reliable.

One of the biggest problems is still hallucination.

Like earlier GPT systems, GPT-4 can confidently generate information that's incorrect, fabricated, or misleading. The model may produce answers that sound highly convincing even when the underlying facts are wrong.

This becomes especially dangerous because GPT-4 is often more fluent and persuasive than previous models. In practice, stronger language generation can sometimes make mistakes harder for users to notice.

The report also discusses reasoning failures.

Although GPT-4 performs much better than GPT-3.5 across many benchmarks, it can still fail at relatively simple logical tasks, make arithmetic mistakes, or break down during longer reasoning chains.

Another important limitation is overconfidence.

GPT-4 doesn't naturally “know when it does not know.” The model can present uncertain or incorrect answers with a high degree of confidence, which creates risks in high-stakes situations like medicine, law, education, or cybersecurity.

The report also notes that GPT-4 has a knowledge cutoff. Most of the model’s training data ends around September 2021, meaning the system lacks reliable awareness of many events that happened afterward.

One particularly interesting section discusses calibration.

According to the report, the pretrained GPT-4 model was actually fairly well calibrated – meaning its confidence often matched the probability of correctness. But post-training alignment and RLHF reduced calibration quality in some cases.

This reveals an important tradeoff: making models more helpful and aligned doesn't automatically make them more truthful or better calibrated.

The paper is also honest about bias and unsafe behavior.

Because GPT-4 learns from large internet-scale datasets, it can still reflect social biases, stereotypes, and problematic patterns present in training data.

OpenAI discusses extensive efforts to reduce harmful outputs, but the report explicitly acknowledges that unsafe behavior is still possible.

One example is jailbreaking: attempts to bypass safety mechanisms using adversarial prompts or clever conversational manipulation. According to the report, GPT-4’s safety systems reduce harmful behavior significantly, but determined users can still sometimes elicit dangerous or policy-violating outputs.

The paper also emphasizes that GPT-4 should not be blindly trusted in high-risk environments without additional safeguards, human oversight, or verification systems.

That honesty is one reason the report remains important: instead of presenting GPT-4 as a solved form of intelligence, OpenAI frames it as a powerful but imperfect system whose growing capabilities also create growing risks.

Historically, this reflects a major shift in AI research culture.

Earlier papers focused mostly on increasing performance. GPT-4 places equal emphasis on capability and failure modes, because once models become widely deployed, understanding limitations becomes just as important as demonstrating strengths.

Safety and Risks

One of the clearest signs that the AI field had changed by the time GPT-4 was released is how much of the report is dedicated to safety, risk analysis, and deployment concerns.

Earlier GPT papers focused primarily on capability improvements, scaling behavior, and benchmark performance. The GPT-4 Technical Report still discusses those topics, but safety becomes a central engineering theme rather than a secondary discussion.

According to the report, OpenAI conducted extensive red teaming and adversarial testing before deployment.

Red teaming involves intentionally trying to break the system, bypass safeguards, trigger unsafe outputs, or expose dangerous behaviors. OpenAI worked with external domain experts to evaluate risks across areas like cybersecurity, misinformation, chemistry, and biological threats.

This type of testing reflects a major shift in mindset.

The goal was no longer simply: “Can the model do impressive things?” But also: “What happens if capable systems are misused at global scale?”

The report repeatedly discusses concerns around dangerous instruction generation.

During internal evaluations, earlier GPT-4 versions were sometimes capable of generating unsafe or harmful information related to dangerous materials, offensive content, or exploitative behavior. OpenAI used RLHF, safety fine-tuning, rule-based reward models, and policy systems to reduce these risks significantly before public deployment.

Cybersecurity concerns also receive substantial attention. The report discusses risks involving:

• phishing assistance

• malware-related guidance

• social engineering

• exploit generation

• automation of cyber abuse workflows

Although GPT-4 isn't presented as an autonomous hacking system, OpenAI clearly recognizes that increasingly capable language models could amplify existing cybersecurity threats if deployed irresponsibly.

Another especially important topic is biosecurity.

The report explains that domain experts evaluated whether GPT-4 could meaningfully assist users with harmful biological or chemical knowledge. OpenAI specifically investigated whether the model could help lower the barrier for dangerous misuse.

This was one of the first times a major AI paper openly treated advanced language models as potential dual-use technologies with real-world security implications.

The report also emphasizes deployment monitoring and iterative safety improvement.

Rather than treating safety as something solved before release, OpenAI frames deployment itself as part of the learning process. Monitoring user interactions, identifying failure modes, updating safeguards, and improving refusal systems became ongoing operational responsibilities rather than one-time research tasks.

Historically, this section may be one of the most important parts of the entire report.

GPT-4 marks the moment when AI safety stopped being a niche research discussion and became a core component of flagship frontier model development.

That shift reflects a deeper realization across the industry: once AI systems become powerful enough for large-scale deployment, increasing capability and managing risk become inseparable engineering problems.

Discussion

Looking back at the GPT series, GPT-4 feels less like the release of a single research model and more like the beginning of a new computing platform.

GPT-1 introduced the idea of large-scale language pretraining. GPT-2 demonstrated zero-shot multitask behavior. GPT-3 showed that models could adapt through prompting and in-context learning.

But GPT-4 changes the conversation again.

According to the technical report, the focus is no longer only about making models larger or improving benchmark scores. The report repeatedly emphasizes reliability, deployment, alignment, infrastructure, multimodal interaction, and safety engineering.

That shift is historically important.

Earlier GPT papers felt like research milestones published mainly for the machine learning community. GPT-4 feels like infrastructure designed for real-world deployment at global scale.

This becomes especially clear through systems like ChatGPT.

GPT-4 was not simply released as a downloadable research artifact or benchmark model. Instead, it became part of an entire AI product ecosystem:

• conversational assistants

• coding copilots

• enterprise APIs

• productivity tools

• educational systems

• multimodal interfaces

In practice, GPT-4 helped transform large language models from isolated research demos into continuously deployed software platforms.

Another major change is the increasing secrecy surrounding frontier AI systems.

Unlike GPT-2 and GPT-3, the GPT-4 report intentionally omits many technical details, including parameter counts, architecture specifics, training compute, and dataset composition.

OpenAI explains this partly through safety concerns and the competitive landscape, but the broader implication is significant: frontier AI models were becoming strategically valuable technologies rather than purely academic research projects.

This marks the beginning of a much more closed era in large-scale AI development.

The report also shows why alignment became such a central concern.

As language models became more capable, the risks associated with hallucinations, harmful outputs, cybersecurity misuse, misinformation, and unsafe reasoning also increased. GPT-4 treats alignment not as an optional improvement layer, but as a core engineering requirement.

This is another major transition in the history of AI systems.

Earlier models were evaluated mostly on capability:

• accuracy

• perplexity

• benchmark scores

• scaling behavior

GPT-4 expands the discussion toward:

• safety

• deployment monitoring

• refusal behavior

• policy enforcement

• human oversight

• operational reliability

The model is no longer judged only by what it can do, but also by how safely and consistently it behaves in real-world environments.

In many ways, GPT-4 also represents the rise of the modern foundation model ecosystem.

Instead of training separate systems for every individual task, one large aligned model can serve as a shared base for many applications:

• coding

• tutoring

• search

• writing

• research assistance

• customer support

• multimodal interaction

• enterprise workflows

That idea fundamentally changed the software industry.

Historically, GPT-4 may ultimately be remembered less for a single benchmark result and more for what it represented: the moment large language models became practical, continuously deployed, general-purpose AI infrastructure.

Conclusion

The GPT-4 Technical Report marks one of the most important turning points in the history of modern AI systems.

According to the report, GPT-4 is not simply a larger language model. It's a multimodal, aligned foundation model designed for real-world deployment at global scale.

The model combines several major ideas that evolved throughout the GPT series:

• large-scale Transformer pretraining

• autoregressive next-token prediction

• scaling laws

• few-shot prompting

• multimodal reasoning

• reinforcement learning from human feedback

• safety-focused post-training

Together, these components produce a system that feels qualitatively different from earlier GPT models.

GPT-4 demonstrates that scaling alone is no longer the entire story.

GPT-3 showed that larger models could develop powerful emergent abilities through scale. GPT-4 shows that alignment, safety engineering, post-training refinement, and deployment infrastructure became equally important parts of building useful AI systems.

This combination of scale and alignment ultimately became the dominant paradigm behind modern frontier AI development.

The report also reflects a broader transition happening across the industry.

Large language models were no longer being treated as isolated research experiments or benchmark systems. GPT-4 pushed AI toward real-world deployment through products, APIs, multimodal assistants, coding systems, enterprise tools, and globally accessible conversational interfaces like ChatGPT.

Historically, GPT-4 represents the moment when foundation models became practical infrastructure for everyday computing.

And that shift continues shaping the direction of modern AI today.

Final Insight

Looking across the entire GPT series, the progression becomes remarkably clear.

GPT-1 introduced the idea that large-scale pretraining could produce transferable language representations. Instead of training separate NLP systems from scratch for every task, models could first learn general language patterns and then adapt through fine-tuning.

GPT-2 pushed this idea further by showing that sufficiently large language models could perform tasks in a zero-shot setting without explicit supervised training. The model was no longer just memorizing tasks – it was beginning to generalize from language itself.

GPT-3 changed the paradigm again. Few-shot prompting and in-context learning showed that models could adapt dynamically during inference simply from examples written inside the prompt. This transformed prompting into a new interface for interacting with AI systems.

Then GPT-4 expanded the idea into something much larger. The focus was no longer only about scaling models or improving benchmarks. GPT-4 introduced the era of aligned multimodal foundation models: systems designed not just to generate language, but to operate safely, follow instructions, reason across modalities, and function as deployable infrastructure for real-world applications.

Historically, that may be the most important shift of all.

GPT-4 was not simply a larger language model.

It marked the transition from experimental large language models to globally deployed AI assistants integrated into everyday computing, software development, education, productivity tools, and multimodal human-computer interaction.

And in many ways, we're still only at the beginning of that transition.

GPT-1 vs GPT-2 vs GPT-3 vs GPT-4: Key Differences

A simple way to see how the GPT series evolved is by looking at what each generation introduced.

GPT-1 introduced modern pretraining, GPT-2 showed that large language models could perform tasks through zero-shot prompting, GPT-3 pushed few-shot prompting and in-context learning into the mainstream, and GPT-4 expanded the idea further through alignment, multimodal reasoning, and real-world deployment.

The comparison below shows how the focus gradually shifted from task-specific NLP models to general-purpose AI systems capable of conversation, coding, reasoning, and multimodal understanding.

PyTorch Implementations of the GPT Architecture Evolution

GPT-1: Pre-training + Fine-Tuning Architecture

class GPT1(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(512, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(d_model)
            for _ in range(n_layers)
        ])

        self.ln_f = nn.LayerNorm(d_model)

        # Language modeling head
        self.lm_head = nn.Linear(d_model, vocab_size)

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.ln_f(x)

        logits = self.lm_head(x)

        return logits

GPT1 inherits from nn.Module, which is the base class used to build neural networks in PyTorch. The constructor (init) defines all trainable layers used by the model.

nn.Embedding(vocab_size, d_model) creates a learnable lookup table that converts token IDs into dense vectors. Each token in the vocabulary is mapped to a vector of size d_model.

The positional embedding layer adds information about token order. Since Transformers process tokens in parallel, they need explicit positional information to understand sequence structure.

nn.ModuleList([...]) stores multiple Transformer blocks while ensuring PyTorch properly tracks their parameters during training. Each TransformerBlock typically contains masked self-attention and a feed-forward network.

nn.LayerNorm(d_model) applies layer normalization before the output projection. This helps stabilize training and improves gradient flow in deep Transformer architectures.

The language modeling head (nn.Linear) projects the hidden representations back into vocabulary space. The output size equals vocab_size, producing prediction scores for every possible next token.

Inside the forward() method, input_ids.size(1) retrieves the sequence length, and torch.arange(...) generates positional indices for each token position.

The token embeddings and positional embeddings are added together to produce the initial Transformer input representation.

The model then passes the representation through each Transformer block sequentially:

for block in self.transformer_blocks:
    x = block(x)

This iterative stacking is what allows GPT models to learn increasingly abstract contextual representations.

After normalization, the final hidden states are passed into lm_head, producing logits. These logits are unnormalized prediction scores used to compute probabilities for next-token generation.

The model finally returns the logits tensor, which is typically passed through softmax during inference or used directly with CrossEntropyLoss during training.

GPT-2: Zero-Shot Multitask Architecture

class GPT2(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(1024, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                pre_layer_norm=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Like GPT-1, the model begins with token embeddings and positional embeddings. nn.Embedding converts token IDs into dense vectors, while positional embeddings provide information about token order in the sequence.

One noticeable difference is the larger positional embedding size (1024 instead of 512), allowing GPT-2 to process longer contexts.

The Transformer layers are stored using nn.ModuleList, but each TransformerBlock now uses:

pre_layer_norm=True

This means layer normalization is applied before attention and feed-forward operations rather than after them. This “Pre-LN” design significantly improves gradient flow and training stability in deeper Transformer models.

The forward pass follows the same overall pipeline:

1. Generate positional indices with torch.arange()

2. Add token and positional embeddings

3. Pass representations through stacked Transformer blocks

4. Apply final normalization

5. Project outputs into vocabulary space

The sequential block processing happens here:

for block in self.transformer_blocks:
    x = block(x)

GPT-2 also introduces a small optimization in the output layer:

self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

The bias term is removed because it provides little benefit in large language modeling setups and slightly reduces parameter count.

Finally, the model returns logits, which contain prediction scores for every token in the vocabulary at each sequence position.

GPT-3: Few-Shot / In-Context Learning Architecture

class GPT3(nn.Module):
    def __init__(
        self,
        vocab_size=50257,
        d_model=12288,
        n_layers=96,
        n_heads=96,
        context_length=2048
    ):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(context_length, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                n_heads=n_heads,
                pre_layer_norm=True,
                sparse_attention=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        self.lm_head = nn.Linear(
            d_model,
            vocab_size,
            bias=False
        )

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Compared to earlier GPT versions, this model dramatically increases scale. The embedding size (d_model=12288) and the number of Transformer layers (96) allow the network to learn highly complex language patterns and long-range dependencies.

The model also uses 96 attention heads:

n_heads=96

Multi-head attention allows the model to focus on different relationships between tokens simultaneously, improving contextual understanding.

The positional embedding length is expanded to 2048, enabling the model to process much longer sequences than GPT-2.

Each Transformer block is configured with:

pre_layer_norm=True,
sparse_attention=True

Pre-layer normalization improves training stability in very deep networks, while sparse attention reduces the computational cost of attention by limiting how many tokens attend to each other. This becomes important at GPT-3 scale, where full attention over long sequences is extremely expensive.

The forward pass follows the standard GPT pipeline:

1. Convert token IDs into embeddings

2. Add positional information

3. Pass representations through stacked Transformer blocks

4. Apply final layer normalization

5. Generate vocabulary logits

The core iterative processing happens here:

for block in self.transformer_blocks:
    x = block(x)

Finally, the output layer projects the hidden states into vocabulary space, producing logits used for next-token prediction during training and text generation.

GPT-4: Aligned Multimodal Foundation Model Architecture

class GPT4(nn.Module):
    def __init__(
        self,
        vocab_size=50257,
        d_model=12288,
        n_layers=120,
        n_heads=96,
        context_length=8192
    ):
        super().__init__()

        # Text embeddings
        self.token_embedding = nn.Embedding(
            vocab_size,
            d_model
        )

        self.position_embedding = nn.Embedding(
            context_length,
            d_model
        )

        # Vision encoder for image inputs
        self.vision_encoder = VisionTransformer(
            embed_dim=d_model
        )

        # Multimodal projection layer
        self.image_projection = nn.Linear(
            d_model,
            d_model
        )

        # Decoder-only Transformer blocks
        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                n_heads=n_heads,
                pre_layer_norm=True,
                flash_attention=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        # Language modeling head
        self.lm_head = nn.Linear(
            d_model,
            vocab_size,
            bias=False
        )

        # RLHF alignment head
        self.reward_head = RewardModel(
            hidden_size=d_model
        )

    def forward(
        self,
        input_ids,
        image_inputs=None
    ):

        positions = torch.arange(
            input_ids.size(1)
        )

        text_embeddings = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        # Encode image if provided
        if image_inputs is not None:

            image_features = self.vision_encoder(
                image_inputs
            )

            image_embeddings = self.image_projection(
                image_features
            )

            x = torch.cat(
                [image_embeddings, text_embeddings],
                dim=1
            )

        else:
            x = text_embeddings

        # Transformer decoding
        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Like previous GPT models, the architecture starts with token embeddings and positional embeddings. nn.Embedding converts token IDs into dense vector representations, while positional embeddings preserve sequence order information.

One major difference is the addition of a vision encoder:

self.vision_encoder = VisionTransformer(
    embed_dim=d_model
)

This module processes image inputs and converts them into visual feature representations that can be understood by the Transformer.

The image features are then passed through a projection layer:

self.image_projection = nn.Linear(
    d_model,
    d_model
)

This aligns image representations with the same embedding space used for text tokens, making multimodal processing possible.

The Transformer stack remains decoder-only, but now uses:

flash_attention=True

Flash Attention is an optimized attention implementation that reduces memory usage and improves training and inference speed, especially for very long context windows like 8192 tokens.

Inside the forward() method, text embeddings are created first. If an image is provided, the image is encoded and projected into embeddings:

image_features = self.vision_encoder(
    image_inputs
)

The image and text embeddings are then combined using:

x = torch.cat(
    [image_embeddings, text_embeddings],
    dim=1
)

torch.cat() concatenates tensors along the sequence dimension, allowing the Transformer to process image and text tokens together as a single sequence.

The combined representations pass through all Transformer blocks sequentially:

for block in self.transformer_blocks:
    x = block(x)

After normalization, the final hidden states are projected into vocabulary space to produce logits for next-token prediction.

The architecture also introduces a reward model head:

self.reward_head = RewardModel(
    hidden_size=d_model
)

This component represents reinforcement learning from human feedback (RLHF), which is used to align model outputs with human preferences and improve response quality and safety.

Resources:

• Pytorch Projects for GPT series

• Attention Is All You Need

• Improving Language Understanding by Generative Pre-Training (GPT-1)

• Language Models are Unsupervised Multitask Learners (GPT-2)

• Language Models are Few-Shot Learners (GPT-3)

• GPT-4 Technical Report

• Scaling Laws for Neural Language Models

• Training Compute-Optimal Large Language Models

• PaLM: Scaling Language Modeling with Pathways

• Training Language Models to Follow Instructions with Human Feedback

• Constitutional AI: Harmlessness from AI Feedback

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

• Self-Consistency Improves Chain of Thought Reasoning in Language Models

• TruthfulQA: Measuring How Models Mimic Human Falsehoods

• HumanEval: Evaluating Large Language Models Trained on Code

• Measuring Massive Multitask Language Understanding (MMLU)

Contact Me

• Github

• X

• Linkedin

You know what it is before you dig into the logs. The AI meeting summaries land in shared Slack channels, the AI-drafted docs show up in shared Google Drive folders, and the AI code review suggestions appear in pull requests that both treated and control engineers read. Behavior changes for the treated users, and a slice of that behavior bleeds back into your control group through the collaboration graph.

This is the collaborator contamination trap. It shows up in every generative AI product that touches shared artifacts: AI meeting notes that teammates read, AI-drafted documents that coworkers edit, AI code suggestions that reviewers evaluate, AI-generated email threads that the whole team replies to. User-level randomization assumes one user's treatment assignment leaves every other user's outcome alone. In a collaborative workspace, that assumption is wrong by design, and the product experiment folds the feature's real effect together with the spillover it creates inside the control group.

Running a collaborative AI feature behind a user-level A/B test is a product experiment that violates the Stable Unit Treatment Value Assumption (SUTVA). The fix is cluster randomization: flip the coin at the workspace level, so entire teams are in or out together, then model the cross-workspace spillover directly.

This tutorial walks through the full pipeline (cluster assignment, a biased, naive user-level OLS, cluster-weighted least squares for honest standard errors, a two-exposure decomposition that identifies direct and spillover effects separately, and cluster-bootstrap confidence intervals) on a 50,000-user synthetic SaaS dataset in which the ground-truth causal effects are known. You'll estimate them, quantify uncertainty, and see where the approach silently breaks.

Companion code: every code block runs end-to-end in the companion notebook at github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm/tree/main/05_cluster_randomization. The notebook (cluster_randomization_demo.ipynb) has all outputs pre-executed, so you can read along on GitHub before running anything locally.

Table of Contents

• Why user-level A/B randomization breaks under collaboration

• What cluster randomization actually does

• Prerequisites

• Setting up the working example

  • Step 1: Build the cluster assignment and spillover exposure

  • Step 2: Naive user-level OLS (biased and overconfident)

  • Step 3: Cluster-weighted least squares (honest standard error)

  • Step 4: Two-exposure decomposition (unbiased direct and spillover)

  • Step 5: Cluster-bootstrap confidence intervals

• When cluster randomization fails

• What to do next

Why User-Level A/B Randomization Breaks Under Collaboration

The math of an A/B test is elegant because one user's treatment assignment has no bearing on another user's outcome. Flip a coin; half your users get the AI feature, and the coin flip breaks every possible confound by construction. Collaboration breaks that guarantee in three ways.

Shared artifacts travel. The AI summary lands in a channel every teammate reads, the AI-drafted doc goes into a folder every teammate edits, and the AI code review suggestion sits on a pull request every reviewer evaluates. Control users consume those artifacts, whether or not the feature is switched on for them, and the behavioral effects of reading AI-assisted content leak into their outcomes.

Shared workflows create interference. A treated user who relies on the AI summarizer writes shorter follow-up notes, assuming teammates have read the summary. A control user on the same team receives those shorter notes and spends less time reading them, which changes their session length. That means the treated user's assignment has shifted the control user's outcome, which is exactly what SUTVA forbids.

Network adoption follows collaboration. Power users on treated teams experiment with the feature first, then nudge teammates in other workspaces through cross-team channels. If your treated group produces AI-assisted content that your control group reads and copies, the control group is partially treated without ever flipping a switch.

All three mechanisms produce the same symptom: the raw user-level comparison understates the feature's direct effect because the control group is no longer a pure counterfactual. On the synthetic dataset in this tutorial, the ground-truth direct effect is +0.80 min of session time for treated users, and the ground-truth spillover effect is +0.20 min for control users who collaborate across workspaces. A naive user-level OLS recovers +0.6723, a 16 percent underestimate of the direct effect, and reports a standard error that is roughly 19 times too small because it treats 50,000 users as independent, even though the treatment was randomized only across 50 clusters. That's not a small error. It's the kind that ships a broken feature launch decision.

What Cluster Randomization Actually Does

Cluster randomization flips the assignment coin at the workspace level so entire teams land in the same arm, confining most interference to where it belongs and making the residual cross-workspace leakage something you can model directly.

Figure 1(image ab: Schematic of the SUTVA violation that cluster randomization targets. Every user in a treated workspace (top row, red) sees the AI feature. Every user in a control workspace (bottom row) should see nothing, but collaborators (orange) read AI artifacts that travel through shared Slack, documents, and code reviews. Those spillover-exposed users are partially treated. Cluster randomization doesn't make interference disappear; it confines it to within workspace boundaries, leaving the remaining cross-workspace leakage as an identifiable component that a two-exposure model can estimate directly.

If a workspace is treated, every user inside it gets the feature. If it's a control workspace, nobody inside it does. Interference within a workspace is fine because all teammates share the same assignment, and the workspace-level mean captures the full treatment package. The design aims to control interference across workspaces.

The estimator works under a stack of assumptions, and each one has a name worth knowing because the failure modes at the end of this tutorial map directly to specific violations.

• Cluster-level random assignment. Treatment is assigned at the cluster level by a genuinely random mechanism. Which workspaces land in the treated arm is independent of workspace-level potential outcomes.

• Partial interference. Interference happens inside clusters but not across them (Hudgens et al.). A treated user in workspace A can affect her teammate in workspace A, but can't affect a user in workspace B. This is the assumption cluster randomization is built around.

• Cluster-level SUTVA. A workspace's treatment is a single, well-defined package. There's one version of the feature, and within-cluster heterogeneity in exposure is absorbed into the cluster-level effect.

• Exchangeability of clusters. Before the coin flip, the treated and control workspaces are exchangeable. Randomization achieves this by construction.

• Sufficient cluster count. Cluster-robust inference relies on a central limit theorem across clusters. Practitioners often use K ≥ 30 as a working floor, though the appropriate threshold depends on cluster-size heterogeneity and the choice of test statistic. Fewer clusters demand a different inference tool, such as randomization inference or a cluster wild bootstrap.

Partial interference is the underlying assumption of load-bearing here. The whole point of cluster randomization is that cross-cluster spillover is smaller and slower than within-cluster spillover, so treating an entire team contains most of the interference where it's supposed to be (Ugander et al.). When cross-cluster spillover is meaningful, a two-exposure model directly identifies and estimates that leakage.

Prerequisites

You'll need Python 3.11 or newer, comfort with pandas and linear regression, and rough familiarity with ordinary least squares.

Install the packages for this tutorial:

pip install numpy pandas statsmodels scipy matplotlib

Here's what's happening: five packages cover the full pipeline. Pandas loads the data and builds the cluster assignment. NumPy handles array arithmetic and bootstrap draws. Statsmodels fits every regression: naive OLS, cluster-weighted least squares, and the two-exposure model with cluster-robust standard errors. Scipy supports the kernel density diagnostic plot, and matplotlib renders it.

Clone the companion repo to get the synthetic dataset:

git clone https://github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm.git
cd product-experimentation-causal-inference-genai-llm
python data/generate_data.py --seed 42 --n-users 50000 --out data/synthetic_llm_logs.csv

Here's what's happening: the clone pulls the companion repo, and generate_data.py produces the shared 50,000-user dataset used across the series. Seed 42 keeps the data reproducible. The 50,000-user scale gives enough users per workspace (about 1,000 each) for the cluster-level inference to behave asymptotically. The output CSV lands at data/synthetic_llm_logs.csv.

Setting up the Working Example

The synthetic dataset simulates a SaaS product with 50,000 users spread across 50 workspaces. The collaborative AI feature ships at full coverage to 25 randomly selected workspaces and stays off for the other 25.

A control user is spillover-exposed when they collaborate across workspaces. In this tutorial, opt_in_agent_mode == 1 serves as a behavioral proxy for that cross-workspace activity: users who actively opt into AI tooling are the ones reading teammate-authored documents, Slack threads, and pull requests where treated-workspace AI output surfaces. In a production deployment, you'd replace this proxy with an observed collaboration graph such as shared-channel membership, doc co-authorship, or reviewer overlap. Because opt_in_agent_mode reflects a voluntary behavioral choice with no random component, the spillover coefficient in a real experiment would absorb selection differences between opting-in and non-opting-in control users. A production spillover flag should be grounded in the observed collaboration graph; behavioral proxies introduce selection bias that the two-exposure model can't correct.

This tutorial constructs session_minutes_obs from scratch by layering known ground-truth effects onto workspace-level baselines. The CSV's session_minutes column is intentionally set aside. That separation lets you verify that every estimator recovers the effects baked in.

The ground-truth effects baked into the scenario are a +0.80-minute direct effect on treated users and a +0.20-minute spillover effect on spillover-exposed control users. Knowing both values is what lets you verify that your estimator recovers them.

Step 1: Build the Cluster Assignment and Spillover Exposure

The first code block loads the data, assigns workspaces to treatment at the cluster level, flags spillover-exposed users, and constructs an observed outcome where the ground truth is known. The outcome starts from a workspace-level baseline so within-workspace correlation is genuine. It then adds the direct effect for treated users, the spillover effect for exposed control users, and Gaussian noise.

import numpy as np
import pandas as pd

DIRECT_EFFECT = 0.80
SPILLOVER_EFFECT = 0.20
DATA_SEED = 42
OUTCOME_NOISE_SD = 0.30

df = pd.read_csv("data/synthetic_llm_logs.csv")
rng = np.random.default_rng(DATA_SEED)

df["treated_workspace"] = (df["workspace_id"] < 25).astype(int)
df["treated_user"] = df["treated_workspace"]
df["spillover_exposed"] = (
    (df["treated_workspace"] == 0) & (df["opt_in_agent_mode"] == 1)
).astype(int)

ws_baseline = pd.DataFrame({
    "workspace_id": np.arange(50),
    "ws_baseline": rng.normal(5.0, 0.30, size=50),
})
df = df.merge(ws_baseline, on="workspace_id")
noise = rng.normal(0, OUTCOME_NOISE_SD, size=len(df))
df["session_minutes_obs"] = (
    df["ws_baseline"]
    + DIRECT_EFFECT * df["treated_user"]
    + SPILLOVER_EFFECT * df["spillover_exposed"]
    + noise
)
df["exposure"] = np.select(
    [df["treated_user"] == 1, df["spillover_exposed"] == 1],
    ["direct", "spillover"],
    default="pure_control",
)

print(f"Total users:             {len(df):,}")
print(f"Treated workspaces:      {df[df.treated_workspace == 1].workspace_id.nunique()}")
print(f"Control workspaces:      {df[df.treated_workspace == 0].workspace_id.nunique()}")
print(f"Treated users:           {df.treated_user.sum():,}")
print(f"Pure-control users:      {(df.exposure == 'pure_control').sum():,}")
print(f"Spillover-exposed users: {(df.exposure == 'spillover').sum():,}")
ws_sizes = df.groupby("workspace_id").size()
print(f"Workspace size: min={ws_sizes.min()} median={int(ws_sizes.median())} max={ws_sizes.max()}")

Expected output:

Total users:             50,000
Treated workspaces:      25
Control workspaces:      25
Treated users:           24,937
Pure-control users:      18,319
Spillover-exposed users: 6,744
Workspace size: min=923 median=1002 max=1052

Here's what's happening: Workspace IDs 0 through 24 become the treated cluster and 25 through 49 become the control cluster, giving you 24,937 treated users and 25,063 control users. Among the controls, 6,744 are flagged as spillover-exposed because they opted into agent mode and sit in a control workspace where they'd plausibly read treated-workspace output through cross-team channels. The remaining 18,319 are pure-control users, untouched by the feature. Workspace sizes range from 923 to 1,052 users, which is close enough to be balanced, so that cluster-weighted and unweighted estimators will behave similarly. The observed outcome session_minutes_obs captures the known ground truth: a treated user adds 0.80 min to their workspace baseline, a spillover-exposed user adds 0.20 min, and every user is subject to Gaussian noise with standard deviation 0.30 min.

Figure 2 (image above): The three exposure groups on the 50,000-user dataset. The top panel shows the observed-outcome distribution for each group, with dashed vertical lines at the group means (5.06 min pure control, 5.27 min spillover-exposed, 5.79 min treated). The spillover distribution sits between the pure-control and treated distributions, which is the contamination a naive user-level estimator would fold into the control baseline. The bottom panel translates the same groups into raw counts: 18,319 pure-control users, 6,744 spillover-exposed control users, and 24,937 treated users. Where Figure 1 schematically showed the SUTVA violation, this figure shows it at the data scale, and the three-group structure is exactly what Step 4's two-exposure model will identify.

Step 2: Naive User-Level OLS (Biased and Overconfident)

The naive analysis ignores clustering entirely and regresses the observed outcome on each user's treatment assignment, reporting a standard error as if every user were an independent draw. Two things go wrong at once.

import statsmodels.formula.api as smf

naive = smf.ols("session_minutes_obs ~ treated_user", data=df).fit()
print(f"Naive estimate:  {naive.params['treated_user']:+.4f} min")
print(f"Naive SE:        {naive.bse['treated_user']:.4f}  (under-reported)")
ci = naive.conf_int().loc["treated_user"].tolist()
print(f"Naive 95% CI:    [{ci[0]:+.4f}, {ci[1]:+.4f}]")
print(f"Ground truth:    +0.80")
print(f"Bias:            {naive.params['treated_user'] - 0.80:+.4f} min")

Expected output:

Naive estimate:  +0.6723 min
Naive SE:        0.0034  (under-reported)
Naive 95% CI:    [+0.6656, +0.6790]
Ground truth:    +0.80
Bias:            -0.1277 min

Here's what's happening: the point estimate lands at +0.6723, 16 percent below the ground-truth direct effect of +0.80. The bias has two components. First, spillover contamination: 6,744 control users who read treated-workspace output lie above the pure-control baseline, raising the control mean and compressing the naive treated-minus-control gap. Second, workspace baseline imbalance: with only 50 clusters, random assignment doesn't guarantee that treated and control workspace pools draw equal mean baselines. This dataset's specific seed produces a treated-pool baseline slightly below the control-pool baseline, adding additional downward pressure on the estimate. The lesson generalizes: at small K, balance checks on observable workspace characteristics before the experiment are the only defense against pre-existing between-arm differences that no standard-error correction can fix.

The standard error is the more alarming number. At 0.0034, it reflects variation across 50,000 users treated as independent observations, and the resulting 95% confidence interval [+0.6656, +0.6790] excludes the ground truth entirely, at roughly one-twentieth the width the design actually supports. An SE 19 times too small inflates the t-statistic by the same factor, making the naive regression's p-value appear orders of magnitude more significant than the design justifies. A stakeholder reading this report would walk away confident that the direct effect is somewhere near 0.67 min. Wrong number, wrong precision.

Step 3: Cluster-Weighted Least Squares (Honest Standard Error)

The fix for the standard error is to aggregate to 50 workspace means, then regress those means on the workspace-level treatment indicator weighted by workspace size. Inference is now based on K = 50 observations.

import statsmodels.api as sm

ws = (
    df.groupby("workspace_id")
    .agg(ws_mean=("session_minutes_obs", "mean"),
         ws_size=("user_id", "count"),
         treated=("treated_workspace", "max"))
    .reset_index()
)
X_ws = sm.add_constant(ws["treated"])
wls = sm.WLS(ws["ws_mean"], X_ws, weights=ws["ws_size"]).fit()
wls_ci = wls.conf_int().loc["treated"].tolist()
print(f"WLS cluster-mean contrast: {wls.params['treated']:+.4f} min")
print(f"WLS SE:          {wls.bse['treated']:.4f}  (based on K=50 clusters)")
print(f"WLS 95% CI:      [{wls_ci[0]:+.4f}, {wls_ci[1]:+.4f}]")

Expected output:

WLS cluster-mean contrast: +0.6723 min
WLS SE:                    0.0652  (based on K=50 clusters)
WLS 95% CI:                [+0.5412, +0.8035]

Here's what's happening: the cluster-mean contrast is identical to the naive estimate at +0.6723, because weighted workspace means are a different aggregation of the same user-level data. What changed is the standard error. At 0.0652, it's roughly 19 times larger than the naive 0.0034 and reflects genuine variation across 50 cluster means (statsmodels WLS uses t(48) critical values in place of z=1.96, which is why the CI bounds differ slightly from a hand calculation with z). The 95% confidence interval expands to [+0.5412, +0.8035], which barely covers the ground truth. WLS has fixed the inference problem, so the standard error now reflects the actual design, but it hasn't fixed the identification problem. Control workspace means still includes spillover-exposed users, so this estimate is a contaminated contrast you can't interpret as a clean ATE. The next step separates the two.

Step 4: Two-Exposure Decomposition (Unbiased Direct and Spillover)

The two-exposure model treats each user's exposure as a three-category variable (direct, spillover, or pure control) and regresses the outcome on the two non-baseline categories (Aronow et al.). Pure control is the omitted reference, so both coefficients are directly interpretable: one is the direct effect of the feature, the other is the spillover effect on control users who collaborate across workspaces.

df["is_direct"] = (df["exposure"] == "direct").astype(int)
df["is_spillover"] = (df["exposure"] == "spillover").astype(int)
two_exp = smf.ols(
    "session_minutes_obs ~ is_direct + is_spillover",
    data=df,
).fit(cov_type="cluster", cov_kwds={"groups": df["workspace_id"]})
direct = two_exp.params["is_direct"]
spillover = two_exp.params["is_spillover"]
direct_ci = two_exp.conf_int().loc["is_direct"].tolist()
spillover_ci = two_exp.conf_int().loc["is_spillover"].tolist()
print(f"Direct effect:     {direct:+.4f} min  (ground truth = +0.80)")
print(f"  SE:              {two_exp.bse['is_direct']:.4f}")
print(f"  95% CI:          [{direct_ci[0]:+.4f}, {direct_ci[1]:+.4f}]")
print(f"Spillover effect:  {spillover:+.4f} min  (ground truth = +0.20)")
print(f"  SE:              {two_exp.bse['is_spillover']:.4f}")
print(f"  95% CI:          [{spillover_ci[0]:+.4f}, {spillover_ci[1]:+.4f}]")
spillover_share = (df["exposure"] == "spillover").mean()
projected = direct + spillover_share * spillover
print(f"Spillover share of all users: {spillover_share:.4f}")
print(f"Projected total under full rollout: {projected:+.4f} min")

Expected output:

Direct effect:     +0.7284 min  (ground truth = +0.80)
  SE:              0.0647
  95% CI:          [+0.6016, +0.8552]
Spillover effect:  +0.2083 min  (ground truth = +0.20)
  SE:              0.0038
  95% CI:          [+0.2008, +0.2158]
Spillover share of all users: 0.1349
Projected total under full rollout: +0.7565 min

Here's what's happening: fitting on the three-category exposure with cluster-robust standard errors keyed to workspace_id yields two clean coefficients. The direct effect is +0.7284, with a 95% CI of [+0.6016, +0.8552], which includes the ground-truth value of +0.80. The spillover effect is +0.2083, with a 95% CI of [+0.2008, +0.2158], which tightly covers the ground-truth +0.20. The spillover SE (0.0038) looks small for cluster-robust inference because the simulated spillover effect is uniform across all 25 control clusters; in real data with heterogeneous spillover intensity, you'll see the cluster-robust SE grow meaningfully larger. The projected total of +0.7565 min accounts for the spillover effect, based on the fraction of users expected to be spillover-exposed at a given deployment scale (0.1349 in this dataset). In a production deployment, you'd replace that fraction with whatever share your collaboration graph predicts will be spillover-exposed under your rollout plan. The projection is a design parameter in your rollout, so state the assumed share explicitly when you report the number.

Step 5: Cluster-Bootstrap Confidence Intervals

The cluster bootstrap resamples entire workspaces to test whether Step 4's analytic confidence intervals hold without assuming the central limit theorem has fully kicked in at K = 50. Analytic standard errors for a cluster design work well when K is large, and workspaces are roughly equal in size; the bootstrap confirms this holds in practice for your actual data. Resampling individual users would undercount variance because users in the same workspace share the cluster assignment and the workspace-level baseline; the cluster bootstrap preserves that correlation structure.

def naive_point(d):
    return smf.ols(
        "session_minutes_obs ~ treated_user", data=d
    ).fit().params["treated_user"]

def wls_point(d):
    w = (d.groupby("workspace_id").agg(
            ws_mean=("session_minutes_obs", "mean"),
            ws_size=("user_id", "count"),
            treated=("treated_workspace", "max")).reset_index())
    X = sm.add_constant(w["treated"])
    return sm.WLS(w["ws_mean"], X, weights=w["ws_size"]).fit().params["treated"]

def two_exp_point(d):
    fit = smf.ols(
        "session_minutes_obs ~ is_direct + is_spillover", data=d
    ).fit(cov_type="cluster", cov_kwds={"groups": d["workspace_id"]})
    return fit.params["is_direct"], fit.params["is_spillover"]

rng_boot = np.random.default_rng(7)
ws_ids = df["workspace_id"].unique()
k = len(ws_ids)
reps = {"naive": [], "cluster_wls": [], "direct": [], "spillover": []}
for _ in range(500):
    draw = rng_boot.choice(ws_ids, size=k, replace=True)
    sample = pd.concat(
        [df[df["workspace_id"] == wid] for wid in draw],
        ignore_index=True,
    )
    reps["naive"].append(naive_point(sample))
    reps["cluster_wls"].append(wls_point(sample))
    d_b, s_b = two_exp_point(sample)
    reps["direct"].append(d_b)
    reps["spillover"].append(s_b)

for key, truth in [("naive", 0.80), ("cluster_wls", 0.80),
                   ("direct", 0.80), ("spillover", 0.20)]:
    arr = np.array(reps[key])
    lo, hi = np.percentile(arr, [2.5, 97.5])
    covers = "covers" if lo <= truth <= hi else "misses"
    print(f"{key:<13} 95% CI: [{lo:+.4f}, {hi:+.4f}]   ({covers} {truth:+.2f})")

Expected output:

naive         95% CI: [+0.5386, +0.7966]   (misses +0.80)
cluster_wls   95% CI: [+0.5386, +0.7966]   (misses +0.80)
direct        95% CI: [+0.5931, +0.8519]   (covers +0.80)
spillover     95% CI: [+0.2008, +0.2164]   (covers +0.20)

Here's what's happening: drawing 50 workspaces with replacement and refitting each estimator 500 times gives you a bootstrap distribution for every point estimate. The naive OLS and cluster WLS estimators produce identical bootstrap intervals because they share the same point estimate under workspace-level resampling, and both intervals exclude the ground-truth +0.80 because both are biased by the two sources identified in Step 2 (spillover contamination and the workspace baseline imbalance). The direct-effect interval from the two-exposure model is [0.5931, 0.8519], which includes 0.80. The spillover interval is [+0.2008, +0.2164], which tightly covers +0.20. The cluster bootstrap confirms what the analytic cluster-robust standard errors in Step 4 already showed: inference holds up without relying on asymptotic approximations at K = 50. Running this takes about one minute on a laptop.

When Cluster Randomization Fails

Cluster randomization solves the SUTVA problem when its assumptions hold, and it produces biased estimates that look clean when they don't. Three failure modes map to a named identification assumption; a fourth addresses estimator efficiency when cluster sizes are unequal.

Too few clusters (violates sufficient cluster count). Cluster-robust standard errors rely on a central limit theorem across clusters, and practitioners often use K ≥ 30 as a working floor, though the appropriate threshold depends on heterogeneity in cluster sizes and the choice of test statistic (MacKinnon & Webb, 2017). A collaborative AI feature rolled out to four customer accounts doesn't clear that bar. Cluster-robust standard errors with K = 4 are anticonservative, and the resulting confidence intervals are too narrow. When K is small, randomization inference or a cluster wild bootstrap gives you valid p-values.

Cluster boundary does not contain the interference graph (violates partial interference). Cluster randomization assumes interference is confined within workspaces. If your users collaborate heavily across workspaces through Slack Connect channels, external shared documents, or customer community forums, partial interference is a fiction, and spillover bleeds across every cluster boundary. The two-exposure model can absorb modest cross-cluster leakage because the spillover coefficient captures whatever spillover your exposure flag measures. When leakage is structural, you need the observed collaboration graph and a graph-cluster randomization design that builds clusters from the collaboration structure itself (Ugander et al.).

Heterogeneous cluster sizes that bias the aggregation (estimator efficiency). Equal-weighted cluster means treat a 50-user workspace the same as a 5,000-user workspace, which is a poor efficiency trade when the variance of a workspace's mean depends on the number of users in it. The fix is weighted least squares by workspace size, or a mixed-effects model with workspace random intercepts. This is an efficiency concern with no bearing on identification, and that distinction matters: the point estimate stays consistent under either weighting choice.

Post-hoc cluster construction (violates exchangeability). Building cluster assignments after observing outcomes is the cleanest way to turn a valid design into p-hacking. You've got to define and commit your clusters before the randomization, ideally in a pre-registered analysis plan. Any post-hoc adjustment to cluster boundaries (dropping a workspace with extreme outcomes, merging small workspaces into a composite, redefining spillover exposure after inspecting the data) reintroduces selection bias that no standard-error correction can fix.

Two additional threats deserve attention in real deployments.

Cluster-level SUTVA fails under partial feature adoption. The cluster-level SUTVA assumption requires that a workspace's treatment is a single, well-defined package. That breaks down when a feature rolls out at different adoption rates within a single workspace, or when multiple feature versions coexist (advanced for power users, basic for casual users). In that case, the cluster-level "treatment" conflates multiple effects, and the estimand is no longer interpretable.

Workspace-level confounders when randomization isn't mechanical. In enterprise deployments, workspace selection into the treated arm is often not fully random. Beta programs attract tech-forward accounts; customer success teams influence which clients get early access. When exchangeability is violated before the coin flip, cluster-robust standard errors cannot correct for pre-existing systematic differences between the treated and control workspace pools. A balance check on observable workspace characteristics (size, industry, baseline engagement) and regression adjustment at the cluster level are the standard remedies.

These failure modes stay invisible in your regression coefficients. They surface later, in the gap between the offline estimate and the production rollout. Cluster counts, collaboration graph audits, and a written pre-registration are your only real defenses.

What To Do Next

Cluster randomization is the right tool when collaboration within a workspace creates spillover effects that break user-level SUTVA, and when your clusters are natural and observable (workspaces, teams, accounts, physical stores). If the interference you care about spans geographic markets or occurs over time inside a two-sided marketplace where drivers and riders clear as a whole, switchback experiments that randomize time slots fit better. If your treatment is assigned at the individual level but you suspect unobserved cross-user confounders, an instrumental variable analysis with a design-based instrument provides a cleaner identification strategy. When interference is known and complex, graph-cluster randomization with Horvitz-Thompson weighted exposure estimators gives you unbiased effect estimates without forcing every cluster boundary to contain every interference path.

The companion notebook for this tutorial lives at github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm/tree/main/05_cluster_randomization. Clone the repo, generate the synthetic dataset, and run cluster_randomization_demo.ipynb (or cluster_randomization_demo.py) to reproduce every code block, every number, and every figure from this tutorial.

When a collaborative AI feature ships to teams who share their work, the user-level A/B estimate is almost always wrong. Cluster randomization plus a two-exposure model gives you the direct effect and the spillover effect separately, and the cluster bootstrap gives you an interval you can defend when a stakeholder asks how much of the lift comes from the feature and how much comes from teammates talking to each other.

Your infrastructure team upgrades every workspace from Claude 4.5 to Claude 4.6 overnight. All 50 production workspaces get the new model at the same time. A week later, task completion climbs across the board. The head of product calls it a win.

But you know something's off. No holdout group ran 4.5 through the upgrade week. The naïve before/after picks up whatever else changed that week alongside the model: a new onboarding flow, a seasonal uptick, a high-profile customer onboarding.

This is the Global Rollout Problem. It appears whenever a team ships a model upgrade to the entire user base simultaneously. For product teams running generative AI features, it's one of the most common measurement traps in the stack. Staged rollouts buy you a control group, global rollouts eliminate it.

In 2026, global model upgrades are the norm: every API provider pushes new versions, and every team using Claude, GPT, or Gemini has experienced the sudden jump from one version to the next with no opt-out.

Synthetic control is the tool that data scientists use when the control group is missing. You build a weighted combination of untreated units (other workspaces or regions that weren't upgraded at the same time) whose pre-upgrade behavior matches that of the treated unit. Compare the treated unit to its synthetic twin after the upgrade, and the gap is the causal estimate, conditional on three identification assumptions that we'll name explicitly.

In this tutorial, you'll build a synthetic control from scratch in Python using scipy.optimize, apply it to a 50,000-user synthetic SaaS dataset, and validate with a placebo permutation test, leave-one-out donor sensitivity, and a cluster bootstrap 95% confidence interval.

Companion code: every code block runs end-to-end in the companion notebook at github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm/tree/main/04_synthetic_control. The notebook (synthetic_control_demo.ipynb) has all outputs pre-executed, so you can read along on GitHub before running anything locally.

Table of Contents

• Why Global Rollouts Break Naïve Measurement

• What Synthetic Control Actually Does

• Prerequisites

• Setting Up the Working Example

• Step 1: Fit Donor Weights with SLSQP

• Step 2: Plot Treated vs Synthetic Control Trajectories

• Step 3: In-Space Placebo Permutation Test

• Step 4: Leave-One-Out Donor Sensitivity

• Step 5: Cluster Bootstrap 95% Confidence Intervals

• When Synthetic Control Fails

• What to Do Next

Why Global Rollouts Break Naïve Measurement

The math of an A/B test is elegant because of one assumption: treatment assignment is independent of everything else. Flip a coin: half your workspaces get Claude 4.6, and half stay on 4.5. The coin flip breaks every possible confound. The global rollout world has no coin.

Three mechanisms make the naive before/after misleading.

1. Co-occurring product changes: Shipping a model upgrade rarely happens in isolation. The same week, the onboarding team ships a redesigned tutorial, the pricing team runs a promotion, or customer success reaches out to enterprise accounts about the new capabilities. Your before/after picks up the sum.

2. Seasonal and market drift: Weekly usage patterns, monthly billing cycles, and quarterly procurement cycles all move outcome metrics. A 3 pp lift in week 20 looks like the model upgrade, but in fact, users returned from spring break.

3. Peer-company dynamics: A competitor releases a buggy update, and your users migrate over for a week. Your task completion rate spikes because the new users had easier queries, with zero contribution from the model itself.

All three produce the same symptom: a raw before/after that folds the upgrade's causal effect together with the causal effect of every other week-20 event.

In this tutorial's dataset, the naïve gap is +0.0515, nearly equal to the ground-truth +0.05. That coincidence is the scariest failure mode: the naive number sometimes lands correctly by accident, and without a counterfactual, you can't tell luck from truth.

What Synthetic Control Actually Does

Figure 1 (above): Schematic of the synthetic control construction. The gray curves are donor workspaces that remain on the old model. The dashed navy curve is the weighted combination of donors that best tracks the treated unit (red) during the pre-treatment window marked by the blue bracket below the x-axis.

After the treatment date (week 20, dotted vertical line), the weights stay frozen, and the dashed curve projects forward as the counterfactual, while the treated unit moves upward. The gap between the two curves in the post-treatment window is the causal-effect estimate.

The key design choice the figure illustrates is that weights are fit once, using only pre-treatment data, and never refit using post-treatment data.

Synthetic control finds a weighted combination of untreated units whose outcome trajectory closely matches the treated unit's in the pre-treatment period. Once the weights are fixed, you project the synthetic unit's trajectory forward into the post-treatment period and read off the gap between the two lines.

In your AI product context: if wave-2 workspaces didn't get the model upgrade at the same time as wave-1 workspaces, each wave-2 workspace is a candidate donor. The optimizer finds the combination of wave-2 workspaces whose weighted pre-upgrade trajectory best matches wave 1's. After week 20 (when wave 1 was upgraded), the gap between wave 1 and its synthetic twin is the causal-effect estimate, provided that the following three identification assumptions hold.

These identification assumptions work together.

• First, pre-period fit (the convex-hull condition): the treated unit's pre-treatment trajectory must lie inside the convex hull of the donor trajectories, which is what the non-negativity and sum-to-1 constraints enforce.

• Second, no interference for donors (SUTVA for the donor pool): the treatment on the treated unit must not affect the donors. Shared API rate-limit pools or users migrating between workspaces both break this.

• Third, stable donor composition: the donors must not experience structural breaks unrelated to the treatment during the post-period. Violate any one, and the gap is biased even when the pre-period fit looks perfect. The failure modes section walks through each.

One geometric note: with T₀ pre-treatment periods and J donors, pre-period overfitting becomes serious when J approaches T₀. This tutorial runs with T₀ = 20 and J = 25, which sits in the danger zone. The LOO sensitivity step later is the right diagnostic for whether the fit reflects genuine comparability or overfitting.

Prerequisites

You'll need Python 3.11 or newer, comfort with pandas and numpy, and familiarity with basic constrained optimization.

Install the packages for this tutorial:

pip install numpy pandas scipy matplotlib

Here's what's happening: four packages cover the full pipeline. Pandas loads the user-level log, NumPy handles panel arithmetic, SciPy provides the SLSQP solver to enforce the convex-combination constraint on the donor weights, and matplotlib renders the trajectory plot and the placebo distribution.

Clone the companion repo to get the synthetic dataset:

git clone https://github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm.git
cd product-experimentation-causal-inference-genai-llm
python data/generate_data.py --seed 42 --n-users 50000 --out data/synthetic_llm_logs.csv

Here's what's happening: the clone pulls the companion repo, and generate_data.py produces the shared synthetic dataset used across the series. Seed 42 keeps the dataset reproducible, and 50,000 users give a clean signal for the estimator in this tutorial. The output CSV lands at data/synthetic_llm_logs.csv.

Setting Up the Working Example

The synthetic dataset simulates a SaaS product with 50,000 users spread across 50 workspaces. Workspaces 0 through 24 are in wave 1, which received the model upgrade at week 20. Workspaces 25 through 49 are in wave 2, which stayed on the old model through week 29.

The ground-truth causal effect baked into the data generator is a +5 percentage-point increase in task completion for wave-1 users in the post-treatment period. You know the truth, so you can check what the synthetic control recovers.

Load the data and aggregate to a workspace-by-week panel:

import numpy as np
import pandas as pd

df = pd.read_csv("data/synthetic_llm_logs.csv")

PRE = 20         # weeks 0-19 are pre-treatment
WINDOW = 30      # analysis window weeks 0-29

df_window = df[df.signup_week < WINDOW].copy()

panel = (
    df_window.groupby(["workspace_id", "signup_week"])
    ["task_completed"].mean().reset_index()
)
panel.columns = ["workspace_id", "week", "task_completed"]

pivot = panel.pivot(
    index="week", columns="workspace_id", values="task_completed"
)
pivot = pivot.interpolate(method="linear", axis=0).ffill().bfill()

ws_wave = df.groupby("workspace_id").wave.first()
wave1_ws = sorted(ws_wave[ws_wave == 1].index.tolist())
wave2_ws = sorted(ws_wave[ws_wave == 2].index.tolist())

treated_series = pivot[wave1_ws].mean(axis=1).values
donor_matrix = pivot[wave2_ws].values

print(f"Treated series shape: {treated_series.shape}")
print(f"Donor matrix shape:   {donor_matrix.shape}")
print(f"Users per workspace-week: ~{len(df_window) / (50 * WINDOW):.1f}")
print(f"Pre-period treated mean  (weeks 0-19):  {treated_series[:PRE].mean():.4f}")
print(f"Post-period treated mean (weeks 20-29): {treated_series[PRE:].mean():.4f}")

Expected output:

Treated series shape: (30,)
Donor matrix shape:   (30, 25)
Users per workspace-week: ~19.2
Pre-period treated mean  (weeks 0-19):  0.5927
Post-period treated mean (weeks 20-29): 0.6421

Here's what's happening: you restrict to the 30-week window, aggregate user rows to a workspace-by-week panel, and reshape so rows are weeks and columns are workspaces. Interpolation fills any missing cells (each cell averages about 19 users). The treated series is the mean across all 25 wave-1 workspaces, pooling roughly 480 users per week to smooth cell-level noise.

The donor matrix keeps each wave-2 workspace as a separate column: 25 time series, each covering weeks 0 through 29. The pre-period treated mean of 0.5927 and the post-period mean of 0.6421 yield a raw before/after gap of +5.15 pp, which coincidentally sits near the ground-truth +5 pp and is contaminated by everything else that moved in weeks 20 through 29.

Figure 2: The diagnostic on the real 50,000-user dataset. Top panel: wave 1's trajectory in red and the fitted synthetic control in navy dashed, with pre-period RMSE of 3.74 pp and a post-treatment gap averaging +8.29 pp. Bottom panel: the placebo distribution built by re-fitting the synthetic control with each of the 25 donor workspaces standing in as the placebo treated unit. The observed gap lies outside the full placebo range, which drives the pseudo p-value in Step 3.

Where Figure 1 schematically showed the method, this figure shows that it produces a pre-period fit tight enough to make the post-period gap interpretable and a placebo distribution that discriminates the observed effect from noise.

Step 1: Fit Donor Weights with SLSQP

The synthetic control weight vector w is the solution to a constrained optimization problem: minimize the pre-period mean squared error between the treated series and the weighted combination of donor series, subject to each weight being in [0, 1] and all weights summing to 1. The non-negativity and sum-to-1 constraints together define a convex combination, which is what prevents extrapolation beyond the support of the donor pool.

from scipy.optimize import minimize

n_donors = len(wave2_ws)
Y_pre = treated_series[:PRE]
D_pre = donor_matrix[:PRE, :]

def objective(w):
    return np.mean((Y_pre - D_pre @ w) ** 2)

w0 = np.ones(n_donors) / n_donors
bounds = [(0, 1)] * n_donors
constraints = [{"type": "eq", "fun": lambda w: w.sum() - 1}]

result = minimize(
    objective, w0, method="SLSQP", bounds=bounds,
    constraints=constraints,
    options={"ftol": 1e-12, "maxiter": 5000},
)
w_opt = result.x

pre_mse = float(np.mean((Y_pre - D_pre @ w_opt) ** 2))
pre_rmse = float(np.sqrt(pre_mse))
nz = int((w_opt > 0.001).sum())

print(f"Optimization converged: {result.success}")
print(f"Non-zero donor weights (|w| > 0.001): {nz}")
print(f"Pre-period MSE:  {pre_mse:.6f}")
print(f"Pre-period RMSE: {pre_rmse:.4f}  "
      f"({pre_rmse * 100:.2f} percentage points)")

synth_full = donor_matrix @ w_opt
gap = float((treated_series[PRE:] - synth_full[PRE:]).mean())
print(f"\nObserved post-period gap: {gap:+.4f}  (ground truth = +0.0500)")

nz_pairs = sorted(
    [(ws, w_opt[i]) for i, ws in enumerate(wave2_ws) if w_opt[i] > 0.001],
    key=lambda x: -x[1]
)
print("\nTop 5 donor weights:")
for ws_id, weight in nz_pairs[:5]:
    print(f"  workspace {ws_id}: w = {weight:.4f}")

Expected output:

Optimization converged: True
Non-zero donor weights (|w| > 0.001): 12
Pre-period MSE:  0.001400
Pre-period RMSE: 0.0374  (3.74 percentage points)

Observed post-period gap: +0.0829  (ground truth = +0.0500)

Top 5 donor weights:
  workspace 35: w = 0.2016
  workspace 40: w = 0.1900
  workspace 25: w = 0.1638
  workspace 32: w = 0.0872
  workspace 36: w = 0.0784

Here's what's happening: the objective function computes the mean squared error between the treated pre-period series and the dot product of the donor matrix with the weight vector.

SLSQP handles the non-negativity bounds and the sum-to-1 equality constraint simultaneously. The w > 0.001 threshold classifies 12 donors as non-zero. SLSQP doesn't guarantee exact zeros at inactive constraints, so the threshold is a display convention. Pre-period RMSE of 3.74 pp measures how closely the weighted donors tracked the treated unit before the upgrade. The observed post-period gap of +0.0829 is the headline estimate, which overshoots the ground-truth +5 pp, as Step 5 quantifies with a confidence interval.

The weights are fixed at the end of the pre-period and never re-estimated using post-treatment data. Any divergence after week 20 reflects movement the optimizer had no opportunity to fit.

Step 2: Plot Treated vs Synthetic Control Trajectories

The primary visual diagnostic for synthetic control is the trajectory overlay: plot both series together, mark the treatment date, and confirm that the synthetic control tracks the treated unit in the pre-period and that a gap opens in the post-period.

A tight pre-period fit is the visible signal that the identification condition holds. A ragged fit means the treated unit is outside the convex hull of the donors, and the whole exercise is suspect.

import matplotlib.pyplot as plt

weeks = np.arange(WINDOW)

fig, ax = plt.subplots(figsize=(9, 4.5))
ax.plot(weeks, treated_series, marker="o", linewidth=1.8,
        color="#C44E52", label="Wave 1 (treated)")
ax.plot(weeks, synth_full, marker="s", linestyle="--",
        linewidth=1.8, color="#4C72B0", label="Synthetic control")
ax.axvline(PRE, color="#555555", linestyle=":", linewidth=1.4,
           label="Model upgrade (week 20)")
ax.set_xlabel("Signup week")
ax.set_ylabel("Mean task completion rate")
ax.set_title("Treated unit vs synthetic control")
ax.legend(frameon=False)
plt.tight_layout()
plt.show()

post_gap = treated_series[PRE:] - synth_full[PRE:]
print("Post-period weekly gaps (treated minus synthetic):")
for wk, g in zip(range(PRE, WINDOW), post_gap):
    print(f"  week {wk}: {g:+.4f}")
print(f"\nMean gap: {post_gap.mean():+.4f}")

Expected output:

Post-period weekly gaps (treated minus synthetic):
  week 20: +0.0398
  week 21: +0.1663
  week 22: +0.1019
  week 23: +0.1535
  week 24: +0.1071
  week 25: +0.1047
  week 26: +0.0424
  week 27: +0.0326
  week 28: +0.0327
  week 29: +0.0479

Mean gap: +0.0829

Here's what's happening: the two lines track each other in the pre-period, confirming the fit assumption. After week 20, the treated series moves above the synthetic control, and the weekly gaps are all positive with a mean of +8.29 pp.

The spread across weeks (from +3.26 pp to +16.63 pp) is how much week-to-week noise the estimator absorbs. A single bad week could swing the mean by a percentage point, which is why the placebo and LOO steps that follow matter more than any single point estimate.

Step 3: In-Space Placebo Permutation Test

You can't run a standard t-test on a single treated unit. The synthetic control has one treated observation (wave 1) and 25 donor observations, which is not a setup for which any conventional p-value applies.

The standard validation is the in-space placebo permutation test. Treat each donor in turn as if it were the "treated" unit, re-fit the synthetic control using the remaining 24 donors as its placebo pool, record the placebo post-period gap, and compare the observed gap to the distribution of placebos.

placebo_gaps = []

for j in range(n_donors):
    placebo_treated = donor_matrix[:, j]
    placebo_pool = np.delete(donor_matrix, j, axis=1)
    n_p = placebo_pool.shape[1]

    def obj_p(w):
        return np.mean((placebo_treated[:PRE] - placebo_pool[:PRE] @ w) ** 2)

    res_p = minimize(
        obj_p, np.ones(n_p) / n_p, method="SLSQP",
        bounds=[(0, 1)] * n_p,
        constraints=[{"type": "eq", "fun": lambda w: w.sum() - 1}],
        options={"ftol": 1e-12, "maxiter": 5000},
    )
    synth_p = placebo_pool @ res_p.x
    placebo_gaps.append((placebo_treated[PRE:] - synth_p[PRE:]).mean())

placebo_gaps = np.array(placebo_gaps)
observed_gap = gap

rank = int((np.abs(placebo_gaps) >= abs(observed_gap)).sum())
pseudo_p = (rank + 1) / (len(placebo_gaps) + 1)

print(f"Observed gap:      {observed_gap:+.4f}")
print(f"Placebo mean gap:  {placebo_gaps.mean():+.4f}")
print(f"Placebo std gap:   {placebo_gaps.std():.4f}")
print(f"Placebo gap range: [{placebo_gaps.min():+.4f}, "
      f"{placebo_gaps.max():+.4f}]")
print(f"|placebo| >= |observed|: {rank} of {len(placebo_gaps)}")
print(f"Pseudo p-value: {pseudo_p:.4f}")

Expected output:

Observed gap:      +0.0829
Placebo mean gap:  -0.0008
Placebo std gap:   0.0380
Placebo gap range: [-0.0748, +0.0707]
|placebo| >= |observed|: 0 of 25
Pseudo p-value: 0.0385

Here's what's happening: the loop iterates over all 25 wave-2 workspaces. For each one, you remove it from the donor pool, treat it as a placebo-treated unit, and re-run the SLSQP optimization. After 25 placebo runs, you count how many placebo gaps meet or exceed the observed gap in absolute value and apply the conservative (count + 1) / (N + 1) correction.

None of the 25 placebos produced a gap as extreme as the observed +0.0829, yielding a pseudo-p-value of 0.0385. That rejects the null of no effect at the 5% level. The placebo distribution centers near zero (mean -0.0008, std 3.80 pp), which is the noise floor to compare the observed gap against.

The correct statistical statement is: the observed gap is more extreme than any placebo drawn from untreated donors at the 5% level. The permutation test's power depends on the donor pool size: with 25 donors, the smallest possible pseudo-p is 1/26 = 0.0385, so you can't get a smaller p-value with this donor count. A wider placebo distribution or a smaller observed gap would rank the observation inside the placebo bulk and push the pseudo p above any useful threshold.

Step 4: Leave-One-Out Donor Sensitivity

A tight point estimate can still be fragile if it hangs on a single donor. The leave-one-out (LOO) sensitivity check drops each non-zero-weight donor in turn, refits the synthetic control on the remaining donors, and records the new gap.

Abadie (2021) recommends this as the first-line robustness check. If removing any single donor swings the gap by a large amount, you don't have a synthetic control – you have a single-donor comparison dressed up with extra weight.

def fit_and_gap(treated, donors, pre=PRE):
    n = donors.shape[1]
    def obj(w):
        return np.mean((treated[:pre] - donors[:pre] @ w) ** 2)
    res = minimize(
        obj, np.ones(n) / n, method="SLSQP",
        bounds=[(0, 1)] * n,
        constraints=[{"type": "eq", "fun": lambda w: w.sum() - 1}],
        options={"ftol": 1e-12, "maxiter": 5000},
    )
    synth = donors @ res.x
    return float((treated[pre:] - synth[pre:]).mean())


nz_idx = np.where(w_opt > 0.001)[0]
loo_rows = []
for j in nz_idx:
    kept = np.delete(donor_matrix, j, axis=1)
    gap_new = fit_and_gap(treated_series, kept)
    loo_rows.append({
        "dropped_workspace": int(wave2_ws[j]),
        "dropped_weight": float(w_opt[j]),
        "new_gap": gap_new,
    })
loo_df = pd.DataFrame(loo_rows).sort_values("dropped_weight", ascending=False)
print(loo_df.round(4).to_string(index=False))
print(f"\nLOO gap range: [{loo_df.new_gap.min():+.4f}, "
      f"{loo_df.new_gap.max():+.4f}]")
print(f"Original gap:  {gap:+.4f}")

Expected output:

 dropped_workspace  dropped_weight  new_gap
                35          0.2016   0.0945
                40          0.1900   0.0756
                25          0.1638   0.0932
                32          0.0872   0.0868
                36          0.0784   0.0739
                31          0.0718   0.0858
                29          0.0648   0.0782
                26          0.0439   0.0786
                27          0.0364   0.0867
                46          0.0350   0.0794
                39          0.0192   0.0848
                42          0.0078   0.0839

LOO gap range: [+0.0739, +0.0945]
Original gap:  +0.0829

Here's what's happening: the loop drops one non-zero-weight donor at a time and refits. All 12 LOO estimates stay positive, with the range [+7.39 pp, +9.45 pp] straddling the original +8.29 pp by about a percentage point in either direction.

No single donor drives the result. Even dropping workspace 35 (the largest weight at 0.2016) only shifts the gap to +9.45 pp because the optimizer redistributes weight across remaining donors.

That redistribution is the point of convex-combination weighting: many near-equivalent donor mixtures produce similar counterfactuals.

Step 5: Cluster Bootstrap 95% Confidence Intervals

Point estimates are only half the story. A stakeholder asking "how sure are you" wants an interval. The classical non-parametric bootstrap doesn't apply cleanly to synthetic control on a single treated unit, because resampling the one treated time series with replacement destroys the time-ordering that the estimator depends on.

A valid substitute is the user-level cluster bootstrap: resample users with replacement, rebuild the workspace-by-week panel from the resampled user log, re-fit the donor weights on the pre-period, and record the post-period gap.

Repeat 500 times. The 2.5th and 97.5th percentiles of the resulting distribution are the 95% CI.

def build_panel(df_inner):
    dfw = df_inner[df_inner.signup_week < WINDOW].copy()
    panel = (dfw.groupby(["workspace_id", "signup_week"])
             ["task_completed"].mean().reset_index())
    panel.columns = ["workspace_id", "week", "task_completed"]
    piv = panel.pivot(index="week", columns="workspace_id",
                      values="task_completed")
    piv = piv.interpolate(method="linear", axis=0).ffill().bfill()
    ws_wave_b = df_inner.groupby("workspace_id").wave.first()
    w1 = sorted(ws_wave_b[ws_wave_b == 1].index.tolist())
    w2 = sorted(ws_wave_b[ws_wave_b == 2].index.tolist())
    return piv[w1].mean(axis=1).values, piv[w2].values


rng = np.random.default_rng(7)
n = len(df)
n_reps = 500
gaps_boot = np.empty(n_reps)
for i in range(n_reps):
    sample = df.iloc[rng.integers(0, n, size=n)]
    t_b, d_b = build_panel(sample)
    gaps_boot[i] = fit_and_gap(t_b, d_b)

lo = float(np.percentile(gaps_boot, 2.5))
hi = float(np.percentile(gaps_boot, 97.5))
print(f"Post-period gap 95% CI: [{lo:+.4f}, {hi:+.4f}]")
print(f"Observed point estimate: {gap:+.4f}")
print(f"Ground truth +0.0500 inside CI: "
      f"{'YES' if lo <= 0.05 <= hi else 'NO'}")
print(f"Zero inside CI: {'YES' if lo <= 0 <= hi else 'NO'}")

Expected output:

Post-period gap 95% CI: [+0.0511, +0.1215]
Observed point estimate: +0.0829
Ground truth +0.0500 inside CI: NO
Zero inside CI: NO

Here's what's happening: you resample the user log 500 times, rebuild the panel from each resample, re-fit the weights on the pre-period, and take the 2.5th and 97.5th percentiles of the 500 resulting gaps. The 95% CI is [+5.11 pp, +12.15 pp]. It excludes zero with room to spare, so the effect is statistically meaningful.

The lower bound sits just above the +5 pp ground truth: a finite-sample upward bias typical of synthetic control on small donor panels, where each donor workspace (about 19 users per week) carries more noise than the 25-workspace treated average.

Placebo, LOO, and bootstrap together confirm a real positive effect. The point-estimate bias is the tradeoff for using single-workspace donors.

For a stakeholder report, cite the interval alongside the point estimate and note the bias direction so the team reads the number with the right calibration.

When Synthetic Control Fails

Synthetic control is a precise tool with narrow failure modes. The four most common map directly to the three identification assumptions.

1. Donor Pool Contamination (Violates No Interference)

If the upgrade shipped to wave 1 spills over to wave 2 (shared API rate-limit pools, shared prompt caches, users migrating between workspaces), the donors are contaminated, and the gap understates the true effect.

The defense is institutional: audit what changed for donor units around the treatment date, explicitly including model-level channels like shared routing, shared caching, and shared monitoring.

2. Fundamentally Different Units (Violates Pre-period Fit)

The convex-hull condition states that the treated unit must lie within the donors' support. If the treated unit is structurally different (for example, enterprise customers where every donor is an SMB), no weighting scheme yields a credible counterfactual, regardless of how tight the pre-period fit appears.

Check the weights: if the optimizer assigns 80 percent to a single donor, that donor is doing the entire job, and you should ask whether it's truly comparable.

3. Post-Treatment Shocks to Donors (Violate Stable Donor Composition)

The synthetic control projects donor behavior forward from pre-period weights. If a key donor experiences a major shock after treatment (a customer churn, an outage, a competitor release), its post-treatment trajectory is no longer a clean counterfactual. Inspect the time series of high-weight donors for unusual post-treatment patterns.

4. Overfitting Risk When J Approaches T₀ (Degrades Pre-period Fit in Practice)

The optimizer can fit the pre-period solely to noise when J ≥ T₀, creating the illusion of comparability. This tutorial runs at T₀/J = 20/25 = 0.8, in the danger zone. The LOO sensitivity check is the practical defense: if the gap holds up across donor drops, the fit reflects genuine comparability.

These failure modes stay invisible in your point estimate. They surface as a synthetic control that looks well-fit on paper and produces a gap that doesn't hold up when treatment rolls out to the next wave. Placebo test, LOO sensitivity, and bootstrap together are your defense.

What to Do Next

Synthetic control is the right tool when your feature ships globally and there's a pool of untreated units resembling the treated unit.

If treated and donor units operate at different scales, augmented synthetic control adds a bias-correction term from a linear outcome model. If you have many treated units with staggered adoption, generalized synthetic control (the gsynth R package) extends the framework.

For production Python work, pysyncon implements the full Abadie-Diamond-Hainmueller estimator with predictor-weighting via a V-matrix outer loop and adds in-time placebo tests (assigning the treatment to a pre-period date and checking for a spurious gap) that this tutorial doesn't cover. The from-scratch implementation here shows that the mechanics pysyncon is what you ship to a reviewer.

The companion notebook for this tutorial lives at github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm/tree/main/04_synthetic_control. Clone the repo, generate the synthetic dataset, and run synthetic_control_demo.ipynb (or synthetic_control_demo.py) to reproduce every code block, every number, and every figure from this tutorial.

When a model upgrade ships to every user at once, the naive before/after is usually the wrong number. Synthetic control builds "users like yours who didn't get the upgrade" from the data you already have, locks in the weights before the treatment week, and gives you a placebo distribution plus a bootstrap interval you can defend when a stakeholder asks how confident you are.

Let's say that your team built a routing layer that splits incoming queries between two models: queries with a confidence score below 0.85 go to a premium model, and those above 0.85 go to a cheaper distilled model. The premium model costs 5x as much as the cheaper one.

Your boss wants the answer that ends the debate: Is the premium model worth it for the queries it sees?

You can't run a clean A/B test, because routing is deterministic: a query at confidence 0.84 always gets premium, a query at 0.86 always gets cheap, and you can't randomize the assignment.

You also can't trust a naïve comparison of premium-routed users against cheap-routed users. Premium handles the harder queries by design (that's the reason you built the gate), so the two groups differ in query difficulty before either model touches them.

The threshold itself is your free experiment. Right at 0.85, the assignment flips, but the queries on either side of that boundary are essentially identical. A query at confidence 0.849 isn't meaningfully different from a query at 0.851. Any differences in outcomes between the two narrow groups stem solely from the routing decision. That's what regression discontinuity design (RDD) reads.

In this tutorial, you'll use Python to estimate the causal effect of premium routing on task completion using sharp RDD with local linear regression. You'll sweep bandwidths to test estimate stability, run a manipulation diagnostic, check robustness with a quadratic specification, and bootstrap 95% confidence intervals around every point estimate.

The LLM telemetry is a 50,000-user synthetic dataset with the ground-truth premium-routing effect baked in at +6 percentage points, so you can verify that RDD recovers it.

Companion code: every code block runs end-to-end in the companion notebook.

Table of Contents

• Why Threshold Routing is a Natural Experiment

• What Regression Discontinuity Actually Does

• Prerequisites

• Setting Up the Working Example

• Step 1: A Sharp RDD with Local Linear Regression

• Step 2: Try Different Bandwidths

• Step 3: Checking for Manipulation at the Threshold

• Step 4: Quadratic Specification as a Robustness Check

• Step 5: Bootstrap Confidence Intervals

• When Regression Discontinuity Fails

• What to Do Next

Why Threshold Routing is a Natural Experiment

The product reason this routing rule exists is to help your team spend the premium model budget where it earns its keep. Low-confidence queries are the harder ones, which is where a stronger model has the most upside. High-confidence queries already look easy enough for the cheap model to handle.

You'll see this routing direction across confidence-score gates for Q&A assistants, query-complexity gates in multi-model gateways like OpenRouter, safety-score gates in content moderation, and latency-budget gates that re-route when the cheap model would exceed a p99 latency budget.

The mechanism is the same in every case: a continuous score, a threshold, and a deterministic routing rule.

What makes this setup useful for causal inference is that users don't pick which model they get. A query lands, the system computes confidence, and the routing layer decides. Right at the threshold, the user's experience flips from premium to cheap based on a difference too small to be meaningful.

Again, a query at 0.849 confidence isn't shipping a different problem to the model than a query at 0.851. Anything that differs in outcomes between those two groups is the routing decision speaking. The underlying query is the same.

That local randomness is the experiment RDD reads from. You don't need a randomized control group, you don't need a propensity score. And you don't need an instrument, you need a sharp threshold that nobody can game.

What Regression Discontinuity Actually Does

The jump at the threshold is the causal effect, which is the number a product team can act on. RDD reads it by fitting two separate regression lines to the outcome: one for users just below the threshold and one for users just above. The vertical difference between those two fitted lines at the cutoff is the local average treatment effect at that point.

Graphically, picture task completion on the y-axis and query confidence on the x-axis. Completion generally trends with confidence (easier queries complete more often). At exactly 0.85, though, users below the cutoff get premium routing, and users above get cheap.

If premium routing helps, you'd see a sharp upward jump in task completion just below 0.85, then disappear just above. Approached from left to right with confidence rising, the visual reads as a downward step at 0.85, because you're moving from the premium-treated zone into the cheap-treated zone.

Figure 1. Conceptual schematic. Two outcome trajectories, one for premium-routed queries (confidence below 0.85) and one for cheap-routed queries (confidence above 0.85), meet at the threshold but don't match. The vertical gap between their endpoints at 0.85 is the local causal effect of premium routing.

That gap is identified under two named assumptions:

1. No manipulation of the running variable: Users (or your system) can't precisely nudge a query's confidence score across the cutoff. If anyone can game their score to land just below 0.85 and grab premium routing, the cutoff is no longer drawn at random, and RDD breaks.

2. Continuity of potential outcomes at the cutoff: Every other factor that affects task completion (query type, user expertise, workspace tenure, time of day) varies smoothly across 0.85. Only the routing assignment changes discontinuously at exactly the threshold. If a second product rule fires at 0.85 (a different logging level, a separate UI treatment, a retry policy), RDD will attribute that rule's effect to the routing decision.

These are the two assumptions you check before you trust the estimate. Step 3 below tests the first one. The second is a structural property of your system that you have to know cold.

Two practical choices shape every RDD: the bandwidth (how close to the cutoff to restrict the analysis) and the functional form (linear, quadratic, or local polynomial).

Narrow bandwidths cut potential bias by staying close to the local-randomization zone, but they shrink the sample. Linear specifications are stable, though they assume the underlying relationship can be approximated by a straight line on each side.

You'll try both linear and quadratic specifications at multiple bandwidths to see whether the answer holds.

The article uses sharp RDD throughout, since assignment is a deterministic function of confidence (below 0.85 always premium, above 0.85 always cheap). When the threshold is probabilistic and compliance is partial, the design is a fuzzy RDD, which requires an instrumental variables framework that you can implement using the rdrobust Python package.

Prerequisites

You need Python 3.11 or newer, comfort with pandas and statsmodels, and rough familiarity with linear regression and interaction terms.

Install the packages used in this tutorial:

pip install numpy pandas statsmodels matplotlib scipy

Here's what's happening: four standard scientific Python libraries plus matplotlib for the diagnostic visualization. Nothing exotic.

Clone the companion repo and generate the synthetic dataset:

git clone https://github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm.git
cd product-experimentation-causal-inference-genai-llm
python data/generate_data.py --seed 42 --n-users 50000 --out data/synthetic_llm_logs.csv

Here's what's happening: the data generator draws 50,000 users with a query_confidence score from a Beta(5,2) distribution, applies the routing rule (routed_to_premium = query_confidence < 0.85), and bakes a +6-percentage-point premium routing effect into task_completed. Same seed, same dataset, every time.

Setting Up the Working Example

The dataset simulates a SaaS product that routes queries between a premium and a cheap model based on confidence score. The threshold is 0.85, and the ground-truth causal effect of premium routing is +6 percentage points on task completion. You know the truth, so you can check whether RDD recovers it.

Load the data and look at the routing breakdown:

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

df = pd.read_csv("data/synthetic_llm_logs.csv")
print(f"Loaded {len(df):,} rows, {df.shape[1]} columns")

print("\nRouting breakdown:")
counts = df.routed_to_premium.value_counts().to_dict()
print(f"  Premium-routed (confidence < 0.85):  {counts.get(1, 0):,}")
print(f"  Cheap-routed   (confidence >= 0.85): {counts.get(0, 0):,}")

print("\nQuery confidence distribution:")
print(df.query_confidence.describe().round(3))

Expected output:

Loaded 50,000 rows, 16 columns

Routing breakdown:
  Premium-routed (confidence < 0.85):  38,874
  Cheap-routed   (confidence >= 0.85): 11,126

Query confidence distribution:
count    50000.000
mean         0.715
std          0.159
min          0.078
25%          0.611
50%          0.736
75%          0.838
max          0.998

Here's what's happening: about 78% of queries land below the 0.85 cutoff and get premium routing. The Beta(5,2) distribution is skewed toward the upper end, with a median of 0.736, and most of its mass still sits below 0.85. The remaining 22% are queries that the model already feels confident about, and they go to the cheap model.

Before any regression, look at the naïve comparison every product team is tempted to run:

naive = (
    df[df.routed_to_premium == 1].task_completed.mean()
    - df[df.routed_to_premium == 0].task_completed.mean()
)
print(f"Naive premium-vs-cheap effect: {naive:+.4f}  (ground truth = +0.06)")

Expected output:

Naive premium-vs-cheap effect: +0.0632  (ground truth = +0.06)

Here's what's happening: the naive estimate sits at +0.0632, which is suspiciously close to the truth. That's a coincidence of this specific synthetic dataset, where the only confounder of premium vs. cheap is query_confidence itself, and the outcome doesn't depend on confidence except through routing.

In production, you almost never get this lucky. User expertise, prompt phrasing, time of day, and a dozen unobserved query traits all correlate with confidence and with completion.

A naïve comparison in a real system can be off by 50% or more in either direction. RDD gives you identification that doesn't depend on the absence of hidden confounders.

Step 1: A Sharp RDD with Local Linear Regression

The basic sharp RDD estimator is a local linear regression. Restrict to users whose confidence sits within a bandwidth of the cutoff, fit separate linear slopes on each side, and read off the jump at 0.85.

cutoff = 0.85
bw = 0.10

near = df[(df.query_confidence > cutoff - bw)
          & (df.query_confidence < cutoff + bw)].copy()
near["below_cutoff"] = (near.query_confidence < cutoff).astype(int)
near["rc"] = near.query_confidence - cutoff

rdd_model = smf.ols(
    "task_completed ~ below_cutoff + rc + below_cutoff:rc",
    data=near,
).fit(cov_type="HC3")

effect = rdd_model.params["below_cutoff"]
print(f"RDD effect at cutoff (LATE): {effect:+.4f}")
print(f"Std error (HC3):             {rdd_model.bse['below_cutoff']:.4f}")
print(f"p-value:                     {rdd_model.pvalues['below_cutoff']:.4f}")
print(f"N users in [0.75, 0.95):     {len(near):,}")

Expected output:

RDD effect at cutoff (LATE): +0.0548
Std error (HC3):             0.0131
p-value:                     0.0000
N users in [0.75, 0.95):     21,689

Here's what's happening: the model fits separate intercepts and slopes on each side of 0.85 (below_cutoff is the side indicator, rc is confidence centered at the cutoff). The coefficient on below_cutoff reads off the vertical jump at the threshold, which is the local average treatment effect (LATE) for queries with confidence near 0.85. You get +0.0548, within sampling noise of the +0.06 ground truth.

Three notes on the specification. First, task_completed is binary, so this is a linear probability model. For RDD with a binary outcome at the cutoff, the linear probability model is standard practice because local linearity is the identifying assumption either way. Logit at the cutoff is an alternative if you need bounded predictions globally.

Second, the standard errors are used cov_type="HC3" to relax the homoskedasticity assumption, which is almost always wrong for binary outcomes.

Third, the dataset has one query per user with no within-user clustering, so cluster-robust standard errors aren't needed here. In a setting with multiple queries per user, you'd cluster on user_id.

The next diagnostic to look at is the confidence distribution near the cutoff. Figure 2 shows what 50,000 queries look like in the bandwidth window:

Figure 2. Real distribution from the 50,000-user synthetic dataset. Unlike the schematic in Figure 1, this shows the actual query density by confidence score, with the routing threshold annotated. The bottom panel counts how many queries land in each 2-percentage-point bin near the cutoff (2,461 / 2,481 / 2,335 / 2,229 / 2,048 across the 0.80–0.90 range). The roughly uniform spread is the visual signal that no manipulation is concentrating users on one side of the threshold.

Step 2: Try Different Bandwidths

Bandwidth choice matters. Too narrow and you have too few observations, so the confidence interval blows up. Too wide and you're extrapolating into regions where the linear specification is no longer a reasonable local approximation.

The honest move is to try multiple bandwidths and report whether the estimate holds.

results = []
for bw in [0.05, 0.10, 0.15, 0.20]:
    sub = df[(df.query_confidence > cutoff - bw)
             & (df.query_confidence < cutoff + bw)].copy()
    sub["below_cutoff"] = (sub.query_confidence < cutoff).astype(int)
    sub["rc"] = sub.query_confidence - cutoff

    m = smf.ols(
        "task_completed ~ below_cutoff + rc + below_cutoff:rc",
        data=sub,
    ).fit(cov_type="HC3")

    results.append({
        "bandwidth": bw,
        "n": len(sub),
        "effect": m.params["below_cutoff"],
        "se": m.bse["below_cutoff"],
        "p": m.pvalues["below_cutoff"],
    })

print(pd.DataFrame(results).round(4).to_string(index=False))

Expected output:

 bandwidth      n  effect     se       p
      0.05  11554  0.0635  0.0183  0.0005
      0.10  21689  0.0548  0.0131  0.0000
      0.15  29137  0.0618  0.0112  0.0000
      0.20  34074  0.0614  0.0107  0.0000

Here's what's happening: four bandwidths from ±0.05 to ±0.20 around the cutoff, refitting the same RDD specification at each. The estimates range from +0.0548 to +0.0635, all in the same neighborhood as the +0.06 ground truth, with standard errors that shrink as the bandwidth widens and grow as it narrows. Every p-value is well below 0.05. Whether the estimates are "stable" depends on the confidence intervals around them, which Step 5 produces with the bootstrap.

Step 3: Checking for Manipulation at the Threshold

RDD is valid only if users can't precisely manipulate the running variable around the cutoff. If your users (or your system) can nudge confidence scores just below 0.85 to force premium routing, you get a density spike at the cutoff, and the RDD estimate is contaminated.

The standard diagnostic is the McCrary density test, which checks whether the distribution of the running variable has a sharp jump at the cutoff. The simple version: bin the data tightly around 0.85 and check whether the counts on the two sides are similar.

print("User counts in 2-percentage-point bins around 0.85:")
for lo in [0.80, 0.82, 0.84, 0.86, 0.88]:
    hi = lo + 0.02
    cnt = ((df.query_confidence >= lo) & (df.query_confidence < hi)).sum()
    print(f"  [{lo:.2f}, {hi:.2f}):  n = {cnt:,}")

Expected output:

User counts in 2-percentage-point bins around 0.85:
  [0.80, 0.82):  n = 2,461
  [0.82, 0.84):  n = 2,481
  [0.84, 0.86):  n = 2,335
  [0.86, 0.88):  n = 2,229
  [0.88, 0.90):  n = 2,048

Here's what's happening: counts trend gently downward across the bandwidth because Beta(5,2) places more mass at higher confidence levels, and the density tapers as it approaches 1.0. There's no spike or dip at the 0.84–0.86 bin that straddles the cutoff. The 433-user spread across all five bins is consistent with smooth tapering of the underlying density.

That's the pattern you want when manipulation is absent. For a more rigorous test, the rddensity Python package implements the formal McCrary procedure with bias-corrected standard errors.

What manipulation looks like when it's real: a spike in users at confidences just barely below 0.85 (they're being nudged into premium routing) and a dip just above. If you see that pattern, the RDD estimate overstates the causal effect because the users right below 0.85 differ in motivation from those right above. They cared enough to manipulate the score, and they'd have shown different outcomes even under random routing.

Step 4: Quadratic Specification as a Robustness Check

If the true relationship between confidence and task completion isn't exactly linear, a local linear RDD can mistake the curvature for a jump. The standard robustness check allows quadratic terms on both sides of the cutoff and tests whether the estimate holds.

near = df[(df.query_confidence > cutoff - 0.10)
         & (df.query_confidence < cutoff + 0.10)].copy()
near["below_cutoff"] = (near.query_confidence < cutoff).astype(int)
near["rc"] = near.query_confidence - cutoff
near["rc2"] = near.rc ** 2

rdd_quad = smf.ols(
    "task_completed ~ below_cutoff + rc + below_cutoff:rc"
    " + rc2 + below_cutoff:rc2",
    data=near,
).fit(cov_type="HC3")

print(f"Linear RDD    (bw=0.10):  effect = +0.0548, p < 0.0001")
print(f"Quadratic RDD (bw=0.10):  effect = "
      f"{rdd_quad.params['below_cutoff']:+.4f}, "
      f"p = {rdd_quad.pvalues['below_cutoff']:.4f}")

Expected output:

Linear RDD    (bw=0.10):  effect = +0.0548, p < 0.0001
Quadratic RDD (bw=0.10):  effect = +0.0569, p = 0.0036

Here's what's happening: the quadratic specification adds squared terms and interactions with the cutoff indicator, allowing the relationship to curve differently on each side. The below_cutoff coefficient still captures the jump at the threshold, now under a more flexible specification.

The two estimates differ by 0.0022, both close to the +0.06 ground truth, and both are significant at p < 0.01. The answer doesn't change when you let the model bend.

When linear and quadratic specifications disagree noticeably, you have a real signal. With small samples (a few thousand at narrow bandwidths), the quadratic version can lose power because four extra parameters need data to be identified.

The standard move is to widen the bandwidth and re-run both specifications. If they still disagree at wider bandwidths, the linear approximation is wrong, and you should report both numbers.

Step 5: Bootstrap Confidence Intervals

Every point estimate in this article is a single number from a finite sample. The bootstrap quantifies how much that number would move under resampling, which is what a confidence interval describes.

def bootstrap_ci(df, cutoff, bw, quadratic=False, n_reps=500, seed=7):
    rng = np.random.default_rng(seed)
    near = df[(df.query_confidence > cutoff - bw)
              & (df.query_confidence < cutoff + bw)].copy()
    near["below_cutoff"] = (near.query_confidence < cutoff).astype(int)
    near["rc"] = near.query_confidence - cutoff
    if quadratic:
        near["rc2"] = near.rc ** 2
        formula = ("task_completed ~ below_cutoff + rc + below_cutoff:rc"
                   " + rc2 + below_cutoff:rc2")
    else:
        formula = "task_completed ~ below_cutoff + rc + below_cutoff:rc"

    n = len(near)
    estimates = np.empty(n_reps)
    for i in range(n_reps):
        sample = near.iloc[rng.integers(0, n, size=n)]
        m = smf.ols(formula, data=sample).fit()
        estimates[i] = m.params["below_cutoff"]
    return (np.percentile(estimates, 2.5), np.percentile(estimates, 97.5))


print("Linear RDD (bw=0.10):")
lo, hi = bootstrap_ci(df, cutoff, bw=0.10)
print(f"  effect = +0.0548   95% CI: [{lo:+.4f}, {hi:+.4f}]")

print("\nBandwidth sensitivity:")
for bw, eff in [(0.05, 0.0635), (0.10, 0.0548), (0.15, 0.0618), (0.20, 0.0614)]:
    lo, hi = bootstrap_ci(df, cutoff, bw=bw)
    print(f"  bw = {bw:.2f}   effect = {eff:+.4f}   "
          f"95% CI: [{lo:+.4f}, {hi:+.4f}]")

print("\nQuadratic RDD (bw=0.10):")
lo, hi = bootstrap_ci(df, cutoff, bw=0.10, quadratic=True)
print(f"  effect = +0.0569   95% CI: [{lo:+.4f}, {hi:+.4f}]")

Expected output:

Linear RDD (bw=0.10):
  effect = +0.0548   95% CI: [+0.0278, +0.0817]

Bandwidth sensitivity:
  bw = 0.05   effect = +0.0635   95% CI: [+0.0244, +0.0986]
  bw = 0.10   effect = +0.0548   95% CI: [+0.0278, +0.0817]
  bw = 0.15   effect = +0.0618   95% CI: [+0.0381, +0.0823]
  bw = 0.20   effect = +0.0614   95% CI: [+0.0420, +0.0808]

Quadratic RDD (bw=0.10):
  effect = +0.0569   95% CI: [+0.0205, +0.0959]

Here's what's happening: the bootstrap resamples the bandwidth-restricted data with replacement 500 times, refits the RDD on each replicate, and collects the below_cutoff coefficient. The 2.5th and 97.5th percentiles of those 500 estimates form the 95% interval. Every interval covers the +0.06 ground truth, every interval excludes zero, and the bandwidth sweep produces overlapping intervals.

That's quantitative stability, verified by resampling across the full bandwidth range. Intervals widen as the bandwidth shrinks and narrow as it grows. The quadratic interval is wider than the linear one because the four extra parameters absorb degrees of freedom.

One thing the intervals do NOT do on this dataset: exclude the naive +0.0632 estimate. That's because the data generator doesn't bake in confounding by query confidence. The only difference between the premium and cheap groups in expectations is the +6pp routing effect itself, so the naïve comparison is close to the truth.

Real systems are messier. In a production setting where unobserved query traits affect both the routing assignment and task completion, the naïve estimate would diverge from the RDD estimate, and the bootstrap intervals would tell you which one to trust.

When Regression Discontinuity Fails

RDD looks clean, but several specific failure modes can destroy the identification. Each one maps to a violation of one of the two named assumptions.

Users manipulate the running variable (violates assumption 1). The whole setup depends on users (or any upstream service) being unable to precisely control which side of the cutoff they land on. Any system that reveals the cutoff and gives users a way to influence their score (a retry mechanism, a prompt engineering workaround, a confidence-inflating trick) breaks RDD.

Run the density check in Step 3 every time. If you find manipulation, switch to a fuzzy RDD that treats the threshold as probabilistic, or abandon the approach.

Other policies fire at the same cutoff (violates assumption 2). If your product has additional rules that activate at 0.85 (a separate UI treatment, a different logging level, a different retry policy), RDD can't separate the routing effect from those other policy effects. Audit the full rule book for anything that shares the threshold.

The threshold has noise or overrides (violates assumption 1, in the structural sense). Maybe routing isn't strictly deterministic at 0.85 – it may have random jitter, or a second rule may override the main rule in some cases.

If assignment to the premium model isn't a deterministic function of query_confidence, you have a fuzzy RDD, which requires an instrumental variables framework. The rdrobust package handles both sharp and fuzzy designs.

Curvature masquerading as a jump (breaks the linear approximation that supports identification at the cutoff). Sharp RDD assumes linearity is a reasonable local approximation. When the underlying outcome-confidence relationship is strongly curved, the linear specification can mistake the bend for a jump.

Step 4's quadratic robustness check is the standard diagnostic. If linear and quadratic disagree, widen the bandwidth and re-run both.

Extrapolation bias (a continuity issue, reframed). RDD estimates are strictly local to the cutoff. The +0.06 effect at 0.85 tells you nothing about what premium routing would do for queries with confidence 0.30 or 0.99.

If you want a global average effect, you need a different technique: propensity methods, regression with confounder adjustment, or an actual experiment.

What to Do Next

RDD is the right tool when your AI feature is gated by a continuous score and a sharp threshold.

If your feature is gated by a user-controlled toggle, propensity score methods are a better fit. If it's gated by a staged rollout across workspaces, difference-in-differences handles it. If it's gated by rules you can't observe directly but that have a random component, instrumental variables is the right choice.

For production RDD analyses, use the rdrobust Python package. It gives you optimal bandwidth selection (Calonico, Cattaneo, and Titiunik 2014), bias-corrected standard errors, and a built-in plotting utility. The companion rddensity package implements the McCrary density test you saw informally in Step 3.

The from-scratch version in this tutorial shows the mechanics. The rd-packages stack is what you ship to a reviewer.

One thing the LATE doesn't do: tell you the effect for users far from the cutoff. If a +0.06 LATE at 0.85 is enough to keep premium routing in the pipeline, you're done. If you need to know what premium would do for the easy queries you're currently sending to cheap (or the hardest queries near the floor), the next step is a small randomized rollout in those zones, scored against the RDD estimate as a calibration check. Don't generalize the LATE without evidence.

The companion notebook for this tutorial lives here on GitHub. Clone the repo, generate the synthetic dataset, and run rdd_demo.ipynb to reproduce every code block from this tutorial.

Threshold routing is one of the most common patterns in production LLM systems, and every confidence-gated routing decision in your stack is a potential RDD. Run the analysis.

Now, not everyone enjoys reading research papers or has the time to comb through and digest all that (sometimes very dense) info. So I decided to do the hard work for you and share the key insights in a series of AI paper reviews.

The goal isn’t to turn this into a heavy academic discussion, but to explain the main ideas in a clear and practical way. You'll learn what problem the paper was trying to solve, what approach it introduced, and why it mattered.

In each article, you’ll get a simple breakdown of the paper, how it works, and what you should take away from it. By the end, you should understand the key idea without needing to go through the full research paper yourself.

Paper Overview

The first paper I'll be reviewing is "Improving Language Understanding by Generative Pre-Training", by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.

Here's the actual paper if you want to read it yourself: Read the paper.

And here's a little infographic of what we'll cover here:

Table of Contents

• Executive Summary

• Goals of the Paper

• Methodology

• Transformer vs. BERT vs. GPT

• Model Architecture

• Key Techniques

• Key Findings

• Conclusions

• Limitations

• Related Work & Context

• Final Insight

• Resources

Prerequisites

To get the most out of this breakdown, it helps to be familiar with a few basic ideas:

• A general understanding of natural language processing (NLP) and how machines work with text

• A high-level idea of what a Transformer model is (you don’t need deep details, just the concept)

• The difference between supervised and unsupervised learning

• Basic machine learning concepts like training data and models

If you’re not fully comfortable with all of these, that’s okay, you can still follow along. The goal here is to keep things clear and intuitive.

Executive Summary

Before models like GPT became what we know today, there was a key limitation: AI systems were good at specific tasks, but struggled with general understanding.

In this paper, the authors introduce a simple but powerful idea. Instead of training a model separately for each task, they first train it on a large amount of unlabeled text to learn the structure of language. Then, they adapt it to specific tasks using smaller labeled datasets.

According to the authors, this two-step approach (pre-training followed by fine-tuning) allows a single model to handle many different tasks with minimal changes.

In practice, this marked a major shift: rather than building a new model for every problem, we can train one general model that learns language itself and then reuse it across tasks.

Goals of the Paper

To understand the motivation behind this work, it helps to look at the main limitations in NLP at the time.

Most models depended heavily on large labeled datasets, which weren’t always available. Many tasks simply didn’t have enough labeled data to train effective systems. On top of that, existing models were usually designed for a single task, making them hard to reuse or adapt.

Because of this, the authors aimed to reduce the reliance on labeled data and move toward a more general approach. Their goal was to build a language model that could learn from large amounts of raw text and then be applied across different tasks.

According to the paper, they also wanted to enable transfer learning: the ability to take knowledge learned from one task and apply it to others. They also wanted to improve performance without needing to redesign a new model each time.

Methodology

To understand how the authors approached this problem, let’s look at the core idea behind their method.

Pre-Training

At the heart of the paper is a simple but powerful approach built in two stages. The first stage is pre-training, where the model learns directly from raw text.

According to the authors, the model is trained on a large corpus of unlabeled text using a language modeling objective (predicting the next word in a sequence) – specifically, predicting the next word based on the previous ones to solve the intractable problem of high dimension probabilities. Through this process, the model gradually learns important aspects of language, such as grammar, context, structure, and general patterns.

The paper highlights that datasets like BooksCorpus are used in this stage because they contain long, continuous text. This is important, since it helps the model understand relationships across sentences rather than just short fragments.

Fine-Tuning (Adapting to Tasks)

Once the model has learned general language patterns, the next step is fine-tuning, where it is adapted to specific tasks using labeled data.

According to the authors, this includes tasks like question answering, text classification, natural language inference, and semantic similarity. Instead of building a new model for each task, the same pre-trained model is reused with only small adjustments.

In practice, this is what makes the approach powerful: the model already understands language at a general level, so it can quickly adapt to different tasks without needing to be redesigned from scratch.

Transformer vs. BERT vs. GPT

Before diving into GPT-1, it helps to understand how modern language models are structured. Most of them are based on the Transformer architecture, but they use it in different ways: encoder-only models (like BERT), decoder-only models (like GPT), or full encoder–decoder models.

The original encoder–decoder Transformer was mainly used for tasks like machine translation. Encoder-only models are typically used for understanding tasks such as text classification and sentiment analysis, while decoder-only models are designed for generation tasks like text creation, powering systems such as ChatGPT, Gemini, and Claude.

Illustration comparing Transformer, GPT, and BERT architectures, adapted from Comparing Large Language Models: GPT vs. BERT vs. T5 showing encoder-decoder, decoder-only, and encoder-only designs

Transformer vs BERT vs GPT: Key Differences

Model Architecture

To support this pre-training and fine-tuning approach, the GPT-1 model is built on a Transformer (decoder) architecture.

According to the authors, this choice is important for a few reasons. Unlike older models such as LSTMs, Transformers handle long-range dependencies more effectively, meaning they can better understand relationships between words that are far apart in a sentence.

They also rely on self-attention, a mechanism that allows the model to focus on the most relevant parts of the text when processing each word. This helps the model capture context more accurately.

Another key advantage is that Transformers make transfer learning more effective, since the same learned representations can be reused across different tasks with minimal changes.

The paper highlights that, in these transfer learning scenarios, Transformers outperform LSTM-based models.

Figure 1 from “Improving Language Understanding by Generative Pre-Training” (Radford et al., 2018), showing the Transformer architecture and task-specific input transformations.

Key Techniques

Along with the main approach, the authors introduce a few practical techniques that make the model more flexible across tasks.

According to the paper, different tasks are handled by converting them into text-based formats, so they can all be processed in a similar way. This makes it easier to use the same model across multiple problems without redesigning it each time.

Another important point is that the model requires only minimal architectural changes when switching between tasks. Most of the knowledge learned during pre-training is reused as-is.

The authors also include an auxiliary language modeling objective during fine-tuning, which helps the model retain its general understanding of language while adapting to specific tasks.

Key Findings

After training and evaluation, the results weren't just strong – they were surprisingly competitive.

According to the authors, the model outperformed state-of-the-art systems in 9 out of 12 tasks. It also showed clear improvements, including +8.9% in commonsense reasoning and +5.7% in question answering.

Another important observation is that the model performed well across datasets of different sizes, although performance was weaker on some smaller datasets.

This suggests that the pre-training step helped it generalize better, even when labeled data was limited.

In practice, what makes these results significant is that a single model was able to compete with specialized systems that were specifically designed for each individual task.

Figure 2 from “Improving Language Understanding by Generative Pre-Training” (Radford et al., 2018), illustrating performance gains from layer transfer and zero-shot learning behavior.

Conclusions

To wrap things up, this paper introduced a major shift in how AI systems are built.

According to the authors, instead of training a new model from scratch for every task, we can first teach a model the structure of language through pre-training, and then adapt it to specific tasks through fine-tuning. This simple idea turns out to be highly effective.

The key takeaway is that language models can develop a general understanding of text, especially when combined with Transformer architectures and large-scale data. This makes transfer learning practical across many different tasks.

In my view, this is what makes the paper so impactful. It doesn’t just improve performance on a few benchmarks. It changes the overall approach to building AI systems.

This idea later became the foundation for models like GPT-2, GPT-3, and ChatGPT, and continues to shape modern large language models today.

Limitations

Like any approach, this method comes with its own limitations.

According to the paper, one of the main challenges is the need for large amounts of unlabeled data during the pre-training stage, which may not always be easy to get. The model’s performance also depends heavily on how well the fine-tuning step is done.

The authors also note that multi-task learning was not fully explored in this work, leaving some open questions about how well the model can handle multiple tasks at the same time.

In practice, another limitation is that performance can be weaker when working with very small datasets, especially if the fine-tuning process is not carefully handled.

Related Work & Context

To better understand where this paper fits, it helps to look at the ideas it builds on.

According to the authors, earlier approaches such as word embeddings (like Word2Vec and GloVe), LSTM-based language models, and semi-supervised learning had already made progress in understanding language. But these methods were often limited to learning representations at the word level or required more task-specific design.

What this paper does differently is move beyond that. Instead of focusing only on individual words, it learns broader language representations that capture context and meaning across entire sequences. This shift is what enables the model to generalize better across different tasks.

Final Insight

If there’s one idea to take away from this paper, it’s this: you don’t need to teach an AI system every task separately.

According to the authors, once a model learns the structure of language, it can adapt to a wide range of tasks with minimal changes. That shift – from task-specific models to general language understanding – is what makes this work so important.

In my view, this is the moment where things really changed. What started here with GPT-1 became the foundation for the systems we use today, including ChatGPT and other modern language models.

Resources:

• Pytorch Projects for GPT series

• Word2Vec (Mikolov et al., 2013)

• GloVe (Pennington et al., 2014)

• Attention Is All You Need (Vaswani et al., 2017)

• Semi-supervised Sequence Learning (Dai and Le, 2015)

• Universal Language Model Fine-tuning for Text Classification (Howard and Ruder, 2018)

• Deep Contextualized Word Representations (Peters et al., 2018)

• Semi-supervised Multitask Learning for Sequence Labeling (Rei, 2017)

• Skip-Thought Vectors (Kiros et al., 2015)

• Supervised Learning of Universal Sentence Representations (Conneau et al., 2017)

Contact Me

• Github

• X

• Linkedin

Let's take the case of a retail shop. If we're looking at the average order value to understand customer spending, we'd load the data, run the code, and get a result of $20 per order.

Done.

Except something looks odd.

When we take a closer look, we see that most customers are buying items worth \(8 - \)15. So where's $20 coming from?

In that case, the problem isn’t data – it’s the average. This is a clean textbook trap where everything works perfectly in the textbook, but real-world data doesn’t behave nicely.

Some customers buy in bulk (very large orders), some return orders (negative quantities), and a few anomalies distort the entire picture.

In this article, we'll use the Online Retail Dataset to answer a simple but tricky question: What does “average” really mean in the real world?

Table Of Contents

• Prerequisites

• The Dataset

• Mean: The Sensitive Giant

• Median: The Robust Middle

• Beyond Averages: Understanding Spread with Quartiles

• Applying IQR to Our Dataset

• Final Comparison and Insights

• Conclusion

• Connect with me

Prerequisites

To follow along here, you'll need:

Basic Python knowledge: Understanding of variables and functions.

The Pandas library: Familiarity with loading data and basic DataFrame operations.

A development environment: Access to a tool like Jupyter Notebook, VS Code, or Google Colab.

A Dataset: For this analysis, I used the Online Retail Dataset, which is available for download here.

The Dataset

We'll work with the Online Retail Dataset, a real-world transactional dataset containing purchase records from a UK-based online retail store.

1. Source: UCI Machine Learning Repository

2. Collected by: UK-based online retail company (2010–2011)

3. Size: 541,909 transactions

4. Features: 8 attributes (InvoiceNo, StockCode, Description, Quantity, InvoiceDate, UnitPrice, CustomerID, Country)

5. Ownership: Public dataset hosted by UCI

6. License: Open for research and educational use

Mean: The Sensitive Giant

In statistics and data analysis, the terms "average" and "arithmetic mean" are often used interchangeably. We aim to find the mean total price in our dataset. Mean in the context of the Online Retail Dataset is given as:

$$\text{Average Order Value} = \frac{\text{Sum of all TotalPrice values}}{\text{Number of transactions}}$$

In our dataset, the mean is calculated by summing all transaction values (including bulk purchases and returns) and dividing by the total number of transactions. This means every value, irrespective of unusually high or any negative values, directly influences the final average.

# Load the dataset
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00352/Online%20Retail.xlsx"
df = pd.read_excel(url, engine='openpyxl')

# Clean and Feature Engineering
df = df.dropna(subset=['CustomerID'])
df['TotalPrice'] = df['Quantity'] * df['UnitPrice']

# Calculate the Mean (Average Order Value)
mean_value = df['TotalPrice'].mean()
print(f"Average Order Value (Mean): {mean_value:.2f}")

The results are as follows:

Average Order Value (Mean): 20.40

At first glance, the results may look promising: every transaction contributes equally. But that’s where the problem lies. Sometimes a few transactions, which are extremely high or low, affect the mean for all customers who lie in the closer range.

Take a look at the graph for the mean below.

The graph shows the mean Total Price for the Online Retail Dataset. We get a mean of 20.42. (Image by Author)

The graph shows a right-skewed distribution where the calculated mean of 20.40 is actually a textbook trap. The tallest bar clearly shows that the majority of transactions lie in the range of \(8 - \)15 range, but the red line is being dragged to the right by the long tail of high-value bulk orders by some customers.

In this scenario, the average price is well above what a typical customer actually spends because it's highly sensitive to outliers – and in reality, the bulk of the data lives in the lower price range.

In simple words, the mean is being pulled by some extreme values to the right, especially by some lying in the range of 200–300, which is noticeable in the graph.

Median: The Robust Middle

When the mean is distorted by extreme values, we need a metric that remains unaffected by such outliers. This is where the median comes into play.

Median is defined as the middle value after sorting the data.

In our dataset, we sort all the transactions and pick the middle one.

The formula for calculating the median is:

$$\text{Median} = \begin{cases} X_{\left[ \frac{n+1}{2} \right]} & \text{if } n \text{ is odd} \ \frac{X_{\left[ \frac{n}{2} \right]} + X_{\left[ \frac{n}{2} + 1 \right]}}{2} & \text{if } n \text{ is even} \end{cases}$$

Unlike the mean, the median doesn't depend on extreme values, and it cares only about the position of the data, not the magnitude.

# Clean and Feature Engineering
df = df.dropna(subset=['CustomerID'])
df['TotalPrice'] = df['Quantity'] * df['UnitPrice']

# Calculate only the Median
median_value = df['TotalPrice'].median()
print(f"Typical Order Value (Median): {median_value:.2f}")

The results are as follows:

Typical Order Value (Median): 11.10

Now you'll notice that the result lies in the \(8 — \)15 range, where most of the transactions lie.

The figure demonstrates the graph for the median, where we get an accurate value of the transactions by the customers. (Image by Author)

In the previous graph, the mean was pulled to the right by large orders, but the median just asks what the middle customer spends. So even if someone spends $300 or some transactions are negative, the median stays stable.

In the above figure the median graph accurately highlights the range where most of the customers lie.

Beyond Averages: Understanding Spread with Quartiles

So far, we've studied the median, but knowing the center is not enough.

To truly understand how customer spending is, we need to understand how the data is spread, and this is where quartiles come into play.

Quartiles divide the dataset into the following parts:

1. Q1(25th percentile): 25% of transactions are below this.

2. Q2 (50th percentile): Median

3. Q3 (75th percentile): 75% of transactions are below this.

This is formally expressed as the Interquartile Range (IQR):

$$IQR = Q_3 - Q_1$$

The IQR: Detecting Outliers

The IQR measures the spread of the middle 50%.

If the IQR is small, then the data is concentrated. If it's large, the data is spread out. The IQR also helps us identify outliers mathematically.

Outlier Rule:

1. Lower Bound = Q1 — 1.5 * IQR

2. Upper Bound = Q3 + 1.5 * IQR

A Simple Example to Understand IQR

Consider the following transaction values:

$$\left[ 5, 8, 10, 12, 15, 18, 20 \right]$$

Step 1: Find the Median (Q2):

The middle value is:

$$Q_2 = 12$$

Step 2: Find Q1 (Lower Quartile):

The lower half is [5, 8, 10]. The median of the lower half is:

$$Q_1 = 8$$

Step 3: Find Q3 (Upper Quartile):

The upper half is [15, 18, 20]. The median of the upper half is:

$$Q_3 = 18$$

Step 4: Calculate IQR:

$$IQR = Q_3 - Q_1 = 18 - 8 = 10$$

Step 5: Find Outlier Bounds:

$$\begin{aligned} \text{Lower Bound} &= Q_1 - 1.5 \times IQR = 8 - 15 = -7 \ \text{Upper Bound} &= Q_3 + 1.5 \times IQR = 18 + 15 = 33 \end{aligned}$$

Any value below -7 or above 33 is an outlier (but in this demo problem, no outliers exist).

Applying IQR to Our Dataset

In our retail dataset, instead of neat values, we have bulk values and even negative returns.

# 1. Calculate IQR and Bounds
Q1 = df['TotalPrice'].quantile(0.25)
Q3 = df['TotalPrice'].quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

When we calculate IQR for our dataset, we get:

Lower Bound: -18.75
Upper Bound: 42.45
Number of Outliers: 33180

The graph demonstrates outliers, which are any values falling outside the range of -18.75 to 42.45. (Image by Author)

As the graph shows, the values outside the range -18.75 to 42.45 are considered outliers. These values will be removed.

Revisiting the Mean After Removing Outliers

Using the IQR method, we've removed extreme transactions that fell outside the typical spending range.

# Clean and Feature Engineering
df = df.dropna(subset=['CustomerID'])
df['TotalPrice'] = df['Quantity'] * df['UnitPrice']

# Original Mean
mean_value = df['TotalPrice'].mean()
print(f"Original Mean: {mean_value:.2f}")

# IQR Calculation
Q1 = df['TotalPrice'].quantile(0.25)
Q3 = df['TotalPrice'].quantile(0.75)
IQR = Q3 - Q1

# Define bounds
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

print(f"Lower Bound: {lower_bound:.2f}")
print(f"Upper Bound: {upper_bound:.2f}")

# Remove Outliers
df_no_outliers = df[(df['TotalPrice'] >= lower_bound) & (df['TotalPrice'] <= upper_bound)]

# New Mean after removing outliers
new_mean = df_no_outliers['TotalPrice'].mean()
print(f"Mean after removing outliers: {new_mean:.2f}")

After recomputing, we get:

Original Mean: 20.40
Lower Bound: -18.75
Upper Bound: 42.45
Mean after removing outliers: 11.63

Removing outliers significantly shifts the mean toward the region where most transactions occur. We now have a much better mean of 11.63 as opposed to the right-stretched mean of 20.40 we got with outliers.

Final Comparison and Insights

Looking at the results from all the graphs, we get a complete understanding of the dataset. The original mean was 20.40, which appeared to be significantly higher than the most transactions that actually occurred. In that case, the mean was pulled upward by some of the high-valued transactions and was distorted by the outliers.

The median, on the other hand, was 11.10, which lies within the range where most transactions are concentrated. This shows that the median is a much better representation of what a typical customer spends, as it's not affected by extreme values.

After removing the outliers using the IQR, the mean dropped to 11.63, bringing it very close to the median. This confirms that the earlier mean was not inherently wrong, but was simply influenced by extreme values in the data. Once those values were handled, the mean became a much more reliable measure of central tendency.

Conclusion

The results show that the mean can be misleading when data contains outliers. In our dataset, the original mean of 20.40 overstated customer spending, while the median (11.10) gave a more realistic picture. After removing outliers, the mean shifted to 11.63, aligning closely with the median.

This highlights a key lesson: The mean isn't wrong, but it must be used with an understanding of the data.

Choosing the right measure of average depends on the dataset, and in messy real-world scenarios, the median or a cleaned mean often tells the true story.

Connect with me

1. Medium

2. LinkedIN

If you want to dive deeper, you can visit: Mean vs Median vs Mode: Understanding Central Tendency in Data Analysis.

Your product shipped a new agent mode last quarter. Users have to tap the "Try agent mode" toggle to enable it. The dashboard numbers look stunning: agent-mode users complete 21 percentage points more tasks than non-users. The CPO calls it the best feature launch of the year.

But you know something's off. Heavy-engagement users opt into new features constantly, while light users ignore toggles entirely. That 21-point gap measures the agent's effect combined with the pre-existing gap between power users and the rest of your base.

This is the Opt-In Trap. It shows up in every generative AI product that ships features behind a user-controlled toggle: "Try our AI assistant," "Enable smart replies," "Turn on code suggestions." Users who click to opt in differ systematically from those who scroll past. Any naïve comparison between the two groups collapses the feature's causal effect into whatever made those users opt in in the first place.

Running an AI feature behind a toggle is a product experiment. The hypothesis: the feature improves outcomes for users who adopt it.

Unlike an A/B test, where the coin flip creates two otherwise-identical populations, the toggle creates two populations that differ before they even make a choice. That pre-existing difference is the measurement problem, and a t-test on dashboard numbers can't fix it.

Propensity score methods are statistical tools that data scientists use to separate adoption bias from the feature's actual effect. They reweight (or rematch) your comparison so that opted-in and non-opted-in groups look comparable on observable characteristics, approximating what a randomized experiment would have given you.

This tutorial walks through the full pipeline (propensity estimation, inverse-probability weighting, nearest-neighbor matching, balance diagnostics, and bootstrap confidence intervals) on a 50,000-user synthetic SaaS dataset where the ground-truth causal effect is known. You'll estimate it, quantify uncertainty, and see where the approach silently breaks.

Companion code: every code block runs end-to-end in the companion notebook at github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm/tree/main/02_propensity_opt_in. The notebook (psm_demo.ipynb) has all outputs pre-executed, so you can read along on GitHub before running anything locally.

Table of Contents

• Why Opt-in Features Break Naïve Comparisons

• What Propensity Scores Actually Do

• Prerequisites

• Setting Up the Working Example

• Step 1: Estimate the Propensity Score

• Step 2: Inverse-Probability Weighting

• Step 3: Nearest-Neighbor Matching

• Step 4: Check Covariate Balance

• Step 5: Bootstrap Confidence Intervals

• When Propensity Score Methods Fail

• What to Do Next

Why Opt-in Features Break Naïve Comparisons

The math of an A/B test is elegant because of one assumption: treatment is assigned independent of everything else. Flip a coin: half your users get agent mode, and the coin flip breaks every possible confound by construction. The opt-in world has no coin.

Three mechanisms make opt-in comparisons misleading.

1. Selection on engagement

Power users click everything. If your heavy-engagement cohort opts into agent mode at 65 percent and your light-engagement cohort opts in at 12 percent, you've stacked the opt-in group with users who were going to complete more tasks anyway.

That compositional imbalance accounts for most of the observed lift on its own, before the agent does any work.

2. Selection on intent

Users who opt into a new feature often have a specific use case in mind. A developer who clicks "Try code suggestions" already has code to write. That user would have shown higher task completion even with the control UI.

3. Selection on risk tolerance

Early adopters tolerate rough edges. A user who clicks "Try beta" and sees slow latency sticks around, but a risk-averse user bounces.

Your opt-in group is enriched for people willing to put up with bad experiences, which affects every downstream metric you might measure.

All three produce the same symptom: a raw comparison of opted-in users against everyone else that can overstate the feature's causal effect by 2x or more, depending on how concentrated opt-in is among your heaviest users.

On the synthetic dataset in this tutorial, the naïve comparison inflates a true +8pp effect to +21pp, a 2.6x overshoot. Propensity score methods exist to correct this.

What Propensity Scores Actually Do

Figure 1: Schematic propensity score distributions for two hypothetical groups. The opted-in group (red) skews toward higher propensities, while the non-opted-in group (blue) skews lower.

In the above figure, the bracketed strip below the x-axis splits the score range into three zones: a control-heavy region at low propensities where few treated users exist, a region of common support in the middle where both groups are well represented, and a treatment-heavy region at high propensities where few controls exist. Propensity score methods operate within the common-support region by reweighting or rematching so that the two groups appear balanced on observables. The extremes are either trimmed out or handled with caution.

The propensity score is the probability that a user opts in given their observable characteristics. Estimate this probability well, and you can use it to reweight your sample so that opted-in and non-opted-in users look similar on observables, just as they would have if opt-in had been randomized.

Two practical strategies use the propensity score:

• Inverse-probability weighting (IPW) assigns each user a weight equal to the inverse of their probability of receiving the treatment they actually received. Opted-in users get weighted by 1/P(opt-in). Non-opted-in users get weighted by 1/P(no opt-in). After weighting, the two groups are balanced on observables, and the weighted difference in outcomes approximates the average treatment effect.

• Matching pairs each opted-in user with one or more non-opted-in users who have similar propensity scores. The average outcome difference between matched pairs estimates the average treatment effect on the treated (ATT): what opt-in users actually gained by opting in.

Both methods rest on three identification assumptions working together.

1. First, unconfoundedness: every observable variable that drives opt-in and affects the outcome is in your propensity model.

2. Second, overlap (also called positivity): every user has some nonzero probability of opting in and some nonzero probability of staying out.

3. Third, no interference: one user's opt-in decision does not affect another user's outcome (the stable-unit-treatment-value assumption, or SUTVA.

Violate any one of these and the estimate is biased even when the other two hold. The failure modes at the end of this tutorial walk through each one.

Prerequisites

You'll need Python 3.11 or newer, comfort with pandas and scikit-learn, and rough familiarity with logistic regression.

Install the packages for this tutorial:

pip install numpy pandas scikit-learn matplotlib

Here's what's happening: four packages cover the full pipeline. Pandas loads the data, NumPy handles weights and array arithmetic, scikit-learn fits the propensity model and runs nearest-neighbor matching, and matplotlib renders the overlap diagnostic.

Clone the companion repo to get the synthetic dataset:

git clone https://github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm.git
cd product-experimentation-causal-inference-genai-llm
python data/generate_data.py --seed 42 --n-users 50000 --out data/synthetic_llm_logs.csv

Here's what's happening: the clone pulls the companion repo, and generate_data.py produces the shared synthetic dataset used across the series. Seed 42 keeps the dataset reproducible, and 50,000 users give clean signal for every estimator in this tutorial. The output CSV lands at data/synthetic_llm_logs.csv.

Setting Up the Working Example

The synthetic dataset simulates a SaaS product where users can opt into an agent mode that uses a more expensive model. With fifty thousand users, opt-in rates differ sharply by engagement tier: heavy users opt in at 65 percent, medium users at 35 percent, and light users at 12 percent.

The ground-truth causal effect baked into the data generator is +8 percentage points on task completion for users who opted in. The naive comparison inflates this to around +21 percentage points because selection bias stacks the opted-in group with your most engaged users.

Knowing the ground truth is what lets you verify that your propensity score method recovers it.

Load the data and see the selection problem:

import pandas as pd

df = pd.read_csv("data/synthetic_llm_logs.csv")

print(df.groupby("engagement_tier").opt_in_agent_mode.mean().round(3))

naive_effect = (
    df[df.opt_in_agent_mode == 1].task_completed.mean()
    - df[df.opt_in_agent_mode == 0].task_completed.mean()
)
print(f"\nNaive opt-in effect: {naive_effect:+.4f}")

Expected output:

engagement_tier
heavy     0.647
light     0.120
medium    0.353
Name: opt_in_agent_mode, dtype: float64

Naive opt-in effect: +0.2106

Here's what's happening: you load 50,000 rows, group by engagement tier, and print the opt-in rate inside each group. Heavy users opt in far more than light users, which is the selection-on-engagement pattern baked into the data. The naïve effect lands at +0.2106 (21 percentage points), nearly three times the ground truth of +0.08. That gap is exactly what propensity score methods have to remove.

Step 1: Estimate the Propensity Score

The propensity score is the output of a model that predicts opt-in from observable characteristics. Logistic regression is the right starting point because it's interpretable and fast, but watch the balance diagnostics in Step 4: if any weighted SMD stays above 0.1, the logistic model is missing an interaction, and gradient boosting is the next move.

For this dataset, the relevant observables are engagement tier and query confidence. In a real product, you'd include every variable you think drives opt-in: device type, tenure, plan tier, and historical usage patterns.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

X = pd.get_dummies(
    df[["engagement_tier", "query_confidence"]],
    drop_first=True
).astype(float)
y_treat = df.opt_in_agent_mode

ps_model = LogisticRegression(max_iter=1000).fit(X, y_treat)
df["propensity"] = ps_model.predict_proba(X)[:, 1]

# Basic sanity checks
print(df.groupby("engagement_tier").propensity.mean().round(3))
print(
    f"\nPropensity range (treated):  "
    f"{df[df.opt_in_agent_mode == 1].propensity.min():.3f} - "
    f"{df[df.opt_in_agent_mode == 1].propensity.max():.3f}"
)
print(
    f"Propensity range (control):  "
    f"{df[df.opt_in_agent_mode == 0].propensity.min():.3f} - "
    f"{df[df.opt_in_agent_mode == 0].propensity.max():.3f}"
)
print(f"Propensity model AUC: {roc_auc_score(y_treat, df.propensity):.3f}")

Expected output:

engagement_tier
heavy     0.646
light     0.120
medium    0.353
Name: propensity, dtype: float64

Propensity range (treated):  0.114 - 0.675
Propensity range (control):  0.114 - 0.673
Propensity model AUC: 0.744

Here's what's happening: you encode the engagement tier as dummy variables, keep query confidence continuous, and fit a logistic regression model. The predicted probability from the model is each user's propensity score.

Scikit-learn LogisticRegression applies L2 regularization by default (C=1.0), which shrinks propensities slightly toward 0.5. For production use, you can set penalty=None if you want an unregularized fit.

Mean propensity inside each engagement tier recovers the true opt-in rate for that tier almost exactly, so the model is calibrated. The AUC of 0.744 confirms the model discriminates between opt-ins and non-opt-ins well above chance (0.5).

And the propensity ranges overlap between treated and control groups (both span roughly 0.11 to 0.67), which is the visual overlap condition.

Figure 2: Two views of the same positivity check on the real 50,000-user synthetic dataset.

In the figure above, the top panel plots smooth kernel density curves of the fitted propensity scores for each group. The three peaks align with the three engagement tiers (light at p ≈ 0.12, medium at p ≈ 0.35, heavy at p ≈ 0.65), as expected, because the opt-in rate is tier-driven. The bottom panel translates that same distribution into raw counts per tier: every tier contains thousands of both opted-in and non-opted-in users, which is exactly what positivity requires.

Where Figure 1 schematically illustrated the idea, this figure shows that it holds for the data, so the weighting and matching that follow will have real counterfactuals to work with.

Step 2: Inverse-Probability Weighting

IPW assigns each user a weight inversely proportional to their propensity. An opted-in user with a 0.12 propensity is rare (a light user who still opted in despite low engagement) and carries information about 1 / 0.12 ≈ 8 similar users in the population. A control user with a 0.12 propensity is the expected case for light users who stayed out, so they're common and get a weight of 1 / (1 - 0.12) ≈ 1.14.

import numpy as np

# ATE weights: 1/P(treat) for treated, 1/P(no treat) for control
df["ipw"] = np.where(
    df.opt_in_agent_mode == 1,
    1 / df.propensity,
    1 / (1 - df.propensity)
)

t = df[df.opt_in_agent_mode == 1]
c = df[df.opt_in_agent_mode == 0]
ate_ipw = (
    (t.task_completed * t.ipw).sum() / t.ipw.sum()
    - (c.task_completed * c.ipw).sum() / c.ipw.sum()
)
print(f"IPW average treatment effect (ATE): {ate_ipw:+.4f}")

# ATT: what opt-in users actually gained
df["ipw_att"] = np.where(
    df.opt_in_agent_mode == 1,
    1,
    df.propensity / (1 - df.propensity)
)
t = df[df.opt_in_agent_mode == 1]   # re-slice now that ipw_att is in df
c = df[df.opt_in_agent_mode == 0]
treated_mean = t.task_completed.mean()
control_w_mean = (c.task_completed * c.ipw_att).sum() / c.ipw_att.sum()
att_ipw = treated_mean - control_w_mean
print(f"IPW average treatment effect on treated (ATT): {att_ipw:+.4f}")

Expected output:

IPW average treatment effect (ATE): +0.0851
IPW average treatment effect on treated (ATT): +0.0770

Here's what's happening: first, you compute ATE weights for every user and take the weighted difference in task completion between opted-in and non-opted-in groups. Then you compute ATT weights, which reweight only the control group to match the treated group's covariate distribution, and compute the average treatment effect on the treated.

ATE answers the population question: what's the effect on a random user who might or might not have opted in anyway? ATT answers the user question: What did opt-in users actually gain? On this dataset, ATE lands at +0.0851 and ATT at +0.0770, both close to the ground-truth +0.08 and a massive improvement over the naive +0.2106.

The distinction matters in practice. Deciding whether to roll the feature out to users who haven't opted in calls for ATE. Reporting on the value opt-in users captured calls for ATT.

Step 3: Nearest-Neighbor Matching

Matching takes a different approach: pair each opted-in user with the non-opted-in user whose propensity score is closest, then take the average outcome difference across matched pairs. The result estimates ATT.

from sklearn.neighbors import NearestNeighbors

treated_ps = df[df.opt_in_agent_mode == 1][["propensity"]].values
control_ps = df[df.opt_in_agent_mode == 0][["propensity"]].values

nn = NearestNeighbors(n_neighbors=1).fit(control_ps)
_, idx = nn.kneighbors(treated_ps)

treated_outcomes = df[df.opt_in_agent_mode == 1].task_completed.values
matched_control_outcomes = (
    df[df.opt_in_agent_mode == 0].task_completed.values[idx.flatten()]
)

att_match = (treated_outcomes - matched_control_outcomes).mean()
print(f"1-NN matching ATT: {att_match:+.4f}")

Expected output:

1-NN matching ATT: +0.0752

Here's what's happening: you extract propensity scores for each group, fit a nearest-neighbor index on the control group, and find the single closest control user for every treated user.

The NearestNeighbors index allows the same control user to be selected as the match for multiple treated users, so this is a matching-with-replacement case.

You pull the outcomes for each treated user and their matched control, take the difference per pair, and average across pairs. The result estimates what opt-in users gained compared to very similar users who did not opt in.

The +0.0752 result lands close to the ground truth of +0.08 but slightly below IPW ATT, typical of 1-NN matching because a single nearest neighbor is a high-variance estimator.

Two variants are worth knowing. Matching with replacement (what you just ran) allows a single control user to serve as a match for multiple treated users, reducing bias when good matches are scarce but inflating variance.

Matching without replacement assigns each control user to at most one treated user, which keeps variance lower but forces poor-quality pairings when the treated group dwarfs the available controls.

For most production analyses, k-nearest-neighbor matching with k = 3-5 and replacement is a sensible default.

Step 4: Check Covariate Balance

Propensity score methods work only if they actually balance the covariates between groups. You need to verify that they did, because if the balance fails, your estimate is wrong.

The standard diagnostic is the standardized mean difference (SMD) for each covariate. SMD compares the treated group mean to the control group mean, divided by the pooled standard deviation.

Before weighting, SMDs tell you how imbalanced the raw groups are. After weighting, they should be small (|SMD| < 0.1 is the conventional cutoff).

def smd(treated_vals, control_vals, treated_w=None, control_w=None):
    """Standardized mean difference, optionally with weights."""
    if treated_w is None:
        treated_w = np.ones(len(treated_vals))
    if control_w is None:
        control_w = np.ones(len(control_vals))
    t_mean = np.average(treated_vals, weights=treated_w)
    c_mean = np.average(control_vals, weights=control_w)
    pooled_std = np.sqrt((treated_vals.var() + control_vals.var()) / 2)
    return (t_mean - c_mean) / pooled_std

engagement_heavy = (df.engagement_tier == "heavy").astype(float).values
qc = df.query_confidence.values
tr = (df.opt_in_agent_mode == 1).values

covariates = {
    "engagement_tier_heavy": engagement_heavy,
    "query_confidence": qc,
}

print(f"{'Covariate':<30} {'Raw SMD':>10} {'Weighted SMD':>15}")
for name, vals in covariates.items():
    smd_raw = smd(vals[tr], vals[~tr])
    smd_weighted = smd(
        vals[tr], vals[~tr],
        treated_w=df[tr].ipw.values,
        control_w=df[~tr].ipw.values,
    )
    print(f"{name:<30} {smd_raw:>+10.3f} {smd_weighted:>+15.3f}")

Expected output:

Covariate                         Raw SMD    Weighted SMD
engagement_tier_heavy              +0.742          +0.002
query_confidence                   -0.032          -0.003

Here's what's happening: the helper computes the standardized mean difference for any covariate, with optional IPW weights.

You then print raw and weighted SMDs for each covariate. The raw SMD on engagement_tier_heavy is +0.742 (heavy users opt in far more than everyone else), and the weighted SMD drops to +0.002, a clean pass. Query confidence was already close to balanced on the raw data, and weighting keeps it that way. If any weighted SMD came back above 0.1 in absolute value, your propensity model would be missing something; the fix is usually richer features or interaction terms in the logistic regression.

Visually, Figure 2 above confirmed what the SMDs now confirm numerically: the overlap condition holds, and balance is achievable.

Step 5: Bootstrap Confidence Intervals

Point estimates are only half the story. Any estimate you report to a product team needs an interval that tells them whether +0.08 is distinguishable from +0.03 or from +0.12. Analytic standard errors for IPW and matching are tricky because of the estimated propensity score, so the simplest and most honest move is the non-parametric bootstrap.

def estimate_all(sample):
    """Return (ATE_IPW, ATT_IPW, ATT_match) on a bootstrap sample."""
    s = sample.copy()
    X_s = pd.get_dummies(
        s[["engagement_tier", "query_confidence"]], drop_first=True
    ).astype(float)
    ps = LogisticRegression(max_iter=1000).fit(X_s, s.opt_in_agent_mode)
    s["p"] = ps.predict_proba(X_s)[:, 1]

    s["w_ate"] = np.where(
        s.opt_in_agent_mode == 1, 1 / s.p, 1 / (1 - s.p)
    )
    s["w_att"] = np.where(
        s.opt_in_agent_mode == 1, 1, s.p / (1 - s.p)
    )
    t, c = s[s.opt_in_agent_mode == 1], s[s.opt_in_agent_mode == 0]

    ate = (
        (t.task_completed * t.w_ate).sum() / t.w_ate.sum()
        - (c.task_completed * c.w_ate).sum() / c.w_ate.sum()
    )
    att = t.task_completed.mean() - (
        (c.task_completed * c.w_att).sum() / c.w_att.sum()
    )
    nn_b = NearestNeighbors(n_neighbors=1).fit(c[["p"]].values)
    _, idx_b = nn_b.kneighbors(t[["p"]].values)
    match = (
        t.task_completed.values
        - c.task_completed.values[idx_b.flatten()]
    ).mean()
    return ate, att, match

rng = np.random.default_rng(7)
n_reps = 500
results = np.zeros((n_reps, 3))
for i in range(n_reps):
    boot = df.iloc[rng.integers(0, len(df), size=len(df))]
    results[i] = estimate_all(boot)

for name, col in zip(["IPW ATE", "IPW ATT", "1-NN ATT"], range(3)):
    lo, hi = np.percentile(results[:, col], [2.5, 97.5])
    print(f"{name:<10} 95% CI: [{lo:+.4f}, {hi:+.4f}]")

Expected output:

IPW ATE    95% CI: [+0.0745, +0.0954]
IPW ATT    95% CI: [+0.0687, +0.0865]
1-NN ATT   95% CI: [+0.0659, +0.0940]

Here's what's happening: you resample the dataset with replacement 500 times, refit the propensity model, and recompute each estimator on each resample, and take the 2.5th and 97.5th percentiles of the bootstrap distribution as the 95% confidence interval. All three intervals cover the ground-truth +0.08 and exclude the naive +0.21 by a wide margin.

The IPW ATT interval is the tightest because ATT reweights only the control group. The 1-NN matching interval is the widest because single-neighbor matching discards control users outside the matched set.

Running this once takes about 90 seconds on a laptop. For a stakeholder report, anchor the headline to the point estimate and cite the interval so the team sees the uncertainty alongside the number.

When Propensity Score Methods Fail

Propensity scores make opt-in comparisons rigorous when their assumptions hold. They produce biased estimates that look clean when those assumptions fail.

Four common failure modes map to the three identification assumptions from earlier.

1. Unmeasured Confounders (Violate Unconfoundedness)

If something drives both opt-in and your outcome but isn't in your propensity model, IPW and matching produce biased estimates. This is the most common failure in practice.

An example: users who opt into agent mode are also the users who follow your engineering blog and read release notes. If blog-reading behavior raises task completion independently of the feature, missing that signal attributes the effect to agent mode, inflating your estimate.

The only real defense is domain knowledge about what drives opt-in, richer feature engineering in your propensity model, and formal sensitivity tools (Rosenbaum bounds, E-values) that quantify how strong an unmeasured confounder would have to be to overturn the result.

2. Positivity (Overlap) Failures (Violates Overlap)

If some users have near-zero probability of opting in (or near-one), you've got no comparable counterfactual for them. I

PW creates extreme weights (1 / 0.001 = 1,000) that let a single outlier dominate the estimate. So matching is forced into poor-quality pairings.

Check propensity histograms and trim propensities outside [0.05, 0.95] before weighting if extreme values exist.

3. Misspecified Propensity Models (Degrade Unconfoundedness in Practice)

A linear logistic regression can't capture nonlinear relationships. If opt-in depends on the interaction between engagement tier and query confidence (power users with complex queries opt in, while light users pass), a main-effects model misses that and produces poor balance.

Use flexible models (for example, gradient boosting on the propensity score or regression adjustment on top of weighting) and always check the balance after weighting. Poor balance after weighting is the primary signal of misspecification.

4. Spillovers Between Users (Violates SUTVA)

Propensity score methods assume your users are independent. If one user opting into agent mode affects another user's task completion (for example, teammates adopting the feature together in shared workspaces), your estimated effect includes the spillover.

This violates the stable-unit-treatment-value-assumption, and handling it cleanly requires a different toolkit: either cluster randomization for features adopted at the workspace level or network-aware experimental designs for user-level spillovers.

These failure modes stay invisible in your regression coefficients. They surface as estimates that look good on paper but don't hold up when the feature rolls out to a broader audience.

Run balance diagnostics, check overlap plots, and document what you might have missed: those are your only real defenses.

What to Do Next

Propensity score methods are the right tool when your feature ships behind an opt-in toggle and you've got rich covariates to model selection with.

If opt-in follows a crisp rule (a threshold on query complexity, a paid-tier gate), regression discontinuity fits better. If you suspect unobserved confounders and have an external randomization source (randomized rollout noise, rate-limit-triggered routing), instrumental variables will do better.

To guard your estimate against propensity misspecification, doubly robust estimators combine propensity weighting with regression adjustment and stay consistent if at least one of the two component models is correctly specified.

The companion notebook for this tutorial lives here. Clone the repo, generate the synthetic dataset, and run psm_demo.ipynb (or psm_demo.py) to reproduce every code block, every number, and every figure from this tutorial.

When an AI feature ships behind a toggle, the naïve opt-in comparison is usually the wrong number. Propensity score methods give you "users comparable to those who clicked this" as your counterfactual, and the bootstrap gives you an interval you can defend when a stakeholder asks how sure you are.