<![CDATA[ Data Science - freeCodeCamp.org ]]> https://www.freecodecamp.org/news/ https://cdn.freecodecamp.org/universal/favicons/favicon.png <![CDATA[ Data Science - freeCodeCamp.org ]]> https://www.freecodecamp.org/news/ Eleventy Sat, 03 Oct 2026 21:16:43 +0000 60 <![CDATA[ What Every Dev Should Know About Tracking Product Data ]]> https://www.freecodecamp.org/news/what-devs-should-know-about-tracking-product-data/ 6a95f5263ba1cbee5d13a5bd Mon, 31 Aug 2026 21:41:58 +0000 Product data is the record of what actually happens inside your app or website. It shows what your users do, how your system behaves, and how your business performs.

In this article, you'll learn what product data is, which parts of it are worth tracking, which parts to leave alone, and why the developer writing the code carries more responsibility for it than anyone tells them.

Table of Contents

What Is Product Data?

Product data refers to what happens inside your app or website. It answers plain questions: Which features do users love? Where do they get stuck? How fast does the app load? Which actions lead to a purchase or a signup?

Product data is verified information. Surveys and feedback tell you what users say, while product data shows you what they actually did.

That difference matters more than it sounds. People misremember their own behavior. They tell you the checkout was fine and then abandon it. Your data doesn't have that problem.

To make product data easier to track and manage, you can split it into three categories.

1. User Behavior Data

This is what users do inside your product. It covers the actions they take and how they move through your app. It shows you usage patterns, feature adoption, friction points, and user journeys.

Questions this data answers:

  • How often do users click the "Add to Cart" button?

  • How many users finish the onboarding flow?

  • At what point in checkout do users give up?

  • How long does it take a new visitor to join the waitlist?

  • Which marketing campaign brought this user in?

2. System and Backend Data

This tracks how your system behaves while users interact with it. It's the under-the-hood view. Here you care about the health of your frontend, your servers, and your code.

Questions this data answers:

  • How long does a page take to load?

  • Which API endpoints return server errors?

  • How much memory is the app using?

  • How long does a database query take to finish?

  • Does your business logic hold up in edge cases?

3. Business Metrics

These are the high-level numbers that tell you whether the product is working as a business. They connect user behavior and system performance to money and growth.

Common ones include:

  • Daily and monthly active users

  • Monthly recurring revenue and revenue per user

  • Conversion rate and churn rate

  • User retention after a set number of days

  • Feature adoption percentage

How the Three Fit Together

All three categories describe the same product from different angles. The useful part is where they meet.

Let's say a business has a payment endpoint that starts returning errors. That's system data. The users who hit those errors give up and leave their carts behind. That's behavior data. The week's revenue comes in lower than the week before. That's a business metric.

Look at any one of them alone and you learn almost nothing. A failing endpoint might be harmless. Abandoned carts might be shoppers who were only browsing. A revenue dip might be seasonal.

But if read all three together, the story tells itself. The endpoint broke, so the carts were abandoned, so the revenue fell.

Why Should You Track Product Data?

You should track product data because it replaces your guesses with facts. Collecting app data shapes your roadmap, settles arguments before they start, and tells you what your users did instead of what you assumed they did.

Building a product without tracking is like driving at night with no headlights. You're moving, but you can't see the road ahead.

Think about how decisions get made without it. The conversation could stall at "I don't think users like this feature." But when you have data, you can change that opinion to a question and have an answer for it. Something like "Why did most of the users who opened this feature never come back to it?" You get to work with data-backed decisions, and that's the main point.

Data also shows you the real user journey. Not the one you designed, the one people actually walk. You see where they stop, what they skip, and which corners of your app they never find.

Tracking data also helps you see acquisition and retention issues. You can't fix a retention problem you can't see. To make matters worse, the retention numbers across the industry are sobering.

According to Adjust's 2024 benchmarks, the median mobile app keeps 26% of its users after one day, 13% after a week, and 7% after 30 days. On the web it's starker. Contentsquare's 2026 Digital Experience Benchmark, built on 99 billion sessions across more than 6,500 sites, found that only 13% of visitors return within 30 days.

Tracking also catches the failures that quietly cost you money. The Baymard Institute maintains a running average of cart abandonment across 50 separate studies, and it sits at 70.22%. When they asked people why they left, 17% said the site had errors or crashed, and another 17% said checkout took too long. Both of those are engineering failures. Both of them show up in your revenue rather than your error logs, and you find them in your data or you don't find them at all. So if you track them, you can avoid such issues.

It's also a high cost to not fully be aware of your market. When people don't know what's going on, it could cost them shutting down. For example, CB Insights tracked 431 startups that shut down since 2023 and found that 43% of them cited poor product-market fit. And this is a problem data could've solved before taking it up to business decisions.

Overall, most teams already believe in the importance of data, and agree that we need data-backed decisions. In March 2025, Salesforce found that 76% of business leaders feel pressure to back their arguments with data. In addition, the majority believe that their career success depends on how "data literate" and data-driven they are.

With that said, let's look at another dimension of data's importance by checking out other team members that use it.

Who Else Uses the Data You Track?

Almost everyone on the team. The data travels further than most developers expect.

You read it first. You use it to see whether last night's release broke something, whether a feature earns the maintenance it costs you, and where the slowness actually lives. Then it keeps traveling.

Your product manager reads it to decide what gets built next. Your designer reads it to find where people struggle. Your analyst reads it to explain why revenue moved. Support reads it to make sense of a spike in tickets. Marketing reads it to see which campaigns brought in users who stayed. Your founder reads it before every hard conversation about the roadmap.

Take one event and follow where it goes. Say it's failed_checkout, with a reason parameter attached.

  • You see an error rate and trace the endpoint that's breaking.

  • Your product manager sees how much conversion is lost and decides whether the fix jumps the queue.

  • Your analyst sees an early churn signal and puts it into a model.

  • Your support lead finally understands why tickets spiked on Tuesday.

  • Your marketer sees paid traffic landing on a checkout that can't complete.

  • Your founder sees revenue that never arrived, and walks into the next board meeting with a better question.

One event, six people, and six questions, all answered by three lines of code you wrote in an afternoon.

Notice what every one of those six people is really doing. They're making a decision.

That's the whole point of tracking. Data doesn't have value sitting in a table. It has value the moment somebody changes their mind because of it, ships something different, or stops doing something that wasn't working.

That need is what created data professions in the first place. Companies got large enough that nobody could hold the whole product in their head, and decisions started needing evidence rather than instinct.

Business intelligence analysts came first. Then data scientists, data engineers, product analysts, and analytics engineers, each a response to more data arriving and more decisions waiting on it. Every one of those roles exists because somebody needed to decide something and couldn't see far enough to do it.

In a small team, or on a product you're building from scratch, all of those jobs are yours. And in the beginning, the jobs are quite small. You don't yet need extensive analysis. Plain tracking, as I'm advocating for here, should do for a start.

Why Should You Start Tracking Early?

You can't go back and collect the past. You can't reconstruct how users behaved in your first month. You can't find out which feature drove your early signups. And you can't explain the drop-off spike from three months ago. Your data starts the day you start collecting, and everything before that is gone for good.

Starting early also prepares you for scale. At some point you'll hire an analyst, or a data scientist, or you'll sit down to do the work yourself. If you've been capturing since day one, you hand them a goldmine. If you haven't, they spend their first quarter waiting for enough data to exist before they can say anything useful.

If you've started recording already, you've done well. Please keep it up. If not, start now. If you're setting up your codebase this week, wire in tracking this week. If your product has been live for two years, the best time to set up tracking was before launch, and the second best time is today.

There's an engineering reason, too. When you instrument from the start, your event schema grows alongside your product. You name things consistently because you're naming them as you build them. Starting tracking two years later is harder when your code has grown so big. So start early.

Starting early doesn't mean tracking everything. It means tracking a few correct things from the beginning and growing from there. You don't need a perfect schema on day one. You need a few good events and the habit of adding more.

Why Tracking Data Never Ends

Setting up tracking is one thing. Keeping it up-to-date and relevant as the product changes is another, and many teams skip this.

Your product won't stay still. Features get renamed, flows get redesigned, and screens get deleted. Every one of those changes can break an event, orphan a parameter, or leave you collecting something that no longer means what it used to mean.

So treat tracking as part of the work, not a one-time setup. It's a living process as much as the product is alive.

When you ship a new feature, add its events in the same pull request. When you change a flow, check which events that flow was firing. When you deprecate a screen, deprecate its events too and tell whomever was querying them. Verify that your events actually fire after you deploy, the same way you'd check that the feature itself works.

Verifying is important. Broken tracking is quiet. Monte Carlo asked data professionals who spots data problems first, and 74% said business stakeholders do, all or most of the time. Almost nobody writes tests for analytics, so a stopped event tends to surface weeks later in somebody else's dashboard.

It's worth auditing the whole schema now and then. Once or twice a year, go through your events and ask three things of each one. Is it still firing? Does anyone read it? Does the name still describe what it does?

What Should You Track in Your Product?

Every product is different, and there's no universal list. The rule of thumb is to track what you'd act on and skip what you wouldn't.

That said, the three categories from earlier map onto concrete things you instrument. Here's how they break down.

User behavior becomes events and user properties. System and backend becomes crashes, performance data, and backend errors. Business metrics are different, and the difference is worth understanding.

You don't instrument business metrics directly. There's no track_mrr call. You compute them from the other two categories. This explains why the quality of your events decides the quality of your business reporting.

Events

Events are the most important thing you track. An event is any meaningful action in your product. Clicking a button, finishing a purchase, submitting a form, sharing content, or skipping onboarding are all examples.

Not every click is an event. Focus on actions that show intent, progress, or value.

Name events in verb_noun format using snake_case. This is the convention Google uses for its own recommended events, including add_to_cart, sign_up, begin_checkout, and join_group. Pick one convention and hold to it. A schema with checkout_failed next to failed_checkout is a schema nobody can query with confidence.

Attach parameters to give events context:

  • failed_checkout with reason: "payment_declined"

  • view_product with product_id: "123" and category: "shoes"

  • use_feature with feature_name: "export_pdf"

Three kinds of event are worth prioritizing. Conversion actions such as sign_up, complete_purchase, and start_trial are your most important numbers. Feature usage such as open_dashboard and share_report tells you which parts of your app earn their keep. Drop-off points such as abandon_checkout and exit_onboarding tell you where people quit, which is every bit as useful as knowing where they succeed.

User Properties

Events describe what a user did. User properties describe who the user is.

A user property is a state that persists between sessions: plan tier, signup month, preferred language, or whether onboarding was ever completed. You set it once and it stays until it changes.

Properties are what let you slice your events. "How many users completed checkout" is a number. "How many users on the free plan completed checkout in their first week" is an answer.

There are two traps to avoid. First, don't put high-cardinality values in user properties, because a property with thousands of distinct values is unusable for grouping. And don't put personal information in them. More on that further down.

Crashes

A crash happens when an app fails during use. A crash is some unforeseen heavy error that causes a poor user experience. Every crash report should tell you the device, the operating system version, the app version, and the stack trace at the moment it went down.

Performance Data

Performance data covers how fast your product actually is for real people, not on your machine. Slow software loses customers, which in turn causes a revenue drop.

Track app start time, screen render time, network request duration, and the time taken by any operation a user waits on. Record these as distributions rather than averages. That way, you have more information when taking action.

Backend Logs and Errors

The logs give insights into what the code was doing. The errors tell where it failed. Overall, these are info from your servers that your frontend can't tell you.

Track logs and error rates per endpoint, status codes, failed background jobs, timeouts, and failures from third-party services you depend on. Log them with enough context to trace a single failing request end to end.

It helps especially when there's no way to get info from the user. They also help to reconcile app-side activities with the system.

Also, backend logs will show when people try to abuse and bombard your server with requests. At that point, you can increase your rate-limits and put measures in to prevent such security problems.

What Should You Not Track?

Don't track what you can't act on. Before adding an event, answer one question. If this number moved, what would I do differently? If you have no answer, you've found a number to skip.

Don't track what you'd hate to leak. Every piece of data you collect is data you're now responsible for protecting. Collecting just what you need is a security best practice.

Don't track anything you can't name precisely. A clear name is a sign that the meaning is settled. user_action and button_click are placeholders rather than events. Work out what the moment means first, then name it.

Don't track the same thing twice. One record per unit event is enough. Two events that fire on the same action introduce a burden of knowing which is a "better source of truth". That stress of choosing which to use isn't worth it.

How Do You Handle User Data Safely?

By collecting just adequately for the job and being deliberate about its handling.

Keep PII Out of Your Analytics.

Personally identifiable information, usually shortened to PII, is any data that can identify a specific person on its own or when combined with something else you hold. Names, email addresses, phone numbers, physical addresses, payment details, and government ID numbers all count.

Analytics tools aren't built to hold PII, and most vendors forbid it in their terms. Use an anonymous or pseudonymous user ID instead, and join to your own database on the rare occasion you genuinely need identity.

Watch What Leaks in Through Parameters.

This is another place where PII usually gets in. A search event that records the query string could record somebody's own name. A URL parameter that carries an email address could directly enter the event log. Sanitize parameter values before they leave the device.

Under the EU's General Data Protection Regulation and similar regimes elsewhere, analytics that isn't strictly necessary generally needs consent.

Set a Retention Period and Let Data Expire.

Most analytics tools default to keeping data for a fixed window and let you shorten it. Shorter is safer. You rarely need event-level detail from three years ago, and aggregate summaries serve you better anyway.

Write Down What You Collect.

Documentation is useful. A single document listing every event, every parameter, and why it exists will save you during a privacy review, an audit, or the day a new engineer asks what evt_flow_2b was for.

Summary

Product data tells you what happened, not what users say happened or what your opinions are.

Collect data from various facets. User behavior tells you what they did, system data tells you how your product held up, and business metrics fall out of the other two. One category on its own might not be as accurate.

Start tracking today, whatever stage you're at. You can't recover last month's data. That window closes permanently, and it closes every single day.

Name your events for the people who'll query them. Use verb_noun, keep it consistent, write down what your parameters mean, and tell your team when you change something.

Keep the schema alive as the product changes. Add events in the same pull request as the feature. Audit it once or twice a year and cut anything nobody reads.

Here's to the many benefits you'll reap in the long run for taking note of data in your product overall.

Cheers!

]]> <![CDATA[ How to Tell If Your Search Console Impressions Came From a Human or a Machine ]]> https://www.freecodecamp.org/news/how-to-tell-if-search-console-impressions-are-human-or-machine/ 6a8f84afb0c3d8eedbfb8fec Thu, 27 Aug 2026 00:28:31 +0000 Your Search Console report says a page earned 3,068 impressions on the first page of Google over 90 days. But it earned zero clicks in that same time period.

The usual reading is that the page has a click-through-rate problem, so you rewrite the title, tighten the meta description, and wait. That reading is likely wrong, and acting on it wastes real work. Nobody saw those results, because no human ever ran those searches.

This tutorial shows you how to separate the two kinds of impressions your site earns. You'll run a short script against your own Search Console data, split the impressions by position band, read the query list for machine signatures, and compare the suspect page against a control page on the same site.

A position band is a bucket of average search positions rather than a single number. This script uses four: the top 3 results, the rest of page 1, page 2, and page 3 and beyond. Bucketing matters because a single average hides the spread, so a page can average position 8 by sitting at 2 for a handful of searches and 30 for everything else.

A control page is simply another page on the same site that you already know has real readers, and it gives you a baseline to hold the suspect page against.

At the end you'll know which of your pages have a human audience and which don't, and you'll stop optimizing for readers who don't exist.

Every number below comes from my own site.

Table of Contents

Prerequisites

You'll need the following before you start:

  • A verified Google Search Console property with at least 90 days of data. The free tier is enough.

  • Node.js 18 or newer. The script uses the built-in fetch, so no HTTP library is needed.

  • A Google Cloud service account with the Search Console API enabled, added as a user on your property. Download its JSON key.

  • Two npm packages: google-auth-library for the token, and tsx to run the TypeScript file directly.

  • A suspect page and a control page. The suspect page is one with high impressions and almost no clicks. The control page is your best-performing article, the one you know real people read.

  • About 20 minutes.

Add the dependency and point the standard credentials variable at your key:

npm i google-auth-library tsx
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your-service-account.json

If you've never enabled the API, turn on "Google Search Console API" in your Google Cloud project, then add the service account's email address as a full user in Search Console under Settings and Users and permissions.

Why Machine Impressions Exist

Google's AI Mode and AI Overviews don't answer a question by running that one question. They run a technique called query fan-out: the model expands your prompt into a set of narrower sub-queries, retrieves sources for each one, and merges the results into an answer.

Each of those sub-queries is a real search against the real index. When your URL is retrieved for one, Search Console logs an impression at the position where it was retrieved.

That impression is genuine. The position is genuine. But no human ever saw a results page, so no human could have clicked. The click isn't missing because your title is weak. The click is structurally impossible.

This matters because the standard tooling can't tell the difference. A click-through-rate gap script that selects rows where actual click-through-rate falls below expected click-through-rate for that position will rank these rows at the very top, since zero divided by anything positive is the largest possible gap. The signal isn't merely absent. It's inverted, and your worst candidates get promoted to your best ones.

Step 1: Pull the Page Totals

Save this as phantom-check.ts. It's the whole tool.

#!/usr/bin/env npx tsx
import { GoogleAuth } from 'google-auth-library';

const SITE = process.argv[2];
const PAGE = process.argv[3];
const DAYS = Number(process.argv[4] ?? 90);

if (!SITE || !PAGE) {
	console.error('Usage: npx tsx phantom-check.ts <site-url> <page-url> [days]');
	process.exit(1);
}

// Search Console data lags about two days, so end the window there.
const iso = (d: Date) => d.toISOString().slice(0, 10);
const endDate = iso(new Date(Date.now() - 2 * 864e5));
const startDate = iso(new Date(Date.now() - (DAYS + 2) * 864e5));

async function getToken(): Promise<string> {
	const auth = new GoogleAuth({
		scopes: ['https://www.googleapis.com/auth/webmasters.readonly']
	});
	const client = await auth.getClient();
	const token = await client.getAccessToken();
	if (!token.token) throw new Error('Could not mint an access token.');
	return token.token;
}

type Row = { keys: string[]; clicks: number; impressions: number; ctr: number; position: number };

async function query(token: string, body: Record<string, unknown>): Promise<Row[]> {
	const url =
		`https://www.googleapis.com/webmasters/v3/sites/${encodeURIComponent(SITE)}/searchAnalytics/query`;
	const res = await fetch(url, {
		method: 'POST',
		headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
		body: JSON.stringify({
			startDate,
			endDate,
			dimensionFilterGroups: [
				{ filters: [{ dimension: 'page', operator: 'equals', expression: PAGE }] }
			],
			...body
		})
	});
	if (!res.ok) throw new Error(`Search Console returned HTTP ${res.status}`);
	const json = (await res.json()) as { rows?: Row[] };
	return json.rows ?? [];
}

function band(position: number): string {
	if (position <= 3) return 'top 3';
	if (position <= 10) return 'rest of page 1';
	if (position <= 20) return 'page 2';
	return 'page 3+';
}

(async () => {
	const token = await getToken();

	const totals = await query(token, { dimensions: ['page'] });
	const t = totals[0];
	console.log(`\n${PAGE}`);
	console.log(`window: ${startDate} to ${endDate} (${DAYS} days)\n`);
	if (!t) {
		console.log('No impressions in this window.');
		return;
	}
	console.log(
		`TOTALS  ${t.impressions} impressions  ${t.clicks} clicks  ` +
			`position ${t.position.toFixed(1)}  CTR ${(t.ctr * 100).toFixed(2)}%\n`
	);

	const rows = await query(token, { dimensions: ['query'], rowLimit: 1000 });

	const bands = new Map<string, { imp: number; clk: number; queries: number }>();
	for (const r of rows) {
		const b = band(r.position);
		const cur = bands.get(b) ?? { imp: 0, clk: 0, queries: 0 };
		cur.imp += r.impressions;
		cur.clk += r.clicks;
		cur.queries += 1;
		bands.set(b, cur);
	}

	console.log('POSITION BANDS');
	for (const b of ['top 3', 'rest of page 1', 'page 2', 'page 3+']) {
		const v = bands.get(b);
		if (!v) continue;
		console.log(
			`  ${b.padEnd(15)} ${String(v.imp).padStart(6)} imp  ` +
				`${String(v.clk).padStart(4)} clk  ${v.queries} queries`
		);
	}

	const top = bands.get('top 3');
	if (top && top.imp >= 100 && top.clk === 0) {
		console.log(
			`\n  VERDICT: ${top.imp} impressions in the top three positions produced zero clicks.`
		);
		console.log('  That is the machine-issued signature. Read the query list below.\n');
	}

	console.log('TOP 20 QUERIES BY IMPRESSIONS');
	for (const r of rows.sort((a, b) => b.impressions - a.impressions).slice(0, 20)) {
		console.log(
			`  ${r.keys[0].slice(0, 60).padEnd(60)} ${String(r.impressions).padStart(5)} imp  ` +
				`${String(r.clicks).padStart(3)} clk  pos ${r.position.toFixed(1)}`
		);
	}
	console.log();
})();

Here is what the script does, part by part.

It takes three arguments off the command line: the Search Console property you own, the single page you want to investigate, and how many days to look back. The lookback defaults to 90. If you leave out the property or the page, it prints a usage line and exits, because every query below is meaningless without both.

Next it builds the date window. Search Console data lags by roughly two days, so the script ends the window two days before today rather than today. Ending it on today would pull a partial, still-filling day and make your most recent numbers look worse than they are. The iso() helper trims a JavaScript Date down to the YYYY-MM-DD string the API expects.

getToken() handles authentication. It creates a GoogleAuth client with exactly one scope, webmasters.readonly, and exchanges your service account key for a short-lived access token. That scope is read-only, so the script can look at your property but can't change anything in it. If Google returns no token, the script throws instead of carrying on with an empty Authorization header.

query() is the one function that talks to the API. It POSTs to the searchAnalytics/query endpoint for your property with the token in an Authorization: Bearer header. The important part is the dimensionFilterGroups block: a single filter on the page dimension with the operator equals. That filter is what turns a whole-site report into a report about one URL. Without it you would get your entire site back, and none of the comparisons in this article would mean anything.

The script then makes two separate calls, and the gap between them is the whole point of this piece. The first call asks for the page dimension, and because the filter has already narrowed the report to one URL, that comes back as a single summary row: total clicks, total impressions, click-through rate, and average position for that page. The second call asks for the query dimension with a row limit of 1000, which returns the individual searches that produced those impressions. A page can look ordinary in the summary and obviously machine-fed in the query list.

Finally, band() sorts the average position of each search into one of four buckets, and the script prints two reports: impressions grouped under POSITION BANDS, and the twenty highest-impression searches under TOP 20 QUERIES BY IMPRESSIONS. If the page had no impressions in the window, it says so and stops.

Run it against your suspect page:

npx tsx phantom-check.ts https://your-site.com https://your-site.com/your-suspect-page 90

Here's what it printed for my article titled "How ChatGPT and Perplexity Decide Which Sources to Cite":

https://chudi.dev/blog/aeo-answer-engine-optimization-explained
window: 2026-05-19 to 2026-08-17 (90 days)

TOTALS  7480 impressions  7 clicks  position 8.5  CTR 0.09%

Seven clicks from 7,480 impressions is a click-through rate of 0.09 percent. Average position 8.5 sits in the middle of the first page.

Google Search Console performance view for chudi.dev filtered to a single article over three months. Total clicks 7, total impressions 7.5K, average CTR 0.1 percent, average position 8.5. The impressions line rises and falls across the whole window while the clicks line stays along the bottom, rising to a single click on a handful of isolated days.

The same figures appear in the Search Console interface if you prefer to start there. The script exists so you can run the comparison in Step 4 without clicking through two properties by hand.

Stop here and you'd conclude that the page ranks fine and converts badly. That's exactly the conclusion that leads to a wasted title rewrite. The page-level average is hiding the distribution, and the distribution is where the answer lives.

Step 2: Split the Impressions by Position Band

The script already did this. Read the next block of its output:

POSITION BANDS
  top 3              100 imp     0 clk  16 queries
  rest of page 1    3068 imp     0 clk  92 queries
  page 2             110 imp     0 clk  25 queries
  page 3+             82 imp     0 clk  23 queries

  VERDICT: 100 impressions in the top three positions produced zero clicks.
  That is the machine-issued signature. Read the query list below.

Look at the second row. On the first page of Google, outside the top three, this page took 3,068 impressions across 92 distinct queries and produced zero clicks.

For scale: a result sitting in positions 4 through 10 normally takes somewhere between 2 and 10 percent of the clicks. At 3,068 impressions, the expected click count is roughly 60 to 300. The observed count is zero. That's not a weak title. A weak title still leaks a few clicks.

Notice also the 100 impressions in the top three positions, again with zero clicks. Positions 1 through 3 convert at 10 to 40 percent for human searchers. One hundred impressions there should have produced something.

Zero clicks in every single band is the signature. Human traffic is noisy and leaks clicks everywhere. Machine traffic is clean and leaks nothing.

One caveat before you go further. The band totals sum to 3,360 impressions while the page total says 7,480, and the page shows 7 clicks while every query row shows zero. That's not a bug in the script. Search Console withholds query rows that are rare enough to identify an individual searcher, so the query dimension never sums to the page dimension. Use the bands for their shape, not as a full accounting.

Step 3: Read the Query List

The position bands tell you something is wrong. The query list tells you what.

Here are real rows from that page, exactly as the script printed them. The text is truncated at 60 characters by the output column:

TOP 20 QUERIES BY IMPRESSIONS
  how do answer engines like chatgpt and perplexity decide whi  1313 imp    0 clk  pos 8.7
  aeo platform that shows which urls chatgpt cites from my sit   206 imp    0 clk  pos 3.7
  as a director of seo at a mid-size company in north america,   193 imp    0 clk  pos 6.2
  what's the minimum viable aeo optimization?                    128 imp    0 clk  pos 3.3
  how can i improve my website's visibility in answer engines    115 imp    0 clk  pos 4.5
  aeo tool that explains chatgpt citation changes                106 imp    0 clk  pos 5.6
  ai content citation criteria answer engine optimization        105 imp    0 clk  pos 7.9
  ai content citation criteria answer engine optimization aeo     95 imp    0 clk  pos 8.3
  how do generative engine optimization (or answer engine opti    61 imp    0 clk  pos 4.6
  aeo tool that explains why a page stopped getting cited         60 imp    0 clk  pos 8.7
  before answer engines can cite your content, what must they     53 imp    0 clk  pos 4.8
  why citations matter for aeo.                                   47 imp    0 clk  pos 6.7
  give me 5 key takeaways from https://wildseo.co/. remember w    44 imp    0 clk  pos 2.8
  aeo tool to diagnose why i dropped out of perplexity citatio    43 imp    0 clk  pos 8.6
  which gpt model uses the same global-scale infrastructure as    43 imp    0 clk  pos 8.3
  ai content citation criteria answer engine source selection     40 imp    0 clk  pos 8.7
  ai content citation criteria answer engine source selection     31 imp    0 clk  pos 9.9
  which generative engine optimization (or answer engine optim    27 imp    0 clk  pos 4.2
  answer engine citations                                         26 imp    0 clk  pos 16.4
  what's the minimum viable aeo program?                          25 imp    0 clk  pos 1.0
Google Search Console queries table for the same page. The rows are long natural-language queries, most of them complete questions, including one reading "as a director of seo at a mid-size company in north america, operating in the technology sector... how do answer engines like chatgpt and perplexity decide which sources to cite?" at 193 impressions. The clicks column reads 0 on every row.

Search Console shows the same rows untruncated, which is worth a look because the persona-framed query is easier to recognize at full length.

The first row deserves a note. That query is my own headline read back to me, near enough word for word. A fan-out that has already selected your page will search for your title, which is why the largest row on a phantom page is so often the page itself. It took 1,313 impressions and returned nothing.

Now compare that against how people actually type into a search box. Four signatures give the machine away:

1. Full natural-language sentences with punctuation. Humans type "aeo tools" and move on. They don't type "as a director of seo at a mid-size company in north america, ..." into Google. That's a persona-framed prompt, and it took 193 impressions.

2. Instructions rather than questions. The row reading "give me 5 key takeaways from https://wildseo.co/. remember w..." is not a search. It's a task given to an assistant, which then went and searched. It sat at position 2.8 and took 44 impressions.

3. Near-identical permutations of one phrase. Count the rows beginning "ai content citation criteria answer engine". Four of the top twenty are the same phrase with the tail swapped: "optimization", "optimization aeo", and "source selection" twice at two different positions. A human asks once. A fan-out asks the same thing several ways, and every variant logs its own impression.

4. Literal machine artifacts. Two rows above are quiz stems rather than searches: "before answer engines can cite your content, what must they..." and "which gpt model uses the same global-scale infrastructure as...". Below the printed top twenty it gets less subtle. The script requests up to 1,000 rows, so raise the slice(0, 20) at the end to see all of them. Mine also contain the string chatgpt://generic-entity?number=6, a bare yes, a placeholder yoursite.com, stems beginning "true or false?", a full-sentence query in German, and a fragment of an assistant's own system prompt beginning "context: location: united states (not for language). do not in...". No person typed any of those into Google.

If you find one of these, it may be coincidence. If you find all four on one page, you're looking at fan-out traffic.

There's one more check worth making in the Search Console interface itself. Open the page, then look at the Search Appearance dimension. Search Appearance is a Search Console breakdown that tags impressions by the kind of result they showed up in, for example an AI Overview, a rich result, a video, or an FAQ, rather than a plain blue link. Google only fills it in for the result types it has chosen to report on, which is why an empty panel tells you nothing on its own.

For this page it returns no rows at all, which means Google isn't reporting any AI Overview appearance for it. Absence there isn't evidence either way, so don't treat an empty Search Appearance panel as proof of anything. Note it and move on.

Step 4: Compare Against a Control Page

A single page in isolation proves little. Your property might simply have poor titles across the board. The control removes that explanation.

Pick your best article, the one you know real people read, and run the identical script over the identical window:

npx tsx phantom-check.ts https://your-site.com https://your-site.com/your-best-page 90

Same site, same script, same 90 days, and my model-comparison article "Fable 5 vs Opus 4.8: Every Reasoning Tier Benchmarked":

https://chudi.dev/blog/claude-fable-5-vs-opus-4-8
window: 2026-05-19 to 2026-08-17 (90 days)

TOTALS  30704 impressions  767 clicks  position 6.8  CTR 2.50%

POSITION BANDS
  top 3              140 imp     6 clk  44 queries
  rest of page 1    6515 imp   273 clk  294 queries
  page 2             790 imp     5 clk  127 queries
  page 3+            222 imp     0 clk  65 queries
Google Search Console performance view for the control article over the same three months. Total clicks 767, total impressions 30.7K, average CTR 2.5 percent, average position 6.8. The clicks line and the impressions line sit at zero until early June, then rise and fall together for the rest of the window.

Compare that chart against the first one. Here the two lines move together, which is what a human audience looks like. On the phantom page the clicks line never leaves the axis.

Put the two rows next to each other:

Page Impressions, positions 4 to 10 Clicks, positions 4 to 10
"How ChatGPT and Perplexity Decide Which Sources to Cite" 3,068 0
"Fable 5 vs Opus 4.8: Every Reasoning Tier Benchmarked" 6,515 273

Roughly twice the impressions produced 273 clicks. Half the impressions produced none at all. Same domain, same author, same publishing pipeline, same 90 days, and the same script.

The query lists differ in kind, not just in performance. The control page ranks for short keyword strings: "fable low vs opus high" at 691 impressions and 38 clicks, "fable high vs opus max" at 280 impressions and 25 clicks. Those are humans typing fragments. The phantom page ranks for grammatically complete sentences that nobody types.

Google Search Console queries table for the control page. The queries are short keyword fragments such as "fable low vs opus high" at 38 clicks from 691 impressions and "fable high vs opus max" at 25 clicks from 280 impressions. Every row shows a non-zero click count.

Set that table beside the one in Step 3. It's the same site, same window, and the same script. One is dominated by long machine-shaped queries that return nothing, the other by short fragments that convert.

Once you see the two outputs side by side, the conclusion is no longer a judgment call.

What I Rejected, and Why

I tried three other approaches first. All three are plausible and all three are worse. Knowing why saves you the detour.

Rejected: a Daily-Impression Variance Test.

My first instinct was that machine traffic should look more regular over time than human traffic, so I compared the coefficient of variation of daily impressions between the two pages. The phantom page scored 0.78 and the human control scored 0.69. By that test the phantom page looked more human than the human page.

The test was simply too weak to resolve the difference, and it contradicted arithmetic that wasn't close. When a subtle instrument disagrees with an overwhelming one, keep the overwhelming one.

Rejected: Filtering Every Zero-click Row.

The obvious automation is to drop any query row with zero clicks. Don't do this. It fires on pages that have only just started to rank and haven't accumulated a click yet, so it silently suppresses your genuine risers. It also fires on a quirk described below.

The correct gate is at page level, not row level: a zero-click row on a page that takes clicks elsewhere is a real opportunity, while a row on a page that takes zero clicks across all of its queries inside position 11 is a phantom.

Rejected: Another Title Rewrite.

Before running any of this I had already rewritten the titles on these pages five separate times over five months. Impressions moved. Clicks didn't. If you have already changed a variable several times with no effect, the next change isn't an experiment, it's a habit. Check whether the thing you're optimizing exists before you optimize it again.

One related trap deserves its own warning, because it manufactures fake phantoms on healthy pages. Search Console reports rows for anchor fragments, meaning URLs of the form page#section, as separate rows. It splits impressions across those rows but attributes the clicks to the parent URL. The result is a set of zero-click rows belonging to a page that converts perfectly well.

When I audited my own candidate list, 64 of 124 entries were duplicates of this kind, and one converting article appeared six times as a supposed click-through-rate gap. Strip any row whose URL contains a # before you analyze anything.

What to Do With a Phantom Page

Nothing, on the page itself. That's the uncomfortable answer, and it's the right one.

Don't rewrite the title. Don't rework the meta description. Don't point internal links at it to give it a push. Every one of those actions optimizes for a reader who will never arrive, and the effort has a real cost measured in the work you didn't do on a page with humans on it.

What the finding actually changes is your instrumentation and your expectations:

  1. Exclude phantom pages from click-through-rate tooling. Any script or agent that proposes title rewrites needs the page-level gate from the previous section, or it will keep nominating your deadest pages as your biggest opportunities.

  2. Stop reading those impressions as demand. A dashboard showing 7,480 impressions looks like an audience. It's a retrieval count. Don't brief a client, or yourself, on machine impressions as though they were interest. Google's own Generative AI report, which went live for every property on 2026-08-11, shows the same impressions without splitting them, and I probed what that report does and does not expose.

  3. Read it as a retrieval signal instead, which is genuinely good news. Your page was selected as a source, repeatedly, at good positions, for questions an assistant was actively researching. Several of the phantom queries on my page are shaped like buying intent: people asking an assistant where to get help with exactly what that page is about. The page is being cited into answers and earning nothing from it. That's not a visibility failure, it's an attribution and hand-off failure, and it's a completely different problem to solve. If you want to measure the citation side directly, that's what I built citability.dev to do.

The distinction is the whole point. "My page is invisible" and "my page is visible to machines and invisible to my analytics" call for opposite responses.

FAQ

Does this mean AI Mode impressions are worthless?

No. It means they're not clicks and must not be counted as reader demand. A retrieval impression says a machine chose your page as a source for a question it was answering. That is worth knowing and worth measuring. It's simply a different metric that happens to share a column name with a human one.

Can I do this in the Search Console interface without the script?

Partly. You can filter to one page and sort queries by impressions, and the sentence-shaped queries will be visible. What the interface won't do is split one page's impressions into position bands, and that split is the step that turns a hunch into a decision. The script exists for that one calculation.

What if a page has both human and machine impressions?

Most pages do, and the bands will show it as clicks in some bands and none in others. Treat the page as mixed, not phantom. The all-zero pattern is what justifies pulling a page out of your click-through-rate tooling. Anything less than that, keep optimizing normally.

Will blocking AI crawlers stop these impressions?

No, and this is the most common mistake I see. Adding GPTBot or Google-Extended to robots.txt blocks training crawlers, which collect text to train models. Query fan-out is retrieval, and it runs against Google's ordinary search index using the same Googlebot crawl that powers every other result. Blocking training access doesn't remove a single one of these impressions. It only removes you from the corpus.

How often should I re-run this?

Once per quarter per suspect page is enough. The classification is stable, since it reflects what kind of queries the page matches rather than a ranking that moves week to week.

What You Accomplished

Search Console shows you impressions. It doesn't tell you whether a person was attached to one.

The four steps separate them with data you already have:

  1. Pull the page totals, and distrust the average position.

  2. Split the impressions into position bands. Zero clicks in every band, especially inside the top ten, is the signature.

  3. Read the query list for full sentences, instructions, permutation families, and literal machine artifacts.

  4. Run the same script over your best page in the same window. The contrast is the proof.

Run it on your own property before your next round of title edits. The 20 minutes it takes is cheaper than a month spent optimizing for an audience that was never there.

I'm Chudi Nnorukam, and I run this split continuously against my own properties. Check out this page to learn more.

]]> <![CDATA[ From Data to Value: Understanding Data Management Through a Real World Use Case [Full Book] ]]> https://www.freecodecamp.org/news/understanding-data-management-with-a-real-world-use-case-book/ 6a8db9c20b9b2c87b7c4049f Tue, 25 Aug 2026 15:50:26 +0000 Today, data has become a particularly valuable resource. It allows companies to compete in the market and drive innovation, improving the quality of products and services offered.

Data processing lets teams automate processes. It also supports decision-making, offers a significantly more personalized experience to the end user, and detects patterns in many areas such as banking fraud or risk mitigation. Companies need to know how to capture and use data effectively, safely, and legally.

You likely are or have been a user of various products and services. And you know that processes involving data are fundamental to almost everything around us. You're likely also already familiar with terms like Big Data, Data Analytics, Artificial Intelligence, and Machine Learning.

But unless you're an expert in one of these fields, some of these concepts might seem overwhelming. These are large areas of study, after all.

And even if you're trained in one of these areas, it's difficult to know all the details about each field, as the data world is vast.

One way to understand this world of data a bit better is by dividing it, and establishing a distinction between the areas of Artificial Intelligence and Data Management. This isn't the only way to proceed, but I've found it helpful to separate the set of disciplines and techniques for information processing into these two blocks.

On one side is Data Management, which encompasses everything related to the capture, storage, protection, and analysis of data.

Meanwhile, on the other side is Artificial Intelligence, which focuses on developing techniques that allow a machine to emulate human capabilities like reasoning or learning to solve a problem, whether interacting with data or not.

Here, interaction refers to an algorithm acquiring "knowledge" from data, but not all artificial intelligence functions.

In any case, this book offers a comprehensive overview of Data Management, helping you understand all the terms and related concepts involved in using, processing, and analyzing data.

It won't just provide an abstract explanation of the field and its contents. It'll instead help you understand it holistically and offer a more practical and realistic view. We'll also study a use case to put into practice everything we discuss.

Table Of Contents

Our Case Study

Our use case involves a fictional university offering international master's programs in Artificial Intelligence and Data Management. It's a public-private institution providing various training programs for different end users, such as recent graduates looking to specialize in this area, working professionals, or international students.

This use case lets us analyze the entire data lifecycle, from student admission to graduation. Also, in a university setting, we can use data alongside artificial intelligence to automate enrollment processes, enhance the student's experience when accessing educational resources, optimize organizational operations, and ultimately help the university differentiate itself from other institutions offering similar programs.

The data lifecycle begins before enrollment in a master's program, as a candidate might discover the program through an advertising campaign, visit the institution's website, or complete an application form. They can then enroll and attend classes, using digital platforms and participating in various educational activities. Finally, they'll complete the program and become part of the alumni community.

Each of these interactions generates different types of data, such as personal, academic, administrative, and financial data. There are also more complex types of data, like activity and digital behavior data, which can include records of access to the virtual campus or consulted resources, among others.

This journey allows us to see how data goes through different phases. We'll see how it's captured, validated, stored, integrated with other systems, protected, analyzed, and finally retained or deleted according to the organization's policies.

As you can imagine, Data Management isn't just about storing data in a database. It's also about ensuring that, throughout its lifecycle, the data is accurate, secure, understandable, accessible to those who need it, and used legitimately.

Also, the university, like any other entity, uses data to identify the needs or problems of its users in order to propose solutions. One such issue could be commuting, as some students in the master's programs live far from campus, others might work, and still others may have poor public transportation options. In these cases, distance or travel time becomes a decisive factor for those students.

Faced with this seemingly complex issue, the university can use data and artificial intelligence techniques to plan and offer suitable transportation services to certain interested students. This means, based on eligibility criteria such as the distance from campus or enrollment in mandatory in-person classes, the university can plan to offer free taxi/VTC services to certain students.

But the idea wouldn't be to provide unlimited taxi services to all students – just to design a controlled, measurable, and sustainable benefit based on clear business rules.

Processing this data effectively would allow the university to offer a more precise service than other competitors, who might offer generic public transportation discounts or fixed bus routes. And while these solutions might be very useful, they don't always adequately meet the needs of all students.

In this scenario, it's clear that a wide variety of data is generated, including data on students, faculty, courses, schedules, attendance records, trips taken, and so on. Using and analyzing this data, we'll be able to learn many Data Management principles. We'll also demonstrate how data pipelines are built, how data is transformed into useful analytical products, and what techniques are involved.

To make these ideas easier to follow in practice, this book is accompanied by a hands-on Jupyter notebook. It uses a compact sample of real taxi-trip data and treats it as a provider feed for the university's transportation service.

Some of the examples discussed throughout the book are reproduced in the notebook with the same dataset, so as you move through the chapters, you can see selected concepts in action, including data profiling, quality rules, integration, transformation, privacy protection, dimensional modeling, SQL analysis, and visualization. It's a focused demonstration rather than a complete implementation of every capability discussed here.

You'll also learn how the university might use artificial intelligence to predict which candidates are most likely to enroll, recommend master's programs, estimate future demand for mobility services, detect unusual patterns in taxi usage, and create conversational assistants to help candidates and students resolve their questions.

This case study will also highlight the university's need to make decisions about privacy, consent, transparency, and security. For example, personal data must be protected, eligibility rules should not unfairly discriminate, and human oversight should be established for decisions that could significantly impact a candidate or student.

Data Management Fundamentals

Data Management is the discipline responsible for capturing, storing, protecting, integrating, understanding, maintaining, and correctly using data throughout its lifecycle. At first glance, management and processing might seem to involve only storage and perhaps later analysis, but nothing could be further from the truth.

There are many more requirements like security (as managing large volumes of information quickly is useless if security is compromised) as well as data integrity and organization.

While researching for this book, I studied the very useful book Data Management Body of Knowledge (DAMA-DMBOK). It's one of the most comprehensive and reputable guides on the world of data. And I highly recommend it if you want to dive even deeper here.

According to the book, Data Management involves the development, execution, and supervision of plans, policies, programs, and practices that enable the delivery, control, protection, and enhancement of the value of data and information assets throughout their lifecycle.

This definition is especially relevant because it highlights two fundamental ideas. One is that data has intrinsic value, allowing it to be treated as an asset. The other is that this value doesn't appear directly in all cases but depends on how the data is managed.

In other words, data alone has no value, but if you process it properly, it has the potential to become usable information and subsequently knowledge.

To achieve this goal, you can think about Data Management as a set of operational capabilities, meaning the various actions a team or organization must undertake regarding its data.

Among the most fundamental are the following:

  • Data Governance: deciding who has access to each piece of data and who sets the access rules.

    • Example: University faculty may have access to certain data about students in their courses, but not about any student in the organization.

  • Data Architecture: designing the processes that data will follow throughout its lifecycle.

    • Example: A data architect defines how data travels from the moment a user enters it into the system, such as during an enrollment form, to where it's stored and processed internally on the university server.

  • Data Storage and Operations: deciding how and where the data is stored.

    • Example: The decision is made to store students' personal data in an internal database, as opposed to alternatives like storing it in an external cloud service. Meanwhile, other data, such as educational materials, are more likely to end up stored in the cloud, although it ultimately depends on the organization's policies.

  • Data Integration: gathering information from different sources to provide a unified view or access to all of them.

    • Example: A data engineer integrates information from different sources about taxi routes, as each company will have its own source with unique characteristics, making it necessary to standardize the data into an intermediate schema.

  • Data Quality: ensuring that the information is accurate, complete, consistent, up-to-date, and reliable.

    • Example: A quality analyst defines the rules that the virtual campus frontend must follow to prevent end users from entering incorrect data into the system, ensuring its quality. They also impose rules on the various internal systems where the information is stored to avoid inconsistencies.

  • Data Security and Privacy: protecting information against unauthorized access and other threats.

    • Example: User access passwords are stored as hashed values, not in plain text, to prevent easy access in case of a potential vulnerability.

  • Metadata Management: specifically managing the data that determines the meaning of other data.

    • Example: A glossary is created with terms that define the meaning of each concept represented in the data. One of them could be "distance to campus in meters." In this case, the meaning is clear, and its inclusion in the glossary allows it to be used in the implementation of storage systems and data processing, facilitating development.

  • Analytics and Business Intelligence: transforming data into reports and visual indicators that facilitate strategic decision-making within the organization.

    • Example: A data analyst creates an interactive dashboard for the administration, displaying graphs of monthly taxi expenses, the number of students benefiting, and how this service has improved the percentage of attendance in in-person classes.

So as you can see, Data Management isn't a specific activity but a collection of many different tasks and processes. When coordinated, these allow data to be transformed into strategic value.

In the university use case, it's clear that the personal data of applicants and students must be protected. Also, to help implement the free taxi service, the data sources from different transportation companies must be well-integrated and of high quality.

Data as an Asset

Data can be defined as a symbolic representation of a quantitative or qualitative attribute or variable. In other words, data are representations of facts, observations, events, or characteristics occurring in an environment, which can later be stored and processed.

This definition of data relates more to its types, such as numbers, dates, text, or images. In our use case, data might include a student's name, address, or the distance from their home to the campus. Each of these, in isolation, is a simple record, but when contextualized and analyzed together, they have the potential to become an asset.

For instance, an isolated piece of data like "18 kilometers" isn't very relevant by itself. But if it's interpreted as the characteristic "distance to campus", it becomes useful for understanding a student's situation and making a decision.

In this context, an asset is any resource expected to yield a return in the future, like buildings, patents, or other elements. Here, we're also including data because of its potential to generate value within the organization.

But this doesn't mean that just any piece of data is an asset. Data can be incorrect, duplicated, or incomplete. So its value mainly depends on how it's managed. For example, at the university, "distance to campus" becomes an asset when it's not used as an isolated number but rather for decision-making.

In our example, the distance from campus along with other student and organizational data can help us decide which students are eligible for this taxi service or how much budget should be allocated for it.

Data that's considered an asset can help drive these decisions only when the quality is adequate, because incomplete, inconsistent, or erroneous data can affect this process negatively or not contribute to the decision.

Ultimately, considering data as assets means treating it as a resource that requires specific management. And this can lead to benefits you wouldn't be able to achieve otherwise, whether it's improved end-user satisfaction or cost optimization.

Data, Information, Knowledge, and Value

From this idea arises the distinction between data, information, knowledge, and value. We'll study the progressive transformation that turns data into useful knowledge and ultimately into value for an organization.

Data

First, data is the most basic unit dealt with in Data Management, and its main function is to represent an aspect of reality. That is, data is what we imagine when we think of something like a number, some text, a date, and so on. Data has types (because of its variety), and also has a basic meaning associated, generally called semantics.

  • Example: "18 kilometers" is a piece of data of the integer type, and its semantics indicate that it represents a quantity of kilometers. Here, it's important to realize that the quantity alone might be considered data, but its semantics allow for interpretation.

Information

Once we have isolated data, we can relate and contextualize it to create a more abstract meaning, which is considered information.

  • Example: To better understand this concept, the previous data "18 kilometers" can be contextualized with other information like a student's name or address, allowing us to infer that the student lives that far from the campus. This is considered information, as its semantics go beyond that of a simple piece of data.

Knowledge

After obtaining information, we can then analyze and interpret it to identify patterns, trends, or cause-and-effect relationships. We do this by integrating the information and observing the prior experience of the organization or similar ones, creating an even more abstract contextualization.

  • Example: If the university observes that students living more than 15 kilometers away and having in-person classes miss more classes, it can conclude that distance and schedule influence attendance. This requires information such as the students' distance from campus or their attendance records and schedules.

Value

Finally, we use knowledge in decision-making and taking actions that can generate a benefit, which is the value derived from the data.

  • Example: The university can offer free taxi services only to specific students who meet certain criteria, improving attendance and user satisfaction while minimizing the impact on the budget. Here, the value lies in the benefit gained from these decisions, which may or may not be easily measurable.

In summary, success doesn't lie solely in storing large volumes of data or processing them at high speed, but in advancing them through this sequence of transformations to turn them into value. This process requires an infrastructure suited to these needs, as well as qualified people who are capable of applying the appropriate Data Management techniques.

The Data Lifecycle

Now let's look at the stages data goes through. Its lifecycle starts when the organization identifies a need for it and ends when the data is no longer useful. Between those points, teams capture, store, maintain, use, and eventually retain or delete the data. The lifecycle describes the phases that keep this journey controlled.

The data lifecycle diagram. Image by author.

As the diagram shows, the lifecycle starts with business needs, not technology choices. Because data is an organizational asset, each phase should help protect it, maintain it, or turn it into value.

The lifecycle consists of the following phases (as in the graphic above):

  • Planning: The organization decides what data it needs, why it needs it, who will be responsible for it, and how it could create value. Before capturing anything, the team should know which data is truly necessary and what they expect to do with it.

    • Example: The university decides it needs to know the distance between a student's home and the campus to evaluate whether it can offer free taxi service, explaining why it's necessary and what decision it will allow later.

  • Design and Enablement: Once the need is clear, the team designs the infrastructure, data flows, and policies that will support it. This work draws on capabilities such as data architecture, modeling, security, quality, and governance, which we'll discuss later.

    • Example: The university defines that the distance to the campus will be calculated from the address provided by the student, that the data will be stored in a specific system, that only certain departments will have access to it, and that it must be updated if the student changes their address.

  • Creation or Acquisition: At this stage, the data enters the organization for the first time. A user might create it through an interaction, or the organization might obtain it from an external source through an API, exchange, purchase, or integration.

    • Example: The data is created when the applicant completes the admission form indicating their address. External data such as geographic information or estimates of distance and travel time from a geographic API could also be obtained.

  • Storage and Maintenance: Once captured, the data must be stored in an appropriate environment and kept ready for later use. Teams may store it in databases, Data Warehouses, Data Lakes, or other systems. They can then clean, integrate, update, document, and protect it as needed.

    • Example: A student's name is stored in a university server database, while the calculated distance to the campus can be saved in a cloud-based analytical database. Additionally, rules are applied to avoid duplicates, incomplete data, or inconsistent formats, ensuring data quality.

  • Use: The organization uses the data for the purpose defined during planning. It might query or analyze the data, generate reports and dashboards, or use it to train AI models.

    • Example: The university uses IP addresses, response times, and virtual campus activity logs in a predictive Machine Learning algorithm to detect behavioral anomalies that indicate potential fraud. This use allows for the detection of identity theft, security issues, and the prevention of fraud in online educational activities.

  • Enrichment: In this phase, teams connect and transform data to add context and uncover patterns or trends that were previously hard to see. This is one way data becomes information and knowledge.

    • Example: A student's access log to the virtual campus can be enriched with data about the time they spent using online resources, the number of material downloads they made, their interaction counts, and their historical statistics. This gives the university more context for studying engagement and its possible relationship with academic progress, without assuming that digital activity alone explains a student's results.

  • Dispose: When the data is no longer needed for its original purpose, the organization decides whether to retain, archive, anonymize, or delete it. Retention policies, business needs, and legal requirements guide that decision. This phase prevents the organization from accumulating unnecessary data, which raises costs and creates extra risk when the data is personal or sensitive.

    • Example: When a student completes their master's program, the university may need to retain grades and other academic records for legal or administrative reasons. Some banking details or operational payment data may no longer be necessary once financial and legal obligations end. The retention policy should identify which fields to keep, delete securely, or anonymize for approved statistical use rather than preserving the full record indefinitely.

Although these phases appear in sequence, real data rarely moves through them only once. Teams may enrich it several times or integrate it with new sources, such as public APIs or partner systems. Think of the lifecycle as a continuous process whose phases can repeat whenever the need changes. It helps keep Data Management consistent, secure, and useful.

Data Management Principles

Now that you understand the lifecycle, you can use a few key principles to guide decisions at every stage. They give teams a shared reference instead of letting each system or department manage data in isolation.

The first fundamental principle already covered is considering data as an asset. From there, another relevant principle emerges: the value of data depends on its quality and context. Incorrect, incomplete, or misinterpreted data can lead to wrong decisions. For example, if a student's name contains a typo, it might not match what's stored in government databases, complicating certain processes. It's also crucial to understand that data needs metadata to be used correctly.

As I explained before, an isolated number like "18" has little value if it's unclear what it represents, in what unit it's expressed, how it was calculated, or when it was updated. Metadata documents this meaning and prevents ambiguities.

Another important principle is the need for planning. As seen in the lifecycle, the first step should be planning which data is expected to be used, among other things. In the case of enrollment, the university shouldn't collect just any student data, but only the relevant information required for the necessary processes.

Another essential principle is to use technology for a clear purpose. A team shouldn't choose a database or a new tool simply because it's the current popular tool. It should choose technology that addresses a real need. At the university, the decision to use a relational database, a Data Warehouse, a geographic API, or a dashboard should depend on the goal of the use case.

Data Management Capabilities

These principles become practical through a set of Data Management capabilities. The capabilities describe what an organization must be able to do with its data throughout the lifecycle.

Data Management main capabilities. Image by author.

The principles guide the work, while capabilities such as Data Governance and Data Modeling put that guidance into practice. The diagram above shows the main capabilities we'll cover in the coming sections.

Some Data Management roles work across several capabilities. One is the Chief Data Officer (CDO), who defines the organization's data strategy and helps ensure that teams manage data as an asset. In our use case, the CDO would help set goals for using data, such as improving attendance, enrollment, or student satisfaction.

Another relevant role is the Data Steward, who helps maintain data definitions, quality, and proper handling within a domain. At the university, they might verify the completeness and consistency of student location and enrollment data. A Chief Privacy Officer (CPO) may also be involved whenever a use of data affects privacy.

Data Governance

Let's start with Data Governance. Data Governance defines how an organization makes decisions about data, who may access or change it, and what responsibilities come with each role. It also helps the organization meet its legal and regulatory obligations.

You can see why this matters at the university: admissions staff, the academic office, faculty, and even AI systems may use student data. Without clear rules, people can gain inappropriate access or make decisions without enough justification.

Not everyone should be able to perform every action on every piece of data. Data Governance provides an organizational control layer across the lifecycle so people use data in an orderly, secure, and legitimate way.

Organizations assign this work to roles such as the CDO and Data Owners. Data Owners usually work within a business area and have authority to make important decisions about the data in their domain.

For example, the director of mobility at a university would be the Data Owner of all data related to the transportation service offered by the university. There may also be other Data Owners like the financial director for all billing and tuition payment information.

Governance tools help teams control, document, and review data use. Their main purpose isn't programming or technical processing.

A CDO or Data Owner might use data catalogs and business glossaries to understand what information exists and what it means. They may also use policy-management platforms, dashboards, and lineage tools that track data from its source to its destination.

Data Ownership

One of the key governance concepts is Data Ownership, which assigns responsibility for different data domains. Ownership doesn't mean that a person literally owns the data. It means that someone has the authority and accountability to make decisions about it.

The main role here is the Data Owner. This is usually a business leader who makes lifecycle and usage decisions for a domain rather than an end user who simply works with the data.

For example, the university may need a student's address to calculate the distance to campus. But the Data Owner of that domain should determine if that address can be accessed by their teachers or shared with an external transportation company, among other decisions.

The Data Owner usually doesn't implement the technical solution. Instead, they use tools such as data catalogs to find and understand the assets in their domain. A catalog organizes those assets through metadata and makes them easier to govern.

Data Stewardship

The Data Owner sets direction for a domain, while a Data Steward supports its day-to-day management. Data Stewardship includes maintaining definitions, monitoring quality, and helping ensure that data is accurate, complete, and handled according to agreed-upon standards.

In practice, a Data Steward might focus on verifying that students' dates and addresses are in a valid and consistent format, ensuring their names are complete, free of illegible characters, and without other issues. Also, this role emphasizes metadata to interpret data and allow other team members to do so without conflicts.

Data Stewards often work with data catalogs and business glossaries. A business glossary standardizes key organizational terms. For example, it might define "distance to campus" as the route distance in meters along public streets rather than a straight-line measurement.

Decision Rights

Another governance concept is Decision Rights: the formal definition of who can make which decisions about data in a given context.

Decision Rights form part of the foundation of governance. Organizations often classify decisions by their scope. Strategic decisions happen at the highest level, for example, when the university decides whether to use mobility data to offer a transportation service.

Then there are tactical decisions, which bridge the gap between the organization's overall strategy and day-to-day operations, such as defining eligibility criteria for candidates for the transportation service.

Finally, there are operational decisions, which are closest to the end users, like accepting or rejecting an enrollment application.

Decision Rights formally assign these choices to specific roles and data domains. The Data Owner and Data Governance Council are especially important here, with the council usually setting the broader decision framework.

A Data Protection Officer (DPO) may advise on a decision and escalate concerns when access would conflict with data-protection requirements. The DPO's exact authority depends on the applicable law and the organization's governance model. Teams often implement Decision Rights through workflow tools and Identity and Access Management (IAM) systems that manage digital identities and permissions.

For example, a university administrator shouldn't have unrestricted database access. They might open a ticket in a workflow tool like Jira to request a specific permission. The appropriate Data Owner reviews the request, and an IAM system such as Microsoft Entra ID grants the approved access to the administrator's verified identity.

Data Policies

While Decision Rights say who can make a decision, Data Policies state how people must manage and use data. They set the limits, principles, and obligations everyone must follow.

At the university, there might be a policy stating that user geolocation data can only be used to calculate eligibility for transportation services and not for other decisions unrelated to academic activities. This is an example of a policy related to privacy, data retention, or its use in AI models.

The Data Governance Council often formalizes these policies, the CDO sponsors them, and Data Stewards help teams apply them. A data catalog can publish the rules and connect them to the affected data assets, while technical systems enforce the controls.

Data Standards

Data Standards are more specific than policies. A standard might define a format, naming convention, or validation rule so teams follow a policy consistently across the organization.

For example, the university might establish that all dates be stored in the same ISO-8601 format or that the distance to the campus is always stored in meters. To better understand, a well-known case in computer science is the storage of decimal numbers, where the IEEE-754 standard is commonly used for binary representation.

Shared standards let systems exchange data with fewer unnecessary transformations. Data Architects and Data Modelers help select and define the standards, while Data Engineers apply them in the implementation. Data Owners and Stewards oversee their use within each domain.

Data Accountability

Data Accountability means that people who have authority over data must also answer for how it's used. Teams need enough monitoring and evidence to trace important actions and understand what happened over time.

If a problem occurs, the organization should be able to establish who accessed the data, when they accessed it, what they did, and whether the action followed policy. Evidence, traceability, and clear responsibilities make governance demonstrable.

At the university, a faculty member may have a legitimate reason to access part of a student's record, but the system should log the access when appropriate. If a privacy issue arises later, audit records can help investigators understand what happened.

The Data Owner is accountable for proper use within the domain, while security, compliance, and platform teams provide controls such as access logs, audit trails, and lineage where relevant.

Data Ethics

Governance alone isn't enough. An organization also needs to ask whether a use of data is fair, proportionate, and justifiable. That's where Data Ethics comes in.

In a data context, ethics applies principles such as transparency, responsibility, privacy, and non-discrimination throughout the lifecycle. This becomes especially important with personal or sensitive data because poor decisions can limit opportunities or deny people services.

For example, in a university, data handling during admission processes can result in discriminatory biases in many ways, some possibly unknown or unexpected. Notable among these are biases based on income, ethnicity, or disability.

Data can introduce these biases in numerous ways, which is why it's important to consider ethics and question whether data should be collected or used and what biases they might introduce.

Ethical Data Use

Ethical data use starts with a clear, legitimate, and proportionate purpose. An organization should know why it needs each piece of data, what value it expects, and what risks the proposed use creates.

Laws such as the General Data Protection Regulation establish legal requirements that overlap with some ethical principles, but legal compliance and ethical judgment aren't identical. The GDPR applies in the European context, and organizations must identify the rules that apply in every region where they operate.

An example of unethical use is when personal data from candidates entered into a form is sold to marketing companies without the candidates' explicit consent. Here, it's evident that personal data can be used to make decisions and improve a service or be used without consent for other purposes unrelated to the user's benefit.

Roles involved in ethical data use can include the CPO, DPO, a Chief Data Ethics Officer or ethics committee, and Data Stewards. Their exact responsibilities vary by organization. Consent management platforms (CMPs) can record and manage the permissions users grant, but consent is only one possible legal basis for processing and one part of ethical review.

Consent and transparency are two important principles. Users should be able to understand what data is collected, why it's needed, how long it will be kept, who can access it, and whether it will be shared. These explanations should use plain language that a non-expert can follow.

In the case of a university, when a candidate applies for enrollment, the form shouldn't just request information and acceptance of terms. Instead, it should provide explanations about why each piece of data is requested. Clear explanations about how the data will be used and whether it will be shared with third parties should be given whenever possible.

Transparency doesn't end when a user submits a form. People should also be able to learn about their rights and use the processes available to request access or corrections when the applicable law provides them.

Fairness and Non-Discrimination

Fairness aims to prevent discrimination and harmful bias in the use of data. It matters especially in AI systems, where complex models and historical data can make bias difficult to detect or explain.

For example, a university might decide to award scholarships based on a candidate's zip code or area of residence. At first glance, this may not seem unjust, but in reality, people with very different incomes or academic records may live within the same zip code, and excluding entire areas could deprive qualified people of scholarship opportunities.

Data ethics requires teams to review their decision criteria. In practice, they may analyze bias, examine sensitive variables and their proxies, validate data quality and representativeness, and monitor outcomes over time. For consequential decisions, the organization should also provide suitable human oversight and a way to challenge errors.

Responsible Data Sharing

Organizations often need to share some data with service providers because they can't deliver every part of a service alone.

Sharing increases risk and needs an appropriate legal basis. That basis isn't always consent. For example, the university may need to share limited data with a taxi/VTC company to provide the service, but the company shouldn't receive the student's full record.

Whenever the use allows it, the organization should share anonymous or pseudonymous data instead of direct identifiers. Properly anonymized data can no longer be linked to a person by reasonably likely means. Pseudonymization replaces identifiers with codes or references, but an authorized party can still reconnect the data to the person using information kept separately, so the data remains personal and protected.

Ethical Risk Management

One practical way to support ethical data use is to assess and manage risk before a new use begins. The review should consider the expected benefits alongside possible harms, bias, privacy effects, and impacts on different groups.

For example, when designing the enrollment application form, before including a field to collect specific data like gender, income, or any other information, it's essential for an ethics committee to evaluate their usefulness, the problems that having this data might cause for students, and whether biases or discrimination could arise.

Data Ethics helps the university improve its services without losing sight of the fact that the data represents real people.

Data Security and Privacy

So far, we've looked at the rules and ethical choices that shape data use. We also need to protect data throughout its lifecycle. Security and privacy work together here, but they solve different problems.

Data Security uses policies, processes, and controls to prevent unauthorized access, alteration, disclosure, or loss. Privacy focuses on whether personal data is collected and used for legitimate purposes, with appropriate transparency and respect for people's rights.

Security commonly aims to preserve the confidentiality, integrity, and availability of data. Only authorized people should access or change it, and it should be available when needed. Those properties alone don't guarantee privacy. An address might be strongly secured, for example, but using or selling it for an unauthorized purpose would still violate privacy.

The university therefore has to address security and privacy together. It handles personal data whose exposure or misuse could cause real harm to students.

A Chief Information Security Officer (CISO) usually leads the security strategy and coordinates technical and defensive policies. The security team uses controls such as Identity and Access Management platforms to centralize identities, authentication, and permissions.

On the privacy side, the previously mentioned CPO helps oversee the organization's privacy program and compliance obligations.

Data Classification

We won't cover every part of security here. But a useful starting point is to identify what information exists and classify it by sensitivity.

Different data can cause very different levels of harm if exposed. A common classification scheme uses public, internal use, confidential, and restricted levels.

The first can be accessed by anyone, while internal use data is intended for organization members, though exposure wouldn't have a particularly severe impact. In contrast, confidential data requires authorization to be accessed, and restricted data needs the highest level of protection.

At the university, schedules published on the website would be public. Faculty work procedures might be for internal use. A student's academic record or travel history could be confidential, while banking information or credentials could be restricted.

When a dataset combines several categories, the organization should classify and protect the result according to the risk of the combined data, which may be as high as or higher than its most sensitive field.

The organization should record the classification as metadata in the data catalog so teams can use it throughout the lifecycle. When a Data Engineer integrates a source or an analyst creates a dashboard, they can see which precautions apply. Data Stewards often help classify the data, Data Owners approve the business decision, and security and privacy teams define the required controls.

Identity and Access Management

Once data is classified, the organization must control who can access it. Identity and Access Management (IAM) covers the processes and technologies used to manage digital identities and grant, review, or revoke permissions. Authentication verifies an identity, while authorization determines what that identity may do.

The fundamental principle guiding data access management is the principle of least privilege, according to which each identity receives only the permissions necessary to perform their job.

For example, an instructor can view the contact information of students enrolled in their courses but shouldn't access the information of unenrolled students. In other words, they have the minimum necessary permissions to perform their duties.

If the number of users to manage is high, it's most common to use Role-Based Access Control (RBAC)BAC), where permissions are associated with roles like instructor, administrative staff, or student, and then each user has a specific role.

As for the professionals responsible for these tasks, the Data Owners decide which roles need access to the data in their domain, while the IAM administrators implement the roles and their permissions with software like Microsoft Entra ID, an IAM technology that centralizes the management of identities, groups, and access policies.

Encryption

Access controls can fail, so organizations also use cryptography. Data encryption transforms readable information into ciphertext that an authorized system can reverse with the correct key.

This encryption should be applied both at rest and in transit, meaning when data is stored and when it is transmitted from one system to another over the network.

For example, the university should encrypt sensitive student data at rest so stolen storage doesn't reveal it in plain text without the required keys. Communications between a student and the university server should also use TLS through HTTPS to protect data in transit. Encryption is effective only when the algorithms, implementation, and key management are sound. Examples include:

Original data Protection applied Protected result
camille.bernard@email.com AES-256 encryption 8A4F2C91B7E03D6A...
ES12 3456 7890 1234 AES-256 encryption D91B70E4A62C8F15...
Password123! Salted hashing using Argon2id $argon2id$v=19$m=65536,t=3,p=4$...

Common approaches use symmetric and asymmetric cryptography, and both depend on strong key management. Keys shouldn't be embedded in source code or stored unprotected beside the data they secure. A Key Management System (KMS) or Hardware Security Module (HSM) can help generate, protect, rotate, and control access to them.

Security Architects and security specialists help select approved encryption standards, protocols, and key-management patterns, while Data Engineers and other developers apply them in each system. Encryption doesn't solve every security problem, so teams combine it with access controls, monitoring, secure development, and usage policies.

Data Masking

Many processes don't need to reveal a complete value. Data Masking transforms or partially hides data to reduce exposure while preserving enough utility for a specific task.

There are mainly two forms of masking. Dynamic Data Masking partially hides the information presented to the user without altering the original stored data. Thus, an authorized person can see the full value, while someone with fewer privileges sees a partial version like **1234.

On the other hand, Persistent Data Masking creates a permanently transformed copy, allowing systems to be tested without using real data.

For example, if the developers of the virtual campus need to test that the application works with thousands of students, subjects, and trips, they don't need to use real data. Instead, they can replace it with fictitious data, shifting dates, changing names to fictitious ones, and so on.

To better understand its purpose, here are some specific examples:

Original Data Technique Applied Displayed Result Purpose
Student’s bank account: ES12 3456 7890 1234 Dynamic masking ES** **** **** 1234 Verify the account without displaying it in full
Student’s email address: lucia.garcia@email.com Partial masking l***@email.com Confirm the student’s identity without exposing the full email address
Student’s full name: Lucía García Persistent substitution Student_1048 Test systems without using real identities
Student’s home address: Calle Mayor 24, Madrid Generalization Madrid Analyze residential areas without knowing the exact address
Student’s date of birth: 18/04/2001 Age-range generalization 20–25 years old Analyze age groups without revealing the exact date of birth
Internal student identifier: STU-45821 Pseudonymization 9F3A-71BC Manage a trip without sharing the student’s full identity

Masking, pseudonymization, and anonymization overlap in some implementations, but they aren't interchangeable. Masking alone doesn't guarantee that a dataset is anonymous. Pseudonymization replaces identifiers with codes while keeping the information needed to reconnect those codes to people separately. Because re-identification remains possible, pseudonymized data is still personal data and needs protection. Anonymization requires reducing identification risk to the point that people are no longer identifiable by reasonably likely means.

In this case, Data Stewards determine which data should be concealed and why, while security and Data Engineering teams implement these decisions at a low level.

Privacy Controls

The previous techniques help prevent unauthorized access. Privacy Controls address a different question: whether the organization has a valid purpose and appropriate rules for processing personal data.

The principles of Privacy by Design and Privacy by Default make privacy part of a system from the start and set privacy-protective defaults. One fundamental control is data minimization, which means collecting only what the stated purpose requires. An enrollment form, for example, shouldn't request a complete medical history unless a specific service and lawful purpose justify it.

Other controls apply to the purpose of the data and its retention. So in use cases, students' personal data shouldn't be kept longer than necessary or reused for other purposes like personalized marketing campaigns without authorization.

In Europe, the GDPR establishes principles and requirements that guide these controls. The organization must also identify the rules that apply in every region where it operates. The DPO monitors and advises on compliance where that role applies, while Data Owners, privacy specialists, security teams, and system designers turn the requirements into practical controls.

Audit and Compliance

The organization must be able to show that its controls and policies work. Auditing independently reviews the available evidence and tests whether controls operate as expected. Compliance covers the ongoing work of meeting internal policies, standards, contractual duties, and applicable regulations.

Logs are one important source of audit evidence. They can record who accessed data, when, from which system, and what action they took. Teams protect these records against tampering and retain them for a defined period based on risk, legal needs, and cost. Security Information and Event Management (SIEM) platforms centralize events from different systems and can generate alerts for unusual behavior.

For example, if a teacher occasionally checks the record of a student enrolled in their course, the behavior may be legitimate. But if they download hundreds of student records with whom they have no connection during the night and from another country, an alert should be generated for the security team to investigate the incident.

An audit might analyze logs, test whether identities have excessive privileges, and review how teams apply encryption and other controls. Independent reviewers and separation of duties help prevent the same administrator from controlling a system and the evidence used to assess their actions.

Roles involved include the CISO, the DPO, the Data Owners, the Data Stewards, and the compliance and audit teams. In summary, security and privacy require knowing what data exists, limiting who can use it, protecting it through controls, and preserving evidence that all of this is correctly followed.

Security Operations (SecOps)

Data security is ongoing work. Beyond policies and encryption mechanisms, SecOps (Security Operations) brings people, processes, and technology together for continuous defense.

SecOps teams monitor systems, detect threats, investigate alerts, and respond to incidents. They try to reduce risk early while staying ready to contain and recover from events that still occur.

In the university context, the SecOps team is responsible for overseeing the digital ecosystem in real time. For example, if a SIEM generates an alert because a teacher has downloaded hundreds of academic records at night or engages in any similar suspicious activity, the SecOps analyst receives the notification, assesses the risk, and takes action, such as temporarily blocking access as a preventive measure.

SecOps teams may also coordinate vulnerability scanning and remediation for the virtual campus and other systems so weaknesses are addressed before attackers exploit them.

In SecOps, key roles include SecOps engineers and security analysts, who work with the CISO to define and implement a defense strategy. These professionals rely on SIEM platforms to centralize event information and SOAR (Security Orchestration, Automation, and Response) tools to automate responses to common threats.

Data Architecture

Once you know who makes decisions about data and how to protect it, you still need to organize the systems that store, move, and process it.

Data Architecture designs the structure that meets those needs. Once the organization defines what it wants to achieve with data, the architecture shows how systems will store, transport, protect, and analyze it.

This capability connects business goals with technical implementation. It goes beyond choosing a database or sketching a pipeline: the design identifies which data the organization needs, where it lives, how it relates, and how it moves. The work can produce data models, flow diagrams, standards, and other architecture decisions.

In this use case, a candidate might enter their home address in a web form during enrollment. This data could then be sent to an admissions system and used in a query to a geographic API to calculate the distance to the campus, for example. It could also be used along with other data present in other systems, like class schedules, to verify eligibility if transportation service is requested.

Here, Data Architecture is responsible for designing how this complete data journey is carried out.

A poorly designed architecture can fail in several ways. Systems may exchange data incorrectly or stop communicating, interrupting a service for users. Even if nothing breaks outright, teams may duplicate data unnecessarily, raising costs and making integration harder. Good architecture reduces these risks and makes tradeoffs explicit.

The Enterprise Data Architect maintains the organization-wide view, while Data Architects and Solution Architects adapt it to particular solutions. Data Modelers, Data Engineers, Data Stewards, Data Owners, and security specialists contribute the design details and help put the architecture into practice.

In simple terms, architects design and document the solution, engineers and developers implement it, and Data Owners and Data Stewards clarify the meaning, rules, and responsibilities of the data.

Enterprise Data Architecture

The broadest level of data architecture is Enterprise Data Architecture, the organization-wide view of how data should be organized, connected, and governed.

At a university, Enterprise Architecture provides a comprehensive view of how systems should be coordinated, what each should do, and how information is exchanged between them.

For example, the web application through which a candidate completes a process must be properly connected with an admissions system or a database where that information is stored. This database or system can also support the operation of other internal systems dedicated to analyzing that data, or parts of it, according to privacy policies.

This work is led by the Enterprise Data Architect with support from the CDO, who aligns the architecture with the data strategy, and other roles like Application Architects or Security Architects. Additionally, Data Owners validate that the architecture meets the needs of their domains.

Data Domains

Data domains are an important part of an organization's architecture. Not all data describes the same part of the business, so teams group related concepts to make the data easier to organize, understand, and govern.

A Data Domain is a logical area containing related organizational concepts and data. A university might define domains for students, faculty, finance, and mobility. Grouping data this way makes its meaning clearer and helps the organization assign a Data Owner to each domain.

Additionally, a domain isn't isolated from others, as data often needs to be contextualized, even if it belongs to different domains. For example, the transportation service may require data from the mobility domain, as well as the schedule of its courses present in another domain.

Each governed domain should have a Data Owner with suitable decision authority. A Data Architect helps design the domain boundaries and relationships, which teams can represent in a conceptual model and document in a data catalog.

Data Flows

Once the domains and systems are clear, the team designs how data moves between them. Data Flows document the source, the systems and processes involved, the transformations applied, and the final storage or consumption point.

You can describe a flow at several levels. A high-level diagram may show data moving from one domain to another. An implementation view names the systems involved, while a more detailed design can show the fields, interfaces, and transformations that each consumer requires.

In the process of enrolling a candidate at the university, the main flow could be as follows:

  1. The candidate accesses the enrollment portal and completes the form with their personal, academic, and contact information.

  2. The enrollment portal validates the required fields and data format. Then, it sends the application to the admissions system via an API.

  3. The admissions system creates the candidate's file and stores documents like the ID, academic degree, and certificates in a document database.

  4. When the application is approved, the admissions system generates an offer that the candidate views and accepts through the enrollment portal.

  5. The portal consults the academic management system to display courses, schedules, and available slots, allowing the candidate to select their options and confirm enrollment.

  6. The payment system sends the transaction to an external payment gateway. The gateway returns the payment status, such as authorized, rejected, or pending. The university stores only a reference to the transaction and its result.

  7. If the payment is successful, the academic management system creates the final enrollment and converts the candidate's file into a student file.

  8. Next, the system updates the identity platform, virtual campus, and billing system. The student receives their credentials, payment receipt, and enrollment confirmation.

  9. Finally, the necessary data can be pseudonymized and sent via a data pipeline to an analytics platform, where statistics on applications, admissions, payments, and enrollments are calculated and displayed on a dashboard.

Some data movements need near-real-time responses, especially in the transportation service, while others can run later in a batch. The flow should state those timing requirements.

The main role that designs the flow and determines which components participate is the Data Architect, while the Data Engineer implements it. But Security Architects also participate, reviewing data protection during the flow, and Data Owners authorize exchanges between domains. Finally, it's important to highlight the significance of data lineage tools for maintaining, monitoring, and auditing the flows.

Operational Data Architecture

The systems in an architecture serve different purposes. It's useful to distinguish between systems that run day-to-day processes and systems designed mainly for analysis.

The first group forms the Operational Data Architecture. This area covers the systems that keep an organization running each day. Online Transactional Processing (OLTP) systems handle frequent operational transactions and use controls that help preserve data integrity and consistency.

The university's operational architecture could include the virtual campus, application services, and a database. The portal would normally use an application or service layer rather than giving the user's browser direct database access. These components support the daily capture and management of data rather than long-running historical analysis.

That is, the operational database can serve as an authorized source to know the current status of enrollments, for example. However, it is not the most suitable place to continuously run complex queries over several years of activity to build statistics, as they could consume the resources needed for daily operations. Therefore, the data required to study trends, compare programs, or create dashboards is handled in another part of the architecture explained later.

For this type of information, it's common to use relational databases like PostgreSQL or MySQL. But you should choose the specific technology based on the volume of your operations, expected availability, existing infrastructure, and other requirements such as maximum response latency.

A Solution Architect or Data Architect designs the operational architecture, Software Engineers build the application components, and Data Engineers help define and implement the data exchanges between them.

Analytical Data Architecture

While operational architecture handles day-to-day activity, Analytical Data Architecture supports the integration, aggregation, and study of historical data. Its systems help teams create reports, discover patterns, and prepare data for AI models without placing unnecessary analytical load on operational services.

At a university, this architecture would be used to combine data on schedules, attendance, and budgets so an analyst can calculate the monthly expenses per master's program or the variation in student attendance over different periods. Similarly, a Data Scientist could use historical data to estimate future demand for transportation services, for example.

A typical analytical flow uses ETL or ELT (which we'll discuss more below) to obtain data from several sources. Teams then transform it before or after loading it into a specialized system such as a Data Warehouse. The result gives Business Intelligence tools and Machine Learning workflows suitable data without competing directly with the virtual campus for the same operational resources.

In this area, the Data Architect or Analytics Architect designs the analytical components of an architecture. Meanwhile, Analytics Engineers and Data Engineers design the processes that prepare data for analysis by Data Analysts or Data Scientists.

Cloud and Hybrid Data Architectures

Architecture also determines where components run: in the cloud, on premises, or across both. Cloud Data Architecture uses cloud computing, storage, database, and analytics services. These services can simplify scaling and reduce the need to manage physical hardware, but the organization still has to configure security, control costs, and govern its data.

On the other hand, a Hybrid Data Architecture combines on-premises systems with cloud services. This approach is common when an organization retains existing applications in its own data center but wants to use the cloud's elasticity or analytical services.

To understand the motivation for a hybrid architecture, in the case of the university, the academic system and the database with records and payments might initially remain in internal infrastructure to prevent third-party access to those data. But some pseudonymized data could be sent to cloud analytics platforms to obtain certain statistics on virtual campus usage or academic metrics.

Nevertheless, keeping certain data on-premises doesn't automatically guarantee greater security, just as using the cloud doesn't automatically mean a loss of control. The decision should consider data sensitivity, latency, availability, scalability, and the total cost of each solution.

In this design, the Enterprise Data Architect and the Data Architect participate, along with the Cloud Architect, who specializes in understanding cloud services to use them correctly in an architecture.

Network Engineers, Cloud Engineers, and Data Engineers also participate in its implementation, while the DPO and Data Owners must review issues like which data can leave the internal infrastructure and for what purpose.

Data Modeling and Design

Data architecture defines which systems manage data and how they exchange it. Data Modeling and Design specifies how those systems represent the information. It identifies the concepts that matter to the organization and describes their attributes, relationships, and rules.

A data model is a simplified representation of part of reality. It gives people a shared structure they can understand and later implement. Before creating the university's database, for example, the team needs to define what a candidate, student, master's program, and enrollment mean, which information each one needs, and how they relate.

Teams commonly describe a design at three levels:

  1. a conceptual model with the main business concepts,

  2. a logical model that adds detail without depending on a particular technology,

  3. and a physical model that maps the design to structures in a specific platform.

Each model can evolve as the team learns more about the requirements.

The Data Modeler leads the design and works with the Data Architect to fit it into the wider architecture. Data Owners, Data Stewards, Business Analysts, and domain experts clarify meaning and rules. Database Administrators (DBAs), Data Engineers, and Software Engineers contribute to the physical design and implementation.

Conceptual Data Models

A conceptual data model gives you a high-level view of an organization's data. It shows the main business concepts and their relationships without technical details about storage or format.

Example of conceptual data model. Image by author.

For example, as shown in the diagram above, in a university, a conceptual model would include concepts like candidate, student, course, or enrollment. (Keep in mind that this is a sketch to help you better understand the concept of a conceptual model, not a diagram used in production.)

At this level, it's sufficient to indicate what each of these concepts is and what they can do in relation to others, such as a student requesting enrollment or an enrollment containing a set of courses. The goal is for both technical teams and academic leaders to understand the same reality before designing a specific solution.

This model is usually developed through interviews or workshops with Data Owners, Data Stewards, Business Analysts, and domain experts, who are generally not very technical given the nature of the task. In this process, the Data Modeler or Data Architect creates diagrams with the model and validates that the concepts match the business glossary.

Logical Data Models

A logical model develops the conceptual model in more detail while remaining independent of a specific technology. It defines entities, attributes, identifiers, relationships, cardinalities, and other business constraints.

Example of logical data model. Image by author.

For example, the Student entity might have attributes like ID, name, email, and address. A student can enroll in several courses, and a course can have many students. This many-to-many relationship could be represented at the logical level with an intermediate entity called Enrollment, which might include attributes like date, status, or academic year.

People often associate logical models with relational databases, but a logical model doesn't have to use that paradigm. Think of it as a technology-independent specification of the information and its connections, even though different paradigms represent entities and relationships in different ways.

These relational models can be refined. For example, in a relational database, its logical model can be normalized to reduce duplications and incorrect dependencies. But in other paradigms or solutions, there will be very different procedures. And the design of this model is led by a Data Modeler, in collaboration with a Data Architect, as mentioned earlier.

Physical Data Models

The physical data model maps the logical design to a specific technology. In a relational database, for example, it turns logical entities and relationships into tables, columns, keys, constraints, partitions, and lower-level structures such as indexes, which often use a B-tree.

At the university, student records could live in a relational table. The DBMS decides how to store the table itself, while the team can create indexes, often B-tree indexes, on selected columns to speed up common queries.

As you can imagine, the same logical model can generate different physical models. For instance, the academic system could be implemented in PostgreSQL or MySQL. So the physical design must consider the DBMS intended for use, data volume, query patterns, security, availability, and operational cost to provide an effective solution.

In this design phase, the Data Modeler or Database Designer, the Data Architect, and the Data Engineers primarily work together with the Software Engineers to implement the solution.

Entity-Relationship Modeling

Entity-relationship diagrams are a common way to represent relational concepts. Depending on how much detail they contain, they can support conceptual or logical modeling. Entities are typically shown as rectangles, while lines represent relationships, cardinality, and optionality.

For example, a student can have many enrollments, and each enrollment belongs to a single student. In contrast, a relationship between Student and Course would be many-to-many because a student can be enrolled in many courses at once.

Keys are also identified to distinguish each instance of an entity and maintain the integrity of their relationships, among other details that aren't as relevant here.

If you're curious, you can read more about database design in my previous book here.

Dimensional Modeling

Another useful approach, especially in Data Warehouses and analytical systems, is dimensional models. These models organize data around facts and dimensions. Facts record measurable events, while dimensions provide the context used to analyze them.

For example, in a transportation service, you might have a fact table called Trip, containing a row for each completed journey, recording measures such as cost, distance, and duration. But instead of storing the traveler's data in the same table, it relates to others representing dimensions like Student, Date, or Transportation Provider. Thus, the fact table models the existence of trips, while other dimensional tables contain specific data for each trip, such as the person or transportation provider, resulting in a structure known as a star schema.

This type of model is primarily used because it simplifies analytical queries and allows studying the same fact from different "perspectives." For example, the university could calculate the total cost of trips by month, student, or provider without having to construct excessively complex queries.

Data Model Governance

Data models also need governance so they stay consistent, current, and aligned with the implementation. Teams should maintain the connection between conceptual, logical, and physical designs as each one changes.

Once teams approve a model, the implementation should follow it or update it through a controlled change. Unexpected differences between an expected and an actual schema are commonly called schema drift.

For example, the university's model might define a numeric age field while the implementation stores it as text. That difference may look small, but downstream systems can fail if they rely on the agreed type. Teams should detect and control schema changes so models, contracts, and implementations stay aligned.

A Data Governance Council or Architecture Review Board may review significant model changes. Data Owners confirm that the design reflects business rules, while database and engineering teams implement approved changes through a controlled process.

Data Storage and Operations

Data models guide the implementation of systems that store data persistently and make it available to applications and other systems. The team has to choose an appropriate storage technology, keep the data accessible when needed, and operate the system at an acceptable level of performance.

Data Storage and Operations covers the design, implementation, and operation of storage systems throughout their lifecycle. This includes choosing databases, file systems, and object stores, then maintaining, monitoring, and optimizing them. As you'll see, a database isn't the right home for every type of data.

Data architecture determines which systems the organization needs and how they communicate. Data modeling specifies how they represent information. Data Storage and Operations turns those designs into working storage systems. A physical model might say that the Student entity maps to a PostgreSQL table with a B-tree index on student_id. This section focuses on implementing and operating that kind of design.

The main objectives of data storage are to maintain availability, integrity, and ensure good performance of the underlying system. To achieve these, you shouldn't always use one technology for all the data in an organization, as the data for an enrollment or a class video, for example, has very different structures, uses, and requirements. So the same organization often combines different storage systems.

Need Example data Most common system Example technologies
Record the current state of operations Students, enrollments, payments, and transportation requests Operational database PostgreSQL, MySQL, SQL Server, Oracle Database, or MongoDB
Store large documents and content Academic certificates, supporting documents, materials, and videos File Storage or Object Storage NFS, SMB, Amazon S3, Azure Blob Storage, Google Cloud Storage, or MinIO
Analyze integrated and historical information Monthly travel costs and attendance trends Data Warehouse Snowflake, BigQuery, Amazon Redshift, Azure Synapse Analytics, or Teradata
Store data for advanced analytics Original provider files, events, and virtual campus logs Data Lake or Lakehouse Object Storage, Parquet, Delta Lake, Apache Iceberg, Spark, or Trino

The team should choose the technology based on its expected volume, access patterns, sensitivity, availability, cost, and other requirements. Every additional technology increases operational complexity, so each one should solve a real problem.

A Database Administrator creates, configures, secures, tunes, and maintains databases. Storage Administrators manage the underlying storage, while Site Reliability Engineers and platform teams monitor services and respond to reliability incidents. The exact division of work depends on the platform and organization.

Databases

A database is an organized collection of data that applications can store, change, and query. A Database Management System (DBMS) is the software that manages databases and provides services for querying, concurrency, security, recovery, and administration. PostgreSQL is a DBMS. The university's academic database would be a particular database managed by a PostgreSQL server or service.

Operational systems often need transactional support, especially for workflows such as enrollment and payment. A transaction groups related operations into one logical unit. The ACID properties (Atomicity, Consistency, Isolation, and Durability) describe guarantees that help applications preserve valid state despite failures and concurrent access.

For example, when making a payment, values must be modified in multiple places corresponding to the users exchanging money. Thus, the atomicity of a transaction allows confirming all these modifications together, and if any fail, reverting them to maintain the previous state.

Database designs make different tradeoffs among data model, scale, consistency, latency, and access patterns. That's why several database paradigms exist:

Paradigm Characteristics Use case example Technologies
Relational Organizes data into related tables, uses predefined schemas, and supports keys, constraints, and transactions Managing students, courses, enrollments, invoices, and transportation requests, where relationships and integrity are important PostgreSQL, MySQL, SQL Server, or Oracle Database
Document-oriented Groups information into documents, usually similar to JSON, which may contain nested structures and evolve more flexibly Storing forms from multiple providers when they don't all submit exactly the same fields MongoDB or Couchbase
Key-value Retrieves a value through a unique key and prioritizes simple, fast access patterns Maintaining portal sessions, temporary results, or a cache of frequent queries Redis or Amazon DynamoDB
Graph-oriented Represents data through nodes and relationships, enabling complex connections to be traversed efficiently Analyzing relationships among students, courses, lecturers, transportation routes, or dependencies between services Neo4j, Amazon Neptune, or ArangoDB

These are only a few database paradigms. A university could use PostgreSQL for an academic system that manages Student, Enrollment, and Course records through tables and relationships. For a specialized route or network analysis, a graph-oriented database could represent locations as nodes and connections as edges. The operational taxi service itself might still use a relational or other transactional store, depending on its access patterns.

The Data Architect and Data Modeler select the database paradigm and design with input from the engineers who will build and operate the solution.

Once operational, the database is maintained by a Database Administrator. Before this, a Database Engineer will have implemented the physical model, created instances, schemas, tables, and other necessary elements to subsequently operate the environment. Software Engineers develop the applications that access these databases and perform queries.

File and Object Storage

Not all data fits naturally in a database. Universities manage diplomas, identity documents, and large files such as class recordings. A DBMS can store binary content, but file or object storage often provides more suitable access, scale, and cost characteristics for these assets.

File Storage organizes files into directories and exposes them through paths and protocols such as NFS or SMB. Teams can implement it with a Network Attached Storage (NAS) system or a cloud service such as Amazon EFS or Azure Files.

Object Storage stores content as objects with identifiers and metadata, usually inside buckets or containers. Its namespace and access model differ from a mounted hierarchical file system, even when tools display folder-like prefixes. Services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage can hold large collections of documents, images, and videos.

The main difference is the access model. File Storage behaves like a shared file system, while applications usually access Object Storage through an API using an object key and metadata.

For example, the university could use File Storage to save administrative documents for each student, like registrations and certificates, in a shared folder. This way, authorized staff could manage them as if they were in a traditional file system.

On the other hand, it could use Object Storage to store a large number of class recordings, images, and multimedia materials in a bucket. Instead of locating a video by navigating folders, the system could retrieve it directly using its identifier or by filtering through its metadata.

The roles responsible for configuring and operating these systems are mainly Storage Administrators, Cloud Engineers, and Platform Engineers, while Software Engineers implement access to these systems from other applications.

Data Warehouses

Operational databases are usually optimized for current transactions and application queries rather than repeated analysis across years of integrated history. Complex analytical workloads can also compete with the applications using the same resources. Organizations therefore often copy suitable data into a separate Data Warehouse.

A Data Warehouse is an analytical repository that integrates data from multiple sources and organizes it for repeatable analysis, reports, and dashboards. These systems support Online Analytical Processing (OLAP) workloads that scan and aggregate many records, in contrast with the Online Transactional Processing (OLTP) workloads common in operational applications.

This difference often affects storage design. Many Data Warehouses use columnar storage because an analytical query may scan a few columns across a large number of rows. To calculate the total cost of taxi rides by date, for example, the engine may only need the cost and date columns.

Many operational relational databases use row-oriented storage because it efficiently retrieves or changes complete records. These are common patterns rather than universal rules. Specific products can support several storage formats.

In practice, the university could have a database and a pipeline where data is periodically extracted to be inserted into a Data Warehouse. There, a dimensional data model could be applied as seen earlier to analyze the data and allow an analyst to answer questions like:

  • What's the average monthly cost of a certain course per student?

  • How has in-person attendance changed over a specific period?

  • How many students enrolled last month?

It's important to understand that a Data Warehouse doesn't replace a database. Rather, it's an auxiliary system focused on data analysis. Among the technologies available for these types of systems are cloud platforms like Snowflake, Google BigQuery, or Amazon Redshift.

The roles that work with them include Data Architects or Analytics Architects, who design the analytical platform, while Data Engineers design the pipelines to extract and load the data.

Data Warehouse Administrators or Platform Engineers manage performance, permissions, reliability, and cost. Data Analysts and Business Intelligence professionals query the governed analytical data without changing the operational source records.

Data Lakes and Lakehouses

A traditional Data Warehouse applies defined schemas and organizes data for known or anticipated analytical needs.

But this isn't always the case, as an organization might also need to retain original files, semi-structured data, logs, images, or events whose future use isn't yet fully defined.

For these situations, we can use a Data Lake, which is a repository designed to store large amounts of data in their original formats or with minimal transformations.

A Data Lake also supports analytical and data-processing needs, but it can retain structured, semi-structured, and unstructured data with fewer transformations at ingestion. It's often associated with schema-on-read, where a query or processing job applies part of the structure, while a traditional Data Warehouse commonly uses schema-on-write before loading curated data.

Schema-on-read doesn't remove the need for metadata, security, quality, and governance. Without them, the lake can become a data swamp.

To understand how information is organized in a Data Lake, in the university's use case, the data could be processed in layers according to their readiness for consumption.

  1. In a specific area of the system, data could be kept in their original formats without modification, such as CSV or JSON files. This would allow for reprocessing the information if an error in a transformation is detected later or if another type of analysis is needed.

  2. In another area, the data could be in a different format, or the same format but with certain transformations applied to remove invalid records or standardize units of measure, for example.

  3. In a curated area, teams could apply further quality checks and transformations until the data meets the requirements for dashboards, with selected statistics pre-calculated.

This separation doesn't imply that all original data is always retained indefinitely, as privacy, security, and retention policies must be followed.

For example, the university may temporarily store documents submitted by a candidate during the admission process. But if the candidate is rejected and enough time has passed, the university must delete those documents, even if derived and anonymized data have been generated to compile statistics on the admission process.

Lakehouses add capabilities such as transactions, schema enforcement, and table management to the flexible storage commonly used for a Data Lake. They can let several analytical workloads share one data foundation, although they don't eliminate every reason to use specialized systems.

Among the technologies used to build a Lakehouse are Delta Lake, Apache Iceberg, and Apache Hudi. They define the data format usually stored on services like Amazon S3, Azure Blob Storage, or Google Cloud Storage and processed using tools like Apache Spark, Databricks, or Trino.

In the case of the university, a Lakehouse could be used to store student data, enrollments, attendance, and taxi rides in one place. This way, the university could securely update this data and use it directly to create reports, such as monthly transportation expenses or the number of students attending classes, without needing separate systems.

Finally, those responsible for designing and implementing data ingestion from different sources in these systems are the Data Engineers. On the other hand, Platform Engineers manage the infrastructure, and Analytics Engineers, along with Data Scientists, consume the data to conduct relevant analyses and research.

Backup and Recovery

Even a well-designed storage system can suffer hardware failures, software defects, corruption, mistakes, or attacks that cause data loss. That's why Backup and Recovery is essential in production.

A backup is a recoverable copy of data kept for loss or corruption scenarios. A backup is useful only if the organization protects it, verifies it, and tests the recovery process. Common mechanisms include:

  • Full backup: Copies the entire dataset. For example, the university could perform a complete weekly copy of the enrollment database. It simplifies restoration, though it requires more time and storage.

  • Incremental backup: Saves only the changes made since a previous copy. After a monthly full backup, only the modified enrollments could be copied daily. It reduces volume, but recovery may require several linked copies.

  • Snapshot: Captures the state of a storage system at a point in time. Depending on the technology, it may share underlying storage and may not be an independent copy. The university could take one before a major academic-system change, while still keeping separate backups for stronger protection.

  • Log backup: A backup that relies on a change log, allowing recovery of the database to a previous point in time if data is accidentally deleted. It's more precise but requires maintaining the entire log sequence.

  • Replication: Maintains a replica of an entire system that can take over if the main system fails. For example, a secondary database could continue serving the enrollment portal, improving availability. But it can also replicate deletions or errors, so it doesn't replace a backup.

A recovery strategy uses two common objectives. The Recovery Point Objective (RPO) expresses the maximum tolerable data loss in time, while the Recovery Time Objective (RTO) states how long service restoration may take before the impact becomes unacceptable.

For example, the university might hypothetically set an RPO of five minutes and an RTO of one hour for the enrollment database during the registration period. This would mean that, in the event of a serious failure, they aim to lose a maximum of five minutes of operations and restore service within an hour. In contrast, a collection of already published videos might allow for a slower recovery if durable copies exist elsewhere.

A well-known practice in designing backup solutions is the 3-2-1 rule, which involves maintaining three copies of important information, using at least two storage media or technologies, and keeping one copy offsite. But you should tailor your solution to the requirements of your organization.

The Data Owners and business leaders are responsible for identifying critical processes and determining acceptable loss or interruption. On a technical level, a DBA implements and validates the database recovery mechanisms. Additionally, Storage Administrators and Cloud or Platform Engineers manage storage and automate backups, while Site Reliability Engineers monitor and conduct tests to ensure recovery functions as expected.

Retention and Archiving

An organization shouldn't keep every piece of data indefinitely. Doing so raises costs, complicates discovery, and increases the impact of a breach.

A retention policy should state how long data stays active, when it moves to an archive, and when it is deleted or anonymized. The policy should reflect business needs, contractual duties, legal requirements, and applicable holds.

In this context, it's important to distinguish between two concepts:

  • Archive: Stores information that's no longer regularly used but must remain accessible. For example, a former student's record might be moved to an archive with lower storage and retrieval costs, in case it's needed to verify their existence when requesting a certificate.

  • Retention: Defines how long data is kept and what happens when that period ends. For example, the personal and academic documentation of a rejected applicant might be retained until the admission process and the appeal period are over. Afterward, those documents would be deleted, although the university might keep anonymous statistics on the number of applications received.

Data Owners, Records Managers, legal counsel, and privacy specialists help establish retention periods. A legal hold can temporarily suspend normal disposal for information related to an investigation or proceeding. The organization therefore needs a documented reason to keep or delete data rather than deciding only by whether it seems useful.

Afterward, Data Stewards classify the data, and DBAs, Storage Administrators, or Cloud Engineers implement the policies. As an interesting technology, Write Once Read Many (WORM) storage is often used for records that must remain unalterable.

Performance and Availability

Stored and protected information must be available when the service needs it and perform within its agreed targets. Performance describes qualities such as response time and throughput, while availability measures whether the expected service can be used.

A system can be technically running yet unusable if it responds too slowly. It can also be fast when online but fail its availability target because of frequent outages. Teams need to manage both qualities.

Some techniques that can improve the performance of a storage system include:

  • Create indexes on frequently queried fields, after ensuring they justify the space cost of the index itself.

  • Analyze the most frequent queries or workloads to try to optimize the query plans generated by the DBMS.

  • Introduce caches whenever possible, especially when results will be needed multiple times.

Performance work depends on the system and workload. Adding hardware won't fix every problem, as software design matters just as much. Unnecessary pipeline transformations, for instance, increase execution time and cost even when they don't cause an outage.

On the other hand, redundancy is often used to improve availability. Essentially, if there are replicas of the same server or system, it's less likely that all will fail simultaneously, leaving end users without service.

You can manage the existence of replicas with failover mechanisms, so if a PostgreSQL instance, for example, stops working, you can redirect traffic to another replica automatically and transparently for the end user.

In the university example, during the last days of the enrollment period, thousands of students might access the portal simultaneously. To maintain good performance, requests would be distributed among several servers, preventing any single one from becoming overloaded and reducing wait times. Also, the database could have replicas so that if one instance fails, another can automatically take over.

This way, the system would remain fast during high demand and stay available even in the event of an unexpected failure.

To measure an organization's performance and availability objectives, observability is especially important. This involves generating metrics, logs, and statistics, and managing them with tools like Prometheus and Grafana to monitor the system and check its availability and performance at any given time.

This analysis and optimization of a storage system is usually performed by the DBA, although certain Software Engineers and Data Engineers may also be involved, optimizing the data pipelines through which various systems exchange information. Regarding availability, SREs, Platform Engineers, and Cloud Engineers automate deployments, monitoring, scaling, and implement failover mechanisms.

Document and Content Management

So far, we've worked with several kinds of data: structured records in tables, semi-structured data such as JSON, and unstructured content such as scans, images, videos, and free-form text.

Documents can contain a mix of structured metadata and unstructured content, so they need their own management practices.

A document usually doesn't follow a rigid row-and-column structure, but it can still have metadata such as a title, author, type, date, or tags. Some digital formats also contain an internal hierarchy. A JSON document, for example, uses named fields and nested objects:

{
  "student_id": "ALU-2026-8942",
  "full_name": "Amélie Dubois",
  "master_program": "Master in Artificial Intelligence",
  "campus_distance_km": 18.2,
  "rideshare_benefit_approved": true,
  "last_trip": {
    "date": "2026-03-09",
    "cost_euros": 24.50
  }
}

Many digital files combine content with descriptive metadata such as a title, author, or creation date. That metadata makes the content easier to identify, organize, secure, and retrieve. Document and Content Management provides the processes and systems for doing this consistently.

Simply placing files in folders isn't enough at organizational scale. Teams need ways to classify documents, describe their content, control access, track versions and retention, and find them later. A basic file system or database can be part of the solution, but a document or content platform adds the management features the organization needs.

For example, the journey of a document in the university systems might be:

  1. The candidate's academic record is captured from a form, an email, or any equivalent means.

  2. It's indexed and metadata is added to provide context.

  3. It's stored in an appropriate repository.

  4. Authorized users and systems can access or share it under the applicable controls. For example, an admissions analyst might query approved extracted fields to count candidates with prior study in a subject area without opening every certificate manually.

  5. Finally, it's deleted or retained according to applicable policies.

Unstructured Data

An important part of the data managed by an organization contains unstructured information. This means that, as mentioned above, its content isn't rigidly structured in clearly identifiable and directly queryable fields. For example, a motivation letter in PDF, a scanned image of a diploma, or a contract may contain information that's difficult to structure.

Documents may have format-specific metadata such as a title or creation date. This helps identify the file but rarely describes everything inside it. The body may contain free-form text, images, tables, or other content that the system must extract or index before it can answer detailed queries.

To perform queries on this information, the system indexes this content or applies techniques like Optical Character Recognition (OCR), Natural Language Processing, or Intelligent Document Processing.

For example, if the university wants to know how many candidates have taken math-related courses before entering the master's program, it must first extract that information from academic certificates, normalize it, and store it in queryable fields. When extracting data from a document, you should maintain a link to the original document to verify its source later.

After extracting useful content, the system can index it in a structure optimized for search. The index may represent a document with fields or key-value pairs such as the candidate identifier, document type, courses taken, and subject area.

{
  "index_id": "idx_cert_2026_0042",
  "student_id": "ALU-2026-8942",
  "student_name": "Amélie Dubois",
  "document_type": "Academic Transcript",
  "extracted_subjects": [
    {
      "original_name": "Algèbre Linéaire",
      "normalized_area": "Mathematics",
      "score": "18/20"
    },
    {
      "original_name": "Introduction à Python",
      "normalized_area": "Computer Science",
      "score": "16/20"
    }
  ],
  "metadata": {
    "issuing_country": "France",
    "language": "fr",
    "confidence_score_ocr": 0.98
  },
  "original_file_url": "https://s3.uni.edu/bucket-cert/2026/8942_transcript.pdf"
}

For example, above you can see what an indexed document might look like. Originally, it could be an academic certificate of a candidate, but for the system, it's a JSON dictionary with this information, meaning the internal content of the document is organized hierarchically.

Representing it this way makes it much easier to perform queries, as you can navigate and access fields like score to see each candidate's grades in the various subjects they've taken at another university.

Document Capture

The first operational step is Document Capture, the controlled process for accepting a document into the organization's systems.

In these processes, it's important to consider the format of the document to be captured, as they're not always digital files. Often, they can be physical documents delivered to an administrative body, which then needs to digitize and upload them to the system.

In any case, assuming a digitized document reaches the data management systems, an adequate capture should perform at least the following actions:

  • Validate the file format and size, and ensure it doesn't contain malicious software.

  • Assign it an identifier and basic metadata, such as its origin and date of receipt, along with a digital fingerprint like a hash to detect changes in the file.

  • Preserve the original and, when necessary, extract a usable representation of its content.

If a document is scanned, its text appears as pixels rather than directly searchable characters. OCR converts visible text into machine-readable text. More advanced Intelligent Document Processing (IDP) systems can also classify documents and extract fields, tables, and layout using rules and Machine Learning models.

For example, a candidate might upload a photo of a diploma issued in another language from their phone. The capture process would detect the language, extract all the corresponding text using OCR, and associate the file with their application so that the document's content can later be reviewed, knowing to whom it belongs.

Document Classification

After capture, the system may need to classify the document so it knows what it is and which workflow, access rules, and retention policy apply. People can do this manually, or software can assist with rules and Machine Learning.

In some workflows, the university may let users attach certificates, reports, and other supporting files. The system can't trust the filename or assume that every upload is safe. It must validate the file, scan it according to security policy, and identify the document type before further processing.

A filename alone isn't reliable: A.pdf could contain almost anything. Classification assigns one of the organization's defined document types and determines the next processing steps. Teams may automate low-risk cases and route uncertain or consequential cases to a person for review.

Content Storage

After capture and classification, the organization stores the original document and its metadata. Object or file storage often holds the binary file, while a document database such as MongoDB, Couchbase, or Amazon DocumentDB may hold flexible metadata or extracted content. The right combination depends on access, retention, search, and scale requirements.

Other alternatives include using a Document Management System (DMS) or a platform with Enterprise Content Management (ECM) capabilities. These document repositories are based on File or Object Storage internally, with additional capabilities that a bucket or folder alone cannot provide, such as advanced metadata management. Lastly, it's worth mentioning the existence of Content Management Systems (CMS), which are designed for creating and publishing content on websites.

Search and Retrieval

A document is useful only if authorized users and systems can find it when needed. After storage and indexing, the platform may support several search methods:

  • Metadata search: Filters by fields such as document_type = Academic Certificate.

  • Full-text search: Finds words or phrases in extracted text and ranks the matching documents.

  • Semantic search: Retrieves documents by meaning, even when they don't contain the exact words in the query.

For example, an authorized employee could search for a certain teacher's employment contract using keywords like "contract" or the person's name, even if they don't remember the exact file name. Alternatively, with a semantic search like the one we can perform on Google, they can also locate that document or any other based on the meaning of its content.

Records Management

Not every document has the same value or lifecycle. Teams may discard drafts quickly, while official evidence of an activity or decision must be preserved as a record. Records Management controls those records throughout their required lifecycle.

Unlike a draft, a record is an official document that must be preserved and kept authentic, complete, and protected. For example, a draft of an admission offer would be disposable, while the accepted and signed offer by the student becomes a record.

Each type of record has an associated retention period that determines how long it must be kept and what should be done afterward. If there's an investigation or legal proceeding, a legal hold may be applied, temporarily suspending its disposal. At a university, official course records or final academic transcripts might be considered records.

Overall, the most common technologies and roles in document management can be summarized as:

Document Phase Key Technologies Roles
Capture Azure AI Document Intelligence, Google Document AI, Amazon Textract, Tesseract OCR Software and Integration Engineers implement the capture pipeline, while ML Engineers design the data extraction models.
Storage OpenText Content Management, MongoDB Information Architects design the logical content structure, while Platform Engineers and ECM/DMS Admins implement and operate the storage systems.
Indexing and Search Elasticsearch, OpenSearch, Apache Solr Information Architects design the indexing strategy, while Search and Software Engineers implement the search engines and queries.
Retention and Maintenance Microsoft Purview Records Management, Amazon S3 Object Lock Records Managers, Data Owners, and the DPO define the policies, rules, and compliance requirements, while Security and Compliance Teams implement security mechanisms and conduct audits.

Reference and Master Data Management

Organizations reuse some data across many processes and systems. The same student may appear in the admissions platform, virtual campus, and billing platform. If each system represents that person differently, duplicates and contradictions quickly appear.

Reference and Master Data Management coordinates this shared data so systems can use consistent, trusted values.

First, you need to distinguish between:

  • Master Data: This describes an entity that is relevant and shared by several processes. For example, the record of the student Amélie Dubois.

  • Reference Data: These are allowed values within a classification or organization of the master data. For example, APPROVED can represent the status of an accepted enrollment application, with the candidate's record considered master data.

The goal isn't to force every piece of data into one database. It's to identify trusted values and systems of record, define who maintains them, and distribute the right representation to each consumer.

Master Data

Master Data represents core entities such as people, organizations, places, or products. At a university, it might include students, faculty, and courses. A trusted student record could contain a global identifier, name, and selected contact attributes, while sensitive payment details remain in the systems that need them.

But payment information won't be used in all processes involving these data. This is why authorized data needs to be distributed to each system so that the entire organization has a consistent view of the data, even if it's used differently.

Not all attributes of a record have to come from the same place. A payment platform may maintain its fiscal information, while the student portal keeps the most recent contact email. Then, a Master Data Management (MDM) platform would integrate these sources to provide a reliable view to other systems.

Platforms used for this purpose include Reltio, SAP Master Data Governance, and IBM InfoSphere MDM. The role that operates them is the MDM or Data Architect, who defines the data model and the architecture used for deployment, while the MDM Engineer configures the platform. Data Engineers and Integration Engineers need to be aware of these authorized sources of truth.

Reference Data

Reference Data supplies controlled values used to classify or organize other data. The university might allow a transportation request to have the status PENDING, APPROVED, or REJECTED. If applications use different terms for the same state, integration and reporting become unreliable. These approved status values are Reference Data.

These values usually change infrequently but aren't immutable. This can happen because new values need to be added to the classification, like CANCELLED.

To make this modification, a Data Steward would document its meaning, while the Data Owner of the corresponding data domain approves the change. Subsequently, the Integration Engineers are responsible for distributing the new value to the systems that consume it.

Golden Records

Information about one entity often appears in several systems, with each system storing what it needs. An MDM platform can combine selected trusted attributes into a unified view called a Golden Record. The goal is a governed, useful representation, not a copy of every piece of information the organization holds.

For example, the university might have an admissions system where a student's personal data, like the name Amelie Dubois, is stored, while their payment information is in a system specialized for processing payments. After verifying they belong to the correct person, they can be linked to provide a single view of the student.

A Golden Record isn't automatically perfect or permanently definitive. It's the best trusted view available under the current matching and survivorship rules.

Entity Resolution

To build that view, the platform must decide which records refer to the same real-world entity. This task is called Entity Resolution.

For example, records named Amélie Dubois and A. Dubois might refer to the same person, or to different people. A resolution process can compare authorized attributes such as email, phone number, or date of birth and apply deterministic rules or probabilistic matching. Because false matches and missed matches can cause harm, teams should review uncertain cases and provide a way to correct decisions.

This is assigned to the MDM Engineer, while the Data Quality Analyst analyzes and supervises the results along with a Data Steward. It's implemented through the functionalities incorporated in MDM platforms, services like AWS Entity Resolution, or record linkage libraries like Splink.

Deduplication

Another issue that drives the need for Entity Resolution is the presence of duplicate data. For example, a candidate might register on the virtual campus with one email and later apply for admission using another. If it's confirmed that both records belong to the same person, they should be handled appropriately in each specific scenario.

This process is called Deduplication and involves using Entity Resolution to detect and manage repeated records, aiming to prevent them from being treated as independent entities. Common approaches include linking, which retains the records in their original systems and creates a correspondence between their identifiers. Alternatively, merging generates a consolidated record, similar to the Golden Record.

Here, responsibilities are divided among several roles. The Data Owner sets the criteria guiding the Deduplication process, the MDM Engineer implements these criteria on the platform, and the Data Quality Analyst, along with the Data Steward, supervises the outcome of the process.

Survivorship Rules

When several source records refer to the same entity, the MDM process must decide which value to use for each attribute in the Golden Record.

Previously, we saw this with the example of the student name Amélie Dubois and A. Dubois, values that may appear in several records. Thus, when creating a Golden Record, it will be necessary to decide which one to keep.

For this, there are Survivorship Rules, which, as their name suggests, are rules that determine the resolution of these situations based on the data involved.

These criteria are designed by a Data Owner, while a Data Steward supervises the process and its application, and an MDM Engineer implements these rules on a platform.

Metadata Management

In the previous section, document metadata helped identify us a file and describe details such as its type or creation date. But metadata applies far beyond documents.

Metadata is data that describes other data. The number 42 is ambiguous by itself. A column name such as age, a unit, a definition, and a timestamp can tell you what it represents and how to interpret it.

At organizational scale, metadata needs deliberate management of its own. Metadata Management collects, connects, maintains, and publishes metadata so people and systems can find and use data correctly.

The goal is to make data understandable and support governance, quality, security, and discovery. People create some metadata manually, while scanners and integrations can collect technical or operational metadata from systems and files. A metadata repository connects these descriptions, and a data catalog makes them available to users.

The CDO and Data Governance Council can set the metadata strategy and governance model. A Metadata Manager or Metadata Engineer operates the platform, while Data Owners and Data Stewards maintain definitions, ownership, and other domain metadata.

Business Metadata

Metadata includes more than column names and file properties. Business Metadata explains data in the language and rules of the organization.

It includes documented definitions, business rules, ownership, and usage constraints. The university might define an "Enrolled Student" as a student with at least one active course enrollment, then specify what "active" means. That definition is business metadata.

The knowledge used to generate the definition is provided by a Business Analyst, who, together with a Data Steward, turns it into a clear and consistent definition.

Technical Metadata

Technical Metadata describes how systems represent data and where it's located. It includes schemas, data types, table and column names, paths, file formats, keys, and interfaces.

For example, in a catalog, it might indicate that a student's address data is located in a certain table attribute, is textual, and doesn't allow null values. All this information is considered metadata because it describes where the data is and how it's represented.

At this level, Data Architects or Data Modelers typically define the data representation so that Data Engineers, Analytics Engineers, and Database Administrators can handle its implementation.

Operational Metadata

Operational Metadata records what happens when systems process or use data. It can include job start and end times, row counts, query activity, freshness, status, and failures.

For example, at the university, it might be recorded that the enrollment request pipeline ran at 6:00 AM, processed 543 students, and completed successfully in 20 seconds.

This metadata is often obtained from orchestrators like Apache Airflow, application logs, and cloud platforms, which are operated by Data Engineers and DataOps or platform professionals who monitor these executions.

Data Catalogs

A Data Catalog is one of the main systems used to bring these metadata types together.

A Data Catalog is a searchable inventory of the organization's data assets. It usually stores metadata and references to source systems rather than copying all the underlying data. Its main purpose is discovery and understanding, although some catalogs also support access-request and governance workflows.

For example, if an analyst is looking for enrollment records from the past 6 months, the catalog should indicate which database or storage system holds that information, who's responsible for it, other metadata like the name of the system or table where it is located, and the access rules.

Among the most well-known commercial solutions are Collibra, Alation, and Microsoft Purview, often deployed on cloud ecosystems like AWS Glue Data Catalog and Google Cloud Knowledge Catalog. Management is handled by the Metadata Manager or Metadata Engineer, who administers this platform.

Business Glossaries

A Business Glossary is a controlled vocabulary that establishes the official meaning of the organization's concepts. It shouldn't be confused with a data dictionary: the dictionary describes tables and columns of a specific system, while the glossary defines business concepts that may be implemented in many systems.

For example, the term Completed Trip might mean a trip that has reached its destination and whose billing has been validated. This definition prevents the mobility area from considering a trip complete when the journey ends, while finance only does so when the invoice is received. The term should include its definition, synonyms, rules, related concepts, owner, steward, and approval status.

A business expert or Business Analyst proposes the term, the Data Steward reviews its clarity and potential conflicts, and the Data Owner approves its use. The glossary can start as a simple document, but as it grows, you should manage it within the data catalog to link each term with its columns, rules, reports, and policies.

Data Lineage

Data Lineage describes where data came from, how it moved, which transformations changed it, and where it's consumed.

At the university, lineage could show that an address enters through an application, passes to a geographic API, produces a route distance, and contributes to a mobility-eligibility decision. A separate operational flow may then share only the minimum trip details with the transportation provider. This metadata helps teams assess the impact of changes, investigate errors, and demonstrate how a result was produced.

Example of data lineage in the use case. Image by author.

Data Engineers, Analytics Engineers, and Metadata Engineers help capture lineage through tools such as dbt, OpenLineage, or Apache Atlas. Automation can collect lineage from supported systems and generate visual paths from sources to dashboards, but teams still need to validate gaps, semantics, and manually implemented processes.

Metadata Standards

Metadata also needs standards, quality controls, and governance. Metadata Standards define how teams document, represent, and exchange it.

The goal is to help people and systems locate, understand, integrate, and exchange data consistently. ISO-8601 is a data representation standard for dates and times. Within an organization, snake_case might be a metadata naming convention, while a defined JSON schema could standardize how a tool exchanges metadata.

Among the most notable external standards are the ISO/IEC 11179 family, used in metadata registries, and the Dublin Core for describing all types of digital resources. The responsibility for applying these standards falls on the Data Architect and the Metadata Manager, who select the standards.

Metadata Quality

Like other data, metadata should meet defined criteria for accuracy, completeness, consistency, and freshness.

Poor metadata can undermine governance and processing because users may interpret otherwise correct data incorrectly. If a catalog says that distance is measured in kilometers while a system stores meters, for example, downstream calculations can be wrong.

Teams can measure metadata quality through checks for completeness, validity, consistency, and freshness. Lineage then helps them see which downstream assets a bad definition or missing field could affect. A Metadata Manager, Data Steward, and Data Quality Analyst may share this work.

Metadata Governance

Metadata Governance defines who can create, approve, change, and retire metadata. Metadata has its own lifecycle, and a controlled process keeps definitions from changing in production without the right review.

For example, if a data analyst proposes changing the description of the concept "distance to campus" to specify that it will now be measured in meters instead of kilometers, they can't modify that definition directly. Governance requires that this proposal first go through the Data Steward to ensure the new wording is clear and consistent with the rest of the glossary, and then be validated by the corresponding Data Owner.

Only after this approval process is the metadata officially updated in production, preventing uncontrolled changes from causing unnecessary failures.

Although the responsibility usually falls on the Data Steward and the Data Owners, this assignment isn't universal. At the executive level, the CDO and the Data Governance Council establish the general policies that guide how governance should be conducted in the organization.

Data Integration and Interoperability

Most organizations don't keep all their data in one system. They use several systems for different jobs, so those systems need reliable ways to exchange and combine information.

Data Integration and Interoperability addresses that need. Interoperability means systems can exchange data and interpret it consistently, while integration combines or connects data for a particular use.

Because each system holds only part of the picture, data integration gathers or virtually connects information from different sources to provide the view a consumer needs.

The goal is to make the right data available in the right place, format, and time. One requirement is latency: the delay between data being created or requested and becoming available to the consumer. The portal may need current taxi availability within seconds, while a monthly cost dashboard can refresh overnight. Integration must also be secure, observable, and auditable.

For example, university systems must agree on the meaning and unit of "distance to campus" or declare a reliable conversion. Without that shared contract, a value in kilometers can be mistaken for meters and cause serious errors.

Once interoperability is ensured, the data can be integrated to generate, for example, dashboards. At the university, data can be obtained from different systems, such as a database with transportation service records and a payment platform, to ultimately generate a dashboard that shows statistics of the cost of that service over a period of time.

Data Architects define interoperability principles and shared patterns. Data Engineers and Integration Engineers design and build ingestion, mappings, and exchanges. Platform Engineering, DataOps, and SRE teams help deploy, monitor, and recover the supporting services.

Data Ingestion

Data Ingestion moves data from a source into a target environment for storage or processing. The target may keep the data temporarily or persistently.

Sources can include databases, APIs, files, applications, and event streams. Destinations can include operational systems, queues, Data Warehouses, Data Lakes, and other platforms. In a push pattern, the source sends data, while in a pull pattern, the destination or connector requests it.

It's also important to mention that there's a distinction in different types of integration depending on whether the data is inserted into a system or queried "directly" from its sources.

One type is physical integration, where data is extracted and stored in a common destination using ETL or ELT processes. For example, the university could load travel and payment records into a Data Warehouse every night using Apache Airflow, Apache Spark, or Azure Data Factory to later generate a cost dashboard.

On the other hand, virtual integration allows querying different sources without having to store their information in a destination environment, as if the sources formed a single system for querying. In this way, the university could combine the travel database and the payment platform in a single query using technologies like Denodo, obtaining integrated data.

Virtual integration doesn't normally persist a separate consolidated copy, although query engines may cache or process data temporarily. Ingestion, by contrast, deliberately moves data into another environment, where further transformations may follow.

For example, the university might want to analyze whether the free taxi service is actually improving attendance at in-person classes. To do this, it integrates data from sources that record travel logs and student attendance, which are likely in different systems. In this process, the sources are queried, and the data is ingested into a Data Warehouse where it's analyzed.

Technologies used for ingestion include Apache NiFi and Kafka Connect, as well as tools like AWS Database Migration Service or Azure Data Factory. The choice depends on the source, destination, volume, frequency, security, and interoperability requirements. A Data Engineer usually designs and implements the ingestion process with the relevant source and platform teams.

Batch Integration

After defining the sources and destination, the team decides when ingestion and processing should run. The answer depends on how fresh the consumer needs the data to be.

In Batch Integration, the system collects and processes groups of records on a schedule or trigger. This approach is often simpler and more cost-efficient when consumers don't need real-time results, although teams still need to manage the concentrated load that a batch can place on source and destination systems.

For example, the university might load completed trips and payments into a Data Warehouse each night to update the transportation service cost dashboard. The process would extract the data, temporarily store it in a staging area, apply the necessary transformations, and load it into the destination. If the frequency is somewhat higher, the batches are called micro-batches, as they contain less data, though the process is exactly the same.

This type of integration is implemented with technologies like Apache Airflow, Apache Spark, AWS Glue, or Azure Data Factory, primarily used by Data Engineers. Additionally, the integration's operation is supervised and monitored by DataOps or Platform Engineering professionals.

Streaming Integration

When consumers need lower latency, Streaming Integration processes events continuously or soon after sources produce them. Instead of waiting for a large scheduled batch, producers publish events that enter ingestion and processing as they arrive.

For example, a transportation company might publish real-time events indicating that a trip has been requested, accepted, started, completed, or canceled, allowing the student portal to be updated immediately.

These events are typically distributed through platforms like Apache Kafka, Apache Pulsar, or Amazon Kinesis, while Apache Flink or Spark Structured Streaming enable filtering, transforming, aggregating, and finally integrating them.

Here, the most important role remains the Data Engineer, although the more specialized role of Streaming Engineer emerges, capable of ensuring these processes run with the necessary low latency.

https://docs.databricks.com/aws/en/data-engineering/batch-vs-streaming

API-Based Integration

Internal and external systems often expose data or operations through an API instead of direct database access.

An Application Programming Interface (API) is a contract through which one system exposes selected data or operations without revealing its internal implementation. You can think of it as a defined set of calls or resources that other software may use.

The university might send a text address to a geographic API and receive coordinates. When a student requests transportation, an internal API could accept an authenticated student identifier and return an eligibility result without exposing the underlying academic record.

Some data platforms expose controlled query APIs, but public services should avoid accepting unrestricted SQL from clients. The API contract should expose only the operations and data that the consumer is authorized to use.

Technologically, the most common practice is to use an API via the HTTP protocol, exchanging data in JSON format and following a REST style, although there are alternatives like gRPC, GraphQL, or SOAP. Regardless of the implementation technology, the API must clearly define its contract, which can be documented using OpenAPI or AsyncAPI.

APIs are usually designed and implemented by a Backend Engineer or API Engineer, while an integration is designed by an Integration Architect, regardless of whether the sources are accessed through an API or not.

ETL and ELT

If we focus on the ingestion process, data must be extracted from a source and inserted into another system. But the target system usually has a different schema than the sources. Each source stores data in a specific organization to solve a problem, while the target system structures data differently, mainly because it integrates information from multiple sources.

For example, a data source might store records with some student information (name, date of birth, email), while the target system where integration is intended stores records with that information along with each student's payment data, possibly changing some fields (name, age, card number). This means student records need to be transformed, such as calculating age from the date of birth.

Real integrations usually need more transformations because source and target structures differ. The boundary isn't always strict: teams may transform data for compatibility, quality, privacy, enrichment, or later analysis at several stages of the flow.

In summary, the transformations referred to here constitute what's known as Extract, Transform, and Load (ETL). Basically, it's a process consisting of a series of steps where data is selected and extracted from a source, transformed to fit the target data model, and loaded.

In the previous example, the only step needed would be converting the date of birth into an age, assuming the data types of the other fields match.

An ETL is suitable when you need to strictly control the information before it enters the destination. But there's also Extract, Load, and Transform (ELT), which first loads the data into the target system and then transforms it once loaded. This approach is common in cloud Data Warehouses and Lakehouses because it allows for preserving an original version and reusing it for various purposes.

For example, with ELT, the university could load authorized student records and raw provider transaction references into a protected Data Lake before applying analytical transformations. It shouldn't copy full card details or bypass security checks simply because the layer is "raw." Keeping source-like data can support reprocessing, but retention, minimization, and access policies still apply.

Teams can implement these processes with Apache Spark, AWS Glue, and Azure Data Factory. Data Engineers usually design the end-to-end flow, while Analytics Engineers often define transformations inside the analytical platform.

Data Exchange Standards

As you've just seen, the differences between source and destination models require transformations.

To reduce the number of transformations needed for integration, there are Data Exchange Standards, which are common rules about the structure, format, and meaning of the data. Their goal is to encourage, whenever possible, the use of a "unique" or common structure so that all systems structure the data as similarly as possible, avoiding transformations when exchanged.

For example, the university could define an exchange model with fields such as (student_id, name, date_of_birth, email), along with their formats and semantics. If a consumer needs age, the contract should define the date on which it's calculated so the value doesn't become ambiguous. Data Exchange Standards don't have to dictate internal storage. They define the representation used at the boundary.

These rules can be grouped into what's known as a Canonical Data Model, documented with OpenAPI or AsyncAPI, among other tools. The responsibility for their definition falls on a Data Architect or Data Modeler, while a Data Engineer or Integration Engineer is the one who ultimately implements the application of these rules in various systems.

Schema Management

Many systems use a schema that defines field names, types, and constraints. A student record might begin as (name, date_of_birth, email) and later gain a phone field. Schemas therefore evolve as requirements change.

Schema Management versions and governs those changes so producers and consumers can coordinate safely. Compatibility policies state which changes a system can accept without breaking existing data or consumers.

Here, we can make a distinction between backward compatibility and forward compatibility. Backward compatibility refers to the ability of a system using a new schema to correctly read or process data saved or emitted with an old schema. Forward compatibility refers to the ability of a system to use an old schema to read, process (or at least safely ignore) data saved or emitted with a new schema without causing errors. In this context, the ideal is to achieve complete compatibility in both directions.

Teams can express schemas with JSON Schema, Apache Avro, Protocol Buffers, and similar technologies, then version compatible formats in Confluent Schema Registry or AWS Glue Schema Registry. Data Architects and Data Modelers define the shared approach with the engineers who produce and consume the data.

Data Quality

Integration can combine data from several sources, but a technically successful integration doesn't guarantee useful results. The output may still contain missing values, incomplete records, contradictions, or duplicates that affect its intended use.

Data Quality is the capability that measures and improves whether data is fit for purpose, in other words, suitable for its intended use.

Quality isn't an absolute label that makes data perfect for every situation. It depends on the intended use. A city of residence may be enough for aggregate demographic statistics but not enough to arrange a pickup. Data should meet measurable requirements for the task at hand.

Generally, the responsibility for maintaining data quality doesn't fall on a single person. Typically, a Data Quality Manager, along with Data Owners, evaluates which data is most critical for an organization, the impact of potential errors, and what level of quality is acceptable.

Then, a Data Quality Analyst analyzes and monitors data practically to ensure its quality, while Data Engineers and development teams implement necessary processes to achieve the required quality. These people don't use specific technologies to manage data quality but rely on other technologies like SQL.

Data Quality Dimensions

Data quality is a measurable property through Data Quality Dimensions, which are observable characteristics of the data. Each one addresses a different question about the data and can apply to a single piece of data or an entire record:

  • Accuracy: Checks if the data correctly represents reality.

    • Example: A student's address is accurate if it matches their real address. Otherwise, it doesn't correctly reflect reality.

  • Completeness: Checks if all necessary data for a specific use is present.

    • Example: Imagine a registration form requires a name, surname, and phone number, and the user doesn't provide their phone number, or that data is lost. The registration record would be incomplete if finalized, as the phone field would be null.

  • Uniqueness: Ensures a piece of data or record doesn't appear more than once.

    • Example: When a student enrolls in a university, the database should have one record with their data, not a duplicate, unless design reasons require it.

  • Consistency: Ensures different representations of data don't contradict each other.

    • Example: If a student's email or phone number must be present in multiple places across one or more systems, its value must be the same. It can't appear as one email in one place and a different email elsewhere for the same student. That wouldn't be consistent.

  • Timeliness: Checks if the data is updated and available when needed.

    • Example: When a student requests a taxi, they should be able to get their real-time location data, available and updated with low latency for use.

  • Validity: Ensures the data respects defined type, format, range, and constraints.

    • Example: If a registration request status can be ACCEPTED or REJECTED, those field values can't be different and must be stored in the defined format. Otherwise, they wouldn't be valid according to defined constraints and business rules.

These dimensions are interrelated, and in practice, some may be more critical for data use. For example, timeliness is crucial when a student requests a taxi, as they expect to see their real-time location immediately. Meanwhile, uniqueness is key for financial data, as a payment record can't exist multiple times, which would be a particularly severe error.

Data Profiling

Data Profiling helps a team understand the current state of a dataset. It inspects structure and content, calculates statistics, and looks for patterns or anomalies. A profile might report null percentages, distinct counts, minimum and maximum values, type patterns, and relationships between fields.

For example, if the university keeps a table with students' personal data, it could be checked that names are stored in a text field, not numeric, or that no record has null values, among other more complex checks.

Relationships between columns and tables can also be analyzed in a relational database, allowing verification that all enrollments are associated with an existing person and subject, as otherwise there would be incomplete and inconsistent data.

Profiling alone can't tell you whether the data is fit for a purpose. A null may be a defect in one field and valid in another. A Data Quality Analyst therefore interprets the profile with Data Stewards and domain experts, using tools such as SQL, pandas, or Apache Spark according to the platform and volume.

Data Quality Rules

Data Quality Rules turn requirements into specific, measurable conditions. They help a team detect when data is unsuitable for an intended use and decide what should happen next.

Profiling discovers what the data looks like, while rules state what acceptable data must look like. Examples include:

  • The student's contact email can't be empty and must match the organization's accepted email format.

  • The distance to the campus must be a decimal number greater than zero.

  • The same taxi ride can't be recorded twice. The student's charge may be zero, while the provider cost must be recorded in the authorized finance system so the university can manage its budget.

The rules are actually treated as a type of metadata, so they must be documented and versioned accordingly. The Data Steward and Data Owners design and validate them based on their business sense, while the Data Quality Analyst and Data Engineer turn them into executable checks. Finally, the rules are expressed in the appropriate technology, such as a query language (SQL, Cypher, and so on).

Data Validation

Data Validation executes rules to decide whether data meets established requirements. Unlike profiling, which explores the data's current state, validation compares values and records with explicit conditions.

The enrollment form may require a student's name, but the API and database should still validate it because client-side checks can be bypassed and data can fail in transit. A relational database can enforce conditions with NOT NULL, UNIQUE, CHECK, foreign keys, and other controls. Application and pipeline checks can handle rules that span systems or require more context.

Data Quality Analysts help define and evaluate these checks, while Data Engineers, Software Engineers, Analytics Engineers, and database specialists implement them at the right layers.

Data Cleansing

Validation may show that all records meet the rules. When some fail, the team needs a defined response: reject, quarantine, correct, enrich, or accept the record with a documented exception.

Data Cleansing detects and corrects known defects so data can meet its requirements. The right transformation depends on the field, the rule, and whether the team can determine the correct value safely. For example:

  • To avoid inconsistencies, a rule might specify that names shouldn't contain spaces at the beginning or end. So, if a name like ' Chloé Moreau ' appears, the rule would determine that the data isn't suitable, and it could be transformed by removing the extra spaces to restore its quality.

  • Another rule might require that all dates use the format YYYY-MM-DD. Thus, if a date like '15/09/2025' appears, the data wouldn't comply with the rule, but it could be transformed to '2025-09-15' to fit the defined format.

Depending on the data, the rule, and the problem it presents, some transformations can be performed automatically, while others may require more supervision to be done correctly. For instance, spaces in a name can be easily detected and removed, but other issues may be more complex and require manual transformation.

Data Engineers, Analytics Engineers, application teams, or operational staff may perform cleansing, while the Data Quality Analyst and Data Steward validate the approach. The process should preserve enough traceability to explain what changed and why. Cleaning a symptom doesn't replace fixing the source of the defect.

Data Quality Monitoring

Validation shouldn't happen only when data first enters a system. Data Quality Monitoring runs relevant rules and measurements over time, stores the results, and alerts teams when quality degrades.

For example, the university can schedule the automatic execution of quality rules on student data every night. The system would check conditions such as complete addresses, non-negative distances to the campus, and valid date formats. These results can be stored and displayed on a dashboard, allowing for the detection of trends like a sudden increase in negative distance values after an update. This way, the team responsible for the change can quickly identify and correct the problem's origin.

These periodic evaluations are carried out with AWS Glue Data Quality or Microsoft Purview, among other technologies maintained by Data Engineers and DataOps teams.

Issue Management

When a quality problem appears, Issue Management records, prioritizes, investigates, and resolves it. Priority depends on the impact on people, decisions, compliance, and business processes, not only on the number of bad rows.

For example, if due to some error, all distances start showing as negative and students are denied access to transportation services, it impacts the user experience and could have more serious consequences if a student can't attend an important exam. So issues must be managed as quickly as possible.

Generally, this management follows these phases:

  1. Registration and classification: When a rule is violated, the incident is documented, including its severity and who is responsible for the affected rule or data domain.

  2. Containment: Depending on the severity or impact of the quality loss, measures are taken to prevent that impact from materializing. For example, if a rule states that payment records must not be duplicated and duplications are detected, the measure might be to temporarily block all payments until the issue is resolved.

  3. Analysis: Data lineage is used to debug processes and locate the cause of the problem.

  4. Correction: Once the cause is identified, the problem is corrected, and the rules are re-executed, validating and documenting the resolution.

If duplicate payment records appear, a Data Quality Analyst may detect and coordinate the issue, the Data Owner sets the business priority, and a Data Engineer or application team fixes the technical cause. Finance and compliance teams may also need to verify the correction.

In short, quality dimensions define what matters for a use case. Profiling shows the current state, rules formalize expectations, validation tests them, cleansing handles suitable corrections, and monitoring detects changes. Issue Management then coordinates the response when a problem reaches production.

Data Engineering

We've discussed systems that store, exchange, protect, and validate data. Now we can look at how teams build the ingestion processes, pipelines, and transformations that connect those systems in practice.

Data Engineering designs, builds, and operates the processes and components that collect and prepare data. It moves data from one or more sources into the systems where people and applications need it, including platforms such as Data Warehouses and Data Lakes.

Data Engineering works across architecture, storage, integration, and quality, although it doesn't replace those disciplines. That overlap is why Data Engineers have appeared in many earlier sections.

The implementation may be as small as a scheduled SQL transformation or as large as a distributed streaming pipeline. In either case, Data Engineering manages dependencies, automates repeatable work, tests changes, and monitors execution.

The goal is to let other professionals use trustworthy data without rebuilding the whole path back to every source.

For example, imagine the university wants to create a dashboard for the management team to analyze the monthly cost of the transportation service. To do this, it's not enough to query a single database, as travel data might be in one database while cost or payment information might be with the transportation company.

Additionally, each source updates at a different frequency and uses its own schema, so Data Engineering here would serve to build a process that performs steps such as:

  1. Extract data from each source.

  2. Validate its quality through rules.

  3. Apply the required transformations, including cleansing defects and standardizing dates, units, and identifiers.

  4. Insert them into a target system, such as a Data Warehouse, Data Lake, or similar.

  5. Once inserted, they may need to be aggregated or processed as required for later use.

The Data Engineer designs and implements these processes with Data Architects, Data Stewards, and Data Quality Analysts. Together, they make sure the solution meets its technical and organizational requirements. Analytics Engineers, Data Analysts, Data Scientists, applications, and other consumers use the results.

And if the infrastructure is large enough, other professionals like Data Platform Engineers, DevOps Engineers, and Site Reliability Engineers (SRE) may be involved to assist in its operation.

Data Pipelines

A Data Pipeline is a sequence of automated tasks that moves and processes data from one or more sources to one or more targets. A task may read, validate, transform, route, or write data, then pass a result to another task.

At the university, a pipeline might extract authorized transaction references, trip records, and enrollment data, transform them into a common target schema, and load them into a Data Warehouse. Analysts can then use the curated result for reports and dashboards.

A pipeline can run in batch or streaming mode. A full load reads the complete selected dataset, while an incremental load processes records that are new or changed since a known point. One valuable design property is idempotence: safely repeating the same input or run shouldn't create unintended duplicates or inconsistent results.

Other significant properties include scalability, so a large volume of data doesn't compromise execution viability, and traceability to know when it's executed and the results it produces.

Pipelines are usually designed and implemented by a Data Engineer, but sometimes Integration Engineers or Analytics Engineers assist, depending on the final use of the data.

The technologies used for implementation vary greatly depending on the infrastructure. A pipeline may include queries in SPARQL, SQL, transformations done in Python, Apache Spark, or Apache Flink, and even use cloud services like Google Cloud Dataflow.

Pipeline Orchestration

After defining a pipeline's tasks, inputs, outputs, sources, and targets, you need to coordinate their dependencies. That coordination is orchestration.

For example, imagine a pipeline where student and travel data is obtained first, followed by payment data, and these are to be inserted into a Data Warehouse that only accepts records with both payment information and personal data of a student. With these requirements, data from all sources must be obtained before insertion, as they need to be combined. This might not be the case in other pipelines where information from each source can be inserted as it's obtained.

These dependencies in a pipeline are commonly represented with a Directed Acyclic Graph (DAG) where each node is a task and each connection indicates a dependency. It can also serve as an internal data structure for orchestration software to precisely decide when a task is ready to execute and what should happen based on its result.

Among the most commonly used technologies for orchestration are Apache Airflow, Dagster, and Prefect, as well as cloud services like Azure Data Factory, AWS Step Functions, or Google Cloud Composer.

Data Transformation

Many pipeline tasks transform the structure, representation, or content of data so a later consumer can use it.

Transformations can be simple, like converting kilometers to meters, normalizing a date to a common format, or renaming a field. Others are more complex or follow more abstract business rules, such as linking taxi routes with academic schedules to automatically validate if a trip coincides with a mandatory in-person class, thus detecting improper use of the service or any issues. Some transformations may also involve filtering, removing duplicates, or aggregating data.

When data transformations are performed, the data transitions from being newly obtained from a source to being ready for use. Here, we can establish a classification based on the level of transformation the data has undergone:

  • Raw: Data kept close to the source representation. For example, a provider supplies the date string 05/03/2026, whose intended day/month order must be documented.

  • Staging: Data is validated and standardized for further processing. Once the source meaning is known, the date could become the unambiguous ISO value 2026-03-05.

  • Curated: At this level, the data is enriched, combined with other data, and considered ready for final use. For example, assuming the previous date corresponds to a trip, it can be combined with other data to create a record of that trip enriched with payment information.

Transformations focus on converting raw data into staging and curated data. Technically, implementation can be done using various technologies depending on the systems involved and company decisions. Primarily, you'll use languages like Python, R, SQL, or frameworks like Apache Spark.

Workflow Automation

A pipeline may also check source availability, validate quality, manage approvals, and send notifications. Workflow Automation coordinates these actions in the required order so repeatable work doesn't depend on someone running every step by hand.

It's important to differentiate between the pipeline and the workflow. The pipeline describes the path of the data and its transformations. On the other hand, the workflow includes tasks that don't directly transform the data but are essential for the execution of a pipeline.

For example, when the university receives a file from the transportation company, the workflow can validate its format, monitor the pipeline execution, and update data lineage tools.

But automating a workflow doesn't always mean eliminating human intervention. For instance, a rule might be set to detect if personal data appears in a source when it shouldn't. If this rule detects personal data, a Data Steward intervenes to approve the change or reject it and take appropriate action.

Finally, workflows are implemented using orchestrators like Apache Airflow, Dagster, or Prefect, along with CI/CD systems and incident management tools.

Data Testing

When automating the execution of a pipeline, even if manual oversight isn't completely eliminated, much of the process will run with the possibility of errors in its implementation. Even with a perfect implementation, errors can occur that affect the data and cause failures in the pipeline tasks.

Data Testing checks both transformation code and the data moving through the pipeline so teams can catch defects before they affect consumers.

The test suite should cover realistic ways that code, schemas, data, dependencies, and infrastructure can fail. Data tests and Data Quality rules overlap, but teams may apply them for different reasons.

A quality rule expresses a business or fitness requirement, while a pipeline test may verify a technical precondition or expected transformation. The same check can serve both purposes.

Common test types include:

  • Unit tests: These verify that the code for a transformation is correct given certain inputs and the respective outputs it should produce. For example, if a transformation converts a distance from kilometers to meters, it could be tested with inputs 18, 4, 6 and outputs 18000, 4000, 6000.

  • Schema tests: These are performed on the data to ensure its structure and format are suitable for a specific task. For instance, when receiving a student's age stored as the number 42, a schema test would verify that this data is of integer type.

  • Integration tests: These check that various components of an architecture or system can interact as expected. For example, an integration test might verify that a university's Data Warehouse can receive data from an academic database.

  • End-to-end tests: These involve executing the entire pipeline to ensure the result is correct given initial data.

  • Reconciliation tests: Compare counts, totals, or control values across stages. If a documented filter should retain 50 of 100 input records, the test verifies both the output count and the reason for the exclusions.

  • Performance tests: Given the complexity of some pipelines, performance tests are conducted to evaluate if their execution is feasible within a certain time and with available resources.

At the university, before deploying a pipeline, datasets with fictional information, also known as synthetic datasets, could be constructed for use in testing. This way, all these types of tests could be executed to verify that tasks are performed correctly, data has the expected properties after each transformation, and the process is completed within a specified time.

The test technology follows the pipeline. A Python transformation could use pytest, while SQL can support reconciliation and schema checks. Data Engineers own most pipeline tests, and Platform or DevOps Engineers help integrate them into automated delivery and runtime environments.

Data Versioning

Data pipelines generally undergo changes due to modifications in business requirements, changes in sources, or other reasons. So it's essential to maintain a history of what has happened with a pipeline over time, allowing you to track its evolution up to a specific point, primarily to facilitate error debugging.

Data Versioning keeps a history of the assets needed to reproduce a result. Depending on the use case, this can include transformation code, schemas, configuration, reference data, model inputs, and snapshots or versions of the dataset itself.

For example, imagine a report states that $10,000 was spent on taxis in a month, but upon checking later, the system says the amount was $8,000 for the same month. This discrepancy could be due to an error or a change in the policies used to calculate that cost, such as no longer counting canceled trips.

To determine if this situation is an error, versioning allows access to previous versions of the pipelines involved in that calculation to see how the figure was obtained.

Teams commonly use Git for code, configuration, and text-based schemas. Table formats such as Apache Iceberg, Delta Lake, and Apache Hudi can preserve data snapshots and change history for supported tables. Reproducibility may require both.

Data Platform Operations

Once implemented and versioned, a pipeline needs an infrastructure to run on, which refers to hardware that can be on university servers or in the cloud. It may require storage for data, computing capacity for transformations, an orchestrator to coordinate tasks, and specialized systems to ensure data and process security. These components together form a Data Platform, which is the technological environment where pipelines and other processes are executed.

The platform itself must be managed and maintained, as it's not a system that operates completely autonomously but requires supervision. This management process is known as Data Platform Operations and encompasses a series of tasks aimed at ensuring the platform is ready to execute pipelines securely, stably, and efficiently.

Some of the most fundamental tasks are:

  • Provisioning and scaling of resources: The number of machines needed by databases and platform components at any given time is configured.

  • Environment management and isolation: Reserved environments are created for testing, development, and production, with the latter providing services to the end user.

  • Permission management: Permissions are determined for each professional to perform their tasks, preventing security breaches.

  • Cost control and optimization: Resource consumption is monitored to avoid overspending, aiming to provide the service with minimal consumption.

For example, a pipeline that calculates the monthly cost of taxi usage might need to connect to a transportation company's API, transform the data, and store it in a Data Warehouse.

To achieve this, the platform must provide the necessary computing resources to perform the transformations, store the data, and allow a secure connection with the API. Thus, proper platform management is critical to ensure the pipeline runs correctly.

A Data Platform Engineer commonly leads this work and understands the services on which the platform runs, such as AWS, Azure, Google Cloud, Databricks, or Snowflake. Docker packages suitable workloads, Kubernetes can orchestrate containers when the complexity justifies it, and Terraform defines infrastructure as code. Infrastructure as code improves repeatability, but it doesn't make services automatically portable between cloud providers.

Data Observability

Data platforms can fail in subtle ways even when every job reports success. Data Observability helps teams understand the health of data and the systems that produce it so they can detect, investigate, and reduce the impact of failures.

Observability lets you infer a system's state from the signals it produces. In a data context, those signals include freshness, volume, schema, distribution, quality results, lineage, job status, logs, metrics, and traces.

Monitoring checks known conditions, such as whether a job completed and whether freshness or volume stayed within expected limits. Infrastructure signals such as CPU and memory can help explain failures, while data-level signals show whether consumers received the right output.

For example, if a data pipeline produces dozens of records when it should produce hundreds, monitoring allows you to detect these changes in results,. It can also show other relevant metrics obtained at those same moments, such as the CPU usage of each task involved in the pipeline, helping you detect if any tasks are failing and preventing data from propagating to the end.

For observability to guide action, teams can define Service Level Indicators (SLIs) for relevant properties and Service Level Objectives (SLOs) for the expected level. An SLI might measure the age of the latest attendance data, while the SLO could state that 99% of daily updates must be available by 7:00 AM. An alert tells the team when the pipeline risks missing that commitment.

The most well-known technologies in observability are Prometheus and Grafana, frequently used to collect and visualize metrics. There are also OpenTelemetry for managing telemetry data and logs, and OpenLineage for monitoring data lineage in real time.

Here, a Data Engineer might be responsible for implementing the appropriate observability mechanisms. But they don't always do it alone, as an SRE, Platform Engineer, or DataOps team may collaborate in maintaining these mechanisms.

Data Contracts

Observability helps detect errors such as failed jobs, stale data, abnormal volumes, and unexpected schema changes. If a taxi provider changes geographic coordinates from numbers to text without notice, for example, downstream processes may fail even though the network connection still works.

Data Contracts reduce this risk by making expectations between producers and consumers explicit. They define the structure and characteristics of the data, along with how teams communicate and version changes. Observability still verifies the contract in operation.

More specifically, a Data Contract can define schema, types, formats, semantics, quality rules, ownership, delivery frequency, latency, and change-management expectations.

For example, the transportation company might agree that each trip event includes (trip_id, student_reference, provider_vehicle_id, price, origin, destination). The contract could define price in euros and coordinates as numeric latitude/longitude pairs, set privacy limits on student_reference, and require a new contract version for an incompatible change.

Also, the contract isn't just documentation. Checks are implemented to verify compliance so that any change, for safety, doesn't affect data pipelines, as changes can impact both availability and security.

To define a Data Contract, data schemas are often represented in JSON Schema, Apache Avro, Protocol Buffers, or similar technologies, although standards like the Open Data Contract Standard (ODCS) are also used.

The contract is developed and reviewed by Data Engineers and Analytics Engineers within the organization, who coordinate with professionals from other companies, such as Software Engineers who know what data their source produces. At a higher level, Data Owners and Data Stewards are involved to validate the semantics, quality, and usage conditions of the data.

DataOps

Data Engineering involves many people and components. Even a pipeline that works today can become unreliable if teams don't coordinate changes to sources, contracts, code, infrastructure, and quality rules.

DataOps is an approach to improving that collaboration and delivery process. It aims to shorten the path from a business need to trustworthy data while maintaining quality, security, and traceability.

DataOps isn't a specific technology. It's a set of practices such as versioning code, automating tests, reviewing and deploying changes through controlled environments, and monitoring production pipelines. It adapts ideas from agile delivery and software operations to data-specific concerns.

For example, imagine the university starts working with a new taxi company. The first step could be creating a Data Contract with the conditions for data delivery. Then, a Data Engineer would implement all the necessary software for obtaining it through a connector and store it in Git.

Also, before deploying it in production, you should conduct data and code tests to ensure functionality. Finally, after deployment, it would be monitored through metrics like the volume of data extracted, its quality, and latency.

Data Engineers, Analytics Engineers, Data Stewards, Data Owners, Platform Engineers, SREs, and consumers all contribute to DataOps. The practices work only when the people who produce, operate, and use data share responsibility for reliable delivery.

Technologically, DataOps relies on tools we've already discussed, like Git for versioning and CI/CD tools for automating tests and deployments, among others. But its value doesn't come from a specific tool. It comes from adopting best practices in their use.

Many of these ideas come from DevOps. Nonetheless, DataOps adapts them to data work, incorporating specific aspects like quality, semantics, lineage, and the relationship between producers and consumers.

DevOps

As I just mentioned, DataOps adopts ideas from DevOps. DevOps refers to a set of best practices that help coordinate software development and the deployment of systems, all with the goal of ensuring that changes can be tested, deployed, and maintained in an automated and reliable manner.

Among its main practices is Continuous Integration (CI), which involves integrating each code change into a repository so tests are automatically conducted. Then there's Continuous Delivery or Continuous Deployment (CD), allowing changes to be deployed automatically in a controlled manner across different environments. Finally we have Infrastructure as Code (IaC), which lets you define infrastructure components programmatically, facilitating their versioning and deployment across various cloud platforms or servers.

For example, when a Data Engineer modifies the connector that extracts data from the taxi company, the change is saved in Git and a CI system automatically runs its tests. If it passes, a new version of the software is built and deployed autonomously in a test environment to continue verifying its functionality until it's deployed in the final production environment.

Common technologies include GitHub Actions, GitLab CI/CD, or Jenkins for automating tests and deployments. Docker is also commonly used for packaging software along with Terraform or OpenTofu for defining infrastructure. Kubernetes can also be used to manage containers when the system's scale and complexity require it.

DevOps Engineers, Platform Engineers, and SREs implement and maintain these mechanisms, while Data Engineers use them to deploy their pipelines. The main difference is that DevOps focuses on software and infrastructure delivery and operation, while DataOps also checks data-specific aspects like quality, semantics, lineage, and availability.

Data Warehousing and Business Intelligence

Organizations capture, integrate, and transform data through pipelines, then store it in systems chosen for particular workloads. Operational databases support the applications and transactions that keep day-to-day services running.

Analysis often needs integrated history, stable definitions, and queries that scan many records. Specialized platforms such as Data Warehouses and Data Lakes support that work. Data Warehousing and Business Intelligence makes governed analytical data available to people who explore it and use the results in decisions.

These are two related concepts. Data Warehousing covers the design and use of a Data Warehouse, which integrates historical data from several sources for repeatable analytical workloads.

Operational and analytical workloads have different priorities and access patterns. Some platforms support both, but teams still face tradeoffs in isolation, performance, freshness, consistency, and cost. Separating the workloads often protects daily operations and gives analysts a model designed for their queries.

Business Intelligence (BI) covers the practices and technologies used to query, analyze, and present data for decision-making. A Data Warehouse often provides the governed analytical foundation for BI, although BI tools can use other sources too.

For example, a university might integrate trip and finance data in a Data Warehouse. Analysts could compare provider costs, usage, attendance, and budget to assess whether the transportation benefit is sustainable and estimate short-term spending.

Also, in order to conduct these data analyses, build dashboards, and ultimately make decisions, the data needs to be of high quality, protected, and maintained with proper lineage. Any issues in these aspects can influence decision-making.

Analytical Data Stores

Analytical workloads often scan long time periods, join several sources, and aggregate large numbers of records. Storage designed mainly for operational transactions may not be the best place to run them repeatedly.

Analytical Data Stores are designed for analytical queries, transformations, and aggregations. They still need security and consistency controls, but their performance priorities usually favor scans and calculations across large datasets rather than high-frequency row-level transactions.

The most representative example of an Analytical Data Store is a Data Warehouse, which stores data in a stable and scalable way so that the same analysis process can be repeated over time with an ever-increasing volume of data.

But this is not the only option, as Data Lakes are also oriented toward this type of use, and Data Marts offer a smaller-scale analytical environment (usually being subsets of data from a Warehouse) specifically designed to meet the needs of a particular department or business area.

For example, the university could create a Data Mart containing mobility measures and the limited financial context needed to analyze service cost, without exposing irrelevant student details. The team should connect the Mart to lineage, security, quality, and audit controls just as it would any other analytical asset.

Among the most used platforms to implement these systems are Snowflake, Google BigQuery, Amazon Redshift, Microsoft Fabric Data Warehouse, and Databricks SQL. Their design and implementation are the responsibility of an Analytics Architect or Data Architect, while Data Engineers maintain the data pipelines that supply them with information, and Analytics Engineers handle the transformations required after ingestion to facilitate subsequent analysis.

At the administration and maintenance level, there are Data Warehouse Administrators or Platform Engineers, who monitor performance, manage permissions, and platform costs.

Facts and Dimensions

An Analytical Data Store may preserve source-like data or organize it into a model, depending on the platform and layer. A Data Lake commonly retains source formats in an early zone, while curated layers and Data Warehouses apply more explicit schemas.

One common analytical approach is the dimensional modeling we talked about earlier. It organizes information into facts and dimensions. A fact records an event such as a trip, while dimensions provide context for filtering, grouping, and comparison.

A particularly important design choice is granularity, or grain: exactly what one row of a fact table represents. The team should define it before choosing dimensions and measures so later aggregations remain valid.

For example, the Trip fact table might have a grain of "one completed trip." If a student takes two trips on the same day, the table stores two rows, each with its cost, distance, duration, and date key. The university can sum those rows by month. It shouldn't add monthly-total rows to the same fact table because they have a different granularity and would cause double counting.

Once the granularity is defined, dimensions should be chosen based on the context describing the fact and the analytical queries expected to be performed. A practical way to identify them is by asking who, what, when, where, and how each fact was involved. For example, if each row represents a trip, dimensions like Student, Date, Provider, Origin, and Destination could be used, each with a unique value for that trip.

These dimensions would allow analysis of the geographical areas where trips occur, which transportation company makes more or fewer trips, and so on. This way, dimensions are incorporated that provide a useful perspective for analyzing the facts.

This data modeling is done by an Analytics Engineer or Data Modeler, along with domain experts like Data Stewards. Then, Data Engineers implement the data ingestion and transformations required to adapt the data to the specific final model of each system.

Metrics and KPIs

In a dimensional model, facts can be seen as rows composed of values, called measures. These measures can help understand what happened during an event over time, but data analysis generally aims to answer questions involving all events over a certain period.

Teams combine measures into repeatable metrics, such as totals, rates, averages, and percentiles. A metric becomes a Key Performance Indicator when it's tied to an important objective and helps show whether the organization is meeting it. Here are some examples:

Concept Meaning Example
Measure A value recorded in a fact A trip cost €18
Metric A repeatable calculation over a set of measures Monthly transportation cost = sum of the cost of trips completed during the month
KPI A metric associated with a business objective Monthly mobility budget consumption, with the hypothetical objective of not exceeding the allocated budget

As is evident, not every metric is always a KPI. For example, a metric that represents the total number of trips made in a month can be useful for describing transportation service usage, but it will only be a KPI when there's a business objective that involves quantifying that number of trips.

KPIs are often used in dashboards and visualizations, although they generally don't appear in isolation. In this regard, when several KPIs with their current values are gathered and compared with established goals, this gathering is called a scorecard.

Despite both concepts being related, a scorecard and a dashboard have different purposes. A scorecard aims to determine if goals are being met, while a dashboard helps understand what's currently happening in the organization and why.

The same metric may appear in dashboards, scorecards, reports, and APIs, so teams need a reusable definition. Its documentation should include:

  • The name, purpose, and business owner.

  • The formula that calculates the resulting value of the metric, the sources of the data, and its granularity.

  • The unit, time period, time zone, and frequency of metric value updates.

  • The filters and inclusion rules, such as excluding canceled trips from the calculation.

  • In the case of a KPI, the objective that originates it is documented.

Here, metrics and KPIs are primarily defined by roles like Business Owners, Data Owners, and Data Stewards. On the other hand, their practical implementation is carried out by Analytics Engineers and BI Developers, and finally, their results are used by Data Analysts, among other professionals.

Semantic Layers

As I mentioned before, metrics are documented to ensure their meaning and calculation method are well understood. But this doesn't guarantee that all systems adhere perfectly to this documentation.

For instance, monthly cost might be calculated excluding canceled trips, while another system might accidentally include them. In both cases, the same "name" is used for a metric that produces different results.

A Semantic Layer addresses this problem by centralizing reusable business definitions between stored data and consumption tools. It presents concepts such as Trip, Student, or Course instead of requiring every consumer to rebuild logic directly from tables and joins.

In this way, the formulas and filtering rules that make up each metric are implemented on the semantic layer, rather than each analyst writing their own code on a database, Data Warehouse, or corresponding system. This layer acts as an intermediary that translates the calculation of a metric expressed in a business-friendly language into the necessary code for specific systems to perform that calculation, facilitating future metric modifications and portability between different systems.

For example, in the Data Warehouse, there might be a Trip fact table, a Date dimension, and a cost measure in each fact. Here, the semantic layer would define the existence of certain concepts like trip and cost, whose calculations are "mapped" in some way onto the technology used to implement each system.

In this case, the calculation of a "Total Cost per Month" metric could be defined on the semantic layer, which would internally translate this into SQL operations, or the corresponding technology, to group trips by month and sum the cost measure of the grouped facts.

The main difference between the documentation of a metric and its implementation in a semantic layer is that the documentation specifies what the metric is and how it is formally calculated, while in the semantic layer this specification is translated into operations in a specific technology that allows the calculation.

Thus, multiple dashboards or reports can reuse the same logic defined on a semantic layer, as sometimes calculations need to be performed on data in different systems.

Technologies used to implement semantic layers include Power BI Semantic Models, LookML, dbt Semantic Layer, and Cube. Analytics Engineers and BI Developers commonly build and maintain these definitions with input from business owners and analysts.

Reports and Dashboards

After implementing the Analytical Data Stores systems in production and defining some metrics or KPIs, the next step is to create Business Intelligence products that present the analysis results to end users, professionals, or executives.

The most common products are reports and dashboards, though they aren't the only ones, as the analysis results can also lead to a visualization or documentation of a decision-making process, for example.

Let's better understand what each one is and their differences:

A report is a document that presents detailed and structured information on a specific topic and time period. It may include graphs, metrics, and explanations. Reports can be generated periodically in static formats, like PDF, or be interactive, allowing users to filter or manipulate the presented information.

For example, a university might prepare a monthly report with the transportation service cost broken down by provider, showing canceled trips, the number of students who used it, and so on.

A dashboard, the other hand, is a view that brings together the most relevant metrics and KPIs to monitor a situation. It typically contains graphs and visual elements that update more frequently than a report.

For example, a dashboard for the administration could show the consumed budget, the number of enrolled students, and the attendance trend, also allowing results to be filtered by training program if it is interactive.

There are some best practices to follow when you're creating dashboards to make sure they're useful. For example, you should display only a few indicators and only those truly relevant to the dashboard's purpose. Also, choosing a visualization isn't merely decorative, as the charts should help people understand the information presented, and they should follow best practices in their design.

In general, you'll use a dashboard when it's necessary to periodically monitor a small set of indicators and quickly detect changes or deviations. You'll use a report when you need a deeper exploration of a topic, although both products can complement each other.

For example, the administration might use a dashboard to detect an increase in transportation expenses and then consult a monthly report to find out which providers, routes, or periods caused it.

For creating these products, the most commonly used technologies are Microsoft Power BI, Tableau, Looker, Apache Superset, and Metabase. These are primarily used by BI Developers, although BI Administrators also collaborate in managing the workspace where the products are built. Finally, the results can be interpreted by a BI Analyst, who also has the knowledge to develop reports or dashboards in certain situations alongside the BI Developers.

Self-Service Analytics

The data analysis process generates products like dashboards or reports, which present specific information structured for a purpose. But sometimes it may be necessary to modify that purpose.

For example, the finance department might have a dashboard designed exclusively to monitor the overall budget the university allocates to taxi services. Yet, the director of a specific master's program might need to cross-reference that transportation data with attendance records from their training program to see if the service provides any benefit, which is a very specific need not addressed by the original dashboard.

The coordinator could ask the technical team to change the dashboard, but every small question would then enter a development queue. Self-Service Analytics lets authorized users explore governed data and create suitable analyses without depending on a technical specialist for every step.

This approach relies on elements we covered earlier, such as semantic layers where metrics are maintained, data catalogs that allow you to quickly locate available information, and business glossaries that standardize the meaning of business concepts. These elements are used by team members who independently build their own visualizations and reports, although they may not have access to all types of information due to existing privacy policies. This is why the process is called managed self-service.

For example, if a dashboard shows an increase in transportation expenses, a Master's coordinator could use a semantic layer to define a filter for their program's data. Thus, the semantic layer would ensure the official cost definition is used, while permissions would prevent access to data from other programs or unnecessary personal information.

Finally, it's worth noting that the original dashboard isn't always modified. Instead, the coordinator creates a new one with their changes.

In practice, the viability of this approach is the result of coordinated work by Analytics Engineers, BI Developers, and BI Administrators, primarily. The end users who consume and leverage this capability are Data Analysts, Business Analysts, and business managers.

Big Data

The data lifecycle runs across an infrastructure of systems and pipelines. Data enters, moves, gets stored and processed, and eventually reaches operational or analytical consumers.

For a moderate workload, a relatively simple architecture may meet the required performance, reliability, and cost targets. As the organization grows, however, it may need to store more data, process events more often, and support more varied formats and use cases.

A database that began on one machine might first scale vertically by gaining more CPU, memory, or storage. At some point, the workload or resilience requirements may justify horizontal scaling across several machines, but that added complexity should solve a measured need.

Big Data deals with datasets and flows whose volume, velocity, variety, or combination pushes beyond the practical limits of conventional tools for a particular organization. The challenge is not simply "a lot of rows". It's meeting the required processing time, reliability, and cost at that scale.

When thinking about Big Data, you might imagine a well-defined threshold beyond which a data set is considered Big Data. But this isn't the case, as the threshold depends on the current infrastructure, the target speed, the cost thr team willing to incur for its management, and the variety in the structure of the information.

A team should adopt a Big Data solution only after assessing whether the current infrastructure misses its performance, reliability, or cost requirements. Distribution may help, but it also adds operational complexity, so the benefits need to justify it.

For example, a university could grow from having 1,000 students to 100,000 due to an expansion of its faculties or the introduction of online classes. If this happens, the databases must support storing all their personal data, as well as the data generated when interacting with various services and platforms like the virtual campus, all at a speed that doesn't compromise service availability or quality.

Big Data draws on many Data Management capabilities at a larger scale. A Big Data Engineer is often a Data Engineer who specializes in distributed storage and processing. They work with Data Architects who design the solution and Data Platform Engineers who operate it.

The 3Vs: Volume, Velocity, and Variety

There's no universal threshold for Big Data, but the 3Vs – Volume, Velocity, and Variety – provide a useful guide. They aren't three boxes every project must check. They describe pressures that can make a workload harder to manage with the current infrastructure.

Volume refers to the total amount of data that must be stored and processed. The first challenge here is that data takes up space, so in a large enough volume, some systems may not be able to handle it all. Also, various management processes slow down as the volume increases because all data must go through pipelines or similar processes.

  • Example: Volume can be associated with the amount of data produced by students, meaning the more students there are, the more data volume needs to be supported. Each student generates data like login events, which must be stored and processed, taking up space and consuming significant computing resources if the volume is high.

Velocity refers to how quickly data arrives, changes, and must become available to consumers.

  • Example: Transportation service taxis must communicate their position and status every few seconds so a student can have a real-time view of available taxis and whether they are near their location. So it's crucial that data is available as quickly as possible to ensure a good user experience.

Variety, as previously mentioned, describes the nature or diversity of data, such as structures, formats, and meanings that data presents.

  • Example: An academic database can store enrollments and students in tables using a relational paradigm, while the virtual campus produces logs in semi-structured JSON documents, or a graph-oriented database represents information about students, drivers, and locations with graphs to optimize transportation routes.

Volume affects storage, transfer, and processing costs. A team may optimize the data model, partitioning, queries, or retention before distributing the workload. When one machine can no longer meet the requirements economically or reliably, horizontal scaling becomes one option.

Not all data needs real-time processing. A live trip-status update may need seconds, while a historical tuition-payment report can refresh on a daily schedule. The required latency should come from the user and business need, not from a desire to make every pipeline real time.

Finally, variety is one of the most significant properties of data because it determines the heterogeneity of the dataset within the organization. With such diverse data stored in different structures, formats, and representations, it becomes necessary to adopt specific techniques for each variety to ensure efficient and viable management.

These are the properties typically attributed to Big Data. But it's also important to highlight other significant properties, such as veracity, which refers to the reliability of the data or value, among others.

Big Data Architectures

When the 3Vs exceed the capacity of a "conventional" solution, there are several ways to increase the capacity of an infrastructure to meet these needs. But first, it's useful to define what infrastructure is.

Infrastructure is the set of computing, storage, networking, and foundational software resources on which the organization's applications and data systems run.

Architecture describes how components use that infrastructure to meet requirements. It defines where systems run, how storage and processing are distributed, and which path data follows from source to consumer.

So if the 3Vs compromise the viability of an existing solution, it may be necessary to modify its architecture. One way to address an increase in volume or velocity, as mentioned before, is vertical scaling. This involves improving the hardware, giving each machine more resources. But this can't scale infinitely, which is why horizontal scaling exists. More machines are added, and storage and processing are distributed.

Another way to increase speed could be the parallel execution of processes across multiple machines, known as Massively Parallel Processing (MPP).

There are many ways to improve the capabilities of an infrastructure, especially when it comes to processing more data at higher speeds. Managing a greater variety of data, though, is often a challenge without established general techniques, although distribution can help.

To better understand what architecture consists of, think of it as a set of layers where each encompasses certain components that together constitute the path data takes throughout its lifecycle within the organization.

Layer Functionality Example Technologies
Sources Origin where data is obtained or generated Taxi company API and payment platform REST APIs, PostgreSQL, IoT sensors
Ingestion Moving data from sources into the platform Receiving virtual-campus events and provider trip updates Apache Kafka, Apache Airflow
Storage Persistently storing data Retaining events, files, and curated analytical tables Amazon S3, Google Cloud Storage
Processing Cleaning and transforming data according to its purpose Removing duplicate trip records in a data pipeline Apache Spark, Apache Flink
Serving Exposing information for querying A Data Warehouse exposes integrated trip information and the associated costs Snowflake, Google BigQuery
Consumption Using information for decision-making or any other purpose Dashboard showing the monthly cost of the transportation service Power BI, Tableau, Jupyter

Another important aspect of any architecture is that its layers must implement security, lineage, and observability mechanisms, also ensuring data privacy.

Imagine a student requests a taxi through the virtual campus. The architecture must protect and trace the event. A streaming flow can update trip status on the portal within seconds, while a later batch process consolidates the relevant records for cost analysis.

This difference in speeds is another way to adjust the architecture so that certain critical functionalities have the required speed or so that analysis processes that don't need to be performed in real time can handle a larger volume of data.

Finally, the architecture is designed by a Data Architect or Big Data Architect and implemented by Data Engineers, Streaming Engineers, or Software Engineers. Its maintenance is the responsibility of Data Platform Engineers, Cloud Engineers, and SREs.

Big Data Storage and Processing

After designing the architecture, its components are implemented, with some dedicated to storing and processing data at the required scale. On one hand, storage is responsible for keeping data persistent, secure, and accessible. On the other, processing uses computing resources to transform and analyze them, primarily.

The university might retain authorized virtual-campus events, attendance records, and trip information for several years, creating a large storage need. Its processing demand may be more variable, with peaks during reporting periods or major academic events.

By separating storage from processing, if we focus on systems that can serve to store data in an infrastructure, we might encounter:

  • Distributed databases: These are databases deployed to operate across multiple machines, using technologies like Cassandra or DynamoDB.

  • Object Storage: These systems are dedicated to storing large volumes of data in independent objects, utilizing Amazon S3, Azure Blob Storage, Google Cloud Storage, or MinIO.

  • Search engines: These systems specialize in quickly indexing and querying logs, texts, and other types of semi-structured information with technologies like Elasticsearch or OpenSearch.

  • Distributed file systems: These store and distribute files across multiple machines using HDFS or CephFS.

On the other hand, data processing in an infrastructure can be distinguished based on the approach taken, which depends on volume and speed:

  • Batch processing: Here, data is accumulated over time and periodically processed in batches. This can be implemented with Apache Spark, for example, which allows tasks like transformation and cleaning to be distributed across multiple machines.

  • Streaming processing: Here, all data generated or arriving at the start of a pipeline is processed continuously, making it suitable when real-time results are needed. Technologies used in this case can be Apache Flink or Spark Structured Streaming.

  • Distributed query and processing: This allows for the analysis of large volumes of data by executing operations in parallel across multiple machines. One of the most common interfaces is SQL, used by tools like Trino or Spark SQL. But in addition to SQL, these systems often offer APIs in languages like Python, Java, or Scala and abstractions like DataFrames, providing greater flexibility for implementing complex transformations or custom logic.

As an example of architecture, the university could use Kafka to receive events generated by the virtual campus or the transportation company, while Flink could process them to keep the status of each journey updated in real time in the application consulted by the end user. Then, with Spark, they would be transformed to be integrated into a Data Warehouse and queried using SQL.

In practice, the central role that implements and optimizes these storage and processing systems is the Big Data Engineer or specialized Data Engineer. For this, they use technologies like Cassandra, Amazon S3, or HDFS, decide how to implement jobs using Spark, and ensure adequate performance.

On the other hand, Data Platform Engineers, Cloud Engineers, and SREs handle the base infrastructure, ensuring its stability, availability, and resilience.

Big Data Analytics

In Big Data, besides storing a large volume of diverse data and processing it at a speed that often needs to be high and in real-time, it must be converted into information, knowledge, and ultimately value. This means that processing refers to the transformations performed on the data to enable storage, clean it, or maintain its quality, primarily.

But processing is also applied after storage to calculate statistics and generally analyze the data. This is the role of Big Data Analytics, an area dedicated to converting data into information, knowledge, and value through analytical processes applied to large volumes of data.

An analysis belongs in a Big Data context when the workload's scale or flow characteristics require distributed or otherwise specialized infrastructure to meet its targets. It doesn't need advanced Machine Learning, and using a scalable cloud platform by itself doesn't make a small analysis "Big Data."

Based on this technological foundation, there are several fundamental analytical approaches you can use, depending on the analysis you need to perform:

  • Descriptive Analytics: Focuses on applying techniques that explore data to understand what has happened. For example, it allows calculating how many trips have been made, how much they have cost, and how many students have used the service each month.

  • Diagnostic Analytics: Here, the analyses aim to understand why a result has occurred. At the university, it could be used to study which supplier time slots are related to an increase in transportation service costs.

  • Predictive Analytics: Uses historical data to make inferences and try to predict what will happen in the future. For example, it could predict how many enrollment applications will be received next term.

  • Prescriptive Analytics: Turns the results of analyses into recommendations. For instance, in this case, it could suggest how to optimize the distribution of taxi fleets and reallocate the monthly budget to ensure service coverage for the maximum number of students.

In big data environments, analysis can be executed in batch or streaming, depending on each process's requirements. For instance, with Apache Spark, you could periodically calculate the evolution of taxi trip costs and class attendance, while with Flink, real-time trips could be analyzed to generate alerts if demand exceeds a certain amount.

For analysis processes to be truly useful, they begin by defining the question to be answered with the obtained knowledge and the value expected to be added, meaning the decision to be made with the result. Then, the necessary data is selected and prepared, ensuring its quality is adequate for analysis. After execution, the result is published via a dashboard, report, alert, API, or predictive model.

It's also important to note that having a larger volume of data doesn't always guarantee "better" conclusions or more value. For example, if students using the taxi service have higher attendance, you can't directly conclude that transportation is the cause, as those students might be taking more in-person classes or have other differences.

So besides handling a large volume of information, it's crucial to interpret results correctly. In this specific case, the problem is that correlation doesn't always imply causation in the analyzed facts, but this isn't the only issue that can arise in an analysis.

Data Engineers build and maintain pipelines and analytical environments. Data Analysts use SQL, Trino, Spark SQL, Power BI, Tableau, and similar tools for a range of analyses, often descriptive and diagnostic. Data Scientists use Python, R, Jupyter, Spark, or MLlib for statistical modeling, experimentation, prediction, and optimization.

BI Developers turn governed metrics and analyses into reports and dashboards. Machine Learning Engineers help train, deploy, and operate models. Domain experts, Data Owners, and Data Stewards help teams interpret and use the results responsibly.

Analytics and Data Science

Organizations analyze data to understand what's happening, support decisions, test ideas, and build models. This is one of the main ways they turn data into knowledge and value.

Analytics and Data Science are overlapping, complementary fields. Analytics often focuses on answering defined questions with descriptive, diagnostic, predictive, or prescriptive methods. At the university, an analyst might study attendance over the past month and investigate which changes coincide with a decline.

Data Science often tackles less-defined or model-heavy questions through statistics, Machine Learning, computation, and domain knowledge. It may explain patterns, estimate effects, segment observations, or make predictions. The university could use it to forecast transportation demand over the next six months.

In practice, both use data to achieve a goal, and the exact boundary varies by organization. Both need governed, suitable, high-quality data and a clear understanding of the decision their result will support.

An analysis should start with a clear question. The university might ask whether the transportation benefit improves class attendance or how many rides students will request next week. The first needs a careful causal design, while the second calls for a forecasting or predictive model.

After formulating the question, a process is established that covers everything from the question to a final analytical product like a dashboard, report, or simply the knowledge produced that contributes to decision-making.

In this process, an analytical dataset is generally built to serve as a source for subsequent analysis. Then, this dataset is explored to understand the data, model it mathematically, or perform transformations on it. In other words, the analysis process begins by applying techniques suited to the business question's needs.

Finally, if you need to train a machine learning model, you'll make certain transformations to prepare the dataset for training, so it's considered model-ready. After training, results are delivered through a report, API, or by deploying the model in the infrastructure to make predictions, for example.

In this process, various roles collaborate, such as Data Analysts, who answer business questions related to Analytics, while Data Scientists formulate hypotheses and develop models to describe data or make predictions. Analytics Engineers focus on building analytical datasets, and Data Engineers construct the pipelines and infrastructure that supply them.

Also, when a machine learning model needs to be integrated into an application, Machine Learning Engineers are involved.

Analytical Datasets

An analytical dataset is prepared for a defined analysis. It isn't a random collection of files: it has a known schema, grain, population, time period, quality criteria, and lineage. The team selects data because it is relevant to the question rather than including every available field.

In the university use case, to study if taxi service improves attendance, a dataset could be built with records of trips and class attendance of students who have or haven't traveled, allowing for a comparison of their attendance statistics.

On the other hand, to predict transportation demand, it would be more appropriate to build another dataset that integrates travel history with class schedules, the academic calendar, or weather conditions. Thus, although both sets may reuse some data sources, their structure, granularity, and quality rules would differ, as each must be designed to address the specific business question.

The design of how a dataset should be is the responsibility of a Data Analyst or Data Scientist, while the implementation of transformations and other processes necessary for its construction is carried out by Analytics Engineers. But if data from multiple sources need to be integrated, a Data Engineer handles this task, as we have seen.

These datasets are usually materialized in the form of tables in a Data Warehouse, Data Lake, or as column-oriented files like Apache Parquet.

Exploratory Data Analysis

Most analyses include Exploratory Data Analysis (EDA) because you rarely understand a new dataset perfectly at the start.

EDA examines the dataset's distributions, patterns, relationships, and unusual values before the team draws conclusions or builds a model. It also reviews types, missing values, duplicates, quality limitations, and possible sources of bias.

Regarding exploration techniques, descriptive statistics and the creation of visualizations are usually key. For example, a Data Analyst might represent the number of enrollments paid per day, compare the payment methods used, and analyze when more incidents occur. This way, they could discover if any of the payment platforms or banks involved in the transactions have caused problems with enrollment payments at any point.

They might also observe phenomena such as students who pay earlier achieving better academic results, but that correlation wouldn't prove that paying in advance is the main cause. Still, exploration serves to generate this hypothesis and detect possible alternative explanations, but not to confirm a causal relationship on its own.

EDA is performed by both Data Analysts and Data Scientists, though in different ways, as analysts seek to make diagnoses, while scientists explore the data to decide how to model it.

The technologies they use for exploration are very diverse, from SQL for querying the dataset, Jupyter notebooks for more easily documenting Python code, to Python libraries like pandas, NumPy, SciPy, Matplotlib, and Seaborn. Other languages that also allow data exploration include R, Julia, or Scala.

Feature Engineering

After exploring the data, transformations are often applied to make them more useful depending on the intended purpose. If we view the data as a set of records where each takes values in a series of attributes called features, sometimes these features may be more or less useful for training a machine learning model or simply for understanding the data.

For example, if we have student records in the form (name, email, 1), having a feature with a fixed value of 1 doesn't contribute to an analysis unless it's a relevant feature that always takes the value 1 for some realistic reason. In this case, it would be ideal to remove the feature and keep only the most useful ones.

Feature Engineering transforms or derives model inputs so they represent the problem usefully. Techniques include normalization or standardization for scale-sensitive algorithms, data imputation for suitable missing values, encoding categories, and discretization. Each choice should follow the business meaning, model type, and evaluation plan rather than a fixed recipe.

For example, imagine the university wants to predict whether a student will finish the master's program. To do this, they have an analytical dataset with records whose features include class attendance, grades, and the number of accesses to the virtual campus, which will later be used to train a machine learning model for prediction.

For a model that is sensitive to feature scale, normalization may help because grades range from 0 to 10 while portal-access counts can reach thousands. Min-max scaling can map them to [0, 1], although other algorithms or scaling methods may be more suitable.

The team must also exclude information that wouldn't be available at prediction time. If a feature reveals the outcome directly or indirectly, data leakage can make evaluation look unrealistically good.

These transformations are usually performed by a Data Scientist, Analytics Engineers, Data Engineers, or a Machine Learning Engineer, primarily. All these roles use technologies like SQL, Apache Spark, or Python to perform them, though these aren't the only ones.

Experimentation

Many analyses test a hypothesis. If the team believes a feature doesn't improve a model, it can state that idea clearly and use experiments to compare a model trained with and without the feature.

Experimentation changes controlled parts of a dataset, method, or training process to test a hypothesis. The work is iterative: one result can reject the original idea or suggest a better question for the next experiment.

In this field, it's important to distinguish between two types of experimentation with different purposes. First, there's analytical experimentation, which is conducted on already collected data and focuses on comparing features, types of models, and training techniques to determine which combination of these elements best answers the business question.

For example, to predict if a student will complete their master's program, the university might start with a simple model using only grades and attendance. Then, they could run another experiment incorporating the number of virtual campus logins or try a different algorithm.

This way, they could determine if the change truly enhances predictive capability or merely increases model complexity.

Also, this model should be evaluated with data not used in its training. Otherwise, it might "cheat," performing well with training data but failing to "generalize" and achieve the same performance with real data.

Controlled experiments introduce a change and compare outcomes between a treatment group that receives it and a control group that doesn't. Random assignment, when feasible and ethical, helps make the groups comparable.

For instance, to see if a taxi service improves attendance, the university could gradually introduce it to a small group of students, provided it's ethically and legally appropriate. Here, the hypothesis would be that the service improves attendance, tested by analyzing treatment data from students who received the service against control data from those who didn't, using metrics like the percentage of classes attended.

Experiments should be reproducible. Teams can version code and configuration with Git, while platforms like MLflow record runs, parameters, metrics, and artifacts.

Data Scientists usually formulate hypotheses and design model experiments with Data Analysts and domain experts. Machine Learning Engineers may help make the training and evaluation workflow reliable at production scale.

Model-Ready Data

Analytical datasets are often ready for analysis but this isn't always the case. If your goal is to train a machine learning model to make predictions, then the dataset must meet additional conditions.

To train a model, the data needs to be model-ready: prepared for the selected algorithm, evaluation design, and production use. A supervised-learning dataset needs a target variable that records the outcome to learn. Unsupervised methods can work without labels, so model-ready requirements depend on the task.

For example, to predict whether a student will leave a master's program, historical training records need an outcome label such as (student_reference, enrolled_subjects, withdrew). The team should exclude direct identifiers such as names from model features unless there's a justified need, and it must review whether the proposed prediction is fair and appropriate to use.

The team also separates data for training, validation, and final testing as the evaluation design requires. It develops the model without using the held-out test data for decisions, then uses that test set for an honest estimate of performance on unseen cases. For time-based predictions, the split should also respect chronology.

Finally, model-ready also implies that the data is representative of the target concept we want the model to "learn." For example, if we train a model to predict master's program dropout using only data from those who have dropped out, it likely won't learn the patterns indicating when someone doesn't drop out, making the dataset unrepresentative.

Thus, ensuring datasets are model-ready is the responsibility of Data Engineers, Data Scientists, and Machine Learning Engineers who may use them.

Analytical Product Delivery

Analysis creates value only when its results reach the right people or systems in a usable form. If the university uses a model to identify unusual exam activity, for example, it should treat the output as a signal for authorized human review rather than proof of misconduct.

Analytical Product Delivery provides the right consumption channel for each result. That channel might be a report, dashboard, alert, file, API, or prediction embedded in an application.

For instance, the university could deliver attendance analysis through a report or dashboard. A carefully governed model that estimates withdrawal risk might provide limited alerts through an internal API to an authorized support team, which would review the context before offering help. The channel and controls should match the intended use and potential impact.

Relevant practices in Analytical Product Delivery include defining the consumers of the results, their update frequency, and quality metrics. All this is documented along with data sources and other aspects, and the delivery mechanisms are monitored.

In the example, a dashboard with attendance analysis would have the rectorate and master's coordinators as consumers, updating with new data monthly. Meanwhile, the dropout prediction model would deliver its alerts to an academic officer via an API, even if this officer accesses it with an application.

The delivery is coordinated by the Data Product Owner or Product Manager, while technical teams with professionals like Analytics Engineers, Software Engineers, or BI Developers are responsible for implementing all the result delivery mechanisms.

Data Products

An analytical result isn't automatically a product. A Data Product packages governed data with a way for defined consumers to use it and an operating model that keeps it useful over time.

It may take the form of a dataset, API, dashboard, or another interface. A dashboard or file alone isn't necessarily a Data Product: it needs a clear purpose, known consumers, ownership, documentation, and defined quality and service expectations.

In our focused use case, the university could create a Mobility Eligibility Data Product. It would combine only the approved enrollment, in-person schedule, distance, and eligibility attributes needed for the transportation benefit. An API could return an eligibility decision and its effective date to the student portal, while a separate governed dataset could provide aggregated service metrics.

Keeping this product narrow avoids exposing a complete student profile to consumers that don't need it.

Product Characteristics

In this context, managing a Data Product should be done just like a commercial product, hence the need to define its consumers and those responsible, and to ensure its quality and availability.

But in the realm of data, there are certain fundamental characteristics for any product:

  • Discoverable: It must be accessible through a data catalog or the appropriate tool.

  • Understandable: The data schema, its semantics, and all aspects that facilitate its comprehension and traceability, such as lineage, must be documented.

  • Reliable: Quality and availability are measured against clear expectations, with monitoring and a response process when the product misses them.

  • Secure: Access controls are implemented, and the exposure of personal data is minimized.

  • Interoperable: The data should be able to be integrated and function correctly in other systems.

  • Stable: This means the data shouldn't undergo frequent changes in its schema, properties, or consumption methods.

A Data Contract can formalize important parts of the product interface, such as schema, semantics, quality rules, and update frequency. The product also needs documentation for ownership, access, support, lifecycle, and consumer expectations.

Ownership and Lifecycle

No single role builds a Data Product alone. The Data Product Owner works with consumers, defines requirements, and sets objectives based on expected value.

On a technical level, there are Data Engineers, Analytics Engineers, or Platform Engineers, among others, who operate the infrastructure for storing and analyzing data, generating the results that become a product.

The lifecycle includes identifying consumer needs, defining the product and its contract, building and releasing it, monitoring service and data quality, improving it, and eventually retiring it.

Adoption is one sign of success, but it isn't enough by itself. The product should help consumers achieve a valuable outcome while maintaining quality, availability, security, and sustainable operating cost.

Data Management Organization

We've covered many capabilities, technologies, and roles. The Data Management Organization defines how these people work together, make decisions, and resolve issues across the lifecycle.

Its operating model assigns authority and responsibility, sets forums and workflows, and gives teams a consistent way to resolve problems and deliver value.

Operating Model

An operating model organizes decision-making and delivery. In a centralized model, one data team handles most of the work. This can improve consistency, but the team may become distant from domain knowledge or turn into a bottleneck.

Another type of operating model is decentralized, where each department or area of the organization manages the data within its domain, increasing autonomy but at the cost of a higher risk of inconsistencies and data silos, making global decision-making more difficult.

Data silos refer to sets of information isolated within an area or system, making them inaccessible or very difficult to reach for the rest of the organization.

Many organizations use a hybrid or federated model, which seeks to combine the advantages of both approaches. Here, each domain maintains a certain degree of autonomy over its data and is responsible for its quality, documentation, and use, while a central unit establishes governance principles, standards, and policies that must be respected throughout the organization.

For example, a hybrid organizational model at a university could have a central Data Management Office led by the CDO, while different domains like Academic Activity, Finance, or Mobility would have their own Data Owners, Data Stewards, and technical teams. If multiple domains need to collaborate, a Data Governance Council could assist in decision-making related to this collaboration.

Roles and Collaboration

The main roles in this context have already been mentioned. But regarding collaboration among them, it's crucial that their responsibilities are clearly defined and documented. This can be formalized through documentation, tools like a RACI matrix, Data Contracts, Governance Charters, or by setting up workflows.

For proper coordination, technologies like Git repositories are used to collaboratively version their work, data catalogs, platforms similar to Jira for communication, and observability tools. But technology doesn't replace the need for authority, communication, and clear responsibilities.

Data Management Maturity

Organizations differ in how consistently they apply these capabilities. Data Management Maturity describes how well practices are embedded, measured, governed, and aligned with organizational goals.

For example, an organization with low maturity would manage data with isolated and ad-hoc actions based on arising needs. As maturity increases, processes and management practices begin to be documented to become standardized, governed, and properly automated. At the highest levels of maturity, a managed approach is adopted, where the management strategy is controlled through quality metrics, audits, and formal risk management.

Maturity focuses not only on the technical aspect but also on the ability to coordinate personnel, their responsibilities, and the tools they use to achieve sustainable results aligned with the organization's strategy.

Maturity Levels

One illustrative maturity model uses the following levels:

  • Level 0 – No Capability: There are no organized practices for managing data. Actions are taken as deemed appropriate at the moment.

  • Level 1 – Initial: Management is assigned to specific professionals, but there's no control over individual actions or collaboration methods.

  • Level 2 – Managed: Processes, roles, and tools begin to be documented to facilitate the replication and automation of management tasks.

  • Level 3 – Defined: Policies and standards are formalized and unified across the organization, ensuring all teams work in a coordinated and scalable manner.

  • Level 4 – Measured: Management is controlled more deeply through audits and metrics to evaluate performance and actively mitigate risks.

  • Level 5 – Optimized: Teams use measurements, feedback, and appropriate automation to improve management continuously and reduce problems before they affect consumers.

Assessment and Roadmap

To determine the maturity level and enhance it within your organization, your team can use a Data Management Maturity Assessment.

This process begins by defining which data domains and management capabilities are to be evaluated. Evidence is then gathered to analyze the maturity level achieved with these capabilities, examining what is documented, which policies are followed, and so on.

By comparing with a target maturity level, a roadmap is developed to reach it, with steps that can vary significantly depending on the specific organization and its current level.

This process is led by the CDO or the Data Governance Office, with participation from Data Owners, Data Stewards, and technical teams.

For example, at the university, the Mobility domain would be at level 1 if student eligibility for the service were reviewed manually and depended on specific individuals' knowledge. At level 2, responsibilities would be assigned, documentation on the concept of eligibility would begin, and basic validations would be automated.

At level 3, Data Products could unify access to selected mobility information under shared rules. At level 4, dashboards could track quality, availability, usage, cost, fairness, and incidents. At level 5, teams would automate low-risk work where appropriate, keep human review and appeal paths for consequential eligibility decisions, and improve the service continuously through metrics and user feedback.

But the goal doesn't have to be reaching level 5 in all capabilities. The university might require high maturity in security and quality capabilities that protect personal data, while a more experimental analysis of classroom usage that doesn't involve personal data might have a lower target.

Conclusions

The Data Management Ecosystem full diagram. Image by author.

Throughout this book, we've treated Data Management as a coordinated set of capabilities that helps an organization capture, integrate, protect, understand, and use data throughout its lifecycle.

The wider university ecosystem shows the scale of a real organization, while our admissions, academic-activity, and transportation examples make the connections concrete. Even a controlled transportation benefit requires much more than a database: it needs governance, quality, privacy, integration, reliable operations, and careful analysis.

Data doesn't generate value automatically. It becomes useful when people give it context, protect it, make it available to the right consumers, and connect it to a real goal. Technology is the means, not the objective. Databases, pipelines, dashboards, models, and Data Products matter only when they solve a genuine need.

]]> <![CDATA[ How to Build AI Applications That Switch Models Automatically ]]> https://www.freecodecamp.org/news/build-ai-applications-that-switch-models-automatically/ 6a69c635b68d550a815570fe Wed, 29 Jul 2026 09:21:57 +0000 Large Language Models (LLMs) have fundamentally changed how we build modern software.

But relying on a single AI model for every user request creates serious production risks. API outages happen. Proprietary models can be expensive for simple tasks. And cheaper open-source models might struggle with complex logical reasoning.

When my team and I built an enterprise-grade AI engine for our customer support platform, we relied on a single top-tier model for everything.

Within a month, we faced two massive issues: a widespread API outage completely froze our app, and our monthly API bill rose because we used expensive reasoning models to answer simple FAQs.

To fix this, I built a resilient, multi-model orchestrator. In this guide, you'll learn how to build an intelligent, multi-tiered AI application using Python that routes prompts dynamically and handles model fallbacks automatically.

Prerequisites and Environment Setup

To follow along with this tutorial, you should have the following setup:

  • Basic proficiency with Python and asynchronous programming.

  • Python 3.9 or higher installed on your system.

  • A code editor such as Visual Studio Code.

  • API keys for at least two model providers (for example, OpenAI and Anthropic), or local models running via Ollama.

Package Installation

Open your terminal and install the required dependencies:

pip install openai anthropic python-dotenv pydantic

Local Directory Structure

Organize your project directory like this to keep your code clean:

ai-model-router/

│

├── .env

├── README.md

└── app.py

Environment Configuration

Create a .env file in the root of your project directory and add your credentials:

Ini, TOML

OPENAI_API_KEY=your_openai_api_key_here ANTHROPIC_API_KEY=your_anthropic_api_key_here ENVIRONMENT=development

The Problem with Single-Model Architectures

A flow diagram illustrating a single-model AI architecture processed by one language model, creating a single point of failure and limiting cost optimization.

If you route every query to a flagship model like GPT-4o or Claude 3.5 Sonnet, you'd be overspending on simple tasks. Conversely, if you route everything to a smaller, faster model like GPT-4o-mini or Claude 3.5 Haiku to save money, your system will fail when users submit complex code-generation or analytical tasks.

On top of cost concerns, single-model systems suffer from single points of failure. When an API provider goes down or rate-limits your account, your entire application crashes.

To solve this, you need an orchestration layer that evaluates prompt complexity before invoking an LLM, routes the request to the most cost-effective model, and falls back to a secondary provider if the primary provider fails.

Understanding the Dynamic Model Routing Lifecycle

Flow diagram of a dynamic multi-model AI system with intelligent model selection and automatic failover.

Here's how a user request journeys through a dynamic multi-model system:

First, you have the complexity analysis. The system inspects the incoming prompt using lightweight metrics to assign a task tier (Simple, Medium, or Complex).

Second, you have the model routing. The system maps the tier to the appropriate model (for example, lightweight tasks go to Haiku/Mini while heavy reasoning goes to Sonnet/GPT-4o).

You also have an automatic fallback: if the primary provider times out or throws an API error, the system automatically redirects the query to an equivalent fallback model.

Step 1: Implementing Tier 1 – Prompt Complexity & Intent Analysis

First, you need a deterministic, fast way to classify prompts without making an expensive API call just to decide which model to use.

Before spending money on an LLM API call just to figure out what the user wants, we can look at the text directly in code. Think of this step as a smart gatekeeper. By checking simple things like text length, code snippets, or tricky keywords, we can figure out how hard the task is in milliseconds and for free.

Here's how we set up our classification rules inside app.py:

import re
from enum import Enum
from pydantic import BaseModel


class TaskComplexity(Enum):
    SIMPLE = "simple"      # FAQs, short summaries, basic translation
    MEDIUM = "medium"      # Standard text generation, content rewriting
    COMPLEX = "complex"    # Code writing, math logic, structural analysis


class PromptAnalyzer:
    def __init__(self):
        # Regex patterns indicative of complex tasks
        self.complex_keywords = [
            r"\brefactor\b",
            r"\bdebug\b",
            r"\bwrite code\b",
            r"\banalyze\b",
            r"\balgorithm\b",
            r"\barchitecture\b",
        ]

    def analyze_complexity(self, prompt: str) -> TaskComplexity:
        """
        Evaluates input text deterministically to output
        a TaskComplexity rating.
        """
        normalized = prompt.lower().strip()
        word_count = len(normalized.split())

        # Check for code blocks or complex request patterns
        contains_code = "```" in prompt
        has_complex_keyword = any(
            re.search(pattern, normalized)
            for pattern in self.complex_keywords
        )

        if contains_code or has_complex_keyword or word_count > 300:
            return TaskComplexity.COMPLEX
        elif word_count > 80:
            return TaskComplexity.MEDIUM
        else:
            return TaskComplexity.SIMPLE


# Example Usage
if __name__ == "__main__":
    analyzer = PromptAnalyzer()

    test_prompt = (
        "Write a Python script that implements a trie "
        "data structure with autocomplete."
    )

    complexity = analyzer.analyze_complexity(test_prompt)
    print(f"Prompt Complexity Tier: {complexity.value}")

Breaking Down the Code Logic for Tier 1

  • TaskComplexity Enum: Defines explicit categories for incoming requests (SIMPLE, MEDIUM, COMPLEX), giving us type safety across our pipeline.

  • Keyword Matching: The PromptAnalyzer class sets up regex patterns looking for action words like refactor, debug, or algorithm that signal a heavy reasoning task.

  • Deterministic Rules in analyze_complexity:

  • Formatting & Length Check: We clean the string, check for Markdown code blocks (```), and calculate word counts.

  • Tier Allocation:

    • If the prompt contains code blocks, trigger words, or exceeds 300 words, it immediately escalates to COMPLEX.

    • If it is between 80 and 300 words without code keywords, it maps to MEDIUM.

    • Anything shorter defaults to SIMPLE.

Running this snippet with a complex query checks the text, spots "write code," and outputs:

Prompt Complexity Tier: complex

Step 2: Implementing Tier2– Dynamic Model Routing Logic

Now that we can successfully label a prompt as simple, medium, or complex, we need a rulebook to decide which AI model actually handles it.

This layer maps each complexity tier to a primary model and a secondary fallback model. For instance, simple queries route to budget models (gpt-4o-mini), while complex requests route to heavyweights (claude-3-5-sonnet).

Add this configuration also:

class ModelConfig(BaseModel):
    provider: str
    model_name: str


class ModelRouter:
    def __init__(self):
        # Map task complexity tiers to primary and fallback models
        self.routing_table = {
            TaskComplexity.SIMPLE: {
                "primary": ModelConfig(
                    provider="openai",
                    model_name="gpt-4o-mini",
                ),
                "fallback": ModelConfig(
                    provider="anthropic",
                    model_name="claude-3-5-haiku-20241022",
                ),
            },
            TaskComplexity.MEDIUM: {
                "primary": ModelConfig(
                    provider="openai",
                    model_name="gpt-4o-mini",
                ),
                "fallback": ModelConfig(
                    provider="anthropic",
                    model_name="claude-3-5-haiku-20241022",
                ),
            },
            TaskComplexity.COMPLEX: {
                "primary": ModelConfig(
                    provider="anthropic",
                    model_name="claude-3-5-sonnet-20241022",
                ),
                "fallback": ModelConfig(
                    provider="openai",
                    model_name="gpt-4o",
                ),
            },
        }

    def get_models_for_tier(
        self, complexity: TaskComplexity
    ) -> tuple[ModelConfig, ModelConfig]:
        """
        Returns the primary and fallback models for a given
        task complexity tier.
        """
        config = self.routing_table[complexity]
        return config["primary"], config["fallback"]

Breaking Down the Code Logic for Tier 2

  • ModelConfig Schema: Uses Pydantic to ensure every model definition includes both a provider (for example, "openai") and a specific model_name string.

  • self.routing_table Mapping: This dictionary acts as our single source of truth for model assignments:

    • SIMPLE & MEDIUM Tiers: Primary target is gpt-4o-mini for high-throughput, low-cost output. If OpenAI fails, it falls back to Anthropic's claude-3-5-haiku-20241022.

    • COMPLEX Tier: Primary target flips to claude-3-5-sonnet-20241022 for top-tier code generation and reasoning, with gpt-4o as the backup.

  • get_models_for_tier: A helper function that takes the analyzed tier and safely returns a tuple of (PrimaryModel, FallbackModel).

Step 3: Implementing Tier3 – Automatic Fallbacks

Even the best AI providers experience downtime, rate limits, or unexpected timeouts. A production-ready app can't just throw an error screen at the user when this happens. We need an execution engine that attempts to call the primary model provider and automatically catches errors. If anything goes wrong, it instantly pivots to the secondary fallback model without breaking the workflow .

Add the execution engine code to the script:

import os
import time

from anthropic import Anthropic, APIError as AnthropicAPIError
from dotenv import load_dotenv
from openai import OpenAI, APIError as OpenAIAPIError

load_dotenv()


class ResilientModelEngine:
    def __init__(self):
        self.openai_client = OpenAI(
            api_key=os.getenv("OPENAI_API_KEY", "dummy")
        )
        self.anthropic_client = Anthropic(
            api_key=os.getenv("ANTHROPIC_API_KEY", "dummy")
        )

    def _call_openai(self, model: str, prompt: str) -> str:
        response = self.openai_client.chat.completions.create(
            model=model,
            messages=[
                {
                    "role": "user",
                    "content": prompt,
                }
            ],
            timeout=10.0,
        )
        return response.choices[0].message.content

    def _call_anthropic(self, model: str, prompt: str) -> str:
        response = self.anthropic_client.messages.create(
            model=model,
            max_tokens=1024,
            messages=[
                {
                    "role": "user",
                    "content": prompt,
                }
            ],
            timeout=10.0,
        )
        return response.content[0].text

    def execute_provider_call(
        self,
        config: ModelConfig,
        prompt: str,
    ) -> str:
        """
        Dispatches prompt execution to the correct provider SDK.
        """
        if config.provider == "openai":
            return self._call_openai(config.model_name, prompt)
        elif config.provider == "anthropic":
            return self._call_anthropic(config.model_name, prompt)
        else:
            raise ValueError(
                f"Unsupported provider: {config.provider}"
            )

    def execute_with_fallback(
        self,
        primary: ModelConfig,
        fallback: ModelConfig,
        prompt: str,
    ) -> tuple[str, str]:
        """
        Attempts execution on the primary model and switches to the
        fallback model if the primary provider fails.

        Returns:
            tuple[str, str]: (Response text, Model used)
        """
        try:
            print(
                f"[Attempt] Calling Primary Provider: "
                f"{primary.provider} ({primary.model_name})"
            )

            result = self.execute_provider_call(primary, prompt)

            return result, (
                f"{primary.provider}:{primary.model_name}"
            )

        except (
            OpenAIAPIError,
            AnthropicAPIError,
            Exception,
        ) as e:
            print(f"[WARNING] Primary call failed due to: {e}")

            print(
                f"[Fallback] Switching to Secondary Provider: "
                f"{fallback.provider} ({fallback.model_name})"
            )

            try:
                result = self.execute_provider_call(
                    fallback,
                    prompt,
                )

                return result, (
                    f"{fallback.provider}:"
                    f"{fallback.model_name} (Fallback)"
                )

            except Exception as fallback_error:
                raise RuntimeError(
                    "Both primary and fallback systems failed. "
                    f"Error: {fallback_error}"
                )

Breaking Down the Code Logic for Tier 3

Provider Clients (_call_openai & _call_anthropic): Helper methods wrap provider SDK calls, establishing a unified strict 10-second timeout. If an API hangs, it aborts fast so the fallback can kick in without making the user wait.

execute_provider_call Dispatcher: Acts as an abstraction bridge, matching the requested provider string to its respective API method.

execute_with_fallback Resiliency Logic: Executes the primary provider first inside a try block. Catches API errors, rate limits, or network timeouts via provider-specific exceptions (OpenAIAPIError, AnthropicAPIError). Logically redirects execution to the fallback provider inside the except block. Only raises an unrecoverable RuntimeError if both primary and fallback providers fail. If your primary provider encounters issues, your console tracks the recovery process transparently:

[Attempt] Calling Primary Provider: anthropic (claude-3-5-sonnet-20241022)

[WARNING] Primary call failed due to: Connection timeout

[Fallback] Switching to Secondary Provider: openai (gpt-4o)

Combining the Architecture into a Unified Execution Pipeline

Now you can combine all three layers into a unified pipeline.

Unified pipeline for executing AI tasks across multiple models and workflows.

Complete your app.py script with this orchestration class:

class SmartAIEngine:
    def __init__(self):
        self.analyzer = PromptAnalyzer()
        self.router = ModelRouter()
        self.executor = ResilientModelEngine()

    def process_request(self, user_prompt: str) -> dict:
        print("\n==========================================")
        print("Processing New Request")
        print("==========================================")

        # Step 1: Analyze prompt complexity
        complexity = self.analyzer.analyze_complexity(
            user_prompt
        )
        print(
            f"[Step 1] Prompt classified as: "
            f"{complexity.value.upper()}"
        )

        # Step 2: Determine routing target
        primary_model, fallback_model = (
            self.router.get_models_for_tier(
                complexity
            )
        )

        print(
            f"[Step 2] Selected Primary: "
            f"{primary_model.model_name}"
        )

        # Step 3: Execute request with resilient fallbacks
        response_text, executed_model = (
            self.executor.execute_with_fallback(
                primary=primary_model,
                fallback=fallback_model,
                prompt=user_prompt,
            )
        )

        return {
            "status": "success",
            "complexity_tier": complexity.value,
            "model_used": executed_model,
            "response": response_text,
        }


# Execution Pipeline Test
if __name__ == "__main__":
    engine = SmartAIEngine()

    # Query 1: Simple task
    simple_query = (
        "What is the capital of Japan? "
        "Answer in one word."
    )

    result_1 = engine.process_request(
        simple_query
    )

    print(f"Model Used: {result_1['model_used']}")
    print(f"Response: {result_1['response']}")

    # Query 2: Complex task
    complex_query = (
        "Write a Python function to debug a "
        "memory leak in a multithreaded "
        "application."
    )

    result_2 = engine.process_request(
        complex_query
    )

    print(f"Model Used: {result_2['model_used']}")
    print(
        f"Response Snippet: "
        f"{result_2['response'][:100]}..."
    )

Breaking Down the Code Logic

  • Unified Orchestration (SmartAIEngine): Initializes all three modular components—PromptAnalyzer, ModelRouter, and ResilientModelEngine—as instance properties.

  • The Pipeline Steps:

    • Analyze: Evaluates the prompt string offline to get the complexity tier.

    • Route: Resolves primary and secondary model pairs based on that tier.

    • Execute: Calls the models resiliently and catches failure scenarios.

  • Normalized Response Payload: Wraps execution details into a consistent output dictionary, keeping track of model usage, complexity categorization, and output text.

Lessons Learnt from Dynamic Model Switching in Production

Building a dynamic AI routing system taught our team critical lessons about enterprise LLM architectures:

First, keep classification light. Never use a large LLM call to classify prompts for small tasks. Use regex, keyword matching, and token-length rules. Your classifier should run in under 5 milliseconds.

Second, normalize system outputs. Different model providers structure outputs differently. Make sure your application wraps responses in a consistent schema before returning data to the user interface.

Third, set a tight timeout. Provider APIs often hang instead of throwing immediate errors. Set tight request timeouts (5 to 10 seconds) on your primary model calls so your fallback triggers quickly without frustrating the end user.

And finally, track usage metrics. Log every routing decision, model fallback, and cost delta. This data will reveal whether your complexity thresholds are properly tuned over time.

Conclusion

As AI applications scale, relying on a single, monolithic LLM becomes unsustainable. Intelligent model routing allows you to balance performance, latency, and cost without sacrificing response quality.

By decoupling your application from specific model providers and introducing automated routing layers, input evaluation, provider abstraction, and resilient fallbacks, you can build production AI systems that are cost-effective, fast, and resilient.

As you deploy your own applications, treat LLM providers as dynamic utilities. Use lightweight models for everyday processing, reserve flagship models for complex tasks, and handle provider transitions cleanly in code.

Thank You for Reading!

I hope this article has given you a practical understanding of how multi-model orchestrators and dynamic routing work in real-world applications and how you can begin implementing them in your own projects.

If you'd like to discuss AI engineering, Agentic AI, LLMs, RAG, MLOps, enterprise AI architecture, or AI governance, feel free to follow, like, share, and connect with me:

]]> <![CDATA[ How to Clean Time Series Data in Python ]]> https://www.freecodecamp.org/news/how-to-clean-time-series-data-in-python/ 6a0ad57ee4a28cf570ec90ac Mon, 18 May 2026 09:01:50 +0000 Real-world time series data is rarely clean. Sensors drop out, systems clock-drift, pipelines duplicate records, and manual data entry introduces mistakes. By the time a dataset reaches your notebook, it has passed through collection, transmission, and storage, each step a potential source of corruption.

Cleaning time series data is harder than cleaning tabular data because time is a structural constraint. You can't shuffle rows or impute a missing value with a column mean without pulling future data into a past observation. Every cleaning decision has to respect temporal ordering, or it breaks the integrity of everything built on top of it.

This guide walks through the full cleaning pipeline in Python: from raw data arrival to a dataset ready for feature engineering or modelling. We'll cover missing value detection and imputation, outlier identification and treatment, duplicate handling, frequency alignment, noise smoothing, and schema validation, applied to sample sensor data throughout.

You can get the Colab notebook from GitHub and follow along.

Prerequisites

To follow along to this guide, you'll need to be:

  • Comfortable working with Python and pandas DataFrames

  • Familiar with time-indexed data

  • Aware of what feature engineering and machine learning modelling involve at a high level

We'll use pandas and numpy for data manipulation, scipy for signal smoothing and statistical tests, scikit-learn for anomaly detection, and statsmodels for seasonal decomposition. Install them before running any code in this guide:

pip install pandas numpy scipy scikit-learn statsmodels

Table of Contents

How to Audit Your Time Series Before Cleaning It

The first rule of data cleaning is: look before you cut. Before imputing, smoothing, or dropping anything, you need a complete picture of what's wrong and where.

A good audit covers the following:

  • The time index: Is it regular? Are there gaps?

  • Missing value distribution: Are missing values random or clustered?

  • Value range: Are there obvious gaps or sensor failures?

  • Duplicate timestamps

Let's spin up a sample dataset (with some of the above problems):

# Simulate one week of smart grid voltage readings (hourly)
# with realistic problems injected
periods = 168
index = pd.date_range("2024-06-01", periods=periods, freq="H")

voltage = (
    230.0
    + 3.5 * np.sin(2 * np.pi * np.arange(periods) / 24)
    + np.random.normal(0, 1.2, periods)
)

# Inject problems
voltage[14:17] = np.nan          # sensor dropout: 3 consecutive missing
voltage[42] = np.nan             # isolated missing
voltage[78] = 312.4              # spike outlier
voltage[101:104] = np.nan        # another dropout
voltage[130] = 187.2             # dip outlier

series = pd.Series(voltage, index=index, name="voltage_v")

# --- Audit ---
print("=== TIME SERIES AUDIT ===")
print(f"Period:        {series.index.min()} → {series.index.max()}")
print(f"Observations:  {len(series)}")
print(f"Expected freq: {pd.infer_freq(series.index)}")
print(f"\nMissing values: {series.isna().sum()} ({series.isna().mean()*100:.1f}%)")
print(f"Value range:    [{series.min():.2f}, {series.max():.2f}]")
print(f"Mean ± Std:     {series.mean():.2f} ± {series.std():.2f}")

# Identify consecutive missing runs
missing_mask = series.isna()
missing_runs = []
run_start = None
for i, (ts, is_missing) in enumerate(missing_mask.items()):
    if is_missing and run_start is None:
        run_start = ts
    elif not is_missing and run_start is not None:
        missing_runs.append((run_start, missing_mask.index[i - 1]))
        run_start = None

print(f"\nMissing runs ({len(missing_runs)} total):")
for start, end in missing_runs:
    print(f"  {start} → {end}")

Output:

=== TIME SERIES AUDIT ===
Period:        2024-06-01 00:00:00 → 2024-06-07 23:00:00
Observations:  168
Expected freq: h

Missing values: 7 (4.2%)
Value range:    [187.20, 312.40]
Mean ± Std:     230.22 ± 7.81

Missing runs (3 total):
  2024-06-01 14:00:00 → 2024-06-01 16:00:00
  2024-06-02 18:00:00 → 2024-06-02 18:00:00
  2024-06-05 05:00:00 → 2024-06-05 07:00:00

This audit gives you a map of the damage before you start cleaning. The key task is distinguishing between isolated missing values, which are imputable with local context, and missing long runs, which may need a different strategy or flagging for downstream consumers.

How to Reindex to a Canonical Frequency

Before imputing missing values, you need to confirm your time index is actually regular. A common problem in ingested time series is that missing timestamps are simply absent rather than represented as NaN rows — which means a .fillna() call will never find them.

# Simulate a sensor feed with missing timestamps (not just missing values)
irregular_index = index.delete([14, 15, 16, 42, 101, 102, 103])
irregular_series = series.dropna().reindex(irregular_index)

print(f"Original length:   {len(series)}")
print(f"Irregular length:  {len(irregular_series)}")
print(f"Inferred freq:     {pd.infer_freq(irregular_series.index)}")  # None = irregular

# Reindex to the full canonical hourly grid
canonical_index = pd.date_range(
    start=irregular_series.index.min(),
    end=irregular_series.index.max(),
    freq="H"
)

reindexed = irregular_series.reindex(canonical_index)

print(f"\nAfter reindex:")
print(f"Length:         {len(reindexed)}")
print(f"Missing values: {reindexed.isna().sum()}")
print(f"Inferred freq:  {pd.infer_freq(reindexed.index)}")

Output:

Original length:   168
Irregular length:  161
Inferred freq:     None

After reindex:
Length:         168
Missing values: 7
Inferred freq:  h

pd.infer_freq returning None is your signal that the index has gaps. After reindexing to the canonical grid, missing timestamps become explicit NaN rows, and now your imputation logic can find them.

How to Handle Missing Values

Not all missing values should be handled the same way. A single isolated missing reading in a smooth signal is best filled with interpolation. A 3-hour sensor dropout in a volatile signal, however, might be better flagged than fabricated. Strategy should match both gap length and signal behavior.

Forward Fill — For Step-Function Signals

Forward fill is appropriate when the variable holds its last known value until something changes it — a machine state, a setpoint, a categorical flag.

# Equipment operating mode — a step signal
mode_data = pd.Series(
    ["running", "running", np.nan, np.nan, "idle", "idle", np.nan, "running"],
    index=pd.date_range("2024-06-01", periods=8, freq="H"),
    name="operating_mode"
)

filled_mode = mode_data.ffill()
print(pd.DataFrame({"original": mode_data, "ffill": filled_mode}))

Output:

                    original    ffill
2024-06-01 00:00:00  running  running
2024-06-01 01:00:00  running  running
2024-06-01 02:00:00      NaN  running
2024-06-01 03:00:00      NaN  running
2024-06-01 04:00:00     idle     idle
2024-06-01 05:00:00     idle     idle
2024-06-01 06:00:00      NaN     idle
2024-06-01 07:00:00  running  running

Time-Weighted Interpolation — For Continuous Signals

For continuous sensor readings, linear interpolation weighted by time handles irregular gaps correctly because it doesn't assume equal spacing.

# Fill the voltage series using time-based interpolation
voltage_clean = reindexed.interpolate(method="time")

# Compare original vs filled around the first gap
gap_window = voltage_clean["2024-06-01 12:00":"2024-06-01 18:00"]
original_window = reindexed["2024-06-01 12:00":"2024-06-01 18:00"]

comparison = pd.DataFrame({
    "original":     original_window,
    "interpolated": gap_window.round(3),
    "was_missing":  original_window.isna(),
})
print(comparison)

Output:

                       original  interpolated  was_missing
2024-06-01 12:00:00  230.290355       230.290        False
2024-06-01 13:00:00  226.798197       226.798        False
2024-06-01 14:00:00         NaN       226.848         True
2024-06-01 15:00:00         NaN       226.897         True
2024-06-01 16:00:00         NaN       226.947         True
2024-06-01 17:00:00  226.996356       226.996        False
2024-06-01 18:00:00  225.410371       225.410        False

Seasonal Decomposition Imputation — For Long Gaps

For gaps longer than a few steps in a seasonal signal, interpolating across the gap ignores the seasonal pattern. A better approach is to decompose the series, impute each component separately, then reconstruct.

from statsmodels.tsa.seasonal import seasonal_decompose

# Use a longer series for decomposition (needs enough periods)
long_voltage = pd.Series(
    230.0
    + 3.5 * np.sin(2 * np.pi * np.arange(336) / 24)
    + np.random.normal(0, 1.0, 336),
    index=pd.date_range("2024-06-01", periods=336, freq="H")
)

# Inject a 6-hour gap
long_voltage.iloc[100:106] = np.nan

# Interpolate first to give decompose a complete series to work with
temp_filled = long_voltage.interpolate(method="time")
decomp = seasonal_decompose(temp_filled, model="additive", period=24)

# Reconstruct: trend + seasonal + zero residual for missing positions
reconstructed = long_voltage.copy()
missing_idx = long_voltage[long_voltage.isna()].index
reconstructed[missing_idx] = (
    decomp.trend[missing_idx].fillna(method="ffill")
    + decomp.seasonal[missing_idx]
)

print(f"Missing before: {long_voltage.isna().sum()}")
print(f"Missing after:  {reconstructed.isna().sum()}")
print("\nFilled values at gap:")
print(reconstructed[missing_idx].round(3))

Output:


                       original  interpolated  was_missing
2024-06-01 12:00:00  230.290355       230.290        False
2024-06-01 13:00:00  226.798197       226.798        False
2024-06-01 14:00:00         NaN       226.848         True
2024-06-01 15:00:00         NaN       226.897         True
2024-06-01 16:00:00         NaN       226.947         True
2024-06-01 17:00:00  226.996356       226.996        False
2024-06-01 18:00:00  225.410371       225.410        False

The seasonal decomposition imputation respects the time-of-day pattern. As you can see, the filled values aren't a flat line across the gap but follow the expected daily curve.

How to Detect and Handle Outliers

Outliers in time series are trickier than in tabular data because context matters. For example, an unusually high or low voltage might be a sensor spike or a genuine grid event. You need methods that use temporal context, not just global statistics.

Z-Score with Rolling Window

A global Z-score misses local anomalies in non-stationary series. A rolling Z-score flags values that are unusual relative to their local neighbourhood.

Note: A non-stationary series is a time series whose statistical properties—such as mean, variance, or trend—change over time instead of remaining constant.

window = 24  # 24-hour rolling window

roll_mean = voltage_clean.rolling(window, center=True, min_periods=1).mean()
roll_std  = voltage_clean.rolling(window, center=True, min_periods=1).std()

rolling_z = (voltage_clean - roll_mean) / roll_std

threshold = 3.0
outliers_z = rolling_z[rolling_z.abs() > threshold]

print(f"Rolling Z-score outliers detected: {len(outliers_z)}")
print(outliers_z.round(3))

Output:

Rolling Z-score outliers detected: 2
2024-06-04 06:00:00    4.646
2024-06-06 10:00:00   -4.484
Name: voltage_v, dtype: float64

Z-score outlier detection works best for approximately Gaussian (normal) distributions because it assumes the data is centered around a mean with symmetric spread measured by standard deviation.

IQR-Based Outlier Detection

The interquartile range (IQR) method is more robust for detecting outliers in non-Gaussian distributions. The interquartile range (IQR) is the difference between the third quartile (Q3) and the first quartile (Q1), representing the spread of the middle 50% of the data.

Q1 = voltage_clean.quantile(0.25)
Q3 = voltage_clean.quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

outliers_iqr = voltage_clean[
    (voltage_clean < lower_bound) | (voltage_clean > upper_bound)
]

print(f"IQR bounds: [{lower_bound:.2f}, {upper_bound:.2f}]")
print(f"Outliers detected: {len(outliers_iqr)}")
print(outliers_iqr.round(2))

Output:

IQR bounds: [220.16, 239.46]
Outliers detected: 2
2024-06-04 06:00:00    312.4
2024-06-06 10:00:00    187.2
Name: voltage_v, dtype: float64

Isolation Forest — For Multivariate Outlier Detection

When you have multiple sensors, an isolated reading on one channel might look normal, but its combination with readings from other channels reveals the anomaly. Isolation Forest handles this naturally.

# Build a multi-sensor DataFrame
np.random.seed(42)
n = 200

sensor_df = pd.DataFrame({
    "voltage_v":    230 + 3 * np.sin(2 * np.pi * np.arange(n) / 24) + np.random.normal(0, 1, n),
    "current_a":    15  + 0.8 * np.sin(2 * np.pi * np.arange(n) / 24) + np.random.normal(0, 0.3, n),
    "frequency_hz": 50  + np.random.normal(0, 0.05, n),
}, index=pd.date_range("2024-06-01", periods=n, freq="H"))

# Inject a multivariate anomaly — voltage drops, current spikes together
sensor_df.iloc[88, 0] = 194.2   # voltage dip
sensor_df.iloc[88, 1] = 28.7    # current surge (consistent with fault)

clf = IsolationForest(contamination=0.02, random_state=42)
sensor_df["anomaly_score"] = clf.fit_predict(sensor_df[["voltage_v", "current_a", "frequency_hz"]])

anomalies = sensor_df[sensor_df["anomaly_score"] == -1]
print(f"Anomalies detected: {len(anomalies)}")
print(anomalies[["voltage_v", "current_a", "frequency_hz"]].round(2))

Output:

Anomalies detected: 4
                     voltage_v  current_a  frequency_hz
2024-06-02 07:00:00     234.75      15.84         49.90
2024-06-04 06:00:00     233.09      15.82         50.15
2024-06-04 16:00:00     194.20      28.70         50.08
2024-06-06 05:00:00     235.09      15.41         49.91

In practice you'd follow up anomaly scores with domain-specific threshold rules.

Outlier Treatment

Once outliers are identified, you can handle them in several ways:

  • Cap them using Winsorization by limiting extreme values to a threshold.

  • Replace them with interpolated or estimated values.

  • Flag them so the model can handle them appropriately.

# Winsorize: cap at the IQR bounds
voltage_winsorized = voltage_clean.clip(lower=lower_bound, upper=upper_bound)

# Replace outliers with time-interpolated values
voltage_outlier_fixed = voltage_clean.copy()
voltage_outlier_fixed[outliers_iqr.index] = np.nan
voltage_outlier_fixed = voltage_outlier_fixed.interpolate(method="time")

print("Outlier treatment comparison:")
for ts in outliers_iqr.index:
    print(f"\n  {ts}")
    print(f"    Original:     {voltage_clean[ts]:.2f}")
    print(f"    Winsorized:   {voltage_winsorized[ts]:.2f}")
    print(f"    Interpolated: {voltage_outlier_fixed[ts]:.2f}")

Output:

Outlier treatment comparison:

  2024-06-04 06:00:00
    Original:     312.40
    Winsorized:   239.46
    Interpolated: 232.01

  2024-06-06 10:00:00
    Original:     187.20
    Winsorized:   220.16
    Interpolated: 231.43

Winsorization preserves the point but clips it to a plausible range — useful when you want to retain the information that something anomalous happened. Interpolation treats the outlier as if it were missing — better when you believe the reading is simply wrong.

How to Remove Duplicates

Duplicate timestamps are common when data pipelines retry on failure. Unlike tabular duplicates, time series duplicates aren't always identical, a retry might deliver a slightly different reading for the same timestamp.

# Inject duplicate timestamps with slightly different values (retry scenario)
dup_index = index.tolist()
dup_index.insert(20, index[20])  # exact duplicate timestamp
dup_index.insert(55, index[55])  # retry duplicate

dup_values = voltage_clean.tolist()
dup_values.insert(20, voltage_clean.iloc[20])
dup_values.insert(55, voltage_clean.iloc[55] + 0.7)  # slightly different value

dup_series = pd.Series(dup_values, index=pd.DatetimeIndex(dup_index), name="voltage_v")

print(f"Length with duplicates: {len(dup_series)}")
print(f"Duplicate timestamps:   {dup_series.index.duplicated().sum()}")

# Strategy 1: keep first (original reading)
dedup_first = dup_series[~dup_series.index.duplicated(keep="first")]

# Strategy 2: keep mean (average across retries)
dedup_mean = dup_series.groupby(level=0).mean()

print(f"\nAfter dedup (keep first): {len(dedup_first)}")
print(f"After dedup (mean):       {len(dedup_mean)}")

# Show the retry duplicate
ts_retry = index[55]
print(f"\nRetry duplicate at {ts_retry}:")
print(f"  Values:      {dup_series[ts_retry].values.round(3)}")
print(f"  Keep first:  {dedup_first[ts_retry]:.3f}")
print(f"  Mean:        {dedup_mean[ts_retry]:.3f}")

Output:

Length with duplicates: 170
Duplicate timestamps:   2

After dedup (keep first): 168
After dedup (mean):       168

Retry duplicate at 2024-06-03 07:00:00:
  Values:      [235.198 234.498]
  Keep first:  235.198
  Mean:        234.848

For most sensor pipelines, keep-first is the right default; the first delivery is the original reading. Mean makes sense when retries come from independent sensors measuring the same quantity.

Frequency Alignment and Resampling

Real pipelines often mix data at different frequencies. For example, you may need a 1-minute meter reading merged with an hourly weather feed. Before joining them, you need to align frequencies explicitly.

# 1-minute power draw readings
power_1min = pd.Series(
    42 + 18 * ((pd.date_range("2024-06-01", periods=1440, freq="T").hour.isin(range(8, 19)))).astype(int)
    + np.random.normal(0, 2, 1440),
    index=pd.date_range("2024-06-01", periods=1440, freq="T"),
    name="power_kw"
)

# Downsample to hourly: mean is appropriate for power (average over the hour)
power_hourly_mean = power_1min.resample("H").mean().round(2)

# Downsample to hourly: max (peak demand within the hour)
power_hourly_max = power_1min.resample("H").max().round(2)

# Downsample to hourly: sum (total energy = kWh)
energy_hourly_kwh = (power_1min.resample("H").sum() / 60).round(3)

comparison = pd.DataFrame({
    "mean_kw":    power_hourly_mean,
    "peak_kw":    power_hourly_max,
    "energy_kwh": energy_hourly_kwh,
}).iloc[7:13]

print(comparison)

Output:

                     mean_kw  peak_kw  energy_kwh
2024-06-01 07:00:00    42.13    46.28      42.133
2024-06-01 08:00:00    60.56    64.81      60.557
2024-06-01 09:00:00    59.91    64.88      59.912
2024-06-01 10:00:00    60.07    65.16      60.066
2024-06-01 11:00:00    60.08    64.99      60.083
2024-06-01 12:00:00    59.72    63.65      59.724

Which aggregation you choose matters enormously for downstream use. Mean power is right for load profiling. Peak power is right for capacity planning. Sum (converted to kWh) is right for billing. You can probably see why the right answer is domain-specific and not technical.

Smoothing Noise

Raw sensor data often contains high-frequency noise that obscures the underlying signal. Smoothing before feature engineering prevents the model from fitting to noise, but over-smoothing destroys real variation.

Exponential Weighted Moving Average

Exponential Weighted Moving Average or EWMA gives more weight to recent observations and adapts quickly to level changes. This is better than a simple moving average for non-stationary signals.

# Noisy temperature sensor (°C)
temp_noisy = pd.Series(
    3.5
    + 1.2 * np.sin(2 * np.pi * np.arange(168) / 24)
    + np.random.normal(0, 0.8, 168),  # high noise
    index=pd.date_range("2024-06-01", periods=168, freq="H"),
    name="temperature_c"
)

temp_ewma = temp_noisy.ewm(span=6, adjust=False).mean()
temp_sma  = temp_noisy.rolling(window=6, center=True).mean()

comparison = pd.DataFrame({
    "raw":  temp_noisy,
    "ewma": temp_ewma.round(3),
    "sma":  temp_sma.round(3),
}).iloc[22:30]

print(comparison)

Output:

                          raw   ewma    sma
2024-06-01 22:00:00  3.212372  2.843  3.035
2024-06-01 23:00:00  3.106840  2.918  3.176
2024-06-02 00:00:00  3.712290  3.145  3.011
2024-06-02 01:00:00  3.344376  3.202  3.294
2024-06-02 02:00:00  2.148946  2.901  3.705
2024-06-02 03:00:00  4.241105  3.284  4.087
2024-06-02 04:00:00  5.677429  3.968  4.381
2024-06-02 05:00:00  5.400083  4.377  4.765

Savitzky-Golay Filter

For signals where you need to preserve peak shapes — not just smooth them away — the Savitzky-Golay filter fits a polynomial over a sliding window and is better at maintaining the height of genuine spikes.

from scipy.signal import savgol_filter

temp_savgol = pd.Series(
    savgol_filter(temp_noisy.values, window_length=11, polyorder=2),
    index=temp_noisy.index,
    name="temp_savgol"
).round(3)

print(pd.DataFrame({
    "raw":    temp_noisy,
    "savgol": temp_savgol,
}).iloc[22:30])

Output:

                          raw  savgol
2024-06-01 22:00:00  3.212372   2.960
2024-06-01 23:00:00  3.106840   2.944
2024-06-02 00:00:00  3.712290   3.114
2024-06-02 01:00:00  3.344376   3.379
2024-06-02 02:00:00  2.148946   3.809
2024-06-02 03:00:00  4.241105   4.288
2024-06-02 04:00:00  5.677429   4.749
2024-06-02 05:00:00  5.400083   5.138

Schema and Sanity Validation

Cleaning without validation is incomplete. You need automated checks that run every time new data arrives — catching problems before they silently corrupt downstream models.

def validate_time_series(series: pd.Series, config: dict) -> dict:
    """
    Run schema and sanity checks on a time series.
    Returns a report dict with pass/fail per check.
    """
    report = {}

    # Frequency check
    inferred = pd.infer_freq(series.index)
    report["freq_regular"] = inferred == config["expected_freq"]

    # Missing value threshold
    missing_rate = series.isna().mean()
    report["missing_below_threshold"] = missing_rate <= config["max_missing_rate"]
    report["missing_rate"] = round(missing_rate, 4)

    # Value range check
    in_range = series.dropna().between(config["min_value"], config["max_value"])
    report["values_in_range"] = in_range.all()
    report["out_of_range_count"] = (~in_range).sum()

    # Duplicate timestamps
    report["no_duplicates"] = not series.index.duplicated().any()

    # Monotonic index
    report["index_monotonic"] = series.index.is_monotonic_increasing

    return report


config = {
    "expected_freq":    "H",
    "max_missing_rate": 0.05,
    "min_value":        210.0,
    "max_value":        250.0,
}

report = validate_time_series(voltage_outlier_fixed, config)

print("=== VALIDATION REPORT ===")
for check, result in report.items():
    if check in ("missing_rate", "out_of_range_count"):
        print(f"  {check}: {result}")
    else:
        status = "✓ PASS" if result else "✗ FAIL"
        print(f"  {status}  {check}")

Output:

=== VALIDATION REPORT ===
  ✗ FAIL  freq_regular
  ✓ PASS  missing_below_threshold
  missing_rate: 0.0
  ✓ PASS  values_in_range
  out_of_range_count: 0
  ✓ PASS  no_duplicates
  ✓ PASS  index_monotonic

This validator is the kind of function you wrap around every data ingestion step in a production pipeline. Run it before cleaning to know what's broken, and after cleaning to confirm everything passed.

The Complete Cleaning Checklist

Here's the full sequence to run on any incoming time series dataset:

Step Technique When to Use
Audit Index check, missing map, value range Always — before anything else
Reindex reindex to canonical frequency When timestamps are absent rather than NaN
Missing: short gaps Time interpolation Continuous signals, gaps ≤ 3 steps
Missing: step signals Forward fill Categorical or setpoint data
Missing: long gaps Seasonal decomposition impute Seasonal signals, gaps > 6 steps
Outliers: univariate Rolling Z-score or IQR Single sensor, local anomalies
Outliers: multivariate Isolation Forest Multiple correlated sensors
Outlier treatment Winsorize or interpolate Depending on whether event is real
Duplicates Keep first or group mean Pipeline retry duplicates
Resampling .resample() with correct aggregation Frequency alignment before joins
Smoothing EWMA or Savitzky-Golay Noisy sensors before feature engineering
Validation Schema + sanity checks After cleaning, and on every new batch

Wrapping Up

The order matters. Reindex before imputing. Impute before smoothing. Validate after everything. Skipping steps or doing them out of order compounds errors in ways that are very difficult to trace back once you're looking at model predictions.

Time series cleaning isn't glamorous work, but a model trained on clean data and thoughtfully engineered features will almost always outperform a more sophisticated model trained on data that wasn't cleaned properly. Getting this pipeline right is the highest-leverage thing you can do before you try running even the simplest algorithm on your time series data.

]]> <![CDATA[ Data Science Insights: Why the Mean Lies When Handling Messy Retail Data ]]> https://www.freecodecamp.org/news/data-science-insights-why-the-mean-lies-when-handling-messy-retail-data/ 69fa21e5a386d7f121b5fe8c Tue, 05 May 2026 16:59:17 +0000 In our daily life, we use the word "average" all the time: average salary, average marks, average age, and so on.

Let's take the case of a retail shop. If we're looking at the average order value to understand customer spending, we'd load the data, run the code, and get a result of $20 per order.

Done.

Except something looks odd.

When we take a closer look, we see that most customers are buying items worth \(8 - \)15. So where's $20 coming from?

In that case, the problem isn’t data – it’s the average. This is a clean textbook trap where everything works perfectly in the textbook, but real-world data doesn’t behave nicely.

Some customers buy in bulk (very large orders), some return orders (negative quantities), and a few anomalies distort the entire picture.

In this article, we'll use the Online Retail Dataset to answer a simple but tricky question: What does “average” really mean in the real world?

Table Of Contents

Prerequisites

To follow along here, you'll need:

Basic Python knowledge: Understanding of variables and functions.

The Pandas library: Familiarity with loading data and basic DataFrame operations.

A development environment: Access to a tool like Jupyter Notebook, VS Code, or Google Colab.

A Dataset: For this analysis, I used the Online Retail Dataset, which is available for download here.

The Dataset

We'll work with the Online Retail Dataset, a real-world transactional dataset containing purchase records from a UK-based online retail store.

  1. Source: UCI Machine Learning Repository

  2. Collected by: UK-based online retail company (2010–2011)

  3. Size: 541,909 transactions

  4. Features: 8 attributes (InvoiceNo, StockCode, Description, Quantity, InvoiceDate, UnitPrice, CustomerID, Country)

  5. Ownership: Public dataset hosted by UCI

  6. License: Open for research and educational use

Mean: The Sensitive Giant

In statistics and data analysis, the terms "average" and "arithmetic mean" are often used interchangeably. We aim to find the mean total price in our dataset. Mean in the context of the Online Retail Dataset is given as:

$$\text{Average Order Value} = \frac{\text{Sum of all TotalPrice values}}{\text{Number of transactions}}$$

In our dataset, the mean is calculated by summing all transaction values (including bulk purchases and returns) and dividing by the total number of transactions. This means every value, irrespective of unusually high or any negative values, directly influences the final average.

# Load the dataset
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00352/Online%20Retail.xlsx"
df = pd.read_excel(url, engine='openpyxl')

# Clean and Feature Engineering
df = df.dropna(subset=['CustomerID'])
df['TotalPrice'] = df['Quantity'] * df['UnitPrice']

# Calculate the Mean (Average Order Value)
mean_value = df['TotalPrice'].mean()
print(f"Average Order Value (Mean): {mean_value:.2f}")

The results are as follows:

Average Order Value (Mean): 20.40

At first glance, the results may look promising: every transaction contributes equally. But that’s where the problem lies. Sometimes a few transactions, which are extremely high or low, affect the mean for all customers who lie in the closer range.

Take a look at the graph for the mean below.

The graph shows the calculated mean for the Online Retail Dataset, where we get a mean of 20.40

The graph shows the mean Total Price for the Online Retail Dataset. We get a mean of 20.42. (Image by Author)

The graph shows a right-skewed distribution where the calculated mean of 20.40 is actually a textbook trap. The tallest bar clearly shows that the majority of transactions lie in the range of \(8 - \)15 range, but the red line is being dragged to the right by the long tail of high-value bulk orders by some customers.

In this scenario, the average price is well above what a typical customer actually spends because it's highly sensitive to outliers – and in reality, the bulk of the data lives in the lower price range.

In simple words, the mean is being pulled by some extreme values to the right, especially by some lying in the range of 200–300, which is noticeable in the graph.

Median: The Robust Middle

When the mean is distorted by extreme values, we need a metric that remains unaffected by such outliers. This is where the median comes into play.

Median is defined as the middle value after sorting the data.

In our dataset, we sort all the transactions and pick the middle one.

The formula for calculating the median is:

$$\text{Median} = \begin{cases} X_{\left[ \frac{n+1}{2} \right]} & \text{if } n \text{ is odd} \ \frac{X_{\left[ \frac{n}{2} \right]} + X_{\left[ \frac{n}{2} + 1 \right]}}{2} & \text{if } n \text{ is even} \end{cases}$$

Unlike the mean, the median doesn't depend on extreme values, and it cares only about the position of the data, not the magnitude.

# Clean and Feature Engineering
df = df.dropna(subset=['CustomerID'])
df['TotalPrice'] = df['Quantity'] * df['UnitPrice']

# Calculate only the Median
median_value = df['TotalPrice'].median()
print(f"Typical Order Value (Median): {median_value:.2f}")

The results are as follows:

Typical Order Value (Median): 11.10

Now you'll notice that the result lies in the \(8 — \)15 range, where most of the transactions lie.

The figure demonstrates the graph for the median, where we get an accurate value of the transactions by the customers.

The figure demonstrates the graph for the median, where we get an accurate value of the transactions by the customers. (Image by Author)

In the previous graph, the mean was pulled to the right by large orders, but the median just asks what the middle customer spends. So even if someone spends $300 or some transactions are negative, the median stays stable.

In the above figure the median graph accurately highlights the range where most of the customers lie.

Beyond Averages: Understanding Spread with Quartiles

So far, we've studied the median, but knowing the center is not enough.

To truly understand how customer spending is, we need to understand how the data is spread, and this is where quartiles come into play.

Quartiles divide the dataset into the following parts:

  1. Q1(25th percentile): 25% of transactions are below this.

  2. Q2 (50th percentile): Median

  3. Q3 (75th percentile): 75% of transactions are below this.

This is formally expressed as the Interquartile Range (IQR):

$$IQR = Q_3 - Q_1$$

The IQR: Detecting Outliers

The IQR measures the spread of the middle 50%.

If the IQR is small, then the data is concentrated. If it's large, the data is spread out. The IQR also helps us identify outliers mathematically.

Outlier Rule:

  1. Lower Bound = Q1 — 1.5 * IQR

  2. Upper Bound = Q3 + 1.5 * IQR

A Simple Example to Understand IQR

Consider the following transaction values:

$$\left[ 5, 8, 10, 12, 15, 18, 20 \right]$$

Step 1: Find the Median (Q2):

The middle value is:

$$Q_2 = 12$$

Step 2: Find Q1 (Lower Quartile):

The lower half is [5, 8, 10]. The median of the lower half is:

$$Q_1 = 8$$

Step 3: Find Q3 (Upper Quartile):

The upper half is [15, 18, 20]. The median of the upper half is:

$$Q_3 = 18$$

Step 4: Calculate IQR:

$$IQR = Q_3 - Q_1 = 18 - 8 = 10$$

Step 5: Find Outlier Bounds:

$$\begin{aligned} \text{Lower Bound} &= Q_1 - 1.5 \times IQR = 8 - 15 = -7 \ \text{Upper Bound} &= Q_3 + 1.5 \times IQR = 18 + 15 = 33 \end{aligned}$$

Any value below -7 or above 33 is an outlier (but in this demo problem, no outliers exist).

Applying IQR to Our Dataset

In our retail dataset, instead of neat values, we have bulk values and even negative returns.

# 1. Calculate IQR and Bounds
Q1 = df['TotalPrice'].quantile(0.25)
Q3 = df['TotalPrice'].quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

When we calculate IQR for our dataset, we get:

Lower Bound: -18.75
Upper Bound: 42.45
Number of Outliers: 33180
The figure demonstrates the outlier range for our dataset

The graph demonstrates outliers, which are any values falling outside the range of -18.75 to 42.45. (Image by Author)

As the graph shows, the values outside the range -18.75 to 42.45 are considered outliers. These values will be removed.

Revisiting the Mean After Removing Outliers

Using the IQR method, we've removed extreme transactions that fell outside the typical spending range.

# Clean and Feature Engineering
df = df.dropna(subset=['CustomerID'])
df['TotalPrice'] = df['Quantity'] * df['UnitPrice']

# Original Mean
mean_value = df['TotalPrice'].mean()
print(f"Original Mean: {mean_value:.2f}")

# IQR Calculation
Q1 = df['TotalPrice'].quantile(0.25)
Q3 = df['TotalPrice'].quantile(0.75)
IQR = Q3 - Q1

# Define bounds
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

print(f"Lower Bound: {lower_bound:.2f}")
print(f"Upper Bound: {upper_bound:.2f}")

# Remove Outliers
df_no_outliers = df[(df['TotalPrice'] >= lower_bound) & (df['TotalPrice'] <= upper_bound)]

# New Mean after removing outliers
new_mean = df_no_outliers['TotalPrice'].mean()
print(f"Mean after removing outliers: {new_mean:.2f}")

After recomputing, we get:

Original Mean: 20.40
Lower Bound: -18.75
Upper Bound: 42.45
Mean after removing outliers: 11.63
The graph demonstrates that the mean improves significantly after all outliers are removed. (Image by Author)

Removing outliers significantly shifts the mean toward the region where most transactions occur. We now have a much better mean of 11.63 as opposed to the right-stretched mean of 20.40 we got with outliers.

Final Comparison and Insights

Looking at the results from all the graphs, we get a complete understanding of the dataset. The original mean was 20.40, which appeared to be significantly higher than the most transactions that actually occurred. In that case, the mean was pulled upward by some of the high-valued transactions and was distorted by the outliers.

The median, on the other hand, was 11.10, which lies within the range where most transactions are concentrated. This shows that the median is a much better representation of what a typical customer spends, as it's not affected by extreme values.

After removing the outliers using the IQR, the mean dropped to 11.63, bringing it very close to the median. This confirms that the earlier mean was not inherently wrong, but was simply influenced by extreme values in the data. Once those values were handled, the mean became a much more reliable measure of central tendency.

Conclusion

The results show that the mean can be misleading when data contains outliers. In our dataset, the original mean of 20.40 overstated customer spending, while the median (11.10) gave a more realistic picture. After removing outliers, the mean shifted to 11.63, aligning closely with the median.

This highlights a key lesson: The mean isn't wrong, but it must be used with an understanding of the data.

Choosing the right measure of average depends on the dataset, and in messy real-world scenarios, the median or a cleaned mean often tells the true story.

Connect with me

  1. Medium

  2. LinkedIN

If you want to dive deeper, you can visit: Mean vs Median vs Mode: Understanding Central Tendency in Data Analysis.

]]> <![CDATA[ The Data Quality Handbook: Data Errors, the Developer's Role, and Validation Layers Explained. ]]> https://www.freecodecamp.org/news/data-quality-handbook-data-errors-the-developer-s-role-validation-layers/ 69dea3b491716f3cfb75fd9d Tue, 14 Apr 2026 20:29:40 +0000 In August 2012, Knight Capital, a major trading firm in the United States, deployed faulty trading software to its production system. The system used this incorrect configuration data and it triggered millions of unintended stock trades.

The company lost about $440 million in just 45 minutes. Knight Capital nearly collapsed and had to be rescued by investors. It was later acquired by another firm.

When Target expanded into Canada, the company relied on a new supply chain system that contained incorrect product and inventory data. Product information in the database was incomplete and inaccurate. Prices, sizes, and product descriptions were entered incorrectly.

Inventory systems reported items in stock that were actually unavailable. Customers found empty shelves in stores despite the system showing stock. The company lost over $2 billion in the Canadian market. Target eventually shut down all Canadian stores in 2015.

One employee made the statement “Even though we had a great supply chain system on paper, we didn’t have accurate data. Bad data leads to bad decisions’’

Another famous example of data-related engineering failures involves the Mars Climate Orbiter spacecraft. One engineering team used metric units (newtons). Another team used imperial units (pounds-force). The system failed to convert the data correctly. The spacecraft entered Mars' atmosphere at the wrong altitude. The mission failed and the spacecraft was destroyed. The loss was about $125 million.

In this article, we'll delve deep into what data quality truly means, the types of data errors that silently break systems, the developer’s responsibility in preventing them, and the validation layers that work together to keep bad data out of production.

What We'll Cover:

Prerequisites

  • A basic understanding of what data is

  • A basic understanding of data structures

  • An understanding of what an API is

  • An understanding of what a database is and what it does

The Importance of Data Quality

As you can see from just these few examples, the quality of the data you're working with really matters.

Gartner reports that organisations attribute around $15 million in annual losses to poor‑quality data. The same research also shows that nearly 60% of companies have no clear idea what bad data actually costs them, largely because they don’t track or measure data‑quality issues at all.

A 2016 study by IBM is even more eye-popping. IBM found that poor data quality strips $3.1 trillion from the U.S. economy annually due to lower productivity, system outages, and higher maintenance costs.

Bad data is, and will continue to be, the kryptonite of any organisation. This is even more concerning as more organisations now depend on data for strategy execution than ever before.

When data is wrong, incomplete, duplicated, or inconsistent, the consequences ripple outward: Incorrect dashboards mislead teams, which leads to making incorrect decisions. Implementing these decisions can lead to faulty strategy and policy implementation.

Eventually, the organisation pays the price, financially, operationally, and reputationally. And while money can be recovered, reputation rarely bounces back so easily.

How Does Bad Data Happen in the First Place?

Form fields are usually the first place where data enters an application, so they’re often where bad data begins. This is why the developer’s role is so critical.

Many of the most damaging data errors don’t originate from malicious users or complex edge cases – they come from simple oversights that the system should never have allowed in the first place.

But it's equally important to recognise that data quality issues often originate before the data ever reaches an application. Upstream processes — how data is collected, measured, recorded, or pre‑validated — can introduce inaccuracies long before the system receives it.

For example, a nurse might weigh a patient using an uncalibrated mechanical scale, record the incorrect value on a paper form, and later have that value transcribed into the hospital system. By the time the data enters the application, the error is already embedded.

This means that maintaining data quality requires attention both to upstream data collection practices and to the system-level validation that developers control.

When the UI, backend, or API layer permits invalid, incomplete, inconsistent, or logically impossible data to enter the pipeline, the organisation inherits a long‑term liability. Even small choices — such as allowing empty fields, ignoring duplicates, or failing to enforce validation rules — can introduce errors that may only surface months later in reports or dashboards, leading to confusion and inaccurate insights.

The Cost of Bad Data

Data quality can also be impacted at any stage of the data pipeline: before ingestion, in production, or even during analysis.

If bad data is caught in the UI, it's almost free, if we're thinking in terms of cost. If it's caught at the API layer, that's still pretty cheap. If it's caught in the database, the cost is moderate. And if it's caught in a report or ML model months later, that's expensive, and sometimes irreversible.

A key principle in modern data management is: the cheapest and safest place to catch bad data is at the source, and that is before ingestion. The well-known 1-10-100 Rule, introduced by George Labovitz and Yu Sang Chang in 1992, clearly illustrates this idea.

According to the rule, it costs about \(1 to validate data at the point of entry, \)10 to correct it after it has entered the system, and $100 per record if the error goes unnoticed and causes problems further down the line.

As the saying goes, an ounce of prevention is worth a pound of cure – and this is especially true when it comes to maintaining high-quality data.

To help buttress my point, I’ve categorised the different types of errors and oversights that developers should never allow that can and should be prevented before they ever reach the database, analytics layer, or reporting systems.

Types of Data Errors

Required Field Errors

If you build a form that allows a user to submit a registration form with important fields left empty (like first name, last name, email address, phone number, date of birth, or address), you're directly letting incomplete data enter the system.

I remember a scenario from my time as a data analyst where I was analysing a dataset containing different types of alarms triggered across several buildings. These alarms fell into categories such as aquarium alarms, intruder alarms, fire alarms, and maintenance alarms.

The purpose of the analysis was simple: identify which buildings had the highest frequency of alarms so that maintenance, resources, or investigations could be allocated appropriately.

Whenever an alarm went off, the security team recorded it using a software system. By the end of each month, we could view the cumulative alarms and generate insights.

But I encountered a major data quality issue. The security officers often selected the alarm category but failed to submit the building where the alarm occurred — and the system allowed this incomplete record to be saved into the database.

Every alarm had to occur in a specific building. Yet during analysis, I would see entries like “20 fire alarms” with no building information attached. Since I couldn’t determine where these alarms happened, the data became unusable. I had no choice but to delete those records because they provided no actionable value.

This is a classic example of poor data validation. If the developer had implemented proper constraints, the system would never allow an alarm to be submitted without a building name.

Required fields should be enforced at the UI and backend levels to prevent missing data from entering the system in the first place. These gaps lead to missing or unusable data in the database, often forcing teams to delete or manually repair records later.

To prevent these errors, you can use required‑field validation, disable the submit button until all mandatory fields are completed, and visually highlight missing fields with inline error messages.

Here's a practical code example of some bad code (no required checks):

<form id="signup">
  <input type="text" id="name" placeholder="Full name">
  <input type="email" id="email" placeholder="Email">
  <button type="submit">Sign up</button>
</form>

<script>
document.getElementById("signup").addEventListener("submit", e => {
  const name = document.getElementById("name").value;
  const email = document.getElementById("email").value;
  console.log("Submitted:", { name, email });
});
</script>

From the above code snippet, the core problem is that the form doesn't enforce required input. Neither HTML‑level validation (using the required attribute) nor JavaScript‑based checks are implemented. This omission allows users to submit the form without providing necessary information, making the form unreliable for collecting valid and complete user data.

From a usability and data quality perspective, this is problematic. Forms are typically designed to collect meaningful and complete information, and fields such as “Full name” and “Email” are usually essential. Without marking these inputs as required or validating them programmatically, we risk receiving blank or invalid submissions, which can compromise the quality of stored data and any processes that depend on it.

Here's an example of a better version (UI prevents empty submission):

<form id="signup">
  <input type="text" id="name" placeholder="Full name" required>
  <input type="email" id="email" placeholder="Email" required>
  <button type="submit">Sign up</button>
</form>

<script>
document.getElementById("signup").addEventListener("submit", e => {
  if (!e.target.checkValidity()) {
    e.preventDefault();
    alert("Please fill in all required fields.");
  }
});
</script>

In this revised version of the code, the addition of the required attribute to both the name and email input elements ensures that the browser won't allow the form to be submitted unless these fields are filled. This is an important step toward maintaining data completeness and improving the overall reliability of the form.

Also, by checking e.target.checkValidity(), we now ensure that the form is evaluated before submission proceeds.

Another positive aspect is the conditional use of e.preventDefault(). When the form is invalid, the default submission behavior is stopped, preventing incomplete or incorrect data from being sent.

Format Validation Errors

If you have a form that allows a user to enter an email without an @ symbol, an email without a domain, a phone number containing letters, or a postcode/ZIP code in the wrong format, that allows invalid data to enter the system.

The same applies when you allow a user to submit an impossible date (32/15/2025) or a credit card number with the wrong length.

These issues will cause the data analyst to spend more time cleaning the data, if it's even cleanable. And such incorrect inputs create unreliable data that breaks downstream processes and increases cleanup costs.

To prevent these types of errors, you can use regex validation, input masks, and field‑type restrictions (for example, numeric‑only fields for phone numbers) to enforce correct formats before submission.

Here's a bad example of allowing format validation errors:

<input id="phone" placeholder="Phone number">
<button onclick="save()">Save</button>

<script>
function save() {
  const phone = document.getElementById("phone").value;
  console.log("Saving phone:", phone);
}
</script>

This code doesn't perform any checks on the format or structure of the phone number. The function simply retrieves whatever value exists – whether valid, invalid, or blank – and logs it to the console without any condition.

Here's the fixed version:

<input id="phone" placeholder="Phone number" required>
<button onclick="save()">Save</button>

<script>
function save() {
  const phone = document.getElementById("phone").value;

  if (!/^\d+$/.test(phone)) {
    alert("Phone number must contain digits only.");
    return;
  }

  console.log("Saving phone:", phone);
}
</script>

This version fixes the earlier mistake by introducing a clear validation rule. Before the system accepts the phone number, it checks whether the input contains only digits. The regular expression ^\d+$ ensures that the value is made up entirely of numbers, with no letters or symbols allowed. If the user enters anything invalid, the function stops and displays an error message instead of saving bad data.

This approach prevents the format error that occurred in the previous example. Instead of blindly trusting whatever the user types, the code now enforces a rule that matches the expected format of a phone number. This is what a responsible developer should do: verify the input before using it.

Range and Limit Errors

Allowing users to enter values outside acceptable limits – such as negative ages, quantities below zero, discounts above 100%, or measurements far beyond realistic ranges – that enables the ingestion of data that violates business rules. These errors distort analytics, break calculations, and create operational inconsistencies.

To mitigate these errors, you can apply min/max constraints, sliders, steppers, and numeric boundaries to ensure values fall within valid ranges.

Here's a bad example of allowing range and limit errors:

<input id="age" type="number">
<button onclick="submitAge()">Submit</button>

<script>
function submitAge() {
  console.log("Age:", document.getElementById("age").value);
}
</script>

As seen above, we've created an input field for age but doesn't specify any limits or constraints. The browser allows the user to type any number — including values that make no sense, such as negative ages, extremely large ages, or decimals. The JavaScript function simply reads the value and logs it without checking whether the age is realistic.

Here's a better version:

<input id="age" type="number" min="0" max="120" required>
<button onclick="submitAge()">Submit</button>

<script>
function submitAge() {
  const ageInput = document.getElementById("age");
  if (!ageInput.checkValidity()) {
    alert("Age must be between 0 and 120.");
    return;
  }
  console.log("Age:", ageInput.value);
}
</script>

Now in this version, the inclusion of the min="0" and max="120" attributes sets clear boundaries for acceptable input values. This ensures that only realistic age values within a defined range are allowed, preventing invalid entries such as negative numbers or excessively large ages.

The JavaScript function further enhances this validation by using the checkValidity() method. This method checks whether the input satisfies all defined constraints, including the required condition and the specified numeric range. If the input doesn't meet these conditions, the function prevents further execution and displays an alert message, informing the user that the entered age must fall within the allowed range.

Logical Consistency Errors

If you allow a user to select an end date before the start date, choose a checkout date earlier than check‑in at a hotel, or enter a delivery date before the order date, this will result in logically impossible data. The same applies when you allow a user to enter a graduation year earlier than their admission to a program, or submit working hours that exceed 24 hours in a day.

You can mitigate this by implementing cross‑field validation, business‑rule checks, and conditional logic that ensures related fields remain consistent.

Here's a bad example of a logical consistency error:

<input type="date" id="start">
<input type="date" id="end">
<button onclick="save()">Save</button>

<script>
function save() {
  console.log({
    start: document.getElementById("start").value,
    end: document.getElementById("end").value
  });
}
</script>

In the code above, the core issue is the complete absence of validation. Although the inputs use type="date", which provides a structured way for users to select dates, the code doesn't enforce that either field is required. This means the user can leave one or both date fields empty, and the save() function will still run and log the values. As a result, the system may end up processing incomplete or meaningless data.

Beyond missing required checks, the code also fails to validate the logical relationship between the two dates. In any scenario involving a start date and an end date, it's expected that the start date shouldn't occur after the end date. But this code performs no such comparison.

This means that the user can select a start date that's later than the end date, and the system will accept it without warning. This leads to inconsistent or impossible data being recorded.

Also, the function simply logs the values without providing any feedback to the user. There's no mechanism to alert the user when a field is empty or when the dates are logically incorrect. This reduces usability and makes it difficult for users to understand or correct their mistakes.

Here's the fixed version:

<input type="date" id="start" required>
<input type="date" id="end" required>
<button onclick="save()">Save</button>

<script>
function save() {
  const startValue = document.getElementById("start").value;
  const endValue = document.getElementById("end").value;

  // Extra safety: check empties (in case required is bypassed)
  if (!startValue || !endValue) {
    alert("Both start and end dates are required.");
    return;
  }

  const start = new Date(startValue);
  const end = new Date(endValue);

  if (end < start) {
    alert("End date cannot be before start date.");
    return;
  }

  console.log({ start, end });
}
</script>

In this improved version, first, both date fields now include the required attribute, ensuring that the user can't leave either field empty without triggering validation.

Second, we've added a logical validation check to ensure that the relationship between the two dates is correct. After retrieving the values, the function converts them into Date objects and compares them to verify that the end date doesn't occur before the start date. If this condition is violated, the function stops execution and displays an alert informing the user of the error.

This prevents inconsistent or impossible date ranges from being accepted.

Duplicate and Data Integrity Errors

When you let a user submit an email that's already registered, choose a username that's already taken, or enter a duplicate employee ID or student number, this results in identity conflicts and duplicate records. Problems also arise when you allow users to upload unsupported file types, oversized files, or corrupted images.

Security risks can emerge when users are able to enter HTML/script tags (XSS), SQL‑injection patterns, or disallowed special characters. These issues compromise data quality, system integrity, and security.

You can prevent these types of issues by using uniqueness checks, file‑type and size validation, and input sanitization to block duplicates, invalid uploads, and malicious inputs.

Here's an example of a duplicate error:

<input id="email" placeholder="Enter email" required>
<button onclick="save()">Save</button>

<script>
const savedEmails = [];

function save() {
  const email = document.getElementById("email").value;
  savedEmails.push(email);
  console.log("Saved emails:", savedEmails);
}
</script>

This code blindly pushes every email into the savedEmails array without checking whether the email already exists. Because there is no duplicate detection, the user can enter the same email multiple times.

Here is the fixed version:

<input id="email" placeholder="Enter email" required>
<button onclick="save()">Save</button>

<script>
const savedEmails = [];

function save() {
  const email = document.getElementById("email").value.trim();

  // Check if the field is empty
  if (!email) {
    alert("Please enter an email before saving.");
    return;
  }

  // Check for duplicate
  if (savedEmails.includes(email)) {
    alert("This email has already been saved.");
    return;
  }

  savedEmails.push(email);
  console.log("Saved emails:", savedEmails);
}
</script>

In this improved version of the code, we've implemented proper validation steps to prevent duplicate email entries. Before saving the email, the function checks whether the value already exists in the savedEmails array using the includes() method. If the email is found, the function stops execution and displays an alert informing the user that the email has already been saved. This ensures that each email is stored only once, maintaining the uniqueness and integrity of the data.

Relational Errors (Reference Integrity)

If you let a user select a city that doesn’t belong to the chosen country, a product ID that no longer exists, a retired SKU, or a shipping method unavailable in the selected region, this can result in broken references.

The same applies when users can select a manager from a different department or choose a fully booked time slot, not setting the right roles and permissions. These errors break relationships between tables and corrupt downstream joins and reports.

Here, you can use dependent dropdowns, real‑time lookups, and foreign‑key validation to help ensure that users can only select valid, existing, and compatible options.

Here's a bad example of a relational error:

<select id="country">
  <option value="uk">United Kingdom</option>
  <option value="usa">United States</option>
</select>

<select id="city">
  <option value="london">London</option>
  <option value="manchester">Manchester</option>
  <option value="newyork">New York</option>
  <option value="losangeles">Los Angeles</option>
</select>

<button onclick="save()">Save</button>

<script>
function save() {
  const country = document.getElementById("country").value;
  const city = document.getElementById("city").value;

  console.log("Saving:", { country, city });
}
</script>

From the above, the mistake in this code is that we've treated country and city as completely independent fields, even though one is supposed to depend on the other. By presenting all cities regardless of the selected country, the interface allows users to create combinations that make no sense — such as choosing “United Kingdom” with “New York” or “United States” with “Manchester.”

Also, because the save() function performs no validation and simply logs whatever the user selects, the system ends up accepting and storing relationships that should never exist. This breaks the logical link between the two fields and leads to invalid, inconsistent data that can corrupt downstream.

Here's the fixed, production-ready version:

<select id="country" onchange="loadCities()" required>
  <option value="">Select country</option>
  <option value="uk">United Kingdom</option>
  <option value="usa">United States</option>
</select>

<select id="city" required disabled>
  <option value="">Select city</option>
</select>

<button onclick="save()">Save</button>

<script>
const citiesByCountry = {
  uk: ["London", "Manchester"],
  usa: ["New York", "Los Angeles"]
};

function loadCities() {
  const country = document.getElementById("country").value;
  const citySelect = document.getElementById("city");

  // Reset city dropdown
  citySelect.innerHTML = '<option value="">Select city</option>';

  // Disable if no country selected
  if (!country) {
    citySelect.disabled = true;
    return;
  }

  // Enable dropdown
  citySelect.disabled = false;

  // Load cities safely
  (citiesByCountry[country] || []).forEach(city => {
    const option = document.createElement("option");
    option.value = city.toLowerCase().replace(/\s+/g, ""); // remove ALL spaces
    option.textContent = city;
    citySelect.appendChild(option);
  });
}

function save() {
  const country = document.getElementById("country").value;
  const city = document.getElementById("city").value;

  // Required validation
  if (!country || !city) {
    alert("Please select both a country and a city.");
    return;
  }

  // Build list of valid cities for this country
  const validCities = (citiesByCountry[country] || [])
    .map(c => c.toLowerCase().replace(/\s+/g, ""));

  // Relational validation
  if (!validCities.includes(city)) {
    alert("Selected city does not belong to the chosen country.");
    return;
  }

  console.log("Saving:", { country, city });
}
</script>

This improved code turns the country–city form into a controlled, relationship‑aware flow instead of two loose dropdowns.

When the user selects a country, the loadCities() function runs. It first clears the city dropdown and, if no country is selected, keeps the city field disabled so the user can't choose a city on its own.

Once a valid country is chosen, the city dropdown is enabled and populated only with the cities that belong to that specific country, using the citiesByCountry mapping. Also, the city values are normalised (lowercased and stripped of spaces) so they’re consistent and safe to compare.

When the user clicks “Save,” the save() function checks that both a country and a city have been selected. If either is missing, it shows an alert and stops. It then rebuilds the list of valid city values for the chosen country and verifies that the selected city is actually in that list.

Structural Errors (Dropdowns, Radio Buttons, Enums)

If users can type a country as “U.S.A”, “USA”, “United States”, or “us”, enter gender as “male”, “Male”, “M”, or “man”, or type a department as “Engineering”, “Eng”, or “engineer”, this can result in inconsistent categorical data.

The same applies to currencies typed as “usd”, “USD”, “US Dollars”, product categories spelled differently, status values like “active”, “Active”, “ACT”, “enabled”, or boolean values like “yes”, “Yes”, “Y”, “1”.

These inconsistencies make analytics, grouping, and reporting unreliable, and the analyst will spend time cleaning and standardizing these files.

You should replace free‑text fields with dropdowns, radio buttons, and enums to enforce standardized categorical values.

Bad example of a structural error:

<form id="profile">
  <label>Country</label>
  <input type="text" id="country" placeholder="Enter country">
  <button type="submit">Save</button>
</form>

<script>
document.getElementById("profile").addEventListener("submit", e => {
  e.preventDefault();
  const country = document.getElementById("country").value;
  console.log("Saving:", country);
});
</script>

The problem with this code is that it pretends to save a country value without doing any real validation or enforcing any rules, which makes the form unreliable and prone to bad data.

The form uses a plain text input for “country,” meaning the user can type anything they want — misspellings, random characters, invalid countries, or even leave it blank. Because the input isn’t marked as required and the JavaScript doesn’t check whether the field contains a meaningful value, the form will happily “save” an empty string or nonsense text.

The submit handler prevents the default form submission but does nothing beyond logging whatever the user typed, so the system accepts invalid, incomplete, or malformed data without question. In short, the code collects input but doesn't validate it, doesn't enforce correctness, and doesn't protect the system from bad or unusable values.

Here's the fixed version:

<form id="profile">
  <label>Country</label>
  <select id="country" required>
    <option value="">Select country</option>
    <option value="uk">United Kingdom</option>
    <option value="usa">United States</option>
    <option value="canada">Canada</option>
  </select>

  <button type="submit">Save</button>
</form>

<script>
document.getElementById("profile").addEventListener("submit", e => {
  e.preventDefault();

  const country = document.getElementById("country").value;

  // Required validation
  if (!country) {
    alert("Please select a country before saving.");
    return;
  }

  console.log("Saving:", country);
});
</script>

The biggest improvement is that we're no longer relying on a free‑text field for the country. By switching to a dropdown, the form now limits the user to a controlled set of valid options. This prevents misspellings, random text, or invalid country names from ever entering the system.

These are the main types of data errors you might come across in your work. Now that we've discussed what causes them and some key fixes/preventative measures you can take, let's move on to data quality itself.

What Makes Good Data?

So what, in fact, is data quality? IBM defines it as the degree of accuracy, consistency, completeness, reliability, and relevance of the data collected, stored, and used within an organization or a specific context.

Let's look at each of these features of quality data a bit more closely to understand what they entail.

Completeness:

Completeness measures how much of the required data is actually present. When large portions of fields are missing, the dataset stops representing reality and any analysis built on it becomes unreliable.

An example would be a sign‑up form that stores users, but half of them are missing an email address. If you run an analysis on “email engagement,” your results will be skewed because a big chunk of users can’t even receive emails. This means that this data is incomplete.

Uniqueness:

Uniqueness checks whether each real‑world entity appears only once in the dataset. Duplicate records inflate counts, break joins, and distort metrics.

An example would be a customer table containing two rows for the same person with the same customer ID. When calculating “active customers,” the system counts them twice, inflating revenue projections.

Validity:

Validity evaluates whether data follows the expected format, type, or business rules. This includes correct data types, allowed ranges, and patterns defined by the system.

An example would be a field meant to store dates contains values like “32/99/2025” or “tomorrow.” These invalid entries break downstream ETL jobs that expect a proper date format.

Timeliness:

Timeliness reflects whether data is available when it’s needed. Even accurate data becomes useless if it arrives too late for the process that depends on it. For example, after a customer places an order, the system should generate an order ID instantly.

Accuracy:

Accuracy measures how closely data matches the real‑world truth. When multiple systems report the same metric, one must be designated as the authoritative source to avoid conflicting values.

Consistency:

Consistency checks whether data aligns across different datasets or within related fields. If two systems describe the same concept, their values shouldn't contradict each other.

For example, a company’s HR system reports 50 employees in Engineering, but the payroll system lists only 42. Since both describe the same group, the mismatch signals a data quality issue.

Fitness for Purpose:

Fitness for purpose assesses whether the data is suitable for the specific business task at hand. Even complete, accurate, and timely data may be unhelpful if it doesn’t answer the intended question.

A dataset of website clicks might be perfect for analysing user engagement, for example, but it’s useless for forecasting revenue because it contains no purchase or pricing information.

Data Validation Layers

Now that we've highlighted the characteristics that ensure quality data, it's important to discuss the layers of data validation.

There are five layers you'll need to check to enforce data quality.

Frontend Layer — “Protect the User, Not the System”

Frontend validation plays an important role in enhancing the user experience – but it doesn't provide real protection for a system.

Since frontend logic operates within the user’s environment, we can't trust it as a mechanism for enforcing data quality. Any code executed in the browser is ultimately under the user’s control, meaning it can be disabled, modified, intercepted, or bypassed entirely.

For instance, a user can simply open browser developer tools, remove validation rules, and submit invalid or malicious data without restriction.

Frontend validation is incapable of enforcing complex business rules. Constraints such as ensuring that a discounted price is lower than the original price, validating that a start date precedes an end date, preventing stock levels from becoming negative, or confirming that a product belongs to a valid category within the database require deeper system-level checks.

At the frontend level, what is being validated is: required fields, email format, password strength, address fields, and payment input format.

So frontend validation doesn't guarantee data quality or security, as it can be bypassed through API tools (like Postman), disabled JavaScript, malicious bots, and third-party integrations.

Because of this, it's best to treat the front-end as a usability layer, not a trust layer.

Backend Validation — “The Real Gatekeeper”

You can only guarantee true data quality and system integrity at the backend and database layers.

The backend is responsible for enforcing request validation, implementing business logic, and managing authentication and authorization.

If validation fails here, invalid data is rejected before it can propagate. Without this layer, data corruption begins at ingestion.

For example:

$request->validate([
   'name' => 'required|string|max:255',
   'price' => 'required|numeric|min:0',
   'stock' => 'required|integer|min:0',
   'category_id' => 'required|exists:categories,id',
]);

The code snippet above demonstrates how you can use request validation in Laravel to ensure that incoming data meets specific requirements before it's processed or stored in the database. This is an essential practice in web development, as it helps maintain data integrity, prevents errors, and enhances application security.

In this example, we're using the $request->validate() method to define a set of validation rules for four input fields: name, price, stock, and category_id. Each field is assigned a series of constraints that the incoming data must satisfy.

The name field is marked as required, meaning it must be included in the request and can't be empty. It must also be a string, ensuring that only textual data is accepted, and it's limited to a maximum length of 255 characters using max:255. This prevents excessively long inputs that could potentially cause issues in the database or user interface.

Similarly, the price field is required and must be numeric, allowing only numbers such as integers or decimal values. The rule min:0 ensures that the price can't be negative, which is logically consistent for most product pricing scenarios.

The stock field is also required and must be an integer, meaning it can only accept whole numbers. This is appropriate for counting physical items. Like the price field, it includes a min:0 rule to prevent negative stock values, which would not make sense in an inventory system.

Finally, the category_id field is validated to ensure it is both present and valid. The required rule ensures that a category is selected, while the exists:categories,id rule checks that the provided value corresponds to an existing id in the categories database table. This prevents invalid or non-existent category references, thereby preserving relational integrity within the database.

This layer validates null values, data types and formats, allowed ranges, and referential integrity (exists).

Database Layer — “Protect the Data at Rest”

Validation at the application level is insufficient on its own. You'll also need to enforce database-level constraints like NOT NULL constraints, UNIQUE constraints (email, SKU, order number), foreign keys (orders.user_id → users.id), and check constraints (for example, price >= 0).

This layer is critical because application bugs may bypass validation, background jobs and imports may skip controllers, and malicious actors may attempt direct access.

The database layer acts as the final line of defense, ensuring structural integrity regardless of application failures. Database constraints are the last hard stop: they enforce correctness even when code is bypassed.

Service Layer / Business Logic — “Validate Real-World Rules”

This layer enforces domain-specific logic that can't be captured by simple validation rules. The service layer is where the application stops asking “Is this data shaped correctly?” and starts asking “Is this allowed to happen in the real world?”.

This layer enforces domain‑specific rules that can't be captured by simple request validation or database constraints. These rules reflect business truth, not structural correctness.

Example:

if (\(product->stock < \)quantity) {
   throw new OutOfStockException();
}

This prevents overselling and ensures the system reflects physical reality.

if (\(cartTotal !== \)calculatedTotal) {
   throw new PriceMismatchException();
}

This protects revenue and prevents tampering.

In this layer, you enforce real‑world business rules by ensuring inventory correctness, recalculating totals, applying discount logic, and checking user‑specific limits.

Jobs / Queues / Data Ingestion — “Validate External Data”

When importing or processing external data (for example, supplier feeds), validation must occur before processing. You'll need to ensure schema conformity, that the required columns are present, that you have the correct data types, that the JSON structure is valid, and that you're detecting duplicate batches.

This is because external data sources are a major source of data quality issues. Without validation here, corrupted data can silently enter the system at scale.

Now that we've discussed the layers of a modern application stack, it should be clear that data quality isn't something you “check once” at the UI.

It must be enforced repeatedly, at multiple depths of the system. Each layer catches a different class of defects, and together they form a defensive wall that prevents bad data from ever reaching storage, analytics, or downstream consumers.

Testing Strategies to Protect Data Quality

To wrap up, here are the three foundational testing strategy every developer should apply to protect data quality.

Unit Testing

Unit tests are the first line of defense in data quality. In this context, a “unit” refers to a single column, a single transformation, or a single validation rule.

The purpose is straightforward: verify that the smallest building blocks of your data logic behave exactly as intended. This matters because if these low‑level rules are not tested and validated, incorrect or inconsistent data will flow into the database and contaminate everything built on top of it.

By isolating each rule or transformation, you can guarantee that schema constraints, field‑level assumptions, and low‑level logic remain correct before data ever flows into larger pipelines or business processes.

Typical questions answered at this layer include:

  1. Does this column allow nulls?

  2. Does this regex correctly strip whitespace from email strings?

  3. Does this transformation produce the expected output for a single row?

This is where you can verify that the data contract is sound. If a column must be non‑null, unique, or follow a specific pattern, the unit test enforces it. When these rules fail here, they fail cheaply – before they can corrupt a table or mislead a dashboard.

To make this concrete, here’s what a unit test looks like in a real codebase. Even though this example comes from Laravel, the testing principle is identical to data‑quality unit tests: one rule, one expectation, isolated from everything else.

Example: Testing a Discount Calculation Rule

Imagine your e‑commerce shop has this rule:

  • If a product costs more than £100, apply a 10% discount.

  • Otherwise, apply no discount.

Let's say this is your discount logic:

<?php

namespace App\Services;

class DiscountService
{
    public function calculate(float $price): float
    {
        if ($price > 100) {
            return $price * 0.10; // 10% discount
        }

        return 0;
    }
}

The unit test for this logic will be:

<?php

namespace Tests\Unit;

use Tests\TestCase;
use App\Services\DiscountService;

class DiscountServiceTest extends TestCase
{
    /** @test */
    public function it_applies_10_percent_discount_when_price_is_above_100()
    {
        $service = new DiscountService();

        \(discount = \)service->calculate(200);

        \(this->assertEquals(20, \)discount);
    }

    /** @test */
    public function it_applies_no_discount_when_price_is_100_or_below()
    {
        $service = new DiscountService();

        \(discount = \)service->calculate(100);

        \(this->assertEquals(0, \)discount);
    }
}

The DiscountService contains a simple rule: if a price is greater than 100, a 10% discount is applied. Otherwise, no discount is applied. The unit test verifies this rule in isolation, without involving controllers, databases, or HTTP requests. By testing the service directly, the developer ensures that the core calculation behaves exactly as intended.

The first test checks the positive case — a price of 200 should produce a discount of 20. The second test checks the boundary condition — a price of 100 should produce no discount. Together, these tests confirm both sides of the rule and protect against regressions if the logic changes in the future.

Now, since this is Laravel example, Laravel tests help you verify both your logic (unit tests) and your full application behaviour (feature tests). You can run them using php artisan test, which executes tests in a separate testing environment, ensuring your real database and main codebase remain safe and unaffected.

Integration Testing: The Flow & Lineage Check

While unit tests validate the correctness of individual rules, integration tests validate the movement of data across components. Integration testing verifies that multiple layers work together as a single data flow.

In this example, the controller receives an order, calls the discount service, applies the transformation, and persists the result to the database. That interaction across layers is what elevates this from a unit test to an integration test. This is where you test the real‑world flow:

  1. Controller → Service → Repository → MySQL

  2. Check if MySQL migrations run correctly

  3. Check foreign keys enforce relationships

  4. Check to ensure services interact with the database as expected

  5. Check to ensure models and repositories behave consistently

Integration tests reveal issues that only appear when components interact: incorrect joins, broken migrations, mismatched field names, or subtle type mismatches that unit tests cannot detect.

This is the layer where you catch the bugs that would otherwise silently corrupt data lineage.

Here's an example:

<?php

namespace Tests\Feature;

use Tests\TestCase;
use App\Models\Order;
use Illuminate\Foundation\Testing\RefreshDatabase;

class ApplyDiscountTest extends TestCase
{
    use RefreshDatabase;

    /** @test */
    public function check_it_persists_the_correct_discounted_total_to_the_database()
    {
        $order = Order::factory()->create(['subtotal' => 150]);

        \(response = \)this->postJson("/orders/{$order->id}/apply-discount");

        $response->assertStatus(200);

        $this->assertDatabaseHas('orders', [
            'id' => $order->id,
            'grand_total' => 135, // 150 - 10% discount
            'discount_total' => 15
        ]);
    }
}

This represents a full flow rather than a single rule:

  • Controller → Service

  • Service → Calculation

  • Controller → Database write

  • Database → Final state

This test begins by creating an order using an Eloquent factory. It immediately steps beyond the boundaries of a unit test, since it interacts with the database and relies on Laravel’s model layer to persist real data.

From there, the test sends an actual HTTP POST request to the /orders/{id}/apply-discount endpoint, which means it's not calling a method directly, but instead it's traveling through Laravel’s routing layer, invoking the controller responsible for handling the request, and triggering whatever business logic is responsible for calculating and applying the discount.

This movement through multiple layers (routing, controller, service logic, and model persistence) is precisely what defines integration testing: the goal is to verify that these components work together correctly as a system.

Once the request is processed, the test asserts that the response returns a successful status code, which confirms that the HTTP layer behaved as expected.

But the most important part comes afterward, when the test checks the database to ensure that the correct grand_total and discount_total were saved. This final assertion proves that the discount logic was executed, the model was updated, and the changes were successfully written to the database.

In other words, the test isn't merely checking whether a calculation is correct. It's also checking whether the entire pipeline – from receiving the request to updating the database – functions as a coherent whole.

Functional Testing: The Business Rule Check

Functional tests validate the entire user experience, from the moment a request enters the system to the moment a response is returned. This includes:

  • HTTP requests

  • Controller logic

  • Validation rules

  • Service operations

  • Database writes

  • Redirects or rendered views

This is where you test the business rules that govern real‑world behaviour:

“A student can't register for two exams at the same time.”

“A cart can't have negative quantities.”

“A user can't update their profile without a valid email.”

Functional tests ensure that the system behaves correctly from the perspective of the user and the business, not just the code.

Here's an example: Functional Test

<?php

namespace Tests\Feature;

use Tests\TestCase;
use App\Models\Product;
use Illuminate\Foundation\Testing\RefreshDatabase;

class CartQuantityFunctionalTest extends TestCase
{
    use RefreshDatabase;

    /** @test */
    public function a_user_cannot_set_a_negative_cart_quantity()
    {
        // Arrange: create a product
        $product = Product::factory()->create(['price' => 40]);

        // Simulate existing cart
        $this->withSession([
            'cart' => [
                $product->id => ['quantity' => 2]
            ]
        ]);

        // Act: user tries to update quantity to a negative number
        \(response = \)this->post('/cart/update', [
            'product_id' => $product->id,
            'quantity' => -5
        ]);

        // Assert: system rejects invalid business behaviour
        $response->assertStatus(302); // redirect back with errors
        $response->assertSessionHasErrors(['quantity']);

        // Assert: cart remains unchanged (business rule preserved)
        \(this->assertEquals(2, session('cart')[\)product->id]['quantity']);
    }
}

The test begins by creating a realistic environment in which a user interacts with a shopping cart. This is essential for understanding the behaviour the system is meant to enforce.

First, it generates a real product in the database using a factory, giving the product a price so that it resembles an item a customer might genuinely add to their cart.

Once the product exists, the test manually seeds the session with a cart containing that product and a quantity of two. This simulates a user who has already added the item to their cart in a previous interaction, and it establishes the baseline state the system must preserve if the user attempts an invalid update.

With the environment prepared, the test then imitates a user action by sending a POST request to the /cart/update endpoint. Instead of calling a method directly, it uses Laravel’s HTTP layer to reproduce the exact behaviour of a browser submitting a form. The request includes the product ID and a deliberately invalid quantity of negative five.

This is the heart of the scenario: the user is attempting something that violates the business rules of the application, and the test is designed to confirm that the system responds appropriately.

Now, when the request is processed, the test expects the application to reject the input, redirect the user back, and attach validation errors to the session. The assertion that the response has a 302 status code and contains validation errors confirms that the validation layer is functioning correctly and that the controller is enforcing the rule that quantities can't be negative.

The final part of the test is where the business rule is truly verified. After the failed update attempt, the test inspects the session to ensure that the cart remains unchanged. This is crucial because rejecting invalid input is only half of the requirement: the system must also protect the integrity of the existing cart data.

Functional tests answer questions like:

  • Does the system prevent invalid real‑world behaviour?

  • Does the user get the correct feedback?

  • Does the data remain consistent after the request?

  • Does the final output match the business expectation?

Conclusion

Data quality is never the result of a single check or a single team. It emerges from a disciplined, layered approach where each testing level catches a different category of defects.

Unit tests safeguard the smallest rules, integration tests validate the flow of data across components, and functional tests enforce the business logic that governs real‑world behaviour.

When these layers operate together, bad data has nowhere to hide. When they don’t, even a minor oversight can slip through the cracks and escalate into a costly downstream failure.

So as you can see, your role in data quality is fundamentally proactive, not reactive. By designing systems with validation, integrity, and monitoring in mind, you ensure that data flowing through the pipeline is accurate, timely, complete, unique, and fit for purpose – supporting reliable analytics, reporting, and intelligent systems.

]]> <![CDATA[ How to Build an End-to-End ML Platform Locally: From Experiment Tracking to CI/CD ]]> https://www.freecodecamp.org/news/build-end-to-end-ml-platform-locally-from-experiment-tracking-to-cicd/ 69b9bab4c22d3eeb8afd5284 Tue, 17 Mar 2026 20:33:56 +0000 Machine learning projects don’t end at training a model in a Jupyter notebook. The hard part is the “last mile”: turning that notebook model into something you can run reliably, update safely, and trust over time.

Most ML systems fail in production for boring (and painful) reasons: the training code and the serving code drift apart, input data changes shape, a “small” preprocessing tweak breaks predictions, or the model silently degrades because real-world behavior shifts. None of these problems are solved by a better algorithm, they’re solved by engineering: repeatable pipelines, validation, versioning, monitoring, and automated checks.

In this hands-on handbook, you’ll build a complete mini ML platform on your local machine, an end-to-end project that takes a model from training to deployment with the core “last mile” infrastructure in place. We’ll use a fraud detection example (predicting fraudulent transactions), but the same workflow works for churn prediction or any binary classification problem. Everything runs locally (no cloud required), and every step is copy-paste runnable so you can follow along and verify outputs as you go.

By the end, you'll have a production-ready ML pipeline running on your machine – from training the model to serving predictions, with the infrastructure to test, monitor, and iterate with confidence. And yes, we'll do it in a hands-on manner with code snippets you can copy-paste and run. Let's dive in!

📦 Get the Complete Code
All code from this handbook is available in a ready-to-run repository:
Repository: https://github.com/sandeepmb/freecodecamp-local-ml-platform
Clone it and follow along, or use it as a reference implementation.

Table of Contents

  1. Project Overview and Setup

  2. Build a Simple Model and API (The Naive Approach)

  3. Where the Naive Approach Breaks

  4. Add Experiment Tracking and Model Registry with MLflow

  5. Ensure Feature Consistency with Feast

  6. Add Data Validation with Great Expectations

  7. Monitor Model Performance and Data Drift

  8. Automate Testing and Deployment with CI/CD

  9. Incident Response Playbook

  10. How to Put It All Together

  11. What’s Next: Scale to Production

  12. Conclusion

  13. References

Project Overview and Setup

Before we jump into coding, let's set the stage. Our use-case is credit card fraud detection – a binary classification problem where we predict whether a transaction is fraudulent (is_fraud = 1) or legitimate (is_fraud = 0). This is a common ML task and a good proxy for production ML challenges because fraud patterns can change over time (allowing us to discuss model drift), and bad input data (for example, malformed transaction info) can cause serious issues if not handled properly.

Tech Stack

We will use Python-based tools that are popular in MLOps but still beginner-friendly:

Tool Purpose Why We Chose It
MLflow Experiment tracking and model registry Open-source, widely adopted, great UI
Feast Feature store for consistent feature serving Production-grade, runs locally, same API for offline/online
FastAPI High-performance web framework for serving predictions Fast, automatic docs, modern Python
Great Expectations Data validation framework Declarative expectations, great reports
Evidently Monitoring for data drift and model decay Beautiful reports, easy to integrate
Docker Containerization for environment consistency Industry standard, works everywhere
GitHub Actions CI/CD automation Free for public repos, tight GitHub integration

Let me explain each tool briefly:

MLflow is an open-source platform designed to manage the ML lifecycle. It provides experiment tracking (logging parameters, metrics, and artifacts), a model registry (versioning models with aliases), and model serving capabilities. We'll use it to ensure our experiments are reproducible and our models are versioned.

Feast (Feature Store) is an open-source feature store that helps manage and serve features consistently between training and inference. This prevents a common problem called "training-serving skew" where the features used in production differ slightly from those used in training, causing silent accuracy degradation.

FastAPI is a modern, fast web framework for building APIs with Python. It's known for being easy to use, efficient, and producing automatic interactive documentation. We'll use it to serve our model predictions.

Great Expectations is an open-source tool for data quality testing. It allows us to define "expectations" on data (like "amount should be positive" or "hour should be between 0 and 23") and test incoming data against them.

Evidently is an open-source library for monitoring data and model performance over time. It can detect data drift (when input distributions change) and model decay (when accuracy drops).

Docker ensures the same environment and dependencies in development and deployment, avoiding the classic "works on my machine" problem.

GitHub Actions provides CI/CD automation. An efficient CI/CD pipeline helps integrate and deploy changes faster and with fewer errors.

💡 Mental Model: Think of this as building a "safety net" around your ML model. Each tool we add catches a different failure mode, like defensive driving for machine learning.

Prerequisites

You'll need:

  • Python 3.9+ installed on your machine

  • Docker Desktop installed and running

  • GitHub account (if you want to try the CI/CD pipeline)

  • Basic familiarity with Python and ML concepts (what training and prediction mean)

You don't need MLOps or Kubernetes experience. Everything will be done locally with just Python and Docker – no cloud and no Kubernetes needed.

Project Structure

Let's set up a basic project structure on your local machine. Open your terminal and run:

# Create project directory and subfolders
mkdir ml-platform-tutorial && cd ml-platform-tutorial
mkdir -p data models src tests feature_repo

# Set up a virtual environment (recommended)
python -m venv venv
source venv/bin/activate   # On Windows: venv\Scripts\activate

Your project structure should look like this:

ml-platform-tutorial/
├── data/              # Training and test datasets
├── models/            # Saved model files
├── src/               # Source code
├── tests/             # Test files
├── feature_repo/      # Feast feature repository
├── venv/              # Virtual environment
└── requirements.txt   # Dependencies

Next, create a requirements.txt with all the necessary libraries:

# requirements.txt

# Core ML libraries
pandas==2.2.0
numpy==1.26.3
scikit-learn==1.4.0

# Experiment tracking and model registry
mlflow==2.10.0

# Feature store
feast==0.36.0

# API framework
fastapi==0.109.0
uvicorn==0.27.0
httpx==0.26.0

# Data validation
great-expectations==0.18.8

# Monitoring
evidently==0.7.20

# Testing
pytest==8.0.0
pytest-cov==4.1.0

# Utilities
pyarrow==15.0.0
pydantic==2.6.0

📌 Version Note: Exact versions are pinned to ensure reproducibility. Newer versions may work, but all examples were tested with the versions listed here.

Install the dependencies:

pip install -r requirements.txt

This might take a few minutes as it installs all the packages. Once complete, we're ready to start building our project step by step.

Checkpoint: You should have a project folder with data/, models/, src/, tests/, and feature_repo/ directories, and an activated virtual environment with all dependencies installed. Verify by running python -c "import mlflow; import feast; import fastapi; print('All imports successful!')".

Figure 1: The Complete ML Platform We'll Build

Don't worry if this looks complex, we'll build each component step by step, starting with the simplest piece and connecting them together.

Architecture diagram of a local end-to-end machine learning platform for fraud detection. Transaction data flows through model training, experiment tracking and model registry in MLflow, feature management in Feast, data validation with Great Expectations, prediction serving through FastAPI, monitoring with Evidently, and automated testing and deployment with Docker and GitHub Actions.

1. Build a Simple Model and API (The Naive Approach)

To illustrate why we need all these tools, let's start by building a naive ML system without any MLOps infrastructure. We'll train a simple model and deploy it quickly, then observe what problems arise. This "naive approach" is how most ML projects start – and understanding its limitations will motivate the solutions we implement later.

1.1 Train a Quick Model

First, we need some data. For simplicity, we'll generate a synthetic dataset for fraud detection so that we don't rely on any external data files. The dataset will have features like:

  • amount: Transaction amount in dollars

  • hour: Hour of the day (0-23) when the transaction occurred

  • day_of_week: Day of the week (0=Monday, 6=Sunday)

  • merchant_category: Type of merchant (grocery, restaurant, retail, online, travel)

  • is_fraud: Label indicating if the transaction is fraudulent (1) or legitimate (0)

We will simulate that only ~2% of transactions are fraud, which is an imbalance typical in real fraud data. This imbalance is important because it affects how we evaluate our model.

Create src/generate_data.py:

# src/generate_data.py
"""
Generate synthetic fraud detection dataset.

This script creates realistic-looking transaction data where fraudulent
transactions have different patterns than legitimate ones:
- Fraud tends to have higher amounts
- Fraud tends to occur late at night
- Fraud is more common for online and travel merchants
"""
import pandas as pd
import numpy as np

def generate_transactions(n_samples=10000, fraud_ratio=0.02, seed=42):
    """
    Generate synthetic fraud detection dataset.
    
    Args:
        n_samples: Total number of transactions to generate
        fraud_ratio: Proportion of fraudulent transactions (default 2%)
        seed: Random seed for reproducibility
    
    Returns:
        DataFrame with transaction features and fraud labels
    
    Fraud transactions have different patterns:
    - Higher amounts (mean \(245 vs \)33 for legit)
    - Late night hours (0-5, 23)
    - More likely to be online or travel merchants
    """
    np.random.seed(seed)
    n_fraud = int(n_samples * fraud_ratio)
    n_legit = n_samples - n_fraud

    # Legitimate transactions: normal shopping patterns
    # - Amounts follow a log-normal distribution (most small, some large)
    # - Hours are uniformly distributed throughout the day
    # - Merchant categories weighted toward everyday shopping
    legit = pd.DataFrame({
        "amount": np.random.lognormal(mean=3.5, sigma=1.2, size=n_legit),  # ~$33 average
        "hour": np.random.randint(0, 24, size=n_legit),
        "day_of_week": np.random.randint(0, 7, size=n_legit),
        "merchant_category": np.random.choice(
            ["grocery", "restaurant", "retail", "online", "travel"],
            size=n_legit,
            p=[0.30, 0.25, 0.25, 0.15, 0.05]  # Weighted toward everyday shopping
        ),
        "is_fraud": 0
    })
    
    # Fraudulent transactions: suspicious patterns
    # - Higher amounts (fraudsters go big)
    # - Late night hours (less scrutiny)
    # - More online and travel (easier to exploit)
    fraud = pd.DataFrame({
        "amount": np.random.lognormal(mean=5.5, sigma=1.5, size=n_fraud),  # ~$245 average
        "hour": np.random.choice([0, 1, 2, 3, 4, 5, 23], size=n_fraud),  # Late night
        "day_of_week": np.random.randint(0, 7, size=n_fraud),
        "merchant_category": np.random.choice(
            ["grocery", "restaurant", "retail", "online", "travel"],
            size=n_fraud,
            p=[0.05, 0.05, 0.10, 0.60, 0.20]  # Weighted toward online/travel
        ),
        "is_fraud": 1
    })
    
    # Combine and shuffle
    df = pd.concat([legit, fraud], ignore_index=True)
    df = df.sample(frac=1, random_state=seed).reset_index(drop=True)
    
    return df

if __name__ == "__main__":
    # Generate dataset
    print("Generating synthetic fraud detection dataset...")
    df = generate_transactions(n_samples=10000, fraud_ratio=0.02)
    
    # Split into train (80%) and test (20%)
    train_df = df.sample(frac=0.8, random_state=42)
    test_df = df.drop(train_df.index)
    
    # Save to CSV files
    train_df.to_csv("data/train.csv", index=False)
    test_df.to_csv("data/test.csv", index=False)
    
    # Print summary statistics
    print(f"\nDataset generated successfully!")
    print(f"Training set: {len(train_df):,} transactions")
    print(f"Test set: {len(test_df):,} transactions")
    print(f"Overall fraud ratio: {df['is_fraud'].mean():.2%}")
    print(f"\nLegitimate transactions - Average amount: ${df[df['is_fraud']==0]['amount'].mean():.2f}")
    print(f"Fraudulent transactions - Average amount: ${df[df['is_fraud']==1]['amount'].mean():.2f}")
    print(f"\nMerchant category distribution (fraud):")
    print(df[df['is_fraud']==1]['merchant_category'].value_counts(normalize=True))

Run the data generation script:

python src/generate_data.py

You should see output like:

Generating synthetic fraud detection dataset...

Dataset generated successfully!
Training set: 8,000 transactions
Test set: 2,000 transactions
Overall fraud ratio: 2.00%

Legitimate transactions - Average amount: $33.45
Fraudulent transactions - Average amount: $245.67

Merchant category distribution (fraud):
online        0.60
travel        0.20
retail        0.10
restaurant    0.05
grocery       0.05

Now you have data/train.csv and data/test.csv with ~8000 training and ~2000 testing transactions.

Why This Matters: The synthetic data has realistic patterns — fraud is rare (2%), high-value, late-night, and concentrated in certain merchant categories. These patterns give our model something to learn.

Now, let's train a quick model. We'll use a simple Random Forest classifier from scikit-learn to predict is_fraud. In this naive version, we won't do much feature engineering – just label encode the categorical merchant_category and feed everything to the model.

Create src/train_naive.py:

# src/train_naive.py
"""
Train a fraud detection model - NAIVE VERSION.

This script demonstrates the "quick and dirty" approach to ML:
- No experiment tracking
- No model versioning
- Just train and save to a pickle file

We'll improve on this in later sections.
"""
import pandas as pd
import pickle
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import (
    accuracy_score, 
    f1_score, 
    precision_score, 
    recall_score,
    confusion_matrix,
    classification_report
)

def main():
    print("Loading data...")
    train_df = pd.read_csv("data/train.csv")
    test_df = pd.read_csv("data/test.csv")
    
    print(f"Training samples: {len(train_df):,}")
    print(f"Test samples: {len(test_df):,}")
    print(f"Training fraud ratio: {train_df['is_fraud'].mean():.2%}")
    
    # Encode the categorical feature
    # We need to save the encoder to use the same mapping at inference time
    print("\nEncoding categorical features...")
    encoder = LabelEncoder()
    train_df["merchant_encoded"] = encoder.fit_transform(train_df["merchant_category"])
    test_df["merchant_encoded"] = encoder.transform(test_df["merchant_category"])
    
    print(f"Merchant category mapping: {dict(zip(encoder.classes_, encoder.transform(encoder.classes_)))}")
    
    # Prepare features and labels
    feature_cols = ["amount", "hour", "day_of_week", "merchant_encoded"]
    X_train = train_df[feature_cols]
    y_train = train_df["is_fraud"]
    X_test = test_df[feature_cols]
    y_test = test_df["is_fraud"]
    
    # Train a Random Forest classifier
    print("\nTraining Random Forest model...")
    model = RandomForestClassifier(
        n_estimators=100,      # Number of trees
        max_depth=10,          # Maximum depth of each tree
        random_state=42,       # For reproducibility
        n_jobs=-1              # Use all CPU cores
    )
    model.fit(X_train, y_train)
    print("Training complete!")
    
    # Evaluate on test data
    print("\n" + "="*50)
    print("MODEL EVALUATION")
    print("="*50)
    
    y_pred = model.predict(X_test)
    y_prob = model.predict_proba(X_test)[:, 1]
    
    print(f"\nAccuracy:  {accuracy_score(y_test, y_pred):.4f}")
    print(f"Precision: {precision_score(y_test, y_pred):.4f}")
    print(f"Recall:    {recall_score(y_test, y_pred):.4f}")
    print(f"F1-score:  {f1_score(y_test, y_pred):.4f}")
    
    print("\nConfusion Matrix:")
    cm = confusion_matrix(y_test, y_pred)
    print(f"  True Negatives:  {cm[0][0]:,} (correctly identified legitimate)")
    print(f"  False Positives: {cm[0][1]:,} (legitimate flagged as fraud)")
    print(f"  False Negatives: {cm[1][0]:,} (fraud missed - DANGEROUS!)")
    print(f"  True Positives:  {cm[1][1]:,} (correctly caught fraud)")
    
    print("\nClassification Report:")
    print(classification_report(y_test, y_pred, target_names=['Legitimate', 'Fraud']))
    
    # Feature importance
    print("\nFeature Importance:")
    for name, importance in sorted(
        zip(feature_cols, model.feature_importances_),
        key=lambda x: x[1],
        reverse=True
    ):
        print(f"  {name}: {importance:.4f}")
    
    # Save the model and encoder together
    print("\nSaving model to models/model.pkl...")
    with open("models/model.pkl", "wb") as f:
        pickle.dump((model, encoder), f)
    
    print("\nModel trained and saved successfully!")
    print("\nWARNING: This naive approach has several problems:")
    print("  - No record of hyperparameters or metrics")
    print("  - No model versioning")
    print("  - No way to reproduce this exact model")
    print("  - We'll fix these issues in the following sections!")

if __name__ == "__main__":
    main()

Run the training script:

python src/train_naive.py

You should see output similar to:

Loading data...
Training samples: 8,000
Test samples: 2,000
Training fraud ratio: 2.00%

Encoding categorical features...
Merchant category mapping: {'grocery': 0, 'online': 1, 'restaurant': 2, 'retail': 3, 'travel': 4}

Training Random Forest model...
Training complete!

==================================================
MODEL EVALUATION
==================================================

Accuracy:  0.9820
Precision: 0.7273
Recall:    0.6154
F1-score:  0.6667

Confusion Matrix:
  True Negatives:  1,956 (correctly identified legitimate)
  False Positives: 4 (legitimate flagged as fraud)
  False Negatives: 32 (fraud missed - DANGEROUS!)
  True Positives:  8 (correctly caught fraud)

Feature Importance:
  amount: 0.5423
  hour: 0.2156
  merchant_encoded: 0.1345
  day_of_week: 0.1076

Important observation: You'll see ~98% accuracy but a lower F1-score (around 0.5-0.7). With only 2% fraud, accuracy is extremely misleading! A model that always predicts "not fraud" would achieve 98% accuracy while catching zero fraud. This is why we focus on F1-score, precision, and recall for imbalanced classification problems.

💡 If you're new to imbalanced classification, remember: high accuracy can be meaningless when the positive class is rare.

The script outputs a file models/model.pkl containing both the trained model and the label encoder (we need both for inference).

Checkpoint: You should now have:

  • data/train.csv (~8,000 rows)

  • data/test.csv (~2,000 rows)

  • models/model.pkl (trained model + encoder)

The model should show ~98% accuracy but F1 around 0.5-0.7. Verify the files exist: ls -la data/ models/

1.2 Serve Predictions with FastAPI

Now that we have a model, let's deploy it as an API so that clients can get predictions. We'll use FastAPI because it's straightforward, very fast, and produces automatic interactive documentation.

FastAPI is known for:

  • Easy to use: Pythonic syntax with type hints

  • High performance: One of the fastest Python frameworks

  • Automatic documentation: Swagger UI out of the box

  • Data validation: Using Pydantic models

Create src/serve_naive.py:

# src/serve_naive.py
"""
Serve fraud detection model as a REST API - NAIVE VERSION.

This is a simple API that:
1. Loads the trained model at startup
2. Accepts transaction data via POST request
3. Returns fraud prediction

We'll improve this with validation, monitoring, and better
model loading in later sections.
"""
import pickle
from fastapi import FastAPI
from pydantic import BaseModel, Field
from typing import Optional

# Load the trained model and encoder at startup
# This is loaded once when the server starts, not on every request
print("Loading model...")
with open("models/model.pkl", "rb") as f:
    model, encoder = pickle.load(f)
print("Model loaded successfully!")

# Create the FastAPI application
app = FastAPI(
    title="Fraud Detection API",
    description="""
    Predict whether a credit card transaction is fraudulent.
    
    This API accepts transaction details and returns:
    - Whether the transaction is predicted to be fraud
    - The probability of fraud (0.0 to 1.0)
    
    **Note:** This is the naive version without validation or monitoring.
    """,
    version="1.0.0"
)

# Define the input schema using Pydantic
# This provides automatic validation and documentation
class Transaction(BaseModel):
    """Schema for a transaction to be evaluated for fraud."""
    amount: float = Field(
        ..., 
        description="Transaction amount in dollars",
        example=150.00
    )
    hour: int = Field(
        ..., 
        description="Hour of the day (0-23)",
        example=14
    )
    day_of_week: int = Field(
        ..., 
        description="Day of week (0=Monday, 6=Sunday)",
        example=3
    )
    merchant_category: str = Field(
        ..., 
        description="Type of merchant",
        example="online"
    )

class PredictionResponse(BaseModel):
    """Schema for the prediction response."""
    is_fraud: bool = Field(description="Whether the transaction is predicted as fraud")
    fraud_probability: float = Field(description="Probability of fraud (0.0 to 1.0)")
    
@app.post("/predict", response_model=PredictionResponse)
def predict(transaction: Transaction):
    """
    Predict whether a transaction is fraudulent.
    
    Takes transaction details and returns a fraud prediction
    along with the probability score.
    """
    # Convert the request to a dictionary
    data = transaction.dict()
    
    # Encode the merchant category using the same encoder from training
    # This ensures consistency between training and serving
    try:
        data["merchant_encoded"] = encoder.transform([data["merchant_category"]])[0]
    except ValueError:
        # Handle unknown merchant categories
        # In production, we'd want better handling here
        data["merchant_encoded"] = 0
    
    # Prepare features in the same order as training
    X = [[
        data["amount"],
        data["hour"],
        data["day_of_week"],
        data["merchant_encoded"]
    ]]
    
    # Get prediction and probability
    prediction = model.predict(X)[0]
    probability = model.predict_proba(X)[0][1]  # Probability of class 1 (fraud)
    
    return PredictionResponse(
        is_fraud=bool(prediction),
        fraud_probability=round(float(probability), 4)
    )

@app.get("/health")
def health_check():
    """
    Health check endpoint.
    
    Returns the status of the API. Useful for:
    - Load balancer health checks
    - Kubernetes liveness probes
    - Monitoring systems
    """
    return {
        "status": "healthy",
        "model_loaded": model is not None
    }

@app.get("/")
def root():
    """Root endpoint with API information."""
    return {
        "message": "Fraud Detection API",
        "version": "1.0.0",
        "docs": "/docs",
        "health": "/health"
    }

A few important things to note about this code:

  1. Pydantic Models: We use BaseModel to define the expected input JSON schema. FastAPI automatically validates incoming requests against this schema.

  2. Type Hints: The type hints (float, int, str) provide both documentation and runtime validation.

  3. Feature Encoding: On each request, we encode the merchant category using the same LabelEncoder we saved from training. This ensures consistency between training and serving.

  4. Health Endpoint: The /health endpoint is standard practice for production APIs - it allows load balancers and monitoring systems to check if the service is running.

To run this API, use Uvicorn (an ASGI server):

uvicorn src.serve_naive:app --reload --host 0.0.0.0 --port 8000

The --reload flag enables auto-reload during development (the server restarts when you change code).

You should see:

Loading model...
Model loaded successfully!
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process

Now open your browser and go to http://localhost:8000/docs. You'll see the Swagger UI – an auto-generated interactive documentation where you can test the API directly from your browser!

Test the API using curl in another terminal:

# Test with a legitimate-looking transaction
curl -X POST "http://localhost:8000/predict" \
  -H "Content-Type: application/json" \
  -d '{"amount": 50.0, "hour": 14, "day_of_week": 3, "merchant_category": "grocery"}'

Expected response:

{"is_fraud": false, "fraud_probability": 0.02}

# Test with a suspicious transaction (high amount, late night, online)
curl -X POST "http://localhost:8000/predict" \
  -H "Content-Type: application/json" \
  -d '{"amount": 500.0, "hour": 3, "day_of_week": 1, "merchant_category": "online"}'

Expected response:

{"is_fraud": true, "fraud_probability": 0.78}

We have a working model served as an API! In a real scenario, we could now integrate this API with a payment processing frontend, mobile app, or any system that needs fraud predictions.

But before we celebrate, let's examine this naive approach for potential pitfalls...

Checkpoint: Your API should be running at http://localhost:8000. The Swagger UI at /docs should show both endpoints (/predict and /health). Test with curl or the Swagger UI to verify predictions are returned.

2. Where the Naive Approach Breaks

Our quick-and-dirty ML pipeline works on the surface: it can train a model and serve predictions. However, hidden problems will emerge if we try to maintain or scale this system in production.

This section is critical: understanding these issues will motivate the solutions we implement in the following sections. Let's go through the problems one by one.

Problem 1: No Experiment Tracking (Reproducibility)

Try this thought experiment: Run train_naive.py again with different hyperparameters (change n_estimators to 200, or max_depth to 15). Would you be able to exactly reproduce the previous model's results if someone asked?

Probably not. Currently, we have no record of:

  • Which hyperparameters we used

  • What metrics we achieved

  • What version of the data we trained on

  • What library versions were installed

  • When the training happened

  • Who ran the training

Three months from now, if your manager asks "How was this model trained? Can you reproduce the results?" – you'd be in trouble. You might have the code, but you don't know which version of the code, which parameters, or which data produced the model that's currently in production.

Experiment tracking is the practice of logging all these details (code versions, parameters, metrics, data versions, artifacts) so experiments can be compared and replicated. Our naive approach lacks this entirely, making our results hard to trust or build upon.

Problem 2: Model Versioning and Deployment Chaos

We trained one model and saved it as model.pkl. Now consider this scenario:

  1. You train a new model with different hyperparameters

  2. You overwrite model.pkl with the new model

  3. You deploy it to production

  4. Users start complaining about more false positives

  5. You want to roll back to the previous model

  6. Problem: The previous model was overwritten and is gone forever

There's no systematic versioning. Questions you cannot answer:

  • Which model version is currently in production?

  • What were the metrics for model v1 vs v2?

  • When was each model trained and by whom?

  • Can we instantly roll back if the new model performs worse?

  • What changed between versions?

Without version control for models, you're flying blind. Imagine deploying code without Git – that's what we're doing with our model.

Problem 3: No Data Validation – Garbage In, Garbage Out

Right now, our API will accept any input and try to make a prediction. Let's see what happens with bad data.

Create a test script src/test_bad_data.py:

# src/test_bad_data.py
"""Test what happens when we send garbage data to the API."""
import requests

BASE_URL = "http://localhost:8000"

print("Testing API with various bad inputs...\n")

# Test 1: Negative amount
print("Test 1: Negative amount")
response = requests.post(f"{BASE_URL}/predict", json={
    "amount": -500.0,        # Negative amount - impossible!
    "hour": 14,
    "day_of_week": 3,
    "merchant_category": "online"
})
print(f"  Status: {response.status_code}")
print(f"  Response: {response.json()}\n")

# Test 2: Invalid hour
print("Test 2: Hour = 25 (should be 0-23)")
response = requests.post(f"{BASE_URL}/predict", json={
    "amount": 100.0,
    "hour": 25,              # Invalid hour!
    "day_of_week": 3,
    "merchant_category": "online"
})
print(f"  Status: {response.status_code}")
print(f"  Response: {response.json()}\n")

# Test 3: Invalid day of week
print("Test 3: day_of_week = 10 (should be 0-6)")
response = requests.post(f"{BASE_URL}/predict", json={
    "amount": 100.0,
    "hour": 14,
    "day_of_week": 10,       # Invalid day!
    "merchant_category": "online"
})
print(f"  Status: {response.status_code}")
print(f"  Response: {response.json()}\n")

# Test 4: Unknown merchant category
print("Test 4: Unknown merchant category")
response = requests.post(f"{BASE_URL}/predict", json={
    "amount": 100.0,
    "hour": 14,
    "day_of_week": 3,
    "merchant_category": "unknown_category"  # Not in training data!
})
print(f"  Status: {response.status_code}")
print(f"  Response: {response.json()}\n")

# Test 5: All bad at once
print("Test 5: Everything wrong")
response = requests.post(f"{BASE_URL}/predict", json={
    "amount": -1000.0,
    "hour": 99,
    "day_of_week": 15,
    "merchant_category": "totally_fake"
})
print(f"  Status: {response.status_code}")
print(f"  Response: {response.json()}\n")

print("Observation: The API happily accepts ALL garbage and returns predictions!")
print("This is dangerous - bad data leads to bad predictions with no warning.")

Run it (make sure your API is still running):

python src/test_bad_data.py

You'll see something like:

Testing API with various bad inputs...

Test 1: Negative amount
  Status: 200
  Response: {'is_fraud': False, 'fraud_probability': 0.15}

Test 2: Hour = 25 (should be 0-23)
  Status: 200
  Response: {'is_fraud': False, 'fraud_probability': 0.08}

...

Observation: The API happily accepts ALL garbage and returns predictions!

The API accepts garbage and returns predictions with no warning! In production, this could mean:

  • Incorrect predictions based on impossible data

  • Fraud going undetected because of malformed input

  • Legitimate transactions blocked based on corrupted data

  • No way to debug why predictions are wrong

As the saying goes: "Garbage in, garbage out." But even worse – we don't even know garbage went in!

Problem 4: Model Drift – Performance Decay Over Time

Here's a scenario that happens in every production ML system:

  1. January: You train your model on historical fraud data. It achieves 98% accuracy and 0.67 F1-score. Everyone's happy.

  2. February: The model is deployed and working well. Fraud is being caught.

  3. March: Fraudsters adapt. They start using different patterns – smaller amounts, different merchant categories, different times of day.

  4. April: Your model's accuracy has dropped from 98% to 85%. F1-score dropped from 0.67 to 0.35. Fraud is slipping through.

  5. May: A major fraud incident occurs. Investigation reveals the model has been underperforming for 2 months.

The problem: Nobody noticed for 2 months because there was no monitoring.

This phenomenon is called data drift (when input data distributions change) or concept drift (when the relationship between inputs and outputs changes). Both are inevitable in real-world systems.

Without monitoring:

  • You don't know when performance degrades

  • You don't know why performance degrades

  • You can't take corrective action until users complain

  • By then, significant damage may have occurred

Problem 5: No CI/CD or Deployment Safety

Our "deployment process" was literally:

  1. SSH into the server (or run locally)

  2. Run python src/train_naive.py

  3. Copy model.pkl to the right place

  4. Restart the API

  5. Hope for the best

There's:

  • No automated testing: A typo could break everything

  • No staging environment: We test directly in production

  • No gradual rollout: 100% of traffic hits the new model immediately

  • No rollback capability: If something breaks, we have to manually fix it

  • No audit trail: Who deployed what and when?

This is how production incidents happen. A rushed deployment at 5 PM on Friday breaks the fraud detection system, and nobody notices until Monday when fraud losses have spiked.

Figure 2: Problems with the Naive Approach

Diagram showing the weaknesses of a naive machine learning setup: manual training and deployment, no experiment tracking, no model versioning, inconsistent features between training and serving, no data validation, no drift or performance monitoring, and no CI/CD safeguards such as automated tests, rollback, or audit trail.

Summary: What We Need to Fix

Our simple ML service is missing critical infrastructure. Here's the mapping of problems to solutions:

Problem Impact Solution Section
No experiment tracking Can't reproduce or compare models MLflow Tracking 3
No model versioning Can't roll back or audit MLflow Registry 3
No feature consistency Training-serving skew Feast Feature Store 4
No data validation Garbage predictions Great Expectations 5
No monitoring Drift goes unnoticed Evidently 6
No CI/CD Risky deployments GitHub Actions + Docker 7

The good news: We can fix each of these by incrementally adding components to our pipeline. Each tool addresses a specific problem, and together they form a robust ML platform.

Let's start fixing these issues, one by one.

3. Add Experiment Tracking and Model Registry with MLflow

What breaks without this: You can't reproduce yesterday's results, can't compare experiments, and can't roll back when a new model fails in production.

Our first fix addresses Problems 1 and 2: experiment reproducibility and model versioning.

MLflow is an open-source platform designed to manage the ML lifecycle. We'll use two of its key components:

  1. MLflow Tracking: Log experiments (parameters, metrics, artifacts) so you can compare runs and reproduce results

  2. MLflow Model Registry: Version your models with aliases (champion, challenger) and manage the deployment lifecycle

Why This Matters: Without tracking, ML is guesswork. With MLflow, every run is logged with parameters, metrics, and artifacts. You can compare runs side-by-side, understand what actually improved your model, and reproduce any past experiment. The Model Registry adds governance – you know exactly which model is in production and can roll back in seconds.

3.1 How to Set Up the MLflow Tracking Server

MLflow can log experiments to a local directory by default, but to use the full UI and model registry, it's best to run the MLflow tracking server.

Open a new terminal (keep it separate from your API terminal) and run:

# Create a directory for MLflow data
mkdir -p mlruns

# Start the MLflow server
mlflow server \
    --host 0.0.0.0 \
    --port 5000 \
    --backend-store-uri sqlite:///mlflow.db \
    --default-artifact-root ./mlruns

Let's break down these parameters:

  • --host 0.0.0.0: Listen on all network interfaces

  • --port 5000: Run on port 5000

  • --backend-store-uri sqlite:///mlflow.db: Store experiment metadata in a SQLite database (for production, you'd use PostgreSQL or MySQL)

  • --default-artifact-root ./mlruns: Store model artifacts (files) in the mlruns directory

You should see:

[INFO] Starting gunicorn 21.2.0
[INFO] Listening at: http://0.0.0.0:5000

Now open your browser and navigate to http://localhost:5000. You'll see the MLflow UI – it should be empty initially since we haven't logged any experiments yet.

3.2 How to Log Experiments in Code

Now let's modify our training script to log everything to MLflow. Create src/train_mlflow.py:

# src/train_mlflow.py
"""
Train fraud detection model with MLflow experiment tracking.

This script demonstrates proper ML experiment tracking:
- Log all hyperparameters
- Log all metrics (train and test)
- Log the trained model as an artifact
- Register the model in the Model Registry

Compare this to train_naive.py to see the difference!
"""
import pandas as pd
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import (
    accuracy_score, 
    precision_score, 
    recall_score, 
    f1_score,
    roc_auc_score
)
import pickle
from datetime import datetime

# Configure MLflow to use our tracking server
mlflow.set_tracking_uri("http://localhost:5000")

# Create or get the experiment
# All runs will be grouped under this experiment name
mlflow.set_experiment("fraud-detection")

def load_and_preprocess_data():
    """Load and preprocess the training and test data."""
    print("Loading data...")
    train_df = pd.read_csv("data/train.csv")
    test_df = pd.read_csv("data/test.csv")
    
    # Encode categorical feature
    encoder = LabelEncoder()
    train_df["merchant_encoded"] = encoder.fit_transform(train_df["merchant_category"])
    test_df["merchant_encoded"] = encoder.transform(test_df["merchant_category"])
    
    # Prepare features
    feature_cols = ["amount", "hour", "day_of_week", "merchant_encoded"]
    X_train = train_df[feature_cols]
    y_train = train_df["is_fraud"]
    X_test = test_df[feature_cols]
    y_test = test_df["is_fraud"]
    
    return X_train, y_train, X_test, y_test, encoder

def train_and_log_model(
    n_estimators: int = 100,
    max_depth: int = 10,
    min_samples_split: int = 2,
    min_samples_leaf: int = 1
):
    """
    Train a model and log everything to MLflow.
    
    Args:
        n_estimators: Number of trees in the forest
        max_depth: Maximum depth of each tree
        min_samples_split: Minimum samples required to split a node
        min_samples_leaf: Minimum samples required at a leaf node
    """
    X_train, y_train, X_test, y_test, encoder = load_and_preprocess_data()
    
    # Start an MLflow run - everything logged will be associated with this run
    with mlflow.start_run():
        # Add a descriptive run name
        run_name = f"rf_est{n_estimators}_depth{max_depth}_{datetime.now().strftime('%H%M%S')}"
        mlflow.set_tag("mlflow.runName", run_name)
        
        # Log all hyperparameters
        # These are the "knobs" we can tune
        mlflow.log_param("n_estimators", n_estimators)
        mlflow.log_param("max_depth", max_depth)
        mlflow.log_param("min_samples_split", min_samples_split)
        mlflow.log_param("min_samples_leaf", min_samples_leaf)
        mlflow.log_param("model_type", "RandomForestClassifier")
        
        # Log data information
        mlflow.log_param("train_samples", len(X_train))
        mlflow.log_param("test_samples", len(X_test))
        mlflow.log_param("fraud_ratio", float(y_train.mean()))
        mlflow.log_param("n_features", X_train.shape[1])
        
        # Train the model
        print(f"\nTraining model: n_estimators={n_estimators}, max_depth={max_depth}")
        model = RandomForestClassifier(
            n_estimators=n_estimators,
            max_depth=max_depth,
            min_samples_split=min_samples_split,
            min_samples_leaf=min_samples_leaf,
            random_state=42,
            n_jobs=-1
        )
        model.fit(X_train, y_train)
        
        # Evaluate and log metrics for BOTH train and test sets
        # This helps detect overfitting
        for dataset_name, X, y in [("train", X_train, y_train), ("test", X_test, y_test)]:
            y_pred = model.predict(X)
            y_prob = model.predict_proba(X)[:, 1]
            
            # Calculate all metrics
            accuracy = accuracy_score(y, y_pred)
            precision = precision_score(y, y_pred, zero_division=0)
            recall = recall_score(y, y_pred, zero_division=0)
            f1 = f1_score(y, y_pred, zero_division=0)
            roc_auc = roc_auc_score(y, y_prob)
            
            # Log metrics with dataset prefix
            mlflow.log_metric(f"{dataset_name}_accuracy", accuracy)
            mlflow.log_metric(f"{dataset_name}_precision", precision)
            mlflow.log_metric(f"{dataset_name}_recall", recall)
            mlflow.log_metric(f"{dataset_name}_f1", f1)
            mlflow.log_metric(f"{dataset_name}_roc_auc", roc_auc)
            
            print(f"  {dataset_name.upper()} - Accuracy: {accuracy:.4f}, F1: {f1:.4f}, ROC-AUC: {roc_auc:.4f}")
        
        # Log feature importance
        for feature, importance in zip(
            ["amount", "hour", "day_of_week", "merchant_encoded"],
            model.feature_importances_
        ):
            mlflow.log_metric(f"importance_{feature}", importance)
        
        # Log the model to MLflow AND register it in the Model Registry
        # This creates a new version of the model automatically
        print("\nRegistering model in MLflow Model Registry...")
        mlflow.sklearn.log_model(
            sk_model=model,
            artifact_path="model",
            registered_model_name="fraud-detection-model",
            input_example=X_train.iloc[:5]  # Example input for documentation
        )
        
        # Save and log the encoder as a separate artifact
        # We need this for inference
        with open("encoder.pkl", "wb") as f:
            pickle.dump(encoder, f)
        mlflow.log_artifact("encoder.pkl")
        
        # Get the run ID for reference
        run_id = mlflow.active_run().info.run_id
        print(f"\nMLflow Run ID: {run_id}")
        print(f"View this run: http://localhost:5000/#/experiments/1/runs/{run_id}")
        
        return model, encoder

def run_experiment_sweep():
    """
    Run multiple experiments with different hyperparameters.
    
    This demonstrates how MLflow helps compare different configurations.
    """
    print("="*60)
    print("RUNNING HYPERPARAMETER EXPERIMENT SWEEP")
    print("="*60)
    
    # Define different configurations to try
    experiments = [
        {"n_estimators": 50, "max_depth": 5},
        {"n_estimators": 100, "max_depth": 10},
        {"n_estimators": 100, "max_depth": 15},
        {"n_estimators": 200, "max_depth": 10},
        {"n_estimators": 200, "max_depth": 20},
    ]
    
    for i, params in enumerate(experiments, 1):
        print(f"\n--- Experiment {i}/{len(experiments)} ---")
        train_and_log_model(**params)
    
    print("\n" + "="*60)
    print("EXPERIMENT SWEEP COMPLETE!")
    print("="*60)
    print("\nView all experiments at: http://localhost:5000")
    print("Compare runs to find the best hyperparameters!")

if __name__ == "__main__":
    run_experiment_sweep()

This script:

  1. Connects to MLflow: mlflow.set_tracking_uri("http://localhost:5000")

  2. Creates an experiment: mlflow.set_experiment("fraud-detection")

  3. Logs parameters: All hyperparameters and data info

  4. Logs metrics: Accuracy, precision, recall, F1, ROC-AUC for both train and test sets

  5. Logs the model: Saves the trained model as an artifact

  6. Registers the model: Adds it to the Model Registry with automatic versioning

Run the experiment sweep:

python src/train_mlflow.py

You'll see output for each experiment:

============================================================
RUNNING HYPERPARAMETER EXPERIMENT SWEEP
============================================================

--- Experiment 1/5 ---
Loading data...
Training model: n_estimators=50, max_depth=5
  TRAIN - Accuracy: 0.9821, F1: 0.6545, ROC-AUC: 0.9234
  TEST - Accuracy: 0.9795, F1: 0.5714, ROC-AUC: 0.8956

Registering model in MLflow Model Registry...
MLflow Run ID: abc123...

--- Experiment 5/5 ---
Training model: n_estimators=200, max_depth=20
  TRAIN - Accuracy: 0.9856, F1: 0.7123, ROC-AUC: 0.9567
  TEST - Accuracy: 0.9810, F1: 0.6667, ROC-AUC: 0.9234

============================================================
EXPERIMENT SWEEP COMPLETE!
============================================================

All 5 runs are now logged to MLflow with full metrics comparison available in the UI.

Now refresh the MLflow UI at http://localhost:5000. You'll see:

  1. Experiments tab: Shows the "fraud-detection" experiment with 5 runs

  2. Each run: Shows parameters, metrics, and artifacts

  3. Compare: You can select multiple runs and compare them side-by-side

  4. Models tab: Shows "fraud-detection-model" with 5 versions

MLflow Tracking UI: Compare runs, metrics, and models at a glance

c5a7d547-31b6-4783-acea-f4e9433d81ef

3.3 How to Use the Model Registry

The Model Registry provides a central hub for managing model versions and their lifecycle stages.

In the MLflow UI:

  1. Click the "Models" tab in the top navigation

  2. Click "fraud-detection-model"

  3. You'll see all 5 versions listed with their metrics

Model Aliases: MLflow now uses aliases instead of stages. If you've seen older tutorials using "Staging" and "Production" stages, aliases are the newer, more flexible approach.

  • @champion: The production model serving live traffic

  • @challenger: Candidate model being tested

  • You can create custom aliases like @baseline, @latest and so on.

Assign an alias:

  1. Open MLflow UI → Models → fraud-detection-model

  2. Click on the version you want to promote

  3. Click "Add Alias"

  4. Enter champion and save

Now you've assigned the @champion alias to your best model. Your API will load whichever version has this alias, making rollbacks as simple as moving the alias to a different version.

Figure 3: MLflow Model Lifecycle — From Training to Production

Diagram showing the MLflow model lifecycle for a fraud detection system: a model is trained with experiment parameters, logged to MLflow tracking with metrics and artifacts, registered in the model registry as multiple versions, assigned aliases such as champion and challenger, and served in production by loading the model through the champion alias. The diagram also shows rollback by moving the alias to an earlier version and restarting the API.

3.4 Update API to Load from Registry

Now let's update our API to load the champion model from the MLflow Registry instead of a pickle file. Create src/serve_mlflow.py:

# src/serve_mlflow.py
"""
Serve fraud detection model from MLflow Model Registry.

This version loads the @champion model from MLflow, which means:
- Always serves the latest @champion model
- Can roll back by changing the @champion alias
- No manual file copying needed
"""
import mlflow
import mlflow.sklearn
import pickle
import os
from fastapi import FastAPI
from pydantic import BaseModel, Field

# Configure MLflow
mlflow.set_tracking_uri("http://localhost:5000")

print("Loading model from MLflow Model Registry...")

# Load the champion model from the registry
# This automatically gets whichever version has the @champion alias
try:
    model = mlflow.sklearn.load_model("models:/fraud-detection-model@champion")
    print("Successfully loaded champion model from MLflow!")
except Exception as e:
    print(f"Error loading from MLflow: {e}")
    print("Make sure you've assigned the @champion alias to a model in the MLflow UI")
    raise

# Load the encoder (saved as an artifact)
# In a real system, you might also version this in MLflow
with open("encoder.pkl", "rb") as f:
    encoder = pickle.load(f)
print("Encoder loaded successfully!")

app = FastAPI(
    title="Fraud Detection API (MLflow)",
    description="""
    Fraud detection API that loads models from MLflow Model Registry.
    
    This version always serves the model with the @champion alias.
    To update the model:
    1. Train a new model with train_mlflow.py
    2. Compare metrics in MLflow UI
    3. Promote the best model to Production
    4. Restart this API
    
    To roll back: Move the @champion alias to a previous version in MLflow UI.
    """,
    version="2.0.0"
)

class Transaction(BaseModel):
    amount: float = Field(..., description="Transaction amount in dollars", example=150.00)
    hour: int = Field(..., description="Hour of the day (0-23)", example=14)
    day_of_week: int = Field(..., description="Day of week (0=Monday, 6=Sunday)", example=3)
    merchant_category: str = Field(..., description="Type of merchant", example="online")

class PredictionResponse(BaseModel):
    is_fraud: bool
    fraud_probability: float
    model_source: str = "MLflow Production"

@app.post("/predict", response_model=PredictionResponse)
def predict(tx: Transaction):
    """Predict whether a transaction is fraudulent using the champion model."""
    data = tx.dict()
    
    try:
        data["merchant_encoded"] = encoder.transform([data["merchant_category"]])[0]
    except ValueError:
        data["merchant_encoded"] = 0
    
    X = [[data["amount"], data["hour"], data["day_of_week"], data["merchant_encoded"]]]
    
    pred = model.predict(X)[0]
    prob = model.predict_proba(X)[0][1]
    
    return PredictionResponse(
        is_fraud=bool(pred),
        fraud_probability=round(float(prob), 4),
        model_source="MLflow Production"
    )

@app.get("/health")
def health():
    return {"status": "healthy", "model_source": "MLflow Registry"}

@app.get("/model-info")
def model_info():
    """Get information about the currently loaded model."""
    return {
        "registry": "MLflow",
        "model_name": "fraud-detection-model",
        "alias": "champion",
        "tracking_uri": "http://localhost:5000"
    }

Stop your old API (Ctrl+C) and start this new one:

uvicorn src.serve_mlflow:app --reload --host 0.0.0.0 --port 8000

Now deploying a new model is a controlled, auditable process:

  1. Train new model → Automatically registered as new version

  2. Compare metrics → Use MLflow UI to compare with current Production

  3. Set as champion → Assign @champion alias in MLflow UI

  4. Restart API → Loads new Production model

  5. Roll back if needed → Move @champion alias to previous version

Checkpoint:

  • MLflow UI (http://localhost:5000) should show the "fraud-detection" experiment with 5 runs

  • The "Models" tab should show "fraud-detection-model" with 5 versions

  • One version should have @champion alias

  • The API should load and serve @champion model

4. Ensure Feature Consistency with Feast

⚠️ First time hearing about feature stores? Don't worry.
You don't need to master every Feast detail on the first read.
Focus on why feature consistency matters — you can revisit the implementation later.
Key takeaway: Training and serving must compute features the same way, or your model silently fails.

What breaks without this: Your model sees different feature values in production than it saw during training. Accuracy drops silently. This is called "training-serving skew" and it's one of the most common causes of ML system failures.

One subtle but critical issue in ML systems is training-serving skew – when data transformations at training time differ from inference time. Even small discrepancies can severely degrade performance.

Why This Matters: Imagine you're computing "average transaction amount per merchant category" as a feature. During training, you compute it using pandas in a notebook. During serving, you compute it using SQL in a different system. Small differences in how these computations handle edge cases (nulls, rounding, time windows) cause the model to see different features in production than it was trained on.

The result? Silent failures where accuracy drops but nothing errors out. Your model is making predictions based on features it's never seen before, and you have no idea.

In our naive implementation, we did handle one simple case: we saved the LabelEncoder to ensure merchant_category is encoded the same way in training and serving. But imagine if we had more complex feature engineering:

  • Rolling averages over time windows

  • User-level aggregations

  • Cross-feature interactions

  • Real-time features from streaming data

Maintaining consistency manually becomes impossible.

4.1 What is Feast and Why Use It?

In production ML platforms, teams use a feature store to guarantee feature consistency between training and serving. Feast is one popular open-source option.

In this tutorial, we use Feast not because you must, but because it makes the training-serving contract explicit and teachable. The principles apply whether you use Feast, Tecton, Featureform, or a custom solution.

Feast provides:

Capability Description
Single source of truth Define features once, use everywhere
Offline/online consistency Same features for training and serving
Point-in-time correctness Prevents data leakage in training
Low-latency serving Millisecond feature retrieval
Feature versioning Track changes to feature definitions

How Feast works:

  1. Define features in Python code (feature definitions)

  2. Materialize features from your data sources to the online store

  3. Retrieve features using the same API for both training (offline) and serving (online)

This ensures that training and serving use exactly the same feature computation logic.

4.2 Install and Initialize Feast

We already installed Feast via requirements.txt. Now let's initialize a feature repository.

# Navigate to the feature_repo directory
cd feature_repo

# Initialize Feast (this creates template files)
feast init . --minimal

# Go back to project root
cd ..

This creates the basic Feast structure:

feature_repo/
├── feature_store.yaml    # Feast configuration
└── __init__.py

4.3 Define Feature Definitions

First, let's create the Feast configuration file:

# feature_repo/feature_store.yaml
project: fraud_detection
registry: ../data/registry.db
provider: local
online_store:
  type: sqlite
  path: ../data/online_store.db
offline_store:
  type: file
entity_key_serialization_version: 3

This configuration:

  • Names our project "fraud_detection"

  • Uses SQLite for the online store (for production, you'd use Redis or DynamoDB)

  • Uses local files for the offline store (for production, you'd use BigQuery or Snowflake)

Now create the feature definitions:

# feature_repo/features.py
"""
Feast feature definitions for fraud detection.

This file defines:
- Entities: The keys we use to look up features (merchant_category)
- Data Sources: Where the raw feature data comes from (Parquet file)
- Feature Views: The features themselves and their schemas

The key insight: These definitions are the SINGLE SOURCE OF TRUTH.
Both training and serving use these exact definitions.
"""
from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource, ValueType
from feast.types import Float32, Int64

# =============================================================================
# ENTITIES
# =============================================================================
# An entity is the "key" we use to look up features.
# For merchant-level features, the entity is merchant_category.

merchant = Entity(
    name="merchant_category",
    description="Merchant category for the transaction (for example, 'online', 'grocery')",
    value_type=ValueType.STRING,
)

# =============================================================================
# DATA SOURCES
# =============================================================================
# Data sources tell Feast where to find the raw feature data.
# For local development, we use a Parquet file.
# For production, this could be BigQuery, Snowflake, S3, etc.

merchant_stats_source = FileSource(
    name="merchant_stats_source",
    path="../data/merchant_features.parquet",  # We'll create this file
    timestamp_field="event_timestamp",       # Required for point-in-time joins
)

# =============================================================================
# FEATURE VIEWS
# =============================================================================
# A Feature View defines a group of related features.
# It specifies:
# - Which entity the features are for
# - The schema (names and types of features)
# - Where the data comes from
# - How long features are valid (TTL)

merchant_stats_fv = FeatureView(
    name="merchant_stats",
    description="Aggregated statistics per merchant category",
    entities=[merchant],
    ttl=timedelta(days=7),  # Features are valid for 7 days
    schema=[
        Field(name="avg_amount", dtype=Float32, description="Average transaction amount"),
        Field(name="transaction_count", dtype=Int64, description="Number of transactions"),
        Field(name="fraud_rate", dtype=Float32, description="Historical fraud rate"),
    ],
    source=merchant_stats_source,
    online=True,  # Enable online serving (low-latency retrieval)
)

4.4 Materialize Features to Online Store

Now we need to:

  1. Compute the features from our training data

  2. Save them in a format Feast can read

  3. Apply the Feast definitions

  4. Materialize features to the online store

Create src/prepare_feast_features.py:

# src/prepare_feast_features.py
"""
Prepare feature data for Feast.

This script:
1. Computes aggregated merchant features from training data
2. Saves them in Parquet format (Feast's offline store format)
3. Applies Feast feature definitions
4. Materializes features to the online store for low-latency serving

Run this whenever your training data changes or you want to refresh features.
"""
import pandas as pd
import numpy as np
from datetime import datetime
import subprocess
import os

def compute_merchant_features(df: pd.DataFrame) -> pd.DataFrame:
    """
    Compute aggregated features by merchant category.
    
    THIS IS THE SINGLE SOURCE OF TRUTH FOR FEATURE COMPUTATION.
    
    Both training and serving will use features computed by this exact logic.
    Any change here automatically applies everywhere.
    
    Args:
        df: Transaction DataFrame with columns: amount, merchant_category, is_fraud
        
    Returns:
        DataFrame with computed features per merchant category
    """
    print("Computing merchant-level features...")
    
    # Group by merchant category and compute aggregates
    stats = df.groupby('merchant_category').agg({
        'amount': ['mean', 'count'],
        'is_fraud': 'mean'
    }).reset_index()
    
    # Flatten column names
    stats.columns = ['merchant_category', 'avg_amount', 'transaction_count', 'fraud_rate']
    
    # Add timestamp for Feast (required for point-in-time correct joins)
    stats['event_timestamp'] = datetime.now()
    
    # Convert types to match Feast schema
    stats['avg_amount'] = stats['avg_amount'].astype('float32')
    stats['transaction_count'] = stats['transaction_count'].astype('int64')
    stats['fraud_rate'] = stats['fraud_rate'].astype('float32')
    
    return stats

def main():
    print("="*60)
    print("FEAST FEATURE PREPARATION")
    print("="*60)
    
    # Load training data
    print("\n1. Loading training data...")
    train_df = pd.read_csv('data/train.csv')
    print(f"   Loaded {len(train_df):,} transactions")
    
    # Compute merchant features
    print("\n2. Computing merchant features...")
    merchant_features = compute_merchant_features(train_df)
    
    print("\n   Computed features:")
    print(merchant_features.to_string(index=False))
    
    # Save as Parquet (required format for Feast file source)
    print("\n3. Saving features to Parquet...")
    os.makedirs('data', exist_ok=True)
    output_path = 'data/merchant_features.parquet'
    merchant_features.to_parquet(output_path, index=False)
    print(f"   Saved to {output_path}")
    
    # Apply Feast feature definitions
    print("\n4. Applying Feast feature definitions...")
    try:
        result = subprocess.run(
            ['feast', 'apply'],
            cwd='feature_repo',
            capture_output=True,
            text=True,
            check=True
        )
        print("   Feature definitions applied successfully!")
        if result.stdout:
            print(f"   {result.stdout}")
    except subprocess.CalledProcessError as e:
        print(f"   Error applying Feast: {e.stderr}")
        raise
    
    # Materialize features to online store
    print("\n5. Materializing features to online store...")
    try:
        result = subprocess.run(
            ['feast', 'materialize-incremental', datetime.now().isoformat()],
            cwd='feature_repo',
            capture_output=True,
            text=True,
            check=True
        )
        print("   Features materialized successfully!")
        if result.stdout:
            print(f"   {result.stdout}")
    except subprocess.CalledProcessError as e:
        print(f"   Error materializing: {e.stderr}")
        raise
    
    print("\n" + "="*60)
    print("FEAST FEATURE PREPARATION COMPLETE!")
    print("="*60)
    print("\nYou can now:")
    print("  - Retrieve features for training: get_training_features()")
    print("  - Retrieve features for serving: get_online_features()")
    print("  - View feature stats: feast feature-views list")

if __name__ == "__main__":
    main()

Run the feature preparation:

python src/prepare_feast_features.py

You should see:

============================================================
FEAST FEATURE PREPARATION
============================================================

1. Loading training data... 8,000 transactions
2. Computing merchant features...
   grocery: avg=$31.24, fraud_rate=0.85%
   online: avg=$98.45, fraud_rate=4.87%
   restaurant: avg=$28.12, fraud_rate=0.50%
   retail: avg=$45.67, fraud_rate=1.02%
   travel: avg=$156.23, fraud_rate=4.18%
3. Saving to data/merchant_features.parquet ✓
4. Applying Feast definitions... ✓
5. Materializing to online store... ✓

FEAST FEATURE PREPARATION COMPLETE!

4.5 Retrieve Features for Training and Serving

Now let's create utilities to retrieve features consistently for both training and serving:

# src/feast_features.py
"""
Feast feature retrieval for training and serving.

This module provides functions to retrieve features from Feast:
- get_training_features(): For offline training (historical features)
- get_online_features(): For real-time serving (low-latency)

IMPORTANT: Both functions use the SAME feature definitions,
ensuring consistency between training and serving.
"""
import pandas as pd
from feast import FeatureStore
from datetime import datetime

# Initialize Feast store (points to our feature_repo)
store = FeatureStore(repo_path="feature_repo")

def get_training_features(df: pd.DataFrame) -> pd.DataFrame:
    """
    Get features for training using Feast's offline store.
    
    Uses point-in-time correct joins to prevent data leakage.
    This means features are looked up as of the time each transaction occurred,
    not as of "now" - preventing you from accidentally using future data.
    
    Args:
        df: DataFrame with at least 'merchant_category' column
        
    Returns:
        DataFrame with original columns plus Feast features
    """
    print("Retrieving training features from Feast offline store...")
    
    # Prepare entity dataframe with timestamps
    # Each row needs: entity key(s) + event_timestamp
    entity_df = df[['merchant_category']].copy()
    entity_df['event_timestamp'] = datetime.now()  # See note below
    entity_df = entity_df.drop_duplicates()
    
    # ⚠️ Simplification: For clarity, we use the current timestamp here.
    # In real systems, this would be the actual event time of each transaction.
    
    # Retrieve historical features
    # Feast handles the point-in-time join automatically
    training_data = store.get_historical_features(
        entity_df=entity_df,
        features=[
            "merchant_stats:avg_amount",
            "merchant_stats:transaction_count",
            "merchant_stats:fraud_rate",
        ],
    ).to_df()
    
    # Merge features back with original dataframe
    result = df.merge(
        training_data[['merchant_category', 'avg_amount', 'transaction_count', 'fraud_rate']],
        on='merchant_category',
        how='left'
    )
    
    print(f"Retrieved features for {len(entity_df)} unique merchants")
    return result

def get_online_features(merchant_category: str) -> dict:
    """
    Get features for real-time serving using Feast's online store.
    
    This is optimized for low-latency retrieval (milliseconds).
    Use this in your prediction API for real-time inference.
    
    Args:
        merchant_category: The merchant category to look up
        
    Returns:
        Dictionary with feature names and values
    """
    # Retrieve from online store (low-latency)
    feature_vector = store.get_online_features(
        features=[
            "merchant_stats:avg_amount",
            "merchant_stats:transaction_count",
            "merchant_stats:fraud_rate",
        ],
        entity_rows=[{"merchant_category": merchant_category}],
    ).to_dict()
    
    # Format the response
    return {
        'merchant_avg_amount': feature_vector['avg_amount'][0],
        'merchant_tx_count': feature_vector['transaction_count'][0],
        'merchant_fraud_rate': feature_vector['fraud_rate'][0],
    }

def get_online_features_batch(merchant_categories: list) -> pd.DataFrame:
    """
    Get features for multiple merchants at once (batch serving).
    
    More efficient than calling get_online_features() in a loop.
    
    Args:
        merchant_categories: List of merchant categories to look up
        
    Returns:
        DataFrame with features for each merchant
    """
    feature_vector = store.get_online_features(
        features=[
            "merchant_stats:avg_amount",
            "merchant_stats:transaction_count",
            "merchant_stats:fraud_rate",
        ],
        entity_rows=[{"merchant_category": mc} for mc in merchant_categories],
    ).to_df()
    
    return feature_vector

if __name__ == "__main__":
    # Test the feature retrieval functions
    print("="*60)
    print("TESTING FEAST FEATURE RETRIEVAL")
    print("="*60)
    
    # Test offline retrieval (for training)
    print("\n1. Testing OFFLINE feature retrieval (for training)...")
    train_df = pd.read_csv('data/train.csv').head(10)
    enriched = get_training_features(train_df)
    print("\n   Sample enriched training data:")
    print(enriched[['amount', 'merchant_category', 'avg_amount', 'fraud_rate']].head())
    
    # Test online retrieval (for serving)
    print("\n2. Testing ONLINE feature retrieval (for serving)...")
    for category in ['online', 'grocery', 'travel', 'restaurant', 'retail']:
        features = get_online_features(category)
        print(f"   {category}: avg_amount=${features['merchant_avg_amount']:.2f}, "
              f"fraud_rate={features['merchant_fraud_rate']:.2%}")
    
    # Test batch retrieval
    print("\n3. Testing BATCH online retrieval...")
    batch_features = get_online_features_batch(['online', 'grocery', 'travel'])
    print(batch_features)
    
    print("\n" + "="*60)
    print("FEAST FEATURE RETRIEVAL TEST COMPLETE!")
    print("="*60)

Test the feature retrieval:

python src/feast_features.py

You should see:

============================================================
TESTING FEAST FEATURE RETRIEVAL
============================================================

1. Testing OFFLINE feature retrieval (for training)...
Retrieving training features from Feast offline store...
Retrieved features for 5 unique merchants

   Sample enriched training data:
   amount merchant_category  avg_amount  fraud_rate
    45.23           grocery       31.24      0.0085
   123.45            online       98.45      0.0487
    ...

2. Testing ONLINE feature retrieval (for serving)...
   online: avg_amount=$98.45, fraud_rate=4.87%
   grocery: avg_amount=$31.24, fraud_rate=0.85%
   travel: avg_amount=$156.23, fraud_rate=4.18%
   restaurant: avg_amount=$28.12, fraud_rate=0.50%
   retail: avg_amount=$45.67, fraud_rate=1.02%

3. Testing BATCH online retrieval...
  merchant_category  avg_amount  transaction_count  fraud_rate
               online       98.45               1234      0.0487
              grocery       31.24               2345      0.0085
               travel      156.23                478      0.0418

Why Feast Over Custom Code?

Aspect Custom Code Feast
Consistency Manual effort to keep in sync Automatic - same definitions everywhere
Point-in-time correctness Must implement yourself Built-in
Online serving Must build your own cache Built-in online store
Feature versioning Not supported Built-in
Scalability Limited Production-ready (BigQuery, Redis, etc.)
Team collaboration Difficult Feature registry with documentation
Monitoring Manual Built-in feature statistics

💡 Mental Model: Treat feature definitions like database schemas.
You wouldn't compute a column one way in your application and a different way in your reports. Features deserve the same discipline — define once, use everywhere.

Checkpoint: After running prepare_feast_features.py, you should have:

  • data/merchant_features.parquet (computed features)

  • data/registry.db (Feast registry)

  • data/online_store.db (SQLite online store)

Running python src/feast_features.py should successfully retrieve features for all merchant categories.

5. Add Data Validation with Great Expectations

What breaks without this: Your API accepts garbage input (negative amounts, invalid hours) and returns meaningless predictions. Worse, you have no idea it happened.

Recall that our API currently trusts input blindly. We saw how garbage data produces a prediction with no warning. Great Expectations is an open-source tool for data quality testing – defining rules (expectations) and testing data against them.

Why This Matters: Data validation acts as a gatekeeper. Bad data is rejected before it can harm predictions. As the saying goes, "Garbage in, garbage out" – feeding unreliable data yields unreliable results. With validation, we transform this to "Garbage in, error out" – much better for debugging and reliability.

5.1 Define Expectations

What are reasonable expectations for our transaction data? Based on domain knowledge:

Field Expectation Reason
amount Positive (> 0) Negative transactions don't make sense
amount Below $50,000 Extremely large amounts are outliers/errors
hour 0-23 inclusive Valid hours in a day
day_of_week 0-6 inclusive Valid days (Mon=0, Sun=6)
merchant_category One of known categories Must match training data
All fields Not null Required for prediction

Create src/data_validation.py:

# src/data_validation.py
"""
Data validation for fraud detection.

This module provides functions to validate input data BEFORE making predictions.
Invalid data is rejected with clear error messages.

The key insight: It's better to reject bad input than to make garbage predictions.
"""
import pandas as pd
from typing import Dict, List, Any, Optional

# Define the valid merchant categories (must match training data!)
VALID_CATEGORIES = ["grocery", "restaurant", "retail", "online", "travel"]

def validate_transaction(data: Dict[str, Any]) -> Dict[str, Any]:
    """
    Validate a single transaction for fraud prediction.
    
    Checks all business rules and data quality requirements.
    Returns a dictionary with 'valid' (bool) and 'errors' (list).
    
    Args:
        data: Dictionary with transaction fields
        
    Returns:
        {"valid": bool, "errors": list of error messages}
        
    Example:
        >>> validate_transaction({"amount": -100, "hour": 25, ...})
        {"valid": False, "errors": ["amount must be positive", "hour must be 0-23"]}
    """
    errors = []
    
    # ==========================================================================
    # Amount Validation
    # ==========================================================================
    amount = data.get("amount")
    if amount is None:
        errors.append("amount is required")
    elif not isinstance(amount, (int, float)):
        errors.append(f"amount must be a number (got {type(amount).__name__})")
    elif amount <= 0:
        errors.append("amount must be positive")
    elif amount > 50000:
        errors.append(f"amount exceeds maximum allowed value of \(50,000 (got \){amount:,.2f})")
    
    # ==========================================================================
    # Hour Validation
    # ==========================================================================
    hour = data.get("hour")
    if hour is None:
        errors.append("hour is required")
    elif not isinstance(hour, int):
        errors.append(f"hour must be an integer (got {type(hour).__name__})")
    elif not (0 <= hour <= 23):
        errors.append(f"hour must be between 0 and 23 (got {hour})")
    
    # ==========================================================================
    # Day of Week Validation
    # ==========================================================================
    day = data.get("day_of_week")
    if day is None:
        errors.append("day_of_week is required")
    elif not isinstance(day, int):
        errors.append(f"day_of_week must be an integer (got {type(day).__name__})")
    elif not (0 <= day <= 6):
        errors.append(f"day_of_week must be between 0 (Monday) and 6 (Sunday) (got {day})")
    
    # ==========================================================================
    # Merchant Category Validation
    # ==========================================================================
    category = data.get("merchant_category")
    if category is None:
        errors.append("merchant_category is required")
    elif not isinstance(category, str):
        errors.append(f"merchant_category must be a string (got {type(category).__name__})")
    elif category not in VALID_CATEGORIES:
        errors.append(
            f"merchant_category must be one of {VALID_CATEGORIES} (got '{category}')"
        )
    
    return {
        "valid": len(errors) == 0,
        "errors": errors
    }

def validate_batch(df: pd.DataFrame) -> Dict[str, Any]:
    """
    Validate a batch of transactions using Great Expectations.
    
    This is useful for validating training data or batch prediction requests.
    Uses Great Expectations for more sophisticated validation.
    
    Args:
        df: DataFrame with transaction data
        
    Returns:
        Dictionary with validation results
    """
    import great_expectations as gx
    
    # Convert to Great Expectations dataset
    ge_df = gx.from_pandas(df)
    
    results = []
    
    # Amount expectations
    r = ge_df.expect_column_values_to_be_between(
        'amount', min_value=0.01, max_value=50000, mostly=0.99
    )
    results.append(('amount_range', r.success, r.result))
    
    # Hour expectations
    r = ge_df.expect_column_values_to_be_between(
        'hour', min_value=0, max_value=23
    )
    results.append(('hour_range', r.success, r.result))
    
    # Day of week expectations
    r = ge_df.expect_column_values_to_be_between(
        'day_of_week', min_value=0, max_value=6
    )
    results.append(('day_range', r.success, r.result))
    
    # Merchant category expectations
    r = ge_df.expect_column_values_to_be_in_set(
        'merchant_category', VALID_CATEGORIES
    )
    results.append(('category_valid', r.success, r.result))
    
    # No nulls in critical fields
    for col in ['amount', 'hour', 'day_of_week', 'merchant_category']:
        r = ge_df.expect_column_values_to_not_be_null(col)
        results.append((f'{col}_not_null', r.success, r.result))
    
    # Summarize results
    passed = sum(1 for _, success, _ in results if success)
    total = len(results)
    
    return {
        'success': passed == total,
        'passed': passed,
        'total': total,
        'pass_rate': passed / total,
        'details': {name: {'passed': success, 'result': result} 
                   for name, success, result in results}
    }

if __name__ == "__main__":
    print("="*60)
    print("TESTING DATA VALIDATION")
    print("="*60)
    
    # Test single transaction validation
    print("\n1. Single Transaction Validation")
    print("-"*40)
    
    test_cases = [
        {
            "name": "Valid transaction",
            "data": {"amount": 50.0, "hour": 14, "day_of_week": 3, "merchant_category": "grocery"}
        },
        {
            "name": "Negative amount",
            "data": {"amount": -100.0, "hour": 14, "day_of_week": 3, "merchant_category": "grocery"}
        },
        {
            "name": "Invalid hour",
            "data": {"amount": 50.0, "hour": 25, "day_of_week": 3, "merchant_category": "grocery"}
        },
        {
            "name": "Unknown merchant",
            "data": {"amount": 50.0, "hour": 14, "day_of_week": 3, "merchant_category": "unknown"}
        },
        {
            "name": "Everything wrong",
            "data": {"amount": -999, "hour": 99, "day_of_week": 15, "merchant_category": "fake"}
        },
    ]
    
    for tc in test_cases:
        result = validate_transaction(tc["data"])
        status = "PASS" if result["valid"] else "FAIL"
        print(f"\n{tc['name']}: {status}")
        if result["errors"]:
            for error in result["errors"]:
                print(f"  - {error}")
    
    # Test batch validation
    print("\n\n2. Batch Validation with Great Expectations")
    print("-"*40)
    
    train_df = pd.read_csv('data/train.csv')
    results = validate_batch(train_df)
    
    print(f"\nTraining data validation: {results['passed']}/{results['total']} checks passed")
    print(f"Pass rate: {results['pass_rate']:.1%}")
    
    if not results['success']:
        print("\nFailed checks:")
        for name, detail in results['details'].items():
            if not detail['passed']:
                print(f"  - {name}")

When to Use Which Validation Approach

Approach Use Case Latency When to Use
Custom Python (validate_transaction) Real-time API requests <1ms Every prediction request
Great Expectations Batch data quality Seconds Training data, periodic audits, CI/CD

We use both in this tutorial because they serve different purposes:

  • Custom validation is your runtime gatekeeper — fast enough for every request

  • Great Expectations is your batch auditor — thorough checks on datasets

5.2 Integrate Validation into FastAPI

Now let's update our API to reject invalid input with clear error messages:

# src/serve_validated.py
"""
Serve fraud detection model with input validation.

This version adds data validation BEFORE making predictions:
- Invalid inputs are rejected with HTTP 400 and clear error messages
- Valid inputs are processed and predictions returned

This is much safer than the naive version which accepted garbage.
"""
import pickle
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from src.data_validation import validate_transaction

# Load model
with open("models/model.pkl", "rb") as f:
    model, encoder = pickle.load(f)

app = FastAPI(
    title="Fraud Detection API (Validated)",
    description="""
    Fraud detection API with input validation.
    
    All inputs are validated before prediction:
    - amount: Must be positive and below $50,000
    - hour: Must be 0-23
    - day_of_week: Must be 0-6
    - merchant_category: Must be one of: grocery, restaurant, retail, online, travel
    
    Invalid inputs return HTTP 400 with detailed error messages.
    """,
    version="3.0.0"
)

class Transaction(BaseModel):
    amount: float = Field(..., description="Transaction amount (must be positive)", example=150.00)
    hour: int = Field(..., description="Hour of day (0-23)", example=14)
    day_of_week: int = Field(..., description="Day of week (0=Mon, 6=Sun)", example=3)
    merchant_category: str = Field(..., description="Merchant type", example="online")

class PredictionResponse(BaseModel):
    is_fraud: bool
    fraud_probability: float
    validation_passed: bool = True

class ValidationErrorResponse(BaseModel):
    detail: dict

@app.post("/predict", response_model=PredictionResponse, responses={400: {"model": ValidationErrorResponse}})
def predict(tx: Transaction):
    """
    Predict whether a transaction is fraudulent.
    
    Input is validated before prediction. Invalid inputs return HTTP 400.
    """
    data = tx.dict()
    
    # VALIDATE INPUT BEFORE MAKING PREDICTION
    validation = validate_transaction(data)
    
    if not validation["valid"]:
        raise HTTPException(
            status_code=400,
            detail={
                "message": "Validation failed",
                "errors": validation["errors"],
                "input": data
            }
        )
    
    # Input is valid - make prediction
    data["merchant_encoded"] = encoder.transform([data["merchant_category"]])[0]
    X = [[data["amount"], data["hour"], data["day_of_week"], data["merchant_encoded"]]]
    
    pred = model.predict(X)[0]
    prob = model.predict_proba(X)[0][1]
    
    return PredictionResponse(
        is_fraud=bool(pred),
        fraud_probability=round(float(prob), 4),
        validation_passed=True
    )

@app.get("/health")
def health():
    return {"status": "healthy", "validation": "enabled"}

Start the validated API:

uvicorn src.serve_validated:app --reload --host 0.0.0.0 --port 8000

Now test with bad data:

curl -X POST "http://localhost:8000/predict" \
  -H "Content-Type: application/json" \
  -d '{"amount": -500, "hour": 25, "day_of_week": 10, "merchant_category": "fake"}'

Response (HTTP 400):

{
  "detail": {
    "message": "Validation failed",
    "errors": [
      "amount must be positive",
      "hour must be between 0 and 23 (got 25)",
      "day_of_week must be between 0 (Monday) and 6 (Sunday) (got 10)",
      "merchant_category must be one of ['grocery', 'restaurant', 'retail', 'online', 'travel'] (got 'fake')"
    ],
    "input": {"amount": -500, "hour": 25, "day_of_week": 10, "merchant_category": "fake"}
  }
}

This is a huge improvement! Instead of silently accepting garbage and returning meaningless predictions, we now:

  • Reject invalid input immediately

  • Provide clear, actionable error messages

  • Return the original input for debugging

  • Use proper HTTP status codes (400 for client error)

Checkpoint: Your validated API should:

  • Accept valid transactions and return predictions

  • Reject invalid transactions with HTTP 400 and detailed error messages

  • Show validation errors for each invalid field

6. Monitor Model Performance and Data Drift

What breaks without this: Your model's accuracy drops from 98% to 70% over two months. Nobody notices until customers complain. By then, significant damage has occurred.

Even with a great model and clean input data, time can be an enemy. Model performance can decline as real-world data evolves – this is known as model drift or model decay.

Why This Matters: In traditional software, you monitor CPU, memory, error rates, and response times. In ML, you must also monitor:

  • Data quality (are inputs within expected ranges?)

  • Model performance (is accuracy holding up?)

  • Data drift (has input distribution changed?)

  • Prediction drift (has the distribution of predictions changed?)

Without monitoring, your model could be silently failing for weeks before anyone notices. By then, significant damage may have occurred – fraud slipping through, good customers blocked, revenue lost.

6.1 The Four Pillars of ML Observability

Pillar What to Monitor Why It Matters
Data Quality Are inputs valid? Nulls? Outliers? Bad data causes bad predictions
Model Performance Accuracy, precision, recall, F1 Is the model still working?
Data Drift Has input distribution changed from training? Model may not generalize to new data
Prediction Drift Has prediction distribution changed? May indicate data or concept drift

6.2 Build a Drift Monitor with Evidently

Evidently is an open-source library specifically designed for ML monitoring. It can detect drift, generate reports, and integrate with monitoring systems.

Create src/monitoring.py:

# src/monitoring.py
"""
Model monitoring with Evidently.

This module provides tools to:
1. Detect data drift between training and production data
2. Generate detailed HTML reports
3. Track drift over time
4. Alert when drift exceeds thresholds

In production, you would run drift checks periodically (hourly, daily)
and alert when significant drift is detected.
"""
import pandas as pd
import numpy as np
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
from evidently.metrics import (
    DatasetDriftMetric,
    DataDriftTable,
    ColumnDriftMetric
)
from datetime import datetime
from typing import List, Dict, Any, Optional

class DriftMonitor:
    """
    Monitor for detecting data drift between reference (training) and current data.
    
    Implementation Note: We use two approaches here:
    1. Scipy's KS-test — A lightweight statistical method that works anywhere (our fallback)
    2. Evidently — A full-featured library with beautiful reports (our primary tool)
    
    The KS-test is included as defensive coding — if Evidently fails to generate 
    a report, we still get drift detection.
    
    Usage:
        monitor = DriftMonitor(training_data)
        result = monitor.check_drift(production_data)
        if result['drift_detected']:
            alert("Drift detected!")
    """
    
    def __init__(self, reference_data: pd.DataFrame, feature_columns: Optional[List[str]] = None):
        """
        Initialize the drift monitor with reference (training) data.
        
        Args:
            reference_data: The training data to compare against
            feature_columns: Columns to monitor (default: all numeric columns)
        """
        self.reference = reference_data
        self.feature_columns = feature_columns or reference_data.select_dtypes(
            include=[np.number]
        ).columns.tolist()
        self.history: List[Dict[str, Any]] = []
        
        print(f"Drift monitor initialized with {len(self.reference):,} reference samples")
        print(f"Monitoring columns: {self.feature_columns}")
    
    def check_drift(self, current_data: pd.DataFrame, threshold: float = 0.1) -> Dict[str, Any]:
        """
        Check for drift between reference and current data.
        
        Args:
            current_data: Current/production data to check
            threshold: Drift share threshold for alerting (default 10%)
            
        Returns:
            Dictionary with drift results
        """
        from scipy import stats
        
        ref_subset = self.reference[self.feature_columns]
        cur_subset = current_data[self.feature_columns]
        
        # Simple statistical drift detection using KS test
        drifted_columns = []
        for col in self.feature_columns:
            statistic, p_value = stats.ks_2samp(
                ref_subset[col].dropna(),
                cur_subset[col].dropna()
            )
            if p_value < 0.05:  # 5% significance level
                drifted_columns.append(col)
        
        n_features = len(self.feature_columns)
        n_drifted = len(drifted_columns)
        drift_share = n_drifted / n_features if n_features > 0 else 0
        
        result = {
            'timestamp': datetime.now().isoformat(),
            'drift_detected': n_drifted > 0,
            'drift_share': drift_share,
            'drifted_columns': drifted_columns,
            'n_features': n_features,
            'n_drifted': n_drifted,
            'current_samples': len(current_data),
            'threshold': threshold,
            'alert': drift_share > threshold
        }
        
        self.history.append(result)
        
        return result
    
    def generate_report(self, current_data: pd.DataFrame, output_path: str = "drift_report.html"):
        """
        Generate a detailed HTML drift report using Evidently.
        
        Opens in browser for visual inspection of drift patterns.
        """
        ref_subset = self.reference[self.feature_columns]
        cur_subset = current_data[self.feature_columns]
        
        try:
            report = Report(metrics=[DataDriftPreset()])
            report.run(reference_data=ref_subset, current_data=cur_subset)
            
            # Save HTML report
            with open(output_path, 'w') as f:
                f.write(report.show(mode='inline').data)
            
            print(f"Drift report saved to {output_path}")
            print(f"Open this file in a browser to view detailed visualizations.")
        except Exception as e:
            print(f"Could not generate Evidently report: {e}")
            print(f"Using simplified drift detection instead.")
    
    def get_alerts(self, threshold: float = 0.1) -> List[Dict[str, Any]]:
        """
        Get all alerts from history where drift exceeded threshold.
        """
        return [
            {
                'timestamp': r['timestamp'],
                'severity': 'HIGH' if r['drift_share'] > 0.3 else 'MEDIUM',
                'drift_share': r['drift_share'],
                'message': f"Drift detected: {r['drift_share']:.1%} of features drifted",
                'drifted_columns': r['drifted_columns']
            }
            for r in self.history
            if r['drift_share'] > threshold
        ]
    
    def summary(self) -> Dict[str, Any]:
        """Get summary statistics from monitoring history."""
        if not self.history:
            return {"message": "No drift checks performed yet"}
        
        drift_shares = [r['drift_share'] for r in self.history]
        alerts = [r for r in self.history if r['alert']]
        
        return {
            'total_checks': len(self.history),
            'total_alerts': len(alerts),
            'avg_drift_share': np.mean(drift_shares),
            'max_drift_share': np.max(drift_shares),
            'first_check': self.history[0]['timestamp'],
            'last_check': self.history[-1]['timestamp']
        }


def simulate_drift_scenarios():
    """
    Demonstrate drift detection with different scenarios.
    
    This simulates what happens when production data differs from training data.
    """
    from src.generate_data import generate_transactions
    
    print("="*70)
    print("DRIFT DETECTION SIMULATION")
    print("="*70)
    
    # Load reference (training) data
    print("\n1. Loading reference data (training set)...")
    reference = pd.read_csv('data/train.csv')
    feature_cols = ['amount', 'hour', 'day_of_week']
    
    # Initialize drift monitor
    monitor = DriftMonitor(reference, feature_cols)
    
    # Scenario 1: Similar data (should show minimal drift)
    print("\n" + "-"*70)
    print("SCENARIO 1: Test data (similar distribution)")
    print("-"*70)
    test_data = pd.read_csv('data/test.csv')
    result = monitor.check_drift(test_data)
    print(f"  Drift detected: {result['drift_detected']}")
    print(f"  Drift share: {result['drift_share']:.1%}")
    print(f"  Drifted columns: {result['drifted_columns']}")
    print(f"  Alert triggered: {result['alert']}")
    
    # Scenario 2: Fraud spike (10% fraud instead of 2%)
    print("\n" + "-"*70)
    print("SCENARIO 2: Fraud spike (10% fraud rate instead of 2%)")
    print("-"*70)
    fraud_spike = generate_transactions(n_samples=2000, fraud_ratio=0.10, seed=101)
    result = monitor.check_drift(fraud_spike)
    print(f"  Drift detected: {result['drift_detected']}")
    print(f"  Drift share: {result['drift_share']:.1%}")
    print(f"  Drifted columns: {result['drifted_columns']}")
    print(f"  Alert triggered: {result['alert']}")
    
    # Scenario 3: Amount inflation (everything costs more)
    print("\n" + "-"*70)
    print("SCENARIO 3: Amount inflation (2x multiplier)")
    print("-"*70)
    inflated = test_data.copy()
    inflated['amount'] = inflated['amount'] * 2
    result = monitor.check_drift(inflated)
    print(f"  Drift detected: {result['drift_detected']}")
    print(f"  Drift share: {result['drift_share']:.1%}")
    print(f"  Drifted columns: {result['drifted_columns']}")
    print(f"  Alert triggered: {result['alert']}")
    
    # Scenario 4: Time shift (more late-night transactions)
    print("\n" + "-"*70)
    print("SCENARIO 4: Time shift (mostly late-night transactions)")
    print("-"*70)
    night_shift = test_data.copy()
    night_shift['hour'] = np.random.choice([0, 1, 2, 3, 22, 23], size=len(night_shift))
    result = monitor.check_drift(night_shift)
    print(f"  Drift detected: {result['drift_detected']}")
    print(f"  Drift share: {result['drift_share']:.1%}")
    print(f"  Drifted columns: {result['drifted_columns']}")
    print(f"  Alert triggered: {result['alert']}")
    
    # Generate detailed report for the most drifted scenario
    print("\n" + "-"*70)
    print("GENERATING DETAILED REPORT")
    print("-"*70)
    monitor.generate_report(night_shift, "drift_report.html")
    
    # Print summary
    print("\n" + "-"*70)
    print("MONITORING SUMMARY")
    print("-"*70)
    summary = monitor.summary()
    print(f"  Total checks: {summary['total_checks']}")
    print(f"  Total alerts: {summary['total_alerts']}")
    print(f"  Average drift share: {summary['avg_drift_share']:.1%}")
    print(f"  Maximum drift share: {summary['max_drift_share']:.1%}")
    
    # Print alerts
    alerts = monitor.get_alerts()
    if alerts:
        print(f"\n  Alerts ({len(alerts)}):")
        for alert in alerts:
            print(f"    [{alert['severity']}] {alert['message']}")
    
    print("\n" + "="*70)
    print("DRIFT DETECTION SIMULATION COMPLETE")
    print("="*70)
    print("\nOpen drift_report.html in your browser to see detailed visualizations!")


if __name__ == "__main__":
    simulate_drift_scenarios()

Run the drift simulation:

python src/monitoring.py

You'll see output showing how drift detection works in different scenarios. Then open drift_report.html in your browser to see beautiful visualizations of the drift patterns.

6.3 Production Monitoring Strategy

In a production environment, you would:

  1. Log all predictions to a database or data warehouse

  2. Run drift checks periodically (hourly for high-traffic systems, daily for lower traffic)

  3. Set up alerts when drift exceeds thresholds (integrate with PagerDuty, Slack, etc.)

  4. Trigger retraining if drift is severe or sustained

  5. Create dashboards to track drift over time (Grafana, Datadog, etc.)

Checkpoint: Running python src/monitoring.py should:

  • Show minimal drift for similar data (test set)

  • Show significant drift for modified data (fraud spike, inflation, time shift)

  • Generate an HTML report that you can view in your browser

7. Automate Testing and Deployment with CI/CD

What breaks without this: A typo in your code breaks the API. You deploy on Friday at 5 PM. Nobody notices until Monday. Fraud losses spike over the weekend.

CI/CD (Continuous Integration/Continuous Deployment) ensures reliable, repeatable releases. As JFrog notes: "A strong CI/CD pipeline enables ML teams to build robust, bug-free models more quickly and efficiently."

Why This Matters: In ML, changes aren't just code – they're also data and models. CI/CD ensures that when you change training logic, data preprocessing, or hyperparameters, tests verify the change doesn't break anything before it reaches production. It's the difference between deploying with confidence and deploying with crossed fingers.

7.1 Write Tests for Data and Model

Create tests/test_data_and_model.py:

# tests/test_data_and_model.py
"""
Tests for data quality and model performance.

These tests run in CI/CD to ensure:
1. Data meets quality requirements
2. Model meets performance thresholds
3. No regressions are introduced

Run with: pytest tests/test_data_and_model.py -v
"""
import pandas as pd
import pickle
import pytest
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score

class TestDataQuality:
    """Tests for training data quality."""
    
    @pytest.fixture
    def train_data(self):
        return pd.read_csv("data/train.csv")
    
    @pytest.fixture
    def test_data(self):
        return pd.read_csv("data/test.csv")
    
    def test_train_data_has_expected_columns(self, train_data):
        """Training data must have all required columns."""
        required_columns = {"amount", "hour", "day_of_week", "merchant_category", "is_fraud"}
        actual_columns = set(train_data.columns)
        missing = required_columns - actual_columns
        assert not missing, f"Missing columns: {missing}"
    
    def test_train_data_not_empty(self, train_data):
        """Training data must have rows."""
        assert len(train_data) > 0, "Training data is empty"
        assert len(train_data) >= 1000, f"Training data too small: {len(train_data)} rows"
    
    def test_no_negative_amounts(self, train_data):
        """Transaction amounts must be non-negative."""
        negative_count = (train_data["amount"] < 0).sum()
        assert negative_count == 0, f"Found {negative_count} negative amounts"
    
    def test_amounts_reasonable(self, train_data):
        """Transaction amounts should be within reasonable bounds."""
        max_amount = train_data["amount"].max()
        assert max_amount <= 100000, f"Max amount {max_amount} exceeds reasonable limit"
    
    def test_hours_valid(self, train_data):
        """Hours must be 0-23."""
        invalid = train_data[(train_data["hour"] < 0) | (train_data["hour"] > 23)]
        assert len(invalid) == 0, f"Found {len(invalid)} invalid hours"
    
    def test_days_valid(self, train_data):
        """Days of week must be 0-6."""
        invalid = train_data[(train_data["day_of_week"] < 0) | (train_data["day_of_week"] > 6)]
        assert len(invalid) == 0, f"Found {len(invalid)} invalid days"
    
    def test_merchant_categories_valid(self, train_data):
        """Merchant categories must be from known set."""
        valid_categories = {"grocery", "restaurant", "retail", "online", "travel"}
        actual_categories = set(train_data["merchant_category"].unique())
        invalid = actual_categories - valid_categories
        assert not invalid, f"Invalid merchant categories: {invalid}"
    
    def test_fraud_ratio_reasonable(self, train_data):
        """Fraud ratio should be realistic (between 0.1% and 50%)."""
        fraud_ratio = train_data["is_fraud"].mean()
        assert 0.001 <= fraud_ratio <= 0.5, f"Fraud ratio {fraud_ratio:.2%} is unrealistic"
    
    def test_no_nulls_in_critical_columns(self, train_data):
        """Critical columns must not have null values."""
        critical = ["amount", "hour", "day_of_week", "merchant_category", "is_fraud"]
        for col in critical:
            null_count = train_data[col].isnull().sum()
            assert null_count == 0, f"Column {col} has {null_count} null values"


class TestModelPerformance:
    """Tests for model performance thresholds."""
    
    @pytest.fixture
    def model_and_encoder(self):
        with open("models/model.pkl", "rb") as f:
            return pickle.load(f)
    
    @pytest.fixture
    def test_data(self):
        return pd.read_csv("data/test.csv")
    
    def test_model_loads_successfully(self, model_and_encoder):
        """Model file must load without errors."""
        model, encoder = model_and_encoder
        assert model is not None, "Model is None"
        assert encoder is not None, "Encoder is None"
    
    def test_model_can_predict(self, model_and_encoder, test_data):
        """Model must be able to make predictions."""
        model, encoder = model_and_encoder
        test_data["merchant_encoded"] = encoder.transform(test_data["merchant_category"])
        X = test_data[["amount", "hour", "day_of_week", "merchant_encoded"]]
        predictions = model.predict(X)
        assert len(predictions) == len(X), "Prediction count mismatch"
    
    def test_accuracy_threshold(self, model_and_encoder, test_data):
        """Model accuracy must be at least 90%."""
        model, encoder = model_and_encoder
        test_data["merchant_encoded"] = encoder.transform(test_data["merchant_category"])
        X = test_data[["amount", "hour", "day_of_week", "merchant_encoded"]]
        y = test_data["is_fraud"]
        accuracy = model.score(X, y)
        assert accuracy >= 0.90, f"Accuracy {accuracy:.2%} below 90% threshold"
    
    def test_f1_threshold(self, model_and_encoder, test_data):
        """Model F1-score must be at least 0.3 (sanity check for imbalanced data)."""
        model, encoder = model_and_encoder
        test_data["merchant_encoded"] = encoder.transform(test_data["merchant_category"])
        X = test_data[["amount", "hour", "day_of_week", "merchant_encoded"]]
        y = test_data["is_fraud"]
        y_pred = model.predict(X)
        f1 = f1_score(y, y_pred)
        assert f1 >= 0.3, f"F1-score {f1:.2f} below 0.3 threshold"
    
    def test_precision_not_zero(self, model_and_encoder, test_data):
        """Model precision must be greater than 0 (catches at least some fraud)."""
        model, encoder = model_and_encoder
        test_data["merchant_encoded"] = encoder.transform(test_data["merchant_category"])
        X = test_data[["amount", "hour", "day_of_week", "merchant_encoded"]]
        y = test_data["is_fraud"]
        y_pred = model.predict(X)
        precision = precision_score(y, y_pred, zero_division=0)
        assert precision > 0, "Model has zero precision (predicts no fraud)"
    
    def test_recall_not_zero(self, model_and_encoder, test_data):
        """Model recall must be greater than 0 (catches at least some fraud)."""
        model, encoder = model_and_encoder
        test_data["merchant_encoded"] = encoder.transform(test_data["merchant_category"])
        X = test_data[["amount", "hour", "day_of_week", "merchant_encoded"]]
        y = test_data["is_fraud"]
        y_pred = model.predict(X)
        recall = recall_score(y, y_pred, zero_division=0)
        assert recall > 0, "Model has zero recall (misses all fraud)"

Create tests/test_api.py:

# tests/test_api.py
"""
Tests for the FastAPI prediction service.

These tests ensure the API:
1. Returns correct responses for valid inputs
2. Rejects invalid inputs with proper error messages
3. Health check works

Run with: pytest tests/test_api.py -v
Note: Requires the API to be running on localhost:8000
"""
import pytest
import httpx

BASE_URL = "http://localhost:8000"

class TestPredictionEndpoint:
    """Tests for the /predict endpoint."""
    
    def test_valid_prediction_returns_200(self):
        """Valid input should return HTTP 200 with prediction."""
        response = httpx.post(f"{BASE_URL}/predict", json={
            "amount": 100.0,
            "hour": 14,
            "day_of_week": 3,
            "merchant_category": "online"
        }, timeout=10)
        
        assert response.status_code == 200
        data = response.json()
        assert "is_fraud" in data
        assert "fraud_probability" in data
        assert isinstance(data["is_fraud"], bool)
        assert 0 <= data["fraud_probability"] <= 1
    
    def test_high_risk_transaction(self):
        """High-risk transaction should have higher fraud probability."""
        response = httpx.post(f"{BASE_URL}/predict", json={
            "amount": 500.0,
            "hour": 3,  # Late night
            "day_of_week": 1,
            "merchant_category": "online"
        }, timeout=10)
        
        assert response.status_code == 200
        data = response.json()
        # High-risk transactions should have elevated probability
        # (not asserting exact value as model may vary)
        assert data["fraud_probability"] >= 0.0
    
    def test_negative_amount_rejected(self):
        """Negative amount should be rejected with 400."""
        response = httpx.post(f"{BASE_URL}/predict", json={
            "amount": -100.0,
            "hour": 14,
            "day_of_week": 3,
            "merchant_category": "online"
        }, timeout=10)
        
        assert response.status_code == 400
        assert "errors" in response.json()["detail"]
    
    def test_invalid_hour_rejected(self):
        """Invalid hour should be rejected with 400."""
        response = httpx.post(f"{BASE_URL}/predict", json={
            "amount": 100.0,
            "hour": 25,  # Invalid
            "day_of_week": 3,
            "merchant_category": "online"
        }, timeout=10)
        
        assert response.status_code == 400
    
    def test_invalid_merchant_rejected(self):
        """Unknown merchant category should be rejected with 400."""
        response = httpx.post(f"{BASE_URL}/predict", json={
            "amount": 100.0,
            "hour": 14,
            "day_of_week": 3,
            "merchant_category": "unknown_category"
        }, timeout=10)
        
        assert response.status_code == 400
    
    def test_missing_field_rejected(self):
        """Missing required field should be rejected."""
        response = httpx.post(f"{BASE_URL}/predict", json={
            "amount": 100.0,
            "hour": 14
            # Missing day_of_week and merchant_category
        }, timeout=10)
        
        assert response.status_code == 422  # Pydantic validation error


class TestHealthEndpoint:
    """Tests for the /health endpoint."""
    
    def test_health_returns_200(self):
        """Health endpoint should return 200."""
        response = httpx.get(f"{BASE_URL}/health", timeout=10)
        assert response.status_code == 200
    
    def test_health_returns_healthy_status(self):
        """Health endpoint should indicate healthy status."""
        response = httpx.get(f"{BASE_URL}/health", timeout=10)
        data = response.json()
        assert data["status"] == "healthy"

Run tests locally:

# Run data and model tests (API not needed)
pytest tests/test_data_and_model.py -v

# Run API tests (requires API to be running)
pytest tests/test_api.py -v

7.2 GitHub Actions Workflow

⚠️ Note for Production Teams
In real ML teams, you typically don't retrain full models inside CI — it's slow and resource-intensive.
Here we do it to keep everything local, reproducible, and self-contained for learning.
Production pipelines usually separate training (scheduled jobs) from testing (CI/CD).

Create .github/workflows/ci.yml:

# .github/workflows/ci.yml
name: ML Pipeline CI/CD

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: 'pip'
      
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      
      - name: Generate training data
        run: python src/generate_data.py
      
      - name: Train model
        run: python src/train_naive.py
      
      - name: Run data quality tests
        run: pytest tests/test_data_and_model.py -v --tb=short
      
      - name: Build Docker image
        run: docker build -t fraud-detection-api .
      
      - name: Run container for API tests
        run: |
          docker run -d -p 8000:8000 --name test-api fraud-detection-api
          sleep 10  # Wait for API to start
          curl -f http://localhost:8000/health || exit 1
      
      - name: Run API tests
        run: pytest tests/test_api.py -v --tb=short
      
      - name: Cleanup
        if: always()
        run: docker stop test-api || true

7.3 Dockerize the Application

Create Dockerfile:

# Dockerfile
FROM python:3.11-slim

# Set working directory
WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY src/ src/
COPY models/ models/
COPY data/ data/

# Expose port
EXPOSE 8000

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

# Run the API
CMD ["uvicorn", "src.serve_validated:app", "--host", "0.0.0.0", "--port", "8000"]

Create .dockerignore:

# .dockerignore
venv/
__pycache__/
*.pyc
.git/
.github/
mlruns/
*.db
*.html
.pytest_cache/

Build and run locally:

# Build the Docker image
docker build -t fraud-detection-api .

# Run the container
docker run -p 8000:8000 fraud-detection-api

# Test it
curl http://localhost:8000/health

Checkpoint:

  • All tests pass: pytest tests/test_data_and_model.py -v

  • Docker image builds successfully

  • Container runs and responds to health checks

8. Incident Response Playbook

When things go wrong in production (and they will), you need a plan. This section provides playbooks for common ML incidents.

Scenario: False Positive Spike

Symptoms: Your fraud model suddenly flags 40% of legitimate transactions as fraud, blocking customers and overwhelming your manual review team.

Severity: HIGH - Direct customer impact

Phase 1: Mitigation (0-5 minutes)

  1. Acknowledge the incident - Notify stakeholders that you're aware and responding

  2. Roll back to previous model - In MLflow UI, move the @champion alias to the previous model version

  3. Restart the API - docker restart fraud-api or redeploy

  4. Verify - Check that false positive rate has returned to normal

  5. Communicate - "Issue detected and mitigated. Investigating root cause."

Phase 2: Diagnosis (5-60 minutes)

  1. Check drift report - Run python src/monitoring.py with recent production data

  2. Check data validation logs - Did upstream data format change?

  3. Check recent deployments - Was there a new model or code deployed recently?

  4. Compare metrics - What's different between the rolled-back and problematic model?

Example root causes:

  • Upstream system sent amounts in cents instead of dollars

  • New merchant category appeared that wasn't in training data

  • Holiday shopping patterns differed significantly from training data

Phase 3: Remediation (1-24 hours)

  1. Fix the root cause - Add validation for the edge case, or update training data

  2. Retrain if needed - Include new patterns in training data

  3. Add test case - Prevent this from happening again

  4. Document - Add to runbook for future reference

Scenario: Gradual Performance Decay

Symptoms: Monitoring shows fraud recall dropping 2% per week over a month. No sudden failures, just slow degradation.

Severity: MEDIUM - Gradual impact, time to respond

Response:

  1. Investigate drift report - Look for gradual distribution changes

    python src/monitoring.py

  2. Collect recent labeled data - Get confirmed fraud cases from the past month

  3. Analyze patterns - What's different about recent fraud?

    • New attack vectors?

    • Different time patterns?

    • New merchant categories?

  4. Retrain on combined data - Include both old and new patterns

    python src/train_mlflow.py

  5. Deploy via canary - Route 10% of traffic to the new model first

    • Monitor metrics for 1-2 days

    • If metrics improve, increase to 50%, then 100%

    • If metrics worsen, roll back

  6. Set up recurring retraining - Schedule weekly or monthly retraining

Scenario: Upstream Data Schema Change

Symptoms: API starts returning 500 errors. Logs show KeyError: 'merchant_category'.

Severity: HIGH - Service is down

Response:

  1. Check error logs - Identify the exact error

    KeyError: 'merchant_category'

  2. Check upstream data - Did the field name change?

    • merchant_category -> category

    • amount -> transaction_amount

  3. Immediate fix - Add field name mapping

    # Quick fix in API
if 'category' in data and 'merchant_category' not in data:
    data['merchant_category'] = data['category']

  4. Long-term fix - Add validation that catches schema changes

    required_fields = ['amount', 'hour', 'day_of_week', 'merchant_category']
missing = [f for f in required_fields if f not in data]
if missing:
    raise ValidationError(f"Missing fields: {missing}")

  5. Add integration test - Test with upstream system in CI/CD

9. How to Put It All Together

Let's step back and appreciate what we've built. Our initial naive system has transformed into a local ML platform with production-grade components.

💡 Mental Model: Each tool in this stack is a "catch net" for a specific failure mode:

  • MLflow catches "which model is this?"

  • Feast catches "are features consistent?"

  • Great Expectations catches "is this data valid?"

  • Evidently catches "has the world changed?"

  • CI/CD catches "did we break something?"

Together, they form defense-in-depth for ML systems.

Component Tool Problem Solved
Experiment Tracking MLflow Every run logged, reproducible
Model Registry MLflow Versioned models, rollback capability
Feature Store Feast Consistent features, no training-serving skew
Data Validation Great Expectations Bad data rejected with clear errors
Monitoring Evidently Drift detected before it causes problems
Containerization Docker Environment consistency everywhere
CI/CD GitHub Actions Automated testing and safe deployments

The Complete Workflow

Here's how all the pieces work together in practice:

  1. Data arrives - New transaction data comes in from upstream systems

  2. Validation gate - Great Expectations rules check data quality. Bad data is rejected with clear error messages before it can cause harm.

  3. Feature computation - Feast computes features using the same definitions for both training and serving. No more training-serving skew.

  4. Training - When you retrain, MLflow logs all parameters, metrics, and artifacts. Every experiment is reproducible and comparable.

  5. Model registry - Trained models are automatically versioned. You can compare metrics, promote the best to Production, and roll back if needed.

  6. Serving - FastAPI loads the @champion model from MLflow. Each request is validated, features are retrieved from Feast, and predictions are returned.

  7. Monitoring - Evidently checks for drift periodically. If input distributions change significantly, alerts are triggered.

  8. Retraining loop - When drift is detected, you retrain on new data, compare metrics, and promote if better. The cycle continues.

  9. CI/CD safety net - All code changes go through automated tests. Docker ensures environment consistency. Nothing reaches production without passing the pipeline.

10. What's Next: Scale to Production

This project runs locally, but the principles and tools extend directly to production deployments. Here's how each component scales:

Scaling Feast for Production

We used Feast with local SQLite stores. For production:

Component Local Production
Online Store SQLite Redis, DynamoDB, or PostgreSQL
Offline Store Parquet files BigQuery, Snowflake, or Redshift
Feature Server Embedded Dedicated Feast serving cluster

Benefits at scale:

  • Sub-10ms feature retrieval

  • Horizontal scaling for high throughput

  • Feature monitoring and statistics

  • Point-in-time joins at petabyte scale

Scaling MLflow for Production

Component Local Production
Backend Store SQLite PostgreSQL or MySQL
Artifact Store Local filesystem S3, GCS, or Azure Blob
Tracking Server Single instance Load-balanced cluster

Kubernetes Deployment

When you outgrow Docker Compose:

  • KServe or Seldon for serverless model serving with auto-scaling

  • Horizontal Pod Autoscaler to scale based on CPU/memory/custom metrics

  • Canary deployments to safely roll out new models (route 10% traffic first)

  • GPU scheduling for inference-heavy models

Advanced Monitoring

Expand observability with:

  • Prometheus + Grafana for real-time dashboards

  • OpenTelemetry for distributed tracing

  • PagerDuty/Slack integration for alerts

  • Labeled data collection for continuous model evaluation

A/B Testing and Multi-Armed Bandits

How to Use the Model Registry:

  • Serve multiple models concurrently (champion vs challengers)

  • Route traffic dynamically based on context

  • Collect metrics for each model variant

  • Automatically promote the best performer

Conclusion

Congratulations on building a production-ready ML system on your local machine!

What we assembled here is a microcosm of real-world ML platforms:

  • We started with just a model saved to a pickle file

  • We ended up with MLOps best practices: experiment tracking, model versioning, feature stores, data validation, monitoring, containerization, and CI/CD

The tools we used are production-grade:

  • MLflow powers ML platforms at companies like Microsoft, Facebook, and Databricks

  • Feast is used by companies like Gojek, Shopify, and Robinhood

  • FastAPI is one of the fastest Python web frameworks

  • Great Expectations is used at companies like GitHub and Shopify

  • Evidently is used for monitoring ML in production at scale

The principles apply at any scale:

  • Always track experiments

  • Always version models

  • Always validate data

  • Always monitor for drift

  • Always containerize for consistency

  • Always automate testing

Next Steps You Can Try

  1. Deploy to the cloud - Push your Docker container to AWS ECS, Google Cloud Run, or Azure Container Instances

  2. Add model explainability - Use SHAP or LIME to explain individual predictions

  3. Implement A/B testing - Serve multiple models and compare performance

  4. Add feature importance monitoring - Track how feature importance changes over time

  5. Set up real-time alerting - Connect Evidently to Slack or PagerDuty

  6. Implement continuous training - Automatically retrain when drift is detected

  7. Add bias and fairness monitoring - Ensure your model treats all groups fairly

Remember that productionizing ML is an iterative process. There's always another layer of robustness to add, another edge case to handle, another metric to track. But with the foundation you've built here, you're well on your way to taking models from promising notebook experiments to deployed, monitored, and maintainable production applications.

Happy building, and may your models be accurate and your pipelines resilient!

Get the Complete Code

The entire project from this handbook is available as a public GitHub repository:

🔗 github.com/sandeepmb/freecodecamp-local-ml-platform

The repository includes:

  • All source code (src/ directory)

  • Test files (tests/ directory)

  • Feast feature definitions (feature_repo/)

  • Docker and CI/CD configuration

  • Ready-to-run scripts

Quick Start:

git clone https://github.com/sandeepmb/freecodecamp-local-ml-platform.git
cd freecodecamp-local-ml-platform
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python src/generate_data.py
python src/train_naive.py

References

]]> <![CDATA[ A Comprehensive Guide to Financial Storytelling using Data Visualization ]]> https://www.freecodecamp.org/news/financial-storytelling-using-data-visualization/ 69b1ced06c896b0519c207be Wed, 11 Mar 2026 20:21:36 +0000 In any analysis project, raw tables of numbers often don’t tell the full story. Visualisations simplify complexity by transforming data into shapes that our brains can quickly understand, emphasising trends, outliers, and regime shifts that might be overlooked in raw data.

This is especially vital in finance and trading, where clear visuals can uncover risks, opportunities, and patterns, directly affecting decisions on position sizing, timing, and confidence.

Today, we'll use FMP APIs to interpret earnings data: extracting announcements, surprises, and price reactions across almost 1,000 stocks to identify actionable patterns in post‑earnings movements.

Here’s exactly what we’ll build:

  • Sector heatmap: Maps strongest 3/10-day post-earnings reactions by sector/market-cap buckets.

  • EPS scatter: Tests if earnings beats drive returns (sector-colored, with regression).

  • Return violins: Shows 3-day post-earnings volatility/skew by sector and market-cap.

  • Mega-tech time series: Tracks AAPL/MSFT/NVDA post-earnings patterns over time.

  • Monthly seasonality: Reveals calendar edges in post-earnings returns/surprises.

  • Regime cross-section: Tests sector robustness across bull/bear/sideways markets.

What we'll cover:

  1. Prerequisites

  2. Data Extraction

  3. Storytelling with Charts and Visuals

  4. What Did We Get Out of All This Storyline?

  5. Final Thoughts

Prerequisites

To follow along, you should be comfortable with Python and basic data manipulation in pandas.

This is a code-first guide. I’ll focus on the workflow and the story the charts reveal, and I won’t explain every line of Python. You should be comfortable reading pandas code, loops, and basic plotting logic so you can follow along without needing a step-by-step breakdown of each block.

You’ll need:

  • Python 3.10+

  • A Financial Modeling Prep (FMP) API key

  • pandas, numpy, matplotlib, seaborn, scipy installed

  • Enough local compute and patience to run API loops across a large stock universe

Data Extraction

In the first part of this article, we need to collect all the data required for our visualisation exercise. Using FMP’s Stock Screener API, we will retrieve NASDAQ stocks. The first API call will return 1,000 stocks.

import requests
import pandas as pd
import numpy as np
import json
from datetime import datetime, timedelta
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats

token = 'YOUR FMP TOKEN'

url = f'https://financialmodelingprep.com/stable/company-screener'
querystring = {"apikey":token,"country":"US", "exchange": "NASDAQ", "isActiveTrading": True, "isEtf": False, "isFund": False}
resp = requests.get(url, querystring).json()

df_universe = pd.DataFrame(resp)
df_universe = df_universe[df_universe['exchangeShortName'] == 'NASDAQ']
df_universe

This will give us 1,000 stocks! Next, we'll bin the market capitalisation to gain a better understanding of the results later on, and we will keep only four columns that are necessary: the symbol, name, market cap, and sector.

bins = [0,
        250_000_000,    # 250M
        2_000_000_000,  # 2B
        10_000_000_000, # 10B
        200_000_000_000,# 200B
        float("inf")]

labels = ["Micro", "Small", "Mid", "Large", "Mega"]

df_universe["marketCap"] = pd.cut(df_universe["marketCap"], bins=bins, labels=labels, right=False)
df_universe = df_universe[['symbol', 'companyName', 'marketCap', 'sector']]
df_universe
0*rAiF7Q5TqSNlRG4h

Now it is time to retrieve the earnings using FMP’s Earnings Report API. We'll loop through each symbol and collect all the earnings the endpoint provides to us.

symbols = df_universe['symbol'].to_list()

all_dfs = []

for symbol in symbols:
    url = f"https://financialmodelingprep.com/stable/earnings?symbol={symbol}"
    params = {"apikey": token}
    resp = requests.get(url, params=params)

    if resp.status_code != 200:
        print(f"Error for {symbol}: {resp.status_code} - {resp.text}")
        continue

    data = resp.json()
    if not data:
        print(f"No data for {symbol}")
        continue

    df_symbol = pd.DataFrame(data)
    df_symbol["symbol"] = symbol
    all_dfs.append(df_symbol)

# Single DataFrame with all earnings
df_earnings = pd.concat(all_dfs, ignore_index=True)
df_earnings = df_earnings.dropna(subset=['epsActual', 'epsEstimated', 'revenueActual','revenueEstimated'])
df_earnings

Now we'll calculate the surprise, both for earnings and revenue in percentage terms, so we can later compare apples with apples! We'll keep everything from 2010 onwards.

df_earnings["eps_surprise"] = ((df_earnings["epsActual"] - df_earnings["epsEstimated"]) /
                               abs(df_earnings["epsEstimated"]) * 100).round(2)

df_earnings["revenue_surprise"] = ((df_earnings["revenueActual"] - df_earnings["revenueEstimated"]) /
                                   abs(df_earnings["revenueEstimated"]) * 100).round(2)

df_earnings = df_earnings[['symbol', 'date', 'eps_surprise', 'revenue_surprise']]

df_earnings["date"] = pd.to_datetime(df_earnings["date"])
df_earnings = df_earnings[df_earnings["date"] > "2009-12-31"]

Lastly, as a final step in gathering the data needed for visualization, using FMP’s Historical Index Full Chart API, we'll loop through the stocks in our dataframe, retrieve the historical daily prices, and calculate the return of the stock 3 and 10 trading days before and after the earnings announcement.

unique_symbols = df_earnings["symbol"].unique()

price_results = []

print(f"Processing {len(unique_symbols)} symbols...")

for symbol in unique_symbols:
    # Fetch full historical prices
    url = f"https://financialmodelingprep.com/stable/historical-price-eod/full"
    params = {"apikey":token, "symbol":symbol, "from":'2009-10-01'}
    resp = requests.get(url, params=params)

    if resp.status_code != 200:
        print(f"Error for {symbol}: {resp.status_code}")
        continue

    data = resp.json()

    hist_df = pd.DataFrame(data)
    hist_df["date"] = pd.to_datetime(hist_df["date"])
    hist_df = hist_df.sort_values("date").reset_index(drop=True)

    # Get matching earnings rows
    earnings_symbol = df_earnings[df_earnings["symbol"] == symbol].copy()

    for _, row in earnings_symbol.iterrows():
        earn_date = pd.to_datetime(row["date"]).date()

        # === 3-DAY WINDOWS ===
        pre3_mask = (hist_df["date"].dt.date < earn_date) & \
                    (hist_df["date"].dt.date >= earn_date - timedelta(days=10))
        pre3 = hist_df[pre3_mask].tail(3)

        post3_mask = (hist_df["date"].dt.date > earn_date) & \
                     (hist_df["date"].dt.date <= earn_date + timedelta(days=10))
        post3 = hist_df[post3_mask].head(3)

        pre3_start = pre3["close"].iloc[0] if len(pre3) >= 3 else None
        pre3_end = pre3["close"].iloc[-1] if len(pre3) >= 1 else None
        post3_end = post3["close"].iloc[-1] if len(post3) >= 3 else None

        pct_pre_3d = ((pre3_end - pre3_start) / pre3_start * 100) if pre3_start and pre3_end else None
        pct_post_3d = ((post3_end - pre3_end) / pre3_end * 100) if pre3_end and post3_end else None

        # === 10-DAY WINDOWS ===
        pre10_mask = (hist_df["date"].dt.date < earn_date) & \
                     (hist_df["date"].dt.date >= earn_date - timedelta(days=20))
        pre10 = hist_df[pre10_mask].tail(10)

        post10_mask = (hist_df["date"].dt.date > earn_date) & \
                      (hist_df["date"].dt.date <= earn_date + timedelta(days=20))
        post10 = hist_df[post10_mask].head(10)

        pre10_start = pre10["close"].iloc[0] if len(pre10) >= 10 else None
        pre10_end = pre10["close"].iloc[-1] if len(pre10) >= 1 else None
        post10_end = post10["close"].iloc[-1] if len(post10) >= 10 else None

        pct_pre_10d = ((pre10_end - pre10_start) / pre10_start * 100) if pre10_start and pre10_end else None
        pct_post_10d = ((post10_end - pre10_end) / pre10_end * 100) if pre10_end and post10_end else None

        price_results.append({
            "symbol": symbol,
            "earn_date": earn_date,
            "month": earn_date.month,
            "pct_pre_3d": round(pct_pre_3d, 2) if pct_pre_3d else None,
            "pct_post_3d": round(pct_post_3d, 2) if pct_post_3d else None,
            "pct_pre_10d": round(pct_pre_10d, 2) if pct_pre_10d else None,
            "pct_post_10d": round(pct_post_10d, 2) if pct_post_10d else None,
            "eps_surprise": row["eps_surprise"],
            "revenue_surprise": row["revenue_surprise"]
        })



df_earnings = pd.DataFrame(price_results)
df_earnings.dropna(inplace=True)
df_earnings = df_universe.merge(df_earnings, on="symbol")
df_earnings

As you can see, at the end of the code, we have also merged the initial dataset, so all the information, such as name, marketCap, and sector, is now in a single dataset.

Storytelling with Charts and Visuals

Sector Heatmap

First, we'll present the Sector Heatmap of average 3-day post-earnings returns segmented by sector and market-cap category. This basic visualisation highlights areas with the most significant reactions, enabling traders to swiftly identify high-alpha sectors and market caps for earnings strategies.

# Aggregate: average post-earnings returns and EPS surprise
agg = (
    df_earnings
    .dropna(subset=['pct_post_3d', 'pct_post_10d', 'eps_surprise', 'marketCap', 'sector'])
    .groupby(['sector', 'marketCap'])
    .agg(
        avg_post3d=('pct_post_3d', 'mean'),
        avg_post10d=('pct_post_10d', 'mean'),
        avg_eps_surprise=('eps_surprise', 'mean')
    )
    .reset_index()
)

# Heatmap: average 3-day post-earnings return
heatmap_3d = agg.pivot(index='sector', columns='marketCap', values='avg_post3d')

plt.figure(figsize=(12, 8))
sns.heatmap(
    heatmap_3d,
    annot=True,
    fmt='.2f',
    cmap='RdYlGn',
    center=0,
    linewidths=0.5,
    linecolor='grey'
)
plt.title('Average 3-Day Post-Earnings Return by Sector and Market-Cap Bucket')
plt.xlabel('Market-cap bucket')
plt.ylabel('Sector')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
Heatmap of average 3-day post-earnings returns by sector and market-cap bucket for NASDAQ stocks

Consumer Cyclical and Materials are performing really well, with small and mid caps seeing positive reactions over 1.1%. Real Estate is also doing great, jumping up to +4.0% in mid caps. Energy and Financials are holding steady, staying close to zero. Technology, on the other hand, is showing more muted gains, under 1.1%, indicating there might be limited immediate upside from the big tech earnings.

Building on the 3‑day heatmap, we'll now look at the Sector Heatmap for average 10‑day post‑earnings returns by sector and market‑cap category. This extends the timeframe to capture momentum persistence, revealing which sectors maintain or reverse short‑term reactions.

# Heatmap: average 10-day post-earnings return
heatmap_10d = agg.pivot(index='sector', columns='marketCap', values='avg_post10d')

plt.figure(figsize=(12, 8))
sns.heatmap(
    heatmap_10d,
    annot=True,
    fmt='.2f',
    cmap='RdYlGn',
    center=0,
    linewidths=0.5,
    linecolor='grey'
)
plt.title('Average 10-Day Post-Earnings Return by Sector and Market-Cap Bucket')
plt.xlabel('Market-cap bucket')
plt.ylabel('Sector')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
Heatmap of average 10-day post-earnings returns by sector and market-cap bucket for NASDAQ stocks

Consumer Cyclical stands out with peaks at 3.2% (mega caps), and Industrials and Health Care show consistent gains in mid and large caps around 1.1%. Real Estate has eased after its 3-day surge. Technology has seen a small boost in mega caps (+1.8%) but remains less active overall compared to cyclicals.

Mega‑Cap Tech Time Series

Extending the heatmaps, we’ll now look at a Mega-Cap Tech time series. It tracks 10-day post-earnings returns over time for AAPL, MSFT, NVDA, and a few other mega-cap tech names.

A bubble chart works well here because it encodes more than one thing at once. The x-axis is the earnings date, the y-axis is the 10-day post-earnings return, the bubble size scales with the absolute EPS surprise magnitude, and the color shows whether the surprise was a beat or a miss. This makes it easy to spot outlier quarters and see whether big surprises consistently lead to bigger post-earnings moves.

# Define mega-cap tech tickers (top ones from data: AAPL, MSFT, NVDA, AMZN, GOOG/GOOGL, META)
tech_tickers = ['AAPL', 'MSFT', 'NVDA', 'AMZN', 'GOOG', 'GOOGL', 'META']

# Filter data for mega-cap tech
df_tech = (
    df_earnings[df_earnings['symbol'].isin(tech_tickers)]
    .dropna(subset=['earn_date', 'pct_post_10d', 'eps_surprise'])
    .sort_values('earn_date')
    .assign(
        earn_date=lambda x: pd.to_datetime(x['earn_date'])
    )
)

# Create time-series plot: pct_post_10d vs earn_date, sized/color by eps_surprise
plt.figure(figsize=(14, 8))

# Scatter plot
scatter = plt.scatter(
    df_tech['earn_date'],
    df_tech['pct_post_10d'],
    s=np.abs(df_tech['eps_surprise']) * 50 + 20,  # Size by abs(eps_surprise)
    c=df_tech['eps_surprise'],
    cmap='RdYlBu_r',
    alpha=0.7,
    edgecolors='black',
    linewidth=0.5
)

plt.colorbar(scatter, label='EPS Surprise (%)')
plt.xlabel('Earnings Date')
plt.ylabel('10-Day Post-Earnings Return (%)')
plt.title('Mega-Cap Tech: 10-Day Post-Earnings Returns vs Time\n(Point size/color by EPS Surprise)')
plt.grid(True, alpha=0.3)

# Add trend line
z = np.polyfit(pd.to_numeric(df_tech['earn_date']), df_tech['pct_post_10d'], 1)
p = np.poly1d(z)
plt.plot(df_tech['earn_date'], p(pd.to_numeric(df_tech['earn_date'])), "r--", alpha=0.8, linewidth=2, label=f'Trend: {z[0]:.3f}x + {z[1]:.1f}')

plt.legend()
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
Bubble chart of 10-day post-earnings returns over time for AAPL, MSFT, NVDA, AMZN, GOOG, GOOGL, META. Bubble size reflects EPS surprise magnitude. Color reflects beat or miss

That large red bubble around 2018 is almost certainly AAPL’s Q4 2018 earnings miss (Jan 2019 announcement, but fiscal Q4 2018 data) and it stands out because:

  • Large size = massive EPS surprise magnitude (Apple cut guidance dramatically, ~10% miss)

  • Red colour = negative surprise

  • Low Y position = poor 10‑day return (~-10% range visible)

This was Apple’s infamous “iPhone demand warning” that triggered the January 2019 market panic. Perfect example of how one outlier event can anchor the whole trend line downward in your visualisation.

EPS Surprise Scatter Plot

After identifying major tech trends, let's now look at the EPS Surprise Scatter plots. This plot checks a simple hypothesis. Do earnings beats lead to positive returns, and do misses lead to negative returns? We plot EPS surprise on the x-axis and post-earnings returns on the y-axis, then add a regression line to show the average relationship.

# Prepare data: drop NaNs and convert earn_date if needed (not used here)
df_plot = (
    df_earnings
    .dropna(subset=['eps_surprise', 'pct_post_3d', 'pct_post_10d', 'sector'])
    .copy()
)

# 1. Scatter: EPS Surprise vs 3-Day Post-Return, colored by sector
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
sns.scatterplot(
    data=df_plot,
    x='eps_surprise',
    y='pct_post_3d',
    hue='sector',
    alpha=0.6,
    s=40
)

# Regression line (overall)
slope, intercept, r_value, p_value, std_err = stats.linregress(df_plot['eps_surprise'], df_plot['pct_post_3d'])
line = slope * df_plot['eps_surprise'] + intercept
plt.plot(df_plot['eps_surprise'], line, 'red', linestyle='--', linewidth=2,
         label=f'y = {slope:.3f}x + {intercept:.2f}\nR²={r_value**2:.3f}')
plt.xlabel('EPS Surprise (%)')
plt.ylabel('3-Day Post-Earnings Return (%)')
plt.title('EPS Surprise vs 3-Day Post-Return by Sector')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(True, alpha=0.3)

# 2. Scatter: EPS Surprise vs 10-Day Post-Return, colored by sector
plt.subplot(1, 2, 2)
sns.scatterplot(
    data=df_plot,
    x='eps_surprise',
    y='pct_post_10d',
    hue='sector',
    alpha=0.6,
    s=40
)

# Regression line (overall)
slope10, intercept10, r_value10, p_value10, std_err10 = stats.linregress(df_plot['eps_surprise'], df_plot['pct_post_10d'])
line10 = slope10 * df_plot['eps_surprise'] + intercept10
plt.plot(df_plot['eps_surprise'], line10, 'red', linestyle='--', linewidth=2,
         label=f'y = {slope10:.3f}x + {intercept10:.2f}\nR²={r_value10**2:.3f}')
plt.xlabel('EPS Surprise (%)')
plt.ylabel('10-Day Post-Earnings Return (%)')
plt.title('EPS Surprise vs 10-Day Post-Return by Sector')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Optional: Summary table of correlations by sector
corr_3d = df_plot.groupby('sector')[['eps_surprise', 'pct_post_3d']].corr().unstack().xs('pct_post_3d', level=1, axis=1)['eps_surprise']
corr_10d = df_plot.groupby('sector')[['eps_surprise', 'pct_post_10d']].corr().unstack().xs('pct_post_10d', level=1, axis=1)['eps_surprise']

corr_df = pd.DataFrame({
    'Corr_EPS_3Day': corr_3d.round(3),
    'Corr_EPS_10Day': corr_10d.round(3)
}).sort_values('Corr_EPS_10Day', ascending=False)
Scatter plot of EPS surprise versus post-earnings returns with sector colors and overall regression line

The red dashed trend line illustrates the typical relationship: for every 1% EPS beat, stocks tend to gain about 0.05–0.1% over 3 to 10 days. The gentle slope suggests that while surprises can give a little boost, they don’t guarantee large moves.

You’ll notice that Consumer Cyclical dots mainly cluster in the upper right (beats leading to gains), and Real Estate shows a steeper increase. The wide spread around the line indicates that other factors often influence stock movements beyond surprises.

Return Distribution Violins

Heatmaps show averages, but averages can hide risk. Violin plots show the full distribution of returns, including how wide the outcomes are and whether the tails are heavy. Here we plot 3-day post-earnings return distributions by sector and by market-cap bucket.

# Prepare data
df_plot = (
    df_earnings
    .dropna(subset=['pct_post_3d', 'sector', 'marketCap'])
    .copy()
)

# 1. Violin plot: 3-day post-returns by sector
plt.figure(figsize=(15, 6))

plt.subplot(1, 2, 1)
sns.violinplot(
    data=df_plot,
    x='sector',
    y='pct_post_3d',
    inner='quartile',
    palette='Set2'
)
plt.title('Distribution of 3-Day Post-Earnings Returns by Sector (Violin)')
plt.xlabel('Sector')
plt.ylabel('3-Day Post-Earnings Return (%)')
plt.xticks(rotation=45, ha='right')
plt.grid(True, alpha=0.3)

# 2. Violin plot: 3-day post-returns by market-cap group
plt.subplot(1, 2, 2)
sns.violinplot(
    data=df_plot,
    x='marketCap',
    y='pct_post_3d',
    inner='quartile',
    palette='Set3'
)
plt.title('Distribution of 3-Day Post-Earnings Returns by Market-Cap (Violin)')
plt.xlabel('Market-cap bucket')
plt.ylabel('3-Day Post-Earnings Return (%)')
plt.xticks(rotation=45, ha='right')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()


plt.show()

# Summary statistics table
summary = df_plot.groupby(['sector', 'marketCap'])['pct_post_3d'].agg(['mean', 'median', 'std', 'count']).round(2)
print("Summary Statistics: Mean/Median/Std/Count of 3-Day Returns by Sector & Market-Cap")
print(summary)
Violin plots showing distribution of 3-day post-earnings returns by sector and by market-cap bucket

All violins concentrate near zero with modest variations (±5%), indicating that post-earnings reactions are generally noisy and lack a clear direction. Markets efficiently incorporate expectations, resulting in little predictable advantage. Consumer Cyclical and Materials sectors display slightly more frequent upside surprises, while small caps exhibit the greatest variability, reflecting higher risk and occasional gains. Not every visualization reveals alpha; this one honestly illustrates the difficulty involved.

Monthly Seasonality

After observing narrow return distributions near zero, let's now look at Monthly Seasonality in four panels: average 3/10‑day post‑returns, EPS surprises, and event counts by month. This reveals calendar effects,  systematic seasonal biases ,  that can influence timing of entries despite noisy individual responses.

# 1. Ensure earn_date is datetime
df_month = (
    df_earnings
    .dropna(subset=['earn_date', 'pct_post_3d', 'pct_post_10d', 'eps_surprise'])
    .copy()
)

df_month['earn_date'] = pd.to_datetime(df_month['earn_date'])

# 2. Derive month number and name
df_month['month_num'] = df_month['earn_date'].dt.month
df_month['month_name'] = df_month['earn_date'].dt.strftime('%b')

# 3. Aggregate averages by month
monthly_agg = (
    df_month
    .groupby('month_num')
    .agg(
        pct_post_3d_mean=('pct_post_3d', 'mean'),
        pct_post_10d_mean=('pct_post_10d', 'mean'),
        eps_surprise_mean=('eps_surprise', 'mean'),
        n_obs=('earn_date', 'count')
    )
    .reset_index()
    .sort_values('month_num')
)

# Keep a stable month order and names
month_order = monthly_agg['month_num'].tolist()
month_labels = df_month.drop_duplicates('month_num').set_index('month_num')['month_name'].reindex(month_order)

monthly_agg['month_name'] = month_labels.values

# 4. Plot bar charts
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
fig.suptitle('Monthly Seasonality of Post-Earnings Returns and EPS Surprise', fontsize=16)

# Avg 3-day return
axes[0, 0].bar(monthly_agg['month_name'], monthly_agg['pct_post_3d_mean'], color='skyblue')
axes[0, 0].set_title('Avg 3-Day Post-Earnings Return by Month')
axes[0, 0].set_ylabel('Return (%)')
axes[0, 0].grid(alpha=0.3)

# Avg 10-day return
axes[0, 1].bar(monthly_agg['month_name'], monthly_agg['pct_post_10d_mean'], color='lightgreen')
axes[0, 1].set_title('Avg 10-Day Post-Earnings Return by Month')
axes[0, 1].set_ylabel('Return (%)')
axes[0, 1].grid(alpha=0.3)

# Avg EPS surprise
axes[1, 0].bar(monthly_agg['month_name'], monthly_agg['eps_surprise_mean'], color='salmon')
axes[1, 0].set_title('Avg EPS Surprise by Month')
axes[1, 0].set_ylabel('EPS Surprise')
axes[1, 0].grid(alpha=0.3)

# Number of observations
axes[1, 1].bar(monthly_agg['month_name'], monthly_agg['n_obs'], color='gold')
axes[1, 1].set_title('Number of Earnings Events by Month')
axes[1, 1].set_ylabel('Count')
axes[1, 1].grid(alpha=0.3)

for ax in axes.ravel():
    ax.set_xlabel('Month')
    ax.tick_params(axis='x', rotation=0)

plt.tight_layout()
plt.show()
Four-panel bar charts showing monthly averages of 3-day returns, 10-day returns, EPS surprise, and event counts

Jan/Oct tend to have the best 3‑day returns, about 0.8%, while May/Jul usually see weaker results. The 10‑day trends show a similar but gentler pattern, with February and August reaching peaks. EPS surprises are slightly negative in January and May, possibly due to tough comparisons, and there are fewer events in July, August, and December because of holidays. While there’s a hint of seasonality, its impact is quite small, around 0.5%.

Regime Cross-Section

Finally, after subtle monthly patterns, we'll look at the Regime Cross‑Section: sector 10‑day post‑earnings returns by market regime (heatmap at the top, bars below). This stress‑tests earlier findings  ( do patterns persist across bull, bear, and COVID eras), revealing rotation opportunities and regime dependence.

# Prepare data with year extraction
df_regimes = (
    df_earnings
    .dropna(subset=['earn_date', 'pct_post_10d', 'sector'])
    .copy()
)

df_regimes['earn_date'] = pd.to_datetime(df_regimes['earn_date'])
df_regimes['year'] = df_regimes['earn_date'].dt.year

# Define market regimes (adjust years based on your data/market history)
# Example: Bull (2023-2025), Bear/Transition (2022), COVID (2020-2021), etc.
def assign_regime(year):
    if year >= 2023:
        return 'Bull (2023+)'
    elif year == 2022:
        return 'Bear (2022)'
    elif 2020 <= year <= 2021:
        return 'COVID Recovery'
    elif 2018 <= year <= 2019:
        return 'Pre-COVID'
    else:
        return 'Earlier'

df_regimes['market_regime'] = df_regimes['year'].apply(assign_regime)

# 1. Aggregate: average 10-day returns by sector and regime/year
agg_data = (
    df_regimes
    .groupby(['sector', 'market_regime'])['pct_post_10d']
    .agg(['mean', 'count'])
    .reset_index()
    .query('count >= 5')  # Filter low-sample regimes
)

# 2. Visualization: Heatmap first (quick overview)
plt.figure(figsize=(12, 8))

plt.subplot(2, 1, 1)
pivot_heatmap = agg_data.pivot(index='sector', columns='market_regime', values='mean')
sns.heatmap(pivot_heatmap, annot=True, fmt='.2f', cmap='RdYlGn', center=0, linewidths=0.5)
plt.title('Average 10-Day Post-Earnings Returns: Sector x Market Regime Heatmap')

# 3. Bar charts: By regime (stacked by sector)
plt.subplot(2, 1, 2)
regime_order = agg_data.groupby('market_regime')['mean'].mean().sort_values(ascending=False).index
sns.barplot(data=agg_data, x='market_regime', y='mean', hue='sector',
            palette='Set2', order=regime_order)
plt.title('Average 10-Day Returns by Market Regime (Colored by Sector)')
plt.ylabel('10-Day Post-Return (%)')
plt.xlabel('Market Regime')
plt.xticks(rotation=45, ha='right')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(axis='y', alpha=0.3)

plt.tight_layout()
plt.show()

# 5. Summary tables
print("Average Returns by Sector x Market Regime (min 5 obs):")
print(agg_data.pivot(index='sector', columns='market_regime', values='mean').round(2))

# 6. Ranking: Best/worst performing sectors by regime
print("\nTop/Bottom Sectors by Regime:")
for regime in regime_order:
    regime_data = agg_data[agg_data['market_regime'] == regime].sort_values('mean', ascending=False)
    print(f"\n{regime}:")
    print(regime_data[['sector', 'mean', 'count']].round(2).head(3))
Heatmap and bar chart showing average 10-day post-earnings returns by sector across market regimes

Consumer Cyclical does well during Bull (2023+) and COVID Recovery (1.5–2%), but it’s less favorable in Bear 2022. Utilities turned negative before COVID. The bottom bars show the COVID era led overall gains (1%), with Basic Materials and Industrials being the strongest. The recent Bull remains positive but less so. Sector leadership shifts depending on the market regime , there are no consistent winners.

What Did We Get Out of All This Storyline?

Guiding you through six interconnected visualizations, we’ve turned 15 years of earnings data into a clear and engaging story.

Each chart responds to a specific question, yet together, they paint a bigger picture: earnings surprises influence markets, but not in the same way everywhere. Some sectors, periods, and regimes often provide consistent advantages, while others don’t.

Here’s what the data shows us:

  • No definitive alpha here, but specific opportunities are present: Markets are mostly efficient,  returns hover near zero with weak surprise correlations ,  yet Consumer Cyclicals and Materials consistently show upside potential across different timeframes and market sizes. Timing your sector choice is important.

  • Timing windows alter the story: 3-day reactions benefit Real Estate mid-caps (+4%), while 10-day reactions shift leadership to Consumer Cyclical mega-caps (+3.2%). Don’t assume all earnings reactions occur at the same pace.

  • Mega-tech hype isn’t eternal: The bubble chart shows AAPL/MSFT/NVDA delivered strong returns from 2020–2022, but the falling trend since then indicates waning market enthusiasm. Don’t chase yesterday’s overhyped stocks.

  • Calendar patterns reward patience: January and October deliver slightly stronger post-earnings returns (~0.8%), while July and August tend to have lower liquidity. Combine seasonal timing with sector choices for additional gains.

  • Market regimes change winners: Cyclicals underperformed during COVID recovery and the bull run (2023+), while Industrials peaked during the recovery. There are no universal “best performers,” only the best performers for now. Adjust to the regime.

  • The actionable setup: Small to mid-cap cyclical longs in January during bull markets combine all these signals for maximum conviction ,  where sector timing, seasonality, and regime alignment converge.

Final Thoughts

This exercise shows why visualization is important in finance: raw tables of returns and surprises wouldn’t reveal these patterns.

  • Heatmaps instantly highlighted sector winners.

  • Scatter plots demonstrated the weak surprise‑return connection. Bubble charts narrated the mega‑tech story over time.

  • Violins unveiled the harsh truth  that markets are noisy. Cross‑sectional regime analysis reminded us that yesterday’s approach doesn’t ensure tomorrow’s returns.

The effort to interpret this data pays off: you shift from passive observation to active pattern recognition. You see not just what occurred, but where and when it happened. In trading and analysis, understanding the shape of complexity often surpasses having a perfect formula.

Visual storytelling turns data into intuition . And intuition, based on evidence, outperforms guesswork every time.

]]> <![CDATA[ How to Build a Spam Email Detector with Python and Naive Bayes Classifier ]]> https://www.freecodecamp.org/news/how-to-build-a-spam-email-detector-with-python-and-naive-bayes-classifier/ 69b0a8f8abc0d95001af6574 Tue, 10 Mar 2026 23:27:52 +0000 Ever wondered how Gmail knows that an email promising you $10 million is spam? Or how it catches those "You've won a free iPhone!" messages before they reach your inbox?

In this tutorial, you'll build your own spam email classifier from scratch using the Naive Bayes algorithm. By the end, you'll have a working model that achieves over 97% accuracy—and you'll understand exactly how it works under the hood.

This project was inspired by the Python Machine Learning Workbook for Beginners by AI Publishing, which offers excellent hands-on ML projects for those starting their journey. (Note: I have no affiliation with the authors — I simply found it a useful resource.)

Table of Contents

What You'll Learn

  • How email spam filters actually work

  • The intuition behind the Naïve Bayes algorithm

  • Text preprocessing techniques for machine learning

  • How to evaluate classification models

  • Building a complete spam detection pipeline in Python

Prerequisites

You should have basic familiarity with Python and some understanding of fundamental machine learning concepts. Don't worry if you're still learning—I'll explain everything as we go.

Why Naive Bayes for Spam Detection?

Before we dive into code, let's understand why Naive Bayes is particularly well-suited for this task.

Imagine you receive an email containing words like "free," "winner," "click here," and "limited time offer." Your brain immediately flags this as suspicious. The Naive Bayes algorithm does something similar—it calculates the probability that an email is spam based on the words it contains.

The algorithm is called "naive" because it makes a simplifying assumption: it treats each word as independent of every other word. In reality, word combinations matter (think "free trial" vs. "free money"), but this simplification works remarkably well in practice.

Why Choose Naive Bayes?

  • Speed: It trains incredibly fast, even on large datasets

  • Efficiency: Requires minimal training data to produce reliable results

  • Simplicity: Easy to implement and interpret

  • Performance: Despite its simplicity, it often outperforms more complex algorithms for text classification

Limitations to keep in mind:

  • The independence assumption means it can't capture relationships between words

  • If a word appears in the test data but never appeared in training, the algorithm assigns it zero probability (though there are ways to handle this)

Now let's build our spam detector.

How to Set Up Your Environment

First, install the required libraries. Open your terminal or run this in a Jupyter notebook cell:


%pip install regex wordcloud numpy pandas seaborn matplotlib scikit-learn

Here's a quick summary of what each library does:

  • regex / re — for cleaning text using pattern matching

  • wordcloud — for visualizing which words appear most frequently

  • numpy and pandas — for data loading and manipulation

  • seaborn and matplotlib — for charts and visualizations

  • scikit-learn — provides the Naive Bayes classifier, vectorizer, and evaluation tools

Once installation is complete, import everything at the top of your script or notebook. Grouping all imports at the top is a Python best practice — it makes dependencies easy to spot at a glance.

# Data manipulation and analysis
import pandas as pd
import numpy as np

# Data visualization
import seaborn as sns
import matplotlib.pyplot as plt

# Natural language processing
import nltk
import re
from nltk.corpus import stopwords

# Machine learning
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

# Word cloud visualization
from wordcloud import WordCloud

How to Load and Explore the Dataset

We'll use a dataset of labeled emails. You can download it from Kaggle or use any similar email dataset with text and spam columns.

Use pandas' read_csv() function to load the dataset from a CSV file into a DataFrame — a table-like structure that makes it easy to inspect and manipulate data. The head() method then displays the first 5 rows so you can confirm the data loaded correctly and understand its structure.

message_dataset = pd.read_csv('emails.csv')
message_dataset.head()

Output:

text spam
0 Subject: naturally irresistible your corporate... 1
1 Subject: the stock trading gunslinger fanny i... 1
2 Subject: unbelievable new homes made easy im ... 1
3 Subject: 4 color printing special request add... 1
4 Subject: do not have money , get software cds ... 1

Next, call shape on the DataFrame to check its dimensions — this returns a tuple of (rows, columns) and is a quick way to confirm you loaded the full dataset without truncation.

# Get the dimensions of our dataset (rows, columns)
message_dataset.shape

Output:

(5728, 2)

The dataset contains 5,728 emails with two columns: text (the email content) and spam (1 for spam, 0 for legitimate emails).

How to Visualize the Data Distribution

Before training any model, it's crucial to understand your data. Let's see how spam and legitimate emails are distributed.

value_counts() tallies how many emails belong to each class (spam vs. legitimate). Chaining .plot(kind="pie") on the result converts those counts directly into a pie chart. The autopct="%1.0f%%" argument tells matplotlib to label each slice with its percentage, rounded to the nearest whole number.

plt.rcParams["figure.figsize"] = [8, 10]
message_dataset.spam.value_counts().plot(kind="pie", autopct="%1.0f%%")

Output:

de29f062-db6b-4f7c-87ad-bf03440cc3fc

You'll see that approximately 24% of emails in the dataset are spam, while 76% are legitimate. This is a moderately imbalanced dataset, which we'll keep in mind when evaluating our model.

How to Analyze Word Patterns with Word Clouds

Word clouds provide an intuitive visualization of the most frequent words in a text corpus. Words that appear more often are rendered larger. Let's create separate word clouds for spam and legitimate emails to identify distinguishing patterns.

First, we need to remove stop words — common words like "the," "is," and "at" that appear everywhere and carry no meaningful signal for classification. NLTK's stopwords.words("english") returns a pre-built list of these words. The apply() method runs a function across every row in the column, and the lambda inside it splits each email into individual words, filters out any stop words, then rejoins the remaining words into a clean string.

stop = stopwords.words("english")

message_dataset["text_without_sw"] = message_dataset["text"].apply(
    lambda x: "".join([item for item in x.split() if item not in stop])
)

Now let's visualize the spam emails. We filter the DataFrame to rows where spam == 1, join all that text into a single large string, and pass it to WordCloud().generate(). The imshow() function renders the resulting image, and axis("off") hides the x/y axes since they're not meaningful for an image display.

message_dataset_spam = message_dataset[message_dataset["spam"] == 1]

plt.rcParams["figure.figsize"] = [8, 10]
text = ' '.join(message_dataset_spam['text_without_sw'])
wordcloud2 = WordCloud().generate(text)

plt.imshow(wordcloud2)
plt.axis("off")
plt.show()

Output:

d33d8b61-7ec4-4044-a98a-c7b9ce95eabf

Now do the same for legitimate emails by filtering to rows where spam == 0:

message_dataset_ham = message_dataset[message_dataset["spam"] == 0]

plt.rcParams["figure.figsize"] = [8, 10]
text = ' '.join(message_dataset_ham['text_without_sw'])
wordcloud2 = WordCloud().generate(text)

plt.imshow(wordcloud2)
plt.axis("off")
plt.show()

Output:

368e5251-3072-423f-b298-fb00d03254f3

Key observations:

  • Spam emails frequently contain promotional language: "free," "money," "offer," "click," "please"

  • Legitimate emails contain more conversational and work-related terms: "company," "time," "thanks"

You'll also notice the word "enron" appearing prominently in the legitimate emails cloud. This is because the non-spam emails in this dataset are drawn from the publicly available Enron email corpus — a large collection of real internal emails from Enron Corporation that was released during their 2001 fraud investigation. It has since become one of the most widely used benchmark datasets in NLP research, which is why "enron" shows up so frequently as a word in legitimate email content.

These patterns give us confidence that word-based classification will work well.

How to Preprocess the Text Data

Raw text needs cleaning before machine learning algorithms can process it effectively. Let's first separate our features from our labels. In ML terminology, X holds the inputs (the email text we use to make predictions) and y holds the target labels (1 for spam, 0 for legitimate).

X = message_dataset["text"]
y = message_dataset["spam"]

Now we'll define a function to clean the text. The re.sub() function from Python's built-in re module performs pattern-based substitution using regular expressions. We call it three times in sequence:

  1. re.sub('[^a-zA-Z]', ' ', doc) — replaces anything that isn't a letter (numbers, punctuation, symbols) with a space. This strips noise that doesn't help with classification.

  2. re.sub(r'\s+[a-zA-Z]\s+', ' ', document) — removes isolated single characters (like "I" or "a" left behind after removing punctuation) by matching any single letter surrounded by whitespace.

  3. re.sub(r'\s+', ' ', document) — collapses multiple consecutive spaces into a single space, tidying up any extra gaps created by the previous two steps.

def clean_text(doc):
    document = re.sub('[^a-zA-Z]', ' ', doc)
    document = re.sub(r'\s+[a-zA-Z]\s+', ' ', document)
    document = re.sub(r'\s+', ' ', document)
    return document

Apply this cleaning function to every email in the dataset. We first convert the pandas Series to a plain Python list using list(), then loop through each email, clean it, and collect the results in X_sentences.

# Create an empty list to store cleaned emails
X_sentences = []

# Convert the pandas Series to a list for iteration
reviews = list(X)

# Clean each email and add it to our list
for rev in reviews:
    X_sentences.append(clean_text(rev))

How to Convert Text to Numerical Features

Machine learning algorithms work with numbers, not text. We need to transform our cleaned text into a numerical representation.

TF-IDF (Term Frequency-Inverse Document Frequency) is a great choice for this. It assigns each word a score that reflects how important it is to a particular document relative to the entire dataset. A word that appears often in one email but rarely across all emails gets a high score — meaning it's distinctive and likely meaningful. Common words that appear everywhere get a lower score.

TfidfVectorizer from scikit-learn handles this transformation. The parameters we set control what gets included:

  • max_features=2500 — only keeps the 2,500 most frequent words, discarding rare ones that don't generalize well

  • min_df=5 — ignores words that appear in fewer than 5 emails (too rare to be useful)

  • max_df=0.7 — ignores words that appear in more than 70% of all emails (too common to be distinctive)

  • stop_words=stopwords.words('english') — removes common English words like "the" and "is"

fit_transform() does two things in one step: it learns the vocabulary from our text (fit), then converts each email into a numerical vector based on that vocabulary (transform). Calling .toarray() on the result converts the sparse matrix output — which stores only non-zero values for efficiency — into a regular dense NumPy array that scikit-learn classifiers expect.

vectorizer = TfidfVectorizer(
    max_features=2500,
    min_df=5,
    max_df=0.7,
    stop_words=stopwords.words('english')
)

X = vectorizer.fit_transform(X_sentences).toarray()

Each email is now represented as a vector of 2,500 numbers, where each number is the TF-IDF score for a specific word.

How to Train the Naive Bayes Classifier

Now comes the exciting part — training our model! First, split the data into training and test sets using train_test_split(). This function randomly shuffles and divides both X and y simultaneously, keeping labels aligned with their corresponding emails. Setting test_size=0.20 reserves 20% of the data for testing. Setting random_state=42 seeds the random number generator so you get the same split every time you run the code, making your results reproducible.

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.20,
    random_state=42
)

Now train the Multinomial Naive Bayes classifier. We use MultinomialNB specifically because it's designed for features that represent counts or frequencies — exactly what TF-IDF scores are. Calling fit(X_train, y_train) trains the model by having it calculate the probability of each word appearing in spam versus legitimate emails across the training set. Those probability tables are what the model uses later to classify new emails.


spam_detector = MultinomialNB()
spam_detector.fit(X_train, y_train)

That's it! The Naive Bayes algorithm is remarkably fast—training completes in milliseconds even with thousands of emails.

How to Evaluate Model Performance

Let's see how well our spam detector performs on emails it has never seen before. The predict() method takes the test set features and returns a predicted label (0 or 1) for each email, based on the probability tables the model learned during training.


y_pred = spam_detector.predict(X_test)

Now evaluate the predictions using three different tools from scikit-learn's metrics module:

  • confusion_matrix() — produces a 2×2 grid comparing actual vs. predicted labels, showing exactly where the model gets things right and wrong

  • classification_report() — prints precision, recall, and F1-score for each class, giving a more complete picture than accuracy alone

  • accuracy_score() — returns the overall percentage of correct predictions


print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
print(accuracy_score(y_test, y_pred))

Output:

[[849   7]
 [ 18 272]]

              precision    recall  f1-score   support

           0       0.98      0.99      0.99       856
           1       0.97      0.94      0.96       290

    accuracy                           0.98      1146
   macro avg       0.98      0.96      0.97      1146
weighted avg       0.98      0.98      0.98      1146

0.9781849912739965

Our model achieves 97.82% accuracy! Let's break down what the confusion matrix tells us:

  • 849: Legitimate emails correctly identified as legitimate (True Negatives)

  • 7: Legitimate emails incorrectly marked as spam (False Positives)

  • 18: Spam emails that slipped through as legitimate (False Negatives)

  • 272: Spam emails correctly caught (True Positives)

The classification report shows:

  • For legitimate emails (class 0): 98% precision, 99% recall

  • For spam emails (class 1): 97% precision, 94% recall

These numbers are impressive, especially considering the simplicity of our approach.

How to Test on Individual Emails

Let's verify our model works by testing it on a specific email. We'll first print the cleaned text at index 56 and its actual label to see what we're working with. Then we'll ask the model to predict it.


print(X_sentences[56])
print(y[56])

Output:

Subject localized software all languages available hello we would like to offer localized software versions german french spanish uk and many others aii iisted software is available for immediate downioad no need to wait week for cd deiivery just few exampies norton lnternet security pro windows xp professionai with sp fuil version corei draw graphics suite dreamweaver mx homesite inciudinq macromedia studio mx just browse our site and find any software you need in your native ianguaqe best reqards kayieen 
1

This is clearly a spam email trying to sell pirated software. The actual label is 1 (spam). Now pass this single email through the same pipeline — first transforming it into a TF-IDF vector using the already-fitted vectorizer, then calling predict() on the result. It's important to use the same vectorizer that was fitted on the training data, so the word-to-index mapping is consistent.


print(spam_detector.predict(vectorizer.transform([X_sentences[56]])))

Output:

[1]

The model correctly identifies this promotional email as spam.

Key Takeaways

  1. Naive Bayes is powerful for text classification despite its simplifying assumptions. For spam detection, it achieves excellent accuracy with minimal computational cost.

  2. Text preprocessing matters. Removing noise (special characters, numbers, extra spaces) helps the algorithm focus on meaningful patterns.

  3. TF-IDF captures word importance effectively. It gives higher weight to distinctive words that help differentiate spam from legitimate emails.

  4. Always evaluate with multiple metrics. Accuracy alone can be misleading, especially with imbalanced datasets. Precision, recall, and F1-score give a complete picture.

  5. Start simple. Before reaching for complex deep learning models, try classical algorithms like Naïve Bayes. They're interpretable, fast, and often surprisingly effective.

Next Steps

Want to improve this spam detector further? Here are some ideas:

  • Experiment with different vectorizers: Try CountVectorizer or word embeddings (Word2Vec, GloVe)

  • Handle class imbalance: Use techniques like SMOTE or adjust class weights

  • Feature engineering: Add features like email length, number of links, or sender domain

  • Try other algorithms: Compare with SVM, Random Forest, or gradient boosting

  • Deploy the model: Build a simple API using Flask or FastAPI

Conclusion

You've built a spam email classifier that achieves over 97% accuracy using the Naïve Bayes algorithm. Along the way, you learned about text preprocessing, feature extraction with TF-IDF, and model evaluation techniques.

The beauty of this approach is its simplicity. With just a few dozen lines of code, you've created something that actually works—and now you understand the principles behind commercial spam filters.

Feel free to experiment with the code, try different parameters, and see how the results change. That's the best way to deepen your understanding.

References

]]> <![CDATA[ How to Create Boxplots and Model Data in R Using ggplot2 ]]> https://www.freecodecamp.org/news/how-to-create-boxplots-and-model-data-in-r/ 69693680d6f0e208b327d21c Thu, 15 Jan 2026 18:48:32 +0000 In this tutorial, you’ll walk through a complete data analysis project using the HR Analytics dataset by Saad Haroon on Kaggle. You’ll start by loading and cleaning the data, then explore it visually using boxplots with ggplot2. Finally, you’ll learn about statistical modelling using linear regression and logistic regression in R.

By the end of this article, you should understand how to create boxplots in R, why they matter, and how they fit into a real-world analytics workflow.

Table of Contents

Prerequisites

Before you begin, you should be comfortable with the following:

  • Basic R syntax (variables, functions, data frames).

  • Installing and loading R packages.

  • Understanding what rows and columns represent in a dataset.

  • Very basic statistics (mean, median, distributions).

How to Set Up Your R Environment

Start by installing and loading the packages you will need.

install.packages(c("tidyverse", "ggplot2"))
library(tidyverse)
library(ggplot2)

tidyverse provides tools for data manipulation and visualization. ggplot2 is the visualization engine you will use for boxplots. Loading the libraries makes their functions available for use

How to Load and Inspect the Data

First, download the HR Analytics dataset by Saad Haroon from Kaggle.

Assuming the downloaded dataset is saved as "C:/Users/johndoe/Downloads/archive (2)/HR_Analytics.csv", load the path file into R.

You can view a sample of the the dataset by running the head function. To view the structure of the dataset, you can run the str function.

hr <- read.csv("C:/Users/johndoe/Downloads/archive (2)/HR_Analytics.csv")
head(hr)
str(hr)

The read.csv function imports the dataset into R. The head function shows the first six rows so you can preview the data. The str function reveals data types, helping you spot categorical versus numeric variables early.

Remember that understanding your data structure early prevents errors later when plotting or modeling. Once you run the head function, you should see the following in your console:

From the head function, you can see:

first-six-rows-of-a-hr-dataset-shown-in-the-r-console

Structure

  • Each row represents one employee.

  • Each column represents a feature/variable about the employee.

Key Columns & Meaning

  • EmpID → Employee identifier

  • Age → Age in years

  • AgeGroup → Age category (for example, 18-25)

  • Attrition → Whether the employee left or not (Yes/No)

  • BusinessTravel → Travel frequency (Travel_Rarely, Travel_Frequently, Non-Travel)

  • Department → Employee department

  • DistanceFromHome → Distance from home to office (km)

  • Education / EducationField → Level and field of education

  • EmployeeCount → Usually 1 per employee (redundant)

  • Gender → Male / Female

  • JobRole / JobSatisfaction → Job title and satisfaction level

  • MonthlyIncome / SalarySlab → Salary amount and category

  • YearsAtCompany / YearsInCurrentRole → Experience metrics

  • OverTime → Works overtime (Yes/No)

  • Other features: PerformanceRating, TrainingTimesLastYear, WorkLifeBalance, StockOptionLevel, and so on.

Data Types

  • Numeric → Age, DistanceFromHome, MonthlyIncome, YearsAtCompany

  • Categorical / Character → Attrition, Gender, Department, JobRole

Observations

  • The dataset is tabular, like a spreadsheet.

  • There are multiple categorical columns

  • There are multiple numeric columns

  • Some columns seem redundant or constant; doesn’t provide useful information because of the same values (for example, EmployeeCount)

From the str function, you can gather that:

r-output-showing-structure-of-hr-dataset

The dataset contains 1,480 observations and 38 variables. Each row represents one employee, and each column represents a feature about that employee.

Each column has a name, data type, and example values. For instance, Age and DistanceFromHome are numeric (int), with values like 28 or 12. EmpID and Department are character strings (chr), with examples like Research & Development or Sales. Other features include JobRole (Analyst, Manager) and Attrition (Yes/No).

The dataset contains mixed data types. Some columns are numeric, such as MonthlyIncome or YearsAtCompany. Some are character or categorical, like Gender (Male/Female) and BusinessTravel (Travel_Rarely, Travel_Frequently). A few columns are redundant or constant. For example, EmployeeCount has the same value of 1 for all rows and does not provide useful information.

How to Clean and Prepare the Data

Before visualization, you must clean your data. In order to find out what you need to clean you can investigate the data.

Run the summary function to view the statistics of the dataset. You also need to run the is.na function to identify missing values to be removed.

summary(hr)
colSums(is.na(hr))

The summary function gives quick statistics and flags suspicious values. The is.na function checks for missing data. Boxplots are sensitive to extreme values, so knowing what you are working with is critical.

After running the summary function, the following will appear in your console:

r-summary-output-of-hr-dataset-showing-statistical-distributions

This shows the basic statistics of each column. After running the is.na function, the following will also appear in your console:

r-output-showing-missing-value-counts-per-column-in-hr-dataset

From this output, you can see that only YearsWithCurrManager has 57, meaning that 57 employees don’t have a value for this column.

You can drop this whole column along with the other redundant columns we saw earlier on. You can do this with the code below.

hr <- hr %>% select(-c(EmployeeCount, Over18, StandardHours, YearsWithCurrManager))

To verify if the columns are gone, use this code:

colnames(hr)

Now we need to convert important categorical variables to factors. Doing this tells R that the column has two categories (‘Yes’ and ‘No’), not continuous text.

hr$Attrition <- as.factor(hr$Attrition)
hr$JobRole <- as.factor(hr$JobRole)
hr$Department <- as.factor(hr$Department)

This also ensures ggplot2 treats them correctly when grouping.

How to Use Boxplots

A boxplot displays key features of a dataset. The median is shown by the line in the middle of the box. The interquartile range is represented by the box itself while the whiskers show the spread of the data. Outliers appear as individual points.

Boxplots are mostly useful when you want to compare distributions across groups, such as income by job role or age by attrition status.

Let’s start with a simple boxplot of monthly income.

ggplot(hr, aes(y = MonthlyIncome)) +
  geom_boxplot(fill = "blue") +
  labs(
    title = "Distribution of Monthly Income",
    y = "Monthly Income")

The aes function tells ggplot what variable to plot. geom_boxplot draws the boxplot. The labs function labels parts of the plot drawn, that is the x axis, y axis, and the title.

A-vertical-boxplot-showing-the-distribution-of-employee-monthly-income.

How to Create Boxplots with ggplot2

Now lets compare income across job roles.

ggplot(hr, aes(x = JobRole, y = MonthlyIncome)) +
  geom_boxplot(fill = "lightblue") +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  labs(
    title = "Monthly Income by Job Role",
    x = "Job Role",
    y = "Monthly Income")

The x aesthetic lists all the job roles. The labels are rotated to improve readability. This visualization quickly reveals income differences across roles.

Multiple-boxplots-comparing-monthly-income-distributions-across-different-job-roles.

How to Perform Exploratory Data Analysis (EDA)

Exploratory data analysis involves using visual methods to ask questions and gain a deeper understanding of the data.

We can use the example of Years at company by department.

ggplot(hr, aes(x = Department, y = YearsAtCompany)) +
  geom_boxplot(fill = "darkblue") +
  labs(
    title = "Years at Company by Department",
    y = "Years at Company")

Boxplots-showing-employee-tenure-across-departments.

How to Build Linear Regression Models

To understand how to build linear regression models, you have to model MonthlyIncome using YearsAtCompany with the command below.

The first one creates the model while the second displays it.

hr_lm<- lm(MonthlyIncome ~ YearsAtCompany, data = hr)
summary(hr_lm)

Linear regression estimates how income changes with tenure. This works when the variables are numeric.

After running the code, your console should show you this output:

Call:
lm(formula = MonthlyIncome ~ YearsAtCompany, data = hr)

Residuals:
   Min     1Q Median     3Q    Max 
 -9506  -2488  -1186   1403  15483 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)     3734.47     159.41   23.43   <2e-16 ***
YearsAtCompany   395.25      17.14   23.07   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4032 on 1478 degrees of freedom
Multiple R-squared:  0.2647,    Adjusted R-squared:  0.2642 
F-statistic:   532 on 1 and 1478 DF,  p-value: < 2.2e-16

Let’s interpret this model.

If an employee has 0 years at the company, their base monthly income is $3734.47. This comes from the intercept.

For each year an employee spends at the company, their monthly income is predicted to increase by $395.25.

Both coefficients have p-values < 2e-16. This means they are highly significant. It strongly shows that the years an employee spends at a company affects their income.

The model’s R-squared is 0.2647. This means about 26% of the variation in monthly income is explained by the years an employee spends at the company. This is low, so other factors like role, department, or education likely affect income too.

The model’s F-statistic is 532, with a p-value < 2.2e-16. This means the model is statistically significant overall.

In general, the longer an employee stays at a company, the more they earn, roughly $395 extra per year. But years at the company alone explain only about a quarter of their income. You need to consider other variables for better predictions.

How to Build Logistic Regression Models

You can now learn how to predict attrition. The first command generates the model while the second displays it.

hr_glm<- glm(
  Attrition ~ MonthlyIncome + YearsAtCompany,
  data = hr,
  family = binomial)


summary(hr_glm)

Your console should show this as an output when you run both commands.

Call:
glm(formula = Attrition ~ MonthlyIncome + YearsAtCompany, family = binomial, 
    data = hr)

Coefficients:
                 Estimate Std. Error z value Pr(>|z|)    
(Intercept)    -8.094e-01  1.375e-01  -5.886 3.96e-09 ***
MonthlyIncome  -9.449e-05  2.302e-05  -4.104 4.05e-05 ***
YearsAtCompany -5.047e-02  1.792e-02  -2.817  0.00485 ** 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1305.4  on 1479  degrees of freedom
Residual deviance: 1252.5  on 1477  degrees of freedom
AIC: 1258.5

Number of Fisher Scoring iterations: 5

Logistic regression is used for binary outcomes, that is, yes or no. It estimates probability.

Let’s interpret this logistic regression model. The model predicts whether an employee is likely to leave the company (Attrition) based on their Monthly Income and Years at Company.

The intercept is -0.809. This is the baseline log-odds of leaving when their income and years at the company are zero.

The employees’ Monthly Income has a coefficient of -0.0000945. This means that as their income increases, their chance of leaving decreases slightly. An increase in income makes them less likely to quit.

The employees’ Years at Company have a coefficient of -0.0505. This shows that the longer they stay, the less likely they are to leave. Each additional year reduces their attrition probability.

All coefficients are statistically significant. Monthly Income and Years at Company both strongly affect their likelihood to stay.

The model’s residual deviance is 1252.5, lower than the null deviance of 1305.4. This means the model explains some of the variation in attrition.

The key takeaway is that if an employee earns more and stays longer at the company, they are less likely to leave. These factors matter, but other elements also influence attrition.

Why Visualization Comes Before Modeling

Boxplots help you to:

  • Detect outliers: Boxplots highlight extreme values that interfere with model results.

  • Compare groups: Boxplots allow quick comparison of distributions across different categories.

  • Form hypotheses: Visual patterns assist in identifying relationships worth testing in a model.

  • Validate modeling assumptions: Boxplots help check distribution shape and variance before modeling.

Modeling without visualization often leads to misinterpretation or false confidence.

Conclusion

In this tutorial, you learned how to load and clean data, understand boxplots and their importance. You also learned how to use ggplot2 to compare distributions, perform exploratory data analysis (EDA), build linear and logistic regression models, and link visualization insights to modeling results.

]]> <![CDATA[ How Neural Networks Work – Explained Using the Straight Line Equation y = ax + b ]]> https://www.freecodecamp.org/news/neural-networks-explained-using-y-ax-b/ 695ef4246f1bfe13bf31abe9 Thu, 08 Jan 2026 00:02:44 +0000 Did you know that every data scientist who builds a complex neural network starts with a fundamental question, “How does the output change when the input changes?“

A straight line equation y = ax+b answers it in the simplest way possible. y can increase, decrease, or stay the same when x changes.

On the other hand, a deep neural network tries to answer it in a flexible way. It’s only possible because of multiple layers of straight line calculations stacked one over another along with non linear adjustments to help the network adapt and produce the desired result.

Since a straight line is the essence of neural networks, I think it’s time we try to understand the subtle details of y = ax+b, which I refer to as the magical equation. We’ll also go through the basics of linear regression and classification, which should help you understand the progression of a simple straight line to a complex deep neural network.

Table of Contents

Prerequisites

  • A basic understanding of linear algebra, particularly y=ax+b.

  • General idea about linear regression and classification.

  • Familiarity with the concept of deep neural networks.

y=ax+b

A straight line simply means that output changes steadily as input changes. There are no surprises (that is, no non linearity). Let’s analyze it properly.

y => Output variable
x => Input variable
a => Amount by which y changes when x changes (slope)
b => Value of y when x is 0 (y intercept)

We can take an example and model it in the same form to understand it better.

Ms. Poly is a math teacher who wants to formulate a study plan for her students to excel in an upcoming final exam. For simplicity, she creates a rule of thumb using only one factor: the number of hours studied per week. It has a direct impact on the marks scored by a student.

Before beginning, she makes certain assumptions:

  • Every student is capable of scoring at least 30 without studying.

  • For every hour a student studies, an additional 3 marks can be scored.

She then comes up with the following equation based on her ideas: y = 3x+30

y => Marks scored.
x => Number of hours studied.
a=3 => Increase in marks for every hour studied
b=30 => Minimum marks

Plot of y=3x+30

In the above graph, she plots the points based on the results of the equation. As expected, it is a straight line. If she needs the marks scored for 9 hours of study, she can get it by just substituting x=9 in y=3x+30. Note that the data (x and y) are just based on her hunch and aren’t real.

But Ms. Poly wants to guide her students on how to prepare for the final exam based on actual data. So she conducts a pop quiz and grades it. In order to formulate a study plan, she interviews her students and collects information on how many hours they study math per week. She creates a table with two columns: number of hours studied (x) per week and marks scored (y). She tries her old formula y=3x+30, but it doesn’t seem to work. Thus, she doesn’t have any sensible equation describing the relation between x and y.

Let’s assume that a new student who hasn’t attended any exam (no y available) joins the class the next day, and Ms. Poly only knows the number of hours dedicated per week (x). How can she answer the question below?

If the new student studies for a certain number of hours (x), what can be the marks scored (y) in the exam?

It’s impossible unless there’s an equation defining the sample data. So, her task is to find one that fits the given points. This process is called curve fitting or regression.

Linear Regression

The core idea of linear regression to find a straight line that captures the trend of the existing data to facilitate predictions for new input data. Now, let’s dive straight into the example to understand the concept better.

Ms. Poly is determined to arrive at a solution. She plots the collected data on a graph to get a better picture.

Input Data

She has absolutely no idea how x and y are related. So, she must figure out a formula, by trial and error, that roughly fits the points. She has to start with an intuitive guess, try to improve it in the subsequent steps and then arrive at the best possible solution.

Trial 1: Ms. Poly begins with her previous straight line equation.

y = 3x+30

She substitutes different values of x and plots it alongside the collected input data. This way she can get a clear picture of the differences in her assumption and reality.

Linear Regression-Trial 1

Trial 2: She observes that the line needs a little more slope. This simply means that, in reality, more marks are being scored for every additional hour of study. By changing it from 3 to 4, the equation becomes:

y = 4x+30

The following graph depicts the new line alongside the sample data:

Linear Regression-Trial 2

Trial 3: It looks better but she feels there is a need to shift the whole line upwards. This means that higher marks are being scored even if a student doesn’t dedicate any time for math in a week. She decides to retain the previous slope but changes the starting marks by 10, thus arriving at:

y = 4x+40

Linear Regression-Trial 3

This particular line covers most of the points and can be considered the best possible solution.

Now, if she wishes to ascertain the marks scored by the new student who studied for 3.5 hours, she pins the value inside the formula and calculates the answer: y = 4*(3.5)+40=54

We saw how Ms. Poly arrived at a straight line equation to predict the output for an unknown input. Now she can chalk out a study plan for her class based on the equation.

Here, an expression is formulated to ascertain the change in output when the input changes. It looks like Ms. Poly is thinking like a data scientist. She has in fact modelled a very simple neural network for regression. The equation y=4x+40 can be considered as the only neuron (processing unit) within it. She’s adjusted the parameters a (weight) and b (bias) to arrive at the final formula which covers most of the points (thus minimizing the loss).

Here’s a breakdown of the y = 4x+40 equation:

y => Marks scored.
x => Number of hours studied.
a=4 => Increase in marks for every hour studied
b=40 => Minimum marks

At present, it is a rudimentary neural network which has no layering and non-linearity.

Now let’s shift our attention to a completely different scenario. Ms. Poly, being a teacher, wants to ensure that all her students pass the exam. Assuming, as an end result, she’s not interested in predicting the marks scored. She just wants to know:

If a student studies for a certain number of hours (x), will the student pass/fail(y) the exam?

This leads her to the process of classification.

Linear Classification

The linear classification process uses a simple straight line to divide the data into categories or classes. The line acts as a boundary so that the classes fall on either side of it. First, Ms. Poly defines the boundary condition for pass and fail.

If marks scored>=50, pass

If marks scored<50, fail

According to the data table, x=3 corresponds to y=52 (boundary condition). Therefore she considers x=3 as the classification line***.***

Linear Classification

x=3 seems to segregate the points into the categories properly. She tries to confirm it by substituting another value. Thus, if a student studied for 9 hours, the score would lie towards the right side of x=3. So, they’d pass as per the classification equation.

Again, she’s arrived at an expression to ascertain the change in output when the input changes. But here, she has modelled a basic neural network for classification. The equation x=3 is the only neuron within it. It can be considered to be having two parts as explained below.

  1. Pre-Activation Part: This portion of the neuron computes an intermediate value which is helpful in further processing. She’s figured out the parameters a (weight) and b (bias) to arrive at the following formula: z = x-3

     z => Intermediate Value.
 x => Number of hours studied.
 a=1 => Influence of the number of hours studied on the marks scored
 b=-3 => Minimum number of hours to study to pass the exam = 3

  2. Activation Part: This portion triggers the neuron to make decisions based on a threshold value. The following equation segregates the points into two classes.

     y = 1 (Pass) if z>=0
 y = 0 (Fail) if z<0

This is a very plain neural network which has no layering and non-linearity but has pre-activation and activation parts inside a neuron.

Comparison

We looked at the examples of both linear regression and classification used by Ms. Poly. Regression helps in predicting a value while Classification helps in decision making. Let’s draw a small table to summarize the differences.

Comparison between Linear Regression and Classification

Upon careful observation we notice that both answer the question of how input change affects output.

But at a slightly higher level of complexity than a straight line. Because in the case of both regression and classification, we try to figure out the equation parameters by trial and error.

Here, since the requirements are simple, Ms. Poly just uses a straight line to solve both. A simple linear equation can handle only one steady trend. But in real life, problems that need solving are far more challenging and unpredictable. Some examples are:

Image Classification: An output label is produced based on the input images.

Text Translation: An English sentence can be given as an input to be translated to say, Spanish.

Chatbots: A text prompt is typed in by a user and a meaningful and relevant output is generated.

She probably should have to use a deep neural network if both data and task were complex. That presents another question: How does one build a deep neural network?

We will explore it further by extending the same example to a more realistic version.

Key Additions to Help Build Deep Neural Networks

In the above sections, we noted that Ms. Poly was interested in predicting the exam results of a student using just one factor - number of hours studied. However, in practice, is that one factor sufficient in determining the marks scored or whether the student passes the exam?

No. It’s not enough. She needs to take into account a lot of aspects like:

  • Number of hours studied

  • Number of hours of sleep/rest

  • Burnout due to over-studying

  • Difficulty level of topics in math

  • Pattern of the exam, and so on.

All the above neither act independently nor do they have a simple linear relation with the marks scored. So, she has to solve this problem by stacking the contributing factors one above the other in layers and also adding the element of non linearity. Let’s take a look at each in detail.

Layering

Burnout leads to lower score whereas good sleep increases score. But burnout can be reduced if the student is well rested. So, the impact on the final score when these two factors interact should be taken into account. This is possible only when the system solves it in layers. The first layer can deal with how they independently influence the score, the next layer can explore the interaction between them.

Non-Linearity

If the number of hours studied increases, the score might increase but when burnout overpowers the effect of study hours, the score reduces. The combined effect results in a non-linear graph. There is a rise and then dip in the score based on number of hours studied. It’s evident that the relationship is not straightforward as in a straight line. That’s where it becomes necessary to add non-linearity in the calculations. It helps the system to respond differently according to the conditions, allowing for flexibility in dealing with real world data and conditions.

Thus, Ms. Poly would have to extend the idea of linear regression/classification by including layering and non-linearity to build a fully functional neural network to help build a practical study plan.

Modelling a Deep Neural Network

Ms. Poly should start the work on modelling a deep neural network by following the steps mentioned below:

Step #1 - Define the Problem Clearly

The following factors should be considered before she begins the process of modelling:

  • What are the input features?

  • What are the output features?

  • What type of problem is it (regression/classification)?

Step #2 - Define the Input Layer

The input features form the first layer. There is no computation in this stage. They are represented as:

x1: Number of hours studied
x2: Number of hours of sleep/rest
x3: Burnout due to over-studying
x4: Difficulty level of topics in Maths
x5: Pattern of the exam

Step #3 - Define the First Hidden Layer

This step consists of two parts:

Apply Linear Transformation: The actual learning begins here. A straight line equation is used to understand the combined effect of the inputs. The general formula is z=Wx+b.

z: Intermediate value or Pre-activation
W: Weight matrix which consists of values corresponding to the impact of
each input feature
x: Matrix consisting of input features, [x1, x2, x3, x4, x5]
b: Bias which represents the initial assumptions of the teacher(when x=0)

It looks similar to a linear regression/classification equation. At first W and b are initialized to random values. Then in the subsequent steps, they are adjusted like it was done in earlier examples. We can consider the following combinations assuming we have two neurons in this layer:

Neuron 1: It can focus on study hours, burnout, and rest, with other features contributing less significantly.

Neuron 2: It can emphasize more on the difficulty level of the topic and the exam type compared to other inputs.

It’s important to note that this layer doesn’t calculate the interactions between the features but only on the way different linear combinations work together but independently. To make it clearer, how they contribute independently are added together. We don’t know how one input feature influences the other. For example, we know sleep increases score and burnout reduces score, but what we don’t know at this stage is if sleep reduces burnout, which in turn can influence the final score.

Add Non-Linearity: This step, also called activation, helps in capturing the complexities in different combinations of the features. Less study results in low marks, and too much burnout also results in low marks. It means there is a curve in the score graph which can’t be represented by a linear equation. The activation function is applied to the intermediate value and can be expressed as:

a = g(z)

a: Activation output
g: Activation function
z: Intermediate value or Pre-activation

For example: ReLU is an activation function which outputs z only if z is positive, else 0.

y = ReLU(z)=max(0,z)

We can see that it has no steady slope and is a non-linear activation function. It can suit this scenario as it lets the value pass through to the next layer only if the combined effect of features is greater than 0. Neuron 1 will let it’s output go to the next layer only if the intermediate value (z) that results from study hours, burnout and rest, is large enough to be influencing the final decision, else it’s ignored. There are multiple options for non-linear activation functions that one can choose from.

Step #4 - Stack Layers One Above the Other

This step helps in learning the mutual interactions between the inferences learned from the first hidden layer. The network attempts to understand the intricate details of the influencing factors and build a stable system. It is here that details of whether sleep reduces burnout are figured out. Every layer consists of linear and non linear transformations applied on the input, which are values obtained from the previous layer. Likewise multiple layers can be stacked one over the other based on the requirements. In this example, for representation, we have taken two hidden layers with two neurons each. The number of layers and neurons can vary based on requirements.

Step #5 - Define the Output Feature(s)

This appears to be the final stage in a deep neural network. Ms. Poly can decide what she wants for output: predict the marks scored by a student or predict if the student passes/fails the exam. If she wants the final marks scored, she just has to apply linear transformation in the neuron in the final layer to produce the output. If she wants pass/fail status, she has to apply both linear and non-linear transformations to achieve the desired results.

The diagram below shows an abstract representation of the deep neural network.

Abstract Representation of a Deep Neural Network

The next steps are:

Training the model: The network is trained in the following way:

  • Random weights and biases are assigned to the linear transformation portions of the network.

  • Then the network makes a prediction which is compared with the expected result.

  • If there are gaps between the actual result and the predicted result, corrections are made in weights and biases (this step is similar to what was done in linear regression and classification).

  • The steps above are repeated until the results improve.

Using the model: After the model has been trained, it is capable of yielding results for new input values.

Final Thoughts

In this article, we began with the basics of a straight line equation. Then we gradually navigated through slightly more elaborate concepts like linear regression and classification. They laid the groundwork for delving into the seemingly mysterious deep neural networks. But they are in fact built by stacking layers of linear transformations and non-linear activations, which help understand sophisticated real world patterns.

Despite all the complexities and layers, we can see that the straight line remains the foundation upon which neural networks are built. As we saw earlier, the equation that a deep neural network begins with is our magical equation: y = ax+b.

]]> <![CDATA[ Common Pitfalls to Avoid When Analyzing and Modeling Data ]]> https://www.freecodecamp.org/news/common-pitfalls-to-avoid-when-analyzing-and-modeling-data/ 68ee54b2edcf5de25dd4bb13 Tue, 14 Oct 2025 13:48:34 +0000 Working with data at any level, whether as an analyst, engineer, scientist, or decision-maker, involves going through a range of challenges. Even experienced teams can run into issues that quietly affect the quality of their work. A mislabeled column, an unclear definition, or a data leak that slips by unnoticed can all lead to results that do not hold up when it matters most.

Reliable analysis depends on how data is handled throughout the process. From collection and preparation to modeling and interpretation, each step carries its own risks. Many of the most persistent problems come not from technical gaps, but from missing checks or assumptions that go unspoken.

This guide highlights some of the most common pitfalls in data analysis and shows where they tend to appear. Along the way, it covers:

  • Biased or unclear inputs that cause trouble early on

  • Validation mistakes that distort model performance

  • Misinterpretation of results that leads to the wrong conclusions

  • Workflow gaps that slow teams down or create confusion

  • Practical steps you can take to catch and correct these issues

Table of Contents

Data Collection Pitfalls

A lot of data issues begin before any modeling takes place. The way data is collected helps shape what your analysis can reveal. Once the inputs are biased or inconsistent, even solid techniques may lead to unreliable results.

One common issue is the bias in data sources. When a large portion of the data comes from digital channels like websites or apps, it creates an imbalance. For instance, if a model is trained only on web traffic, it could miss users who engage through offline means, like in-person visits or phone support. This then results in blind spots that limit how well the model performs once deployed.

Inconsistent definitions across systems also pose a major challenge. A simple label like “customer” could represent various things - it could refer to an active user in one database, a prospect in another, or even a past buyer elsewhere. Without shared definitions, one can end up using the same terms to mean very different things, and this leads to confusion and misaligned metrics.

A third issue is the lack of metadata or data provenance. Without clear records of where the data came from or how well it has changed over time, it becomes harder to trace issues, explain outputs, or reproduce results.

The way out:

  • Combine data from multiple sources to build a more complete and representative picture

  • Use stratified sampling to reduce bias where possible

  • Set up regular audits to catch data drift or gaps early

  • Maintain a shared data dictionary and align terms across teams

  • Track data lineage with tools like dbt, Apache Atlas, or OpenMetadata

Getting data collection right sets a strong foundation for analysis and helps prevent issues down the line.

Data Preparation Pitfalls

Once the data has been collected, the next step involves cleaning and shaping it for use. This is another delicate stage where data analysts often encounter an issue. Some choices that seem helpful at first can create problems later, especially when they aren’t documented or tested properly.

Silent Data Leakage

Data leakage occurs when a model learns from information that it would not have access to at prediction time. Let’s say for example, you’re building a model in January to predict whether a customer will make a purchase in February. If your dataset includes transactions from February, and you use that to calculate a feature like “days since last purchase”, then your model is learning from data it wouldn’t realistically have at prediction time.

Improper Handling of Missing Values

Quite a number of data explorers think missing values are just gaps to be filled. In certain cases, the fact that data is missing can be just as meaningful as the value itself. In a customer churn dataset, some users might have blank entries for recent activities because they have already stopped engaging with the product. Filling those gaps with averages and zeros without context could make the model treat them the same as users who simply haven’t generated enough data yet, which can be misleading.

Over-aggressive Outlier Removal

It’s tempting to remove extreme values to simplify modeling, but outliers often represent, although rare, yet important events. In fraud detection, for instance, the anomalies are the very signals the models need to learn from. Discarding them automatically based on z-scores or quantiles may improve the short-term accuracy while weakening long-term reliability.

The way out

  • To avoid data leakage, create training and test splits before engineering features. Make use of chronological splits when modeling time-based behavior, and regularly audit feature logic.

  • For missing values, go through the missingness patterns first. Use indicator variables where necessary, and treat the missingness as a signal, rather than just a defect.

  • With outliers, analyze their sources before removing them. If they are recognized, try using robust models that can handle skewed data or flag them for downstream use instead of deleting them.

Getting this stage right protects your models from brittle and unstable behavior.

Modeling and Validation Pitfalls

A common thought in this field is that models are only as reliable as the assumptions built into them. Mistakes at this phase are often reflected late, sometimes after the models have been deployed, making them harder to catch and more expensive to fix.

Overfitting Through Hyperparameter Tuning

Trying to make a model perfect with the training data can lead to patterns that don’t hold up in practice. When one tests hundreds of hyperparameter combinations without proper checks, the model often ends up learning noise rather than signals in the data, thereby resulting in excellent scores during cross-validation but weak performance in production. For instance, a churn model might show an excellent performance during development, but once it is deployed to a new region with a slight difference in customer behavior, it then starts to miss the mark.

Validation Leakage

Leakage can occur when the validation process accidentally gives the model access to target-related information. One common case is target encoding, where features like average purchase per customer group are calculated on the full dataset rather than only on the training set. This can lead to inflated validation scores and a false sense of confidence.

Ignoring Data Drift and Concept Drift

Data changes over time, and so do the basic relationships that models rely on. A model trained on behavior from eight months ago may not reflect current realities. Imagine a fraud detection model built before a major policy shift or change of product; the possibility that the model may fail to catch new fraud patterns that arise afterwards is extremely high.

The Way Out

  • Use nested cross-validation (a technique that separates hyperparameter tuning from final evaluation by using two loops of cross-validation) to avoid overfitting during the model selection. After this, you can then compare results against simple baselines to keep complexity in check.

  • Treat feature engineering as part of the pipeline and apply it within each training fold to avoid leakage. For time-sensitive data, validate progressively to reflect real-world use.

  • Check for drift using techniques like the Kolmogorov-Smirnov test or the Population Stability Index, and link alerts to retraining processes so models can evolve with data.

These steps go a long way in keeping your models solid in production and ready for whatever the data throws at them.

Interpretation and Communication Pitfalls

Clear, responsible communication is just as important as accurate modeling. But it is very easy to slip into habits that make results look more certain, more compelling, more reliable than they really are. These missteps can lead teams to act on insights that don’t hold up.

Overconfidence in Statistical Significance

Testing lots of variables without making adjustments can make weak signals look important. Imagine you run a dozen A/B tests and pick the one with a p-value below 0.05. Without correcting for multiple comparisons, there’s a good chance that result is just noise.

Ignoring Practical Significance

A result can be significant statistically but still meaningless when viewed in context. For example, finding a 0.1% lift in clickthrough rate, which is technically real but not worth the cost of rolling out a change across the product.

Model Explainability Missteps

When explanation tools are used without context, they can confuse rather than clarify. Showing a ranked list of SHAP values might look impressive, but if the stakeholders don’t understand what the features mean or how they interact, the takeaway is lost.

The Way Out

  • Be cautious with statistical significance. If you’re running several tests, apply corrections for multiple comparisons (Bonferroni or Benjamini-Hochberg methods, for instance) and avoid selectively reporting only the findings that look significant and ignoring those that don’t.

  • Look beyond what is statistically true and ask whether it is practically useful. A small, significant change might not be worth acting on at the end of the day.

  • When using explainability tools like SHAP or LIME, don’t assume the outputs speak for themselves. Add plain-language summaries, relevant examples, and business contexts to make them actionable. It is better to explain less with clarity than more with confusion.

These habits make your results easier to trust, interpret, and apply, which is ultimately the point of the work.

Organizational and Workflow Pitfalls

A major fact is that analytics is most effective when it is collaborative and responsive. Gaps in team structure or feedback processes can slow progress and limit the value of your work.

Teams working in isolation are a frequent issue. When analysts, engineers, and business stakeholders do not share tools or goals, efforts get duplicated and insights become fragmented. For example, one team might define active users based on weekly logins, while another uses monthly engagements, resulting in mismatched reports.

Lack of feedback from deployed models is another pitfall. If no one tracks what happens after predictions are made, teams miss the opportunity to refine and improve their processes. Imagine if a loan approval model is deployed, but there’s no follow-up on repayment behavior, it becomes difficult to tell whether the model is supporting sound lending decisions or increasing default risk.

The way out

  • Encourage collaboration by forming cross-functional teams and coordinating around shared planning cycles. Align on definitions early and rely on centralized dashboards to ensure that everyone is working from the same source of truth.

  • Create feedback loops and make them a standard part of your workflow, Track real-world outcomes, and schedule regular post-deployment reviews to understand what is working and what is not.

  • Include end users alongside data teams and treat their input as essential to improving the system.

Taking these actions helps analytics stay practical, consistent, and responsive to real needs.

Conclusion

Each stage of the data workflow benefits from clarity, structure, and shared understanding. The table below shows all the mentioned pitfalls, together with the way out to help teams build more reliable models and deliver results that hold up in real-world settings.

Category

Pitfall

Consequences

Recommended Approach

Data collection

Unreliable sources

Skewed insights

Validate source quality and apply consistent standards

Data preparation

Silent data leakage

Inflated model performance without real-world value

Use proper data splits and audit derived features

Modeling & validation

Overfitting through hyperparameter tuning

Strong validation results that don’t translate to reality

Use nested cross-validation (a structure where tuning happens inside training folds) and keep simple baselines for comparison

Interpretation & communication

Overconfidence in statistical significance

Misleading conclusions from small or selective effects

Adjust for multiple comparisons and report confidence intervals alongside p-values

Organizational & workflow

Fragmented teams

Redundant work and inconsistent metrics

Encourage collaboration with shared planning, dashboards, and definitions

Strong analytic practice is built over time. Keeping these pitfalls in view helps teams stay consistent, improve delivery, and create results that stay useful across projects and contexts.

]]> <![CDATA[ How to Forecast Time Series Data with Python Darts ]]> https://www.freecodecamp.org/news/how-to-forecast-time-series-data-with-python-darts/ 68e40c4dd441014d7e52dc0d Mon, 06 Oct 2025 18:37:01 +0000 When analyzing time series data, your main objective is to consider the period during which the data is collected and how your variable of interest changes over time.

There are various libraries for time series forecasting in Python, and Darts is one of them. Unlike other forecasting libraries, Darts is a high-level forecasting library with algorithms to handle various time series data, regardless of the kind of trend they portray.

This tutorial will walk you through how you can forecast time series data using Python Darts. This will help you make meaningful insights whenever you come across time series data such as stock prices, weather measurements, and so on.

Here’s what we’ll cover:

What is Python Darts?

Python Darts is an open-source library for time series analysis and forecasting. It has various models ranging from statistical time series models like ARIMA, and SARIMA, to machine learning and deep learning models like Prophet, and LSTM.

It has various algorithms for handling missing imputations in time series data, and can handle time series problems ranging from univariate, multivariate to hierarchical time series.

Prerequisites

Before we proceed, you will need to have the following:

  • Python 3.9+ installed.

  • Jupyter Notebook, Google Colab, or Positron to run your code.

  • Download the Netflix stock data.

  • Have the following libraries installed:

    • darts for time series analysis

    • pandas for data wrangling

    • matplotlib for data visualization.

How to Set Up Dependencies

Load the following libraries.

import matplotlib.pyplot as plt
import pandas as pd
import darts
from darts import TimeSeries
from darts.models import ARIMA
from darts.models import RegressionModel
from lightgbm import LGBMRegressor
from darts.models import RNNModel
from darts.metrics import mape
import itertools

Understanding the Dataset

The Netflix stock data contains historical daily prices of Netflix stock from the year 2002 till date.

Load the data and have a preview of it.

netflix = pd.read_csv("/kaggle/input/netflix-stock-data-live-and-latest/Netflix_stock_history.csv")
netflix['Date'] = pd.to_datetime(netflix['Date'], utc=True).dt.tz_convert(None)
netflix.head()

Image showing the first 5 rows of the Netflix stock data

To forecast a time series data, we need a Date column, which we already have, and then the variable of interest. We have several variables, but for this tutorial, we will focus on the Close variable of Netflix stocks.

Let’s visualize the data to see how Netflix closing price performed over the years.

netflix.plot(x='Date', y='Close', figsize=(10,5))
plt.show()

Image showing a line chart of Netflix stock data from 2000 to date

From the chart above, you can see that Netflix stock showed exponential growth in recent years. This means that the data is non-stationary, implying that there are no consistent changes over time.

There are a lot of random fluctuations in the data, which might make it difficult to forecast. Such data usually requires advanced models to handle the various fluctuations or noise present in the data.

How to Prepare the Data for Darts

Before preparing the data for Darts, you need to take note of few things.

First of all, if you look at our data preview earlier on, you would notice that it is recorded daily, we also need to fill in missing dates.

Copy and paste this code into your notebook.

start = netflix['Date'].min()
end = netflix['Date'].max()

netflix = (
    netflix.set_index('Date')
           .reindex(pd.date_range(start=start, end=end, freq='D'))
           .ffill()
           .reset_index()
           .rename(columns={'index': 'Date'})
)
netflix.head()

The code above ensures the netflix dataset has a continuous daily time series by filling in missing dates.

First, it finds the earliest start and latest end dates in the data, then creates a full daily date range between them.

By setting the Date column as the index and using .reindex() method, it inserts rows for any missing dates, which initially contain NaN.

The .ffill() method (forward fill) replaces these gaps by carrying forward the last known value, which is common for stock data when markets are closed, such as weekends.

Finally, the index is reset, and the column is renamed back to Date, producing a clean, continuous dataset ready for time series analysis.

Next, we need to convert the data to a Darts Timeseries object to make it usable by the Darts library.

 = TimeSeries.from_dataframe(
    netflix,
    time_col='Date',
    value_cols='Close',
)

The code above converts the netflix DataFrame into a Darts TimeSeries object, which is optimized for time series modeling and forecasting.

It takes the Date column (time_col='Date') as the timeline and the Close column (value_cols='Close') as the target values to forecast.

The resulting series object is now structured for use with Darts’ advanced forecasting models like ARIMA, Prophet, RNNs, and other time series algorithms.

Just like you would with any other machine learning model, you need to split your data into a training set and a validation set.

train, val = series.split_before(0.8)

How to Build a Forecasting Model

When building a forecasting model, you have the privilege of trying various models and picking the best-performing one.

The Darts library has various algorithms for time series analysis, from popular statistical algorithms like the Auto Regressive Integrated Moving Average (ARIMA) and Moving Average (MA) models, to machine learning and deep learning algorithms like Prophet and Long Short Term Memory (LSTM).

Note, I will only demonstrate how these algorithms work - it’s not necessary that we get accurate model metrics. But with further feature engineering, hyperparameter tuning, and cross-validation, you can get good results on your own.

Classical Model

The classical mode is the use of statistical time series models such as ARIMA. ARIMA is made up of the following components:

  • AR (AutoRegressive): Predict past values by looking at previous ones.

  • I (Integrated): Remove trends by focusing on changes instead of raw values.

  • MA (Moving Average): Learn from the errors of past predictions to improve accuracy.

Run the code below in your notebook to fit an ARIMA model.

arima_model = ARIMA()
arima_model.fit(train)
arima_forecast = arima_model.predict(len(val))

To visualize the forecast by the model, call the .plot() method on the forecast object.

series.plot(label='actual')
arima_forecast.plot(label='forecast')
plt.legend()

Image showing the ARIMA model forecast of netflix stock

You can improve the model by adding some additional parameters to the ARIMA() class. You can read more about that in the Darts documentation.

Machine Learning Models

Classical models like ARIMA can’t handle non-linear data. Machine learning models fill this gap. We’ll use the LightGBM model as an example.

The LightGBM is a machine learning model that builds models sequentially based on decision trees. It adds new decision trees that correct the errors of previous trees.

Although it was not designed to handle time series, with some feature engineering such as lags, rolling statistics, and seasonal indicators, you can make it learn patterns from time series data.

Run this code on your notebook to fit a LightGBM model on the Netflix data.

lgbm = LGBMRegressor()
lgbm_model = RegressionModel(lags=12, model=lgbm)
lgbm_model.fit(train)
lgbm_forecast = lgbm_model.predict(len(val))

From the code above, the lag argument is set to 12, which is the value of the Netflix stock price for 12 days before a selected day.

Let’s have a view of the forecast by running the following code.

series.plot(label='actual')
lgbm_forecast.plot(label='forecast')
plt.legend()

Image showing the LightGBM model forecast of netflix stock

You can read more about tuning the LightGBM model from the Darts documentation to improve the above model.

How to Forecast with Deep Learning models

You can go for deep learning models designed for time series, such as LSTM, a kind of Recurrent Neural Network (RNN) designed to capture long-term dependencies in sequential data.

Run the following code to build the LSTM model.

lstm_model = RNNModel(model='LSTM', input_chunk_length=12, output_chunk_length=6, n_epochs=100)
lstm_model.fit(train)
lstm_forecast = rnn_model.predict(len(val))

Now let’s visualize the forecast and see what we have.

series.plot(label='actual')
lstm_forecast.plot(label='forecast')
plt.legend()

Image showing the LSTM model forecast of Netflix stock

You can look up the Darts documentation to improve the model and check out other deep learning models also.

Model Evaluation

Now that you have three models, you need to select the best one among them using the Mean Absolute Percentage Error (MAPE).

It expresses the average absolute error as a percentage of the actual values, and the closer your value is to 0, the better your model.

Run the following to print the MAPE of each respective model.

arima_error = mape(val, arima_forecast)
print("MAPE:", arima_error)
lgbm_error = mape(val, lgbm_forecast)
print("MAPE:", lgbm_error)
lstm_error = mape(val, lstm_forecast)
print("MAPE:", lstm_error)

> MAPE: 38.33262525601514
> MAPE: 39.00241495209449
> MAPE: 38.82910057097827

The model with the lowest MAPE is the ARIMA model with approximately 38.33, which means it’s our best-performing model.

BackTesting

Darts has a feature called backtesting that allows you to evaluate your models based on historical data, using a rolling forecast.

Backtesting is like a time machine for forecasting. It simulates how your model would have performed in the past by repeatedly training it on historical data up to a certain point, making a prediction for the next step, then moving forward, and repeating the process.

This rolling evaluation simulates how the model would behave in real-world conditions, where future data is unknown, helping you measure its consistency and reliability over time, instead of just testing it once on a single validation set.

Since the ARIMA model is currently our best-performing model, run the code below to implement backtesting.


# Perform backtesting on the training + validation series
backtest_series = train.concatenate(val)

# Backtest
backtest_forecast = arima_model.historical_forecasts(
    series=backtest_series,
    start=0.8,          # fraction of the series to start forecasting from
    forecast_horizon=len(val),
    stride=1,           # step size of rolling forecast
    retrain=True,       # retrain the model at each step
    verbose=True
)

# Compute metrics
error = mape(backtest_series[-len(val):], backtest_forecast)
print(f"MAPE: {error:.2f}%")

> historical forecasts: 100%|██████████| 1/1 [00:02<00:00,  2.69s/it]MAPE: 47.27%

In the code above,

  • The start argument defines where to start backtesting, which in this case is the last 20% series of the data.

  • The forecast_horizon is how many steps ahead to forecast at each point.

  • The stride is how frequently to retrain/forecast.

  • The retrain=True refits the model at each step for realistic evaluation.

You can see that the MAPE, after backtesting, is higher because backtesting is more realistic, and it is more difficult to achieve a lower MAPE.

On your own, you can try to replicate backtesting for the other models.

Hyper Parameter Tuning

The ARIMA model has three hyperparameter:

  • p which is the AR order

  • d which is the differencing order

  • q which is the MA order

You can use either grid or random search to tune your ARIMA model in Darts.

# Define possible values
p_values = range(0, 4)
d_values = range(0, 3)
q_values = range(0, 4)

best_mape = float('inf')
best_params = None

for p, d, q in itertools.product(p_values, d_values, q_values):
    try:
        arima_model = ARIMA(p=p, d=d, q=q)
        arima_model.fit(train)
        arima_forecast = arima_model.predict(len(val))
        arima_error = mape(val, arima_forecast)
        if arima_error < best_mape:
            best_mape = arima_error
            best_params = (p, d, q)
    except Exception as e:
        # Some combinations may fail
        continue

print(f"Best ARIMA params: p={best_params[0]}, d={best_params[1]}, q={best_params[2]} with MAPE={best_mape:.2f}%")

> Best ARIMA params: p=2, d=0, q=3 with MAPE=35.95%

In the above code, you define a range of possible values for the p, d , and q components, iterating over each combination of those values and choosing the model with the best MAPE among them.

Note that each model has its specific parameter you would have to tune, and you will need to check the Darts documentation for the hyperparameters of other models.

Real-World Use Cases

Forecasting time series data has a lot of real-world applications, some of which are:

  • Stock price prediction: Like the dataset used in this tutorial, forecasting is used in finance for stock price prediction, allowing investors to manage risk.

  • Demand forecasting for inventory: As a store owner, you can forecast product demands based on past sales of a product. This lets you know products that are in high demand.

  • Energy consumption prediction: Governments, industries, and consumers can plan and manage energy production, distribution, and consumption efficiently, based on data from past usage. This helps to avoid blackouts and wastage, enabling them to prepare ahead.

Best Practices

  • Always visualize residuals: Residuals are the difference between forecasted values and actual values. You must visualize them to detect outliers and unusual events.

  • Perform proper backtesting: Backtesting lets you see a more realistic model, subjected to various changes that can occur in real life. When you backtest all your models, you end up getting a model that performs well when forecasting.

  • Avoid data leakage: Do not train your models on validation sets to avoid bias, and always use cross-validation where necessary.

  • Use domain knowledge for feature engineering: Ensure you understand the data you are working with. This comes in handy in feature engineering, when you want to come up with new features to help your forecasting model, especially in multivariate time series forecasting.

Conclusion

This tutorial is more like an overview, especially if you are new to time series, but you can build a lot just from what you have learned.

You already have an idea of what time series and forecasting are, and how you can use the Darts Python library to achieve that.

You also learned of various models for forecasting time series data, and how you can apply techniques such as backtesting and hyperparameter tuning to achieve better results.

Another interesting thing with Darts is its ability to handle hierarchical time series. Here, data is structured at aggregated levels.

Darts is one of the most powerful time series libraries in Python and has a lot of models to handle various cases. You can proceed to explore models such as Transformers and also multi-series forecasting, which are used for special use cases.

If you are interested in more data science and statistics articles, don’t forget to check out my blog.

]]> <![CDATA[ Graph Algorithms in Python: BFS, DFS, and Beyond ]]> https://www.freecodecamp.org/news/graph-algorithms-in-python-bfs-dfs-and-beyond/ 68b86be0956e509211153b48 Wed, 03 Sep 2025 16:25:04 +0000 Have you ever wondered how Google Maps finds the fastest route or how Netflix recommends what to watch? Graph algorithms are behind these decisions.

Graphs, made up of nodes (points) and edges (connections), are one of the most powerful data structures in computer science. They help model relationships efficiently, from social networks to transportation systems.

In this guide, we will explore two core traversal techniques: Breadth-First Search (BFS) and Depth-First Search (DFS). Moving on from there, we will cover advanced algorithms like Dijkstra’s, A*, Kruskal’s, Prim’s, and Bellman-Ford.

Table of Contents:

  1. Understanding Graphs in Python

  2. Ways to Represent Graphs in Python

  3. Breadth-First Search (BFS)

  4. Depth-First Search (DFS)

  5. Dijkstra’s Algorithm

  6. A* Search

  7. Kruskal’s Algorithm

  8. Prim’s Algorithm

  9. Bellman-Ford Algorithm

  10. Optimizing Graph Algorithms in Python

  11. Key Takeaways

Understanding Graphs in Python

A graph consists of nodes (vertices) and edges (relationships).

For examples, in a social network, people are nodes and friendships are edges. Or in a roadmap, cities are nodes and roads are edges.

There are a few different types of graphs:

  • Directed: edges have direction (one-way streets, task scheduling).

  • Undirected: edges go both ways (mutual friendships).

  • Weighted: edges have values (distances, costs).

  • Unweighted: edges are equal (basic subway routes).

Now that you know what graphs are, let’s look at the different ways they can be represented in Python.

Ways to Represent Graphs in Python

Before diving into traversal and pathfinding, it’s important to know how graphs can be represented. Different problems call for different representations.

Adjacency Matrix

An adjacency matrix is a 2D array where each cell (i, j) shows whether there is an edge from node i to node j.

  • In an unweighted graph, 0 means no edge, and 1 means an edge exists.

  • In a weighted graph, the cell holds the edge weight.

This makes it very quick to check if two nodes are directly connected (constant-time lookup), but it uses more memory for large graphs.

graph = [
    [0, 1, 1],
    [1, 0, 1],
    [1, 1, 0]
]

Here, the matrix shows a fully connected graph of 3 nodes. For example, graph[0][1] = 1 means there is an edge from node 0 to node 1.

Adjacency List

An adjacency list represents each node along with the list of nodes it connects to.

This is usually more efficient for sparse graphs (where not every node is connected to every other node). It saves memory because only actual edges are stored instead of an entire grid.

graph = {
    'A': ['B','C'],
    'B': ['A','C'],
    'C': ['A','B']
}

Here, node A connects to B and C, and so on. Checking connections takes a little longer than with a matrix, but for large, sparse graphs, it’s the better option.

Using NetworkX

When working on real-world applications, writing your own adjacency lists and matrices can get tedious. That’s where NetworkX comes in, a Python library that simplifies graph creation and analysis.

With just a few lines of code, you can build graphs, visualize them, and run advanced algorithms without reinventing the wheel.

import networkx as nx
import matplotlib.pyplot as plt

G = nx.Graph()
G.add_edges_from([('A','B'), ('A','C'), ('B','C')])
nx.draw(G, with_labels=True)
plt.show()

This builds a triangle-shaped graph with nodes A, B, and C. NetworkX also lets you easily run algorithms like shortest paths or spanning trees without manually coding them.

Now that we’ve seen different ways to represent graphs, let’s move on to traversal methods, starting with Breadth-First Search (BFS).

Breadth-First Search (BFS)

The basic idea behind BFS is to explore a graph one layer at a time. It looks at all the neighbors of a starting node before moving on to the next level. A queue is used to keep track of what comes next.

BFS is particularly useful for:

  • Finding the shortest path in unweighted graphs

  • Detecting connected components

  • Crawling web pages

Here’s an example:

from collections import deque

def bfs(graph, start):
    visited = {start}
    queue = deque([start])

    while queue:
        node = queue.popleft()
        print(node, end=" ")
        for neighbor in graph[node]:
            if neighbor not in visited:
                visited.add(neighbor)
                queue.append(neighbor)


graph = {
    'A': ['B','C'],
    'B': ['A','D','E'],
    'C': ['A','F'],
    'D': ['B'],
    'E': ['B','F'],
    'F': ['C','E']
}

bfs(graph, 'A')

Here’s what’s going on in this code:

  • graph is a dict where each node maps to a list of neighbors.

  • deque is used as a FIFO queue so we visit nodes level-by-level.

  • visited keeps track of nodes we’ve already processed so we don’t loop forever on cycles.

  • In the loop, we pop a node, print it, then for each unvisited neighbor, we mark it visited and enqueue it.

And here’s the output:

A B C D E F

Now that we have seen how BFS works, let’s turn to its counterpart: Depth-First Search (DFS).

Depth-First Search (DFS)

DFS works differently from BFS. Instead of moving level by level, it follows one path as far as it can go before backtracking. Think of it as diving deep down a trail, then returning to explore the others.

We can implement DFS in two ways:

  • Recursive DFS, which uses the function call stack

  • Iterative DFS, which uses an explicit stack

DFS is especially useful for:

  • Cycle detection

  • Maze solving and puzzles

  • Topological sorting

Here’s an example of recursive DFS:

def dfs_recursive(graph, node, visited=None):
    if visited is None:
        visited = set()
    if node not in visited:
        print(node, end=" ")
        visited.add(node)
        for neighbor in graph[node]:
            dfs_recursive(graph, neighbor, visited)

graph = {
    'A': ['B','C'],
    'B': ['A','D','E'],
    'C': ['A','F'],
    'D': ['B'],
    'E': ['B','F'],
    'F': ['C','E']
}

dfs_recursive(graph, 'A')

  • visited is a set that tracks nodes already processed so you don’t loop forever on cycles.

  • On each call, if node hasn’t been seen, it’s printed, marked visited, then the function recurses into each neighbor.

Traversal order:

A B D E F C

Explanation: DFS visits B after A, goes deeper into D, then backtracks to explore E and F, and finally visits C.

And here’s an example of iterative DFS:

def dfs_iterative(graph, start):
    visited = set()
    stack = [start]

    while stack:
        node = stack.pop()
        if node not in visited:
            print(node, end=" ")
            visited.add(node)
            stack.extend(reversed(graph[node]))

dfs_iterative(graph, 'A')

  • visited tracks nodes you’ve already processed so you don’t loop on cycles.

  • stack is LIFO (last in, first out) – you pop() the top node, process it, then push its neighbors.

  • reversed(graph[node]) pushes neighbors in reverse so they’re visited in the original left-to-right order (mimicking the usual recursive DFS).

Here’s the output:

A B D E F C

With BFS and DFS explained, we can now move on to algorithms that solve more complex problems, starting with Dijkstra’s shortest path algorithm.

Dijkstra’s Algorithm

Dijkstra’s algorithm is built on a simple rule: always visit the node with the smallest known distance first. By repeating this, it uncovers the shortest path from a starting node to all others in a weighted graph that doesn’t have negative edges.

import heapq

def dijkstra(graph, start):
    heap = [(0, start)]
    shortest_path = {node: float('inf') for node in graph}
    shortest_path[start] = 0

    while heap:
        cost, node = heapq.heappop(heap)
        for neighbor, weight in graph[node]:
            new_cost = cost + weight
            if new_cost < shortest_path[neighbor]:
                shortest_path[neighbor] = new_cost
                heapq.heappush(heap, (new_cost, neighbor))
    return shortest_path

graph = {
    'A': [('B',1), ('C',4)],
    'B': [('A',1), ('C',2), ('D',5)],
    'C': [('A',4), ('B',2), ('D',1)],
    'D': [('B',5), ('C',1)]
}

print(dijkstra(graph, 'A'))

Here’s what’s going on in this code:

  • graph is an adjacency list: each node maps to a list of (neighbor, weight) pairs.

  • shortest_path stores the current best-known distance to each node (∞ initially, 0 for start).

  • heap (priority queue) holds frontier nodes as (cost, node), always popping the smallest cost first.

  • For each popped node, it relaxes its edges: for each (neighbor, weight), compute new_cost. If new_cost beats shortest_path[neighbor], update it and push the neighbor with that cost.

And here’s the output:

{'A': 0, 'B': 1, 'C': 3, 'D': 4}

Moving on, let’s look at an extension of this algorithm: A Search.*

A* works like Dijkstra’s but adds a heuristic function that estimates how close a node is to the goal. This makes it more efficient by guiding the search in the right direction.

import heapq

def heuristic(node, goal):
    heuristics = {'A': 4, 'B': 2, 'C': 1, 'D': 0}
    return heuristics.get(node, 0)

def a_star(graph, start, goal):
    g_costs = {node: float('inf') for node in graph}
    g_costs[start] = 0
    came_from = {}

    heap = [(heuristic(start, goal), start)]

    while heap:
        f, node = heapq.heappop(heap)

        if f > g_costs[node] + heuristic(node, goal):
            continue

        if node == goal:
            path = [node]
            while node in came_from:
                node = came_from[node]
                path.append(node)
            return path[::-1], g_costs[path[0]]

        for neighbor, weight in graph[node]:
            new_g = g_costs[node] + weight
            if new_g < g_costs[neighbor]:
                g_costs[neighbor] = new_g
                came_from[neighbor] = node
                heapq.heappush(heap, (new_g + heuristic(neighbor, goal), neighbor))

    return None, float('inf')

graph = {
    'A': [('B',1), ('C',4)],
    'B': [('A',1), ('C',2), ('D',5)],
    'C': [('A',4), ('B',2), ('D',1)],
    'D': []
}

print(a_star(graph, 'A', 'D'))

This one’s a little more complex, so here’s what’s going on:

  • graph: adjacency list – each node maps to [(neighbor, weight), ...].

  • heuristic(node, goal): returns an estimate h(node) (lower is better). It’s passed goal but in this demo uses a fixed dict.

  • g_costs: best known cost from start to each node (∞ initially, 0 for start).

  • heap: min-heap of (priority, node) where priority = g + h.

  • came_from: backpointers to reconstruct the path once we pop the goal.

Then in the main loop:

  • We pop the node with smallest priority.

  • If it’s the goal, we backtrack via came_from to build the path and return it with g_costs[goal].

  • Otherwise, we relax the edges: for each (neighbor, weight), compute new_cost = g_costs[node] + weight. If new_cost improves g_costs[neighbor], update it, set came_from[neighbor] = node, and push (new_cost + heuristic(neighbor, goal), neighbor).

Output:

(['A', 'B', 'C', 'D'], 4)

Next up, let’s move from shortest paths to spanning trees. This is where Kruskal’s algorithm comes in.

Kruskal’s Algorithm

Kruskal’s algorithm builds a Minimum Spanning Tree (MST) by sorting all edges from smallest to largest and adding them one at a time, as long as they don’t create a cycle. This makes it a greedy algorithm as it always picks the cheapest option available at each step.

The implementation uses a Disjoint Set (Union-Find) data structure to efficiently check whether adding an edge would create a cycle. Each node starts in its own set, and as edges are added, sets are merged.

class DisjointSet:
    def __init__(self, nodes):
        self.parent = {node: node for node in nodes}
        self.rank = {node: 0 for node in nodes}
    def find(self, node):
        if self.parent[node] != node:
            self.parent[node] = self.find(self.parent[node])
        return self.parent[node]
    def union(self, node1, node2):
        r1, r2 = self.find(node1), self.find(node2)
        if r1 != r2:
            if self.rank[r1] > self.rank[r2]:
                self.parent[r2] = r1
            else:
                self.parent[r1] = r2
                if self.rank[r1] == self.rank[r2]:
                    self.rank[r2] += 1

def kruskal(graph):
    edges = sorted(graph, key=lambda x: x[2])
    mst, ds = [], DisjointSet({u for e in graph for u in e[:2]})
    for u,v,w in edges:
        if ds.find(u) != ds.find(v):
            ds.union(u,v)
            mst.append((u,v,w))
    return mst

graph = [('A','B',1), ('A','C',4), ('B','C',2), ('B','D',5), ('C','D',1)]
print(kruskal(graph))

Output:

[('A','B',1), ('C','D',1), ('B','C',2)]

Here, the MST includes the smallest edges that connect all nodes without forming cycles. Now that we have seen Kruskal’s, we can move further to analyze another algorithm.

Prim’s Algorithm

Prim’s algorithm also finds an MST, but it grows the tree step by step. It starts with one node and repeatedly adds the smallest edge that connects the current tree to a new node. Think of it as expanding a connected “island” until all nodes are included.

This implementation uses a priority queue (heapq) to always select the smallest available edge efficiently.

import heapq

def prim(graph, start):
    mst, visited = [], {start}
    edges = [(w, start, n) for n,w in graph[start]]
    heapq.heapify(edges)

    while edges:
        w,u,v = heapq.heappop(edges)
        if v not in visited:
            visited.add(v)
            mst.append((u,v,w))
            for n,w in graph[v]:
                if n not in visited:
                    heapq.heappush(edges, (w,v,n))
    return mst

graph = {
    'A':[('B',1),('C',4)],
    'B':[('A',1),('C',2),('D',5)],
    'C':[('A',4),('B',2),('D',1)],
    'D':[('B',5),('C',1)]
}
print(prim(graph,'A'))

Output:

[('A','B',1), ('B','C',2), ('C','D',1)]

Notice how the algorithm gradually expands from node A, always picking the lowest-weight edge that connects a new node.

Let’s now look at an algorithm that can handle graphs with negative edges: Bellman-Ford.

Bellman-Ford Algorithm

Bellman-Ford is a shortest path algorithm that can handle negative edge weights, unlike Dijkstra’s. It works by relaxing all edges repeatedly: if the current path to a node can be improved by going through another node, it updates the distance. After V-1 iterations (where V is the number of vertices), all shortest paths are guaranteed to be found.

This makes it slightly slower than Dijkstra’s but more versatile. It can also detect negative weight cycles by checking for further improvements after the main loop.

def bellman_ford(graph, start):
    dist = {node: float('inf') for node in graph}
    dist[start] = 0
    for _ in range(len(graph)-1):
        for u in graph:
            for v,w in graph[u]:
                if dist[u] + w < dist[v]:
                    dist[v] = dist[u] + w
    return dist

graph = {
    'A':[('B',4),('C',2)],
    'B':[('C',-1),('D',2)],
    'C':[('D',3)],
    'D':[]
}
print(bellman_ford(graph,'A'))

Output:

{'A': 0, 'B': 4, 'C': 2, 'D': 5}

Here, the shortest path to each node is found, even though there’s a negative edge (B → C with weight -1). If there had been a negative cycle, Bellman-Ford would detect it by noticing that distances keep improving after V-1 iterations.

With the main algorithms explained, let’s move on to some practical tips for making these implementations more efficient in Python.

Optimizing Graph Algorithms in Python

When graphs get bigger, little tweaks in how you write your code can make a big difference. Here are a few simple but powerful tricks to keep things running smoothly.

1. Use deque for BFS
If you use a regular Python list as a queue, popping items from the front takes longer the bigger the list gets. With collections.deque, you get instant (O(1)) pops from both ends. It’s basically built for this kind of job.

from collections import deque

queue = deque([start])  # fast pops and appends

2. Go Iterative with DFS
Recursive DFS looks neat, but Python doesn’t like going too deep – you’ll hit a recursion limit if your graph is very large. The fix? Write DFS in an iterative style with a stack. Same idea, no recursion errors.

def dfs_iterative(graph, start):
    visited, stack = set(), [start]
    while stack:
        node = stack.pop()
        if node not in visited:
            visited.add(node)
            stack.extend(graph[node])

3. Let NetworkX Do the Heavy Lifting
For practice and learning, writing your own graph code is great. But if you’re working on a real-world problem – say analyzing a social network or planning routes – the NetworkX library saves tons of time. It comes with optimized versions of almost every common graph algorithm plus nice visualization tools.

import networkx as nx

G = nx.Graph()
G.add_edges_from([('A','B'), ('A','C'), ('B','D'), ('C','D')])

print(nx.shortest_path(G, source='A', target='D'))

Output:

['A', 'B', 'D']

Instead of worrying about queues and stacks, you can let NetworkX handle the details and focus on what the results mean.

Key Takeaways

  • An adjacency matrix is fast for lookups but is memory-heavy.

  • An adjacency list is space-efficient for sparse graphs.

  • NetworkX makes graph analysis much easier for real-world projects.

  • BFS explores layer by layer, DFS explores deeply before backtracking.

  • Dijkstra’s and A* handle shortest paths.

  • Kruskal’s and Prim’s build spanning trees.

  • Bellman-Ford works with negative weights.

Conclusion

Graphs are everywhere, from maps to social networks, and the algorithms you have seen here are the building blocks for working with them. Whether it is finding paths, building spanning trees, or handling tricky weights, these tools open up a wide range of problems you can solve.

Keep experimenting and try out libraries like NetworkX when you are ready to take on bigger projects.

]]>