<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
    <channel>
        
        <title>
            <![CDATA[ automation - freeCodeCamp.org ]]>
        </title>
        <description>
            <![CDATA[ Browse thousands of programming tutorials written by experts. Learn Web Development, Data Science, DevOps, Security, and get developer career advice. ]]>
        </description>
        <link>https://www.freecodecamp.org/news/</link>
        <image>
            <url>https://cdn.freecodecamp.org/universal/favicons/favicon.png</url>
            <title>
                <![CDATA[ automation - freeCodeCamp.org ]]>
            </title>
            <link>https://www.freecodecamp.org/news/</link>
        </image>
        <generator>Eleventy</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:34:05 +0000</lastBuildDate>
        <atom:link href="https://www.freecodecamp.org/news/tag/automation/rss.xml" rel="self" type="application/rss+xml" />
        <ttl>60</ttl>
        
            <item>
                <title>
                    <![CDATA[ What a CAPTCHA Farm Attack Actually Looks Like ]]>
                </title>
                <description>
                    <![CDATA[ CAPTCHAs are something most of us have come across while browsing the internet. Sometimes we're asked to identify traffic lights in pictures, solve a small puzzle, or simply tick a box to confirm we'r ]]>
                </description>
                <link>https://www.freecodecamp.org/news/captcha-farm-attack/</link>
                <guid isPermaLink="false">6aeba47df91e889efe09a17a</guid>
                
                    <category>
                        <![CDATA[ cybersecurity ]]>
                    </category>
                
                    <category>
                        <![CDATA[ captcha ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Web Security ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Ethical Hacking ]]>
                    </category>
                
                    <category>
                        <![CDATA[ internet security ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ online fraud ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Reetain Raina ]]>
                </dc:creator>
                <pubDate>Wed, 04 Nov 2026 21:32:45 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/392975d9-8fb2-4311-bbd9-8ec42af322ef.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>CAPTCHAs are something most of us have come across while browsing the internet. Sometimes we're asked to identify traffic lights in pictures, solve a small puzzle, or simply tick a box to confirm we're human. Websites use these challenges to help distinguish real users from automated bots.</p>
<p>But there's an interesting problem with this approach. What happens when a bot doesn't need to solve the CAPTCHA itself because a real person can solve it for them?</p>
<p>This is how CAPTCHA farms work. Attackers use services where real people solve CAPTCHA challenges on behalf of automated programs. The person completing the challenge is human, but the activity happening behind it may still be controlled by a bot. This makes it harder for websites to identify suspicious requests, even when their CAPTCHA system appears to be working correctly.</p>
<p>Here, we'll look at how CAPTCHA farm attacks work, why attackers use human solvers, and what makes these attacks difficult to detect. We'll also go through some of the security measures developers can use to identify suspicious activity and better protect their websites.</p>
<h3 id="heading-what-well-cover">What We'll Cover:</h3>
<ul>
<li><a href="#heading-what-is-a-captcha-farm-attack">What Is a CAPTCHA Farm Attack?</a></li>
<li><a href="#heading-behind-the-scenes-the-four-part-infrastructure">Behind the Scenes: The Four-Part Infrastructure</a></li>
<li><a href="#heading-how-an-attack-unfolds-a-step-by-step-look">How an Attack Unfolds: A Step-by-Step Look</a></li>
<li><a href="#heading-why-attackers-rely-on-human-solvers">Why Attackers Rely on Human Solvers</a></li>
<li><a href="#heading-what-captcha-farms-target">What CAPTCHA Farms Target</a></li>
<li><a href="#heading-why-these-attacks-are-difficult-to-stop">Why These Attacks Are Difficult to Stop</a></li>
<li><a href="#heading-how-websites-can-defend-themselves">How Websites Can Defend Themselves</a><ul>
<li><a href="#heading-behavioral-telemetry-and-biometrics">Behavioral Telemetry and Biometrics</a></li>
<li><a href="#heading-network-and-device-fingerprinting">Network and Device Fingerprinting</a></li>
<li><a href="#heading-modern-rate-limiting-models">Modern Rate-Limiting Models</a></li>
<li><a href="#heading-post-authentication-anomaly-detection">Post-Authentication Anomaly Detection</a></li>
</ul>
</li>
<li><a href="#heading-security-beyond-the-checkbox">Security Beyond the Checkbox</a></li>
<li><a href="#heading-conclusion">Conclusion</a></li>
</ul>
<h2 id="heading-what-is-a-captcha-farm-attack">What Is a CAPTCHA Farm Attack?</h2>
<p>To understand how these attacks work, it helps to distinguish between the two primary ways attackers deal with verification gates: machine-driven recognition and human-driven outsourcing.</p>
<p>In an automated CAPTCHA-solving attack, an attacker uses computer vision models, optical character recognition (OCR), or neural networks to parse the challenge image and guess the answer without human intervention.</p>
<p>This approach is fast and computationally cheap, but it frequently fails when security vendors update their visual distortions, introduce interactive logic puzzles, or assess human-like mouse kinematics.</p>
<p>A CAPTCHA farm attack takes the opposite route. Instead of attempting to outsmart the defensive algorithm with software, the attacker routes the challenge to a human solving service. These commercial operations maintain large networks of manual workers who solve puzzles continuously around the clock. By inserting a human into the loop, the attacker converts what should be a technical roadblock into an outsourced labor pipeline.</p>
<p>The attack doesn't break the cryptography or reverse-engineer the underlying challenge code. Instead, it relies on a CAPTCHA relay attack: software manages the speed and scale of the attack, while real human eyes provide the cognitive validation the website expects.</p>
<h2 id="heading-behind-the-scenes-the-four-part-infrastructure">Behind the Scenes: The Four-Part Infrastructure</h2>
<p>A functional CAPTCHA farm is neither an ad-hoc group of rogue hackers nor an obscure black-box script. It's a commercial, multi-tiered infrastructure built around standard web development protocols.</p>
<p>Four distinct components work together to make the relay happen in a matter of seconds.</p>
<p>First is the attacker’s automated script. This client bot typically runs headlessly via browser automation tools like Puppeteer, Playwright, or custom HTTP client libraries. The script navigates to a target platform, such as a user registration portal, and fills in usernames, passwords, and demographic fields at machine speed until it hits a defensive challenge.</p>
<p>Second is the CAPTCHA-solving platform's API. Commercial solver platforms act as brokers between abuse operators and manual laborers. When the attacker's script encounters a challenge, it intercepts the site's public key, the challenge URL, and the visual assets, packaging them into an HTTP POST request directed at the solving service’s application programming interface (API).</p>
<p>Third is the distributed human workforce. Workers access these services through dedicated web dashboards, mobile apps, or browser plug-ins. When the API receives a task from an attacker, it pushes the image directly onto a worker’s screen. The worker identifies the bicycles or fire hydrants, types the distorted characters, or drags the puzzle slider into place, earning a microscopic payout, often ranging from $0.50 to $2.00 per thousand puzzles solved.</p>
<p>Fourth is the payload delivery back to the target. Once the worker submits the correct solution, the broker API returns the generated verification token back to the attacker’s script. The script injects this validation token into the target website's hidden form field and fires the final submission request. To the receiving server, the request bears an authentic verification signature generated by a real human session.</p>
<h2 id="heading-how-an-attack-unfolds-a-step-by-step-look">How an Attack Unfolds: A Step-by-Step Look</h2>
<p>To see how these parts interact in practice, consider a credential-stuffing or mass-account registration campaign against an online service.</p>
<ol>
<li><strong>Initiation:</strong> The attacker's script dispatches an HTTP request to establish a session with the target registration page, populating the input fields with generated credentials.</li>
<li><strong>Challenge trigger:</strong> The target application evaluates the incoming request. Detecting an unfamiliar IP address or an incomplete browser fingerprint, it triggers a CAPTCHA modal, pausing the registration sequence until a valid solution token is supplied.</li>
<li><strong>Extraction and dispatch:</strong> The bot reads the page DOM, captures the puzzle iframe parameters, site key, and challenge payload, and forwards this data to the CAPTCHA farm API.</li>
<li><strong>Human execution:</strong> The farm's queuing system allocates the challenge to an active human worker. The worker completes the puzzle within an average response window of 10 to 25 seconds.</li>
<li><strong>Token retrieval:</strong> The worker’s response yields an official validation token from the CAPTCHA provider's origin server. The solving farm's API returns this signed string to the attacker’s script.</li>
<li><strong>Form finalization:</strong> The script injects the token into the page's response payload (such as g-recaptcha-response) and submits the form. The website’s backend queries the CAPTCHA provider's verification API, receives a confirmation that the challenge was successfully passed, and creates the account.</li>
</ol>
<p>The entire loop completes well before typical session timeouts kick in, leaving the target server with no obvious visual anomaly to flag.</p>
<h2 id="heading-why-attackers-rely-on-human-solvers">Why Attackers Rely on Human Solvers</h2>
<p>Building and maintaining custom neural networks to break modern visual challenges requires constant development effort. Defensive providers frequently alter noise patterns, camera angles, distortion algorithms and semantic puzzles specifically to break automated image classifiers.</p>
<p>Human solvers bypass this arms race entirely because the human visual cortex processes semantic context effortlessly. A person instantly understands what constitutes a "motorcycle," even if the image is cropped awkwardly, partially obscured by street foliage, or rendered under strange lighting.</p>
<p>This human adaptability is supported by extensive security research. In a foundational study, <a href="https://www.researchgate.net/publication/333536967_Towards_understanding_the_security_of_modern_image_captchas_and_underground_captcha-solving_services"><u>Towards Understanding the Security of Modern Image Captchas and Underground Captcha-Solving Services</u></a>, researchers analyzed the commercial ecosystem across 152 solving platforms. They found that human-solving services achieved near-flawless accuracy rates across complex, diverse image challenges where purely automated computer vision solvers repeatedly struggled or degraded over time.</p>
<p>Outsourcing the problem is also remarkably inexpensive. Because labor pools are drawn from regions with low living costs, attackers can bypass thousands of protective checkpoints for less than the price of a cup of coffee. The workflow is highly scalable: software handles millions of connections, while human labor is summoned on-demand only for the few seconds required to clear a barrier.</p>
<h2 id="heading-what-captcha-farms-target">What CAPTCHA Farms Target</h2>
<p>CAPTCHA farms rarely serve as self-contained attacks. Instead, they act as an operational enabler for wider abusive campaigns across the web.</p>
<p>First, they enable mass fake account creation. Malicious actors generate millions of disposable profiles on social media, dating networks, and developer platforms to distribute phishing links, spam search engines, or run coordinated disinformation campaigns.</p>
<p>They also facilitate credential stuffing. When attackers test databases of leaked usernames and passwords against banking or retail portals, defensive systems usually throw a CAPTCHA after a handful of failed attempts. Solver APIs allow the credential-testing bots to continue cycling through login credentials uninterrupted.</p>
<p>Inventory hoarding and scalping is another common use case. High-demand retail drops, concert tickets, and travel bookings are frequent targets. Security researchers tracking real-world bot traffic at <a href="https://datadome.co/threat-research/how-datadome-stopped-a-captcha-farm-behind-18-million-ticket-scalping-attempts/"><u>DataDome</u></a> documented an operation where an attacker combined bot automation with a CAPTCHA farm to orchestrate 18 million automated ticket-scalping attempts, successfully bypassing front-line interactive verification.</p>
<p>And they enable data scraping at scale. Web crawlers harvesting competitive pricing, directory databases, or intellectual property frequently utilize solver APIs whenever protected search and discovery pages present challenge gates.</p>
<p>In every case, the CAPTCHA farm doesn't execute the underlying abuse: it merely removes the specific verification tripwire meant to halt it.</p>
<h2 id="heading-why-these-attacks-are-difficult-to-stop">Why These Attacks Are Difficult to Stop</h2>
<p>Stopping CAPTCHA farms is inherently difficult because they exploit the core assumption behind standard challenge-response security: that the entity answering the challenge is the entity making the request.</p>
<p>This systemic blind spot is analyzed in the research paper <a href="https://link.springer.com/chapter/10.1007/978-3-319-66399-9_24"><u>On the Security and Usability of Dynamic Cognitive Game CAPTCHAs</u></a>, which evaluated how puzzle formats hold up against relay systems. The study demonstrated that regardless of how cognitively engaging, gamified, or technically dynamic a test is, relay architectures successfully decouple cognitive processing from the requesting client. The server receives a mathematically legitimate, human-generated solution token, making the puzzle itself effectively irrelevant.</p>
<p>Also, attackers route their client requests through residential proxy networks. The initial HTTP request comes from one residential IP address, the CAPTCHA is rendered by an automated browser mimicking common hardware profiles, and the solution is completed remotely. The defender’s server can't verify the physical proximity between the solver and the requesting client, creating an asymmetric advantage for the attacker.</p>
<h2 id="heading-how-websites-can-defend-themselves">How Websites Can Defend Themselves</h2>
<p>Because human solvers can easily clear visual and cognitive challenges, treating a solved puzzle as a blanket guarantee of trustworthiness is no longer a viable defensive strategy. Web applications require a defense-in-depth model that monitors behaviors before, during, and long after a verification challenge appears.</p>
<h3 id="heading-behavioral-telemetry-and-biometrics">Behavioral Telemetry and Biometrics</h3>
<p>Automated scripts can easily spoof screen resolutions and user-agent strings, but reproducing organic human cursor movement across an entire page requires far more work. Real users demonstrate micro-deviations, variable acceleration curves, pauses, and continuous mouse tremors.</p>
<p>When a CAPTCHA farm attack takes place, there's often a distinct behavioral mismatch: the form fields are filled out instantaneously or via scripted keyboard events, followed by an abrupt pause while the remote worker solves the puzzle, followed by an instantaneous token injection. Monitoring telemetry across the whole lifecycle of the page exposes this artificial stitching.</p>
<h3 id="heading-network-and-device-fingerprinting">Network and Device Fingerprinting</h3>
<p>Defenders must assess context beyond the single page interaction. Inspecting TCP/IP packet behavior, TLS fingerprints (JA3/JA4 fingerprints), HTTP/2 frame parameters, and device APIs reveals whether the underlying runtime environment is a standard consumer browser or an orchestrated automation tool running inside a data center.</p>
<h3 id="heading-modern-rate-limiting-models">Modern Rate-Limiting Models</h3>
<p>Relying entirely on puzzle solving often introduces accessibility barriers for real users while doing little to stop funded attackers. As explored in research published by the Association for Computing Machinery regarding <a href="https://dl.acm.org/doi/10.1145/3638759"><u>mCaptcha and Proof-of-Work Rate Limiting</u></a>, defensive architectures can enforce cryptographic proof of work puzzles at the transport layer.</p>
<p>By forcing the connecting client's CPU to compute complex mathematical problems proportional to the platform's current traffic load, defense systems raise the economic and computational cost of running automated campaigns, neutralizing attacks without demanding manual human puzzle-solving.</p>
<h3 id="heading-post-authentication-anomaly-detection">Post-Authentication Anomaly Detection</h3>
<p>Passing an authentication challenge should never grant unconditional trust. Web architectures should monitor post-login activity for anomalous actions:</p>
<ul>
<li>Does an account immediately attempt hundreds of data queries per minute right after registering?</li>
<li>Does it immediately navigate directly to high-value purchase endpoints without loading associated asset stylesheets or images?</li>
<li>Does the session change IP ranges and geographic locations between the initial page load and the final transaction?</li>
</ul>
<p>Identifying these signals allows security teams to invalidate sessions even if the initial challenge was solved cleanly.</p>
<h2 id="heading-security-beyond-the-checkbox">Security Beyond the Checkbox</h2>
<p>CAPTCHA farms show why websites can't rely on a single security check to keep automated attacks away. Even when someone successfully completes a CAPTCHA, the request may still be part of an automated operation.</p>
<p>CAPTCHAs are designed to check whether a person can solve a particular challenge. But they can't tell whether that person is the actual user trying to access the website or someone solving the challenge on behalf of a bot. This is one of the reasons CAPTCHA farms can bypass a security measure that would normally stop automated traffic.</p>
<p>For developers, this means a completed CAPTCHA should be treated as just one part of the verification process. Websites also need to monitor user behavior, network activity, and suspicious requests to identify possible attacks. Combining these checks makes it easier to spot activity that a CAPTCHA alone might miss.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>CAPTCHA farms make it difficult for websites to tell whether a completed verification challenge actually came from a legitimate user. Even when a real person solves the puzzle, the request may still be part of an automated attack.</p>
<p>For developers, this means it's worth paying attention to what happens before and after a CAPTCHA is completed. Things like unusual login attempts, repeated requests, unexpected changes in user behavior, and suspicious account activity can provide useful clues.</p>
<p>Using CAPTCHA alongside rate limiting, behavioral monitoring, and other security checks can help websites identify these attacks more effectively. It's also important to review these protections regularly as attackers change how they operate.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Automate Docker Image Creation for Postgres Using Shell Commands ]]>
                </title>
                <description>
                    <![CDATA[ If you have certain lines of code that repeat over and over again in a project, there's a pretty standard solution for it: using a function. But what about when you have the same project file setup th ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-automate-docker-image-creation-for-postgres-using-shell/</link>
                <guid isPermaLink="false">6aea2aaffbe759481e70cb16</guid>
                
                    <category>
                        <![CDATA[ shell ]]>
                    </category>
                
                    <category>
                        <![CDATA[ shell scripting ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Bash ]]>
                    </category>
                
                    <category>
                        <![CDATA[ docker images ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Docker compose ]]>
                    </category>
                
                    <category>
                        <![CDATA[ postgres ]]>
                    </category>
                
                    <category>
                        <![CDATA[ vibe coding ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Workflow Automation ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ jabo Landry ]]>
                </dc:creator>
                <pubDate>Tue, 03 Nov 2026 18:40:47 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/60aa4c00-42d7-4e36-84b4-89f55b140dca.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>If you have certain lines of code that repeat over and over again in a project, there's a pretty standard solution for it: using a function.</p>
<p>But what about when you have the same project file setup that gets repeated over and over? How do you solve that?</p>
<p>Meet bash scripting. Bash scripts are written to execute certain sequences of commands on your machine automatically without you having to do anything. This is known as workflow automation, and that's what I'll be teaching you how to do in this article.</p>
<h2 id="heading-table-of-contents">Table Of Contents</h2>
<ul>
<li><a href="#heading-prerequisites">Prerequisites</a></li>
<li><a href="#heading-what-motivated-this-project">What Motivated This Project?</a></li>
<li><a href="#heading-introduction">Introduction</a></li>
<li><a href="#heading-conditions-in-bash">Conditions in bash</a></li>
<li><a href="#heading-interactive-shell">Interactive Shell</a></li>
<li><a href="#heading-how-to-create-and-write-to-a-file">How to Create and Write to a File</a></li>
<li><a href="#heading-bash-arguments-and-flags">Bash Arguments and Flags</a></li>
<li><a href="#heading-how-to-make-your-scripts-executable-and-available-on-your-pcs-path">How to Make Your Scripts Executable and Available on Your PC's Path</a></li>
<li><a href="#heading-conclusion">Conclusion</a></li>
</ul>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>To follow along with this article, you should have:</p>
<ul>
<li>Git-bash installed on your machine</li>
<li>Some experience using a terminal or command prompt to run and execute commands (helpful but not required)</li>
<li>Understanding of basic programming concepts (like conditions and loops)</li>
<li>Docker installed on your machine</li>
<li>Basic understanding of Docker concepts (like images and containers)</li>
<li>Experience using Postgres (optional but recommended)</li>
</ul>
<h2 id="heading-what-motivated-this-project">What Motivated This Project?</h2>
<p>I was recently setting up a Docker image for Postgres on a few projects I was working on, and I noticed that I was following the same pattern in creating the images. The only difference was that they had different database credentials and database names.</p>
<p>I also realized that a large portion of today's rapidly developed apps tend to use Supabase as their backend, which can come with resource limits such as database storage, bandwidth, concurrent connections, and API request quotas (depending on the service tier and size of the project). And these issues can be solved by setting up your database.</p>
<p>So I thought, why not make a CLI tool that could automate creating Docker images for Postgres? That's when I learned about writing executable shell commands using bash to create workflow automations.</p>
<p>You can find the full working example used in this guide in this <a href="https://github.com/jabo-arnold-landry/Postgres-docker-image-generator">git-repo</a>.</p>
<h2 id="heading-introduction">Introduction</h2>
<p>Bash is a shell and scripting language used to write and execute commands or scripts that perform operations on a computer, such as creating directories and files, manipulating files, automating tasks, and managing system processes.</p>
<p>In this guide, you'll learn to automate the process of creating a Docker image for a Postgres database so that you don't have to create the image from scratch for each new project.</p>
<h3 id="heading-bash-scripting-file-setup">Bash Scripting File Setup</h3>
<p>To get started, open you preferred text-editor and create a directory or folder there. Then within that directory, create a file with the extension <code>.sh</code>.</p>
<p>At the beginning of a file, type:</p>
<pre><code class="language-shell">#!/bin/bash
</code></pre>
<p><code>#!/bin/bash</code> (known as a "shebang") tells the operating system to run the script using the Bash shell, so the commands inside the file are executed as Bash/terminal commands.</p>
<h2 id="heading-conditions-in-bash">Conditions in bash</h2>
<p>After you've finished creating a script file and adding shebang, let's start by exploring how if statements are used in bash.</p>
<p>The syntax of <code>if</code> looks like this:</p>
<pre><code class="language-shell">if [[ condition ]];
    then
        command to execute
fi
</code></pre>
<p>The <code>if</code> in bash is no different from what you're used to in other programming languages:</p>
<ul>
<li>You use double square braces ([]) to add a condition you're checking against and a semicolon (;).</li>
<li>You use <code>then</code> after the <code>if</code> declaration to specify what should be executed if the condition is true.</li>
<li>You use the <code>fi</code> to close the condition statement.</li>
</ul>
<p>When you're writing the condition inside the square braces, make sure you leave a space before the condition and another one after it (as you can see in the code above). If you omit these spaces, you'll get errors.</p>
<h3 id="heading-working-example">Working Example:</h3>
<p>Since we are automating Postgres database Docker image creation, first check if the file exists and has content. If it does have content, end the script. If not, we'll proceed with the next script.</p>
<pre><code class="language-shell">#!/bin/bash

if [[ -s docker-compose.yml ]];
    then
        echo "The file already exists"
        exit 0
fi

echo "Next command"
</code></pre>
<p>In the if condition, we use the flag <code>-s</code> to check if the file exists and has contents. File flags check file availability and permissions, and include but aren't limited to:</p>
<ul>
<li>-f: used to check if the file exists</li>
<li>-s: used to check if the file exists and has contents</li>
<li>-d: used to check if a directory exists</li>
<li>-r: used to check if a file is readable</li>
<li>-w: used to check if a file is writable</li>
<li>-x: used to check if a file is executable</li>
</ul>
<p>In the script, we're using <code>-s</code> to check if a file named <code>docker-compose.yml</code> exists in the directory where we're executing the command. If it's false, we use the <code>echo</code> keyword to print the text "Next command". If it's true, we print the text "The file already exists".</p>
<p><code>exit</code> acts as a return keyword for bash scripting, 0 is used for success, and 1 is used for failure.</p>
<p>Let's test all this by creating a Docker Compose file and adding some content in the same directory where we're writing the bash script. In your terminal type <code>bash name-of-file.sh</code>, and you should see "The file already exists":</p>
<img src="https://cdn.hashnode.com/uploads/covers/69c7bcff7cf27065100ae8be/14d2b684-5e37-4e70-b6be-05386777fd68.png" alt="output after creating the already existing file using bash" style="display: block;" width="1470" height="213" loading="lazy">

<p>But if we delete the content in the file and run the command again, we should see the output saying "Next command":</p>
<img src="https://cdn.hashnode.com/uploads/covers/69c7bcff7cf27065100ae8be/130698ee-7309-4f3e-9eb0-37632b954ded.png" alt="output of a successful file creation in bash" style="display: block;" width="1459" height="138" loading="lazy">

<h2 id="heading-interactive-shell">Interactive Shell</h2>
<p>To make an automation dynamic and flexible you will need to receive inputs, take an example of a database name you won't always have the same database name, the name will change based on the project,</p>
<p>you do need to setup away to use a given name of the database when creating a new docker image for that you read inputs using the <code>read</code> keyword.</p>
<p>There are different types of inputs based on the operation. And there are flags that are used to specify which type of input you're expecting, such as:</p>
<ul>
<li>-p: for prompting the user to write a text</li>
<li>-s: used on passwords when you want the user input to stay hidden when writing it in the terminal</li>
<li>-a: used when you're expecting the user to pass in multiple inputs or if you're expecting an array from user.</li>
</ul>
<p>Here's the syntax for reading user input:</p>
<pre><code class="language-shell">read -p 'input_label: ' variable_name
</code></pre>
<h3 id="heading-working-example">Working Example:</h3>
<p>Now let's create scripts to receive user input about the Postgres database image you're going to create.</p>
<pre><code class="language-shell">#!/bin/bash

# database credentials
read -p 'DataBase-name(*): ' db
read -p 'DataBase-user(*): ' user
read -sp 'DataBase-password(*): ' pwd
echo
echo "your database is -&gt; $db with a username of $user and a password of $pwd"
</code></pre>
<p>The script uses <code>-p</code> on the <code>read</code> keyword that prompts a string of text and <code>-sp</code> that prompts the user to enter a string of text that's hidden (because the <code>-s</code> hides what's being typed).</p>
<p>If you don't use the <code>-p</code> option together with <code>-s</code>, you won't be able to display a prompt asking the user to enter a password. With <code>-s</code> alone, the input is hidden, but no prompt is shown unless you print one using <code>echo</code> or <code>printf</code>.</p>
<p>To print or use the input the user has added, use the dollar sign ($) followed with a variable name for the input you're expecting from the user. In the example above, we named the variable for getting the database name <code>db</code>, so we'll use <code>$db</code> when we want to access the name of the database the user has input.</p>
<p>Now if you run the script in the terminal, you should see a sequence of terminal prompts waiting for inputs. Once they're filled, you'll see all the input's values in the terminal:</p>
<img src="https://cdn.hashnode.com/uploads/covers/69c7bcff7cf27065100ae8be/9871cea4-07f0-4626-ba2a-f4ca738e447b.png" alt="output on how to get and display user inputs " style="display: block;" width="1342" height="286" loading="lazy">

<h3 id="heading-how-to-validate-and-sanitize-input">How to Validate and Sanitize Input</h3>
<p>You'll also need to validate input before executing anything. This is important because sometimes crucial field might be missing or mistyped. For example, you might forget to input the database name for the image you're trying to create.</p>
<p>When validating input, you'll use one of these flags (among others):</p>
<ul>
<li>-z: checks if a variable value is empty</li>
<li>-n: checks if a variable value is not empty</li>
<li>-lt: checks if a number is less than a given number</li>
<li>-gt: checks if a number is greater than a given number</li>
<li>-eq: checks for equality of two values</li>
</ul>
<p>You can use the <code>-z</code> flag on every input that you need to make sure isn't empty before moving on to the next step:</p>
<pre><code class="language-shell">#!/bin/bash

# database credentials
read -p 'DataBase-name(*): ' db
if [[ -z $db ]]
    then
        echo "The database field cannot be empty!"
        exit 1
fi

read -p 'DataBase-user(*): ' user
if [[ -z $user ]]
    then
        echo "The user field cannot be empty!"
        exit 1
fi

read -sp 'DataBase-password(*): ' pwd
if [[ -z $pwd ]]
    then
        echo "The password field cannot be empty!"
        exit 1
fi
</code></pre>
<p>This script helps you make sure that all critical information about the database is present before proceeding to another input.</p>
<h3 id="heading-default-values">Default Values</h3>
<p>You may want some fields to use default value (for example, the container name of the image, port, or volume of the container). For such scenarios you can use <code>:=</code> to assign a variable default value:</p>
<pre><code class="language-shell">#!/bin/bash

# container information
read -p 'Container-name: ' container_name
read -p 'Container-port(if omitted, the container will use default postgresql default port of 5432): ' port
read -p 'Volume-name: ' vol_name

echo "container-name-&gt; ${container_name:=cont_one}, port-&gt;${port:=5432}, volume_name-&gt;${vol_name:=backend_data}"
</code></pre>
<p>When you run the script and skip all three of those inputs, bash will use the default value(s) you've used as fallbacks.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69c7bcff7cf27065100ae8be/63c5a5df-10f6-4b9e-b3f1-28460b0db3e4.png" alt="Output for setting up a default value on variables" style="display: block;" width="1340" height="204" loading="lazy">

<p>But if you pass value for those inputs, bash uses the provided values. For example, supply all three inputs and see the output you get:</p>
<img src="https://cdn.hashnode.com/uploads/covers/69c7bcff7cf27065100ae8be/d62949f4-b87e-448e-b702-393a04ed19ab.png" alt="Receiving user inputs and displaying the recevied input" style="display: block;" width="1348" height="222" loading="lazy">

<h2 id="heading-how-to-create-and-write-to-a-file">How to Create and Write to a File</h2>
<p>Now that you know how to create a Docker image for a Postgres database, you need to create a <code>docker-compose.yml</code> file that will be executed when starting and stopping your containers.</p>
<p>In Bash, there are multiple ways of creating a file:</p>
<ul>
<li>First, you can use <code>&gt; filename</code>. This creates a file if it doesn't exist. If it exists, its contents are overwritten (replaced).</li>
<li>There's also <code>&gt;&gt; filename</code>: This appends contents to a file or creates a file if one doesn't exist.</li>
<li>Finally, there's <code>touch filename</code>: This creates a file if it doesn't exist. If the file already exists, its contents are preserved and only the file's timestamps are updated.</li>
</ul>
<h3 id="heading-how-to-use-gt-to-create-and-write-to-a-file">How to Use <code>&gt;</code> to Create and Write to a File</h3>
<p>When you want to create a file and write content to it at the same time, you'll want to use echo in front of <code>&gt;</code> and then add the filename, like this:</p>
<pre><code class="language-shell">#!/bin/bash

echo Hello&gt; test.txt
</code></pre>
<p>This will create a <code>test.txt</code> file with "hello" as content inside.</p>
<h3 id="heading-how-to-use-gtgt-to-append-contents-to-an-existing-file">How to Use <code>&gt;&gt;</code> to Append Contents to an Existing File</h3>
<p>To append contents to an existing file, add the contents before the <code>&gt;&gt;</code> and then follow with the filename you want to append content onto. Let's add Africa after Hello in the <code>test.txt</code> file we created in the previous script.</p>
<pre><code class="language-shell">#!/bin/bash

echo Africa!&gt;&gt; test.txt
</code></pre>
<p>Now test.txt will have Hello and Africa! on the new line.</p>
<p>Using those symbols, let's create a <code>docker-compose.yml</code> file that will hold contents about our Postgres database setup.</p>
<pre><code class="language-shell">#!/bin/bash

&gt; docker-compose.yml
</code></pre>
<p>And just like that you have your docker-compose file created in your directory.</p>
<h3 id="heading-how-to-write-multiple-different-contents-to-a-file">How to Write Multiple Different Contents to a File</h3>
<p>You can also use the <code>cat</code> keyword to write to a file. Typically, <code>cat</code> is used for reading the contents of a file. But when you combine it with <code>&lt;&lt;EOF</code> and the sign for appending contents to a file <code>&gt;&gt;</code> (that we just discussed), you can add multiple contents to a file in the format you want. The syntax looks like this:</p>
<pre><code class="language-shell">
# writing docker compose contents

cat &lt;&lt;EOF&gt;&gt; docker-compose.yml

  # file contents
EOF
</code></pre>
<p>The <code>&lt;&lt;EOF</code> tells bash to write contents to a file until it finds the EOF keyword (end of file). Then you use <code>&gt;&gt; filename</code> to append the contents to the file. With a Docker compose setup it would look like this:</p>
<pre><code class="language-shell">#!/bin/bash

#file creation and writing
 &gt;  docker-compose.yml

cat&lt;&lt;EOF&gt;&gt;docker-compose.yml
 services:
    postgres:
        image: postgres:18-alpine
        container_name: ${container_name:= $db}

        environment:
            POSTGRES_USER: $user
            POSTGRES_PASSWORD: $pwd
            POSTGRES_database: $db

        ports:
            - ${port:=5432}:5432
        volumes:
            - ${vol_name:=database-data}:/var/postgresql

 volumes:
    ${vol_name:=database-data}:

EOF
</code></pre>
<h2 id="heading-bash-arguments-and-flags">Bash Arguments and Flags</h2>
<p>With Bash arguments, you can pass values in the same line while executing a script. To use arguments, add the values while executing the script on the same line, like this:</p>
<pre><code class="language-shell">bash script val1 val2 
</code></pre>
<p>To read arguments value(s), use the dollar sign ($) followed by the position in which you inputted your values (positions start at 1, not 0).</p>
<p>In the example, if you want to echo the content of the second argument value, you can use <code>echo $2</code>.</p>
<p>To understand this better, let's enhance data input experience by passing the database credentials on the fly using arguments:</p>
<pre><code class="language-shell">#!/bin/bash

# database credentials

if [[ -z $1 ]]; then
    read -p 'DataBase-name(*): ' db
 else
    db="$1"
fi

# database checking and validating

if [[ -z $2 ]]; then
    read -p 'DataBase-user(*): ' user
 else
    user="$2"
fi

# user checking and validating

if [[ -z $1 ]]; then
    read -sp 'DataBase-password(*): ' pwd
 else
    pwd=$3
fi

# password vaidation and checking
</code></pre>
<p>The snippet first checks if there's an argument. If there aren't any, you use the prompt to input the value. If there are arguments, you assign the argument value to a variable that represents the given database credentials (database name, password, and so on...). Now you can run the script with arguments:</p>
<pre><code class="language-shell">bash image.sh customers ABC-Company secret123!
</code></pre>
<p>The image that will be created from the above script will have a database of <code>customers</code>, user <code>ABC-Company</code>, and a password of <code>secret123!</code>.</p>
<h3 id="heading-bash-flagsoptions">Bash Flags/Options</h3>
<p>The script has a section that asks for information about the image that you're creating, but the info can be optional. With bash option/flags, you can skip the step that prompts for container information input using the <code>-D</code> option. This lets you explicitly keep the default container info before the script runs.</p>
<p>Here's an example:</p>
<pre><code class="language-bash">
#!/bin/bash

while getopts "ny" options; do
	case "$options" in
		n) echo "You choose no";;
		y) echo "You choose yes";;
		*) echo "No such option/flag"; exit 1;;
    esac
done

shift $((OPTIND - 1))
</code></pre>
<p>In the above script:</p>
<ol>
<li>The while loop keeps running as long as <code>getopts</code> isn't empty</li>
<li><code>getopts</code> gets the passed option in the script</li>
<li><code>"ny"</code> are the options we're expecting from user when running the script</li>
<li><code>options</code> is a variable that will keep track of the input option between "n" and "y"</li>
<li><code>do</code> starts the loop</li>
<li><code>case</code> checks if the selected option is in the defined option using the <code>in</code> keyword</li>
<li>We then define what happens when we call the script with the option <code>-n</code> or <code>-y</code> and use a double semi-colon (;;) to close the option definition</li>
<li>The <code>*</code> is used in case the input options aren't defined.</li>
<li><code>esac</code> then closes the case</li>
<li><code>done</code> ends the loop so the loop doesn't continue running forever</li>
<li><code>shift $((OPTIND - 1))</code> helps avoid a conflict that may occur when the script also expects arguments by shifting the remaining arguments left <em>over</em> the flags, effectively discarding the flags from the parameter list.</li>
</ol>
<p>You can then apply the syntax in the script to skip the step that asks for container info when <code>-D</code> is passed as an option during the script execution.</p>
<pre><code class="language-shell">
#!/bin/bash

ask_container_info=true

while getopts "D" options; do
	case "$options" in
		D) ask_container_info=false;;
		*) echo "No such option/flag"; exit 1;;
    esac
done

shift $((OPTIND - 1))

if [[ $ask_container_info == true ]]; then
	#Prompt the user to input custom container information
    read -p 'Container-name: ' container_name
    read -p 'Container-port(if omitted, the container will use default postgresql default port of 5432): ' port
    read -p 'Volume-name: ' vol_name
fi
</code></pre>
<p>There is one option here, which is why the option section only has <code>D</code>. <code>ask_container_info</code> starts as true, meaning unless the option is passed, the script will prompt the user to enter the container information. We then change the value of <code>ask_container_info</code> to false when the <code>-D</code> is passed with the script's execution.</p>
<h2 id="heading-how-to-make-your-scripts-executable-and-available-on-your-pcs-path">How to Make Your Scripts Executable and Available on Your PC's Path</h2>
<p>The next step is to make the script available and executable in every terminal you use on your machine. At the moment, the script just works in the current directory you're using to create and write your scripts.</p>
<p>We'll start by making the script file executable and remove the file extension from the file, so we don't need to always call our script with the <code>.sh</code> extension.</p>
<h3 id="heading-how-to-make-a-file-executable">How to Make a File Executable</h3>
<p>To make a file executable, use the Linux <code>chmod</code> command. It stands for change mode. Mode includes file permissions like write, read, and execute.</p>
<pre><code class="language-shell">chmod +x image
</code></pre>
<p>Here, <code>x</code> makes the file executable, and then you've listed the file name you want to make executable (<code>image</code>).</p>
<h3 id="heading-how-to-add-your-directory-to-the-systemmachine-path">How to Add Your Directory to the <code>system/machine</code> Path</h3>
<p>To make this available in every terminal you open, regardless of the directory, you'll need to register your scripts in the system paths where all global scripts and commands are stored.</p>
<p>To do that, copy the full path of your directory and then in your terminal type the following:</p>
<pre><code class="language-shell">export PATH="$PATH:&lt;YOUR_DIRECTORY_LOCATION&gt;"
</code></pre>
<p>Replace <code>&lt;YOUR_DIRECTORY_LOCATION&gt;</code> with your directory. This command will add the script directory into the current working terminal. But if you try to switch to another terminal and run your script, it'll fail.</p>
<p>Now let's configure the directory globally so every terminal will be able to execute the script:</p>
<pre><code class="language-shell">nano ~/.bash_profile
</code></pre>
<p>After this you should see something that looks like this:</p>
<img src="https://cdn.hashnode.com/uploads/covers/69c7bcff7cf27065100ae8be/eb9d4588-4d21-43d5-a666-d406cdd9946d.png" alt="Nano editor for bash profiles and its contents" style="display: block;" width="1496" height="387" loading="lazy">

<p>You can see in the image above that I already have my path declared in the file (see the selected text).</p>
<p>If it's not in you're file, you can copy-paste <code>export PATH="$PATH:&lt;YOUR_DIRECTORY_LOCATION&gt;"</code> inside the file. When you're done, press <code>Ctrl+O</code> and then <code>enter</code> to save. Then press <code>Ctrl+x</code> to exit the nano editor.</p>
<p>Then re-load the machine path again to include your scripts there:</p>
<pre><code class="language-shell">source ~/.bash_profile
</code></pre>
<p>After this, you'll be able to run your script in any terminal on your machine using the script file name only (I named my file <code>image</code>):</p>
<img src="https://cdn.hashnode.com/uploads/covers/69c7bcff7cf27065100ae8be/09dcf19f-0712-48f7-9ac6-f00723aa267c.png" alt="Output of a successfully script execution for the docker image creation script" style="display: block;" width="1468" height="662" loading="lazy">

<h2 id="heading-conclusion">Conclusion</h2>
<p>In this guide, you learned how to create your own scripts to automate your workflows. I hope it helps you reduce the amount of energy and time you waste on repetitive tasks that can be automated using script commands.</p>
<p>You can find the full working CLI tool that we used as an example in this guide on this <a href="https://github.com/jabo-arnold-landry/Postgres-docker-image-generator">git-repo</a>. There you'll find the installation guide and usage tips. If you find it helpful, please consider starring the repo and sharing it with your collogues who might benefit from it.</p>
<p>Thanks!</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ Python in Marketing: A Practical Guide to Marketing Analytics, Automation, and Data-Driven Growth [Full Book] ]]>
                </title>
                <description>
                    <![CDATA[ Marketing has always involved wrangling lots of data. Before digital marketing, businesses had to track sales, customer purchases, coupon redemptions, store traffic, and survey responses to understand ]]>
                </description>
                <link>https://www.freecodecamp.org/news/python-in-marketing-a-practical-book/</link>
                <guid isPermaLink="false">6ae36961d5f7580678ebe2d1</guid>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ data analytics ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Data Science ]]>
                    </category>
                
                    <category>
                        <![CDATA[ software development ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ book ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Eva J Patel ]]>
                </dc:creator>
                <pubDate>Thu, 29 Oct 2026 15:42:25 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/edbaff87-7932-42b2-8c28-61bb67972244.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Marketing has always involved wrangling lots of data. Before digital marketing, businesses had to track sales, customer purchases, coupon redemptions, store traffic, and survey responses to understand what their customers wanted.</p>
<p>And while digital marketing didn't invent marketing analytics, it dramatically increased the amount of information out there.</p>
<p>Today, almost every interaction can produce data. A person might see an advertisement, click it, visit a website, read an article, subscribe to an email list, open several emails, add a product to a shopping cart, leave the website, return through a search engine, and eventually make a purchase. And every one of those interactions can potentially become a data point.</p>
<p>For a marketer, this creates an incredible opportunity. It also creates a problem: having more data doesn't automatically result in better decisions.</p>
<p>A company can have hundreds of thousands of customer records and still not know which customers are most valuable. A marketing team can have dozens of dashboards and still struggle to answer a simple question such as, "Which campaign should we invest more money into?"</p>
<p>This is where programming becomes useful. Python gives marketers a way to work with data programmatically. Instead of manually opening spreadsheets, copying formulas, filtering rows, and creating the same report every week, you can build reusable processes that perform those tasks automatically.</p>
<p>More importantly, Python allows you to go beyond simple reporting. You can use it to investigate customer behavior, discover patterns, perform statistical analysis, create predictive models, automate workflows, and build systems that continuously transform raw marketing data into useful information.</p>
<p>That doesn't mean every marketer needs to become a software engineer. It just means that programming can become another tool in a marketer's toolbox and this tutorial explains how.</p>
<h3 id="heading-prerequisites">Prerequisites</h3>
<p>You don't need to be an experienced programmer or marketer to use this book, but a few basics will make the examples easier to follow.</p>
<ul>
<li><strong>A computer with Python installed:</strong> Use a recent Python 3 version. If you're new to Python, an editor such as VS Code or PyCharm can help you run the examples.</li>
<li><strong>Basic computer and file skills:</strong> You should be comfortable opening files, creating folders, and working with CSV or Excel spreadsheets.</li>
<li><strong>A willingness to learn Python fundamentals:</strong> Part I starts with variables, strings, numbers, conditions, loops, and functions, so prior Python experience is helpful but not required.</li>
<li><strong>A little familiarity with marketing concepts:</strong> Terms such as campaign, conversion, revenue, clicks, and advertising spend will appear throughout. You can learn the metrics as you go. The book explains key terms where they're introduced.</li>
<li><strong>The ability to install packages:</strong> Later chapters use libraries such as pandas, NumPy, Matplotlib, scikit-learn, and requests. Examples show how to install them.</li>
</ul>
<p>You can follow the book using sample data. You don't need access to a company's private customer data or paid marketing APIs. If you use real data, make sure you have permission to access it and avoid exposing personal or confidential information.</p>
<h3 id="heading-what-this-book-will-teach-you">What This Book Will Teach You</h3>
<p>This book focuses on three major ideas:</p>
<ol>
<li><strong>Marketing analytics:</strong> using data to understand what happened and why.</li>
<li><strong>Marketing automation:</strong> using Python to perform repetitive marketing tasks more efficiently.</li>
<li><strong>Marketing intelligence:</strong> combining data, analytics, automation, and predictive techniques to support better decisions.</li>
</ol>
<p>You'll begin with Python fundamentals and gradually move into more advanced applications.</p>
<p>By the end, you'll build a marketing intelligence system that combines many of these concepts into one project.</p>
<h3 id="heading-heres-what-well-cover">Here's What We'll Cover:</h3>
<ul>
<li><a href="#heading-part-i-python-fundamentals-for-marketers">Part I: Python Fundamentals for Marketers</a></li>
<li><a href="#heading-part-ii-working-with-marketing-data">Part II: Working With Marketing Data</a></li>
<li><a href="#heading-part-iii-data-cleaning-and-analysis-with-pandas">Part III: Data Cleaning and Analysis With pandas</a></li>
<li><a href="#heading-part-iv-marketing-visualization">Part IV: Marketing Visualization</a></li>
<li><a href="#heading-part-v-customer-and-campaign-analytics">Part V: Customer and Campaign Analytics</a></li>
<li><a href="#heading-part-vi-marketing-automation">Part VI: Marketing Automation</a></li>
<li><a href="#heading-part-vii-machine-learning-for-marketing">Part VII: Machine Learning for Marketing</a></li>
<li><a href="#heading-part-viii-building-a-marketing-intelligence-system">Part VIII: Building a Marketing Intelligence System</a></li>
<li><a href="#heading-capstone-project-build-a-python-marketing-intelligence-platform">Capstone Project: Build a Python Marketing Intelligence Platform</a></li>
<li><a href="#heading-conclusion-next-steps">Conclusion / Next Steps</a></li>
</ul>
<h2 id="heading-the-philosophy-of-this-book">The Philosophy of This Book</h2>
<p>There's an important difference between learning Python and learning to <strong>use</strong> Python.</p>
<p>Learning Python means understanding syntax, data types, functions, loops, libraries, and other programming concepts. Using Python means taking those concepts and applying them to an actual problem.</p>
<p>This book focuses heavily on the second approach.</p>
<p>You'll still learn Python fundamentals, but the examples will consistently connect those fundamentals to marketing.</p>
<p>Instead of simply writing:</p>
<pre><code class="language-python">x = 10
y = 20
print(x + y)
</code></pre>
<p>we'll work with things like:</p>
<pre><code class="language-python">campaign_budget = 10000
campaign_spend = 7350

remaining_budget = campaign_budget - campaign_spend

print(remaining_budget)
</code></pre>
<p>The programming concept is the same, but the context is different.</p>
<p>That context is what makes the skill useful.</p>
<h2 id="heading-part-i-python-fundamentals-for-marketers">Part I: Python Fundamentals for Marketers</h2>
<p>Python doesn't know what a campaign is or what a customer is. And it doesn't know whether an advertisement is successful. Python simply processes the instructions you give it.</p>
<p>That means your first job is learning how to communicate with the computer.</p>
<p>Fortunately, you don't need to learn everything about Python before you can start working with marketing data. But you do need a practical foundation.</p>
<h3 id="heading-thinking-like-a-programmer">Thinking Like a Programmer</h3>
<p>Programming can initially seem intimidating because computers are extremely literal.</p>
<p>If you tell a person: "Calculate how much money we have left from the campaign budget," they can probably figure out what you mean.</p>
<p>A computer needs explicit instructions. You need to tell it which values to use and which operation to perform.</p>
<p>For example:</p>
<pre><code class="language-python">budget = 10000
spent = 7350

remaining = budget - spent

print(remaining)
</code></pre>
<p>The output is:</p>
<pre><code class="language-text">2650
</code></pre>
<p>The computer didn't need to understand marketing. It simply followed the instructions.</p>
<p>This is the basic relationship between you and a programming language.</p>
<p>You provide instructions, and Python translates those instructions into operations the computer can perform.</p>
<h3 id="heading-breaking-problems-into-steps">Breaking Problems Into Steps</h3>
<p>One of the most important programming skills is learning to break a large problem into smaller steps.</p>
<p>Imagine a marketing manager asks: "Can you tell me which campaigns are performing poorly?"</p>
<p>That sounds simple, but it actually contains several smaller questions.</p>
<p>What does "poorly" mean?</p>
<p>Is it:</p>
<ul>
<li>Low revenue?</li>
<li>Low conversion rate?</li>
<li>High CPA?</li>
<li>Low ROAS?</li>
<li>Low engagement?</li>
<li>Low profit?</li>
</ul>
<p>You'll need to know what data you need, where that data is stored, how you should clean it, how to calculate the relevant metrics, and what threshold should determine whether a campaign is considered weak.</p>
<p>Once you break the problem apart, the programming becomes much easier.</p>
<p>For example:</p>
<pre><code class="language-text">Load campaign data.
Clean the data.
Calculate ROAS.
Calculate conversion rate.
Filter campaigns below the chosen thresholds.
Display the results.
</code></pre>
<p>Python can then turn those steps into code.</p>
<p>This way of thinking is called decomposition. You'll use it constantly throughout this book.</p>
<h3 id="heading-chapter-1-learning-and-implementing-the-basics">Chapter 1: Learning and Implementing the Basics</h3>
<h4 id="heading-variables">Variables</h4>
<p>Variables allow you to store information under a name.</p>
<pre><code class="language-python">campaign = "Back to School"
budget = 5000
clicks = 1250
conversions = 84
</code></pre>
<p>Now Python knows that:</p>
<ul>
<li><code>campaign</code> contains text.</li>
<li><code>budget</code> contains a number.</li>
<li><code>clicks</code> contains a number.</li>
<li><code>conversions</code> contains a number.</li>
</ul>
<p>You can use those variables later.</p>
<pre><code class="language-python">print(campaign)
print(budget)
print(clicks)
print(conversions)
</code></pre>
<p>You can also change their values.</p>
<pre><code class="language-python">budget = 5000

budget = 7500

print(budget)
</code></pre>
<p>The final value is:</p>
<pre><code class="language-text">7500
</code></pre>
<p>Variables are useful because marketing programs constantly work with changing values.</p>
<p>Today's revenue is different from yesterday's revenue. This month's advertising budget is different from last month's. A customer's order count changes as they make purchases. And a campaign's performance changes over time.</p>
<p>Variables allow your program to represent those values.</p>
<h4 id="heading-naming-variables-clearly">Naming Variables Clearly</h4>
<p>You could technically write:</p>
<pre><code class="language-python">x = 5000
</code></pre>
<p>But what does <code>x</code> mean? You would have to look elsewhere to find out.</p>
<p>A better approach is:</p>
<pre><code class="language-python">campaign_budget = 5000
</code></pre>
<p>The name immediately communicates its purpose.</p>
<p>Similarly:</p>
<pre><code class="language-python">total_revenue = 25000
total_cost = 8000
total_conversions = 450
</code></pre>
<p>Clear variable names make your code easier to understand. This becomes increasingly important as your programs grow.</p>
<h4 id="heading-strings">Strings</h4>
<p>Strings represent text.</p>
<pre><code class="language-python">campaign_name = "Holiday Sale"
channel = "Email"
customer_name = "Ava"
</code></pre>
<p>You can combine strings, for example:</p>
<pre><code class="language-python">message = "Welcome, " + customer_name

print(message)
</code></pre>
<p>The result is:</p>
<pre><code class="language-text">Welcome, Ava
</code></pre>
<p>But f-strings are usually easier to read.</p>
<pre><code class="language-python">message = f"Welcome, {customer_name}"

print(message)
</code></pre>
<p>You can include multiple variables:</p>
<pre><code class="language-python">customer_name = "Ava"
discount = 20

message = (
    f"Hi {customer_name}, "
    f"you have a {discount}% discount."
)

print(message)
</code></pre>
<p>This becomes particularly useful when generating personalized marketing messages.</p>
<h4 id="heading-numbers">Numbers</h4>
<p>Python supports integers and floating-point numbers.</p>
<p>An integer is a whole number:</p>
<pre><code class="language-python">customers = 1500
</code></pre>
<p>A float contains a decimal:</p>
<pre><code class="language-python">conversion_rate = 4.75
</code></pre>
<p>You can perform mathematical operations:</p>
<pre><code class="language-python">revenue = 15000
cost = 5000

profit = revenue - cost

print(profit)
</code></pre>
<p>You can multiply:</p>
<pre><code class="language-python">average_order_value = 75
orders = 120

revenue = average_order_value * orders

print(revenue)
</code></pre>
<p>And divide:</p>
<pre><code class="language-python">revenue = 12000
orders = 150

average_order_value = revenue / orders

print(average_order_value)
</code></pre>
<p>These basic operations form the foundation for many marketing metrics.</p>
<h4 id="heading-percentages">Percentages</h4>
<p>Marketing analytics uses percentages everywhere.</p>
<p>Suppose 500 people clicked an advertisement out of 10,000 impressions.</p>
<p>The click-through rate is:</p>
<pre><code class="language-python">clicks = 500
impressions = 10000

ctr = clicks / impressions * 100

print(ctr)
</code></pre>
<p>The result is:</p>
<pre><code class="language-text">5.0
</code></pre>
<p>You could format it like this:</p>
<pre><code class="language-python">print(f"CTR: {ctr:.2f}%")
</code></pre>
<p>The <code>.2f</code> tells Python to display two decimal places.</p>
<h4 id="heading-boolean-values">Boolean Values</h4>
<p>Boolean values represent two states:</p>
<pre><code class="language-python">True
False
</code></pre>
<p>For example:</p>
<pre><code class="language-python">campaign_active = True
</code></pre>
<p>You can use Booleans with conditions.</p>
<pre><code class="language-python">campaign_active = True

if campaign_active:
    print("Campaign is currently running.")
</code></pre>
<p>Booleans become particularly useful when your program needs to make decisions.</p>
<h4 id="heading-comparison-operators">Comparison Operators</h4>
<p>In marketing, one metric you'll often compare is <strong>return on ad spend (ROAS)</strong>. ROAS measures the revenue attributed to advertising relative to the amount spent on that advertising:</p>
<p><strong>ROAS = Revenue Attributed to Ads ÷ Advertising Cost</strong></p>
<p>For example, if a campaign generates $4,000 in attributed revenue from $1,000 in ad spend, its ROAS is 4.0, often described as 4:1: the campaign generated $4 in revenue for every $1 spent on ads.</p>
<p>ROAS isn't the same as profit because it doesn't automatically subtract product costs, agency fees, or other business expenses. The right target depends on the company's margins and goals.</p>
<p>Using Python, you can compare the values of ROAS using this simple line of code:</p>
<pre><code class="language-python">roas = 4.2

print(roas &gt; 3)
</code></pre>
<p>The result is:</p>
<pre><code class="language-text">True
</code></pre>
<p>Other comparisons include:</p>
<pre><code class="language-python">roas == 3
roas != 3
roas &gt;= 3
roas &lt;= 3
</code></pre>
<p>These comparisons allow you to build rules.</p>
<pre><code class="language-python">if roas &gt;= 3:
    print("Campaign is performing well.")
</code></pre>
<h4 id="heading-combining-conditions">Combining Conditions</h4>
<p>Marketing decisions often involve multiple conditions.</p>
<p>Suppose a campaign is considered promising when both ROAS is above 3 and conversion rate is above 4 percent.</p>
<pre><code class="language-python">roas = 3.8
conversion_rate = 5.2

if roas &gt;= 3 and conversion_rate &gt;= 4:
    print("Campaign meets performance criteria.")
</code></pre>
<p>The <code>and</code> operator requires both conditions to be true.</p>
<p>You can also use <code>or</code>:</p>
<pre><code class="language-python">if roas &gt;= 4 or conversion_rate &gt;= 6:
    print("Campaign has at least one strong performance signal.")
</code></pre>
<p>These simple logical operators become extremely useful when creating automated marketing rules.</p>
<h4 id="heading-lists">Lists</h4>
<p>A list stores multiple values.</p>
<pre><code class="language-python">channels = [
    "Email",
    "Search",
    "Social Media",
    "Display"
]
</code></pre>
<p>You can access an item using its position, like this:</p>
<pre><code class="language-python">print(channels[0])
</code></pre>
<p>The result is:</p>
<pre><code class="language-text">Email
</code></pre>
<p>Python uses zero-based indexing, meaning the first item is at position <code>0</code>.</p>
<p>You can add an item:</p>
<pre><code class="language-python">channels.append("Affiliate")
</code></pre>
<p>You can see how many items are in the list:</p>
<pre><code class="language-python">print(len(channels))
</code></pre>
<p>Lists are useful for storing collections of marketing information.</p>
<h4 id="heading-dictionaries">Dictionaries</h4>
<p>Dictionaries store information as key-value pairs.</p>
<pre><code class="language-python">campaign = {
    "name": "Holiday Sale",
    "channel": "Email",
    "budget": 5000,
    "conversions": 125
}
</code></pre>
<p>You can retrieve a value:</p>
<pre><code class="language-python">print(campaign["channel"])
</code></pre>
<p>You can also update a value:</p>
<pre><code class="language-python">campaign["budget"] = 7500
</code></pre>
<p>Dictionaries are particularly useful when representing a single customer, campaign, product, or marketing event.</p>
<p>For example:</p>
<pre><code class="language-python">customer = {
    "name": "Ava",
    "age": 24,
    "orders": 7,
    "total_spent": 850
}
</code></pre>
<p>You could access:</p>
<pre><code class="language-python">print(customer["total_spent"])
</code></pre>
<h4 id="heading-lists-of-dictionaries">Lists of Dictionaries</h4>
<p>Things become much more interesting when you combine lists and dictionaries.</p>
<pre><code class="language-python">campaigns = [
    {
        "name": "Spring Sale",
        "channel": "Email",
        "revenue": 8500
    },
    {
        "name": "Summer Sale",
        "channel": "Social Media",
        "revenue": 6200
    },
    {
        "name": "Holiday Sale",
        "channel": "Search",
        "revenue": 12000
    }
]
</code></pre>
<p>Now you have a simple representation of multiple campaigns.</p>
<p>You can loop through them:</p>
<pre><code class="language-python">for campaign in campaigns:
    print(campaign["name"])
</code></pre>
<p>This produces:</p>
<pre><code class="language-text">Spring Sale
Summer Sale
Holiday Sale
</code></pre>
<p>This is the beginning of working with collections of structured data.</p>
<h4 id="heading-loops">Loops</h4>
<p>A loop repeats an operation.</p>
<pre><code class="language-python">channels = [
    "Email",
    "Search",
    "Social Media",
    "Display"
]

for channel in channels:
    print(channel)
</code></pre>
<p>You could also perform calculations:</p>
<pre><code class="language-python">revenues = [5000, 7500, 9000, 6200]

for revenue in revenues:
    print(revenue * 0.1)
</code></pre>
<p>The loop processes each value.</p>
<p>Imagine having 10,000 customers. You wouldn't want to manually write the same operation 10,000 times.</p>
<p>A loop gives Python instructions for handling each record. Later in this book, you'll see how pandas can make this kind of work even more powerful and efficient.</p>
<h4 id="heading-conditional-logic-inside-loops">Conditional Logic Inside Loops</h4>
<p>You can combine loops and conditions.</p>
<pre><code class="language-python">campaigns = [
    {"name": "Spring Sale", "roas": 2.5},
    {"name": "Summer Sale", "roas": 4.8},
    {"name": "Holiday Sale", "roas": 1.2}
]

for campaign in campaigns:
    if campaign["roas"] &gt;= 3:
        print(
            f"{campaign['name']} is performing well."
        )
</code></pre>
<p>Only the campaigns meeting the condition are printed.</p>
<p>This is conceptually similar to filtering a marketing dataset.</p>
<h4 id="heading-functions">Functions</h4>
<p>Functions package reusable logic.</p>
<p>Suppose you repeatedly need to calculate ROAS. Instead of writing:</p>
<pre><code class="language-python">roas = revenue / cost
</code></pre>
<p>every time, create a function.</p>
<pre><code class="language-python">def calculate_roas(revenue, cost):
    return revenue / cost
</code></pre>
<p>Then:</p>
<pre><code class="language-python">roas = calculate_roas(
    15000,
    5000
)

print(roas)
</code></pre>
<p>You can reuse the same function:</p>
<pre><code class="language-python">campaign_one = calculate_roas(12000, 3000)
campaign_two = calculate_roas(8000, 4000)
campaign_three = calculate_roas(15000, 2500)
</code></pre>
<p>This is much cleaner.</p>
<h4 id="heading-adding-validation-to-functions">Adding Validation to Functions</h4>
<p>A good function should account for problematic inputs.</p>
<p>Consider:</p>
<pre><code class="language-python">def calculate_roas(revenue, cost):
    return revenue / cost
</code></pre>
<p>What happens if <code>cost</code> is zero?</p>
<p>Python will raise an error because division by zero is undefined.</p>
<p>A safer version is:</p>
<pre><code class="language-python">def calculate_roas(revenue, cost):
    if cost == 0:
        return None

    return revenue / cost
</code></pre>
<p>Now:</p>
<pre><code class="language-python">print(calculate_roas(10000, 0))
</code></pre>
<p>returns:</p>
<pre><code class="language-text">None
</code></pre>
<p>rather than crashing.</p>
<p>This idea of protecting your code from unexpected input becomes increasingly important when we begin working with real datasets.</p>
<h4 id="heading-returning-multiple-values">Returning Multiple Values</h4>
<p>A function can return multiple results.</p>
<pre><code class="language-python">def campaign_metrics(
    revenue,
    cost,
    clicks,
    conversions
):
    if cost == 0:
        roas = None
    else:
        roas = revenue / cost

    if clicks == 0:
        conversion_rate = None
    else:
        conversion_rate = (
            conversions / clicks * 100
        )

    return roas, conversion_rate
</code></pre>
<p>You can use it like this:</p>
<pre><code class="language-python">roas, conversion_rate = campaign_metrics(
    12000,
    4000,
    1000,
    80
)

print(roas)
print(conversion_rate)
</code></pre>
<p>Later, we'll replace or integrate functions like into reusable analytics modules.</p>
<h3 id="heading-chapter-2-libraries-in-python">Chapter 2: Libraries in Python</h3>
<h4 id="heading-importing-libraries">Importing Libraries</h4>
<p>Python's built-in functionality is useful, but marketing analytics requires additional tools.</p>
<p>Libraries provide pre-built functionality.</p>
<p>For example:</p>
<pre><code class="language-python">import pandas as pd
</code></pre>
<p>The <code>as pd</code> part gives pandas a shorter alias.</p>
<p>Now you can use:</p>
<pre><code class="language-python">pd.read_csv()
</code></pre>
<p>rather than:</p>
<pre><code class="language-python">pandas.read_csv()
</code></pre>
<p>You'll see this convention throughout data analysis projects.</p>
<p>NumPy is commonly imported as:</p>
<pre><code class="language-python">import numpy as np
</code></pre>
<p>Matplotlib's plotting module is commonly imported as:</p>
<pre><code class="language-python">import matplotlib.pyplot as plt
</code></pre>
<h4 id="heading-installing-libraries">Installing Libraries</h4>
<p>If a library isn't already installed, use pip.</p>
<pre><code class="language-bash">pip install pandas
</code></pre>
<p>For several libraries:</p>
<pre><code class="language-bash">pip install pandas numpy matplotlib scikit-learn requests openpyxl
</code></pre>
<p>The exact installation process can vary depending on your Python environment.</p>
<p>If you use a virtual environment, it's a good practice to install project dependencies inside that environment rather than globally.</p>
<h4 id="heading-virtual-environments">Virtual Environments</h4>
<p>Suppose one project requires one version of a library and another project requires a different version. Installing everything globally can create conflicts.</p>
<p>A virtual environment gives a project its own isolated Python environment.</p>
<p>You can create one like this:</p>
<pre><code class="language-bash">python -m venv marketing-env
</code></pre>
<p>And then activate it like this on Windows:</p>
<pre><code class="language-bash">marketing-env\Scripts\activate
</code></pre>
<p>Or like this on macOS or Linux:</p>
<pre><code class="language-bash">source marketing-env/bin/activate
</code></pre>
<p>Then install your packages:</p>
<pre><code class="language-bash">pip install pandas numpy matplotlib scikit-learn
</code></pre>
<p>You can save your dependencies:</p>
<pre><code class="language-bash">pip freeze &gt; requirements.txt
</code></pre>
<p>Another developer can then install them with:</p>
<pre><code class="language-bash">pip install -r requirements.txt
</code></pre>
<p>This becomes useful when you turn your capstone into a shareable project.</p>
<h2 id="heading-part-ii-working-with-marketing-data">Part II: Working With Marketing Data</h2>
<p>Marketing data rarely arrives in the exact format you want. That's one of the first realities you need to accept.</p>
<p>A spreadsheet might contain missing values, an exported report might have inconsistent capitalization, a CRM might use one customer ID format while an advertising platform uses another, dates might appear in different formats, or numbers might be stored as strings.</p>
<p>A dataset can look perfectly normal while containing subtle problems that produce incorrect analysis.</p>
<p>Before you calculate anything, understand what you're working with.</p>
<h3 id="heading-understanding-the-marketing-data-lifecycle">Understanding the Marketing Data Lifecycle</h3>
<p>A useful way to think about marketing analytics is as a lifecycle.</p>
<ol>
<li>Data is first collected.</li>
<li>Then it's stored.</li>
<li>Then it's cleaned.</li>
<li>Then it's transformed.</li>
<li>Then it's analyzed.</li>
<li>Then the findings are communicated.</li>
<li>Finally, someone uses those findings to make a decision.</li>
</ol>
<p>Python can participate in almost every stage.</p>
<p>For example, Python can retrieve information from an API, store it in a DataFrame, clean it with pandas, calculate KPIs, create visualizations, train a machine learning model, and export a report.</p>
<p>That's why Python becomes more valuable as your marketing workflow becomes more complex.</p>
<h3 id="heading-structured-and-unstructured-data">Structured and Unstructured Data</h3>
<p>Structured data is information organized into predictable fields.</p>
<p>A spreadsheet containing:</p>
<pre><code class="language-text">customer_id
date
campaign
channel
revenue
</code></pre>
<p>is structured data.</p>
<p>Each row represents a record, and each column represents a variable. This is the kind of data pandas handles particularly well.</p>
<p>Unstructured data is different. Examples include:</p>
<ul>
<li>Social media posts</li>
<li>Customer reviews</li>
<li>Email text</li>
<li>Survey responses</li>
<li>Images</li>
<li>Videos</li>
</ul>
<p>Python can also work with unstructured data, but that often requires additional techniques such as natural language processing or computer vision.</p>
<p>This book primarily focuses on structured marketing data because it provides a strong foundation for analytics, automation, and machine learning.</p>
<h3 id="heading-rows-and-columns">Rows and Columns</h3>
<p>Suppose your dataset contains:</p>
<pre><code class="language-text">customer_id | channel | clicks | conversions | revenue
101         | Email   | 500    | 40          | 3200
102         | Search  | 800    | 65          | 5100
103         | Social  | 300    | 18          | 1400
</code></pre>
<p>Each row represents an observation, while each column represents an attribute.</p>
<p>Understanding this distinction is important because many pandas operations operate either across rows, across columns, or across groups of rows.</p>
<h4 id="heading-the-unit-of-analysis">The Unit of Analysis</h4>
<p>One concept that deserves special attention is the unit of analysis.</p>
<p>Ask: <strong>What does one row represent?</strong> Does it represent:</p>
<ul>
<li>A customer?</li>
<li>A purchase?</li>
<li>A campaign?</li>
<li>An advertisement?</li>
<li>A website session?</li>
<li>An email?</li>
<li>A day?</li>
</ul>
<p>This matters enormously.</p>
<p>Imagine you have a dataset where every row represents a purchase. A customer who made ten purchases will appear ten times.</p>
<p>If you count rows and call that the number of customers, your analysis will be wrong.</p>
<p>You would instead need the number of unique customer IDs.</p>
<pre><code class="language-python">df["customer_id"].nunique()
</code></pre>
<p>The unit of analysis should always be understood before calculating metrics.</p>
<h3 id="heading-marketing-metrics-are-context-dependent">Marketing Metrics Are Context-Dependent</h3>
<p>Suppose Campaign A has a 10 percent conversion rate.</p>
<p>That sounds excellent, but what if it generated only ten visitors and one conversion?</p>
<p>Now suppose Campaign B has a 4 percent conversion rate but generated 100,000 visitors and 4,000 conversions.</p>
<p>The percentage alone doesn't tell the complete story.</p>
<p>This is why good marketing analytics considers other factors, like:</p>
<ul>
<li>Volume</li>
<li>Rate</li>
<li>Cost</li>
<li>Revenue</li>
<li>Profitability</li>
<li>Customer quality</li>
<li>Time period</li>
<li>Business objective</li>
</ul>
<p>A metric isn't an answer by itself. It's evidence.</p>
<h3 id="heading-chapter-3-working-with-csv-data">Chapter 3: Working With CSV Data</h3>
<p>CSV stands for Comma-Separated Values, and it's one of the simplest and most common ways of exchanging tabular data.</p>
<p>Suppose <code>campaigns.csv</code> contains:</p>
<pre><code class="language-text">campaign,channel,impressions,clicks,conversions,cost,revenue
Spring Sale,Email,10000,850,120,500,6000
Summer Sale,Social Media,25000,1100,75,1200,4200
Holiday Sale,Search,18000,1400,180,2000,9500
</code></pre>
<p>Load it:</p>
<pre><code class="language-python">import pandas as pd

df = pd.read_csv("campaigns.csv")
</code></pre>
<p>Inspect the first five rows:</p>
<pre><code class="language-python">df.head()
</code></pre>
<p>You can see more rows if you want:</p>
<pre><code class="language-python">df.head(10)
</code></pre>
<p>Or the last five:</p>
<pre><code class="language-python">df.tail()
</code></pre>
<h4 id="heading-understanding-dataframe-shape">Understanding DataFrame Shape</h4>
<p>Use:</p>
<pre><code class="language-python">df.shape
</code></pre>
<p>You might receive:</p>
<pre><code class="language-text">(1000, 8)
</code></pre>
<p>That means the DataFrame contains 1,000 rows and 8 columns.</p>
<p>This is one of the quickest ways to understand the size of your dataset.</p>
<h4 id="heading-viewing-column-names">Viewing Column Names</h4>
<pre><code class="language-python">df.columns
</code></pre>
<p>You might see:</p>
<pre><code class="language-text">Index([
    'campaign',
    'channel',
    'impressions',
    'clicks',
    'conversions',
    'cost',
    'revenue'
], dtype='object')
</code></pre>
<p>If you're writing reusable code, knowing the exact column names is important.</p>
<h4 id="heading-selecting-a-column">Selecting a Column</h4>
<p>Select one column:</p>
<pre><code class="language-python">df["revenue"]
</code></pre>
<p>Or select multiple columns:</p>
<pre><code class="language-python">df[
    [
        "campaign",
        "revenue",
        "cost"
    ]
]
</code></pre>
<p>This lets you create smaller views of your dataset.</p>
<h4 id="heading-selecting-rows">Selecting Rows</h4>
<p>You can select rows using conditions.</p>
<pre><code class="language-python">df[df["revenue"] &gt; 5000]
</code></pre>
<p>This returns campaigns where revenue is greater than $5,000.</p>
<p>You can combine conditions:</p>
<pre><code class="language-python">df[
    (df["revenue"] &gt; 5000) &amp;
    (df["cost"] &lt; 3000)
]
</code></pre>
<p>Parentheses are important when combining pandas conditions because each comparison needs to be evaluated as a complete condition before pandas combines the results. Operators such as <code>&amp;</code> and <code>|</code> have different precedence rules from Python's ordinary <code>and</code> and <code>or</code>, so leaving out parentheses can cause an error or an unintended expression.</p>
<p>Put each condition in its own parentheses, and use <code>&amp;</code> for AND and <code>|</code> for OR when filtering a DataFrame.</p>
<p>For example:</p>
<pre><code class="language-python"># Select campaigns with more than 1,000 clicks
# and a conversion rate above 2%
filtered_campaigns = df[
    (df["clicks"] &gt; 1000) &amp;
    (df["conversion_rate"] &gt; 0.02)
]
</code></pre>
<p>Here, both conditions must be true for a row to be included in <code>filtered_campaigns</code>.Sorting Data</p>
<p>Sort by revenue:</p>
<pre><code class="language-python">df.sort_values(
    "revenue",
    ascending=False
)
</code></pre>
<p>This places the highest-revenue campaigns first.</p>
<p>Sort by cost:</p>
<pre><code class="language-python">df.sort_values(
    "cost",
    ascending=True
)
</code></pre>
<p>You can sort by multiple columns:</p>
<pre><code class="language-python">df.sort_values(
    ["channel", "revenue"],
    ascending=[True, False]
)
</code></pre>
<p>This first groups the results by channel and then orders revenue within each channel.</p>
<h3 id="heading-chapter-4-working-with-excel">Chapter 4: Working With Excel</h3>
<p>Many marketing departments still rely heavily on Excel.</p>
<p>Fortunately, pandas makes it relatively easy to work with Excel files.</p>
<pre><code class="language-python">df = pd.read_excel(
    "marketing_data.xlsx"
)
</code></pre>
<p>You can specify a worksheet:</p>
<pre><code class="language-python">df = pd.read_excel(
    "marketing_data.xlsx",
    sheet_name="Campaigns"
)
</code></pre>
<p>You can inspect available sheets using:</p>
<pre><code class="language-python">excel_file = pd.ExcelFile(
    "marketing_data.xlsx"
)

print(excel_file.sheet_names)
</code></pre>
<p>This is useful when working with large workbooks.</p>
<h4 id="heading-exporting-data-to-excel">Exporting Data to Excel</h4>
<p>You can export a DataFrame:</p>
<pre><code class="language-python">df.to_excel(
    "marketing_report.xlsx",
    index=False
)
</code></pre>
<p>You can create multiple worksheets:</p>
<pre><code class="language-python">with pd.ExcelWriter(
    "marketing_report.xlsx"
) as writer:

    df.to_excel(
        writer,
        sheet_name="Data",
        index=False
    )

    channel_summary.to_excel(
        writer,
        sheet_name="Channels"
    )
</code></pre>
<p>This capability will become important in the capstone because the final system will produce a structured marketing report.</p>
<h3 id="heading-chapter-5-dates-and-time">Chapter 5: Dates and Time</h3>
<p>Marketing performance is highly dependent on time.</p>
<p>A campaign that generated $20,000 in revenue over six months is very different from one that generated $20,000 in revenue over two days.</p>
<p>Start by converting date columns:</p>
<pre><code class="language-python">df["date"] = pd.to_datetime(
    df["date"]
)
</code></pre>
<p>Now you can extract components.</p>
<pre><code class="language-python">df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["day"] = df["date"].dt.day
</code></pre>
<p>You can extract the month name:</p>
<pre><code class="language-python">df["month_name"] = (
    df["date"]
    .dt.month_name()
)
</code></pre>
<p>And weekday:</p>
<pre><code class="language-python">df["weekday"] = (
    df["date"]
    .dt.day_name()
)
</code></pre>
<h4 id="heading-time-based-marketing-questions">Time-Based Marketing Questions</h4>
<p>Once dates are properly formatted, you can answer questions such as:</p>
<ul>
<li>Which month generated the most revenue?</li>
<li>Which weekday has the highest conversion rate?</li>
<li>Did campaign performance improve after a specific change?</li>
<li>Is there seasonal behavior?</li>
<li>Are customer purchases becoming less frequent?</li>
</ul>
<p>For example:</p>
<pre><code class="language-python">monthly_revenue = (
    df.groupby(
        df["date"].dt.to_period("M")
    )["revenue"]
    .sum()
)
</code></pre>
<p>This produces monthly revenue totals.</p>
<h3 id="heading-chapter-6-working-with-marketing-apis">Chapter 6: Working With Marketing APIs</h3>
<p>A file is useful when someone manually exports data. But what if you want your marketing intelligence system to collect data automatically?</p>
<p>This is where APIs become important.</p>
<p>An API, or Application Programming Interface, provides a structured way for one program to communicate with another.</p>
<p>For example, a platform might expose an endpoint that returns campaign information.</p>
<p>Python can send a request:</p>
<pre><code class="language-python">import requests

response = requests.get(
    "https://example.com/api/campaigns"
)

print(response.status_code)
</code></pre>
<p>A successful response commonly has a status code such as:</p>
<pre><code class="language-text">200
</code></pre>
<p>You can inspect the response body:</p>
<pre><code class="language-python">print(response.text)
</code></pre>
<p>If the response contains JSON:</p>
<pre><code class="language-python">data = response.json()
</code></pre>
<p>You can inspect it:</p>
<pre><code class="language-python">print(data)
</code></pre>
<h4 id="heading-api-authentication">API Authentication</h4>
<p>Real APIs often require authentication.</p>
<p>For example:</p>
<pre><code class="language-python">api_key = "your-api-key"
</code></pre>
<p>But you should <strong>not</strong> place actual secret keys directly inside code that you intend to share publicly.</p>
<p>Instead, use environment variables.</p>
<pre><code class="language-python">import os

api_key = os.getenv(
    "MARKETING_API_KEY"
)
</code></pre>
<p>You can then construct headers:</p>
<pre><code class="language-python">headers = {
    "Authorization": f"Bearer {api_key}"
}
</code></pre>
<p>And make the request:</p>
<pre><code class="language-python">response = requests.get(
    "https://example.com/api/campaigns",
    headers=headers
)
</code></pre>
<p>The exact authentication system depends on the API.</p>
<h4 id="heading-api-errors">API Errors</h4>
<p>Your program shouldn't assume that every request succeeds.</p>
<pre><code class="language-python">response = requests.get(
    api_url,
    timeout=30
)

if response.status_code == 200:
    data = response.json()
else:
    print(
        f"Request failed: "
        f"{response.status_code}"
    )
</code></pre>
<p>When you send an API request, receiving a response doesn't necessarily mean the request succeeded. The server might return an error status, such as <code>404 Not Found</code> or <code>500 Internal Server Error</code>.</p>
<p>The <code>raise_for_status()</code> method checks the HTTP response status. If the response indicates an unsuccessful request, it raises an exception instead of allowing your program to continue as if everything worked.</p>
<p>A <code>try</code> block lets you handle that exception:</p>
<pre><code class="language-python">import requests

try:
    response = requests.get(
        "https://example.com/api/campaigns",
        timeout=30
    )
    response.raise_for_status()
    campaigns = response.json()

except requests.exceptions.RequestException as error:
    print(f"Could not retrieve campaign data: {error}")
</code></pre>
<p>Here, <code>requests.get()</code> sends the request, <code>timeout=30</code> prevents the program from waiting indefinitely, and <code>raise_for_status()</code> raises an exception for an unsuccessful HTTP response. The <code>except</code> block catches request-related errors and displays a message.</p>
<p>In a production workflow, you should also log the error and decide whether the process should retry, skip the failed source, or stop report generation. Avoid silently treating a failed request as an empty dataset because that could produce misleading marketing reports.</p>
<p>This becomes especially important when your system runs automatically. If an API goes down at 3 AM, you want the program to report the problem rather than silently producing an empty marketing report.</p>
<h4 id="heading-api-pagination">API Pagination</h4>
<p>Some APIs return only a limited number of records per request. If an account contains thousands of campaign records, for example, the API might return the first 100 and provide a link or token for retrieving the next group. This process is called <strong>pagination</strong>.</p>
<p>The pagination method depends on the API. Some services use page numbers, others use offsets, and some provide a continuation token or a URL for the next page. Always check the API's documentation before implementing pagination.</p>
<p>The following example demonstrates an API that returns a <code>results</code> list and a <code>next</code> URL:</p>
<pre><code class="language-python">import requests

def fetch_all_campaigns(api_url, headers):
    all_campaigns = []
    next_url = api_url

    while next_url:
        response = requests.get(
            next_url,
            headers=headers,
            timeout=30
        )
        response.raise_for_status()
        payload = response.json()

        all_campaigns.extend(payload["results"])
        next_url = payload.get("next")

    return all_campaigns
</code></pre>
<p>The function starts with the initial API URL and continues requesting pages while <code>next_url</code> contains a URL. The <code>extend()</code> method adds the records from each page to the same list. When the API returns no next-page URL, the loop stops.</p>
<p>This is an illustrative pattern, not a universal solution. Your API might use different field names or require a different pagination method. For a real integration, follow the service's documentation and consider safeguards such as retry limits and maximum page counts.</p>
<h4 id="heading-api-rate-limits">API Rate Limits</h4>
<p>APIs often limit how many requests a client can make within a particular period. These restrictions are called <strong>rate limits</strong>. For example, a service might allow only a certain number of requests per minute.</p>
<p>If your marketing automation script sends too many requests, the API may return an HTTP <code>429 Too Many Requests</code> response. Instead of repeatedly sending requests immediately, your program should pause and try again according to the service's rules.</p>
<p>Some APIs provide a <code>Retry-After</code> response header that indicates how long the client should wait. Here's a simplified example:</p>
<pre><code class="language-python">import time
import requests

def get_with_rate_limit(url, headers, max_retries=5):
    for attempt in range(max_retries):
        response = requests.get(
            url,
            headers=headers,
            timeout=30
        )

        if response.status_code == 429:
            retry_after = response.headers.get(
                "Retry-After", "2"
            )
            time.sleep(float(retry_after))
            continue

        response.raise_for_status()
        return response

    raise RuntimeError(
        "API request failed after repeated rate-limit responses."
    )
</code></pre>
<p>This function retries a limited number of times when the API returns a <code>429</code> response. If the response includes a <code>Retry-After</code> header, the function uses that value to determine how long to wait. If the header is absent, it waits for two seconds.</p>
<p>This is a starting point rather than a complete production retry system. Some APIs use different retry rules, and production systems should also consider temporary server errors, logging, exponential backoff, and safeguards against excessive retries. Always follow the API provider's documentation.</p>
<p>Rate-limit handling is particularly important when automating reports across multiple campaigns or accounts. Respecting these limits makes your integration more reliable and reduces the chance of interrupted data collection.</p>
<h2 id="heading-part-iii-data-cleaning-and-analysis-with-pandas">Part III: Data Cleaning and Analysis with Pandas</h2>
<p>The first rule of marketing analytics should be simple: <strong>Don't trust your dataset until you've inspected it.</strong></p>
<p>That doesn't mean your data is bad. It just means you should verify it before analyzing it and making decisions based off of it.</p>
<p>A dataset can contain millions of perfectly formatted rows and still represent the wrong thing.</p>
<h3 id="heading-chapter-7-inspecting-data-quality">Chapter 7: Inspecting Data Quality</h3>
<p>Start with:</p>
<pre><code class="language-python">df.info()
</code></pre>
<p>This provides information about:</p>
<ul>
<li>Number of rows</li>
<li>Column names</li>
<li>Missing values</li>
<li>Data types</li>
<li>Memory usage</li>
</ul>
<p>You can also use:</p>
<pre><code class="language-python">df.describe()
</code></pre>
<p>This gives statistical information for numerical columns.</p>
<p>For example, you might discover that:</p>
<pre><code class="language-text">revenue
count    1000
mean     5820
min      -250
max      85000
</code></pre>
<p>The negative minimum should immediately make you investigate.</p>
<p>A negative revenue value might be legitimate if it represents a refund, or it could indicate a data-entry error.</p>
<p>The number alone doesn't tell you which.</p>
<h4 id="heading-missing-values">Missing Values</h4>
<p>Check for missing values with:</p>
<pre><code class="language-python">df.isnull().sum()
</code></pre>
<p>You might see:</p>
<pre><code class="language-text">customer_id       0
campaign          3
channel           0
revenue          12
cost              4
</code></pre>
<p>Now you know exactly where missing information exists.</p>
<h4 id="heading-why-missing-data-matters">Why Missing Data Matters</h4>
<p>Suppose you calculate average revenue:</p>
<pre><code class="language-python">df["revenue"].mean()
</code></pre>
<p>pandas generally excludes missing values from the calculation.</p>
<p>That may be convenient, but you still need to understand why the values are missing.</p>
<p>If the missing values are random, one approach may work. If all missing revenue values come from one specific sales channel, simply ignoring them could bias your conclusions.</p>
<p>Missing data isn't merely a technical problem. It can change the conclusions you draw from your marketing analysis.</p>
<p>Imagine that several campaign records have no revenue value. If you replace every missing value with zero, you're assuming those campaigns generated no revenue. But a missing value could mean that revenue tracking failed, the data hasn't arrived yet, or the purchase information is stored somewhere else.</p>
<p>Those situations aren't equivalent.</p>
<p>Before filling or removing missing values, ask yourself:</p>
<ul>
<li>Why is the value missing?</li>
<li>Does a missing value genuinely mean zero, or is the information unknown?</li>
<li>How many records are affected?</li>
<li>Could removing those records bias the analysis?</li>
</ul>
<p>For example, filling missing advertising costs with zero could cause a campaign's return on ad spend to become misleading or undefined. Similarly, removing every row with a missing customer attribute could exclude an important group of customers from your analysis.</p>
<p>Use <code>fillna()</code> or <code>dropna()</code> only after deciding what the missing values mean in the context of your dataset. Document important assumptions so that other people can understand how the results were produced.</p>
<h4 id="heading-duplicate-records">Duplicate Records</h4>
<p>Check duplicates like this:</p>
<pre><code class="language-python">df.duplicated().sum()
</code></pre>
<p>If the result is 25, you have 25 rows that are exact duplicates.</p>
<p>You can remove them like this:</p>
<pre><code class="language-python">df = df.drop_duplicates()
</code></pre>
<p>But remember the unit of analysis.</p>
<p>Two rows with the same customer ID aren't necessarily duplicates, as a customer can make multiple purchases.</p>
<p>Two rows with the same campaign name aren't necessarily duplicates, as a campaign may have data recorded across multiple days.</p>
<p>Never delete records merely because they share one or two values. Always check those values first.</p>
<h3 id="heading-chapter-8-cleaning-text-data">Chapter 8: Cleaning Text Data</h3>
<p>Marketing datasets often contain inconsistent text.</p>
<p>You may find:</p>
<pre><code class="language-text">Email
email
EMAIL
Email Marketing
</code></pre>
<p>Python treats these as different values.</p>
<p>Start by standardizing case:</p>
<pre><code class="language-python">df["channel"] = (
    df["channel"]
    .str.lower()
)
</code></pre>
<p>Then remove unnecessary whitespace:</p>
<pre><code class="language-python">df["channel"] = (
    df["channel"]
    .str.strip()
)
</code></pre>
<p>Now this:</p>
<pre><code class="language-text">Email
email
EMAIL
</code></pre>
<p>will all become standardized to this:</p>
<pre><code class="language-text">email
</code></pre>
<p>You can also standardize specific labels:</p>
<pre><code class="language-python">df["channel"] = df["channel"].replace({
    "email marketing": "email",
    "paid social": "social media",
    "social": "social media"
})
</code></pre>
<h4 id="heading-why-standardization-matters">Why Standardization Matters</h4>
<p>Imagine you calculate revenue by channel before cleaning the data.</p>
<p>You might get:</p>
<pre><code class="language-text">email            $20,000
Email            $15,000
EMAIL             $8,000
</code></pre>
<p>The business may actually have generated $43,000 from one channel, but if you haven't standardized, your analysis would incorrectly treat those as separate channels.</p>
<p>This is a small example of how data quality can affect business decisions.</p>
<h3 id="heading-chapter-9-cleaning-numeric-data">Chapter 9: Cleaning Numeric Data</h3>
<p>Numbers sometimes arrive in a dataset as text instead of as numerical values. This is especially common when working with marketing reports exported from spreadsheets, advertising platforms, or customer relationship management (CRM) systems.</p>
<p>For example, a revenue column might contain these values:</p>
<pre><code class="language-python">"$5,000"
"$7,500"
"$2,300"
</code></pre>
<p>Although these values represent money, Python may store them as strings because they contain characters such as dollar signs and commas. A string is a sequence of characters, not a number that Python can directly use in mathematical calculations.</p>
<p>For example, Python can't reliably calculate a total or average from strings containing currency symbols. Before calculating marketing metrics, you need to clean these values and convert them into an appropriate numeric type.</p>
<h4 id="heading-removing-currency-symbols-and-separators">Removing Currency Symbols and Separators</h4>
<p>The first step is to remove characters that prevent Python from interpreting the values as numbers.</p>
<pre><code class="language-python">df["revenue"] = (
    df["revenue"]
    .str.replace("$", "", regex=False)
    .str.replace(",", "", regex=False)
)
</code></pre>
<p>Here, “remove symbols” means removing formatting characters such as <code>$</code> and <code>,</code> from the text. The code transforms <code>"$5,000"</code> into <code>"5000"</code>. Notice that the result is still a string. Cleaning the text doesn't automatically turn it into a numeric value.</p>
<h4 id="heading-converting-the-cleaned-values-into-numbers">Converting the Cleaned Values Into Numbers</h4>
<p>After removing the formatting characters, convert the cleaned strings into numeric values:</p>
<pre><code class="language-python">df["revenue"] = pd.to_numeric(
    df["revenue"],
    errors="coerce"
)
</code></pre>
<p>In this example, <strong>convert</strong> means changing the cleaned text into a numeric data type that Python can use for addition, subtraction, multiplication, division, and statistical calculations.</p>
<p>The <code>errors="coerce"</code> argument tells pandas to replace values it can't interpret as numbers with <code>NaN</code>, which represents a missing or unavailable numeric value. This prevents invalid text from silently being treated as a valid number, but it also means you should investigate any values that become <code>NaN</code>.</p>
<p>You can identify the affected records like this:</p>
<pre><code class="language-python">df[df["revenue"].isna()]
</code></pre>
<p>If the original column might contain missing values, save a copy before cleaning so you can distinguish values that were already missing from values that became invalid during conversion.</p>
<pre><code class="language-python">original_revenue = df["revenue"].copy()

cleaned_revenue = (
    original_revenue
    .str.replace("$", "", regex=False)
    .str.replace(",", "", regex=False)
)

numeric_revenue = pd.to_numeric(
    cleaned_revenue,
    errors="coerce"
)

invalid_revenue = (
    cleaned_revenue.notna()
    &amp; numeric_revenue.isna()
)

df["revenue"] = numeric_revenue

print(df.loc[invalid_revenue, "revenue"])
</code></pre>
<p>For more detailed investigations, keep the original values in a separate column before replacing them.</p>
<h4 id="heading-checking-for-invalid-values">Checking for Invalid Values</h4>
<p>Converting text into numbers doesn't guarantee that the resulting values make sense. A dataset might contain negative conversion rates or unusually large percentages.</p>
<p>For example:</p>
<pre><code class="language-text">conversion_rate
5.2
4.8
-17
250
</code></pre>
<p>A conversion rate of <code>250%</code> is suspicious if the metric represents conversions divided by clicks and each click can produce no more than one conversion. But it might be valid under a different definition, such as when the metric counts multiple conversion events per click. A negative conversion rate would generally require investigation.</p>
<p>You can find values outside the usual range for a conventional conversion rate with:</p>
<pre><code class="language-python">df[
    (df["conversion_rate"] &lt; 0)
    | (df["conversion_rate"] &gt; 100)
]
</code></pre>
<p>This code identifies records for review. It doesn't automatically determine whether they're errors. Check the metric definition, source data, and business context before correcting or removing any records.</p>
<p>So remember: clean formatting characters, convert the cleaned values into the correct data type, and validate the results before using them in marketing calculations.</p>
<h3 id="heading-chapter-10-creating-marketing-metrics-with-pandas">Chapter 10: Creating Marketing Metrics With pandas</h3>
<p>Raw marketing data tells you what happened, but calculated metrics help you evaluate performance. With pandas, you can create new columns that measure how efficiently campaigns turn impressions into clicks, clicks into conversions, and advertising spending into revenue.</p>
<p>Before calculating these metrics, make sure the columns contain numeric values and that the data uses consistent units.</p>
<h4 id="heading-calculating-common-marketing-metrics">Calculating Common Marketing Metrics</h4>
<p><strong>Click-through rate (CTR)</strong> measures the percentage of impressions that resulted in clicks.</p>
<pre><code class="language-python">df["ctr"] = (
    df["clicks"]
    .div(df["impressions"])
    .mul(100)
)
</code></pre>
<p><strong>Conversion rate</strong> measures the percentage of clicks that resulted in conversions.</p>
<pre><code class="language-python">df["conversion_rate"] = (
    df["conversions"]
    .div(df["clicks"])
    .mul(100)
)
</code></pre>
<p><strong>Cost per click (CPC)</strong> measures the average advertising cost for each click.</p>
<pre><code class="language-python">df["cpc"] = (
    df["cost"]
    .div(df["clicks"])
)
</code></pre>
<p><strong>Cost per acquisition (CPA)</strong> measures the average cost of generating a conversion or acquisition, depending on how the business defines an acquisition.</p>
<pre><code class="language-python">df["cpa"] = (
    df["cost"]
    .div(df["conversions"])
)
</code></pre>
<p><strong>Return on advertising spend (ROAS)</strong> measures the revenue attributed to advertising relative to the advertising cost.</p>
<p>You can calculate ROAS in pandas like this (as we covered before):</p>
<pre><code class="language-python">df["roas"] = (
    df["revenue"]
    .div(df["cost"])
)
</code></pre>
<p>The correct interpretation of each metric depends on the definitions used in your dataset. For example, a conversion might represent a purchase, a signup, a qualified lead, or another desired action.</p>
<h4 id="heading-protecting-against-zero-values">Protecting Against Zero Values</h4>
<p>Marketing datasets can contain campaigns with zero clicks, zero conversions, or zero advertising cost. These values require careful handling because division by zero is undefined.</p>
<p>Consider this calculation:</p>
<pre><code class="language-python">df["conversion_rate"] = (
    df["conversions"]
    .div(df["clicks"])
    .mul(100)
)
</code></pre>
<p>If a campaign has zero clicks, the conversion-rate calculation has a denominator of zero. That means there's no valid click-based rate to calculate for that campaign. Similarly, CPA can't be calculated when there are zero conversions, and ROAS can't be calculated when advertising cost is zero.</p>
<p>A safer approach is to replace zero denominators with missing values before performing the division:</p>
<pre><code class="language-python">df["conversion_rate"] = (
    df["conversions"]
    .div(df["clicks"].replace(0, float("nan")))
    .mul(100)
)

df["cpa"] = (
    df["cost"]
    .div(df["conversions"].replace(0, float("nan")))
)

df["roas"] = (
    df["revenue"]
    .div(df["cost"].replace(0, float("nan")))
)
</code></pre>
<p>This approach is safer because it avoids presenting an undefined calculation as a meaningful result. When clicks are zero, the conversion rate becomes missing rather than being reported as a valid percentage. When conversions are zero, CPA is unavailable rather than a finite cost per acquisition. When cost is zero, ROAS is unavailable rather than being treated as a valid ratio.</p>
<p>A missing value doesn't mean the metric is zero. For example:</p>
<ul>
<li><strong>Zero conversions with 100 clicks:</strong> the conversion rate is 0%.</li>
<li><strong>Zero clicks:</strong> the click-based conversion rate can't be calculated.</li>
<li><strong>Zero advertising cost:</strong> ROAS can't be calculated using revenue divided by cost.</li>
</ul>
<p>These situations have different business meanings and shouldn't be treated as interchangeable.</p>
<p>You can check which records have unavailable metrics:</p>
<pre><code class="language-python">df[
    df[
        ["conversion_rate", "cpa", "roas"]
    ].isna().any(axis=1)
]
</code></pre>
<p>Investigate these records before using the metrics in reports or making campaign decisions.</p>
<h4 id="heading-interpreting-the-results">Interpreting the Results</h4>
<p>Creating metrics is only the beginning. A high CTR doesn't necessarily mean a campaign generates valuable customers, and a high ROAS doesn't automatically mean the campaign is profitable.</p>
<p>Use multiple metrics together. CTR can help evaluate engagement, conversion rate can help evaluate how effectively clicks lead to the desired action, CPA can help evaluate acquisition efficiency, and ROAS can help evaluate attributed revenue relative to advertising cost.</p>
<p>The goal isn't just maximizing every metric independently. You want to understand how the metrics work together and whether they support the business objective.</p>
<h3 id="heading-chapter-11-filtering-data">Chapter 11: Filtering Data</h3>
<p>Filtering allows you to ask targeted questions. You could specifically search for:</p>
<p>Campaigns with ROAS above 3:</p>
<pre><code class="language-python">high_roas = df[
    df["roas"] &gt; 3
]
</code></pre>
<p>Campaigns with high conversion rates:</p>
<pre><code class="language-python">high_conversion = df[
    df["conversion_rate"] &gt; 5
]
</code></pre>
<p>Campaigns that satisfy both:</p>
<pre><code class="language-python">strong_campaigns = df[
    (df["roas"] &gt; 3) &amp;
    (df["conversion_rate"] &gt; 5)
]
</code></pre>
<p>Campaigns that satisfy either:</p>
<pre><code class="language-python">promising_campaigns = df[
    (df["roas"] &gt; 4) |
    (df["conversion_rate"] &gt; 7)
]
</code></pre>
<h4 id="heading-filtering-by-text">Filtering by Text</h4>
<p>Email campaigns:</p>
<pre><code class="language-python">email = df[
    df["channel"] == "email"
]
</code></pre>
<p>You can search for text:</p>
<pre><code class="language-python">holiday = df[
    df["campaign"]
    .str.contains(
        "holiday",
        case=False,
        na=False
    )
]
</code></pre>
<p>This is useful when campaign names follow patterns.</p>
<h3 id="heading-chapter-12-groupby-the-core-of-marketing-analysis">Chapter 12: GroupBy, The Core of Marketing Analysis</h3>
<p>One of the most useful pandas operations for marketing analytics is <code>groupby()</code>. It allows you to organize rows into groups that share a value and calculate summaries for each group.</p>
<p>Imagine that your dataset contains campaign results from several marketing channels, including email, search, social media, and display advertising. Looking at every campaign separately can make it difficult to understand which channels contribute the most revenue or generate the most conversions.</p>
<p>With <code>groupby()</code>, you can combine the campaign records by channel and calculate metrics for each channel.</p>
<h4 id="heading-grouping-campaigns-by-channel">Grouping Campaigns by Channel</h4>
<p>Suppose your dataset contains a column named <code>channel</code> and another named <code>revenue</code>. You can calculate total revenue for each channel:</p>
<pre><code class="language-python">channel_revenue = (
    df.groupby("channel")["revenue"]
    .sum()
)

print(channel_revenue)
</code></pre>
<p>The code groups all rows with the same channel name, selects the revenue column, and adds the revenue values within each group.</p>
<p>You can use the same approach to calculate average revenue:</p>
<pre><code class="language-python">channel_average = (
    df.groupby("channel")["revenue"]
    .mean()
)
</code></pre>
<p>Count the number of campaign records in each channel:</p>
<pre><code class="language-python">campaign_count = (
    df.groupby("channel")
    .size()
)
</code></pre>
<p>Or calculate total conversions:</p>
<pre><code class="language-python">channel_conversions = (
    df.groupby("channel")["conversions"]
    .sum()
)
</code></pre>
<p>Each result answers a different question. Total revenue measures the revenue associated with a channel, average revenue describes the average revenue per record, campaign count indicates how many records are present, and total conversions measures the number of conversions attributed to the channel.</p>
<h4 id="heading-creating-a-complete-marketing-report">Creating a Complete Marketing Report</h4>
<p>Marketing decisions usually require more than one metric. Instead of calculating each summary separately, you can use <code>agg()</code> to calculate multiple totals in one operation.</p>
<pre><code class="language-python">channel_summary = (
    df.groupby("channel")
    .agg({
        "impressions": "sum",
        "clicks": "sum",
        "conversions": "sum",
        "cost": "sum",
        "revenue": "sum"
    })
)
</code></pre>
<p>This creates a table with one row per channel and a column for each total. You can then calculate ROAS and CPA using those totals:</p>
<pre><code class="language-python">channel_summary["roas"] = (
    channel_summary["revenue"]
    .div(channel_summary["cost"].replace(0, float("nan")))
)

channel_summary["cpa"] = (
    channel_summary["cost"]
    .div(
        channel_summary["conversions"].replace(
            0, float("nan")
        )
    )
)

print(channel_summary.round(2))
</code></pre>
<p>For example, imagine that your data produces the following illustrative report:</p>
<table>
<thead>
<tr>
<th>Channel</th>
<th align="right">Impressions</th>
<th align="right">Clicks</th>
<th align="right">Conversions</th>
<th align="right">Cost</th>
<th align="right">Revenue</th>
<th align="right">ROAS</th>
<th align="right">CPA</th>
</tr>
</thead>
<tbody><tr>
<td>Email</td>
<td align="right">50,000</td>
<td align="right">4,000</td>
<td align="right">500</td>
<td align="right">$2,000</td>
<td align="right">$10,000</td>
<td align="right">5.00</td>
<td align="right">$4.00</td>
</tr>
<tr>
<td>Search</td>
<td align="right">80,000</td>
<td align="right">6,000</td>
<td align="right">300</td>
<td align="right">$3,000</td>
<td align="right">$9,000</td>
<td align="right">3.00</td>
<td align="right">$10.00</td>
</tr>
<tr>
<td>Social Media</td>
<td align="right">100,000</td>
<td align="right">5,000</td>
<td align="right">250</td>
<td align="right">$2,500</td>
<td align="right">$7,500</td>
<td align="right">3.00</td>
<td align="right">$10.00</td>
</tr>
</tbody></table>
<p><em>Illustrative data shown to demonstrate the report structure.</em></p>
<p>This table is more useful than a list of isolated totals because it allows you to compare reach, engagement, conversions, cost, and attributed revenue in one place.</p>
<p>In this example, email has the highest ROAS and the lowest CPA among the three channels. Social media has the most impressions, but that reach doesn't translate into the highest revenue or conversion total. These results suggest questions to investigate, but they don't automatically prove why one channel performs better than another.</p>
<h4 id="heading-why-calculate-ratios-from-totals">Why Calculate Ratios From Totals?</h4>
<p>When calculating an overall channel ROAS, divide total attributed revenue by total advertising cost. When calculating CPA, divide total cost by total conversions.</p>
<p>Don't simply average the individual campaign ROAS or CPA values if you want the overall channel metric. Campaigns may have very different budgets and conversion volumes, so a simple average can give small campaigns the same weight as large campaigns.</p>
<p>The key lesson here is this: <code>groupby()</code> helps turn individual campaign records into summaries that support channel comparisons and reporting. The results become more meaningful when you calculate the relevant metrics from the grouped totals and interpret them in context.</p>
<h3 id="heading-chapter-13-exploratory-data-analysis">Chapter 13: Exploratory Data Analysis</h3>
<p>Exploratory data analysis, commonly called EDA, is the process of examining a dataset to understand its structure, quality, patterns, and possible relationships before drawing conclusions or building a predictive model.</p>
<p>The purpose of EDA is both to generate statistics as well as to answer practical questions about the data:</p>
<ul>
<li>What information does the dataset contain?</li>
<li>Are any values missing, duplicated, or incorrectly formatted?</li>
<li>Which campaigns or channels appear to perform differently?</li>
<li>Are there unusual values that need investigation?</li>
<li>Do variables appear to move together?</li>
<li>Are there patterns that could help explain marketing performance?</li>
</ul>
<p>EDA helps you discover what the data can support and what still needs to be investigated. It can also reveal data-quality problems that would otherwise lead to misleading reports or unreliable machine learning models.</p>
<h4 id="heading-step-1-understand-the-dataset">Step 1: Understand the Dataset</h4>
<p>Start by checking the number of rows and columns, previewing records, and examining data types.</p>
<pre><code class="language-python">print(df.shape)
print(df.head())
print(df.info())
</code></pre>
<p><code>df.shape</code> shows the dataset's dimensions. <code>df.head()</code> displays the first few rows, while <code>df.info()</code> summarizes the columns, data types, and non-missing values.</p>
<p>These checks can reveal problems such as a revenue column stored as text or a column containing fewer non-missing records than expected.</p>
<h4 id="heading-step-2-examine-summary-statistics">Step 2: Examine Summary Statistics</h4>
<pre><code class="language-python">print(df.describe())
</code></pre>
<p>For numeric columns, <code>describe()</code> reports statistics such as count, mean, standard deviation, minimum, quartiles, and maximum.</p>
<p>These values help you understand typical values and identify possible anomalies. For example, a campaign with unusually high advertising cost may be legitimate, or it may indicate a data-entry error. The summary alone can't determine which explanation is correct.</p>
<h4 id="heading-step-3-check-missing-and-duplicate-records">Step 3: Check Missing and Duplicate Records</h4>
<pre><code class="language-python">print(df.isnull().sum())
print(df.duplicated().sum())
</code></pre>
<p>Missing values can affect calculations and models. Duplicate rows may indicate repeated exports or duplicated records, although repeated values aren't automatically duplicates. Before removing records, determine whether they represent genuinely repeated observations or valid separate events.</p>
<h4 id="heading-step-4-examine-categories">Step 4: Examine Categories</h4>
<p>Marketing data often contains categories such as channel, campaign, region, audience, or device type.</p>
<pre><code class="language-python">print(df["channel"].value_counts())
print(df["campaign"].value_counts())
</code></pre>
<p>These summaries show how many records belong to each category. If a channel appears only once while another appears hundreds of times, that imbalance may affect how you interpret comparisons.</p>
<h4 id="heading-step-5-explore-distributions-and-relationships">Step 5: Explore Distributions and Relationships</h4>
<p>You can summarize the distribution of revenue:</p>
<pre><code class="language-python">print(df["revenue"].describe())
</code></pre>
<p>You can also examine correlations between numeric variables:</p>
<pre><code class="language-python">numeric_columns = [
    "impressions",
    "clicks",
    "conversions",
    "cost",
    "revenue"
]

print(df[numeric_columns].corr())
</code></pre>
<p>Correlation describes how two numeric variables vary together. A positive correlation means higher values of one variable tend to occur alongside higher values of the other; a negative correlation means higher values of one tend to occur alongside lower values of the other.</p>
<p>For example, advertising cost and revenue may be positively correlated because campaigns with larger budgets also generate more revenue. That doesn't prove that increasing spending caused the revenue increase. Campaign quality, audience size, seasonality, product demand, and other factors may also contribute.</p>
<h4 id="heading-what-insights-can-eda-produce">What Insights Can EDA Produce?</h4>
<p>EDA can help you identify:</p>
<ul>
<li><strong>Data-quality issues:</strong> missing values, invalid data types, duplicate records, or suspicious measurements.</li>
<li><strong>Performance differences:</strong> channels or campaigns with higher or lower revenue, conversions, or costs.</li>
<li><strong>Distribution patterns:</strong> whether most orders have similar values or whether a few unusually large orders affect the average.</li>
<li><strong>Potential relationships:</strong> variables that appear to change together and deserve further analysis.</li>
<li><strong>Questions for further investigation:</strong> unexpected results that need business context, a visualization, or a statistical test.</li>
</ul>
<p>EDA doesn't automatically establish cause and effect. Instead, it provides a better understanding of the dataset and helps you decide which analyses should come next.</p>
<h3 id="heading-chapter-14-cohort-analysis">Chapter 14: Cohort Analysis</h3>
<p>Customer behavior becomes more interesting when you analyze it over time.</p>
<p>A cohort groups customers based on a shared starting point. For example, you might group customers based on the month they first purchased.</p>
<p>Suppose one customer first purchased in January and another first purchased in March. They belong to different acquisition cohorts.</p>
<p>First, identify each customer's first purchase:</p>
<pre><code class="language-python">df["purchase_date"] = pd.to_datetime(
    df["purchase_date"]
)

first_purchase = (
    df.groupby("customer_id")[
        "purchase_date"
    ].min()
)
</code></pre>
<p>You can attach that information to the dataset:</p>
<pre><code class="language-python">df["first_purchase"] = (
    df["customer_id"]
    .map(first_purchase)
)
</code></pre>
<p>Then extract the cohort month:</p>
<pre><code class="language-python">df["cohort_month"] = (
    df["first_purchase"]
    .dt.to_period("M")
)
</code></pre>
<p>Now you can compare customer behavior across acquisition periods.</p>
<p>Cohort analysis is particularly useful for understanding retention. A company might discover that customers acquired during one campaign remain active much longer than customers acquired through another campaign.</p>
<p>That insight can be much more valuable than simply knowing which campaign generated the most initial conversions.</p>
<h3 id="heading-chapter-15-retention-analysis">Chapter 15: Retention Analysis</h3>
<p>Acquiring a customer is only one part of marketing. Keeping that customer matters, too.</p>
<p>Suppose a company acquires 1,000 customers in January. If only 50 return in February, the business may have a retention problem.</p>
<p>You can calculate returning customers like this:</p>
<pre><code class="language-python">monthly_customers = (
    df.groupby(
        df["purchase_date"].dt.to_period("M")
    )["customer_id"]
    .nunique()
)
</code></pre>
<p>More detailed retention analysis requires defining exactly what counts as an active customer.</p>
<p>For an e-commerce company, activity might mean a purchase. For a SaaS product, it might mean logging in. And for an email campaign, it might mean opening or clicking.</p>
<p>The definition should match the business model.</p>
<h2 id="heading-part-iv-marketing-visualization">Part IV: Marketing Visualization</h2>
<p>Once you've cleaned and analyzed your data, the next challenge is communication.</p>
<p>A marketing analyst might understand a dataset perfectly, while a marketing manager might not want to look at 50,000 rows of data.</p>
<p>Visualization provides a bridge.</p>
<h3 id="heading-chapter-16-visualization-as-communication">Chapter 16: Visualization as Communication</h3>
<p>A visualization should answer a question. For example:</p>
<ol>
<li><strong>Which channel generates the most revenue?</strong> In this case, a bar chart makes sense.</li>
<li><strong>How is revenue changing over time?</strong> Here, a line chart makes sense.</li>
<li><strong>Is advertising spend associated with revenue?</strong> And for this, a scatter plot can help investigate the relationship.</li>
</ol>
<p>And while charts certainly do look professional, that's not the only goal here. Our true goal is to make the important information easier to understand.</p>
<h4 id="heading-how-to-choose-the-right-chart">How to Choose the Right Chart</h4>
<p>So how do you know which type of chart to use?</p>
<p>You should use line charts for trends over time, bar charts for category comparisons, scatter plots for relationships between numerical variables, histograms for distributions, and tables when exact values matter.</p>
<p>A visualization should match the question.</p>
<h3 id="heading-chapter-17-revenue-over-time">Chapter 17: Revenue Over Time</h3>
<p>Revenue is more informative when you examine how it changes over time instead of looking at a single total. A company might generate substantial revenue in one month but experience a decline in the following months. Another company might have lower total revenue but show steady growth.</p>
<p>Analyzing revenue over time helps you investigate trends, recurring seasonal patterns, sudden changes, and periods that may require additional attention.</p>
<p>Before plotting revenue, make sure the dataset contains a valid date column and a numeric revenue column. The example below assumes the dataset has a <code>purchase_date</code> column and that each row represents a revenue event or transaction.</p>
<h4 id="heading-preparing-monthly-revenue">Preparing Monthly Revenue</h4>
<p>First, convert the date column to a datetime format:</p>
<pre><code class="language-python">import pandas as pd
import matplotlib.pyplot as plt

df["purchase_date"] = pd.to_datetime(
    df["purchase_date"],
    errors="coerce"
)

df = df.dropna(
    subset=["purchase_date", "revenue"]
)
</code></pre>
<p>Next, group the revenue by month:</p>
<pre><code class="language-python">monthly_revenue = (
    df.set_index("purchase_date")["revenue"]
    .resample("MS")
    .sum()
)
</code></pre>
<p>The <code>"MS"</code> frequency groups the data by month, with each period labeled by the start of the month. The <code>sum()</code> operation adds the revenue values within each month.</p>
<p>This calculation assumes that each row represents a revenue amount that should be added. If the dataset already contains monthly revenue totals, don't sum repeated monthly totals without checking the structure, because doing so could double-count revenue.</p>
<h4 id="heading-visualizing-the-trend">Visualizing the Trend</h4>
<pre><code class="language-python">monthly_revenue.plot(
    kind="line",
    marker="o"
)

plt.title("Illustrative Monthly Marketing Revenue")
plt.xlabel("Month")
plt.ylabel("Revenue ($)")
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
</code></pre>
<p>A line chart makes changes over time easier to see. The horizontal axis represents time, and the vertical axis represents revenue. Each point represents the total revenue for a month, while the connecting lines make the overall pattern easier to follow.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6a581501af6af179dc1987d5/7be7abc8-7b4d-43c4-b8b7-42991d9a3901.png" alt="Illustrative Monthly Marketing Revenue Potential Output" style="display: block;" width="1440" height="810" loading="lazy">

<p><em>Illustrative data for learning purposes</em></p>
<h4 id="heading-interpreting-the-chart">Interpreting the Chart</h4>
<p>When examining the graph, ask these questions:</p>
<ul>
<li>Is revenue generally increasing, decreasing, or staying stable?</li>
<li>Are there sudden increases or decreases?</li>
<li>Do similar changes appear during the same season each year?</li>
<li>Could a campaign launch, price change, promotion, or product release help explain a change?</li>
<li>Are the periods being compared complete and measured consistently?</li>
</ul>
<p>A rising line doesn't automatically mean a marketing campaign caused revenue to increase. Other factors, including seasonality, pricing, product availability, and changes in customer demand, may contribute.</p>
<p>If the dataset contains only a few months, you may not have enough history to identify a reliable seasonal pattern. Use the chart to identify questions, then compare the pattern with campaign activity and other business information.</p>
<p>So remember: a revenue-over-time chart shows how performance changes, giving you a starting point for investigating trends and understanding what may be driving them.</p>
<h3 id="heading-chapter-18-channel-comparisons">Chapter 18: Channel Comparisons</h3>
<p>Marketing teams often use several channels at the same time. Email, search advertising, social media, and display advertising can each contribute to revenue, but they may perform differently in terms of reach, conversions, and cost.</p>
<p>Channel comparisons help answer questions such as:</p>
<ul>
<li>Which channels generate the most revenue?</li>
<li>Which channels produce the most conversions?</li>
<li>Which channels use the advertising budget most efficiently?</li>
<li>Are high-reach channels also generating meaningful results?</li>
</ul>
<p>A visualization makes these comparisons easier to understand, especially when the report contains several channels.</p>
<h4 id="heading-comparing-revenue-by-channel">Comparing Revenue by Channel</h4>
<p>Start by grouping revenue by channel:</p>
<pre><code class="language-python">channel_revenue = (
    df.groupby("channel")["revenue"]
    .sum()
    .sort_values(ascending=False)
)
</code></pre>
<p>Then create a bar chart:</p>
<pre><code class="language-python">import matplotlib.pyplot as plt

channel_revenue.plot(
    kind="bar"
)

plt.title("Illustrative Revenue by Marketing Channel")
plt.xlabel("Marketing Channel")
plt.ylabel("Revenue ($)")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()
</code></pre>
<p>Each bar represents the total revenue associated with one channel. Sorting the values from highest to lowest makes it easier to identify which channels contribute the most revenue in the dataset.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6a581501af6af179dc1987d5/8637e85d-a950-49fa-8d4e-91188f3d0826.png" alt="Illustrative Revenue by Marketing Channel Potential Output" style="display: block;" width="1440" height="810" loading="lazy">

<p><em>Illustrative data for learning purposes</em></p>
<h4 id="heading-comparing-conversions-by-channel">Comparing Conversions by Channel</h4>
<p>Revenue doesn't tell the whole story. A channel might generate fewer dollars but produce a large number of signups, leads, or purchases, depending on the business objective.</p>
<pre><code class="language-python">channel_conversions = (
    df.groupby("channel")["conversions"]
    .sum()
    .sort_values(ascending=False)
)

channel_conversions.plot(
    kind="bar"
)

plt.title("Illustrative Conversions by Marketing Channel")
plt.xlabel("Marketing Channel")
plt.ylabel("Conversions")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/6a581501af6af179dc1987d5/191e1863-9e06-4299-9787-49b494a5efb0.png" alt="Illustrative Conversions by Marketing Channel Potential Output" style="display: block;" width="1440" height="810" loading="lazy">

<p><em>Illustrative data for learning purposes</em></p>
<p>The first chart compares revenue volume, while the second compares conversion volume. Together, they provide more context than either chart alone.</p>
<h4 id="heading-why-one-metric-isnt-enough">Why One Metric Isn't Enough</h4>
<p>Imagine the following results:</p>
<table>
<thead>
<tr>
<th>Channel</th>
<th align="right">Revenue</th>
<th align="right">Conversions</th>
<th align="right">Advertising Cost</th>
</tr>
</thead>
<tbody><tr>
<td>Email</td>
<td align="right">$50,000</td>
<td align="right">1,000</td>
<td align="right">$40,000</td>
</tr>
<tr>
<td>Search</td>
<td align="right">$45,000</td>
<td align="right">300</td>
<td align="right">$5,000</td>
</tr>
</tbody></table>
<p>Email generates more revenue and more conversions. But search generates $45,000 in revenue with only $5,000 in advertising cost, while email spends $40,000 to generate $50,000 in revenue.</p>
<p>Their ROAS values are:</p>
<ul>
<li>Email: $50,000 ÷ $40,000 = 1.25</li>
<li>Search: $45,000 ÷ $5,000 = 9.00</li>
</ul>
<p>Search has the higher ROAS in this example, even though email generates more total revenue. Remember that ROAS measures attributed revenue relative to advertising cost. It's not the same as profit.</p>
<p>These numbers illustrate why you should compare related metrics rather than automatically choosing the channel with the tallest revenue bar.</p>
<h3 id="heading-chapter-19-scatter-plots">Chapter 19: Scatter Plots</h3>
<p>Suppose you want to explore whether advertising spend and revenue are related.</p>
<pre><code class="language-python">plt.scatter(
    df["cost"],
    df["revenue"]
)

plt.title(
    "Advertising Cost vs Revenue"
)

plt.xlabel("Advertising Cost")
plt.ylabel("Revenue")

plt.tight_layout()
plt.show()
</code></pre>
<p>The resulting pattern can help you investigate relationships.</p>
<p>But don't immediately conclude that spending more causes revenue to increase.</p>
<p>The relationship could be affected by:</p>
<ul>
<li>Campaign quality</li>
<li>Audience size</li>
<li>Seasonality</li>
<li>Product demand</li>
<li>Brand recognition</li>
<li>Existing customer behavior</li>
<li>Other marketing channels</li>
</ul>
<p>Visualization helps you ask better questions, but it doesn't replace the actual analysis of the data.</p>
<h3 id="heading-chapter-20-distribution-analysis">Chapter 20: Distribution Analysis</h3>
<p>Distribution analysis examines how values are spread across a dataset. Instead of focusing only on a total or average, it helps you understand which values occur frequently, how much variation exists, and whether a small number of unusually high or low observations influence the results.</p>
<p>In marketing, distribution analysis can help you understand customer order values, campaign costs, website sessions, conversion rates, and other numeric measurements.</p>
<p>This is different from simply analyzing older values. Historical analysis compares measurements across time, while distribution analysis examines the spread and frequency of values within a dataset or a selected period. You can use both approaches together, but they answer different questions.</p>
<p>For example, historical analysis asks whether average order value increased from one month to the next. Distribution analysis asks whether most orders have similar values or whether a few unusually large orders pull the average upward.</p>
<h4 id="heading-examining-order-values">Examining Order Values</h4>
<p>Suppose your dataset contains an <code>order_value</code> column. Begin by examining its summary statistics:</p>
<pre><code class="language-python">print(df["order_value"].describe())
</code></pre>
<p>The output includes the count, mean, standard deviation, minimum, quartiles, and maximum.</p>
<p>The <strong>mean</strong> is the arithmetic average: add all order values and divide by the number of orders. The <strong>median</strong> is the middle value when the orders are sorted. The median is less affected by a small number of extremely large purchases.</p>
<p>For example, consider these five order values:</p>
<pre><code class="language-text">$20, $25, $30, $35, $390
</code></pre>
<p>The mean is $100, but the median is $30. The $390 purchase raises the mean substantially, even though four of the five orders are below $40.</p>
<p>This is why the mean doesn't always describe a typical customer's order particularly well. If order values are strongly skewed, the median and the distribution itself may provide a more representative picture.</p>
<h4 id="heading-visualizing-the-distribution">Visualizing the Distribution</h4>
<p>A histogram groups numeric values into ranges called bins and shows how many observations fall into each range.</p>
<pre><code class="language-python">import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({
    "order_id": range(1, 11),
    "order_value": [20, 25, 30, 35, 390, 28, 32, 40, 45, 50]
})

plt.figure(figsize=(9, 5))

plt.hist(
    df["order_value"],
    bins=8,
    edgecolor="black"
)

plt.title("Illustrative Distribution of Customer Order Values")
plt.xlabel("Order Value ($)")
plt.ylabel("Number of Orders")
plt.grid(axis="y", alpha=0.3)
plt.tight_layout()
plt.show()
</code></pre>
<p>Each bar represents a range of order values, and the height indicates how many orders fall within that range.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6a581501af6af179dc1987d5/3a0d7be3-9fc8-41b1-8732-9c1a9be80433.png" alt="Histogram Output" style="display: block;" width="1323" height="732" loading="lazy">

<p>A histogram can reveal whether most orders cluster around a particular range, whether the distribution has a long tail of high-value purchases, or whether multiple groups of order values appear to exist.</p>
<h4 id="heading-what-should-you-look-for">What Should You Look For?</h4>
<p>When interpreting a distribution, consider:</p>
<ul>
<li><strong>Center:</strong> Where do the values tend to fall? The mean and median can help describe the center.</li>
<li><strong>Spread:</strong> How much do values vary? A wide distribution indicates more variation than a tightly clustered one.</li>
<li><strong>Skew:</strong> Are values concentrated toward one end with a longer tail on the other?</li>
<li><strong>Outliers:</strong> Are some values unusually far from the rest?</li>
<li><strong>Multiple groups:</strong> Does the distribution suggest different types of customers or transactions?</li>
</ul>
<p>An unusual value isn't necessarily an error. A very large order could be a legitimate purchase from a valuable customer. Check the source and business context before removing or changing it.</p>
<h4 id="heading-why-distribution-analysis-matters">Why Distribution Analysis Matters</h4>
<p>Suppose average order value rises substantially in a particular month. Looking only at the mean might lead you to assume that customers generally began spending more. But the increase could be caused by a handful of unusually large purchases while most customers continue to spend the same amount.</p>
<p>Comparing the mean, median, histogram, and historical trends can help distinguish these possibilities.</p>
<p>The key lesson is this: distribution analysis helps you understand what's typical, how much values vary, and whether extreme observations affect the story told by a summary statistic.</p>
<h2 id="heading-part-v-customer-and-campaign-analytics">Part V: Customer and Campaign Analytics</h2>
<p>Marketing analytics becomes significantly more interesting when you stop asking only "How did our campaign perform?" and start asking "Who responded, why did they respond, and what should we do next?"</p>
<h3 id="heading-chapter-21-customer-segmentation">Chapter 21: Customer Segmentation</h3>
<p>Customer segmentation is the process of dividing customers into groups based on shared characteristics or behaviors. Rather than treating every customer as identical, a business can identify groups that may respond differently to marketing campaigns.</p>
<p>For example, some customers purchase frequently, some make only one purchase, and others have not purchased for several months. A business may want to reward frequent customers, encourage one-time buyers to return, or reconnect with customers who have become inactive.</p>
<p>Segmentation helps marketers develop more relevant strategies for different customer groups. It doesn't automatically tell you what each customer wants, but it provides a structured way to understand patterns in customer behavior.</p>
<h4 id="heading-rfm-analysis">RFM Analysis</h4>
<p>One common approach is RFM analysis, which uses three measurements:</p>
<ul>
<li><strong>Recency:</strong> How recently did the customer make a purchase?</li>
<li><strong>Frequency:</strong> How many purchases has the customer made during the period being analyzed?</li>
<li><strong>Monetary value:</strong> How much money has the customer spent during that period?</li>
</ul>
<p>These measurements provide different perspectives on customer activity. A customer with low recency in days has purchased recently, while a customer with high recency in days hasn't purchased for longer. Higher frequency indicates more purchases, and higher monetary value indicates greater recorded spending.</p>
<p>Suppose a dataset contains <code>customer_id</code>, <code>purchase_date</code>, and <code>order_value</code>. First, convert the purchase dates to datetime values:</p>
<pre><code class="language-python">df["purchase_date"] = pd.to_datetime(
    df["purchase_date"],
    errors="coerce"
)
</code></pre>
<p>Choose an analysis date and calculate each customer's last purchase, purchase frequency, and total recorded spending:</p>
<pre><code class="language-python">analysis_date = pd.Timestamp("2026-09-05")

customer_rfm = (
    df.groupby("customer_id")
    .agg(
        last_purchase=("purchase_date", "max"),
        frequency=("purchase_date", "count"),
        monetary=("order_value", "sum")
    )
)

customer_rfm["recency"] = (
    analysis_date - customer_rfm["last_purchase"]
).dt.days

customer_rfm = customer_rfm[
    ["recency", "frequency", "monetary"]
]

print(customer_rfm)
</code></pre>
<p>This example assumes that each row represents one purchase and that <code>order_value</code> contains the value of that purchase. If your data has multiple rows per order, use a unique order identifier to count orders correctly. Choose an analysis date that matches the reporting period you're studying.</p>
<h4 id="heading-understanding-the-rfm-table">Understanding the RFM Table</h4>
<p>The following illustrative table shows what an RFM summary might look like:</p>
<table>
<thead>
<tr>
<th>Customer ID</th>
<th align="right">Recency (days)</th>
<th align="right">Frequency (orders)</th>
<th align="right">Monetary Value</th>
</tr>
</thead>
<tbody><tr>
<td>C001</td>
<td align="right">8</td>
<td align="right">12</td>
<td align="right">$960</td>
</tr>
<tr>
<td>C002</td>
<td align="right">75</td>
<td align="right">4</td>
<td align="right">$240</td>
</tr>
<tr>
<td>C003</td>
<td align="right">180</td>
<td align="right">1</td>
<td align="right">$45</td>
</tr>
<tr>
<td>C004</td>
<td align="right">20</td>
<td align="right">7</td>
<td align="right">$525</td>
</tr>
</tbody></table>
<p><em>Illustrative values for learning purposes.</em></p>
<p>Customer C001 purchased recently, has placed many orders, and has spent the most in this example. Customer C003 hasn't purchased for a much longer period, has placed only one order, and has the lowest recorded spending.</p>
<p>These differences may suggest different marketing approaches. A business might consider a loyalty offer for a frequent customer, a follow-up offer for a one-time buyer, or a re-engagement campaign for a customer who has been inactive.</p>
<p>But RFM values should be interpreted in context. A customer with low frequency might be perfectly normal for a product that's purchased only once a year. The length of the observation period also matters because a customer who has been active for two months has had less time to accumulate purchases than a customer who has been active for two years.</p>
<h3 id="heading-chapter-22-customer-lifetime-value">Chapter 22: Customer Lifetime Value</h3>
<p>Customer lifetime value estimates how much value a customer generates over their relationship with the business.</p>
<p>A simple historical measure is:</p>
<pre><code class="language-python">customer_ltv = (
    df.groupby("customer_id")[
        "order_value"
    ].sum()
)
</code></pre>
<p>This tells you how much each customer has spent historically.</p>
<p>A more sophisticated model might estimate future value.</p>
<p>For example, a conceptual lifetime value formula could consider:</p>
<pre><code class="language-text">Average Order Value
× Purchase Frequency
× Expected Customer Lifespan
</code></pre>
<p>But real lifetime value calculations can become much more complex.</p>
<p>They may need to consider:</p>
<ul>
<li>Gross margin</li>
<li>Retention</li>
<li>Churn</li>
<li>Discounting</li>
<li>Acquisition cost</li>
<li>Product category</li>
<li>Customer behavior</li>
</ul>
<p>The important lesson is that LTV is an estimate, not an unquestionable truth.</p>
<h3 id="heading-chapter-23-customer-acquisition-cost">Chapter 23: Customer Acquisition Cost</h3>
<p>Customer acquisition cost, or CAC, estimates how much it costs to acquire a customer.</p>
<p>A simple formula is:</p>
<pre><code class="language-python">cac = total_marketing_cost / new_customers
</code></pre>
<p>Suppose:</p>
<pre><code class="language-python">marketing_cost = 20000
new_customers = 400

cac = marketing_cost / new_customers

print(cac)
</code></pre>
<p>The result is:</p>
<pre><code class="language-text">50.0
</code></pre>
<p>That means the average acquisition cost was $50 per customer.</p>
<p>Comparing CAC with customer value can help determine whether acquisition is economically sustainable.</p>
<h4 id="heading-cac-and-ltv-together">CAC and LTV Together</h4>
<p>Suppose:</p>
<pre><code class="language-text">CAC = $50
LTV = $300
</code></pre>
<p>That looks potentially healthy.</p>
<p>Now suppose:</p>
<pre><code class="language-text">CAC = $250
LTV = $180
</code></pre>
<p>The business may be acquiring customers for more than they're worth.</p>
<p>These metrics should still be interpreted carefully. LTV may be estimated, and CAC may vary substantially by channel.</p>
<p>Some acquisition costs may be excluded from a simple calculation. But together, they provide a useful framework for thinking about marketing economics.</p>
<h3 id="heading-chapter-24-funnel-analysis">Chapter 24: Funnel Analysis</h3>
<p>A marketing funnel represents the stages a customer moves through before completing a desired action, such as making a purchase, signing up for a service, or submitting a lead form.</p>
<p>A typical online shopping funnel might include:</p>
<ol>
<li>Visiting a website</li>
<li>Viewing a product</li>
<li>Adding the product to a cart</li>
<li>Starting checkout</li>
<li>Completing a purchase</li>
</ol>
<p>At each stage, some people continue while others leave. Funnel analysis measures how many people progress from one stage to the next and helps you identify where additional investigation may be useful.</p>
<h4 id="heading-calculating-stage-conversion-rates">Calculating Stage Conversion Rates</h4>
<p>Suppose a website records the following counts during the same reporting period:</p>
<pre><code class="language-python">visitors = 50000
product_views = 30000
cart_adds = 8000
checkouts = 4000
purchases = 2500
</code></pre>
<p>A <strong>stage conversion rate</strong> is the percentage of people who reach one stage and then progress to the next stage.</p>
<p>Stage Conversion Rate = Number Reaching the Next Stage ÷ Number Reaching the Previous Stage × 100</p>
<p>For example, the product-view rate is calculated by dividing product views by visitors:</p>
<pre><code class="language-python">product_view_rate = (
    product_views / visitors * 100
)

cart_rate = (
    cart_adds / product_views * 100
)

checkout_rate = (
    checkouts / cart_adds * 100
)

purchase_rate = (
    purchases / checkouts * 100
)

print(f"Visitor to product view: {product_view_rate:.1f}%")
print(f"Product view to cart: {cart_rate:.1f}%")
print(f"Cart to checkout: {checkout_rate:.1f}%")
print(f"Checkout to purchase: {purchase_rate:.1f}%")
</code></pre>
<p>With these illustrative values, 60% of visitors view a product, about 26.7% of product viewers add an item to their cart, 50% of cart additions progress to checkout, and 62.5% of checkouts result in a purchase.</p>
<p>Each percentage describes a specific transition. Don't confuse these stage conversion rates with the overall purchase rate, which would compare purchases directly with visitors.</p>
<p>These calculations also assume that the stage counts are defined consistently and refer to comparable users or sessions within the same reporting period. If the counts represent different populations or use different tracking rules, the rates may not accurately describe customer progression.</p>
<h4 id="heading-investigating-customer-experience">Investigating Customer Experience</h4>
<p>A low stage conversion rate tells you where people are dropping out, but it doesn't explain why they're leaving. The next step is to investigate the customer experience at that stage.</p>
<p>For example, if many customers abandon checkout, possible issues to examine include:</p>
<ul>
<li>Unexpected shipping fees or taxes</li>
<li>A checkout process that requires too many steps</li>
<li>Forms that are difficult to complete on a mobile device</li>
<li>Limited payment options</li>
<li>Slow-loading pages or technical errors</li>
<li>Confusing delivery or return information</li>
<li>A mismatch between the advertisement and the landing page</li>
</ul>
<p>You can investigate these possibilities using analytics events, device-level comparisons, customer support feedback, usability testing, and error logs. These sources help you identify potential explanations rather than assuming the funnel rate itself proves the cause.</p>
<p>If the evidence suggests that a checkout change might help, you could test a simpler checkout flow or clearer pricing information. This connects funnel analysis to A/B testing: funnel analysis identifies where to investigate, while an experiment can help assess whether a proposed change improves the outcome.</p>
<p>Funnel analysis shows where customers stop progressing. Investigating the customer experience helps you develop explanations and determine which changes are worth testing.</p>
<h3 id="heading-chapter-25-ab-testing">Chapter 25: A/B Testing</h3>
<p>A/B testing is a controlled experiment used to compare two versions of a marketing experience. A business might compare two landing pages, email subject lines, advertisements, checkout designs, or calls to action to find out which version performs better for a defined objective.</p>
<p>Version A is the control or existing version, while Version B is the alternative being tested. Ideally, eligible users are randomly assigned to one version so that the comparison is less likely to be distorted by differences between the groups.</p>
<h4 id="heading-calculating-conversion-rates">Calculating Conversion Rates</h4>
<p>Suppose you're testing two landing pages and measuring whether visitors complete a desired action.</p>
<pre><code class="language-python">visitors_a = 5000
conversions_a = 240

visitors_b = 5000
conversions_b = 290
</code></pre>
<p>Calculate each conversion rate:</p>
<pre><code class="language-python">rate_a = (
    conversions_a / visitors_a
)

rate_b = (
    conversions_b / visitors_b
)

print(f"Version A: {rate_a:.2%}")
print(f"Version B: {rate_b:.2%}")
</code></pre>
<p>In this example, Version A has a conversion rate of 4.8%, while Version B has a conversion rate of 5.8%. The observed difference is one percentage point.</p>
<p>But an observed difference <strong>doesn't automatically prove that Version B is genuinely better</strong>. Results can vary because of random chance, even when two versions perform similarly in the wider population.</p>
<h4 id="heading-understanding-statistical-uncertainty">Understanding Statistical Uncertainty</h4>
<p>An A/B test uses a sample of visitors to estimate how the versions might perform across a larger population. Because the test doesn't usually include every possible customer, the measured conversion rates are estimates rather than perfectly known population values.</p>
<p>Statistical uncertainty describes the fact that these estimates can vary from sample to sample. A different group of visitors might produce a slightly different result, even if the underlying performance of the two versions hasn't changed.</p>
<p>Several factors influence the uncertainty of an experiment:</p>
<ul>
<li><strong>Sample size:</strong> smaller samples generally produce less precise estimates than larger samples.</li>
<li><strong>Conversion frequency:</strong> when the desired action is rare, more observations may be needed to estimate the conversion rate precisely.</li>
<li><strong>Random variation:</strong> individual visitor behavior isn't perfectly predictable.</li>
<li><strong>Experimental design:</strong> uneven assignment, tracking problems, or changes made during the test can undermine the comparison.</li>
</ul>
<p>A small observed difference in a small sample may be due to random variation. A larger, well-designed experiment can provide stronger evidence, but sample size alone doesn't fix problems such as biased assignment or incorrect tracking.</p>
<h4 id="heading-statistical-significance-and-business-significance">Statistical Significance and Business Significance</h4>
<p>Statistical significance addresses whether the observed result would be unusual under a specified statistical model in which there's no true difference between the versions. It doesn't directly tell you how valuable the difference is to the business, nor does it prove that the result will repeat in every future setting.</p>
<p>Business significance concerns whether the size of the improvement is worth acting on. For example, a small conversion-rate improvement might generate meaningful additional revenue for a high-traffic website but very little additional value for a low-traffic site. The costs and risks of implementing the change also matter.</p>
<p>A responsible A/B testing process should:</p>
<ol>
<li>Define the primary outcome before the test begins.</li>
<li>Randomly assign eligible visitors when appropriate.</li>
<li>Make sure conversion tracking works consistently for both versions.</li>
<li>Decide how much data and what test duration are appropriate before evaluating the result.</li>
<li>Consider statistical uncertainty and the practical size of the observed difference.</li>
<li>Avoid repeatedly checking results and stopping as soon as one version looks better unless the testing method explicitly accounts for sequential monitoring.</li>
<li>Document the result, limitations, and decision.</li>
</ol>
<p>A/B testing isn't simply a matter of choosing whichever version has the larger percentage. The goal is to gather reliable evidence about whether a change improves a meaningful outcome.</p>
<p>The key lesson here is this: statistical uncertainty matters because sample results can differ from the true underlying performance. Understanding that uncertainty helps you avoid making expensive marketing decisions based on random fluctuations.</p>
<h2 id="heading-part-vi-marketing-automation">Part VI: Marketing Automation</h2>
<p>Marketing teams often spend enormous amounts of time repeating processes that follow predictable rules.</p>
<p>That makes automation one of the strongest applications of Python.</p>
<h3 id="heading-chapter-26-identifying-automation-opportunities">Chapter 26: Identifying Automation Opportunities</h3>
<p>A good candidate for automation usually has several characteristics:</p>
<ul>
<li>It happens repeatedly</li>
<li>It follows predictable steps</li>
<li>It uses structured data</li>
<li>It requires little subjective judgment</li>
<li>Mistakes are common when performed manually</li>
<li>Automating it saves meaningful time</li>
</ul>
<p>For example:</p>
<p>"Every Monday, download campaign data, calculate ROAS, identify campaigns below a threshold, and generate a report."</p>
<p>That's an excellent automation candidate.</p>
<h3 id="heading-chapter-27-building-a-reporting-function">Chapter 27: Building a Reporting Function</h3>
<p>Instead of manually running every step, create a function.</p>
<pre><code class="language-python">def generate_campaign_report(
    input_file,
    output_file
):
    df = pd.read_csv(input_file)

    df["roas"] = (
        df["revenue"] /
        df["cost"]
    )

    df["conversion_rate"] = (
        df["conversions"] /
        df["clicks"] *
        100
    )

    df.to_csv(
        output_file,
        index=False
    )
</code></pre>
<p>Then:</p>
<pre><code class="language-python">generate_campaign_report(
    "campaigns.csv",
    "campaign_report.csv"
)
</code></pre>
<p>Now the workflow can be repeated.</p>
<h3 id="heading-chapter-28-logging">Chapter 28: Logging</h3>
<p>When a script runs automatically, <code>print()</code> statements are often not enough.</p>
<p>Logging provides a more structured approach.</p>
<pre><code class="language-python">import logging

logging.basicConfig(
    level=logging.INFO
)

logging.info(
    "Marketing report started."
)
</code></pre>
<p>Then:</p>
<pre><code class="language-python">logging.info(
    "Campaign data loaded successfully."
)
</code></pre>
<p>If an error occurs:</p>
<pre><code class="language-python">logging.error(
    "Unable to load campaign data."
)
</code></pre>
<p>Logging becomes particularly useful when a scheduled job runs without someone watching it.</p>
<h3 id="heading-chapter-29-scheduling">Chapter 29: Scheduling</h3>
<p>Once your script works, you can schedule it.</p>
<p>For example, you might configure your operating system to run the reporting program every Monday morning.</p>
<p>The scheduling system itself depends on your environment. On Windows, Task Scheduler can execute Python programs. On macOS and Linux, cron is a common option. Cloud platforms can also run scheduled jobs.</p>
<p>The important concept is that <strong>automation isn't complete until the process can run reliably without someone manually starting it.</strong></p>
<h3 id="heading-chapter-30-automated-alerts">Chapter 30: Automated Alerts</h3>
<p>A marketing system shouldn't always wait for someone to open a report.</p>
<p>It can identify important conditions.</p>
<pre><code class="language-python">if overall_roas &lt; 1:
    alert = (
        "Warning: overall ROAS is below 1."
    )
</code></pre>
<p>Or:</p>
<pre><code class="language-python">if conversion_rate &lt; previous_rate * 0.8:
    alert = (
        "Conversion rate has fallen "
        "significantly compared with the "
        "previous period."
    )
</code></pre>
<p>The alert can eventually be connected to email, Slack, Microsoft Teams, or another notification system.</p>
<p>The important thing is that the system identifies situations requiring attention.</p>
<h3 id="heading-chapter-31-responsible-automation">Chapter 31: Responsible Automation</h3>
<p>Automation can make marketing work faster and more consistent, but it can also repeat mistakes at scale. A script that calculates an incorrect metric once can produce the same incorrect result every time it runs. If the report feeds into campaign decisions or automated customer communications, the consequences can grow quickly.</p>
<p>Suppose a report accidentally labels customers as eligible for a promotion when they don't meet the eligibility rules. A manual review might catch the problem. An automated system without validation could distribute the incorrect list repeatedly.</p>
<p>For this reason, production automation should include input validation, error handling, logging, testing, and monitoring. Important actions may also require human approval.</p>
<h4 id="heading-a-simple-production-ready-workflow">A Simple Production-Ready Workflow</h4>
<p>A basic marketing-report automation workflow can follow these steps:</p>
<ol>
<li><strong>Load the source data:</strong> Read the latest campaign export or approved input file.</li>
<li><strong>Validate the data:</strong> Check that required columns exist, numeric fields contain valid values, and unexpected records are flagged.</li>
<li><strong>Calculate the metrics:</strong> Generate conversion rate, CPA, and ROAS while handling zero denominators appropriately.</li>
<li><strong>Save the report:</strong> Write the results to a designated output location.</li>
<li><strong>Log the outcome:</strong> Record whether the report succeeded or failed so someone can investigate problems.</li>
<li><strong>Schedule and monitor the process:</strong> Use an operating-system scheduler or an approved orchestration tool, and check that the expected output is produced.</li>
</ol>
<p>The following example demonstrates a simple workflow for a campaign report. It assumes the input CSV contains one row per campaign and includes <code>campaign</code>, <code>clicks</code>, <code>conversions</code>, <code>cost</code>, and <code>revenue</code> columns.</p>
<pre><code class="language-python">import logging
from pathlib import Path

import pandas as pd

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s: %(message)s"
)

REQUIRED_COLUMNS = {
    "campaign",
    "clicks",
    "conversions",
    "cost",
    "revenue"
}


def generate_report(input_file, output_file):
    input_path = Path(input_file)
    output_path = Path(output_file)

    try:
        if not input_path.exists():
            raise FileNotFoundError(
                f"Input file not found: {input_path}"
            )

        df = pd.read_csv(input_path)

        missing = REQUIRED_COLUMNS - set(df.columns)
        if missing:
            raise ValueError(
                f"Missing required columns: {sorted(missing)}"
            )

        numeric_columns = [
            "clicks",
            "conversions",
            "cost",
            "revenue"
        ]

        for column in numeric_columns:
            df[column] = pd.to_numeric(
                df[column],
                errors="coerce"
            )

        if df[numeric_columns].isna().any().any():
            raise ValueError(
                "Required numeric fields contain missing "
                "or invalid values."
            )

        if (df[numeric_columns] &lt; 0).any().any():
            raise ValueError(
                "Negative values were found in metrics "
                "that should not be negative."
            )

        if (df["conversions"] &gt; df["clicks"]).any():
            logging.warning(
                "Some campaigns have more conversions "
                "than clicks. Check the metric definitions."
            )

        df["conversion_rate"] = (
            df["conversions"]
            .div(df["clicks"].replace(0, float("nan")))
            .mul(100)
        )

        df["cpa"] = (
            df["cost"]
            .div(
                df["conversions"].replace(
                    0, float("nan")
                )
            )
        )

        df["roas"] = (
            df["revenue"]
            .div(df["cost"].replace(0, float("nan")))
        )

        output_path.parent.mkdir(
            parents=True,
            exist_ok=True
        )

        df.to_csv(output_path, index=False)

        logging.info(
            "Report saved successfully to %s",
            output_path
        )

    except Exception:
        logging.exception(
            "Marketing report generation failed"
        )
        raise


if __name__ == "__main__":
    generate_report(
        "campaigns.csv",
        "reports/campaign_report.csv"
    )
</code></pre>
<h4 id="heading-what-makes-this-workflow-safer">What Makes This Workflow Safer?</h4>
<p>The script checks that the input file exists and that required columns are present. It attempts to convert metric columns into numeric values and stops if required fields contain missing or invalid data. It also flags negative values in metrics that shouldn't normally be negative.</p>
<p>When conversions exceed clicks, the example logs a warning rather than automatically changing the data. Depending on the business's definition of a conversion, that result may be possible, so it should be investigated rather than silently corrected.</p>
<p>The calculated rates use missing values for zero denominators. This prevents undefined ratios from being reported as valid numbers. The script also creates the output directory if necessary, records successful completion, and logs failures before raising the error so that a scheduler or monitoring system can detect the failure.</p>
<p>This is a practical starting point, not a complete monitoring platform. A real deployment should also consider data freshness, duplicate imports, expected row counts, access permissions, secure storage, backups, alerting, and whether a human should approve the report before it influences a campaign.</p>
<h4 id="heading-scheduling-the-workflow">Scheduling the Workflow</h4>
<p>The script can be run manually while it's being tested. Once it works reliably, it can be scheduled using a tool such as Windows Task Scheduler or cron on Linux and macOS. The scheduler determines when the script runs. The Python code determines what the script does.</p>
<p>Before scheduling it, test the workflow using sample data and deliberately introduce problems such as a missing column or an invalid numeric value. Confirm that the script fails clearly rather than producing a misleading report.</p>
<p>The key lesson over here: reliable automation isn't just about running code without human involvement. You also need to worry about about validating inputs, handling failures, documenting results, and ensuring that the automated output can be trusted.</p>
<h2 id="heading-part-vii-machine-learning-for-marketing">Part VII: Machine Learning for Marketing</h2>
<p>Machine learning becomes useful when you want to move from describing what happened to estimating what might happen next.</p>
<h3 id="heading-chapter-32-from-analytics-to-prediction">Chapter 32: From Analytics to Prediction</h3>
<p>Traditional analytics might answer a question like, "Which customers purchased last month?"</p>
<p>Machine learning, on the other hand, might answer: "Which customers are most likely to purchase next month?"</p>
<p>Traditional analytics asks "Which leads converted?"</p>
<p>Machine learning asks "Which leads are most likely to convert?"</p>
<p>Traditional analytics asks "Which customers churned?"</p>
<p>Machine learning asks "Which customers appear most likely to churn?"</p>
<p>This distinction is important: Machine learning doesn't replace analytics. It builds on it.</p>
<h3 id="heading-chapter-33-features-and-targets">Chapter 33: Features and Targets</h3>
<p>A machine learning model typically needs input variables called features.</p>
<p>Suppose you want to predict customer churn. Features might include:</p>
<pre><code class="language-text">purchase_frequency
total_spend
days_since_last_purchase
email_opens
website_sessions
</code></pre>
<p>The target might be:</p>
<pre><code class="language-text">churned
</code></pre>
<p>In Python:</p>
<pre><code class="language-python">features = [
    "purchase_frequency",
    "total_spend",
    "days_since_last_purchase",
    "email_opens",
    "website_sessions"
]

X = df[features]
y = df["churned"]
</code></pre>
<p>Here:</p>
<p><code>X</code> contains information used for prediction.</p>
<p><code>y</code> contains the outcome you want to predict.</p>
<h3 id="heading-chapter-34-splitting-data">Chapter 34: Splitting Data</h3>
<p>Before training a machine learning model, you need to separate the data used to teach the model from the data used to evaluate it.</p>
<p>This is important because a model can perform well on examples it has already seen without performing equally well on new examples. If you evaluate a model using the same records it learned from, the evaluation may give an overly optimistic impression of how well the model will perform in practice.</p>
<p>A common approach is to create a <strong>training set</strong> and a <strong>test set</strong>:</p>
<ul>
<li><strong>Training set:</strong> the records the model uses to learn patterns.</li>
<li><strong>Test set:</strong> records held back until evaluation to estimate how the trained model performs on data it didn't use for learning.</li>
</ul>
<p>For example, an 80/20 split allocates approximately 80% of the records to training and 20% to testing. This is a common starting point, not a universal rule. The appropriate split depends on the dataset size, the prediction task, and how the model will be used.</p>
<h4 id="heading-creating-the-split-with-scikit-learn">Creating the Split With scikit-learn</h4>
<p>Assume that <code>X</code> contains the model's input features and <code>y</code> contains the target you want to predict.</p>
<pre><code class="language-python">from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)
</code></pre>
<p>Here, <code>test_size=0.2</code> reserves 20% of the records for testing. <code>random_state=42</code> makes the random split reproducible, so you can obtain the same split again when the dataset and code remain unchanged.</p>
<p>For a classification problem, you may also want to preserve the approximate proportion of each class in both sets:</p>
<pre><code class="language-python">X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)
</code></pre>
<p>Using <code>stratify=y</code> is often helpful when the target classes are imbalanced. But stratification requires enough examples in each class and isn't appropriate for every task.</p>
<h4 id="heading-avoiding-time-related-leakage">Avoiding Time-Related Leakage</h4>
<p>Random splitting isn't always suitable for marketing data. If you're predicting future purchases or future churn, the records have a time order. Training on later observations and testing on earlier observations can allow information from the future to influence the evaluation.</p>
<p>For time-dependent predictions, consider training on an earlier period and testing on a later period. The split should reflect the information that would actually have been available when the prediction was made.</p>
<p>Also remember that splitting data does not, by itself, prevent data leakage. Any preprocessing steps that learn information from the dataset (like imputing missing values or scaling features) should generally be fitted using the training data only and then applied to the test data.</p>
<p>Remember: data splitting helps estimate how a model may perform on unseen data, but the split must reflect the way the model will be used in the real world.</p>
<h3 id="heading-chapter-35-classification">Chapter 35: Classification</h3>
<p>Classification is a machine learning task in which a model predicts a category or class. Marketing examples include predicting whether a customer will churn, whether a lead will convert, whether an email is likely to be marked as spam, or whether a customer will purchase within a defined period.</p>
<p>Classification differs from regression. Classification predicts categories, while regression predicts a numeric value such as expected revenue or customer lifetime value.</p>
<p>A classification model learns patterns from historical examples in which the correct outcomes are already known. It then uses those patterns to predict outcomes for new records. The model's predictions are estimates, not guarantees.</p>
<h4 id="heading-preparing-the-features-and-target">Preparing the Features and Target</h4>
<p>Suppose you want to predict whether a customer will churn. Your dataset might include features such as:</p>
<pre><code class="language-text">purchase_frequency
total_spend
days_since_last_purchase
email_opens
website_sessions
</code></pre>
<p>The target might be a column named <code>churned</code>, where each record is labeled to indicate whether the customer churned during a clearly defined period.</p>
<p>In Python:</p>
<pre><code class="language-python">features = [
    "purchase_frequency",
    "total_spend",
    "days_since_last_purchase",
    "email_opens",
    "website_sessions"
]

X = df[features]
y = df["churned"]
</code></pre>
<p>Here, <code>X</code> contains the information used to make predictions, and <code>y</code> contains the outcome the model is supposed to predict. The features should only include information that would genuinely be available at the time the prediction is made.</p>
<p>Before fitting the model, handle missing values and categorical variables as appropriate, and split the dataset into training and test sets as described in the previous chapter.</p>
<h4 id="heading-training-a-logistic-regression-classifier">Training a Logistic Regression Classifier</h4>
<p>Logistic regression is a common starting point for binary classification, such as predicting one of two outcomes: churn or no churn.</p>
<pre><code class="language-python">from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    max_iter=1000
)

model.fit(
    X_train,
    y_train
)
</code></pre>
<p>The <code>fit()</code> method trains the model using the training features and their known target labels. The model learns relationships between the input variables and the outcome.</p>
<p>Use the trained model to predict classes for the held-back test features:</p>
<pre><code class="language-python">predictions = model.predict(
    X_test
)
</code></pre>
<p>The <code>predict()</code> method produces predicted class labels. For a binary churn model, those labels might indicate churn or no churn.</p>
<p>You can evaluate the predictions with a classification report:</p>
<pre><code class="language-python">from sklearn.metrics import classification_report

print(
    classification_report(
        y_test,
        predictions,
        zero_division=0
    )
)
</code></pre>
<p>This report summarizes metrics such as precision, recall, and F1 score for each class.</p>
<h4 id="heading-other-classification-models">Other Classification Models</h4>
<p>Logistic regression isn't the only option. Depending on the data and problem, you might also investigate:</p>
<ul>
<li><strong>Decision trees:</strong> learn a series of decision rules based on feature values.</li>
<li><strong>Random forests:</strong> combine predictions from multiple decision trees.</li>
<li><strong>Gradient boosting:</strong> build a sequence of models that aims to improve predictive performance.</li>
<li><strong>Support vector machines:</strong> identify decision boundaries that separate classes.</li>
<li><strong>Naive Bayes:</strong> use probabilistic assumptions to estimate class membership.</li>
</ul>
<p>Different models have different strengths, computational requirements, and interpretability. A more complex model isn't automatically better. Compare suitable models using the same data split and evaluation criteria, and avoid choosing a model based only on training performance.</p>
<p>So remember: Classification models learn from labeled historical examples to predict categories for new records. Their usefulness depends on the quality of the data, the timing of the features, the model choice, and how well the predictions are evaluated.</p>
<h3 id="heading-chapter-36-why-accuracy-can-be-misleading">Chapter 36: Why Accuracy Can Be Misleading</h3>
<p>Accuracy measures the proportion of predictions a classification model gets correct. It's easy to understand, but it can be misleading when the classes are imbalanced or when different mistakes have different business costs.</p>
<p>Suppose only 1% of customers churn. A model that predicts “not churned” for every customer would achieve approximately 99% accuracy. But it would fail to identify any of the customers who actually churn.</p>
<p>That model has high accuracy but may be practically useless if the purpose is to identify customers who need retention efforts.</p>
<p>To evaluate classification models more effectively, consider precision, recall, F1 score, ROC-AUC, and the confusion matrix.</p>
<h4 id="heading-the-confusion-matrix">The Confusion Matrix</h4>
<p>A confusion matrix compares predicted labels with the actual labels. For a binary classifier, it contains four outcomes:</p>
<ul>
<li><strong>True positives (TP):</strong> the model predicts the positive class, and the actual class is positive.</li>
<li><strong>True negatives (TN):</strong> the model predicts the negative class, and the actual class is negative.</li>
<li><strong>False positives (FP):</strong> the model predicts the positive class, but the actual class is negative.</li>
<li><strong>False negatives (FN):</strong> the model predicts the negative class, but the actual class is positive.</li>
</ul>
<p>For a churn model, a true positive is a customer correctly identified as someone who churns. A false positive is a customer incorrectly flagged as likely to churn. A false negative is a customer who churns but wasn't identified by the model.</p>
<p>These error types have different consequences. A false positive might cause the business to spend money on a retention offer for someone who would have stayed anyway. A false negative might mean the business misses an opportunity to reach a customer who is about to leave.</p>
<h4 id="heading-accuracy">Accuracy</h4>
<p>Accuracy measures the proportion of all predictions that are correct.</p>
<p>Accuracy = (TP + TN) ÷ (TP + TN + FP + FN)</p>
<p>It can be useful when the classes and error costs are reasonably balanced. But when one class is much more common than the other, accuracy may hide poor performance on the less common class.</p>
<h4 id="heading-precision">Precision</h4>
<p>Precision measures the proportion of predicted positives that are actually positive.</p>
<p>Precision = TP ÷ (TP + FP)</p>
<p>For a churn model, precision answers: “Of the customers the model flagged as likely to churn, what proportion actually churned?”</p>
<p>High precision can be valuable when contacting a customer is expensive or when false alarms waste limited campaign resources.</p>
<h4 id="heading-recall">Recall</h4>
<p>Recall measures the proportion of actual positives that the model correctly identifies.</p>
<p>Recall = TP ÷ (TP + FN)</p>
<p>For a churn model, recall answers: “Of all customers who actually churned, what proportion did the model identify?”</p>
<p>High recall may be important when missing a customer at risk of churn is particularly costly. But increasing recall can also increase false positives, depending on the model and decision threshold.</p>
<h4 id="heading-f1-score">F1 Score</h4>
<p>The F1 score combines precision and recall using their harmonic mean:</p>
<p>F1 = 2 × (Precision × Recall) ÷ (Precision + Recall)</p>
<p>It can be useful when both precision and recall matter. It doesn't account for true negatives directly, and it shouldn't be treated as the best metric for every business problem.</p>
<h4 id="heading-roc-auc">ROC-AUC</h4>
<p>ROC-AUC summarizes how well a model ranks positive examples above negative examples across different classification thresholds. A value closer to 1 indicates stronger ranking performance, while a value near 0.5 indicates performance similar to random ranking under the usual interpretation.</p>
<p>ROC-AUC doesn't tell you whether a particular threshold is appropriate for a marketing action. When positive cases are rare, precision-recall analysis may also provide useful insight. In either case, evaluate the model on data that wasn't used to train it.</p>
<h4 id="heading-calculating-the-metrics-in-python">Calculating the Metrics in Python</h4>
<p>You can calculate several of these metrics using scikit-learn:</p>
<pre><code class="language-python">from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    roc_auc_score,
    confusion_matrix
)

predictions = model.predict(X_test)

print("Accuracy:", accuracy_score(y_test, predictions))
print(
    "Precision:",
    precision_score(
        y_test, predictions, zero_division=0
    )
)
print(
    "Recall:",
    recall_score(
        y_test, predictions, zero_division=0
    )
)
print(
    "F1 score:",
    f1_score(
        y_test, predictions, zero_division=0
    )
)
print("Confusion matrix:")
print(confusion_matrix(y_test, predictions))
</code></pre>
<p>For ROC-AUC, use the model's probability estimates or decision scores rather than its final predicted labels:</p>
<pre><code class="language-python">probabilities = model.predict_proba(X_test)[:, 1]

print(
    "ROC-AUC:",
    roc_auc_score(y_test, probabilities)
)
</code></pre>
<p>This assumes the model supports <code>predict_proba()</code> and that the positive class is represented by the second probability column. Verify the class ordering for your specific model and target labels.</p>
<h4 id="heading-choosing-the-right-metric">Choosing the Right Metric</h4>
<p>There's no universally best classification metric. The appropriate choice depends on the objective, the class distribution, and the cost of different errors.</p>
<p>For example, a retention team with a limited budget may prioritize precision because it can contact only a small number of customers. Another team may prioritize recall if missing a high-risk customer is especially costly.</p>
<p>Consider the business decision the model supports before selecting a metric. A useful evaluation should explain not only how well the model performs but also whether its mistakes are acceptable for the intended use.</p>
<p>The key lesson here is this: accuracy is only one part of model evaluation. Precision, recall, F1, ROC-AUC, and the confusion matrix reveal different aspects of performance and help connect model quality to business consequences.</p>
<h3 id="heading-chapter-37-customer-segmentation-with-machine-learning">Chapter 37: Customer Segmentation With Machine Learning</h3>
<p>Rule-based segmentation groups customers using predefined conditions. Machine learning offers another approach: clustering. Clustering algorithms look for patterns in data and group observations that are similar according to the features and distance measures used.</p>
<p>For marketing teams, clustering may help explore groups of customers with similar purchasing behavior, spending levels, or engagement patterns. The resulting groups can provide starting points for further analysis, but they don't automatically represent meaningful customer types.</p>
<h4 id="heading-using-k-means-clustering">Using K-Means Clustering</h4>
<p>K-Means is a clustering algorithm that divides observations into a chosen number of clusters. It repeatedly assigns observations to cluster centers and updates those centers to reduce the within-cluster squared distances.</p>
<p>For this example, suppose the dataset already contains one row per customer and includes <code>recency</code>, <code>frequency</code>, and <code>monetary</code> columns.</p>
<pre><code class="language-python">from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

features = df[
    [
        "recency",
        "frequency",
        "monetary"
    ]
].copy()

scaler = StandardScaler()

scaled_features = scaler.fit_transform(
    features
)

model = KMeans(
    n_clusters=4,
    random_state=42,
    n_init=10
)

df["segment"] = model.fit_predict(
    scaled_features
)
</code></pre>
<p>The example standardizes the three features before clustering. This matters because monetary values may be much larger numerically than recency or frequency. Without scaling, features with larger numeric ranges can dominate the distance calculations.</p>
<p>Standardization changes the scale of the input variables, but it doesn'tt make the variables equally important from a business perspective. If the data contains extreme values, you should also consider whether standard scaling is suitable and whether the features need additional preprocessing.</p>
<p>The <code>n_clusters=4</code> setting asks K-Means to produce four clusters. The algorithm assigns each customer a numeric label, which the code stores in the <code>segment</code> column.</p>
<h4 id="heading-cluster-numbers-are-identifiers-not-rankings">Cluster Numbers Are Identifiers, Not Rankings</h4>
<p>The labels <code>0</code>, <code>1</code>, <code>2</code>, and <code>3</code> are simply identifiers for the four groups. <strong>Cluster 0 isn't automatically better, more valuable, or more important than cluster 1.</strong></p>
<p>The numbers don't rank customer quality. If the model is rerun with different settings or data, the numeric labels may be assigned differently even when the overall group patterns are similar.</p>
<p>To understand what each cluster represents, summarize the original feature values for each group:</p>
<pre><code class="language-python">cluster_summary = (
    df.groupby("segment")
    .agg(
        customer_count=("segment", "size"),
        average_recency=("recency", "mean"),
        average_frequency=("frequency", "mean"),
        average_monetary=("monetary", "mean")
    )
    .round(2)
)

print(cluster_summary)
</code></pre>
<p>This summary lets you compare the typical behavior of customers in each cluster. One group might have more recent purchases and higher frequency, while another might contain customers with low frequency and long gaps since their last purchase.</p>
<p>Use those observed patterns to develop descriptive names if they're justified by the data. For example, a group with recent purchases and high frequency might be described as “frequent active customers.” A group with high recency values and previously high spending might be worth investigating for re-engagement.</p>
<p>Don't assign names based only on the cluster number. Inspect the actual values and make sure the description fits the group's behavior.</p>
<h4 id="heading-choosing-and-evaluating-clusters">Choosing and Evaluating Clusters</h4>
<p>The number of clusters is a modeling choice. Four clusters may be useful for one dataset but too many or too few for another. You can compare candidate values using methods such as the elbow method or silhouette score, alongside whether the resulting groups are interpretable and useful for the business.</p>
<p>Clustering results are also sensitive to the features, scaling method, outliers, and selected number of clusters. The groups aren't guaranteed to correspond to natural customer types, and different algorithms may produce different results.</p>
<p>The key lesson: clustering can reveal patterns that help you explore customer groups. The cluster labels themselves have no built-in meaning. Interpret each group by examining its features and validating whether the pattern is useful for the business.</p>
<h3 id="heading-chapter-38-feature-engineering">Chapter 38: Feature Engineering</h3>
<p>Machine learning models often become more useful when raw data is transformed into meaningful features.</p>
<p>Suppose you have:</p>
<pre><code class="language-text">purchase_date
</code></pre>
<p>Instead of giving the raw date directly to the model, you might create:</p>
<pre><code class="language-text">days_since_last_purchase
</code></pre>
<p>Similarly, instead of raw transaction records, you might calculate:</p>
<pre><code class="language-text">total_orders
average_order_value
total_spend
days_since_first_purchase
email_open_rate
</code></pre>
<p>These engineered features can better represent customer behavior.</p>
<p>Feature engineering is often one of the most important parts of a machine learning project.</p>
<h3 id="heading-chapter-39-avoiding-data-leakage">Chapter 39: Avoiding Data Leakage</h3>
<p>Data leakage occurs when information that wouldn't legitimately be available to a model at prediction time influences its training or evaluation. Leakage can make a model appear much more accurate than it will be when used with real, future data.</p>
<p>Imagine that you're building a model to predict whether a customer will make a purchase in the next 30 days. Your goal is to use information available today to estimate what the customer will do during the following month.</p>
<p>If you accidentally include the customer's actual purchase outcome from that future period as an input feature, the model receives information that wouldn't have been available when the prediction was supposed to be made. This makes the evaluation misleading.</p>
<h4 id="heading-an-example-of-data-leakage">An Example of Data Leakage</h4>
<p>Suppose the dataset contains these columns:</p>
<pre><code class="language-text">past_purchases
website_visits
purchased_next_30_days
</code></pre>
<p>The first two columns describe historical behavior. The last column records the outcome you want to predict.</p>
<p>The following code contains a leakage problem:</p>
<pre><code class="language-python"># Incorrect: the target is included as an input feature.
X = df[
    [
        "past_purchases",
        "website_visits",
        "purchased_next_30_days"
    ]
]

y = df["purchased_next_30_days"]
</code></pre>
<p>The target, <code>purchased_next_30_days</code>, appears in both <code>X</code> and <code>y</code>. The model is given the answer it's supposed to predict. It may achieve an extremely high score, but that score doesn't demonstrate that the model can predict future purchases from legitimate historical information.</p>
<p>This is an example of direct target leakage. Other forms of leakage can be less obvious, such as using information collected after the prediction date or preprocessing the entire dataset before splitting it into training and test sets.</p>
<h4 id="heading-fixing-the-leakage">Fixing the Leakage</h4>
<p>The solution is to exclude the target and any other information that wouldn't be available at prediction time from the input features.</p>
<pre><code class="language-python"># Correct: use only information available before prediction.
X = df[
    [
        "past_purchases",
        "website_visits"
    ]
]

y = df["purchased_next_30_days"]
</code></pre>
<p>Next, split the data into training and test sets:</p>
<pre><code class="language-python">from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

model = RandomForestClassifier(
    random_state=42
)

model.fit(
    X_train,
    y_train
)

predictions = model.predict(
    X_test
)

print(
    "Accuracy:",
    accuracy_score(y_test, predictions)
)
</code></pre>
<p>This example is safer because the model receives only the two historical features specified in <code>X</code>. The future purchase outcome remains the target that the model is learning to predict.</p>
<p>But removing the target from the features is only one part of preventing leakage. You must also confirm that <code>past_purchases</code> and <code>website_visits</code> contain information recorded before the prediction date. If either column includes activity from the future 30-day outcome period, leakage can still occur.</p>
<h4 id="heading-preventing-leakage-during-preprocessing">Preventing Leakage During Preprocessing</h4>
<p>Leakage can also happen when preprocessing learns information from records that should be held back for evaluation. For example, if you calculate scaling parameters or fill missing values using the entire dataset before splitting it, information from the test set can influence the training process.</p>
<p>A safer pattern is to fit preprocessing steps using the training data only, then apply the learned transformations to the test data. Scikit-learn pipelines can help enforce this separation.</p>
<p>For time-dependent prediction problems, use a time-based split when appropriate: train using earlier observations and evaluate using later observations. This better reflects the information available when future predictions are made.</p>
<h4 id="heading-a-practical-leakage-checklist">A Practical Leakage Checklist</h4>
<p>Before trusting a model's evaluation, ask:</p>
<ul>
<li>Were all input features available at the intended prediction time?</li>
<li>Does any feature contain the target itself or a direct proxy for the outcome?</li>
<li>Were preprocessing steps fitted using training data only?</li>
<li>Does the train/test split reflect the real prediction setting?</li>
<li>Are records or customers duplicated across the split in a way that could inflate performance?</li>
<li>Is the evaluation based on genuinely unseen data?</li>
</ul>
<p>A surprisingly high score isn't automatically evidence of leakage, but it's a reason to investigate the features, data collection process, and evaluation design carefully.</p>
<p>The key lesson here: a model must be evaluated using only the information it could legitimately access when making a prediction. Preventing leakage helps ensure that reported performance is meaningful rather than artificially inflated.</p>
<h2 id="heading-part-viii-building-a-marketing-intelligence-system">Part VIII: Building a Marketing Intelligence System</h2>
<p>At this point, you've learned many individual techniques. Now we can combine them.</p>
<p>A marketing intelligence system shouldn't simply be a collection of unrelated Python scripts. It should have a logical workflow.</p>
<h3 id="heading-step-1-defining-the-business-questions">Step 1: Defining the Business Questions</h3>
<p>Before writing the system, you should define what it should answer.</p>
<p>For our capstone, we'll ask:</p>
<ol>
<li>How much revenue are our marketing campaigns generating?</li>
<li>Which channels are most efficient?</li>
<li>Which campaigns are performing best?</li>
<li>Which campaigns may need attention?</li>
<li>Which customers are most valuable?</li>
<li>Which customers may need re-engagement?</li>
<li>How is performance changing over time?</li>
<li>Where are customers dropping out of the funnel?</li>
<li>Are there meaningful customer segments?</li>
<li>Can machine learning identify customers at risk of churn?</li>
<li>Can the system automatically generate a report?</li>
<li>Can the system alert the marketing team when important metrics change?</li>
</ol>
<p>These questions determine the architecture of the project.</p>
<h3 id="heading-step-2-project-structure">Step 2: Project Structure</h3>
<p>A practical project can be organized like this:</p>
<pre><code class="language-text">marketing_intelligence/
    data/
        raw/
        processed/

    reports/

    notebooks/

    models/

    src/
        data_loader.py
        cleaning.py
        metrics.py
        customer_analysis.py
        campaign_analysis.py
        segmentation.py
        reporting.py
        alerts.py

    tests/

    main.py

    requirements.txt
</code></pre>
<p>You don't have to create this entire structure on day one. The purpose is to show how a larger project can eventually be organized.</p>
<h3 id="heading-step-3-data-loading-module">Step 3: Data Loading Module</h3>
<p>Create:</p>
<pre><code class="language-text">src/data_loader.py
</code></pre>
<p>Then:</p>
<pre><code class="language-python">import pandas as pd


def load_marketing_data(path):
    return pd.read_csv(path)
</code></pre>
<p>Now the main program can import it:</p>
<pre><code class="language-python">from src.data_loader import (
    load_marketing_data
)

df = load_marketing_data(
    "data/raw/marketing_data.csv"
)
</code></pre>
<p>This keeps data-loading logic separate from analysis logic.</p>
<h3 id="heading-step-4-data-cleaning-module">Step 4: Data Cleaning Module</h3>
<p>Create:</p>
<pre><code class="language-text">src/cleaning.py
</code></pre>
<p>Then:</p>
<pre><code class="language-python">import pandas as pd


def clean_marketing_data(df):
    df = df.copy()

    df = df.drop_duplicates()

    df["date"] = pd.to_datetime(
        df["date"],
        errors="coerce"
    )

    numeric_columns = [
        "impressions",
        "clicks",
        "conversions",
        "cost",
        "revenue"
    ]

    for column in numeric_columns:
        df[column] = pd.to_numeric(
            df[column],
            errors="coerce"
        )

    df["channel"] = (
        df["channel"]
        .str.strip()
        .str.lower()
    )

    return df
</code></pre>
<p>This function performs common cleaning tasks.</p>
<h3 id="heading-step-5-metrics-module">Step 5: Metrics Module</h3>
<p>Create:</p>
<pre><code class="language-text">src/metrics.py
</code></pre>
<p>Then:</p>
<pre><code class="language-python">def calculate_metrics(df):
    df = df.copy()

    df["ctr"] = (
        df["clicks"]
        .div(
            df["impressions"]
            .replace(0, None)
        )
        .mul(100)
    )

    df["conversion_rate"] = (
        df["conversions"]
        .div(
            df["clicks"]
            .replace(0, None)
        )
        .mul(100)
    )

    df["cpc"] = (
        df["cost"]
        .div(
            df["clicks"]
            .replace(0, None)
        )
    )

    df["cpa"] = (
        df["cost"]
        .div(
            df["conversions"]
            .replace(0, None)
        )
    )

    df["roas"] = (
        df["revenue"]
        .div(
            df["cost"]
            .replace(0, None)
        )
    )

    return df
</code></pre>
<p>Now your calculations live in one reusable location.</p>
<h3 id="heading-step-6-campaign-analysis-module">Step 6: Campaign Analysis Module</h3>
<p>Create:</p>
<pre><code class="language-text">src/campaign_analysis.py
</code></pre>
<p>Then:</p>
<pre><code class="language-python">def analyze_campaigns(df):
    summary = (
        df.groupby("campaign")
        .agg({
            "impressions": "sum",
            "clicks": "sum",
            "conversions": "sum",
            "cost": "sum",
            "revenue": "sum"
        })
    )

    summary["roas"] = (
        summary["revenue"] /
        summary["cost"]
    )

    summary["cpa"] = (
        summary["cost"] /
        summary["conversions"]
    )

    return summary.sort_values(
        "roas",
        ascending=False
    )
</code></pre>
<p>This produces a reusable campaign-performance table.</p>
<h3 id="heading-step-7-channel-analysis">Step 7: Channel Analysis</h3>
<pre><code class="language-python">def analyze_channels(df):
    summary = (
        df.groupby("channel")
        .agg({
            "impressions": "sum",
            "clicks": "sum",
            "conversions": "sum",
            "cost": "sum",
            "revenue": "sum"
        })
    )

    summary["roas"] = (
        summary["revenue"] /
        summary["cost"]
    )

    summary["cpa"] = (
        summary["cost"] /
        summary["conversions"]
    )

    return summary.sort_values(
        "roas",
        ascending=False
    )
</code></pre>
<p>Now campaign analysis and channel analysis are separated. That makes the project easier to maintain.</p>
<h3 id="heading-step-8-detecting-underperforming-campaigns">Step 8: Detecting Underperforming Campaigns</h3>
<p>Create a reusable rule:</p>
<pre><code class="language-python">def identify_underperforming_campaigns(
    campaign_summary,
    roas_threshold=1,
    conversion_threshold=2
):
    return campaign_summary[
        (campaign_summary["roas"] &lt; roas_threshold) &amp;
        (
            campaign_summary["conversion_rate"]
            &lt; conversion_threshold
        )
    ]
</code></pre>
<p>But do you notice an issue? Our campaign summary doesn't currently calculate conversion rate.</p>
<p>This is a good example of why building software requires testing your assumptions.</p>
<p>We can fix the summary:</p>
<pre><code class="language-python">def analyze_campaigns(df):
    summary = (
        df.groupby("campaign")
        .agg({
            "impressions": "sum",
            "clicks": "sum",
            "conversions": "sum",
            "cost": "sum",
            "revenue": "sum"
        })
    )

    summary["conversion_rate"] = (
        summary["conversions"] /
        summary["clicks"] *
        100
    )

    summary["roas"] = (
        summary["revenue"] /
        summary["cost"]
    )

    summary["cpa"] = (
        summary["cost"] /
        summary["conversions"]
    )

    return summary.sort_values(
        "roas",
        ascending=False
    )
</code></pre>
<p>This is an important development habit: <strong>Test each stage before building the next stage on top of it.</strong></p>
<h3 id="heading-step-9-building-the-customer-intelligence-layer">Step 9: Building the Customer Intelligence Layer</h3>
<p>The customer layer should summarize behavior at the customer level.</p>
<pre><code class="language-python">def analyze_customers(df):
    summary = (
        df.groupby("customer_id")
        .agg({
            "revenue": "sum",
            "conversions": "sum",
            "campaign": "count"
        })
        .rename(
            columns={
                "campaign":
                "purchase_activity"
            }
        )
    )

    summary["average_order_value"] = (
        summary["revenue"] /
        summary["purchase_activity"]
    )

    return summary
</code></pre>
<p>This gives the system a customer-level perspective.</p>
<h3 id="heading-step-10-identifying-high-value-customers">Step 10: Identifying High-Value Customers</h3>
<p>We can create a simple rule:</p>
<pre><code class="language-python">def identify_high_value_customers(
    customer_summary,
    threshold=5000
):
    return customer_summary[
        customer_summary["revenue"]
        &gt;= threshold
    ]
</code></pre>
<p>Again, the threshold isn't universal.</p>
<p>A $5,000 threshold might be appropriate for one company and absurd for another.</p>
<p>A strong marketing intelligence system should make important thresholds configurable.</p>
<h3 id="heading-step-11-the-recommendation-engine">Step 11: The Recommendation Engine</h3>
<p>Now we can create recommendations.</p>
<pre><code class="language-python">def generate_channel_recommendation(
    row
):
    if row["roas"] &gt;= 4:
        return (
            "Strong performance. "
            "Investigate opportunities "
            "to scale efficiently."
        )

    if row["roas"] &gt;= 2:
        return (
            "Moderate performance. "
            "Continue monitoring and "
            "test optimization opportunities."
        )

    return (
        "Low return. Review targeting, "
        "creative, costs, and conversion "
        "performance before increasing spend."
    )
</code></pre>
<p>Apply it:</p>
<pre><code class="language-python">channel_summary[
    "recommendation"
] = channel_summary.apply(
    generate_channel_recommendation,
    axis=1
)
</code></pre>
<p>This is a basic rule-based intelligence layer.</p>
<p>It doesn't pretend that Python knows the perfect marketing strategy.</p>
<p>Instead, it translates predefined analytical conditions into useful prompts for human decision-makers.</p>
<h2 id="heading-capstone-project-build-a-python-marketing-intelligence-platform">Capstone Project: Build a Python Marketing Intelligence Platform</h2>
<p>The capstone project is the final application that brings the entire book together.</p>
<p>You're going to build a Python-based system that can take raw marketing data and transform it into a structured intelligence report.</p>
<p>The finished system will combine:</p>
<ul>
<li>Python fundamentals</li>
<li>pandas</li>
<li>Data cleaning</li>
<li>Marketing KPIs</li>
<li>Campaign analytics</li>
<li>Channel analytics</li>
<li>Customer analytics</li>
<li>RFM segmentation</li>
<li>Cohort analysis</li>
<li>Funnel analysis</li>
<li>Visualization</li>
<li>Automated alerts</li>
<li>Machine learning</li>
<li>Report generation</li>
<li>Automation</li>
</ul>
<p>The goal isn't to create a fictional "AI marketing platform" that magically knows what customers want. The goal is to build a realistic analytical pipeline.</p>
<h3 id="heading-capstone-objective">Capstone Objective</h3>
<p>The system should be able to answer:</p>
<blockquote>
<p>"What is happening in our marketing data, which areas deserve attention, and what evidence can we provide to support the next decision?"</p>
</blockquote>
<p>That's the central question behind the entire project.</p>
<h3 id="heading-step-1-create-the-dataset">Step 1: Create the Dataset</h3>
<p>For learning purposes, we can generate synthetic data like this:</p>
<pre><code class="language-python">import random
import pandas as pd

random.seed(42)

campaigns = [
    "Spring Sale",
    "Summer Sale",
    "Back to School",
    "Holiday Sale"
]

channels = [
    "Email",
    "Search",
    "Social Media",
    "Display"
]

rows = []

for _ in range(5000):
    impressions = random.randint(
        1000,
        50000
    )

    clicks = random.randint(
        20,
        5000
    )

    conversions = random.randint(
        0,
        500
    )

    cost = round(
        random.uniform(
            100,
            5000
        ),
        2
    )

    revenue = round(
        random.uniform(
            100,
            15000
        ),
        2
    )

    rows.append({
        "customer_id": random.randint(
            1,
            1000
        ),
        "campaign": random.choice(
            campaigns
        ),
        "channel": random.choice(
            channels
        ),
        "impressions": impressions,
        "clicks": clicks,
        "conversions": conversions,
        "cost": cost,
        "revenue": revenue
    })

df = pd.DataFrame(rows)

df.to_csv(
    "marketing_data.csv",
    index=False
)
</code></pre>
<p>This gives us a starting dataset.</p>
<h3 id="heading-step-2-load-the-data">Step 2: Load the Data</h3>
<pre><code class="language-python">df = pd.read_csv(
    "marketing_data.csv"
)
</code></pre>
<p>Immediately inspect it:</p>
<pre><code class="language-python">print(df.head())
print(df.shape)
print(df.info())
</code></pre>
<h3 id="heading-step-3-validate-the-data">Step 3: Validate the Data</h3>
<p>Check missing values:</p>
<pre><code class="language-python">print(
    df.isnull().sum()
)
</code></pre>
<p>Check duplicates:</p>
<pre><code class="language-python">print(
    df.duplicated().sum()
)
</code></pre>
<p>Check channels:</p>
<pre><code class="language-python">print(
    df["channel"].value_counts()
)
</code></pre>
<p>Check campaigns:</p>
<pre><code class="language-python">print(
    df["campaign"].value_counts()
)
</code></pre>
<p>The goal is to understand what entered the system.</p>
<h3 id="heading-step-4-clean-the-data">Step 4: Clean the Data</h3>
<pre><code class="language-python">df = df.drop_duplicates()

numeric_columns = [
    "impressions",
    "clicks",
    "conversions",
    "cost",
    "revenue"
]

for column in numeric_columns:
    df[column] = pd.to_numeric(
        df[column],
        errors="coerce"
    )

df["channel"] = (
    df["channel"]
    .str.strip()
    .str.lower()
)
</code></pre>
<p>Now the data is ready for the analytical layer.</p>
<h3 id="heading-step-5-calculate-kpis">Step 5: Calculate KPIs</h3>
<pre><code class="language-python">df["ctr"] = (
    df["clicks"] /
    df["impressions"] *
    100
)

df["conversion_rate"] = (
    df["conversions"] /
    df["clicks"] *
    100
)

df["cpc"] = (
    df["cost"] /
    df["clicks"]
)

df["cpa"] = (
    df["cost"] /
    df["conversions"]
)

df["roas"] = (
    df["revenue"] /
    df["cost"]
)
</code></pre>
<p>In the production version, we should protect every division from zero values.</p>
<h3 id="heading-step-6-build-the-channel-report">Step 6: Build the Channel Report</h3>
<pre><code class="language-python">channel_summary = (
    df.groupby("channel")
    .agg({
        "impressions": "sum",
        "clicks": "sum",
        "conversions": "sum",
        "cost": "sum",
        "revenue": "sum"
    })
)
</code></pre>
<p>Then:</p>
<pre><code class="language-python">channel_summary["ctr"] = (
    channel_summary["clicks"] /
    channel_summary["impressions"] *
    100
)

channel_summary["conversion_rate"] = (
    channel_summary["conversions"] /
    channel_summary["clicks"] *
    100
)

channel_summary["roas"] = (
    channel_summary["revenue"] /
    channel_summary["cost"]
)

channel_summary["cpa"] = (
    channel_summary["cost"] /
    channel_summary["conversions"]
)
</code></pre>
<p>Sort it:</p>
<pre><code class="language-python">channel_summary = (
    channel_summary
    .sort_values(
        "roas",
        ascending=False
    )
)
</code></pre>
<h3 id="heading-step-7-build-the-campaign-report">Step 7: Build the Campaign Report</h3>
<pre><code class="language-python">campaign_summary = (
    df.groupby("campaign")
    .agg({
        "impressions": "sum",
        "clicks": "sum",
        "conversions": "sum",
        "cost": "sum",
        "revenue": "sum"
    })
)
</code></pre>
<p>Then calculate:</p>
<pre><code class="language-python">campaign_summary["ctr"] = (
    campaign_summary["clicks"] /
    campaign_summary["impressions"] *
    100
)

campaign_summary["conversion_rate"] = (
    campaign_summary["conversions"] /
    campaign_summary["clicks"] *
    100
)

campaign_summary["roas"] = (
    campaign_summary["revenue"] /
    campaign_summary["cost"]
)

campaign_summary["cpa"] = (
    campaign_summary["cost"] /
    campaign_summary["conversions"]
)
</code></pre>
<p>Now you can rank campaigns by performance.</p>
<pre><code class="language-python">campaign_summary.sort_values(
    "roas",
    ascending=False
)
</code></pre>
<h3 id="heading-step-8-build-the-customer-report">Step 8: Build the Customer Report</h3>
<pre><code class="language-python">customer_summary = (
    df.groupby("customer_id")
    .agg({
        "revenue": "sum",
        "conversions": "sum",
        "campaign": "count"
    })
)
</code></pre>
<p>Rename the activity column:</p>
<pre><code class="language-python">customer_summary = (
    customer_summary
    .rename(
        columns={
            "campaign":
            "purchase_activity"
        }
    )
)
</code></pre>
<p>Calculate average value:</p>
<pre><code class="language-python">customer_summary[
    "average_order_value"
] = (
    customer_summary["revenue"] /
    customer_summary[
        "purchase_activity"
    ]
)
</code></pre>
<h3 id="heading-step-9-customer-segmentation">Step 9: Customer Segmentation</h3>
<p>Create RFM-style measurements.</p>
<pre><code class="language-python">rfm = customer_summary.copy()

rfm["frequency"] = (
    rfm["purchase_activity"]
)

rfm["monetary"] = (
    rfm["revenue"]
)
</code></pre>
<p>If your dataset contains actual purchase dates, add recency:</p>
<pre><code class="language-python">today = pd.Timestamp(
    "2026-09-05"
)

last_purchase = (
    df.groupby("customer_id")[
        "purchase_date"
    ].max()
)

rfm["recency"] = (
    today -
    last_purchase
).dt.days
</code></pre>
<p>Then rank customers.</p>
<pre><code class="language-python">rfm["monetary_score"] = pd.qcut(
    rfm["monetary"],
    5,
    labels=False,
    duplicates="drop"
) + 1
</code></pre>
<p>The same concept can be applied to frequency and recency.</p>
<h3 id="heading-step-10-create-customer-segments">Step 10: Create Customer Segments</h3>
<p>You could define simple behavioral groups.</p>
<p>For example:</p>
<pre><code class="language-python">def classify_customer(row):
    if (
        row["monetary"] &gt;= 5000 and
        row["frequency"] &gt;= 5
    ):
        return "High Value"

    if row["frequency"] &gt;= 5:
        return "Frequent"

    if row["monetary"] &gt;= 5000:
        return "High Spending"

    return "Developing"
</code></pre>
<p>Then:</p>
<pre><code class="language-python">rfm["segment"] = (
    rfm.apply(
        classify_customer,
        axis=1
    )
)
</code></pre>
<p>Summarize:</p>
<pre><code class="language-python">segment_summary = (
    rfm.groupby("segment")
    .agg({
        "monetary": "mean",
        "frequency": "mean"
    })
)
</code></pre>
<p>Now the marketing team can see the characteristics of each segment.</p>
<h3 id="heading-step-11-build-the-funnel">Step 11: Build the Funnel</h3>
<p>Add funnel information to your dataset.</p>
<p>For example:</p>
<pre><code class="language-python">funnel = {
    "Visitors": 50000,
    "Product Views": 32000,
    "Cart Adds": 9000,
    "Checkouts": 4500,
    "Purchases": 2700
}
</code></pre>
<p>Calculate stage-to-stage conversion:</p>
<pre><code class="language-python">funnel_rates = {}

previous = None

for stage, value in funnel.items():

    if previous is None:
        funnel_rates[stage] = 100
    else:
        funnel_rates[stage] = (
            value /
            previous *
            100
        )

    previous = value
</code></pre>
<p>This creates a simple funnel analysis.</p>
<h3 id="heading-step-12-create-visualizations">Step 12: Create Visualizations</h3>
<p>Revenue by channel:</p>
<pre><code class="language-python">import matplotlib.pyplot as plt

channel_summary[
    "revenue"
].plot(
    kind="bar"
)

plt.title(
    "Revenue by Marketing Channel"
)

plt.xlabel("Channel")
plt.ylabel("Revenue")

plt.tight_layout()
plt.show()
</code></pre>
<p>ROAS by channel:</p>
<pre><code class="language-python">channel_summary[
    "roas"
].plot(
    kind="bar"
)

plt.title(
    "ROAS by Marketing Channel"
)

plt.xlabel("Channel")
plt.ylabel("ROAS")

plt.tight_layout()
plt.show()
</code></pre>
<p>Customer value distribution:</p>
<pre><code class="language-python">plt.hist(
    rfm["monetary"],
    bins=30
)

plt.title(
    "Customer Monetary Value"
)

plt.xlabel("Customer Revenue")
plt.ylabel("Customers")

plt.tight_layout()
plt.show()
</code></pre>
<h3 id="heading-step-13-create-performance-alerts">Step 13: Create Performance Alerts</h3>
<p>Suppose ROAS below 1 is considered a warning.</p>
<pre><code class="language-python">alerts = []

for channel, row in (
    channel_summary.iterrows()
):

    if row["roas"] &lt; 1:
        alerts.append(
            f"{channel}: ROAS is below 1."
        )
</code></pre>
<p>Display the alerts:</p>
<pre><code class="language-python">for alert in alerts:
    print(alert)
</code></pre>
<p>You could also identify sudden changes in conversion rate or revenue.</p>
<h3 id="heading-step-14-create-recommendations">Step 14: Create Recommendations</h3>
<pre><code class="language-python">recommendations = []

for channel, row in (
    channel_summary.iterrows()
):

    if row["roas"] &gt;= 4:
        recommendations.append(
            f"{channel}: Strong ROAS. "
            "Investigate whether the channel "
            "can be scaled efficiently."
        )

    elif row["roas"] &lt; 1:
        recommendations.append(
            f"{channel}: ROAS is below break-even. "
            "Review targeting, creative, "
            "costs, and conversion performance."
        )

    else:
        recommendations.append(
            f"{channel}: Moderate performance. "
            "Continue testing and optimization."
        )
</code></pre>
<p>This gives the final report an interpretation layer.</p>
<h3 id="heading-step-15-add-machine-learning">Step 15: Add Machine Learning</h3>
<p>Once the analytical system works, we can add a predictive layer.</p>
<p>Suppose the dataset contains a <code>churned</code> column.</p>
<p>Select features:</p>
<pre><code class="language-python">features = [
    "purchase_frequency",
    "total_spend",
    "email_opens",
    "website_sessions"
]

X = df[features]
y = df["churned"]
</code></pre>
<p>Split:</p>
<pre><code class="language-python">from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = (
    train_test_split(
        X,
        y,
        test_size=0.2,
        random_state=42
    )
)
</code></pre>
<p>Train:</p>
<pre><code class="language-python">from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    max_iter=1000
)

model.fit(
    X_train,
    y_train
)
</code></pre>
<p>Predict:</p>
<pre><code class="language-python">predictions = model.predict(
    X_test
)
</code></pre>
<p>Evaluate:</p>
<pre><code class="language-python">from sklearn.metrics import classification_report

print(
    classification_report(
        y_test,
        predictions
    )
)
</code></pre>
<p>The model now becomes an additional layer of intelligence.</p>
<h3 id="heading-step-16-export-the-report">Step 16: Export the Report</h3>
<pre><code class="language-python">with pd.ExcelWriter(
    "marketing_intelligence_report.xlsx"
) as writer:

    df.to_excel(
        writer,
        sheet_name="Processed Data",
        index=False
    )

    channel_summary.to_excel(
        writer,
        sheet_name="Channels"
    )

    campaign_summary.to_excel(
        writer,
        sheet_name="Campaigns"
    )

    customer_summary.to_excel(
        writer,
        sheet_name="Customers"
    )

    rfm.to_excel(
        writer,
        sheet_name="Segments"
    )
</code></pre>
<p>Now the system produces a reusable report.</p>
<h3 id="heading-step-17-turn-the-project-into-a-pipeline">Step 17: Turn the Project Into a Pipeline</h3>
<p>The final step is combining everything.</p>
<p>The main program should eventually look conceptually like:</p>
<pre><code class="language-python">from src.data_loader import (
    load_marketing_data
)

from src.cleaning import (
    clean_marketing_data
)

from src.metrics import (
    calculate_metrics
)

from src.campaign_analysis import (
    analyze_campaigns
)

from src.channel_analysis import (
    analyze_channels
)

from src.customer_analysis import (
    analyze_customers
)


def main():

    df = load_marketing_data(
        "data/raw/marketing_data.csv"
    )

    df = clean_marketing_data(df)

    df = calculate_metrics(df)

    campaigns = analyze_campaigns(
        df
    )

    channels = analyze_channels(
        df
    )

    customers = analyze_customers(
        df
    )

    print(
        "Marketing intelligence analysis "
        "completed."
    )


if __name__ == "__main__":
    main()
</code></pre>
<p>This is the direction we want the capstone to reach.</p>
<p>Instead of one enormous notebook containing hundreds of disconnected cells, the system becomes a collection of reusable components.</p>
<h3 id="heading-what-makes-this-a-marketing-intelligence-system">What Makes This a Marketing Intelligence System?</h3>
<p>At this point, it's worth stepping back. We haven't just created a report. The system can do a whole bunch of things, like:</p>
<ul>
<li>Ingest data</li>
<li>Validate data</li>
<li>Clean data</li>
<li>Transform data</li>
<li>Calculate KPIs</li>
<li>Analyze channels</li>
<li>Analyze campaigns</li>
<li>Analyze customers</li>
<li>Segment customers</li>
<li>Examine funnels</li>
<li>Detect problems</li>
<li>Generate recommendations</li>
<li>Run predictive models</li>
<li>Export reports</li>
</ul>
<p>That combination is what makes the project an intelligence system rather than a simple analytics script.</p>
<p>The system takes raw information and creates a structured process for turning that information into decisions.</p>
<h3 id="heading-what-the-system-still-cant-do">What the System Still Can't Do</h3>
<p>Even a sophisticated marketing intelligence system has limitations.</p>
<ul>
<li>It can't automatically understand every reason a campaign succeeded.</li>
<li>It can't know whether a customer is happy simply because they purchased.</li>
<li>It can't guarantee that a machine learning prediction is correct.</li>
<li>It can't determine the perfect marketing strategy without business context.</li>
<li>It can't replace marketers.</li>
</ul>
<p>Instead, it provides evidence. That distinction should remain at the center of the project.</p>
<h3 id="heading-the-final-challenge">The Final Challenge</h3>
<p>Once you complete the basic capstone, there are various ways you can improve it. You could:</p>
<ul>
<li>Add a database</li>
<li>Connect an API</li>
<li>Add automated scheduling</li>
<li>Create a dashboard</li>
<li>Add cohort analysis</li>
<li>Improve the RFM segmentation</li>
<li>Train a churn model</li>
<li>Add model evaluation</li>
<li>Create automated alerts</li>
<li>Generate a PDF report</li>
<li>Send authorized notifications</li>
<li>Add unit tests</li>
<li>Create configuration files so thresholds can be changed without modifying the source code</li>
</ul>
<p>Eventually, the project should become something that could realistically serve as the foundation of an internal marketing analytics tool.</p>
<p>That's the real purpose of the capstone. It demonstrates that you can take the individual techniques learned throughout this book and combine them into a coherent system.</p>
<h2 id="heading-conclusion-next-steps">Conclusion / Next Steps</h2>
<p>Learning Python for marketing doesn't mean that you have to memorize Python. It just means that you need to learn to think differently about marketing problems.</p>
<p>A spreadsheet can tell you what happened, but Python can help you investigate why. An analytics dashboard can show that revenue fell, but a Python workflow can help determine which channel, campaign, customer segment, or time period contributed to the change. A manual report can summarize the past, but an automated intelligence system can continuously monitor what's happening and bring important changes to someone's attention.</p>
<p>That progression is the real opportunity.</p>
<h3 id="heading-continue-beyond-the-capstone">Continue Beyond the Capstone</h3>
<p>The capstone shouldn't be the final project you ever build. It should be the project that gives you enough confidence to build the next one.</p>
<p>Take the system apart, understand every function, change the metrics, and break the code intentionally.</p>
<p>Try different datasets, add features, and remove features.</p>
<p>Test your assumptions, then rebuild it. The goal is to develop the ability to solve new problems.</p>
<h3 id="heading-build-a-marketing-analytics-portfolio">Build a Marketing Analytics Portfolio</h3>
<p>Once you're comfortable with Python, consider building several smaller projects.</p>
<p>For example:</p>
<ol>
<li><strong>Campaign Performance Analyzer:</strong> Create a program that imports campaign data and automatically calculates KPIs.</li>
<li><strong>Customer Segmentation Tool:</strong> Build an RFM analysis system that categorizes customers based on behavior.</li>
<li><strong>Marketing Funnel Analyzer:</strong> Analyze where users drop out of a conversion process.</li>
<li><strong>A/B Testing Calculator:</strong> Create a tool that compares two experiment groups and reports relevant statistical measurements.</li>
<li><strong>Marketing Report Generator:</strong> Automatically generate a recurring marketing report.</li>
<li><strong>Customer Churn Predictor:</strong> Train a machine learning model that identifies customers who may be at risk of churning.</li>
</ol>
<p>Each project demonstrates a different skill. But together, they create a portfolio that shows you understand more than Python syntax.</p>
<h3 id="heading-keep-the-business-question-first">Keep the Business Question First</h3>
<p>One of the easiest mistakes to make when learning technology is starting with the tool.</p>
<p>You might think: "I learned pandas. What can I do with pandas?"</p>
<p>Instead ask: "What marketing problem could pandas help me solve?"</p>
<p>The same applies to machine learning.</p>
<p>Don't begin with: "I want to use machine learning."</p>
<p>Begin with: "We need to predict which customers are likely to churn."</p>
<p>Then determine whether machine learning is actually appropriate.</p>
<p>Technology should support the problem. The problem shouldn't exist simply to justify the technology.</p>
<h3 id="heading-data-literacy-is-a-marketing-skill">Data Literacy Is a Marketing Skill</h3>
<p>You don't need to become a data scientist to benefit from data literacy.</p>
<p>You should be comfortable asking:</p>
<ul>
<li>Where did this number come from?</li>
<li>What exactly does this metric measure?</li>
<li>What is the denominator?</li>
<li>How was the data collected?</li>
<li>What's missing?</li>
<li>Could there be bias?</li>
<li>Is this correlation or causation?</li>
<li>Is the sample large enough?</li>
<li>Does this result actually matter to the business?</li>
</ul>
<p>These questions are often more important than knowing another Python library.</p>
<h3 id="heading-python-gives-you-leverage">Python Gives You Leverage</h3>
<p>A marketer who can analyze a dataset manually can produce an analysis. But a marketer who can write Python can potentially turn that analysis into a reusable process.</p>
<p>Instead of analyzing one report, they can analyze hundreds. Instead of manually calculating a metric, they can build a function. Instead of manually checking performance every morning, they can create an automated monitoring system. Instead of simply reporting what happened, they can build models that help estimate what might happen next.</p>
<p>That's leverage.</p>
<h3 id="heading-the-future-of-marketing-analytics">The Future of Marketing Analytics</h3>
<p>Marketing technology will continue to change, new platforms will appear, and existing APIs will change.</p>
<p>Machine learning models will improve, and artificial intelligence will become increasingly integrated into marketing workflows.</p>
<p>But the fundamental skills will remain valuable: understanding data, asking good questions, designing experiments, interpreting results, writing reliable code, communicating insights, thinking critically about predictions, and protecting customer information.</p>
<p>These skills aren't tied to one specific platform.</p>
<h3 id="heading-final-thoughts">Final Thoughts</h3>
<p>You don't need to become a programmer before you can use Python. Start with one problem, find the data, write a small program, make a mistake, and fix it. Then make the program better.</p>
<p>Eventually, those small programs become workflows, those workflows become systems, and those systems can become powerful tools for understanding customers, improving campaigns, automating repetitive work, and making better decisions.</p>
<p>Python is the tool, marketing is the context, and data is the evidence. The real skill is connecting all three.</p>
<p>Happy coding!</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Separate Operational Intent from the Executor ]]>
                </title>
                <description>
                    <![CDATA[ In my previous article, I argued that software automation needs a layer between operational intent and execution. The reason is simple: the specification should describe what success means, and the ex ]]>
                </description>
                <link>https://www.freecodecamp.org/news/separate-operational-intent-from-executor/</link>
                <guid isPermaLink="false">6ac242565423a0b19b9f5c6d</guid>
                
                    <category>
                        <![CDATA[ software architecture ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Artificial Intelligence ]]>
                    </category>
                
                    <category>
                        <![CDATA[ TypeScript ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ sdops ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Hugo Teijiz ]]>
                </dc:creator>
                <pubDate>Sun, 04 Oct 2026 12:11:02 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/8b78496e-2ca5-4585-8cfa-f25cb56b56c7.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>In <a href="https://www.freecodecamp.org/news/software-automation-intent-execution-layer/">my previous article</a>, I argued that software automation needs a layer between operational intent and execution.</p>
<p>The reason is simple: the specification should describe what success means, and the executor should decide how to achieve it.</p>
<p>That separation creates a useful possibility.</p>
<p>If the operational specification is independent from the execution mechanism, then more than one executor should be able to satisfy the same specification.</p>
<p>That sounds straightforward. But in practice, it raises several difficult questions:</p>
<pre><code class="language-text">Can two different executors achieve the same operational outcome?

How do we compare them?

What must remain stable when the executor changes?

Which differences are acceptable?

What evidence should each executor produce?

How do we know whether executor independence is real or just theoretical?
</code></pre>
<p>These questions matter because modern software systems rarely keep one execution mechanism forever.</p>
<p>Teams change:</p>
<pre><code class="language-text">CI/CD platforms
cloud providers
deployment systems
infrastructure tools
orchestration engines
incident automation
AI agents
</code></pre>
<p>If changing the executor also changes the meaning of the operation, then the system isn't really specification-driven. The executor still owns too much of the intent.</p>
<p>In this article, I’ll show you how to separate an operational specification from its executors, run the same specification through two different implementations, collect evidence, compare outcomes, and identify where executor independence breaks down.</p>
<p>The goal isn't to prove that two executors behave identically internally.</p>
<p>The goal is to determine whether they can satisfy the same operational contract.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>You should be comfortable with:</p>
<ul>
<li><p>software architecture</p>
</li>
<li><p>interfaces and dependency inversion</p>
</li>
<li><p>TypeScript or a similar language</p>
</li>
<li><p>CI/CD and deployment concepts</p>
</li>
<li><p>observability</p>
</li>
<li><p>basic testing</p>
</li>
<li><p>operational specifications</p>
</li>
</ul>
<p>You don't need Kubernetes, Terraform, or any specific cloud platform. The examples are intentionally small and in-memory so the architecture stays visible.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-what-executor-independence-actually-means">What Executor Independence Actually Means</a></p>
</li>
<li><p><a href="#heading-why-executor-independence-matters">Why Executor Independence Matters</a></p>
</li>
<li><p><a href="#heading-start-with-one-stable-operational-specification">Start with One Stable Operational Specification</a></p>
</li>
<li><p><a href="#heading-define-an-executor-contract">Define an Executor Contract</a></p>
</li>
<li><p><a href="#heading-build-a-first-executor">Build a First Executor</a></p>
</li>
<li><p><a href="#heading-build-a-second-executor">Build a Second Executor</a></p>
</li>
<li><p><a href="#heading-run-the-same-specification-through-both-executors">Run the Same Specification Through Both Executors</a></p>
</li>
<li><p><a href="#heading-compare-evidence-not-internal-steps">Compare Evidence, Not Internal Steps</a></p>
</li>
<li><p><a href="#heading-normalize-executor-specific-evidence">Normalize Executor-Specific Evidence</a></p>
</li>
<li><p><a href="#heading-separate-execution-evidence-from-conformance">Separate Execution Evidence from Conformance</a></p>
</li>
<li><p><a href="#heading-what-counts-as-equivalent-execution">What Counts as Equivalent Execution?</a></p>
</li>
<li><p><a href="#heading-where-executor-independence-usually-breaks">Where Executor Independence Usually Breaks</a></p>
</li>
<li><p><a href="#heading-how-to-test-executor-portability">How to Test Executor Portability</a></p>
</li>
<li><p><a href="#heading-why-this-matters-for-ai-agents">Why This Matters for AI Agents</a></p>
</li>
<li><p><a href="#heading-a-small-end-to-end-example">A Small End-to-End Example</a></p>
</li>
<li><p><a href="#heading-a-practical-workflow">A Practical Workflow</a></p>
</li>
<li><p><a href="#heading-what-executor-independence-does-not-mean">What Executor Independence Does Not Mean</a></p>
</li>
<li><p><a href="#heading-from-replaceable-tools-to-stable-operational-intent">From Replaceable Tools to Stable Operational Intent</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
</ul>
<h2 id="heading-what-executor-independence-actually-means">What Executor Independence Actually Means</h2>
<p>Executor independence means that the operational specification remains valid even if the mechanism that performs the work changes.</p>
<p>For example, suppose the specification says:</p>
<pre><code class="language-text">Deploy Orders version v42.

Constraints:
- at least 3 replicas available
- error rate &lt;= 1%
- p95 latency &lt;= 400 ms
- rollback must remain possible
</code></pre>
<p>One executor may implement that using:</p>
<pre><code class="language-text">Kubernetes rolling deployment
</code></pre>
<p>Another may use:</p>
<pre><code class="language-text">blue/green deployment
</code></pre>
<p>Another may use:</p>
<pre><code class="language-text">a managed cloud deployment service
</code></pre>
<p>And later, an AI agent may generate its own execution plan.</p>
<p>If every executor can be evaluated against the same specification, then the specification is doing its job.</p>
<p>Conceptually:</p>
<pre><code class="language-text">                    ┌── Executor A
Operational Spec ───┼── Executor B
                    ├── Executor C
                    └── AI Agent
</code></pre>
<p>The specification stays stable while the execution mechanism changes.</p>
<p>That's executor independence.</p>
<h2 id="heading-why-executor-independence-matters">Why Executor Independence Matters</h2>
<p>Software teams replace tools constantly. A deployment workflow might move from:</p>
<pre><code class="language-text">Jenkins
↓
GitHub Actions
↓
Argo CD
↓
Kubernetes operator
</code></pre>
<p>If every migration requires rediscovering:</p>
<pre><code class="language-text">what success means
what constraints matter
what evidence is required
when rollback is allowed
</code></pre>
<p>then the operational meaning was never truly independent from the old tool.</p>
<p>This creates several risks.</p>
<ol>
<li><p>Tool lock-in: The system may be technically portable while the operational rules are not.</p>
</li>
<li><p>Hidden behavior changes: A new executor may preserve the same deployment steps but lose an important constraint.</p>
</li>
<li><p>Reimplementation drift: Teams may recreate the old behavior approximately rather than exactly.</p>
</li>
<li><p>Audit gaps: It becomes difficult to prove that the new executor preserves the same operational contract.</p>
</li>
</ol>
<p>Executor independence gives you a stronger migration target: preserve the specification, replace the mechanism.</p>
<h2 id="heading-start-with-one-stable-operational-specification">Start with One Stable Operational Specification</h2>
<p>To test executor independence, first define something stable.</p>
<p>For example:</p>
<pre><code class="language-typescript">type DeploymentSpec = {
  service: string;
  version: string;
  minReplicas: number;
  maxErrorRate: number;
  maxP95LatencyMs: number;
};
</code></pre>
<p>Then:</p>
<pre><code class="language-typescript">const spec: DeploymentSpec = {
  service: "orders",
  version: "v42",
  minReplicas: 3,
  maxErrorRate: 0.01,
  maxP95LatencyMs: 400,
};
</code></pre>
<p>This specification shouldn't contain:</p>
<pre><code class="language-text">kubectl
helm
terraform
AWS
Azure
Argo
GitHub Actions
</code></pre>
<p>Those belong to executors.</p>
<p>The specification describes the operational contract.</p>
<h2 id="heading-define-an-executor-contract">Define an Executor Contract</h2>
<p>Now define the minimum interface an executor must satisfy.</p>
<p>For example:</p>
<pre><code class="language-typescript">type ExecutionEvidence = {
  deployedVersion: string;
  availableReplicas: number;
  errorRate: number;
  p95LatencyMs: number;
};

interface DeploymentExecutor {
  execute(
    spec: DeploymentSpec
  ): Promise&lt;ExecutionEvidence&gt;;
}
</code></pre>
<p>This interface doesn't say how deployment happens.</p>
<p>It only says:</p>
<pre><code class="language-text">given a specification,
perform the operation,
return evidence
</code></pre>
<p>In this article, <strong>evidence</strong> means the observable facts produced or collected after execution that let us evaluate what actually happened. For a deployment, that might include the version that is running, the number of available replicas, the measured error rate, and p95 latency.</p>
<p>Evidence isn't the executor's opinion that the operation succeeded. It's the data we can compare against the specification.</p>
<p>That's important. If the executor interface includes tool-specific concepts, portability starts leaking.</p>
<p>For example, this would be more coupled:</p>
<pre><code class="language-typescript">interface DeploymentExecutor {
  executeKubectlCommand(
    namespace: string,
    manifestPath: string
  ): Promise&lt;void&gt;;
}
</code></pre>
<p>Now the interface already assumes Kubernetes.</p>
<p>That's not executor-independent.</p>
<h2 id="heading-build-a-first-executor">Build a First Executor</h2>
<p>Let’s create a simple in-memory executor.</p>
<pre><code class="language-typescript">class RollingDeploymentExecutor
  implements DeploymentExecutor {
  async execute(
    spec: DeploymentSpec
  ): Promise&lt;ExecutionEvidence&gt; {
    return {
      deployedVersion: spec.version,
      availableReplicas:
        spec.minReplicas,
      errorRate: 0.004,
      p95LatencyMs: 280,
    };
  }
}
</code></pre>
<p>This executor simulates a rolling deployment.</p>
<p>Internally, you can imagine that it:</p>
<pre><code class="language-text">starts new replicas
waits for health
gradually replaces old replicas
</code></pre>
<p>But none of that appears in the specification. The executor owns the mechanism.</p>
<h2 id="heading-build-a-second-executor">Build a Second Executor</h2>
<p>Now create a different strategy.</p>
<pre><code class="language-typescript">class BlueGreenExecutor
  implements DeploymentExecutor {
  async execute(
    spec: DeploymentSpec
  ): Promise&lt;ExecutionEvidence&gt; {
    return {
      deployedVersion: spec.version,
      availableReplicas:
        spec.minReplicas + 2,
      errorRate: 0.003,
      p95LatencyMs: 260,
    };
  }
}
</code></pre>
<p>This executor may conceptually:</p>
<pre><code class="language-text">create a parallel environment
verify it
switch traffic
keep the old environment available
</code></pre>
<p>Its internal process is different, but its evidence shape is the same. That means both can be evaluated against the same operational specification.</p>
<h2 id="heading-run-the-same-specification-through-both-executors">Run the Same Specification Through Both Executors</h2>
<p>Now execute both.</p>
<pre><code class="language-typescript">const rolling =
  new RollingDeploymentExecutor();

const blueGreen =
  new BlueGreenExecutor();

const rollingEvidence =
  await rolling.execute(spec);

const blueGreenEvidence =
  await blueGreen.execute(spec);
</code></pre>
<p>At this point, we have:</p>
<pre><code class="language-text">same specification
different executors
different internal behavior
different evidence values
</code></pre>
<p>The important question isn't whether they performed the same steps. They didn't. The question is if they both satisfied the same operational contract.</p>
<h2 id="heading-compare-evidence-not-internal-steps">Compare Evidence, Not Internal Steps</h2>
<p>Executor independence depends on comparing outcomes rather than implementation details.</p>
<p>Suppose:</p>
<pre><code class="language-text">Rolling deployment:
replicas = 3
error rate = 0.4%
p95 = 280 ms

Blue/green:
replicas = 5
error rate = 0.3%
p95 = 260 ms
</code></pre>
<p>Those outputs aren't identical, but both may be conformant. That matters.</p>
<p>Executor independence doesn't require:</p>
<pre><code class="language-text">same commands
same number of steps
same topology
same timing
same infrastructure
</code></pre>
<p>It requires:</p>
<pre><code class="language-text">same operational intent
satisfied constraints
required evidence
acceptable outcome
</code></pre>
<p>This is similar to interface-based programming. Two implementations can behave differently internally while satisfying the same contract.</p>
<h2 id="heading-normalize-executor-specific-evidence">Normalize Executor-Specific Evidence</h2>
<p>Real executors often return different evidence formats.</p>
<p>Suppose executor A returns:</p>
<pre><code class="language-json">{
  "readyReplicas": 3,
  "image": "orders:v42",
  "latencyP95": 280
}
</code></pre>
<p>Executor B returns:</p>
<pre><code class="language-json">{
  "instancesHealthy": 5,
  "releaseVersion": "v42",
  "p95Ms": 260
}
</code></pre>
<p>These can't be compared directly. You need adapters.</p>
<p>For example:</p>
<pre><code class="language-typescript">type CanonicalEvidence = {
  version: string;
  availableReplicas: number;
  p95LatencyMs: number;
};
</code></pre>
<p>Adapter A:</p>
<pre><code class="language-typescript">function normalizeRolling(
  raw: {
    readyReplicas: number;
    image: string;
    latencyP95: number;
  }
): CanonicalEvidence {
  return {
    version:
      raw.image.split(":")[1],
    availableReplicas:
      raw.readyReplicas,
    p95LatencyMs:
      raw.latencyP95,
  };
}
</code></pre>
<p>This adapter translates the rolling executor's native output into the canonical evidence model. It extracts the version from the image tag, maps <code>readyReplicas</code> to <code>availableReplicas</code>, and renames <code>latencyP95</code> to <code>p95LatencyMs</code>.</p>
<p>The important point is that the adapter doesn't change the operational meaning. It only converts executor-specific field names and formats into the shared representation expected by the specification layer.</p>
<p>Adapter B:</p>
<pre><code class="language-typescript">function normalizeBlueGreen(
  raw: {
    instancesHealthy: number;
    releaseVersion: string;
    p95Ms: number;
  }
): CanonicalEvidence {
  return {
    version:
      raw.releaseVersion,
    availableReplicas:
      raw.instancesHealthy,
    p95LatencyMs:
      raw.p95Ms,
  };
}
</code></pre>
<p>This adapter does the same job for the blue/green executor. Its raw output uses different names, but those values represent the same operational concepts: version, available capacity, and p95 latency.</p>
<p>With both adapters in place, the rest of the system no longer needs to understand the executor-specific shapes. It can evaluate both results using the same canonical model.</p>
<p>Now both executors produce evidence that can be evaluated using the same model.</p>
<p>This is an important architectural boundary. Executor-specific evidence stays near the executor, while canonical evidence belongs to the specification layer.</p>
<h2 id="heading-separate-execution-evidence-from-conformance">Separate Execution Evidence from Conformance</h2>
<p>The executor should produce evidence. It shouldn't decide whether the operation was successful from the perspective of the specification.</p>
<p>That distinction prevents another form of coupling.</p>
<p>For example, avoid:</p>
<pre><code class="language-typescript">return {
  success: true,
};
</code></pre>
<p>A generic success flag tells you very little.</p>
<p>Instead, return evidence:</p>
<pre><code class="language-typescript">return {
  deployedVersion: "v42",
  availableReplicas: 3,
  errorRate: 0.004,
  p95LatencyMs: 280,
};
</code></pre>
<p>Then evaluate it separately.</p>
<pre><code class="language-typescript">type ConformanceResult = {
  conformant: boolean;
  failures: string[];
};

function evaluateConformance(
  spec: DeploymentSpec,
  evidence: ExecutionEvidence
): ConformanceResult {
  const failures: string[] = [];

  if (
    evidence.deployedVersion !==
    spec.version
  ) {
    failures.push(
      "wrong-version"
    );
  }

  if (
    evidence.availableReplicas &lt;
    spec.minReplicas
  ) {
    failures.push(
      "insufficient-replicas"
    );
  }

  if (
    evidence.errorRate &gt;
    spec.maxErrorRate
  ) {
    failures.push(
      "error-rate-too-high"
    );
  }

  if (
    evidence.p95LatencyMs &gt;
    spec.maxP95LatencyMs
  ) {
    failures.push(
      "latency-too-high"
    );
  }

  return {
    conformant:
      failures.length === 0,
    failures,
  };
}
</code></pre>
<p>The evaluator receives two things: the specification that defines what should be true, and the evidence that describes what was actually observed.</p>
<p>It checks each constraint independently. A wrong version adds <code>wrong-version</code>, too few replicas adds <code>insufficient-replicas</code>, and the error-rate and latency checks work the same way.</p>
<p>At the end, <code>conformant</code> is <code>true</code> only when no failures were recorded. Returning the individual failure names is useful because it explains <em>why</em> an execution didn't conform instead of collapsing everything into a generic <code>false</code>.</p>
<p>This is why the executor should return facts rather than a final verdict. The same evidence can be reevaluated later if the specification changes, if an audit needs to reconstruct the decision, or if you want to compare multiple executors using exactly the same rules.</p>
<p>Now:</p>
<pre><code class="language-text">executor → evidence

specification + evidence → conformance
</code></pre>
<p>This separation will become important later.</p>
<h2 id="heading-what-counts-as-equivalent-execution">What Counts as Equivalent Execution?</h2>
<p>Two executors don't need to produce identical evidence. They need to produce evidence that satisfies the same specification.</p>
<p>Suppose:</p>
<pre><code class="language-text">Executor A
replicas: 3
error rate: 0.4%
latency: 280 ms

Executor B
replicas: 5
error rate: 0.3%
latency: 260 ms
</code></pre>
<p>Both may pass.</p>
<p>Now suppose executor B produces:</p>
<pre><code class="language-text">replicas: 2
error rate: 0.3%
latency: 260 ms
</code></pre>
<p>It fails one constraint.</p>
<p>That means:</p>
<pre><code class="language-text">Executor A:
conformant

Executor B:
non-conformant
</code></pre>
<p>The executors are still independent implementations, but only one satisfied the specification for this execution.</p>
<p>That distinction matters.</p>
<p>Executor independence doesn't guarantee executor correctness. It only gives you a stable contract against which correctness can be evaluated.</p>
<h2 id="heading-where-executor-independence-usually-breaks">Where Executor Independence Usually Breaks</h2>
<p>There are several common failure modes.</p>
<h3 id="heading-tool-specific-fields-in-the-specification">Tool-Specific Fields in the Specification</h3>
<p>For example:</p>
<pre><code class="language-yaml">kubernetesNamespace: production
helmChart: orders
</code></pre>
<p>If those are truly implementation details, they shouldn't live in the operational specification.</p>
<h3 id="heading-executor-owned-success-rules">Executor-Owned Success Rules</h3>
<p>If each executor defines its own thresholds, the specification is no longer authoritative.</p>
<h3 id="heading-missing-evidence-normalization">Missing Evidence Normalization</h3>
<p>Different executors may expose different concepts that are never mapped into a common model.</p>
<h3 id="heading-hidden-preconditions">Hidden Preconditions</h3>
<p>Executor A may require an approval. Executor B may not.</p>
<p>If approval is part of operational intent, that rule shouldn't live only inside one executor.</p>
<h3 id="heading-hidden-recovery-behavior">Hidden Recovery Behavior</h3>
<p>One executor may rollback automatically, while another may leave the failed state running.</p>
<p>If recovery behavior matters, the specification should express that expectation.</p>
<h3 id="heading-semantic-mismatch">Semantic Mismatch</h3>
<p>Two tools may use the same words differently.</p>
<p>For example:</p>
<pre><code class="language-text">healthy
ready
available
running
</code></pre>
<p>Those concepts need explicit definitions. Otherwise executor portability is superficial.</p>
<h2 id="heading-how-to-test-executor-portability">How to Test Executor Portability</h2>
<p>You can test executor independence explicitly.</p>
<p>Start with a suite of specifications.</p>
<p>For example:</p>
<pre><code class="language-typescript">const cases: DeploymentSpec[] = [
  {
    service: "orders",
    version: "v42",
    minReplicas: 3,
    maxErrorRate: 0.01,
    maxP95LatencyMs: 400,
  },
  {
    service: "payments",
    version: "v18",
    minReplicas: 5,
    maxErrorRate: 0.005,
    maxP95LatencyMs: 250,
  },
];
</code></pre>
<p>Then run each specification against every executor.</p>
<pre><code class="language-typescript">const executors:
  DeploymentExecutor[] = [
    new RollingDeploymentExecutor(),
    new BlueGreenExecutor(),
  ];

for (const spec of cases) {
  for (const executor of executors) {
    const evidence =
      await executor.execute(spec);

    const result =
      evaluateConformance(
        spec,
        evidence
      );

    console.log({
      spec: spec.service,
      executor:
        executor.constructor.name,
      result,
    });
  }
}
</code></pre>
<p>This produces a useful matrix:</p>
<pre><code class="language-text">               Rolling    BlueGreen
Orders v42     PASS       PASS
Payments v18   PASS       FAIL
</code></pre>
<p>Now you have evidence about executor portability.</p>
<p>That is much stronger than assuming that both tools support deployments, so they're equivalent.</p>
<h2 id="heading-why-this-matters-for-ai-agents">Why This Matters for AI Agents</h2>
<p>AI agents make executor independence even more interesting.</p>
<p>An AI agent may generate a new plan every time.</p>
<p>For example:</p>
<pre><code class="language-text">Execution 1:
scale
deploy
verify
route

Execution 2:
create parallel environment
verify
switch traffic

Execution 3:
deploy canary
observe
expand rollout
</code></pre>
<p>The plans differ, and the executor behavior is dynamic. That makes step-by-step equivalence unrealistic.</p>
<p>But the specification can still remain stable.</p>
<p>For example:</p>
<pre><code class="language-text">Deploy Orders v42.

Constraints:
replicas &gt;= 3
error rate &lt;= 1%
latency &lt;= 400 ms

Evidence:
running version
replicas
error rate
latency
</code></pre>
<p>The AI agent can choose any acceptable plan, and its result is still evaluated using the same contract.</p>
<p>This creates a powerful boundary:</p>
<pre><code class="language-text">agent autonomy
inside
operational constraints
</code></pre>
<p>The agent can optimize execution. It can't silently redefine success.</p>
<h2 id="heading-a-small-end-to-end-example">A Small End-to-End Example</h2>
<p>Let’s put the pieces together.</p>
<p>Start with the specification:</p>
<pre><code class="language-typescript">const spec: DeploymentSpec = {
  service: "orders",
  version: "v42",
  minReplicas: 3,
  maxErrorRate: 0.01,
  maxP95LatencyMs: 400,
};
</code></pre>
<p>Create two executors:</p>
<pre><code class="language-typescript">const executors:
  DeploymentExecutor[] = [
    new RollingDeploymentExecutor(),
    new BlueGreenExecutor(),
  ];
</code></pre>
<p>Run them:</p>
<pre><code class="language-typescript">for (const executor of executors) {
  const evidence =
    await executor.execute(spec);

  const conformance =
    evaluateConformance(
      spec,
      evidence
    );

  console.log(
    executor.constructor.name,
    evidence,
    conformance
  );
}
</code></pre>
<p>Possible result:</p>
<pre><code class="language-text">RollingDeploymentExecutor

evidence:
version = v42
replicas = 3
error rate = 0.004
p95 = 280

conformance:
PASS
</code></pre>
<p>and:</p>
<pre><code class="language-text">BlueGreenExecutor

evidence:
version = v42
replicas = 5
error rate = 0.003
p95 = 260

conformance:
PASS
</code></pre>
<p>The executors didn't behave identically, and they didn't need to. They satisfied the same operational specification.</p>
<p>Now imagine the second executor reports:</p>
<pre><code class="language-text">replicas = 2
</code></pre>
<p>The result becomes:</p>
<pre><code class="language-text">BlueGreenExecutor

conformance:
FAIL

reason:
insufficient-replicas
</code></pre>
<p>That's exactly what we want: the specification remains stable, and the executor changes.</p>
<p>Evidence reveals whether the execution satisfies the contract.</p>
<h2 id="heading-a-practical-workflow">A Practical Workflow</h2>
<p>If you want to test executor independence in a real system, I would use this sequence.</p>
<h3 id="heading-1-pick-one-operation">1. Pick One Operation</h3>
<p>For example:</p>
<pre><code class="language-text">deploy service
restore backup
rotate certificate
scale worker pool
</code></pre>
<h3 id="heading-2-extract-the-operational-specification">2. Extract the Operational Specification</h3>
<p>Define:</p>
<pre><code class="language-text">objective
constraints
evidence requirements
recovery expectations
</code></pre>
<h3 id="heading-3-remove-tool-specific-language">3. Remove Tool-Specific Language</h3>
<p>Look for:</p>
<pre><code class="language-text">kubectl
Terraform
GitHub Actions
AWS CLI
specific resource IDs
</code></pre>
<p>Keep only what is truly part of the operational intent.</p>
<h3 id="heading-4-define-an-executor-interface">4. Define an Executor Interface</h3>
<p>Make the executor responsible for:</p>
<pre><code class="language-text">performing the operation
producing evidence
</code></pre>
<h3 id="heading-5-build-the-first-adapter">5. Build the First Adapter</h3>
<p>Wrap the current implementation, don't rewrite it.</p>
<h3 id="heading-6-build-a-second-executor">6. Build a Second Executor</h3>
<p>Use:</p>
<pre><code class="language-text">another tool
another deployment strategy
a simulator
a test double
an AI agent
</code></pre>
<h3 id="heading-7-normalize-evidence">7. Normalize Evidence</h3>
<p>Map executor-specific observations into a canonical evidence model.</p>
<h3 id="heading-8-evaluate-conformance-separately">8. Evaluate Conformance Separately</h3>
<p>Don't let the executor decide whether it passed.</p>
<h3 id="heading-9-compare-portability">9. Compare Portability</h3>
<p>Run the same specifications through multiple executors.</p>
<h3 id="heading-10-investigate-differences">10. Investigate Differences</h3>
<p>Ask:</p>
<pre><code class="language-text">Is the executor wrong?

Is the specification incomplete?

Is evidence missing?

Are the concepts actually equivalent?
</code></pre>
<p>Those differences are useful. They expose hidden coupling.</p>
<h2 id="heading-what-executor-independence-does-not-mean">What Executor Independence Does Not Mean</h2>
<p>Executor independence doesn't mean that all tools are interchangeable.</p>
<p>Different executors may have different:</p>
<pre><code class="language-text">capabilities
costs
latency
failure modes
security models
operational complexity
</code></pre>
<p>A specification may also require a capability that one executor simply can't provide.</p>
<p>For example:</p>
<pre><code class="language-text">zero-downtime deployment
</code></pre>
<p>may be feasible in one platform and impossible in another.</p>
<p>That's not a failure of the specification. It's useful information. The executor can't satisfy the contract.</p>
<p>Executor independence also doesn't mean ignoring implementation details. Implementation details still matter for:</p>
<pre><code class="language-text">performance
security
cost
reliability
maintainability
</code></pre>
<p>The point is narrower: the operational meaning shouldn't depend on one specific execution mechanism.</p>
<h2 id="heading-from-replaceable-tools-to-stable-operational-intent">From Replaceable Tools to Stable Operational Intent</h2>
<p>Software infrastructure changes constantly.</p>
<p>Tools come and go, execution strategies evolve, cloud platforms change, and agents become more capable.</p>
<p>If operational intent is embedded inside each executor, every change risks becoming a semantic migration.</p>
<p>But if intent is represented independently:</p>
<pre><code class="language-text">Operational specification
          ↓
     stable contract
          ↓
   replaceable executor
</code></pre>
<p>then the system becomes easier to evolve.</p>
<p>This also gives us something else:</p>
<pre><code class="language-text">evidence from multiple executors
evaluated against the same specification
</code></pre>
<p>At that point, we can stop asking:</p>
<blockquote>
<p>Did the tool finish?</p>
</blockquote>
<p>and start asking:</p>
<blockquote>
<p>To what degree did this execution conform to the specification?</p>
</blockquote>
<p>That's the next problem.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Executor independence isn't about pretending every tool is the same. It's about protecting operational intent from implementation churn.</p>
<p>A specification describes:</p>
<pre><code class="language-text">what should happen
what constraints must hold
what evidence is required
</code></pre>
<p>An executor decides:</p>
<pre><code class="language-text">how to make it happen
</code></pre>
<p>Then execution produces evidence. And evidence can be evaluated independently.</p>
<p>That gives us:</p>
<pre><code class="language-text">Specification
      ↓
Executor A ──→ Evidence A
Executor B ──→ Evidence B
      ↓
Evaluation
</code></pre>
<p>Two executors can use completely different strategies and still satisfy the same operational contract.</p>
<p>That's a useful property for ordinary automation.</p>
<p>It becomes even more important when executors are autonomous agents whose internal plans may change from one run to the next.</p>
<p>But once multiple executors can operate against the same specification, a new question becomes unavoidable:</p>
<blockquote>
<p>How should we measure conformance between the specification and what actually happened?</p>
</blockquote>
<p>That's where I want to go next.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ Why Software Automation Needs a Layer Between Intent and Execution ]]>
                </title>
                <description>
                    <![CDATA[ Software automation is good at turning instructions into actions. A pipeline can deploy a release, Terraform can provision infrastructure, a runbook can restart a service, or an agent can inspect tele ]]>
                </description>
                <link>https://www.freecodecamp.org/news/software-automation-intent-execution-layer/</link>
                <guid isPermaLink="false">6abb57d97eb0e171af50644b</guid>
                
                    <category>
                        <![CDATA[ software architecture ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Artificial Intelligence ]]>
                    </category>
                
                    <category>
                        <![CDATA[ TypeScript ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ sdops ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Hugo Teijiz ]]>
                </dc:creator>
                <pubDate>Tue, 29 Sep 2026 06:16:57 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/eb21a403-1e0d-46a4-a8b5-2dbb98095cdd.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Software automation is good at turning instructions into actions.</p>
<p>A pipeline can deploy a release, Terraform can provision infrastructure, a runbook can restart a service, or an agent can inspect telemetry and choose a remediation step.</p>
<p>But all of those systems share the same weakness: they're usually better at executing instructions than at representing the intent behind those instructions.</p>
<p>That gap is easy to ignore while automation is simple. If a script only runs one command, the script itself may be enough to explain what should happen.</p>
<p>But as systems become more distributed, more policy-driven, and more autonomous, execution logic starts carrying responsibilities it was never designed to own.</p>
<p>A deployment workflow may now encode:</p>
<pre><code class="language-text">what should be deployed
where it may run
which constraints must hold
which evidence must be checked
who may approve the change
when rollback is required
which failures are acceptable
</code></pre>
<p>At that point, the automation is no longer just executing. It's also becoming the place where operational intent is defined.</p>
<p>And that's a problem.</p>
<p>Because when intent and execution are fused together, changing the executor can also change the meaning of the operation.</p>
<p>In <a href="https://www.freecodecamp.org/news/executable-operational-specifications-software-automation/">my previous article</a>, I introduced the idea of an executable operational specification: a machine-readable description of what an operation is supposed to achieve and how its outcome can be evaluated.</p>
<p>In this article, I want to go one level deeper.</p>
<p>The central question is:</p>
<blockquote>
<p>What should exist between human intent and the system that executes it?</p>
</blockquote>
<p>I will explore:</p>
<ul>
<li><p>why automation tools often become accidental specifications</p>
</li>
<li><p>why intent and execution evolve at different speeds</p>
</li>
<li><p>what gets lost when intent is embedded in pipelines and scripts</p>
</li>
<li><p>why desired state is useful but still incomplete</p>
</li>
<li><p>how to introduce an explicit operational intent layer</p>
</li>
<li><p>how preconditions, constraints, evidence, and recovery rules fit into that layer</p>
</li>
<li><p>how execution plans differ from operational specifications</p>
</li>
<li><p>how this structure helps with AI agents</p>
</li>
<li><p>and what this separation still can't solve</p>
</li>
</ul>
<p>The goal isn't to add another abstraction for its own sake. The goal is to make operational decisions easier to understand, evaluate, and change without rewriting the meaning of the operation every time the execution mechanism changes.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>You should be comfortable with:</p>
<ul>
<li><p>key software architecture concepts</p>
</li>
<li><p>CI/CD</p>
</li>
<li><p>infrastructure automation</p>
</li>
<li><p>observability</p>
</li>
<li><p>distributed systems</p>
</li>
<li><p>basic policy concepts</p>
</li>
<li><p>TypeScript or a similar language</p>
</li>
</ul>
<p>You don't need a specific orchestration tool or cloud platform. The examples are deliberately small and tool-independent.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-automation-tools-often-become-accidental-specifications">Automation Tools Often Become Accidental Specifications</a></p>
</li>
<li><p><a href="#heading-intent-and-execution-change-at-different-speeds">Intent and Execution Change at Different Speeds</a></p>
</li>
<li><p><a href="#heading-what-gets-lost-when-intent-lives-inside-the-executor">What Gets Lost When Intent Lives Inside the Executor</a></p>
</li>
<li><p><a href="#heading-desired-state-helps-but-its-not-the-whole-intent">Desired State Helps, But It's Not the Whole Intent</a></p>
</li>
<li><p><a href="#heading-the-missing-layer">The Missing Layer</a></p>
</li>
<li><p><a href="#heading-a-small-deployment-example">A Small Deployment Example</a></p>
</li>
<li><p><a href="#heading-separate-preconditions-from-execution-steps">Separate Preconditions from Execution Steps</a></p>
</li>
<li><p><a href="#heading-separate-constraints-from-the-mechanism">Separate Constraints from the Mechanism</a></p>
</li>
<li><p><a href="#heading-make-evidence-requirements-explicit">Make Evidence Requirements Explicit</a></p>
</li>
<li><p><a href="#heading-define-recovery-before-execution-starts">Define Recovery Before Execution Starts</a></p>
</li>
<li><p><a href="#heading-operational-specification-vs-execution-plan">Operational Specification vs Execution Plan</a></p>
</li>
<li><p><a href="#heading-why-this-matters-for-ai-agents">Why This Matters for AI Agents</a></p>
</li>
<li><p><a href="#heading-a-minimal-typescript-model">A Minimal TypeScript Model</a></p>
</li>
<li><p><a href="#heading-a-practical-workflow">A Practical Workflow</a></p>
</li>
<li><p><a href="#heading-what-this-separation-doesnt-solve">What This Separation Doesn't Solve</a></p>
</li>
<li><p><a href="#heading-from-automation-logic-to-operational-contracts">From Automation Logic to Operational Contracts</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
</ul>
<h2 id="heading-automation-tools-often-become-accidental-specifications">Automation Tools Often Become Accidental Specifications</h2>
<p>Consider a deployment workflow. It may begin as something simple:</p>
<pre><code class="language-yaml">steps:
  - build
  - test
  - deploy
</code></pre>
<p>Then production requirements appear.</p>
<p>You add:</p>
<pre><code class="language-text">approval
region validation
health checks
canary percentage
rollback logic
error-rate thresholds
latency thresholds
artifact verification
</code></pre>
<p>Eventually, the pipeline becomes the only place where the operation is fully described.</p>
<p>That creates an accidental architecture:</p>
<pre><code class="language-text">Operational intent
      ↓
pipeline configuration
      ↓
execution
</code></pre>
<p>The pipeline is now doing two jobs: it describes <strong>what the operation means</strong> and <strong>how the operation is executed</strong>.</p>
<p>Those responsibilities are related, but they're not identical.</p>
<p>Suppose this workflow says:</p>
<pre><code class="language-yaml">deploy:
  image: orders:v42

verify:
  errorRateBelow: 0.01

rollback:
  whenVerificationFails: true
</code></pre>
<p>That may work perfectly. But now imagine moving the deployment to another platform.</p>
<p>If the operational rules only exist inside this pipeline, you must rediscover and reimplement them while moving the executor.</p>
<p>The real migration becomes:</p>
<pre><code class="language-text">move execution mechanism
+
reconstruct operational intent
</code></pre>
<p>That's much more dangerous than just changing tooling.</p>
<h2 id="heading-intent-and-execution-change-at-different-speeds">Intent and Execution Change at Different Speeds</h2>
<p>Operational intent is often relatively stable.</p>
<p>For example:</p>
<pre><code class="language-text">Deploy the approved version.

Maintain availability.

Do not exceed the error-rate threshold.

Keep rollback possible.

Do not deploy outside the approved region.
</code></pre>
<p>Those expectations may remain valid for years. But execution mechanisms change much more frequently.</p>
<p>A team may move through:</p>
<pre><code class="language-text">shell scripts
Jenkins
GitHub Actions
Argo CD
Kubernetes operators
cloud-native deployment services
AI agents
</code></pre>
<p>If intent is tightly coupled to those mechanisms, every tooling change risks becoming an operational redesign.</p>
<p>That creates unnecessary churn.</p>
<p>You want:</p>
<pre><code class="language-text">stable intent
     ↓
replaceable execution
</code></pre>
<p>not:</p>
<pre><code class="language-text">new executor
     ↓
rediscover the meaning of the operation
</code></pre>
<p>This separation becomes more important as automation becomes more sophisticated.</p>
<p>The more intelligence the executor contains, the easier it becomes for intent to disappear inside implementation details.</p>
<h2 id="heading-what-gets-lost-when-intent-lives-inside-the-executor">What Gets Lost When Intent Lives Inside the Executor</h2>
<p>When operational intent is embedded directly in scripts and pipelines, several things become harder.</p>
<h3 id="heading-1-review">1. Review</h3>
<p>A reviewer may see:</p>
<pre><code class="language-yaml">if: error_rate &lt; 0.01
</code></pre>
<p>but not know whether that threshold is:</p>
<pre><code class="language-text">a business requirement
an SRE policy
a temporary rollout rule
a historical workaround
</code></pre>
<p>The value is executable. Its meaning isn't explicit.</p>
<h3 id="heading-2-auditability">2. Auditability</h3>
<p>Six months later, somebody asks:</p>
<blockquote>
<p>Why was this deployment allowed?</p>
</blockquote>
<p>The answer may require reconstructing:</p>
<pre><code class="language-text">pipeline version
feature flags
runtime variables
dashboards
approvals
operator decisions
</code></pre>
<h3 id="heading-3-portability">3. Portability</h3>
<p>Changing tools means reimplementing rules.</p>
<h3 id="heading-4-testing">4. Testing</h3>
<p>You can test whether the pipeline works. But it's harder to test whether the operational intent itself is complete.</p>
<h3 id="heading-5-governance">5. Governance</h3>
<p>Authorization rules may become buried in workflow code.</p>
<p>For example:</p>
<pre><code class="language-text">production changes require approval
</code></pre>
<p>may be implemented as a platform-specific branch protection rule instead of being represented as part of the operation.</p>
<h3 id="heading-6-ai-execution">6. AI Execution</h3>
<p>An AI agent may generate a valid sequence of commands while violating an unstated operational constraint.</p>
<p>The problem isn't that the executor is weak. It's that the intent isn't independently represented.</p>
<h2 id="heading-desired-state-helps-but-its-not-the-whole-intent">Desired State Helps, But It's Not the Whole Intent</h2>
<p>Many modern systems already use desired-state models.</p>
<p>For example:</p>
<pre><code class="language-yaml">replicas: 3
image: orders:v42
</code></pre>
<p>That's useful.</p>
<p>The system can compare:</p>
<pre><code class="language-text">desired state
vs.
observed state
</code></pre>
<p>and reconcile the difference.</p>
<p>But operational intent often includes more than final configuration.</p>
<p>Suppose the desired state is:</p>
<pre><code class="language-text">orders:v42
replicas: 3
</code></pre>
<p>The operation may also require:</p>
<pre><code class="language-text">do not reduce availability below 99.9%
do not deploy if payment dependency is degraded
keep error rate below 1%
require approval for production
rollback if latency exceeds 400 ms
</code></pre>
<p>Those aren't simply target-state properties.</p>
<p>Some are:</p>
<pre><code class="language-text">preconditions
runtime constraints
evidence requirements
governance rules
recovery rules
</code></pre>
<p>Desired state is part of operational intent, but it's not always the entire operational intent.</p>
<h2 id="heading-the-missing-layer">The Missing Layer</h2>
<p>A more explicit architecture looks like this:</p>
<pre><code class="language-text">Human / organizational intent
             ↓
Operational specification
             ↓
Execution planning / binding
             ↓
Executor
             ↓
Actions
             ↓
Evidence
             ↓
Evaluation
</code></pre>
<p>The operational specification is the missing layer.</p>
<p>Its job is not to execute commands. Its job is to preserve the meaning of the operation independently from the mechanism performing it.</p>
<p>That layer can contain:</p>
<pre><code class="language-text">objective
scope
preconditions
constraints
required evidence
authorization rules
recovery conditions
</code></pre>
<p>The executor then answers a different question:</p>
<blockquote>
<p>How can I satisfy this specification using the tools available to me?</p>
</blockquote>
<p>That's a cleaner separation.</p>
<h2 id="heading-a-small-deployment-example">A Small Deployment Example</h2>
<p>Suppose we want to deploy version <code>v42</code> of the Orders service.</p>
<p>The operational objective is:</p>
<pre><code class="language-text">Deploy Orders v42 to production.
</code></pre>
<p>But that's not enough.</p>
<p>We also know:</p>
<pre><code class="language-text">the payment dependency must be healthy
at least 3 replicas must remain available
error rate must remain below 1%
p95 latency must remain below 400 ms
production deployment requires approval
rollback must remain possible
</code></pre>
<p>We can represent those concerns separately.</p>
<pre><code class="language-typescript">type DeploymentIntent = {
  service: string;
  version: string;
  environment: string;
  preconditions: {
    paymentDependencyHealthy: boolean;
    approved: boolean;
  };
  constraints: {
    minAvailableReplicas: number;
    maxErrorRate: number;
    maxP95LatencyMs: number;
  };
  recovery: {
    rollbackOnViolation: boolean;
  };
};
</code></pre>
<p>Then:</p>
<pre><code class="language-typescript">const intent: DeploymentIntent = {
  service: "orders",
  version: "v42",
  environment: "production",
  preconditions: {
    paymentDependencyHealthy: true,
    approved: true,
  },
  constraints: {
    minAvailableReplicas: 3,
    maxErrorRate: 0.01,
    maxP95LatencyMs: 400,
  },
  recovery: {
    rollbackOnViolation: true,
  },
};
</code></pre>
<p>Notice what's missing. There's no:</p>
<pre><code class="language-text">kubectl
Terraform
GitHub Actions
Argo
cloud API
</code></pre>
<p>The intent doesn't yet care how the deployment happens.</p>
<p>That's deliberate.</p>
<h2 id="heading-separate-preconditions-from-execution-steps">Separate Preconditions from Execution Steps</h2>
<p>A precondition answers:</p>
<blockquote>
<p>Is this operation allowed to start?</p>
</blockquote>
<p>That's different from:</p>
<blockquote>
<p>Which step should run first?</p>
</blockquote>
<p>For example:</p>
<pre><code class="language-text">payment dependency must be healthy
production approval must exist
artifact must be signed
maintenance window must be open
</code></pre>
<p>These aren't execution instructions. They're conditions that should be true before execution begins.</p>
<p>A simple evaluator might look like this:</p>
<pre><code class="language-typescript">type Preconditions = {
  paymentDependencyHealthy: boolean;
  approved: boolean;
};

function canStart(
  preconditions: Preconditions
): boolean {
  return (
    preconditions.paymentDependencyHealthy &amp;&amp;
    preconditions.approved
  );
}
</code></pre>
<p>If:</p>
<pre><code class="language-typescript">canStart({
  paymentDependencyHealthy: false,
  approved: true,
});
</code></pre>
<p>returns:</p>
<pre><code class="language-text">false
</code></pre>
<p>the operation shouldn't begin.</p>
<p>That decision shouldn't depend on whether the executor is Kubernetes, a script, or an AI agent. The precondition belongs to the operational intent.</p>
<h2 id="heading-separate-constraints-from-the-mechanism">Separate Constraints from the Mechanism</h2>
<p>Constraints answer what must remain true while or after this operation runs.</p>
<p>Examples:</p>
<pre><code class="language-text">replicas &gt;= 3
error rate &lt;= 1%
latency &lt;= 400 ms
cost increase &lt;= 20%
region must remain us-east-1
</code></pre>
<p>An executor may use many different strategies to satisfy them.</p>
<p>For example, to maintain three replicas:</p>
<pre><code class="language-text">Kubernetes may use a rolling deployment
another platform may use blue/green deployment
an agent may temporarily scale capacity before switching traffic
</code></pre>
<p>The mechanism differs, but the constraint does not.</p>
<p>That's exactly why the constraint should live outside the executor.</p>
<h2 id="heading-make-evidence-requirements-explicit">Make Evidence Requirements Explicit</h2>
<p>A constraint is only useful if you can evaluate it.</p>
<p>Suppose you define:</p>
<pre><code class="language-text">error rate &lt;= 1%
</code></pre>
<p>Then you need a source of evidence.</p>
<p>For example:</p>
<pre><code class="language-text">Prometheus
Datadog
CloudWatch
OpenTelemetry
custom application metrics
</code></pre>
<p>The specification shouldn't necessarily hard-code one vendor. But it should say that evidence is required.</p>
<p>For example:</p>
<pre><code class="language-typescript">type EvidenceRequirement = {
  name: string;
  metric: string;
};

const evidenceRequirements:
  EvidenceRequirement[] = [
    {
      name: "availability",
      metric: "availableReplicas",
    },
    {
      name: "errors",
      metric: "errorRate",
    },
    {
      name: "latency",
      metric: "p95LatencyMs",
    },
  ];
</code></pre>
<p>The executor or evidence collector can decide where those values come from. The specification defines what must be known. That distinction is important.</p>
<h2 id="heading-define-recovery-before-execution-starts">Define Recovery Before Execution Starts</h2>
<p>Many operational workflows define rollback only after something goes wrong. But that's too late. Recovery is part of operational intent.</p>
<p>For example:</p>
<pre><code class="language-text">If error rate exceeds 1%, stop rollout.

If p95 latency exceeds 400 ms, restore previous version.

If evidence is unavailable, require human review.
</code></pre>
<p>These aren't just implementation details. They express acceptable operational behavior.</p>
<p>A simple representation might be:</p>
<pre><code class="language-typescript">type RecoveryPolicy = {
  rollbackOnConstraintViolation: boolean;
  requireHumanReviewWhenEvidenceMissing:
    boolean;
};
</code></pre>
<p>The executor can implement rollback differently. The policy still belongs to the operation.</p>
<h2 id="heading-operational-specification-vs-execution-plan">Operational Specification vs Execution Plan</h2>
<p>This distinction is important.</p>
<p>An operational specification answers:</p>
<pre><code class="language-text">What should happen?

What must be true?

What evidence is required?

What happens if constraints fail?
</code></pre>
<p>An execution plan answers:</p>
<pre><code class="language-text">Which actions will we perform?

In what order?

Using which tools?
</code></pre>
<p>For example:</p>
<h3 id="heading-specification">Specification</h3>
<pre><code class="language-text">Deploy Orders v42.

Keep at least 3 replicas available.

Error rate &lt;= 1%.

Rollback if the constraint is violated.
</code></pre>
<h3 id="heading-execution-plan">Execution plan</h3>
<pre><code class="language-text">1. Scale Orders to 6 replicas.
2. Deploy v42 to 1 replica.
3. Route 10% of traffic.
4. Wait 5 minutes.
5. Check metrics.
6. Increase traffic.
7. Remove old replicas.
</code></pre>
<p>The plan is one possible way to satisfy the specification.</p>
<p>Another executor might produce a different plan. That's useful. It means the same operational intent can survive changes in tooling and execution strategy.</p>
<h2 id="heading-why-this-matters-for-ai-agents">Why This Matters for AI Agents</h2>
<p>This separation becomes especially important when the executor is an AI agent.</p>
<p>A traditional script usually has a predictable path while an agent may decide dynamically.</p>
<p>For the same incident, it might choose:</p>
<pre><code class="language-text">restart
scale
reroute
disable feature
change configuration
</code></pre>
<p>If correctness is defined by following one exact sequence, dynamic execution becomes difficult to verify.</p>
<p>But if correctness is defined by an operational specification, the agent has flexibility.</p>
<p>For example:</p>
<pre><code class="language-text">Objective:
restore checkout availability

Preconditions:
payment database reachable

Constraints:
authentication remains enabled
no payment duplication
error rate &lt;= 1%
cost increase &lt;= 20%

Evidence:
checkout success rate
duplicate payment count
error rate
cost delta

Recovery:
escalate if constraints cannot be satisfied
</code></pre>
<p>The agent can choose different actions, but its behavior is still bounded.</p>
<p>That gives us:</p>
<pre><code class="language-text">Intent
↓
Specification
↓
Agent-generated plan
↓
Execution
↓
Evidence
↓
Evaluation
</code></pre>
<p>The agent is free to decide <strong>how</strong>. It's not free to redefine <strong>what success means</strong>.</p>
<p>That distinction will become increasingly important as autonomous systems gain more operational authority.</p>
<h2 id="heading-a-minimal-typescript-model">A Minimal TypeScript Model</h2>
<p>We can combine these ideas into a small generic model.</p>
<pre><code class="language-typescript">type OperationalSpec&lt;TContext, TEvidence&gt; = {
  name: string;

  canStart(
    context: TContext
  ): boolean;

  evaluate(
    evidence: TEvidence
  ): EvaluationResult;

  recovery: RecoveryPolicy;
};

type EvaluationResult = {
  conformant: boolean;
  failures: string[];
};

type RecoveryPolicy = {
  rollbackOnConstraintViolation: boolean;
};
</code></pre>
<p>For a deployment:</p>
<pre><code class="language-typescript">type DeploymentContext = {
  paymentDependencyHealthy: boolean;
  approved: boolean;
};

type DeploymentEvidence = {
  version: string;
  replicas: number;
  errorRate: number;
  p95LatencyMs: number;
};
</code></pre>
<p>Then:</p>
<pre><code class="language-typescript">const deploymentSpec:
  OperationalSpec&lt;
    DeploymentContext,
    DeploymentEvidence
  &gt; = {
    name: "deploy-orders-v42",

    canStart: (context) =&gt;
      context.paymentDependencyHealthy &amp;&amp;
      context.approved,

    evaluate: (evidence) =&gt; {
      const failures: string[] = [];

      if (evidence.version !== "v42") {
        failures.push(
          "wrong-version"
        );
      }

      if (evidence.replicas &lt; 3) {
        failures.push(
          "insufficient-replicas"
        );
      }

      if (evidence.errorRate &gt; 0.01) {
        failures.push(
          "error-rate-too-high"
        );
      }

      if (
        evidence.p95LatencyMs &gt; 400
      ) {
        failures.push(
          "latency-too-high"
        );
      }

      return {
        conformant:
          failures.length === 0,
        failures,
      };
    },

    recovery: {
      rollbackOnConstraintViolation:
        true,
    },
  };
</code></pre>
<p>Now execution can be handled elsewhere.</p>
<p>For example:</p>
<pre><code class="language-typescript">interface DeploymentExecutor {
  execute(
    spec: typeof deploymentSpec
  ): Promise&lt;DeploymentEvidence&gt;;
}
</code></pre>
<p>That executor could be backed by:</p>
<pre><code class="language-text">Kubernetes
a cloud platform
a test harness
a local simulator
an AI agent
</code></pre>
<p>The specification remains the reference point.</p>
<h2 id="heading-a-practical-workflow">A Practical Workflow</h2>
<p>If you want to introduce this separation into an existing automation system, start with one operation.</p>
<h3 id="heading-1-pick-a-repeated-operation">1. Pick a Repeated Operation</h3>
<p>For example:</p>
<pre><code class="language-text">deploy service
rotate certificate
scale worker pool
restore backup
</code></pre>
<h3 id="heading-2-extract-the-objective">2. Extract the Objective</h3>
<p>Write down what the operation is trying to achieve. Avoid commands.</p>
<h3 id="heading-3-extract-preconditions">3. Extract Preconditions</h3>
<p>Ask:</p>
<blockquote>
<p>What must already be true before this operation can begin?</p>
</blockquote>
<h3 id="heading-4-extract-constraints">4. Extract Constraints</h3>
<p>Ask:</p>
<blockquote>
<p>What must remain true during or after execution?</p>
</blockquote>
<h3 id="heading-5-define-evidence">5. Define Evidence</h3>
<p>For every constraint, identify the observation required to evaluate it.</p>
<h3 id="heading-6-define-recovery">6. Define Recovery</h3>
<p>Decide what should happen when evidence shows that a constraint was violated.</p>
<h3 id="heading-7-leave-execution-separate">7. Leave Execution Separate</h3>
<p>Let the current automation tool remain the executor. Don't rewrite everything at once.</p>
<h3 id="heading-8-evaluate-the-existing-executor">8. Evaluate the Existing Executor</h3>
<p>Ask whether the current script, pipeline, or agent satisfies the extracted specification.</p>
<p>This is useful because the separation can be introduced incrementally. You don't need a new operations platform to begin.</p>
<h2 id="heading-what-this-separation-doesnt-solve">What This Separation Doesn't Solve</h2>
<p>Separating intent from execution doesn't automatically make operations correct.</p>
<p>You can still have:</p>
<pre><code class="language-text">wrong objectives
bad thresholds
missing evidence
misleading metrics
conflicting constraints
unsafe executors
security bugs
incorrect recovery policies
</code></pre>
<p>The specification itself can be wrong. That matters.</p>
<p>A formal representation doesn't make an assumption true. It only makes the assumption easier to inspect and evaluate.</p>
<p>There's also a cost. More explicit operational intent means more artifacts to maintain.</p>
<p>If specifications aren't versioned and reviewed, they can become stale. If evidence definitions are weak, conformance results can become misleading. And some operational decisions remain difficult to formalize.</p>
<p>Human judgment doesn't disappear. The value comes from making more of that judgment explicit.</p>
<h2 id="heading-from-automation-logic-to-operational-contracts">From Automation Logic to Operational Contracts</h2>
<p>There's another way to think about this separation.</p>
<p>A mature operational specification starts to behave like a contract.</p>
<p>It says:</p>
<pre><code class="language-text">This operation may begin under these conditions.

It must preserve these constraints.

It must produce this evidence.

Its outcome will be evaluated using these rules.

If it fails, these recovery conditions apply.
</code></pre>
<p>That's much stronger than:</p>
<pre><code class="language-text">run this workflow
</code></pre>
<p>The workflow is an implementation. The operational contract describes what that implementation is responsible for achieving.</p>
<p>This gives us a structure like:</p>
<pre><code class="language-text">Intent
↓
Operational contract
↓
Execution plan
↓
Executor
↓
Evidence
↓
Evaluation
</code></pre>
<p>That separation creates room for something important:</p>
<pre><code class="language-text">multiple execution strategies
for the same operational intent
</code></pre>
<p>And once that's possible, another question appears:</p>
<blockquote>
<p>Can two different executors satisfy the same operational specification?</p>
</blockquote>
<p>That's the question I want to explore next.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Automation has made software operations faster and more repeatable.</p>
<p>But execution is only one part of an operation. We still need to know:</p>
<pre><code class="language-text">why the operation is allowed
what outcome is intended
what constraints must hold
what evidence is required
what happens when the constraints fail
</code></pre>
<p>When all of that logic lives inside a pipeline, script, or agent, the executor becomes the accidental owner of operational meaning.</p>
<p>That coupling makes systems harder to review, migrate, audit, and verify.</p>
<p>A better separation is:</p>
<pre><code class="language-text">Intent
↓
Operational specification
↓
Execution plan
↓
Executor
↓
Evidence
↓
Evaluation
</code></pre>
<p>The specification describes <strong>what success means</strong>. The executor decides <strong>how to achieve it</strong>.</p>
<p>That distinction becomes increasingly valuable as execution mechanisms become more dynamic. Especially when those executors are AI agents.</p>
<p>Because the more freedom we give a system to decide how work gets done, the more explicit we need to be about what the work is supposed to achieve.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Build a Deployment Checklist That Actually Prevents Production Incidents ]]>
                </title>
                <description>
                    <![CDATA[ Count the items on your deployment checklist. If there are more than a handful, the length is worth a second look. Part of it is likely your team being careful. Part of it may be a record of how much  ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-build-a-deployment-checklist-that-actually-prevents-production-incidents/</link>
                <guid isPermaLink="false">6aa1a7edf42d135b6e7c6de3</guid>
                
                    <category>
                        <![CDATA[ deployment ]]>
                    </category>
                
                    <category>
                        <![CDATA[ production ]]>
                    </category>
                
                    <category>
                        <![CDATA[ software development ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Manish Shivanandhan ]]>
                </dc:creator>
                <pubDate>Wed, 09 Sep 2026 18:39:41 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/b2e654b3-dbd0-40af-a21b-c4b63cda4fcb.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Count the items on your deployment checklist. If there are more than a handful, the length is worth a second look.</p>
<p>Part of it is likely your team being careful. Part of it may be a record of how much infrastructure your product team is still checking by hand.</p>
<p>A deployment checklist grows based on application risk: how your own code, data, and rollout can hurt users. Every other item, like certificates, load balancers, image provenance, autoscaling, and connection draining, is a piece of plumbing you're verifying by hand because nothing else verifies it for you.</p>
<p>Those items don't need better wording. They need an owner, and it's worth asking whether that owner should be your product team. This assumes you're already running applications in production, with a pipeline, a rollback story of some kind, and an incident history worth reading.</p>
<p>The question isn't how to start deploying safely. It's which of the risks you currently carry actually need a human carrying them.</p>
<p>In this article, I argue that a deployment checklist should be as long as your application risk requires, and no longer. Most of the extra length is infrastructure work that a person is still checking by hand.</p>
<p>So we'll start by sorting the list into two columns: application risk and infrastructure risk. Then we'll treat each column differently.</p>
<p>Build the application half from your own incidents. Then add the checks that catch even strong teams: background workers, cached data, schema changes, and rollback triggers.</p>
<p>For the infrastructure half, each item still definitely matters. The question is just why a person is the one confirming it, and what it costs you to keep answering that by hand.</p>
<h3 id="heading-what-well-cover">What We'll Cover:</h3>
<ul>
<li><p><a href="#heading-two-kinds-of-risk-in-one-list">Two Kinds Of Risk In One List</a></p>
</li>
<li><p><a href="#heading-column-one-build-it-from-your-own-incidents">Column One: Build It From Your Own Incidents</a></p>
</li>
<li><p><a href="#heading-the-application-questions-worth-asking">The Application Questions Worth Asking</a></p>
</li>
<li><p><a href="#heading-three-application-risks-that-are-rarely-on-the-list">Three Application Risks That Are Rarely On The List</a></p>
</li>
<li><p><a href="#heading-database-migrations-are-where-rollbacks-go-to-die">Database Migrations Are Where Rollbacks Go To Die</a></p>
</li>
<li><p><a href="#heading-separate-deploying-from-releasing">Separate Deploying From Releasing</a></p>
</li>
<li><p><a href="#heading-decide-your-rollback-trigger-before-you-ship">Decide Your Rollback Trigger Before You Ship</a></p>
</li>
<li><p><a href="#heading-column-two-take-it-off-the-list">Column Two: Take It Off The List</a></p>
</li>
<li><p><a href="#heading-keep-it-short-keep-it-in-the-repo">Keep It Short, Keep It In The Repo</a></p>
</li>
</ul>
<h2 id="heading-two-kinds-of-risk-in-one-list">Two Kinds Of Risk In One List</h2>
<p>Open your current checklist and sort every item into two columns.</p>
<p>The first column is application risk. Can this schema change be undone? Will the old consumer understand the new job payload? What happens to sessions already in the cache?</p>
<p>These come out of decisions your team made in a pull request. Nobody outside the team can answer them, because nobody else knows what your code means.</p>
<p>The second column is infrastructure risk. Is the certificate valid? Did the image come from the right commit? Are autoscaling rules correct? Are logs shipping? Is the previous version still around?</p>
<p>These are recurring platform responsibilities, not application-specific judgment calls. Every production team faces them, which is why they should be enforced consistently instead of rechecked by hand.</p>
<p>Here is what that split looks like on a real list. Say your wiki page currently holds eleven items in one flat run. Sorted, it reads like this:</p>
<table>
<thead>
<tr>
<th>Application risk (changes every deploy)</th>
<th>Infrastructure risk (identical every deploy)</th>
</tr>
</thead>
<tbody><tr>
<td>Is the <code>full_name</code> migration reversible?</td>
<td>Is the TLS certificate valid for another 30 days?</td>
</tr>
<tr>
<td>Will the old invoice worker read the new job payload?</td>
<td>Did the image build from the commit we're deploying?</td>
</tr>
<tr>
<td>What happens to sessions already in Redis?</td>
<td>Are autoscaling rules right for this instance size?</td>
</tr>
<tr>
<td>Is <code>PAYMENT_TIMEOUT_MS</code> set in production?</td>
<td>Are logs and metrics shipping from the new instances?</td>
</tr>
<tr>
<td>Has rollback been tested on this service this week?</td>
<td>Is the previous release still there to roll back to?</td>
</tr>
<tr>
<td></td>
<td>Did old instances drain connections before shutdown?</td>
</tr>
</tbody></table>
<p>Read the two columns and the difference jumps out. The left column mentions your column names, your worker, your config keys. It would be meaningless at another company.</p>
<p>The right column would read the same at any company shipping a web app. That's the tell. Anything you could copy into a stranger's wiki without editing it isn't really your team's judgment call.</p>
<p>The first column is a checklist. The second is a list of things that should be answering themselves. Most teams write both into the same document, then wonder why it's too long to use.</p>
<h2 id="heading-column-one-build-it-from-your-own-incidents">Column One: Build It From Your Own Incidents</h2>
<p>Pull your last ten incidents. For each one, ask what single check, run before the deploy, would have caught it. A pattern should appear fast.</p>
<p>Most teams find that three or four causes explain the bulk of their outages: bad config, an unsafe schema change, a dependency that wasn't ready, or a rollback that didn't work.</p>
<p>Four items you use beat forty you ignore. Google's SRE team makes a related point in its chapter on <a href="https://sre.google/sre-book/release-engineering/">release engineering</a>, where the goal is releases that are boring and repeatable rather than careful and heroic.</p>
<p>A working <a href="https://sre.google/sre-book/postmortem-culture/">postmortem practice</a> keeps the list current, since every incident should add an item or delete one. One structural warning. Any item that depends on a human remembering will eventually fail, because people get tired and pipelines do not.</p>
<p>Each item should be automated, or explain why it can't be. The ones that can't be automated are usually judgement calls about your data model, which is exactly what belongs in column one.</p>
<h2 id="heading-the-application-questions-worth-asking">The Application Questions Worth Asking</h2>
<p>Add these to whatever your incidents produced. They cover the failure modes that hit nearly every team, and all of them are about your code rather than your plumbing.</p>
<ul>
<li><p><strong>Can I undo this in under two minutes?</strong> Not whether a rollback plan exists on paper. Whether it has been tested this week, on this service. An untested rollback is a hope, not a control.</p>
</li>
<li><p><strong>Is the config this code needs already in place?</strong> A new variable must be set before the code that reads it arrives, not after. The <a href="https://12factor.net/config">twelve-factor approach</a> keeps config in the environment rather than the code, which makes the ordering explicit.</p>
</li>
<li><p><strong>Is the database change safe on its own?</strong> That one gets its own section below.</p>
</li>
<li><p><strong>Does the app report its own health honestly?</strong> A running process isn't a working app, so the health endpoint has to check what the app actually depends on. Kubernetes formalises the distinction with <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/">readiness and liveness probes</a>: readiness decides whether traffic reaches the instance, while liveness decides whether it gets restarted. Confusing the two turns a slow start into an endless restart loop. Writing an honest health check is application work. Checking by hand that the rollout waited for it is not.</p>
</li>
<li><p><strong>How many users see it first?</strong> Shipping to everyone at once turns every mistake into a full outage. Shipping to five percent turns most mistakes into a blip. None of these ask whether the code is correct. That's what review and tests are for. A deployment checklist asks a narrower question: can delivering correct code still hurt you?</p>
</li>
</ul>
<h2 id="heading-three-application-risks-that-are-rarely-on-the-list">Three Application Risks That Are Rarely On The List</h2>
<p>These catch experienced teams, because nothing in the pull request hints at them.</p>
<p><strong>Background workers are the first.</strong> Web processes and queue consumers ship from the same codebase but not always at the same moment. For a few minutes, a new producer may write payloads an old consumer can't read.</p>
<p>Here's the change that causes it. A developer tidies up a job payload:</p>
<pre><code class="language-python"># NEW producer code. Looks harmless in review.
enqueue("send_invoice", {
    "customer": {"id": 42, "email": "ada@example.com"},
})
</code></pre>
<p>The consumer still running in production is the old one, and it reads a key that no longer exists:</p>
<pre><code class="language-python"># OLD consumer, still live for the next few minutes
def send_invoice(payload):
    customer = db.get_customer(payload["customer_id"])  # KeyError
</code></pre>
<p>Those jobs fail quietly into a dead letter queue and get found hours later.</p>
<p>The fix is to send both shapes for one release, so either version of the consumer can read the message:</p>
<pre><code class="language-python"># Safe producer: old key stays until every consumer is new
enqueue("send_invoice", {
    "customer_id": 42,                                   # old consumers read this
    "customer": {"id": 42, "email": "ada@example.com"},  # new consumers read this
})
</code></pre>
<p>Then deploy the new consumer, let the queue drain, and drop <code>customer_id</code> in a later release. Same rule as a database column: add, migrate, and remove, never all at once.</p>
<p><strong>Serialised data is the second.</strong> Payloads need the same backward compatibility rule as database columns. Anything written into a cache, a session store, or a message body is effectively a schema.</p>
<p>The trap is that no migration file exists, so nothing warns you:</p>
<pre><code class="language-python"># Yesterday's code wrote this shape, and it's still sitting in Redis
cache.set(f"user:{user.id}", {"name": "Ada Lovelace"}, ex=86400)

# Today's code reads it
profile = cache.get(f"user:{user.id}")
first = profile["first_name"]   # KeyError on every object cached before the deploy
</code></pre>
<p>Change the shape of a cached object and old readers break, even though no database was touched. Use a new cache key instead:</p>
<pre><code class="language-python">CACHE_VERSION = "v2"   # bump this whenever the stored shape changes

def cache_key(user_id):
    return f"user:{CACHE_VERSION}:{user_id}"

cache.set(cache_key(user.id), {"first_name": "Ada", "last_name": "Lovelace"}, ex=86400)
</code></pre>
<p>New code misses on the new key, rebuilds from the database, and moves on. The old <code>user:*</code> entries expire on their own. One line of version bump replaces an outage.</p>
<p><strong>Cold-start capacity is the third,</strong> and it sits on the boundary. A rolling deploy briefly serves normal traffic with fewer warm instances.</p>
<p>If your app needs forty seconds to warm its connection pool and the grace period is thirty, you find out at the worst time. This config is the version that bites:</p>
<pre><code class="language-yaml">readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 6      # gives up after ~30s. The app needs 40.
</code></pre>
<p>The instance is killed and restarted right as it was about to be ready, so the rollout loops while traffic piles onto the old instances. Split the two jobs apart instead:</p>
<pre><code class="language-yaml"># Generous window for the first boot only
startupProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 18     # allows 90s to warm the pool

# Tight window once the app is up
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3
</code></pre>
<p>Knowing the warm-up time is application work. Enforcing it on every deploy is a configuration you should set once, not a line you reread each time.</p>
<h2 id="heading-database-migrations-are-where-rollbacks-go-to-die">Database Migrations Are Where Rollbacks Go To Die</h2>
<p>You can roll back code in seconds, but you can't roll back a dropped column.</p>
<p>No tooling anywhere will make this someone else's problem, which is why it's the strictest item on the list.</p>
<p>Never ship a schema change and the code that depends on it in the same deploy. Split it. Add the new column and leave the old one alone, then deploy. Ship code that writes to both. Then ship code that reads the new one. Only much later, once nothing references it, should you drop the old column.</p>
<p>The pattern is called expand and contract. Renaming <code>users.name</code> to <code>users.full_name</code> looks like one line of work. Done safely, it's four deploys.</p>
<p>Deploy one adds the column and nothing else:</p>
<pre><code class="language-sql">-- Nullable, no default, so there is no table rewrite and no long lock
ALTER TABLE users ADD COLUMN full_name varchar(255);
</code></pre>
<p>Deploy two writes both columns and still reads the old one. Nothing user-facing changes yet:</p>
<pre><code class="language-python">def save_user(user, name):
    user.name = name          # still the source of truth
    user.full_name = name     # new column starts filling
</code></pre>
<p>Then backfill the rows that existed before deploy two (in batches so you never hold a lock for long):</p>
<pre><code class="language-sql">-- Repeat until it reports 0 rows
UPDATE users SET full_name = name
WHERE full_name IS NULL
LIMIT 5000;
</code></pre>
<p>Deploy three reads the new column, with a fallback so a rollback to deploy two is still safe:</p>
<pre><code class="language-python">display_name = user.full_name or user.name
</code></pre>
<p>Deploy four is the contract step, and it comes weeks later once nothing in the codebase mentions <code>name</code> and you're sure you won't roll back that far:</p>
<pre><code class="language-sql">ALTER TABLE users DROP COLUMN name;
</code></pre>
<p>It's more work. It also means that at every step, old code and new code can both still run, and that property is what makes fast rollback possible at all.</p>
<p><a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">Blue-green deployment</a> keeps two full environments and switches traffic between them, which only works if your data layer serves both versions. Migration discipline is the foundation. The deploy strategy sits on top.</p>
<p>One more rule: never run a long migration inside the deploy. A change that locks a large table takes your app down while the pipeline reports success. Run it ahead of time and let the deploy be a fast swap of code.</p>
<h2 id="heading-separate-deploying-from-releasing">Separate Deploying From Releasing</h2>
<p>Deploying means the code is on the server, while releasing means users can reach the behaviour. Most teams treat these as one event, which is why every deploy feels risky.</p>
<p>Put new behaviour behind a flag and the two come apart. In the code, that's one branch and one place that decides:</p>
<pre><code class="language-python">def checkout(request, user):
    if flags.enabled("new_checkout", user=user):
        return new_checkout(request, user)
    return legacy_checkout(request, user)
</code></pre>
<p>Deploy with the flag off. The new path is on the server and reachable by nobody:</p>
<pre><code class="language-yaml"># flags.yaml — deploy 1: code is live, behaviour is not
new_checkout:
  enabled: false
</code></pre>
<p>Confirm the app is healthy, then turn it on for your own team:</p>
<pre><code class="language-yaml">new_checkout:
  enabled: true
  audience: internal      # your staff accounts only
</code></pre>
<p>Then one percent of real users, and watch your error rate and latency at that slice:</p>
<pre><code class="language-yaml">new_checkout:
  enabled: true
  audience: percentage
  percentage: 1
</code></pre>
<p>Then everyone, with the cleanup date written down at the same time:</p>
<pre><code class="language-yaml">new_checkout:
  enabled: true
  audience: percentage
  percentage: 100
  remove_by: 2026-11-01   # delete the flag and legacy_checkout by this date
</code></pre>
<p>If something looks wrong at any step, you set <code>enabled: false</code> instead of redeploying, and recovery takes seconds.</p>
<p>This also shortens the list. A change behind a default-off flag really is low risk, so it can skip most items, and the checks move to the flag flip where the risk lives.</p>
<p>Just keep in mind that flags cost something too, since old ones become dead code. So make sure you set a removal date when you add one. That <code>remove_by</code> field is the whole reason it's in the config rather than in someone's head.</p>
<h2 id="heading-decide-your-rollback-trigger-before-you-ship">Decide Your Rollback Trigger Before You Ship</h2>
<p>Rollbacks happen late mostly because nobody agreed in advance on what "bad" looks like. Under pressure, people negotiate and wait one more minute to see if the errors settle.</p>
<p>Write the trigger down before the deploy. Pick two or three numbers, each with a threshold and a window: for example</p>
<ol>
<li><p>an error rate above one percent for two minutes</p>
</li>
<li><p>latency at the ninety-fifth percentile above double its normal value</p>
</li>
<li><p>queue depth climbing without recovery</p>
</li>
</ol>
<p>Roll back when any one is crossed, with no discussion. Watch percentiles, not averages. If one request in twenty now takes eight seconds, the average barely moves while a real share of users suffer.</p>
<p>Then be honest about detection. How long would it take you to notice each signal today? If that's longer than your watch window, the problem isn't your checklist. It's your monitoring.</p>
<h2 id="heading-column-two-take-it-off-the-list">Column Two: Take It Off The List</h2>
<p>Now go back to the infrastructure column and ask a different question of each item. Not "is this worth checking". It is. Instead, ask "why is a person checking it?"</p>
<p>The obvious answer is to automate. Certificate renewal becomes automatic. Image provenance becomes a pipeline control that refuses to promote an unverified build. Config drift becomes a deployment-time check.</p>
<p>Each line you convert genuinely leaves the list, and a team that does this for a quarter will have a visibly shorter document.</p>
<p>Then look at what's left after the easy passes, because the remainder is where the argument actually is. You'll have items like health-gated rollout, where traffic shifts only after new instances pass their checks. Or zero-downtime replacement: draining connections, waiting for in-flight requests, sequencing instance turnover, and honouring readiness before moving traffic.</p>
<p>Keeping the previous release built and warm enough to return to in seconds. A preview environment per pull request, built the way production is built, so that "it worked in staging" stops being a category of incident.</p>
<p>These don't respond to a scripting afternoon. Each is a distributed systems problem with edge cases you discover in production, and each stays on the list as a human verification step until it's fully solved.</p>
<p>Teams that commit to solving these issues properly aren't writing scripts anymore. They're building and staffing an internal delivery platform, which is a real product with a roadmap, an on-call rotation, and a maintenance cost that doesn't end.</p>
<p>The work is legitimate. It's also the same work, done once per company, that a <a href="https://www.freecodecamp.org/news/my-team-s-experience-moving-from-aws-to-a-paas/">platform as a service</a> has already amortised across everyone using it, with the edge cases found by someone else's outages.</p>
<p>Rollback shows the gap most clearly. Owned in-house, it's a procedure a person performs under pressure, and the only thing that makes it trustworthy is rehearsing it often enough that the rehearsal itself becomes a standing cost.</p>
<p>Owned by the platform, the checklist item collapses to confirming that the previous release exists, which the platform also guarantees. The reliability difference isn't a matter of how well the runbook is written.</p>
<p>There's a real limit to this. Opinionated platforms constrain unusual networking, specialised hardware, and fine-grained tuning of the layers they abstract. So teams with those requirements will hit the edges and should stay where they are. It's worth checking whether that describes you, because most teams shipping web applications and APIs assume it does long before it's true.</p>
<h2 id="heading-keep-it-short-keep-it-in-the-repo">Keep It Short, Keep It In The Repo</h2>
<p>What survives is a list about your application, and it fits in a pull request template. Six tick boxes in front of the reviewer beat a wiki page every time.</p>
<p>A workable version reads like this:</p>
<ul>
<li><p>Confirm the change is backwards compatible with the running version, including job payloads and cached data.</p>
</li>
<li><p>Confirm any schema change ships in an earlier deploy.</p>
</li>
<li><p>Confirm new config already exists in the target environment.</p>
</li>
<li><p>Name the rollback trigger.</p>
</li>
<li><p>Say how the change reaches users.</p>
</li>
<li><p>Name who watches for the first fifteen minutes.</p>
</li>
</ul>
<p>That last item is underrated. A deploy nobody is watching is a bet, which is also why shipping at the end of the day hurts so often. The deploy is fine. The empty room is the problem.</p>
<p>Then measure whether the list works. The four <a href="https://dora.dev/guides/dora-metrics/">DORA metrics</a> give you the read you need. Change failure rate says whether your checks catch real problems. Recovery time says whether your rollback story is honest. And deployment frequency says whether the process has grown too heavy to use.</p>
<p>The teams that deploy most often break production least often, because speed and safety both come from a delivery path so automated that little is left for a human to get wrong.</p>
<p>So treat the checklist as a running record of risks a human is still absorbing on the system's behalf. Automate what automation can reach, and for the rest, be clear-eyed that keeping the item means keeping the infrastructure that produced it. A shrinking checklist is a maturing system. A list that stays long is a decision, not an accident.</p>
<p>Hope you enjoyed this article. You can <a href="https://linkedin.com/in/manishmshiva">connect with me on LinkedIn</a>.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ Learn the AI SDLC – The Complete Guide to Building Agent Skills ]]>
                </title>
                <description>
                    <![CDATA[ Perhaps you can relate to this scenario: you've explained AI models four times using the same instructions this week. You've talked repeatedly about how your team structures a deck. Which checks run b ]]>
                </description>
                <link>https://www.freecodecamp.org/news/learn-the-ai-sdlc-the-complete-guide-to-building-agent-skills/</link>
                <guid isPermaLink="false">6a958dafb6e371335a86ea8c</guid>
                
                    <category>
                        <![CDATA[ AI ]]>
                    </category>
                
                    <category>
                        <![CDATA[ ai-sdlc ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ MathJax ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Sarvesh Talele ]]>
                </dc:creator>
                <pubDate>Mon, 31 Aug 2026 14:20:31 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/eeed67e6-32c9-45df-a2f4-9894f2e49f00.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Perhaps you can relate to this scenario: you've explained AI models four times using the same instructions this week.</p>
<p>You've talked repeatedly about how your team structures a deck. Which checks run before a deploy. Why the staging database isn't the one in the README.</p>
<p>Every time, you type it out again. Every time, the agent does a decent job. And every time, the next session starts from zero.</p>
<p>That's the problem <strong>Agent Skills</strong> solves.</p>
<p>A skill is a folder with one Skill.md file in it. The agent reads a one-line summary at startup, and opens the full instructions only when a task actually needs them. Write the explanation once, commit it next to your code, and every agent on your team can access it now and automate any boring task.</p>
<p>The format started at <a href="https://www.anthropic.com/">Anthropic</a>, was released as an open standard, and now it supports more than 45 tools that work with it, including Claude Code, VS Code, GitHub Copilot, Cursor, Gemini CLI, Codex, Goose, and JetBrains Junie. Google Antigravity supports it too. One folder, every tool.</p>
<h2 id="heading-automate-repeatable-tasks-with-agent-skills">Automate Repeatable Tasks with Agent Skills</h2>
<p>Most guides scatter a dozen half-finished examples across a dozen sections. Here, we'll build an actual agent skill from scratch which will be compatible across all the AI tools.</p>
<p>We'll build <strong>exactly one skill</strong>, called <code>deck-builder</code>, and it's the only example in this article. It teaches an agent to turn a vague request like <em>"make me a deck about the Q3 migration"</em> into a real presentation outline by <strong>brainstorming first and writing slides second</strong>.</p>
<p>That ordering is the whole point. Ask any model for a deck, and it starts generating slide one immediately. You get twelve slides of tidy, generic bullets that never decide what the presentation is <em>for</em>.</p>
<p>A human who's good at this does something different. They ask who is in the room, what's the one message they want to convey to the audience, and which outline or framework they want to use. Then they'll start creating the presentation based on a mental model.</p>
<p>The skill starts as 28 lines of Markdown you can write in ten minutes. By the end, it has a tuned description, a false positives list, a bundled validator, an on-demand reference file, an eval suite, and a clean security scan. Every idea in this guide gets demonstrated on that same folder as it grows.</p>
<p>Nothing here needs an account, an API key, or PowerPoint. You'll paste the skill in the skills folder, and it will create a presentation based on the skill you have created.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-automate-repeatable-tasks-with-agent-skills">Automate Repeatable Tasks with Agent Skills</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-what-agent-skills-actually-are">What Agent Skills Actually Are</a></p>
</li>
<li><p><a href="#heading-why-your-giant-system-prompt-stops-working">Why Your Giant System Prompt Stops Working</a></p>
</li>
<li><p><a href="#heading-token-optimisation-for-agent-skills">Token Optimisation for Agent Skills</a></p>
</li>
<li><p><a href="#heading-anatomy-of-a-skill">Anatomy of a Skill</a></p>
</li>
<li><p><a href="#heading-build-version-1-in-ten-minutes">Build Version 1 in Ten Minutes</a></p>
</li>
<li><p><a href="#heading-run-it-in-vs-code-or-claude-code">Run It in VS Code or Claude Code</a></p>
</li>
<li><p><a href="#heading-version-2-make-it-trigger-every-time">Version 2: Make It Trigger Every Time</a></p>
</li>
<li><p><a href="#heading-version-3-write-a-body-that-earns-its-tokens">Version 3: Write a Body That Earns Its Tokens</a></p>
</li>
<li><p><a href="#heading-version-4-bundle-a-script-that-checks-the-work">Version 4: Bundle a Script That Checks the Work</a></p>
</li>
<li><p><a href="#heading-version-5-move-the-deep-material-out-of-the-way">Version 5 Move the Deep Material Out of the Way</a></p>
</li>
<li><p><a href="#heading-where-skills-live-and-how-agents-find-them">Where Skills Live and How Agents Find Them</a></p>
</li>
<li><p><a href="#heading-skills-vs-rules-vs-mcp-vs-hooks-vs-plugins">Skills vs Rules vs MCP vs Hooks vs Plugins</a></p>
</li>
<li><p><a href="#heading-prove-the-skill-actually-helps">Prove the Skill Actually Helps</a></p>
</li>
<li><p><a href="#heading-scan-before-you-install">Scan Before You Install</a></p>
</li>
<li><p><a href="#heading-mistakes-that-quietly-waste-your-tokens">Mistakes That Quietly Waste Your Tokens</a></p>
</li>
<li><p><a href="#heading-the-pre-flight-checklist">The Pre-Flight Checklist</a></p>
</li>
<li><p><a href="#heading-key-takeaways">Key Takeaways</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
</ul>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>You don't need deep knowledge of AI agents to follow this guide, but a few basics will make the examples easier to work through.</p>
<p>You should be comfortable working with Markdown files, navigating a project directory, and running simple commands in a terminal. The examples use Python for the bundled validator, so you'll also need <strong>Python 3.9 or later</strong> installed on your machine.</p>
<p>For the walkthrough, you'll work with an AI coding client that supports Agent Skills, such as Claude Code, VS Code with GitHub Copilot, or Google Antigravity. The skill itself doesn't require an account, API key, or PowerPoint.</p>
<p>That's all you need. We'll start with a single <code>SKILL.md</code> file, then build it up step by step as the workflow becomes more useful and more reliable.</p>
<h2 id="heading-what-agent-skills-actually-are">What Agent Skills Actually Are</h2>
<p>A skill is a directory. Inside it, there's one required file called <code>SKILL.md</code>.</p>
<p>That file has two parts: a short YAML block at the top, and Markdown instructions below it. Everything else is optional.</p>
<p>Note: If you add more files in this folder, it'll consume more tokens, as the context window will increase due to irrelevant or dead files.</p>
<p>Here's where <code>deck-builder</code> ends up by the end of this guide:</p>
<pre><code class="language-python">deck-builder/
├── SKILL.md                                
├── scripts/
│   └── validate_deck.py          
├── references/
│   └── narrative-patterns.md
└── evals/
    └── evals.json                
</code></pre>
<p>You need to install Python for execution of scripts. A skill is a Markdown file with metadata and instructions specific to the goal for which we want to create the skill.</p>
<p>What makes the format worth using is what it captures:</p>
<ul>
<li><p><strong>Expertise the agent can't guess:</strong> Your review gates, presentation style, deck conventions, frameworks, and so on.</p>
</li>
<li><p><strong>Workflows that repeat:</strong> Multi-step tasks become a consistent procedure instead of improvisation.</p>
</li>
<li><p><strong>Reuse across tools:</strong> Build once, run in any compatible client (Claude Code, VS Code, and so on)</p>
</li>
</ul>
<p>The mental model that works best is an <strong>executable runbook</strong>. Think of a good skill as the document you would hand to a sharp new hire on day one: it explains the exact workflow, calls out the common traps, provides a script for the tedious parts, and shows you how to verify that everything worked.</p>
<h2 id="heading-why-your-giant-system-prompt-stops-working">Why Your Giant System Prompt Stops Working</h2>
<p>Most teams start by dumping everything into one always-on file: <code>AGENTS.md</code>, <code>CLAUDE.md</code>, or a system prompt in code. Deck conventions go in there, wedged between the deploy runbook and the style guide.</p>
<p>With two conventions, this is fine. But it breaks for three reasons.</p>
<h3 id="heading-1-it-costs-you-on-every-single-request">1. It costs you on every single request</h3>
<p>The system prompt loads with every API call. Twenty runbooks can reach tens of thousands of tokens before the user types anything.</p>
<p>You pay that on turn one. You pay it again on turn forty. You pay it when the conversation has nothing to do with presentations.</p>
<h3 id="heading-2-long-context-doesnt-mean-even-attention">2. Long context doesn't mean even attention</h3>
<p>A big context window isn't the same as uniform attention across it.</p>
<p>Instruction-following degrades when the line that matters is buried in thousands of unrelated lines. "<em>Brainstorm before you write slides</em>" is obeyed reliably in a 2,000-token prompt. In an 80,000-token one, the main points get skipped. Loading more context isn't the same as being understood.</p>
<h3 id="heading-3-prose-cant-enforce-a-procedure">3. Prose can't enforce a procedure</h3>
<p>Ask a model to "<em>check the outline</em>", and you get a different check each time. Sometimes a thorough one, and other times a sentence of praise.</p>
<p>Prose makes an agent <em>likely</em> to do the right thing. Only a script makes it <em>verifiable</em>.</p>
<p>Progressive disclosure fixes the first two. The validator we'll add in version 4 fixes the third point.</p>
<h2 id="heading-token-optimisation-for-agent-skills">Token Optimisation for Agent Skills</h2>
<p>Agents load skills in three stages. Understanding where each stage ends is what keeps a skill efficient.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69e9e11d16b084c068eb65a1/74d9273f-6adb-44a6-8de1-aa544f84613e.png" alt="Diagram showing an AI-native skill execution workflow with three stages: Discovery → Activation → Execution. A user request triggers skill discovery, where the agent initially sees only skill names and descriptions at a low token cost. When a matching skill is activated, the full SKILL.md is loaded along with workflow templates and examples, increasing context cost. During execution, the agent loads only the specific reference files or scripts needed for the task, such as narrative-patterns.md or validate-deck.py, minimizing token usage while keeping the required context available." style="display: block;" width="4990" height="1135" loading="lazy">

<p>The specification puts real numbers on this. Roughly 100 tokens of metadata per skill at discovery. A recommended ceiling of 500 lines or 5,000 tokens for the body at activation. Anything heavier belongs in <code>references/</code>.</p>
<h3 id="heading-activation-isnt-a-one-time-charge">Activation isn't a one-time charge</h3>
<p>This detail changes how you should write a skill. <strong>Activation isn't a one-time cost that disappears after the agent finishes the task.</strong> Once the skill is activated, its instructions remain in the conversation and continue consuming context for the rest of the session.</p>
<p>Claude Code documents it precisely: the rendered <code>SKILL.md</code> enters the conversation as a single message and <strong>stays there for the rest of the session</strong>. The file isn't re-read on later turns.</p>
<p>There are three consequences here:</p>
<ul>
<li><p><strong>Every line is a recurring cost:</strong> A bloated body is charged on turn twenty, not just the turn that triggered it.</p>
</li>
<li><p><strong>Write standing instructions, not one-time steps:</strong> The agent sees the same text all session.</p>
</li>
<li><p><strong>Compaction can evict your skill:</strong> When Claude Code compacts a long conversation, it re-attaches the most recent invocation of each skill, keeping the first 5,000 tokens of each within a combined 25,000-token budget, most recent first. Invoke many skills, and the earliest ones drop out entirely.</p>
</li>
</ul>
<p>That last point explains a symptom people often misread. When a skill seems to stop working mid-session, the instructions haven't necessarily disappeared. The model may simply be favouring another approach. In that case, strengthen the skill’s instructions or re-invoke it after context compaction.</p>
<h3 id="heading-the-math-briefly">The Math, Briefly</h3>
<p>Let N be the number of skills you have installed, T (descending) the metadata cost of one, T (full) the cost of one body, and k the number of skills a task actually opens.</p>
<p>Load everything up front:</p>
<p>$$C_{\text{static}} = N \times T_{\text{full}}$$</p>
<p>Load progressively:</p>
<p>$$C_{\text{progressive}} = N \times T_{\text{desc}} + k \times T_{\text{full}}$$</p>
<p>$$Use\ the\ spec's\ own\ budgets: T_{desc} = 100\ and\ T_{full} = 5,000,\ for\ 50\ skills\ where\ one\ activates.$$</p>
<p>$$C_{\text{static}} = 50 \times 5{,}000 = 250{,}000\text{ tokens}$$</p>
<p>$$C_{\text{progressive}} = (50 \times 100) + 5{,}000 = 10{,}000\text{ tokens}$$</p>
<p>$$\rho = 1 - \frac{10{,}000}{250{,}000} = 0.96$$</p>
<p><strong>A 96% reduction</strong> for the same task with the same fifty capabilities available.</p>
<p>The formula also shows where it breaks. Progressive disclosure wins while k ≪ N, meaning while most skills stay shut. If every request opens half your catalogue, your skills are scoped too broadly, and you have rebuilt the monolith skill file. Keeping k small is a design goal.</p>
<h2 id="heading-anatomy-of-a-skill">Anatomy of a Skill</h2>
<h3 id="heading-the-frontmatter">The Frontmatter</h3>
<p>The YAML block at the top has two required fields and four optional ones.</p>
<table>
<thead>
<tr>
<th>Field</th>
<th>Required</th>
<th>Rules</th>
</tr>
</thead>
<tbody><tr>
<td><code>name</code></td>
<td>Yes</td>
<td>1–64 characters. Lowercase letters, digits, hyphens. No leading, trailing, or doubled hyphens. Must match the folder name.</td>
</tr>
<tr>
<td><code>description</code></td>
<td>Yes</td>
<td>1–1024 characters. Says what the skill does <strong>and</strong> when to use it.</td>
</tr>
<tr>
<td><code>license</code></td>
<td>No</td>
<td>A license name, or a bundled license file.</td>
</tr>
<tr>
<td><code>compatibility</code></td>
<td>No</td>
<td>Up to 500 characters. Environment needs: product, packages, and network access.</td>
</tr>
<tr>
<td><code>metadata</code></td>
<td>No</td>
<td>Free-form string map for your own tooling.</td>
</tr>
<tr>
<td><code>allowed-tools</code></td>
<td>No</td>
<td>Space-separated list of pre-approved tools. Marked experimental by the spec, Claude Code implements it fully. See the note below.</td>
</tr>
</tbody></table>
<p>Here is what <code>deck-builder</code> ends up with:</p>
<pre><code class="language-yaml">---
name: deck-builder
description: &gt;-
  Turn a request for a presentation into a slide outline. Brainstorm the
  audience, core message, and narrative arc first, then write slides. Use
  when someone asks for a deck, slides, a presentation, a readout, a board
  update, or a talk, including when they only say "put something together
  for Thursday" without naming a format.
license: Apache-2.0
compatibility: Requires Python 3.9+. No network access needed.
---
</code></pre>
<p>Two rules are easy to overlook, especially when you start making a skill portable across clients. The first is the <code>name</code> field: it isn't just a label. Under the specification, it must match the parent folder exactly. So if your skill lives at <code>deck-builder/SKILL.md</code>, the frontmatter must use <code>name: deck-builder</code>, not <code>name: deckBuilder</code>.</p>
<p>The second is the <code>description</code> limit. The specification allows up to 1024 characters, but that limit becomes surprisingly easy to cross once you start tuning the description for better triggering in Version 2. Some clients are more permissive and accept variations of these rules, but that flexibility can create a portability problem. Write against the strict specification first, and your skill has a much better chance of behaving consistently everywhere.</p>
<h3 id="heading-where-clients-extend-the-six-fields">Where Clients Extend the Six Fields</h3>
<p>That table is the <strong>specification</strong>. Individual clients relax parts of it and add their own fields. This is worth knowing before you write frontmatter you plan to share.</p>
<p>Claude Code is the most extended implementation. It treats <code>name</code> as optional and defaults to the folder name. In a personal or project skill, <code>name</code> sets only the display label while the folder name sets the command. It treats <code>description</code> as recommended rather than required, falling back to the body's first paragraph. It truncates the description in its skill list at 1,536 characters. It accepts <code>license</code> and <code>compatibility</code> without acting on them.</p>
<p>On top of the six, it adds about a dozen more, including <code>when_to_use</code>, <code>argument-hint</code>, <code>arguments</code>, <code>disable-model-invocation</code>, <code>user-invocable</code>, <code>disallowed-tools</code>, <code>model</code>, <code>effort</code>, <code>context</code>, <code>agent</code>, <code>hooks</code>, <code>paths</code>, and <code>shell</code>.</p>
<p>Here’s the catch. These extensions aren’t portable and aren’t simply disregarded.</p>
<p>For example, Claude Code supports extra frontmatter fields such as <code>argument-hint</code> and <code>when_to_use</code>. A skill can work perfectly in Claude Code with those fields. But if you move that same <code>SKILL.md</code> to a stricter validation path such as Claude Desktop or the Claude API, those extra fields can cause validation to fail.</p>
<p>Consider this frontmatter:</p>
<pre><code class="language-plaintext">--- 
name: deck-builder 
description: Build presentation outlines. argument-hint: Give me a topic for the deck. 
---
</code></pre>
<p>Claude Code can accept <code>argument-hint</code> as a client-specific extension. A stricter validator can reject the file because <code>argument-hint</code> is not part of the six fields defined by the specification.</p>
<p>That is why the error looks like this:</p>
<pre><code class="language-text">Unexpected key(s) in SKILL.md frontmatter: argument-hint.
Allowed properties are: allowed-tools, compatibility, description, license, metadata, name
</code></pre>
<p>The practical rule is simple: use client-specific extensions when a skill is meant to stay within one client. Otherwise, keep the frontmatter limited to the six fields defined by the specification. That distinction matters because portability isn't just about whether another client can read the file. It's about whether the same skill can be discovered and validated without client-specific fields causing failures.</p>
<p>For <code>deck builder</code>, there's no reason to take that risk. It uses only the six specification fields, which Claude Code loads without modification. That keeps the skill aligned with the portable subset of the format and makes the same <code>SKILL.md</code> easier to move across compatible clients.</p>
<h3 id="heading-the-conventional-folders">The Conventional Folders</h3>
<p>Beyond<code>SKILL.md</code>, the spec names three conventions:</p>
<ul>
<li><p><code>scripts/</code>: code the agent runs. Self-contained, clear errors, no hanging. Ours holds <code>validate_deck.py</code>.</p>
</li>
<li><p><code>references/</code>: docs the agent opens on demand. Keep each file narrow. Ours holds <code>narrative-patterns.md</code>.</p>
</li>
<li><p><code>assets/</code>: templates, schemas, and images. <code>deck-builder</code> doesn't need one.</p>
</li>
</ul>
<p>Different clients may recognise additional directories. For example, Antigravity also documents <code>examples/</code> and <code>resources/</code>. But these directories are conventions, not a fixed requirement of the skill format.</p>
<p>What actually matters is how <code>SKILL.md</code> points to those files. The agent follows the relative paths you define, so you can organise supporting material around the way your skill works. Keep <code>scripts/</code>, <code>references/</code>, <code>assets/</code>, or client-specific directories wherever they make sense, then reference them explicitly from <code>SKILL.md</code>.</p>
<p>The conventions still have an important human benefit. When someone opens a skill they didn't write, familiar directories immediately tell them where to look for executable code, reference material, examples, or other supporting resources</p>
<h3 id="heading-pointing-at-your-own-files">Pointing at Your Own Files</h3>
<p>Always use paths relative to the skill root, never absolute paths.</p>
<pre><code class="language-markdown">See [the narrative patterns guide](references/narrative-patterns.md) when the instructions are unclear.

Check the outline before writing slides:

    python3 scripts/validate_deck.py --file outline.md
</code></pre>
<p>Keep references one level deep. Files pointing at files pointing at more files make the agent burn turns navigating instead of working. And <code>/Users/you/dev/skills/...</code> breaks the moment a teammate clones the repo.</p>
<p><strong>One client-specific refinement:</strong></p>
<p>Claude Code substitutes <code>${CLAUDE_SKILL_DIR}</code>, the folder holding<code>SKILL.md</code>, into both the body and the Bash rules in <code>allowed-tools</code>. Using it in both places lets a skill run its own bundled script without a permission prompt. It's a Claude Code extension, so keep plain relative paths in a portable skill.</p>
<h3 id="heading-validate-before-you-commit">Validate Before You Commit</h3>
<p>The reference library that ships with the standard checks frontmatter and naming:</p>
<pre><code class="language-bash">skills-ref validate ./deck-builder
</code></pre>
<p>Run the validator in CI, so malformed skills are caught before they reach runtime.</p>
<p>Frontmatter mistakes are easy for a machine to detect but much harder to diagnose once the agent is running. The frustrating part is that clients don't always surface these failures in the same way. A strict validator may reject the skill immediately with a clear error, while another client may simply fail to list the skill at all, leaving you with no obvious indication that the frontmatter is the problem.</p>
<h2 id="heading-build-version-1-in-ten-minutes">Build Version 1 in Ten Minutes</h2>
<h3 id="heading-first-pick-the-right-folder">First, Pick the Right Folder</h3>
<p>The spec says what goes <em>inside</em> a skill. It does <strong>not</strong> say where the folder lives, and clients differ. Getting this wrong is the most common reason a first skill never fires, so check before you run <code>mkdir</code>.</p>
<table>
<thead>
<tr>
<th>Client</th>
<th>Project scope</th>
<th>Personal scope</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Claude Code</strong></td>
<td><code>.claude/skills/</code></td>
<td><code>~/.claude/skills/</code></td>
</tr>
<tr>
<td><strong>VS Code / Copilot</strong></td>
<td><code>.github/skills/</code>, <code>.claude/skills/</code>, <code>.agents/skills/</code></td>
<td><code>~/.copilot/skills/</code>, <code>~/.claude/skills/</code>, <code>~/.agents/skills/</code></td>
</tr>
<tr>
<td><strong>Google Antigravity</strong></td>
<td><code>.agents/skills/</code></td>
<td><code>~/.gemini/config/skills/</code></td>
</tr>
</tbody></table>
<p><code>.agents/skills/</code> is the emerging cross-client convention. Claude Code is the big exception: its docs name <code>.claude/skills/</code> and <code>~/.claude/skills/</code>, and do not list <code>.agents/skills/</code> at all.</p>
<p>The overlap is the useful part. <strong>VS Code scans</strong> <code>.claude/skills/</code> <strong>as well</strong>, so one folder satisfies both of the two most typical clients. That's why this guide defaults to it.</p>
<pre><code class="language-bash"># Claude Code, and VS Code / Copilot, which both scan this location
SKILLS_DIR=.claude/skills

# Antigravity, or VS Code if you prefer the cross-client convention
# SKILLS_DIR=.agents/skills

mkdir -p "$SKILLS_DIR/deck-builder"
</code></pre>
<h3 id="heading-then-write-the-file">Then Write the File</h3>
<p>Create <code>deck-builder/SKILL.md</code>:</p>
<pre><code class="language-markdown">---
name: deck-builder
description: Helps make presentations.
---

# Deck Builder

Never start writing slides immediately. Decide what the deck is for first.

## Workflow

### Step 1: Brainstorm

Before any slide exists, write a `## Brainstorm` block answering three questions:

- **Audience.** Who is in the room, how long do you have, what do they
  already know, and what decision do they need to make?
- **Core message.** One sentence. If the audience remembers nothing else,
  what is it?
- **Arc.** How the deck moves from opening to ask.

If the request does not give you enough to answer these, ask the user
before continuing. Do not guess the audience.

### Step 2: Write slides

One idea per slide. Six bullets maximum. End on the ask, never on a
body slide.
</code></pre>
<p>That's a complete, working skill. Twenty-eight lines in one file with no dependencies.</p>
<p>Three things about it generalise to everything you'll build:</p>
<ul>
<li><p><strong>It skips the background:</strong> No explanation of what a slide is. The agent knows.</p>
</li>
<li><p><strong>The rules are checkable:</strong> "Six bullets maximum" can be verified by looking. "Follow presentation best practices" can't.</p>
</li>
<li><p><strong>The description is bad on purpose:</strong> <code>Helps make presentations.</code> is exactly the vague summary that fails to fire. Fixing it is version 2.</p>
</li>
</ul>
<h2 id="heading-run-it-in-vs-code-or-claude-code">Run It in VS Code or Claude Code</h2>
<p>The file exists. Now confirm your client can see it.</p>
<p>This matters more than it might seem. A skill that the client never discovers fails to load into the context, and the symptom can look exactly like a skill with a weak or poorly tuned description.</p>
<p>Before changing the triggering logic, first confirm that the client can actually see the skill. That simple check tells you whether you're debugging discovery or triggering.</p>
<p>Both clients give you two ways to run a skill, and they test different things:</p>
<ul>
<li><p><strong>Invoke it by name:</strong> This skips the description and tests the <strong>body</strong>.</p>
</li>
<li><p><strong>Ask a question that should trigger it</strong>, without naming it. This tests the <strong>description</strong>.</p>
</li>
</ul>
<p>Test both paths. <strong>Explicit invocation</strong> confirms that the skill itself is valid and its instructions are being followed. <strong>Automatic triggering</strong> tests whether the <code>description</code> is specific enough for the agent to recognise when the skill applies.</p>
<p>That distinction makes debugging much easier. If explicit invocation works but automatic triggering fails, the skill body is doing its job and the problem is the description. That's the signal to move to <strong>Version 2</strong> and tune the triggering logic.</p>
<h3 id="heading-in-claude-code">In Claude Code</h3>
<ol>
<li><p>Restart Claude Code so the new folder is discovered. It picks up edits to an existing skills folder live, but a folder that didn't exist at session start needs a restart.</p>
</li>
<li><p>Type <code>/skills</code> and confirm <code>deck-builder</code> is listed.</p>
</li>
<li><p>Invoke it directly with <code>/deck-builder</code> or even if you mention create presentation, it will automatically detect <code>/deck-builder</code> and start executing the workflow based on the skill instructions.</p>
</li>
<li><p>In a fresh session, test triggering. Ask, without naming the skill:</p>
<blockquote>
<p>Can you put together something for Thursday's board meeting about the Q3 migration?</p>
</blockquote>
</li>
</ol>
<p>The command name comes from the <strong>folder</strong> name, not the frontmatter <code>name</code>. Rename the folder and the command changes. Claude Code also merged custom commands into skills, so <code>.claude/commands/deploy.md</code> and <code>.claude/skills/deploy/SKILL.md</code> both produce <code>/deploy</code>.</p>
<h3 id="heading-in-vs-code">In VS Code</h3>
<ol>
<li><p>Open the project with GitHub Copilot installed.</p>
</li>
<li><p>Open the Copilot Chat panel. The Agent Skills quickstart recommends selecting <strong>Agent</strong> mode from the mode dropdown, which is where the agent can run terminal commands. VS Code's own skills docs don't state a mode requirement, so if skills don't appear in another mode, switch to Agent before assuming the file is wrong.</p>
</li>
<li><p>Type <code>/</code> to list available skills and prompts. Skills appear as slash commands alongside prompt files. <code>/skills</code> opens the <strong>Configure Skills</strong> menu, where you can confirm <code>deck-builder</code> was picked up.</p>
</li>
<li><p>Select it from the <code>/</code> list to run it. You can append context, as in <code>/deck-builder for the board meeting</code>.</p>
</li>
<li><p>Ask the same untargeted question.</p>
</li>
</ol>
<p>Because VS Code also scans<code>.claude/skills/</code>, you can reuse the same <code>deck builder</code> directory there without changing its structure. That gives you a convenient shared location when the same skill needs to work across Claude Code and VS Code.</p>
<p>If you prefer to follow VS Code’s own convention, place the skill under <code>.github/skills/</code> instead. The important part isn't choosing a single universal folder. It's placing the skill where the target client actually looks for it and verifying discovery before debugging the skill itself.</p>
<h3 id="heading-what-success-looks-like">What Success Looks Like</h3>
<p>Without the skill, a model typically opens with slide one:</p>
<pre><code class="language-plaintext">Slide 1: Q3 Migration Overview
Slide 2: Goals
Slide 3: Timeline
Slide 4: Challenges
Slide 5: Results
Slide 6: Thank You
</code></pre>
<p>Without the skill, the agent can produce something that looks perfectly reasonable: a clean sequence of slides with familiar sections, but no clear decision about who the deck is actually serving or what it needs to accomplish.</p>
<p>Load the skill, and the behaviour changes before a single slide is written. The agent first creates a <code>Brainstorm</code> block covering the audience, core message, and narrative arc. It may even stop and ask a clarifying question, because a request like “<em>put something together for Thursday’s board meeting</em>” still leaves one critical detail unanswered: <strong>what does the board need to decide?</strong></p>
<p>That pause is the real payoff of <em>Version 1</em>. Instead of rushing into slide generation, the agent takes a moment to establish the thinking that should guide the deck.</p>
<h3 id="heading-when-it-doesnt-work">When it Doesn't Work</h3>
<table>
<thead>
<tr>
<th>Symptom</th>
<th>Likely cause</th>
<th>Fix</th>
</tr>
</thead>
<tbody><tr>
<td>Skill missing from <code>/skills</code></td>
<td>Folder is somewhere this client doesn't scan, or the session predates it</td>
<td>Recheck the path table, then restart</td>
</tr>
<tr>
<td>Listed, but <code>/deck-builder</code> not found</td>
<td>Command comes from the folder name</td>
<td>Rename the folder to <code>deck-builder</code></td>
</tr>
<tr>
<td>Invocation works, triggering does not</td>
<td>Description is too narrow. Expected in version 1</td>
<td>Continue to version 2</td>
</tr>
<tr>
<td>Skill loads but the agent writes slides anyway</td>
<td>Instruction-following varies by model</td>
<td>Try a different model before editing the skill</td>
</tr>
<tr>
<td>Frontmatter rejected as an unexpected key</td>
<td>A client-specific field on a stricter path</td>
<td>Restrict to the spec's six fields</td>
</tr>
</tbody></table>
<p>That fourth row deserves a pause. When output disappoints, the instinct is to rewrite the skill. Change the model first. Otherwise you're debugging two variables at once, and only one of them is your file.</p>
<h2 id="heading-version-2-make-it-trigger-every-time">Version 2: Make It Trigger Every Time</h2>
<p>A skill that never fires is worth nothing. The <code>description</code> carries that entire burden. It's the only part of your skill the agent sees before deciding.</p>
<p>Version 1 says <code>Helps make presentations.</code> That fires on the word "<em>presentation</em>" and stays quiet on "<em>put something together for Thursday,</em>" which is how people actually ask.</p>
<h3 id="heading-four-principles">Four Principles</h3>
<ul>
<li><p><strong>Write it as an instruction:</strong> "Use this skill when…" beats "This skill does…". The agent is making a decision, so address the decision.</p>
</li>
<li><p><strong>Describe intent:</strong> The agent matches what the user asked for, not your architecture.</p>
</li>
<li><p><strong>Be a little pushy:</strong> Name the situations where it applies, including the ones where the user avoids your vocabulary.</p>
</li>
<li><p><strong>Watch the ceiling:</strong> 1024 characters is a hard limit, and descriptions grow during tuning.</p>
</li>
</ul>
<p>One nuance is easy to miss when tuning a skill description: <strong>not every request needs a skill</strong>. Agents tend to reach for skills when the task involves multiple steps, domain-specific judgment, or a procedure that's difficult to reproduce reliably on their own.</p>
<p>A question like “<em>What is a good font size for slides?</em>” usually doesn't need <code>deck builder</code>. The agent can answer it directly. The description becomes valuable when the request requires a repeatable workflow, such as deciding the audience, defining the core message, choosing the right narrative arc, and then building the deck. That's exactly the kind of multi-step work where a well-written skill earns its place.</p>
<h3 id="heading-test-triggering-instead-of-guessing">Test Triggering Instead of Guessing</h3>
<p>You can measure this. Build about 20 realistic prompts labelled with whether they <em>should</em> fire the skill: 8 to 10 positive, 8 to 10 negative. Store them in <code>evals/trigger_queries.json</code>.</p>
<pre><code class="language-json">[
  {
    "query": "can you put something together for thursday's board meeting on the q3 migration",
    "should_trigger": true
  },
  {
    "query": "I need to walk the new hires through how our deploy pipeline works, 20 mins",
    "should_trigger": true
  },
  {
    "query": "make the font bigger on slide 4 of this pptx",
    "should_trigger": false
  },
  {
    "query": "write me a one-page summary of the q3 migration for the wiki",
    "should_trigger": false
  }
]
</code></pre>
<p>The valuable positives are the ones where the skill helps, but the wording doesn't say so. The first example never says "<em>deck</em>," "<em>slides</em>," or "<em>presentation</em>." If a prompt already asks for exactly what the skill does, any description passes, and you learn nothing.</p>
<p>The most valuable negative tests are <strong>near misses</strong>: prompts that contain the same vocabulary as the skill, but actually require a different task. A prompt like “<em>Write a Fibonacci function</em>” tells you almost nothing because there's no overlap with presentation work. “<em>Make the font bigger on slide 4</em>” is much more useful because it clearly mentions a slide, yet the task is editing an existing slide rather than designing a deck.</p>
<p>The wiki summary is another strong boundary case. It may be about the same migration, but the user is asking for a written document, not a presentation. These near misses tell you whether the description understands the <strong>intent of the request</strong>, rather than simply matching familiar words.</p>
<p>Model behaviour varies run to run, so run each query three times and compute a <strong>trigger rate</strong>. Positives pass above 0.5, negatives below.</p>
<h3 id="heading-dont-overfit">Don't Overfit</h3>
<p>Tuning against every query you wrote produces a description that works on those phrasings and fails on real users.</p>
<p>Split the set: about 60% train, 40% validation, proportional positives and negatives in each. Use train failures to guide edits. Use validation only to check that the edits generalise.</p>
<ol>
<li><p>Evaluate on both sets.</p>
</li>
<li><p>Find train failures. Missing triggers mean too narrow. False triggers mean too broad.</p>
</li>
<li><p>Revise toward the <em>general category</em> the failures represent. Never paste keywords from a failed query, which is overfitting.</p>
</li>
<li><p>Repeat, usually no more than five times.</p>
</li>
<li><p>Pick the iteration with the best <strong>validation</strong> rate. It's often not the last one.</p>
</li>
</ol>
<h3 id="heading-the-result">The Result</h3>
<pre><code class="language-yaml"># Version 1 — fires on the word "presentation" and little else
description: Helps make presentations.

# Version 2 — tuned against 20 labelled queries
description: &gt;-
  Turn a request for a presentation into a slide outline. Brainstorm the
  audience, core message, and narrative arc first, then write slides. Use
  when someone asks for a deck, slides, a presentation, a readout, a board
  update, or a talk, including when they only say "put something together
  for Thursday" without naming a format. Not for editing existing slide
  files or for writing prose documents.
</code></pre>
<p>Two changes did the work. It became more specific about what it does by naming the brainstorm-then-build procedure, and broader about when it applies by covering requests that never say "<em>deck</em>."</p>
<p>That closing sentence is the part most people skip. Saying what a skill is <em>not</em> for is how you stop near-misses from false-triggering.</p>
<h2 id="heading-version-3-write-a-body-that-earns-its-tokens">Version 3: Write a Body That Earns Its Tokens</h2>
<p>Once a skill activates, its whole body competes for attention with the conversation, the system context, and every other active skill. Treat it as a budget.</p>
<h3 id="heading-start-from-real-expertise">Start from Real Expertise</h3>
<p>The most common failure in skill writing is asking an LLM to write a skill with no domain input. What comes back is fluent and useless: "<em>consider your audience</em>," "<em>keep slides clear</em>."</p>
<p>Effective skills come from things that already exist. Take the deck your team actually praised and write down why it worked. Take the review comments where someone said: "<em>this is three decks, not one</em>." Take the readout that got cut short because slide two lost the room.</p>
<p>For <code>deck-builder</code>, the raw material is the feedback you already give people on drafts.</p>
<h3 id="heading-cut-what-the-agent-already-knows">Cut What the Agent Already Knows</h3>
<p>Ask of every line: <em>would the agent get this wrong without it?</em> If not, cut.</p>
<pre><code class="language-markdown">&lt;!-- Too verbose — the agent knows what a slide is --&gt;
## Slide design

A slide is a single screen in a presentation. Slides should be visually
clear and not too crowded. Audiences find it hard to read a lot of text
on a screen, so you should use bullet points to summarize your ideas.

&lt;!-- Better — starts where the agent goes wrong --&gt;
## Slide design

One idea per slide. Six bullets maximum, each under 120 characters.
Anything longer belongs in speaker notes, not on the slide.
</code></pre>
<h3 id="heading-scope-it-as-one-coherent-job">Scope it as One Coherent Job</h3>
<p>Scope too narrow and one task drags in four skills. Scope too broad and the description can't fire precisely.</p>
<p>Brainstorming an outline and writing the slides is one job, because the second depends on the first. Adding chart design, speaker coaching, and editing existing <code>.pptx</code> files would be four jobs wearing one hat. That's exactly why our tuned description disclaims them.</p>
<h3 id="heading-match-how-firmly-you-instruct-to-how-fragile-the-task-is">Match How Firmly You Instruct to How Fragile the Task is</h3>
<p><strong>Leave room</strong> where several approaches work. Explaining <em>why</em> beats rigid commands, because an agent that understands the purpose adapts well.</p>
<pre><code class="language-markdown">## Choosing an arc

Pick the arc that matches what the audience needs to do:

- They must decide something: Situation, Complication, Resolution
- They are skeptical: lead with the objection, then dismantle it
- They need to learn: chronological, simplest case first
- They already agreed: skip persuasion, go straight to the plan
</code></pre>
<p><strong>Be prescriptive</strong> where order matters:</p>
<pre><code class="language-markdown">## Order of operations

Do these in order. Do not write slides before the brainstorm exists.

1. Write the `## Brainstorm` block
2. Run `python3 scripts/validate_deck.py --file outline.md`
3. Write slides only after it exits 0
</code></pre>
<p>Most skills need both. Calibrate section by section.</p>
<h3 id="heading-give-default-options">Give Default Options</h3>
<p>Listing five options invites deliberation:</p>
<pre><code class="language-markdown">&lt;!-- Too many options --&gt;
You could use SCR, PAS, AIDA, the pyramid principle, the hero's journey...

&lt;!-- One default, one escape hatch --&gt;
Default to Situation, Complication, Resolution. It fits most internal
readouts. For a skeptical audience, lead with the objection instead.
</code></pre>
<h3 id="heading-four-patterns-that-do-most-of-the-work">Four Patterns That Do Most of the Work</h3>
<h4 id="heading-false-positives">False Positives</h4>
<p>The highest-value section in most skills. Not advice, but corrections to mistakes the agent will otherwise make:</p>
<pre><code class="language-markdown">## False Positives

- "Put something together" is not a brief. It gives you no audience and
  no decision. Ask before you build; do not invent an audience.
- A deck for a 15-minute slot is not a shorter version of the 45-minute
  deck. Fewer slides with the same message, not the same slides compressed.
- If the core message needs an "and" to state it, it is two decks. Split
  it or pick one.
- Our leadership readouts open with the ask, not the background. Reverse
  the arc for anyone above director level.
</code></pre>
<p>That last one is the kind of thing no model can guess. It's your presentation styling and designing conventions, and it's the single most valuable line in the file.</p>
<p>Keep false positives in <code>SKILL.md</code>, not a reference file. The agent needs them <em>before</em> it hits the situation, and it can't know to open a file describing a trap it doesn't know exists.</p>
<p>Every time you correct an agent mid-task, that correction belongs here.</p>
<h4 id="heading-templates">Templates</h4>
<p>When output shape matters, show the shape. Agents pattern-match against structure far better than they follow prose about structure.</p>
<pre><code class="language-markdown">## Outline format

Produce exactly this structure:

```markdown
# Deck: &lt;title&gt;

## Brainstorm
- Audience: &lt;who, how long, what they know, what they must decide&gt;
- Core message: &lt;one sentence&gt;
- Arc: &lt;which pattern and why&gt;

## Slides

### &lt;slide title, under 60 characters&gt;
- &lt;bullet, under 120 characters&gt;
```
</code></pre>
<h4 id="heading-checklists">Checklists</h4>
<p>Explicit progress lists stop steps from being skipped:</p>
<pre><code class="language-markdown">## Progress

- [ ] 1. Brainstorm block written
- [ ] 2. Missing audience info asked about, not invented
- [ ] 3. `validate_deck.py` exits 0
- [ ] 4. Slides written
- [ ] 5. Validator re-run on the finished outline
</code></pre>
<h4 id="heading-validation-loops">Validation Loops</h4>
<p>Tell the agent to check its own work and iterate. This turns one-shot generation into a self-correcting process:</p>
<pre><code class="language-markdown">## Validation loop

1. Write or edit the outline.
2. Run `python3 scripts/validate_deck.py --file outline.md`.
3. If it exits 1, read the rule ID and message, fix it, run again.
4. Only continue when it exits 0.
</code></pre>
<h3 id="heading-when-the-body-legitimately-needs-more">When the Body Legitimately Needs More</h3>
<p>Our body is now near the useful limit. The full catalogue of narrative arcs, with worked examples, would push it well past 500 lines.</p>
<p>That goes to <code>references/</code>, which is <em>Version 5</em>. The rule: tell the agent <em>when</em> to open each file. "Read <code>references/narrative-patterns.md</code> when the audience is skeptical or the arc is unclear" is actionable. "See references/ for details" is not.</p>
<h2 id="heading-version-4-bundle-a-script-that-checks-the-work">Version 4: Bundle a Script That Checks the Work</h2>
<p>Prose can make an agent <strong>more likely to follow a procedure</strong>, but it can't prove that the procedure was actually followed. A script gives you that missing verification layer.</p>
<p>Our skill already tells the agent to validate the outline. Now we're giving that instruction a concrete mechanism to validate against. Instead of trusting the agent to remember every rule, <code>validate_deck.py</code> checks the outline and returns a clear pass or failure signal. That turns “<em>please check your work</em>” from a suggestion into something measurable and repeatable</p>
<h3 id="heading-declare-dependencies-inline">Declare Dependencies Inline</h3>
<p>A bundled script should declare its own dependencies so the agent runs it in one command with no install step. In Python, <a href="https://peps.python.org/pep-0723/">PEP 723</a> does this:</p>
<pre><code class="language-python"># /// script
# requires-python = "&gt;=3.9"
# dependencies = []
# ///
</code></pre>
<p><code>uv run scripts/validate_deck.py</code> would then build an isolated environment and execute. Ours uses only the standard library, so the list is empty and plain <code>python3</code> works.</p>
<p>That's a design choice worth copying. A zero-dependency validator still works in a locked-down CI container in three years.</p>
<h3 id="heading-design-it-for-an-agent">Design it For an Agent</h3>
<p>The agent reads stdout and stderr to decide what to do next. Six choices decide whether that goes well:</p>
<ul>
<li><p><strong>Never prompt interactively:</strong> Hard requirement. Agents run in non-interactive shells and can't answer a TTY prompt. A script that blocks on input hangs until killed.</p>
</li>
<li><p><strong>Document through</strong> <code>--help</code><strong>:</strong> That output is how the agent learns your interface. Purpose, flags, exit codes. Keep it short. It lands in the context window.</p>
</li>
<li><p><strong>Write errors that suggest the fix:</strong> <code>Error: invalid input</code> costs a turn. Naming the rule and the remedy costs nothing.</p>
</li>
<li><p><strong>Emit structured output:</strong> JSON on stdout, diagnostics on stderr.</p>
</li>
<li><p><strong>Be idempotent:</strong> Agents retry. A static checker is naturally safe to re-run.</p>
</li>
<li><p><strong>Bound your output:</strong> Many harnesses truncate tool output past roughly 10–30K characters, silently dropping the part that mattered. Report findings, not the whole file.</p>
</li>
</ul>
<h3 id="heading-the-validator">The Validator</h3>
<p>Create <code>deck-builder/scripts/validate_deck.py</code>:</p>
<pre><code class="language-python">#!/usr/bin/env python3
"""Static checker for deck outlines produced by the deck-builder skill.

Checks that an outline brainstormed before it built, and that no slide is
overloaded. Reads the outline file only; nothing is rendered or uploaded.

Usage:
  scripts/validate_deck.py --file outline.md
  scripts/validate_deck.py --file outline.md --format json

Exit codes:
  0  Outline passes every check.
  1  One or more problems found.
  2  The file could not be read.
"""

import argparse
import json
import re
import sys
from typing import Dict, List

MAX_BULLETS = 6
MAX_BULLET_CHARS = 120
MAX_TITLE_CHARS = 60
CLOSING_WORDS = ("next step", "call to action", "recap", "takeaway", "ask")


def parse(outline: str) -&gt; List[Dict]:
    """Split the outline into slides. A slide starts at a '### ' heading."""
    slides, current = [], None
    for lineno, line in enumerate(outline.split("\n"), 1):
        heading = re.match(r"^###\s+(.*\S)\s*$", line)
        if heading:
            current = {"title": heading.group(1), "line": lineno, "bullets": []}
            slides.append(current)
            continue
        bullet = re.match(r"^\s*[-*]\s+(.*\S)\s*$", line)
        if bullet and current is not None:
            current["bullets"].append({"text": bullet.group(1), "line": lineno})
    return slides


def analyze(path: str) -&gt; List[Dict]:
    with open(path, "r", encoding="utf-8") as handle:
        outline = handle.read()

    findings: List[Dict] = []

    def add(rule, line, message, snippet=""):
        findings.append(
            {"rule": rule, "line": line, "message": message, "snippet": snippet}
        )

    if not re.search(r"^##\s+Brainstorm\s*$", outline, re.M | re.I):
        add(
            "DECK001", 1,
            "No '## Brainstorm' section. The skill must think before it builds: "
            "record audience, core message, and arc before writing slides.",
        )

    slides = parse(outline)
    if not slides:
        add("DECK006", 1, "No slides found. Each slide is a '### ' heading.")

    for slide in slides:
        if len(slide["title"]) &gt; MAX_TITLE_CHARS:
            add(
                "DECK004", slide["line"],
                f"Slide title is {len(slide['title'])} characters; keep it under "
                f"{MAX_TITLE_CHARS} so it fits one line at presentation size.",
                slide["title"][:70],
            )
        if len(slide["bullets"]) &gt; MAX_BULLETS:
            add(
                "DECK002", slide["line"],
                f"Slide has {len(slide['bullets'])} bullets; split it. More than "
                f"{MAX_BULLETS} reads as a document, not a slide.",
                slide["title"][:70],
            )
        for bullet in slide["bullets"]:
            if len(bullet["text"]) &gt; MAX_BULLET_CHARS:
                add(
                    "DECK003", bullet["line"],
                    f"Bullet is {len(bullet['text'])} characters. Tighten it to "
                    f"under {MAX_BULLET_CHARS} or move it to speaker notes.",
                    bullet["text"][:70],
                )

    if slides:
        tail = " ".join(
            [slides[-1]["title"]] + [b["text"] for b in slides[-1]["bullets"]]
        ).lower()
        if not any(word in tail for word in CLOSING_WORDS):
            add(
                "DECK005", slides[-1]["line"],
                "Last slide has no recap, takeaway, or next step. End on the ask, "
                "not on the final body slide.",
                slides[-1]["title"][:70],
            )

    return sorted(findings, key=lambda f: (f["line"], f["rule"]))


def main() -&gt; int:
    parser = argparse.ArgumentParser(
        description="Check a deck outline for structure and slide density.",
        epilog="Exit codes: 0 clean, 1 findings, 2 unreadable file.",
    )
    parser.add_argument("--file", required=True, help="Path to the outline")
    parser.add_argument(
        "--format", choices=["text", "json"], default="text",
        help="Output format (default: text)",
    )
    args = parser.parse_args()

    try:
        findings = analyze(args.file)
    except OSError as exc:
        print(f"Error: could not read {args.file}: {exc}", file=sys.stderr)
        return 2

    if args.format == "json":
        json.dump({"file": args.file, "findings": findings}, sys.stdout, indent=2)
        sys.stdout.write("\n")
    elif findings:
        for f in findings:
            print(f"{args.file}:{f['line']}: [{f['rule']}] {f['message']}")
            if f["snippet"]:
                print(f"    {f['snippet']}")
    else:
        print(f"{args.file}: outline passes all checks.")

    return 1 if findings else 0


if __name__ == "__main__":
    sys.exit(main())
</code></pre>
<pre><code class="language-bash">chmod +x "$SKILLS_DIR/deck-builder/scripts/validate_deck.py"
</code></pre>
<p>Three details make this validator <strong>agent-friendly</strong>, not merely technically correct.</p>
<p>First, every error should tell the agent how to fix it. Reporting that a rule was violated is only half the job. A message such as “<em>slide exceeds six bullets; split it into smaller ideas</em>” gives the agent enough information to correct the outline and try again, without sending the work back to a human.</p>
<p>Second, enforce the skill’s core workflow mechanically. Version 1 told the agent to brainstorm before writing slides, but prose turned that requirement into a suggestion.</p>
<p>The validator changes that. If the brainstorm section is missing, the script returns a non-zero exit code, giving the agent an objective signal that it must fix the workflow before continuing.</p>
<p>Third, make a clear distinction between conventions and hard requirements. Six bullets per slide and 120 characters per bullet are not universal laws of presentation design. They're this team’s conventions. Treating them as universal rules can make the agent push back when a user has a legitimate reason to break them.</p>
<p>Be explicit about which rules are mandatory and which rules simply reflect your team’s preferred way of working.</p>
<h3 id="heading-wire-it-in">Wire it in</h3>
<p>Add to <code>SKILL.md</code>:</p>
<pre><code class="language-markdown">## Available scripts

- **`scripts/validate_deck.py`** — checks one outline. Exits 0 clean,
  1 on findings, 2 if the file cannot be read.

Run it after the brainstorm and again after writing slides:

```bash
python3 scripts/validate_deck.py --file outline.md
```
</code></pre>
<p>The loop is now closed. The agent drafts, the script judges, the agent fixes, and the cycle ends on exit code 0.</p>
<h2 id="heading-version-5-move-the-deep-material-out-of-the-way">Version 5 Move the Deep Material Out of the Way</h2>
<p>Our <code>SKILL.md</code> covers the workflow, the format, the false positives, and the validator. What it doesn't cover is the narrative theory, and it shouldn't. That's needed on maybe one deck in five, and paying for it on every activation is the exact waste progressive disclosure exists to prevent.</p>
<p>Create <code>deck-builder/references/narrative-patterns.md</code>:</p>
<pre><code class="language-markdown"># Narrative Patterns

Pick the arc from what the audience must do, not from what feels natural
to write.

## Situation, Complication, Resolution

The default for internal readouts. Works when the audience needs to
approve or fund something.

- **Situation.** What everyone already agrees is true. Keep it short.
- **Complication.** What changed, or what broke. This is the slide that
  earns attention.
- **Resolution.** What you did or propose, and the ask.

Failure mode: spending four slides on Situation. If the audience lived
through it, one slide is enough.

## Objection first

For a skeptical audience, or a proposal that was rejected before.

Open with the strongest argument against you, stated fairly. Then
dismantle it. An audience that hears its own objection spoken aloud
stops rehearsing it and starts listening.

## Chronological

For teaching, onboarding, and post-incident reviews. Simplest case
first, then complications in the order they were discovered.

Failure mode: chronological order is rarely the persuasive order. Do
not reach for it just because it matches how the work happened.

## Ask first

For leadership above director level, and for any slot under 10 minutes.

State the decision on slide one. Then support it. If they say yes on
slide one, you have saved everyone twenty minutes, and the rest of the
deck becomes optional backup.

## Choosing quickly

| Audience state | Arc |
| :--- | :--- |
| Needs to decide | Situation, Complication, Resolution |
| Doubts you | Objection first |
| Needs to learn | Chronological |
| Already agrees | Ask first |
| Very senior, short slot | Ask first |
</code></pre>
<p>Now add the conditional pointer to <code>SKILL.md</code>:</p>
<pre><code class="language-markdown">Read [references/narrative-patterns.md](references/narrative-patterns.md)
when the audience is skeptical, when the slot is under 10 minutes, when
the audience is above director level, or when Situation-Complication-
Resolution does not obviously fit.
</code></pre>
<p>That sentence is the whole trick. Four named conditions, each one something the agent can recognise when it happens.</p>
<p>Compare it to "see references/ for more information," which gives no signal about when the file becomes relevant and gets ignored.</p>
<p>The skill is now complete:</p>
<img src="https://cdn.hashnode.com/uploads/covers/69e9e11d16b084c068eb65a1/0a844b3c-8a22-4bc9-a8fc-bee84c3b8ea8.png" alt="Diagram of a deck-builder/ skill directory showing five components: SKILL.md containing instructions and workflow, scripts/validate_deck.py containing validation logic, references/narrative-patterns.md containing reference guidance, evals/trigger_queries.json containing trigger tests, and evals/evals.json containing output-quality tests." style="display: block;" width="1064" height="292" loading="lazy">

<h2 id="heading-where-skills-live-and-how-agents-find-them">Where Skills Live and How Agents Find Them</h2>
<p>Version 1 gave you the one path you needed. This is the full picture, because discovery is where the open standard stops and per-client behaviour starts.</p>
<p>Clients generally scan a project scope and a user scope. Which folders fill those scopes is where they diverge:</p>
<table>
<thead>
<tr>
<th>Client</th>
<th>Project scope</th>
<th>User scope</th>
<th>Also scans</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Claude Code</strong></td>
<td><code>.claude/skills/</code></td>
<td><code>~/.claude/skills/</code></td>
<td>Nested <code>.claude/skills/</code> below the working directory. <code>.claude/skills/</code> inside any <code>--add-dir</code> directory. Plugin skills under <code>&lt;plugin&gt;/skills/</code>. An enterprise directory via managed settings.</td>
</tr>
<tr>
<td><strong>VS Code / Copilot</strong></td>
<td><code>.github/skills/</code>, <code>.claude/skills/</code>, <code>.agents/skills/</code></td>
<td><code>~/.copilot/skills/</code>, <code>~/.claude/skills/</code>, <code>~/.agents/skills/</code></td>
<td>Any folder added through <code>chat.agentSkillsLocations</code></td>
</tr>
<tr>
<td><strong>Google Antigravity</strong></td>
<td><code>.agents/skills/</code></td>
<td><code>~/.gemini/config/skills/</code></td>
<td>Legacy <code>.agent/skills/</code></td>
</tr>
</tbody></table>
<p>Two asymmetries decide where to put a skill.</p>
<p>First, Claude Code doesn't scan <code>.agents/skills/</code>. A skill placed there never appears. If your first skill isn't triggering and you're on Claude Code, check this before rewriting the description.</p>
<p>Second, VS Code scans <code>.claude/skills/</code> as well as its own conventions. Combined with the point above, <code>.github/skills/</code> is the one folder both major clients read. Reach for <code>.agents/skills/</code> when your audience is on Antigravity.</p>
<p>Some clients also walk parent folders up to the git root, so a monorepo subproject inherits skills defined at the root.</p>
<h3 id="heading-project-or-personal">Project or Personal?</h3>
<p>For <code>deck-builder</code> the choice is clear. That false positives about leadership readouts opening with the ask is a decision about <em>this</em> organisation. It belongs in the project, committed so everyone gets the same rules without setup.</p>
<p>A skill encoding personal taste rather than team policy belongs in the personal scope.</p>
<h3 id="heading-name-collisions-never-assume-which-skill-wins">Name Collisions: Never Assume Which Skill Wins</h3>
<p>Two skills can have the same name, and that is where things get subtle. Different clients use different precedence rules to decide which skill takes priority. If you assume the order is the same everywhere, you may invoke a different skill than the one you intended without getting an obvious error.</p>
<p>That makes name collisions more than a naming issue. They're a <strong>behavioural risk</strong>: the command may work exactly as expected while executing the wrong skill.</p>
<p>The Agent Skills client-implementation guidance describes the common convention as <em>project overrides user</em>, reasoning that a runbook in version control represents a team decision.</p>
<p>Claude Code documents the opposite:</p>
<blockquote>
<p><strong>Enterprise overrides personal, and personal overrides project.</strong></p>
</blockquote>
<p>So imagine you have a <code>deck builder</code> skill in <code>~/.claude/skills/</code>, and then you clone a repository that contains another skill with the same name. In Claude Code, the personal skill wins, so the version you already trust continues to run instead of being silently replaced by the repository copy.</p>
<p>That behaviour is useful, but it's not universal. The broader Agent Skills convention treats project skills as higher priority than user skills, while Claude Code uses the opposite precedence. This is exactly the kind of difference that can make a portable skill behave unexpectedly.</p>
<p>The safest approach is to <strong>never assume which skill wins</strong>. Check the precedence rules documented by the client you're using. Claude Code also provides explicit namespacing for plugin skills, such as <code>plugin-name:skill-name</code>, and can distinguish nested monorepo skills with paths such as <code>apps/web:deploy</code>.</p>
<h3 id="heading-a-trust-boundary-worth-naming">A Trust Boundary Worth Naming</h3>
<p>Precedence has a sharp edge either direction. Project skills come from the repo you're working in, possibly one you cloned five minutes ago and haven't read. Loading them means loading instructions written by whoever wrote that repo, straight into your agent's context.</p>
<p>Claude Code adds an important trust boundary around project skills. A skill from <code>.claude/skills/</code> must pass the workspace trust check before the client discovers it. But that protection stops at discovery. <strong>Trust determines whether the skill becomes available, not what an already invoked skill is allowed to do.</strong></p>
<p>That distinction matters because a trusted or explicitly invoked skill can still contain instructions, scripts, or tool permissions that affect the agent’s behaviour. In other words, passing the trust dialogue shouldn't be treated as a security review of the skill itself. The security section looks at that second layer.</p>
<h2 id="heading-skills-vs-rules-vs-mcp-vs-hooks-vs-plugins">Skills vs Rules vs MCP vs Hooks vs Plugins</h2>
<p>Skills are one primitive among several. Picking correctly between them is most of the architecture work.</p>
<table>
<thead>
<tr>
<th>Primitive</th>
<th>Purpose</th>
<th>Loading</th>
<th>Where <code>deck-builder</code> fits</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Rules</strong> (<code>AGENTS.md</code>)</td>
<td>Always-on constraints and standards</td>
<td>Always-on or path-matched</td>
<td>"All readouts live in <code>docs/decks/</code>" is a rule. <em>How</em> to build one is not.</td>
</tr>
<tr>
<td><strong>Skills</strong> (<code>SKILL.md</code>)</td>
<td>Domain procedures and runbooks</td>
<td>Progressive disclosure</td>
<td>Our entire skill</td>
</tr>
<tr>
<td><strong>MCP servers</strong></td>
<td>Connections to live external tools</td>
<td>Active process</td>
<td>A server that renders the outline into Google Slides</td>
</tr>
<tr>
<td><strong>Hooks</strong></td>
<td>Shell commands on lifecycle events</td>
<td>Event-triggered</td>
<td>Run <code>validate_deck.py</code> automatically on every write to <code>docs/decks/</code></td>
</tr>
<tr>
<td><strong>Plugins</strong></td>
<td>Bundles of the above</td>
<td>Ingested on discovery</td>
<td>Ship the skill, the rule, and the hook as one package</td>
</tr>
</tbody></table>
<p>The confusing pair is Skills versus MCP. Our example separates them cleanly:</p>
<ul>
<li><p><strong>MCP is capability:</strong> It gives the agent a hand: create this file, call this API, render these slides.</p>
</li>
<li><p><strong>Skills are judgment:</strong> They supply the strategy, the ordering, and the checks that decide <em>how</em> and <em>when</em> that hand should move.</p>
</li>
</ul>
<p><code>deck builder</code> defines the judgment layer of the workflow. It decides that the agent should brainstorm first, identify the audience and core message, and keep each slide within the six bullet limit.</p>
<p>An MCP server provides a different layer of capability: it gives the agent the tools needed to turn that finished outline into an actual slide deck.</p>
<p>These two pieces work together, but they solve different problems. The skill decides <strong>how the work should be done</strong>, while MCP provides the capabilities needed to execute that work. And the skill remains useful even without MCP, because a well-structured outline still has value even when nothing renders it into slides</p>
<img src="https://cdn.hashnode.com/uploads/covers/69e9e11d16b084c068eb65a1/4d2fe7a0-5589-4dc3-9e2d-119793586059.png" alt="Diagram showing how a deck-builder skill responds to a natural-language request without requiring a “deck” keyword. The skill recognizes the user's intent, checks that its description matches the task, thinks through the task, asks about the audience, reads narrative-pattern guidance, builds and writes the slides, and validates them. It then calls slides-provider.create_presentation(), producing a board deck that opens with brainstorming." style="display: block;" width="1058" height="424" loading="lazy">

<h3 id="heading-hooks-and-plugins-are-client-specific">Hooks and Plugins Are Client-Specific</h3>
<p>Rules, MCP, and Skills are broadly portable. <strong>Hooks and Plugins are not.</strong> The manifests live in different places, and the event sets differ in size:</p>
<table>
<thead>
<tr>
<th></th>
<th>Google Antigravity</th>
<th>Claude Code</th>
</tr>
</thead>
<tbody><tr>
<td>Plugin manifest</td>
<td><code>plugin.json</code> at plugin root</td>
<td><code>.claude-plugin/plugin.json</code></td>
</tr>
<tr>
<td>Bundled skills</td>
<td><code>skills/&lt;name&gt;/SKILL.md</code></td>
<td><code>skills/&lt;name&gt;/SKILL.md</code></td>
</tr>
<tr>
<td>Hook config</td>
<td><code>hooks.json</code> in <code>.agents/</code> or <code>~/.gemini/config/</code></td>
<td><code>hooks/hooks.json</code> in plugin root, or inline in <code>plugin.json</code></td>
</tr>
<tr>
<td>Lifecycle events</td>
<td>5: <code>PreToolUse</code>, <code>PostToolUse</code>, <code>PreInvocation</code>, <code>PostInvocation</code>, <code>Stop</code></td>
<td>13+, including <code>SessionStart</code>, <code>UserPromptSubmit</code>, <code>PreToolUse</code>, <code>PostToolUse</code>, <code>PostToolUseFailure</code>, <code>PermissionRequest</code>, <code>PermissionDenied</code>, <code>Notification</code>, <code>SubagentStop</code>, <code>Stop</code>, <code>StopFailure</code>, <code>PreCompact</code>, <code>SessionEnd</code></td>
</tr>
</tbody></table>
<p>A bundle built for one won't load in the other. One convergence is worth knowing: in Claude Code, dropping a <code>.claude-plugin/plugin.json</code> into a folder inside a skills directory promotes it to a plugin named <code>&lt;name&gt;@skills-dir</code>, letting a skill grow into a bundle with no install step.</p>
<p>The examples below are Antigravity's, documented at <a href="https://antigravity.google/docs/plugins">antigravity.google/docs</a>. Check your client before copying.</p>
<pre><code class="language-json">{
  "$schema": "https://antigravity.google/schemas/v1/plugin.json",
  "name": "presentation-suite",
  "description": "Deck brainstorming and outline validation for the platform team."
}
</code></pre>
<p>A hook that runs the validator automatically, so the agent can't forget step 2:</p>
<pre><code class="language-json">{
  "outline-validator": {
    "PostToolUse": [
      {
        "matcher": "run_command",
        "hooks": [
          {
            "type": "command",
            "command": "./scripts/validate-changed-outlines.sh",
            "timeout": 10
          }
        ]
      }
    ]
  }
}
</code></pre>
<p><code>PreToolUse</code> is the interesting one for safety. Hooks receive JSON on stdin and return JSON on stdout, and a <code>PreToolUse</code> response requires a <code>decision</code> field that gates the call: <code>allow</code> proceeds, <code>deny</code> blocks, <code>ask</code> prompts while respecting "always allow" settings, and <code>force_ask</code> prompts regardless. That field is the mechanism behind automated guardrails.</p>
<h2 id="heading-prove-the-skill-actually-helps">Prove the Skill Actually Helps</h2>
<p>You built a skill. It produced a better outline once. That alone isn't evidence that the skill is actually improving the agent. Model outputs vary from run to run, and what looks like an improvement in one run may simply be noise.</p>
<p>This is also a different measurement from Version 2. There, the question was <strong>“Does the skill fire when it should?”</strong> Here, the question is <strong>“Once it fires, does it actually improve the result?”</strong> A skill can trigger perfectly and still add no value. It can also improve the output when invoked explicitly while failing to trigger reliably. You need to measure both.</p>
<h3 id="heading-write-test-cases">Write Test Cases</h3>
<p>Each case has a realistic prompt, a description of success, and optionally input files. Store them in <code>evals/evals.json</code>:</p>
<pre><code class="language-json">{
  "skill_name": "deck-builder",
  "evals": [
    {
      "id": 1,
      "prompt": "can you put something together for thursday's board meeting on the q3 migration",
      "expected_output": "A brainstorm block first, and a clarifying question about what the board needs to decide, before any slides exist.",
      "assertions": [
        "A '## Brainstorm' section appears before any slide",
        "The brainstorm names an audience, a core message, and an arc",
        "The agent asks what decision the board needs to make, rather than inventing one",
        "The core message is one sentence with no 'and' joining two claims"
      ]
    },
    {
      "id": 2,
      "prompt": "20 min onboarding walkthrough of our deploy pipeline for new hires, here are my notes",
      "expected_output": "A chronological outline. No slide over 6 bullets. Ends on next steps.",
      "assertions": [
        "No slide has more than 6 bullets",
        "No bullet exceeds 120 characters",
        "The final slide contains a recap or next step",
        "validate_deck.py exits 0 on the produced outline"
      ]
    },
    {
      "id": 3,
      "prompt": "I already know the audience and the message, just give me the 5 slides for the migration retro",
      "expected_output": "Slides, without re-interrogating the user. The skill should not force a brainstorm the user already did.",
      "assertions": [
        "The agent does not ask questions the prompt already answered",
        "The outline still records the supplied audience and message in the brainstorm block",
        "The agent produces slides in this turn rather than stopping to plan"
      ]
    }
  ]
}
</code></pre>
<p>Start with two or three test cases. Vary the wording, tone, and level of detail so you're not testing the skill against one narrow style of request. Include at least one boundary case that looks relevant on the surface but shouldn't trigger the skill. Keep the prompts grounded in realistic situations you would actually expect users to bring to the agent.</p>
<p>Case 3 deserves special attention because it's a <strong>negative capability test</strong>. It checks that the skill doesn't make the agent worse by applying its workflow when it's no longer necessary.</p>
<p>For example, our skill requires a brainstorm before creating slides, but a user may have already provided the audience and core message. Forcing that user through the same discovery process adds friction instead of value. A good evaluation should catch that behaviour, not just reward the skill for following its own rules.</p>
<h3 id="heading-compare-against-no-skill">Compare Against No Skill</h3>
<p>Run every case twice: <strong>with the skill</strong> and <strong>without it</strong>. The baseline is the entire point. A skill is only worth its context if it beats the bare model, and plenty do not.</p>
<p>Give each run a clean context so it follows only <code>SKILL.md</code>. In clients with subagents, each child task starts fresh, otherwise use a separate session. When improving an existing skill, snapshot the old version and use that as baseline. Version 1 makes a natural baseline for version 5.</p>
<h3 id="heading-write-assertions-and-grade-them">Write Assertions and Grade Them</h3>
<p>Write assertions after you've seen the first few outputs. Before the skill runs, it's often difficult to define what “good” looks like in a way that's both useful and testable. Early outputs expose the behaviours your evaluation actually needs to measure.</p>
<p>Good assertions are <strong>specific, observable, and checkable</strong>. For example, “a <code>Brainstorm</code> section appears before any slide” gives you a concrete condition to verify. Likewise, requiring the exact phrase <code>core message</code> is too brittle because the agent could satisfy the wording without actually doing the required thinking.</p>
<p>Grade every assertion as <strong>PASS or FAIL</strong>, and require evidence from the actual output. A brainstorm that says <code>Audience: everyone</code> should still fail if it provides no meaningful audience analysis. The presence of the label isn't proof that the required reasoning happened.</p>
<p>Use scripts wherever the condition can be checked mechanically. <code>validate_deck.py</code> can handle structural checks consistently, leaving human review for qualities that are harder to reduce to rules, such as whether the narrative is persuasive or whether the core message actually lands with the audience.</p>
<h3 id="heading-read-the-numbers">Read the Numbers</h3>
<pre><code class="language-json">{
  "run_summary": {
    "with_skill":    { "pass_rate": 0.85, "time_seconds": 38.0, "tokens": 3900 },
    "without_skill": { "pass_rate": 0.29, "time_seconds": 26.0, "tokens": 2200 },
    "delta":         { "pass_rate": 0.56, "time_seconds": 12.0, "tokens": 1700 }
  }
}
</code></pre>
<p>The delta states the trade honestly: 12 seconds and 1,700 tokens buy 56 points of pass rate. That's clearly worth it. A skill that doubles tokens for two points is not, and the benchmark tells you which one you built.</p>
<p>Then read past the averages:</p>
<ul>
<li><p><strong>Assertions passing in both</strong> measure the model, not your skill. Remove them.</p>
</li>
<li><p><strong>Assertions failing in both</strong> are usually broken assertions or impossible cases. Fix them.</p>
</li>
<li><p><strong>Assertions passing only with the skill</strong> are where the value is. Here it's the brainstorm block, which baselines rarely produce.</p>
</li>
<li><p><strong>High variance across runs</strong> means ambiguous instructions, not a flaky test.</p>
</li>
</ul>
<h3 id="heading-close-the-loop">Close the Loop</h3>
<p>Revision should be driven by three signals: failed assertions, human feedback, and execution transcripts. Of the three, transcripts often reveal the most because they show <strong>where the agent went wrong and what led it there</strong>.</p>
<p>Suppose the agent tries three different approaches before producing the outline. That usually points to an instruction that leaves too much room for interpretation. Or suppose it opens <code>narrative</code> <code>patterns.md</code> for a simple standup deck. That's a sign that the reference condition is too broad. The transcript exposes the reasoning path behind the failure, not just the failure itself.</p>
<p>Use those signals to improve the underlying pattern rather than patching one failing example at a time. Also be willing to remove instructions. If the pass rate stops improving while the skill keeps getting larger, the skill may be carrying too many constraints. And when the agent repeatedly recreates the same helper logic during execution, that's a strong signal that the logic belongs in <code>scripts/</code> instead. That's precisely how the validator earned its place in <code>deck builder</code>.</p>
<h2 id="heading-scan-before-you-install">Scan Before You Install</h2>
<p>Everything so far assumes you wrote the skill. Increasingly, you did not.</p>
<p>Skills spread the way npm packages did: copied from marketplaces, cloned from GitHub, or pasted from a colleague. A skill isn't inert data. It's instructions that enter your agent's context and scripts that execute on your machine, usually with whatever credentials your shell already has.</p>
<p>Picture finding <code>deck-builder-pro</code> on a marketplace. Everything ours does, plus chart generation and brand templates. It has stars. Would you read all four files before dropping it into a repo your agent can write to?</p>
<h3 id="heading-the-frontmatter-is-part-of-the-attack-surface">The Frontmatter is Part of the Attack Surface</h3>
<p>Before the scripts, look at the frontmatter. A skill can grant <em>itself</em> permissions.</p>
<p>Claude Code's <code>allowed-tools</code> pre-approves tools for the turn that invokes the skill. That's convenient when you wrote it, but it's a capability grant when you did not.</p>
<p>Claude Code's docs are direct: <strong>workspace trust doesn't gate this field.</strong> A project skill's grant applies whenever the skill is invoked, including in a <code>-p</code> run inside a folder you never trusted.</p>
<p>So a hostile skill needs no exploit and no obfuscated payload. It needs one line:</p>
<pre><code class="language-yaml">---
name: deck-builder-pro
description: Brainstorm and build presentation outlines with brand templates.
allowed-tools: Bash
---
</code></pre>
<p>Read the <code>allowed-tools</code> of any skill checked into a repo before running an agent there. It's the cheapest review step available, and the one most often skipped, because frontmatter reads like configuration instead of code.</p>
<p>The two rules sound contradictory but are not: Claude Code gates <em>discovery</em> of project skills behind the trust dialogue, but doesn't gate the <em>grant</em> of an invoked skill.</p>
<h3 id="heading-what-the-data-says">What the Data Says</h3>
<p>The first large-scale study of this ecosystem, <a href="https://arxiv.org/abs/2601.10338">"Agent Skills in the Wild"</a> (Liu et al., January 2026), collected 42,447 skills from two major marketplaces and analysed 31,132 of them:</p>
<ul>
<li><p><strong>26.1%</strong> contained at least one vulnerability.</p>
</li>
<li><p><strong>13.3%</strong> showed data exfiltration patterns, and <strong>11.8%</strong> showed privilege escalation.</p>
</li>
<li><p><strong>5.2%</strong> showed high-severity patterns the authors describe as strongly suggesting malicious intent.</p>
</li>
<li><p>Skills bundling executable scripts were <strong>2.12× more likely</strong> to be vulnerable than instruction-only skills (OR = 2.12, p &lt; 0.001).</p>
</li>
</ul>
<p>Roughly one in four has a problem. One in twenty looks deliberate.</p>
<p>Note that last finding applies squarely to skills shaped like ours. The moment we added <code>validate_deck.py</code>, <code>deck-builder</code> joined the higher-risk category. That's not an argument against bundling scripts. It's an argument for scanning them and a security audit for each agent skill we create for any task.</p>
<h3 id="heading-skillspector">SkillSpector</h3>
<p><a href="https://github.com/NVIDIA/skillspector">SkillSpector</a> is NVIDIA's open-source answer: a static security scanner built for agent skills. Apache-2.0, written in Python, and it exists to answer one question before you install anything: <em>is this safe?</em></p>
<p>It's part of NVIDIA's <a href="https://docs.nvidia.com/skills/">Verified Skills pipeline</a>, which scans, evaluates, and signs skills before they reach the NVIDIA skills catalogue.</p>
<p>As of v2.9.6, it has <strong>70 vulnerability patterns across 17 categories</strong>:</p>
<table>
<thead>
<tr>
<th>Category</th>
<th>Representative patterns</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Prompt injection</strong></td>
<td>Instruction override, hidden directives in comments or invisible text, whitespace padding that pushes text out of visual range</td>
</tr>
<tr>
<td><strong>Anti-refusal</strong></td>
<td>"Never refuse," "omit all disclaimers," jailbreak framing</td>
</tr>
<tr>
<td><strong>Data exfiltration</strong></td>
<td>Environment-variable harvesting, filesystem enumeration, sending conversation context to external endpoints</td>
</tr>
<tr>
<td><strong>Privilege escalation</strong></td>
<td>Sudo and root invocation, reads of SSH keys, tokens, password stores</td>
</tr>
<tr>
<td><strong>Supply chain</strong></td>
<td>Remote execution via <code>curl</code> piped to a shell, base64 payloads, typosquatted packages, shipped <code>.pyc</code>known CVEs via live <a href="https://osv.dev">OSV.dev</a> lookup</td>
</tr>
<tr>
<td><strong>Excessive agency</strong></td>
<td>Unrestricted tool access, high-impact decisions with no human in the loop</td>
</tr>
<tr>
<td><strong>Memory poisoning</strong></td>
<td>Content engineered to persist across sessions</td>
</tr>
<tr>
<td><strong>Rogue agent</strong></td>
<td>Runtime self-modification, persistence via cron or startup scripts</td>
</tr>
<tr>
<td><strong>Trigger abuse</strong></td>
<td>Descriptions crafted to shadow built-ins or maximise spurious activation</td>
</tr>
<tr>
<td><strong>Behavioural AST</strong></td>
<td><code>exec</code>, <code>eval</code>, dynamic imports, <code>subprocess</code>, reflective <code>getattr</code> sinks</td>
</tr>
<tr>
<td><strong>Taint tracking</strong></td>
<td>Credentials flowing to network sinks, file reads reaching network output</td>
</tr>
<tr>
<td><strong>MCP-specific</strong></td>
<td>Tool poisoning via Unicode homoglyphs, parameter-description injection</td>
</tr>
</tbody></table>
<p>Two deserve more attention because they're invisible to ordinary code review.</p>
<p><strong>Trigger abuse</strong> attacks the description field, the very thing we spent Version 2 optimising. The same techniques that make a description fire reliably can be pushed into one engineered to activate on everything. A hostile <code>deck-builder-pro</code> might describe itself as applying to "any document, file, or planning task" precisely so it loads on turns where it has no business loading.</p>
<p><strong>Whitespace padding</strong> hides instructions below the visible area of a file, where a reviewer scrolling casually never sees them.</p>
<p>Both exploit the fact that <code>SKILL.md</code> is read by a model, not compiled by a parser.</p>
<h3 id="heading-scanning">Scanning</h3>
<pre><code class="language-bash">uv tool install git+https://github.com/NVIDIA/skillspector.git
</code></pre>
<pre><code class="language-bash">skillspector scan "$SKILLS_DIR/deck-builder/"
</code></pre>
<p>It accepts folders, single <code>SKILL.md</code> files, Git URLs, and zips. That last capability matters most, because you can scan a skill <strong>before</strong> it reaches your filesystem:</p>
<pre><code class="language-bash">skillspector scan https://github.com/someone/deck-builder-pro
</code></pre>
<p>The analysis happens in two stages. The first is always static: SkillSpector checks the skill with regex patterns, Python AST inspection, YARA signatures, and live CVE lookups. An optional LLM stage then examines intent, filters false positives, and turns the findings into clearer explanations, bringing precision to roughly 87%. Use <code>--no-llm</code> when you want a faster scan or when the skill contents must stay on your machine.</p>
<p>Our <code>deck builder</code> should pass this review cleanly. It makes no network calls, reads no environment variables, starts no subprocesses, and has no external dependencies. A suspicious skill tells a very different story: its scan can expose environment harvesting or unexpected external transmission before you ever let it run.</p>
<pre><code class="language-plaintext"> SkillSpector Security Report

 Skill: deck-builder-pro
 Score           78/100
 Severity        HIGH
 Recommendation  DO NOT INSTALL

 Issues (2)

   HIGH: Env Variable Harvesting (E2)
     Location: scripts/brand_sync.py:23
     Finding: for key, val in os.environ.items():...
     Confidence: 94%

   HIGH: External Transmission (E1)
     Location: scripts/brand_sync.py:45
     Finding: requests.post("https://api.deckmetrics.io/telemetry"...
     Confidence: 89%
</code></pre>
<p>Neither finding is conclusive on its own. A deck skill reading environment variables could have a legitimate reason, such as locating brand assets. Sending data to an external endpoint could also be ordinary analytics.</p>
<p>The concern appears when the two behaviours are considered together: environment data is being collected and transmitted, which can indicate credential exfiltration disguised as telemetry. That's why the scanner correlates related findings instead of treating every match as an isolated problem.</p>
<p>The resulting score falls into a defined risk band. Scores from 0 to 20 are marked <code>LOW</code> or <code>SAFE</code>, 21 to 50 as <code>MEDIUM</code> or <code>CAUTION</code>, 51 to 80 as <code>HIGH</code>, and 81 to 100 as <code>CRITICAL</code>. Both <code>HIGH</code> and <code>CRITICAL</code> findings lead to a <code>DO_NOT_INSTALL</code> recommendation.</p>
<p>The score is built from weighted findings: critical contributes 50 points, high 25, medium 10, and low 5. When a skill includes scripts, the total receives an additional 1.3 multiplier, reflecting the higher risk that executable code introduces.</p>
<h3 id="heading-gate-installs-and-ci">Gate Installs and CI</h3>
<p>Exit codes are a stable contract:</p>
<table>
<thead>
<tr>
<th>Code</th>
<th>Meaning</th>
</tr>
</thead>
<tbody><tr>
<td><code>0</code></td>
<td>Scan completed, score ≤ 50 (<code>SAFE</code> or <code>CAUTION</code>)</td>
</tr>
<tr>
<td><code>1</code></td>
<td>Scan completed, score &gt; 50 (<code>DO_NOT_INSTALL</code>)</td>
</tr>
<tr>
<td><code>2</code></td>
<td>Error: bad input, unreadable source, internal failure</td>
</tr>
</tbody></table>
<p>Because <code>0</code> collapses <code>SAFE</code> and <code>CAUTION</code>, read the <code>recommendation</code> field from JSON when you need to distinguish them:</p>
<pre><code class="language-bash">skillspector scan ./candidate-skill/ --format json --output report.json
</code></pre>
<p><code>--format sarif</code> emits SARIF 2.1.0, which GitHub Advanced Security and most static-analysis dashboards ingest directly. Add it to the CI job already running <code>skills-ref validate</code>.</p>
<p>Repeat scans of your own skill produce findings you already triaged. A baseline suppresses them so re-scans surface only what is new:</p>
<pre><code class="language-bash">skillspector baseline "$SKILLS_DIR/deck-builder/" -o .skillspector-baseline.yaml
</code></pre>
<pre><code class="language-bash">skillspector scan "$SKILLS_DIR/deck-builder/" --baseline .skillspector-baseline.yaml
</code></pre>
<p>Commit the baseline. Fingerprint entries are evidence-bound: changing the source or the scanner version reactivates the finding until reviewed again.</p>
<h3 id="heading-as-a-runtime-gate">As a Runtime Gate</h3>
<p>The most interesting deployment is as an MCP server, moving scanning from an audit to a gate:</p>
<pre><code class="language-bash">uv tool install --force 'skillspector[mcp] @ git+https://github.com/NVIDIA/skillspector.git'
</code></pre>
<pre><code class="language-bash">claude mcp add skillspector -- skillspector mcp
</code></pre>
<p>It exposes one tool, <code>scan_skill</code>, returning <code>risk_score</code>, <code>severity</code>, <code>recommendation</code>, <code>safe_to_install</code>, and findings. It also reports <code>llm_used</code> and <code>scan_mode</code>, so a low score from a static-only pass is never mistaken for a clean full scan.</p>
<p>One caution worth reading twice: <strong>the HTTP transport ships without authentication.</strong> Over stdio or on <code>127.0.0.1</code> that matches the CLI's trust boundary. Bind to a routable interface, and you need an authenticating reverse proxy in front. Local paths and <code>file://</code> URLs are auto-rejected over HTTP, but that's not a substitute for authentication.</p>
<h3 id="heading-know-what-it-doesnt-do">Know What it Doesn't Do</h3>
<p>There's an important boundary to understand before trusting the scan. <strong>SkillSpector never executes the skill it's analysing.</strong> It examines the files statically through regex patterns, AST inspection, and YARA signatures, with an optional LLM pass that evaluates the file contents. The scanner can flag suspicious behaviour before installation, but it can't contain a skill after you choose to install and run it.</p>
<p>That also means the scan has clear blind spots. Patterns may miss non-English content, text embedded inside images isn't analysed, and compiled or encrypted payloads can't be inspected. Runtime behaviour is outside the scanner’s scope. Without access to <code>api.osv.dev</code>, the CVE check falls back to the smaller bundled vulnerability list.</p>
<p>There's also a data handling detail worth checking before you run the scan. With LLM analysis enabled, file contents are sent to your configured provider. Use <code>--no-llm</code> when you need the analysis to remain local. The supply chain check is separate: even with<code>--no-llm</code>, it sends dependency names and versions to <a href="http://OSV.dev">OSV.dev</a>, not the contents of your files. Our <code>deck builder</code> skill declares no dependencies, so there's nothing to send for that check</p>
<h3 id="heading-three-checkpoints">Three Checkpoints</h3>
<ol>
<li><p><strong>Before installing anything you didn't write:</strong> Scan the Git URL before it touches your disk. This is the highest-value scan you'll run.</p>
</li>
<li><p><strong>In CI, on every change to your own skills:</strong> A skill clean last quarter may have picked up a dependency with a fresh CVE.</p>
</li>
<li><p><strong>At install time, via the MCP gate:</strong> An automated gate beats a documented policy nobody follows.</p>
</li>
</ol>
<p>The paper's conclusion is worth restating plainly: this ecosystem needs capability-based permissions and mandatory vetting before the attack surface is exploited more widely. Until permissions arrive, scanning is the practical control, and it takes about ten seconds.</p>
<h2 id="heading-mistakes-that-quietly-waste-your-tokens">Mistakes That Quietly Waste Your Tokens</h2>
<p>Every one of these is a mistake an earlier draft of <code>deck-builder</code> actually made.</p>
<table>
<thead>
<tr>
<th>Mistake</th>
<th>How it showed up</th>
<th>The fix</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Vague description</strong></td>
<td>Version 1's <code>Helps make presentations.</code> never fired on "put something together for Thursday"</td>
<td>Name the operations and trigger conditions, disclaim adjacent domains, then measure the trigger rate</td>
</tr>
<tr>
<td><strong>The monolithic runbook</strong></td>
<td>Inlining every narrative arc pushed <code>SKILL.md</code> past 500 lines, paid on every activation</td>
<td>Move depth to <code>references/</code>, and name when to open it</td>
</tr>
<tr>
<td><strong>Restating general knowledge</strong></td>
<td>An early body explained what a slide is</td>
<td>Write only what the agent gets wrong</td>
</tr>
<tr>
<td><strong>A rule with no owner</strong></td>
<td>"Six bullets max" read like a law of design, so the agent argued with users who wanted seven</td>
<td>Say which rules are house convention and which are hard requirements</td>
</tr>
<tr>
<td><strong>Enforcing in prose only</strong></td>
<td>"Brainstorm first" was a suggestion the model skipped under pressure</td>
<td>Make it a validator rule with a non-zero exit</td>
</tr>
<tr>
<td><strong>Over-applying its own rule</strong></td>
<td>The skill interrogated users who had already supplied audience and message</td>
<td>Add a negative capability test to the eval set</td>
</tr>
<tr>
<td><strong>Absolute paths</strong></td>
<td><code>/Users/you/dev/...</code> Broke on every other machine</td>
<td>Paths relative to the skill root, always</td>
</tr>
<tr>
<td><strong>Interactive scripts</strong></td>
<td>A draft validator prompted <code>Continue? [y/N]</code> and hung the agent</td>
<td>Flags only. Fail with a message naming the missing flag.</td>
</tr>
<tr>
<td><strong>Assuming one discovery path</strong></td>
<td>An early draft used <code>.agents/skills/</code> throughout. On Claude Code the tutorial silently produced nothing.</td>
<td>Check your client's paths, confirm with its skill-listing command</td>
</tr>
<tr>
<td><strong>Skipping the baseline eval</strong></td>
<td>"It produced a better outline" was one run</td>
<td>Evaluate with and without. Delete skills that lose.</td>
</tr>
<tr>
<td><strong>Installing unscanned skills</strong></td>
<td>Adding a script moved the skill into the 2.12× risk category</td>
<td><code>skillspector scan</code> before installing anything you didn't write</td>
</tr>
<tr>
<td><strong>Treating frontmatter as config</strong></td>
<td><code>allowed-tools</code> is a permission grant, and workspace trust doesn't gate it</td>
<td>Read third-party frontmatter as carefully as scripts</td>
</tr>
</tbody></table>
<h2 id="heading-the-pre-flight-checklist">The Pre-Flight Checklist</h2>
<p>Run this before committing a skill others will clone.</p>
<p><strong>Format</strong></p>
<ul>
<li><p>[ ] <code>SKILL.md</code> has valid YAML frontmatter with <code>name</code> and <code>description</code>.</p>
</li>
<li><p>[ ] <code>name</code> is lowercase, hyphenated, ≤ 64 characters, matching the folder exactly.</p>
</li>
<li><p>[ ] <code>description</code> is under 1024 characters and says what it does <strong>and</strong> when to use it.</p>
</li>
<li><p>[ ] <code>skills-ref validate ./deck-builder</code> passes.</p>
</li>
<li><p>[ ] The folder sits somewhere your client actually scans, confirmed with its skill-listing command.</p>
</li>
<li><p>[ ] You know your client's collision precedence, so you know which copy wins.</p>
</li>
<li><p>[ ] Frontmatter uses only the spec's six fields if the skill may travel.</p>
</li>
</ul>
<p><strong>Content</strong></p>
<ul>
<li><p>[ ] The body is under 500 lines and ~5,000 tokens, remembering it stays in context all session.</p>
</li>
<li><p>[ ] Guidance that must hold all task long is written as a standing instruction.</p>
</li>
<li><p>[ ] Nothing explains what the model already knows.</p>
</li>
<li><p>[ ] Environment-specific traps live in a false positives section inside <code>SKILL.md</code>.</p>
</li>
<li><p>[ ] Every reference file has an explicit condition for opening it.</p>
</li>
<li><p>[ ] Rules that are house convention are labelled as such.</p>
</li>
<li><p>[ ] All paths are relative to the skill root.</p>
</li>
</ul>
<p><strong>Scripts</strong></p>
<ul>
<li><p>[ ] Executable (<code>chmod +x</code>) and dependencies declared inline.</p>
</li>
<li><p>[ ] Nothing prompts interactively.</p>
</li>
<li><p>[ ] <code>--help</code> documents purpose, flags, and exit codes.</p>
</li>
<li><p>[ ] Error messages state the remedy, not just the failure.</p>
</li>
<li><p>[ ] Structured output on stdout, diagnostics on stderr.</p>
</li>
<li><p>[ ] No credentials, tokens, or keys anywhere in the skill.</p>
</li>
<li><p>[ ] Any <code>allowed-tools</code> grant is the minimum needed, reviewed as a permission.</p>
</li>
</ul>
<p><strong>Evidence</strong></p>
<ul>
<li><p>[ ] Trigger rate measured against labelled queries with a train/validation split.</p>
</li>
<li><p>[ ] Output quality evaluated with and without the skill, and the skill wins.</p>
</li>
<li><p>[ ] At least one test confirms the skill doesn't over-apply its own rules.</p>
</li>
<li><p>[ ] Token and latency cost known and judged acceptable.</p>
</li>
</ul>
<p><strong>Security</strong></p>
<ul>
<li><p>[ ] <code>skillspector scan</code> reports <code>SAFE</code>, or every <code>CAUTION</code> finding is reviewed and in a committed baseline.</p>
</li>
<li><p>[ ] The scan runs in CI on every change.</p>
</li>
</ul>
<h2 id="heading-key-takeaways">Key Takeaways</h2>
<p>We built one skill, <code>deck-builder</code>, from 28 lines to a validated, evaluated, and scanned package. Every lesson here is visible in that one folder.</p>
<ul>
<li><p><strong>A skill is a folder with a</strong> <code>SKILL.md</code> <strong>file:</strong> No build step, registry, or runtime.</p>
</li>
<li><p><strong>Progressive disclosure is the whole economic argument:</strong> ~100 tokens per skill at discovery instead of ~5,000 is roughly a 96% reduction at 50 skills, and it holds only while most skills stay shut.</p>
</li>
<li><p><strong>Activation persists:</strong> Once invoked, the body stays in context all session. Every line is a recurring cost, and compaction can evict it from a long conversation.</p>
</li>
<li><p><strong>The description does all the triggering:</strong> Version 1 failed on "put something together for Thursday." Version 2 worked because it named the procedure, covered how people actually ask, and disclaimed the near-misses.</p>
</li>
<li><p><strong>Bundle a script wherever correctness matters:</strong> Prose made "brainstorm first" a suggestion. <code>validate_deck.py</code> made skipping it a non-zero exit.</p>
</li>
<li><p>False positives <strong>are the highest-value content you will write:</strong> The most useful line in our skill is the one saying leadership readouts open with the ask. No model can guess your organisation.</p>
</li>
<li><p><strong>Label your conventions as conventions:</strong> A skill that presents house style as universal truth teaches the agent to argue with users who work differently.</p>
</li>
<li><p><strong>Evaluate against a baseline, or you're guessing:</strong> Include a test that catches over-application, not just under-performance.</p>
</li>
<li><p><strong>Know where your client looks, and which skill wins:</strong> <code>.agents/skills/</code> is the cross-client convention, but Claude Code uses <code>.claude/skills/</code> and doesn't scan <code>.agents/</code>. Precedence differs too, and Claude Code documents the reverse of the convention.</p>
</li>
<li><p><strong>The spec is the portable subset:</strong> Clients extend frontmatter heavily, and stricter validation paths reject unknown keys rather than ignoring them.</p>
</li>
<li><p><strong>Scan what you didn't write:</strong> With 26.1% of published skills carrying a vulnerability and 5.2% showing likely malicious intent, an unscanned install is an unexamined trust decision.</p>
</li>
</ul>
<p>Start where we started: with one folder, one file, and one rule your team already repeats out loud. Version 1 took ten minutes. Everything after that was refinement driven by measurement.</p>
<p>The next time you catch yourself explaining the same thing for the fourth time, stop and write it down as a skill instead.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>A useful agent skill isn't about writing more instructions. It's about turning the knowledge your team repeats into a reliable, reusable workflow.</p>
<p>We built <code>deck builder</code> from a small <code>SKILL.md</code> into a validated, evaluated, and security-scanned package. Along the way, the important lessons became clear: keep discovery lightweight, treat activated context as a continuing cost, make descriptions precise enough to trigger on real requests, capture team-specific knowledge in false positives, and move repeatable checks into scripts so correctness can be verified rather than assumed.</p>
<p>Just as importantly, a good skill knows its boundaries. It shouldn't force its workflow when the user has already thought. It shouldn't pretend team conventions are universal rules. It shouldn't be trusted simply because it looks like harmless configuration, especially when it comes from a repository you didn't write.</p>
<p>The pattern is simple: <strong>write down the knowledge, make the workflow explicit, test whether it actually helps, and scan what you didn't write.</strong></p>
<p>The next time you catch yourself explaining the same process for the fourth time, that's probably your signal. Stop repeating it and turn it into a skill.</p>
<p>Happy building, and scan before you install.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ The ETL Pipeline Handbook: How to Build a Production-Grade Pipeline in Python ]]>
                </title>
                <description>
                    <![CDATA[ Tracking flood risk takes one unglamorous but essential thing: clean and structured data. In this tutorial, you'll build a data pipeline yourself. You'll create a Python ETL (Extract, Transform, Load) ]]>
                </description>
                <link>https://www.freecodecamp.org/news/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python/</link>
                <guid isPermaLink="false">6a679921f4d9ad6845fede20</guid>
                
                    <category>
                        <![CDATA[ data-engineering ]]>
                    </category>
                
                    <category>
                        <![CDATA[ handbook ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Tutorial ]]>
                    </category>
                
                    <category>
                        <![CDATA[ ETL ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ python projects ]]>
                    </category>
                
                    <category>
                        <![CDATA[ pandas ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Pipeline ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ brooklyn ]]>
                </dc:creator>
                <pubDate>Mon, 27 Jul 2026 17:45:05 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/36906090-d056-4207-8632-fcdf35843018.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Tracking flood risk takes one unglamorous but essential thing: clean and structured data.</p>
<p>In this tutorial, you'll build a data pipeline yourself. You'll create a Python ETL (Extract, Transform, Load) pipeline that pulls daily water-level readings from <a href="https://hubeau.eaufrance.fr/">Hub'Eau</a>, France's official open water-data API. Then, you'll clean that data and publish it as a public dataset, just like the <a href="https://www.kaggle.com/code/grimespoint/paris-flood-dataset-weekly-updater">live version</a> does.</p>
<p>This tutorial is based on a real pipeline that runs once a week, on a schedule, and it keeps the <a href="https://www.kaggle.com/datasets/grimespoint/paris-flood-dataset">Paris Flood Dataset</a> updated automatically.</p>
<p>You won't just copy and paste code, though. The real goal is to understand <em>why</em> the pipeline works the way it does. You'll walk through the design decisions that separate a script meant to run once from a script that keeps working, unattended, for years.</p>
<p>You can code along with this <a href="https://www.kaggle.com/code/grimespoint/data-engineering-with-python-etl-pipeline">notebook</a>. For most of the tutorial, the pipeline runs on <strong>simulated (mock) API data</strong>. This lets you run every cell safely, without hammering a real server. A later section shows how to switch to the live API.</p>
<p>By the end, you'll be able to:</p>
<ul>
<li><p>Explain and implement the Extract, Transform, Load pattern</p>
</li>
<li><p>Manage configuration with Python <code>@dataclass</code> instead of scattering constants everywhere</p>
</li>
<li><p>Write API-fetching code that survives network failures and paginated responses</p>
</li>
<li><p>Apply robust type-coercion so one bad row can't crash a whole pipeline run</p>
</li>
<li><p>Deduplicate and merge incremental data safely</p>
</li>
<li><p>Wire everything into a single, idempotent, schedulable <code>main()</code> entry point</p>
</li>
</ul>
<h3 id="heading-table-of-contents">Table of Contents:</h3>
<ul>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-part-1-the-design-logic">Part 1: The Design Logic</a></p>
<ul>
<li><p><a href="#heading-what-is-an-etl-pipeline">What is an ETL pipeline?</a></p>
</li>
<li><p><a href="#heading-the-architecture-at-a-glance">The architecture, at a glance</a></p>
</li>
<li><p><a href="#heading-two-patterns-that-make-it-production-grade">Two patterns that make it production-grade</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-2-setup-and-dependencies">Part 2: Setup and Dependencies</a></p>
</li>
<li><p><a href="#heading-part-3-manage-configuration-with-dataclasses">Part 3: Manage Configuration with Dataclasses</a></p>
<ul>
<li><p><a href="#heading-why-bother-with-a-config-layer-at-all">Why bother with a config layer at all?</a></p>
</li>
<li><p><a href="#heading-code-level-walkthrough">Code-level walkthrough</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-4-the-extraction-step">Part 4: The Extraction Step</a></p>
<ul>
<li><p><a href="#heading-graceful-file-loading">Graceful file loading</a></p>
</li>
<li><p><a href="#heading-incremental-update-logic">Incremental update logic</a></p>
</li>
<li><p><a href="#heading-simulate-the-api">Simulate the API</a></p>
</li>
<li><p><a href="#heading-fetch-data-for-real">Fetch data for real</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-5-the-transform-step">Part 5: The Transform Step</a></p>
<ul>
<li><p><a href="#heading-type-parsing-and-graceful-coercion">Type parsing and graceful coercion</a></p>
</li>
<li><p><a href="#heading-schema-translation-with-bidirectional-mappings">Schema translation with bidirectional mappings</a></p>
</li>
<li><p><a href="#heading-compute-flood-alerts">Compute flood alerts</a></p>
</li>
<li><p><a href="#heading-column-ordering">Column ordering</a></p>
</li>
<li><p><a href="#heading-deduplication">Deduplication</a></p>
</li>
<li><p><a href="#heading-put-it-all-together-in-postprocess">Put it all together inpostprocess()</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-6-the-load-step">Part 6: The Load Step</a></p>
<ul>
<li><p><a href="#heading-design-logic">Design logic</a></p>
</li>
<li><p><a href="#heading-code-level-walkthrough">Code-level walkthrough</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-7-assemble-the-full-pipeline">Part 7: Assemble the Full Pipeline</a></p>
<ul>
<li><p><a href="#heading-the-global-rehearsal-mock-mode">The global rehearsal (mock mode)</a></p>
</li>
<li><p><a href="#heading-main-pipeline-orchestration">main(): pipeline orchestration</a></p>
</li>
<li><p><a href="#heading-the-if-name-main-guard">Theif name == "main":guard</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-8-go-live-and-switch-to-the-real-api">Part 8: Go Live and Switch to the Real API</a></p>
<ul>
<li><a href="#heading-post-run-validation">Post-run validation</a></li>
</ul>
</li>
<li><p><a href="#heading-summary">Summary</a></p>
</li>
<li><p><a href="#heading-references">References</a></p>
</li>
</ul>
<h2 id="heading-prerequisites">Prerequisites</h2>
<ul>
<li><p><strong>Python 3.10+</strong>. The code uses type hints and dataclasses. These work from Python 3.7 onward, but 3.10+ is best.</p>
</li>
<li><p><a href="https://leetcode.com/studyplan/introduction-to-pandas/"><strong>Knowledge of pandas DataFrames</strong></a>: read data, filter rows, and basic column operations.</p>
</li>
<li><p>Comfort with <strong>functions and basic OOP</strong> (<a href="https://realpython.com/python3-object-oriented-programming/">Object-Oriented programming</a>) in Python. Don't worry, this guide explains every non-obvious piece as you go.</p>
</li>
<li><p>Optional: a free <a href="https://www.kaggle.com/">Kaggle</a> account and the <a href="https://github.com/Kaggle/kaggle-api">Kaggle CLI</a>, only if you want to run the final publishing step for real.</p>
</li>
</ul>
<p>Install the dependencies:</p>
<pre><code class="language-bash">pip install requests pandas numpy ipykernel
</code></pre>
<p>You'll do the work inside a <a href="https://www.kaggle.com/code/grimespoint/data-engineering-with-python-etl-pipeline">Jupyter notebook</a>. Download the <code>.ipynb</code> file from Kaggle, or click "Copy and Edit" to work directly on Kaggle. You'll need a Kaggle account for that.</p>
<img src="https://cdn.hashnode.com/uploads/covers/67c84561e3f229edf2351ba2/681c54fc-c12a-4c01-83ba-17368d4ccc49.png" alt="The menu button that provides options to download the data engineering follow along notebook on Kaggle." style="display: block;" width="695" height="502" loading="lazy">

<p><strong>Optional:</strong> If you plan to run the notebook locally, install the <code>notebook</code> package too:</p>
<pre><code class="language-bash">pip install notebook
</code></pre>
<h2 id="heading-part-1-the-design-logic">Part 1: The Design Logic</h2>
<p>Before you touch a single line of code, we'll spend three minutes on <em>why</em> the pipeline is shaped this way.</p>
<p>This is the <strong>big-picture view</strong>. Every code-level decision later traces back to one of these ideas. Read this section even if you skim everything else.</p>
<h3 id="heading-what-is-an-etl-pipeline">What is an ETL Pipeline?</h3>
<p>ETL stands for <strong>Extract, Transform, Load</strong>. It's the standard pattern for moving data from a source to a destination in a reliable, repeatable way.</p>
<ul>
<li><p><strong>Extract</strong>: pull data from a source, like an API, a database, or files.</p>
</li>
<li><p><strong>Transform</strong>: clean, standardize, enrich, and validate the data.</p>
</li>
<li><p><strong>Load</strong>: write the result to a destination, like a warehouse, a CSV, or a public platform.</p>
</li>
</ul>
<p>Here's how those three stages map onto this project:</p>
<table>
<thead>
<tr>
<th>Stage</th>
<th>What happens here</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Extract</strong></td>
<td>Load existing data if it exists, then call the Hub'Eau API (via <code>requests</code>) for each gauging station</td>
</tr>
<tr>
<td><strong>Transform</strong></td>
<td>Translate French columns and entries to English, fix data types, remove duplicates</td>
</tr>
<tr>
<td><strong>Load</strong></td>
<td>Write a CSV and a metadata file, then publish to Kaggle via the CLI</td>
</tr>
</tbody></table>
<h3 id="heading-the-architecture-at-a-glance">The Architecture, at a Glance</h3>
<img src="https://cdn.hashnode.com/uploads/covers/67c84561e3f229edf2351ba2/6bede708-bdb9-4cae-bfd8-1dc1aa0348f6.png" alt="Schema of a simple ETL pipeline." style="display: block;" width="2040" height="900" loading="lazy">

<p>Notice that "Extract" already touches two different sources: the <em>existing</em> dataset (what you already have) and the <em>new</em> data from the API. That distinction is the seed of the next idea.</p>
<h3 id="heading-two-patterns-that-make-it-production-grade">Two Patterns that Make it Production-Grade</h3>
<p>There two patterns that make this pipeline safe to run unattended, scheduled, for years.</p>
<h4 id="heading-1-idempotency">1. Idempotency</h4>
<p><a href="https://en.wikipedia.org/wiki/Idempotence"><strong>Idempotence</strong></a> means when the same operation runs twice it gives the same result as if it runs once. Deduplication is what makes this pipeline idempotent. If the scheduler accidentally triggers twice, or a network retry fetches the same day again, the second run won't create duplicate rows. This matters a lot for anything that runs on a schedule with no one watching it.</p>
<h4 id="heading-2-incremental-loading">2. Incremental loading</h4>
<p>A naïve pipeline would re-download the <em>entire</em> history on every run. That's slow, it wastes your API quota, and it's fragile: the more data you transfer, the more chances something fails.</p>
<p>An <strong>incremental</strong> pipeline avoids this. Instead, it:</p>
<ul>
<li><p>Checks the most recent date already in the dataset</p>
</li>
<li><p>Requests only the data <em>from that date onward</em></p>
</li>
<li><p>Merges the new records into the existing dataset</p>
</li>
</ul>
<p>You'll find this logic built explicitly in <a href="#heading-incremental-update-logic"><code>determine_update_range</code></a>.</p>
<p>Always ask yourself: <strong>"Do I actually need to do this?"</strong> That one habit separates a fragile script from a production pipeline. For example, every "expensive" or "external" step in this pipeline (network calls, disk writes, publishing) is guarded by a cheap, local check first.</p>
<h2 id="heading-part-2-setup-and-dependencies">Part 2: Setup and Dependencies</h2>
<p>Here's the import block. It looks unremarkable, but how it's organized is itself a best practice worth calling out.</p>
<pre><code class="language-python"># Standard library imports
import json
import os
import random
import subprocess
from dataclasses import dataclass, field
from datetime import date, timedelta
from pathlib import Path
from typing import Dict, List, Optional, Set, Tuple

# Third-party libraries
import numpy as np
import pandas as pd
import requests

# Display config for notebooks
pd.set_option('display.max_columns', None)   # All columns will show
pd.set_option('display.max_colwidth', None)  # Prevents cutting long column text with ...
</code></pre>
<p><strong>Code-level notes:</strong></p>
<ul>
<li><p>Imports fall into three blocks: <strong>standard library, then third-party, then local</strong>, with blank lines between them. This follows <a href="https://peps.python.org/pep-0008/#imports">PEP 8's import-ordering convention</a>. Most <a href="https://en.wikipedia.org/wiki/Pretty-printing">auto-formatters</a> (<code>isort</code>, <code>ruff</code>) enforce this same grouping.</p>
</li>
<li><p>Nothing is imported with <code>from module import *</code>. That syntax pollutes the current namespace. It makes it hard to trace where a name came from when someone debugs the code six months later. Python style guides echo this advice, including <a href="https://google.github.io/styleguide/pyguide.html">Google's Python Style Guide</a>.</p>
</li>
<li><p>The <code>pd.set_option(...)</code> calls exist purely for <em>notebook readability</em>, so wide DataFrames don't get truncated with <code>...</code>. They have zero effect on the pipeline's logic. You'd typically remove or scope them differently in a <code>.py</code> script.</p>
</li>
</ul>
<h2 id="heading-part-3-manage-configuration-with-dataclasses">Part 3: Manage Configuration with Dataclasses</h2>
<h3 id="heading-why-bother-with-a-config-layer-at-all">Why Bother with a Config Layer at All?</h3>
<p>Every pipeline has <strong>knobs</strong>: which stations to monitor, what the flood threshold is, and where to publish. The bad approach sprinkles these values as literals throughout the code as you write it. You end up with an <code>if level &gt; 6000:</code> buried three functions deep. Then changing <em>any</em> setting means hunting through the whole file, and it's easy to update one spot and miss another.</p>
<p>The fix: <strong>centralize all settings in one place.</strong> Python's <a href="https://docs.python.org/3/library/dataclasses.html"><code>@dataclass</code></a> decorator is a natural fit for that.</p>
<table>
<thead>
<tr>
<th>Feature</th>
<th>Why it matters</th>
</tr>
</thead>
<tbody><tr>
<td>Auto-generated <code>__init__</code>, <code>__repr__</code>, <code>__eq__</code></td>
<td>You don't need to write them yourself</td>
</tr>
<tr>
<td><a href="https://www.geeksforgeeks.org/python/type-hints-in-python/">Type hints</a></td>
<td>Gives you IDE autocomplete and self-documenting code</td>
</tr>
<tr>
<td>Optional <code>frozen=True</code></td>
<td>Gives you <a href="https://stackoverflow.com/questions/66194804/what-does-frozen-mean-for-dataclasses">true immutability</a> if you want config knobs that can't change after creation</td>
</tr>
<tr>
<td><code>__post_init__</code> hook</td>
<td>Validates or computes derived fields once, right after construction</td>
</tr>
</tbody></table>
<p>Compare that to a configuration written as a plain <code>dict</code>:</p>
<pre><code class="language-python">config = {
    "stations": ["STN001", "STN002"],
    "flood_threshold": 6000,
    "publish_url": "https://example.com/alerts",
    "retry_count": 2,
    "timeout_seconds": 5,
}
</code></pre>
<p>A plain dict gives you none of that: no immutability, no type checking, and no autocomplete.</p>
<h3 id="heading-code-level-walkthrough">Code-Level Walkthrough</h3>
<p>The pipeline defines three configuration classes. Each one has <em>a single responsibility</em>: API details, station rules, and publishing destination. Each class becomes one <strong>module-level singleton instance</strong>. Every other function in the pipeline reads from these singletons.</p>
<pre><code class="language-python">@dataclass
class APIConfig:
    """API configuration for HubEau data fetching.

    Think of this as the "address book" for the API.
    """
    use_mock: bool = True
    base_url: str = "https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab"
    metric: str = "HIXnJ"  # Daily max water level (elaborated observations)
    # Pagination: fetch 20k records per request (API Limit)
    max_per_page: int = 20000
    timeout_seconds: int = 60  # Network timeout

    def __post_init__(self):
        """Validate configuration after initialization."""
        if self.max_per_page &lt;= 0:
            raise ValueError("max_per_page must be positive")
        if self.timeout_seconds &lt;= 0:
            raise ValueError("timeout_seconds must be positive")
</code></pre>
<p>Notice the validation inside <code>__post_init__</code>. It runs right after the auto-generated <code>__init__</code>. A misconfigured <code>APIConfig(max_per_page=-1)</code> fails loudly and immediately at startup, instead of surfacing as a bug later during an actual pipeline run.</p>
<pre><code class="language-python">@dataclass
class StationConfig:
    """Station monitoring configuration.

    The 'what' of data collection: which stations, what's a flood?
    """
    station_codes: List[str] = field(default_factory=lambda: [
        "F700000109", "F700000110", "F700000111",
        "F700000102", "F700000103",
    ])
    flood_threshold_mm: int = 6000  # Flood alert threshold
    earliest_date: str = "1900-01-01"  # How far back to go
</code></pre>
<p><strong>The mutable-default trap:</strong> Look closely at <code>station_codes</code>. It isn't written as <code>station_codes: List[str] = [...]</code>. That's deliberate, and it dodges one of the most common gotchas in Python. If you use a plain mutable object (a list, dict, or set) as a default argument or dataclass field, <strong>every instance shares the same underlying object</strong>. Mutate it on one instance, and you silently mutate it everywhere else too.</p>
<p>Stack Overflow covers this at length in its <a href="https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument">"least astonishment" mutable-default-argument discussion</a>, as does <a href="https://realpython.com/python-optional-arguments/">Real Python's guide to optional arguments</a>. The fix is <code>field(default_factory=...)</code>. It calls a <em>fresh</em> factory function (here, a <code>lambda</code>) for every new instance, so each one gets its own independent list.</p>
<p>Explanation:</p>
<pre><code class="language-python"># Bad: shared mutable default
@dataclass
class BadConfig:
    station_codes: list[str] = []

a = BadConfig()
b = BadConfig()

a.station_codes.append("ALERT")
print(a.station_codes)  # ['ALERT']
print(b.station_codes)  # ['ALERT']  &lt;-- same list!!


# Good: fresh list per instance
@dataclass
class GoodConfig:
    station_codes: list[str] = field(default_factory=list)

x = GoodConfig()
y = GoodConfig()

x.station_codes.append("ALERT")
print(x.station_codes)  # ['ALERT']
print(y.station_codes)  # []  &lt;-- independent list
</code></pre>
<p>And the last Config singleton, the Kaggle settings:</p>
<pre><code class="language-python">@dataclass
class KaggleConfig:
    """Kaggle dataset publishing configuration."""
    dataset_slug: str = "grimespoint/paris-flood-dataset"
    input_csv: str = "kaggle/input/datasets/{slug}/paris_flood_dataset.csv"
    output_dir: Path = field(default_factory=lambda: Path("kaggle/working/kaggle_dataset"))
    mock_output_dir: Path = field(default_factory=lambda: Path("mock_output"))
    output_filename: str = "paris_flood_dataset.csv"
    mock_output_filename: str = "mock_flood_dataset.csv"
    metadata_filename: str = "dataset-metadata.json"

    # Metadata
    title: str = "Paris flood dataset"
    keywords: list = field(default_factory=lambda: [
        "tabular", "weather and climate", "environment", "europe", "time series analysis"
    ])
    geospatial_coverage: str = "Paris, France"
    update_frequency: str = "Weekly"
    license_name: str = "CC0-1.0"

    # Computed fields (set in __post_init__)
    output_csv_path: Path = field(init=False)
    metadata_path: Path = field(init=False)

    def __post_init__(self):
        """Compute derived paths after initialization."""
        self.input_csv = self.input_csv.format(slug=self.dataset_slug)
        self.output_csv_path = self.output_dir / self.output_filename
        self.metadata_path = self.output_dir / self.metadata_filename
        self.mock_output_filename = self.mock_output_dir / self.mock_output_filename

# Initialize configs - module-level singletons
API_CONFIG = APIConfig()
STATION_CONFIG = StationConfig()
KAGGLE_CONFIG = KaggleConfig()
</code></pre>
<p>This is the most common use of <code>__post_init__</code>. <code>output_csv_path</code> and <code>metadata_path</code> are marked <code>field(init=False)</code>, so you can't set them directly through the constructor. Instead, <code>__post_init__</code> computes them from other fields (<code>output_dir</code> and <code>output_filename</code>).</p>
<p>Use this pattern for <strong>derived, computed values</strong>: compute them once, in one place, instead of recomputing <code>output_dir / output_filename</code> every time you need the path elsewhere in the codebase.</p>
<p>See <a href="https://realpython.com/python-data-classes/#comparing-cards">Real Python's data classes guide</a> or the data classes chapter of O'Reilly's <em>Fluent Python</em> for more on this pattern.</p>
<p><strong>Tip:</strong> the auto-generated <code>__repr__</code> gives you a readable printout for free. Call <code>print(API_CONFIG)</code>: it shows every field and value without a single line of formatting code. It's handy for quick sanity checks when you're debugging a pipeline run.</p>
<pre><code class="language-python">print(APIConfig)
# prints APIConfig(use_mock=True, base_url='https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab', ...)
</code></pre>
<h2 id="heading-part-4-the-extraction-step">Part 4: The Extraction Step</h2>
<p>The Extract phase pulls data from source systems and reads it into memory. Here, you genuinely have <strong>two</strong> sources to extract from: the <em>existing</em> dataset (what you already published last time) and the <em>new</em> (recent) data from the Hub'Eau API.</p>
<h3 id="heading-graceful-file-loading">Graceful File Loading</h3>
<h4 id="heading-design-logic">Design logic</h4>
<p>What should happen the very first time this pipeline runs (and there's no existing dataset yet)? A naïve implementation would crash with a <code>FileNotFoundError</code>.</p>
<p>Here's the trick: <code>load_csv()</code> follows the <a href="https://en.wikipedia.org/wiki/Null_object_pattern"><strong>Null Object pattern</strong></a>. Instead of raising an error, it returns an <em>empty</em> DataFrame. Every downstream function can then treat "no existing data" and "some existing data" the same way, with no special-casing needed.</p>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p>This function loads a CSV file into a DataFrame. If the file doesn't exist, it returns an empty DataFrame instead of crashing.</p>
<p><code>low_memory=False</code> tells pandas to read the file carefully, so it avoids mixed-type guesses. <code>parse_dates=True</code> tries to automatically convert date-like columns into dates. <code>delimiter=","</code> tells pandas the file is comma-separated.</p>
<pre><code class="language-python">def load_csv(path: str) -&gt; pd.DataFrame:
    """Load CSV file or return empty DataFrame if file does not exist.

    Args:
        path (str): Full path to the CSV file.

    Returns:
        pd.DataFrame: Loaded data, or empty DataFrame if file not found.

    Raises:
        pd.errors.ParserError: If the CSV is malformed.
    """
    if os.path.exists(path):
        return pd.read_csv(path, low_memory=False, parse_dates=True, delimiter=",")
    return pd.DataFrame()   # Null Object: consistent return type
</code></pre>
<p>To test, try it against a path that doesn't exist:</p>
<pre><code class="language-python">df_missing = load_csv("/tmp/does_not_exist.csv")
print(df_missing.empty)  # True: no crash

# Callers can always do this, instead of an `is None` check:
if df_missing.empty:
    print("No existing data. Will run a full fetch from earliest date.")
</code></pre>
<p><strong>Best practice:</strong> return a <strong>consistent type</strong> from every code path in a function. A function that sometimes returns a <code>DataFrame</code> and sometimes <code>None</code> forces every caller to add a <code>None</code> check before using the result.</p>
<p>Return a <code>DataFrame</code>, empty or not, which keeps things simpler. You never have to ask <em>"did I get a real result, or</em> <code>None</code><em>?"</em> before using it.</p>
<h3 id="heading-incremental-update-logic">Incremental Update Logic</h3>
<h4 id="heading-design-logic">Design logic</h4>
<p>This is the "incremental loading" pattern from Part 1 in action. Before you fetch anything, ask: <em>"What's the most recent record I already have, and do I actually need more?"</em></p>
<p><strong>Strategy:</strong> check whether the existing data already covers <em>yesterday</em>.</p>
<ul>
<li><p>Yes: skip the update entirely. Nothing to do, the dataset is current.</p>
</li>
<li><p>No: fetch starting from the day after the last known date.</p>
</li>
</ul>
<pre><code class="language-text">Existing data: Jan. 1 – Jan. 15
Yesterday: Jan. 19

Decision: fetch from Jan. 16 onwards (not from Jan. 1)
</code></pre>
<p>Why <em>yesterday</em> and not <em>today</em>? Today's measurement might not be finalized on the source system yet. Hub'Eau's "elaborated observations" are a processed daily aggregate, so the safest check is against the last <strong>fully completed</strong> day.</p>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p><code>determine_update_range()</code> checks the newest saved date. It tells you either "you're already up to date" or "start downloading from this next date."</p>
<p>Step by step:</p>
<ol>
<li><p><strong>The function starts with existing data</strong></p>
<ul>
<li><code>existing</code> is a pandas <code>DataFrame</code> that already has some rows of data.</li>
</ul>
</li>
<li><p><strong>If there is no data at all</strong></p>
<ul>
<li><p><code>if existing.empty:</code></p>
</li>
<li><p>If the DataFrame has zero rows, it says: <em>"Nothing is saved yet, so fetch everything."</em></p>
</li>
<li><p>It returns:</p>
<ul>
<li><p><code>True</code> = update needed</p>
</li>
<li><p><code>STATION_CONFIG.earliest_date</code> = start from the earliest allowed date</p>
</li>
</ul>
</li>
</ul>
</li>
<li><p><strong>Find the date column</strong></p>
<ul>
<li><p>The code checks which column contains dates:</p>
<ul>
<li><p>first tries <code>"date_obs_elab"</code></p>
</li>
<li><p>then <code>"record_date"</code></p>
</li>
</ul>
</li>
<li><p>If neither exists, it raises an error because it doesn’t know which column to use.</p>
</li>
</ul>
</li>
<li><p><strong>Convert the date column into real dates</strong></p>
<ul>
<li><p><code>pd.to_datetime(...)</code> turns the column into date objects pandas can work with.</p>
</li>
<li><p><code>errors="coerce"</code> means bad date values become missing values instead of crashing.</p>
</li>
</ul>
</li>
<li><p><strong>Find the latest date in the data</strong></p>
<ul>
<li><p><code>last_day = s.max().date()</code></p>
</li>
<li><p>This gets the newest date already in the dataset.</p>
</li>
</ul>
</li>
<li><p><strong>Compare it with yesterday</strong></p>
<ul>
<li><p><code>yesterday = date.today() - timedelta(days=1)</code></p>
</li>
<li><p>The function checks whether data already includes yesterday.</p>
</li>
</ul>
</li>
<li><p><strong>If data is already up to date</strong></p>
<ul>
<li><p>If <code>last_day &gt;= yesterday</code>, it returns:</p>
<ul>
<li><p><code>False</code> = no update needed</p>
</li>
<li><p><code>None</code> = no start date needed</p>
</li>
</ul>
</li>
</ul>
</li>
<li><p><strong>If data is behind</strong></p>
<ul>
<li><p>It sets <code>next_day</code> to the day after the last saved date.</p>
</li>
<li><p>Then it returns:</p>
<ul>
<li><p><code>True</code> = update needed</p>
</li>
<li><p>that next day as a string like <code>"2026-07-12"</code></p>
</li>
</ul>
</li>
</ul>
</li>
</ol>
<pre><code class="language-python">def determine_update_range(existing: pd.DataFrame) -&gt; Tuple[bool, Optional[str]]:
    """Determine whether an update is needed and from what date.

    Logic:
    1. Check if existing data covers yesterday's date
    2. If yes → no update needed
    3. If no → start fetching from day after last data

    Returns:
        Tuple[should_update, start_date]
    """
    # Case 1: nothing on disk yet
    if existing.empty:
        print("No existing data found. Will fetch all data from earliest date.")
        return True, STATION_CONFIG.earliest_date

    if "date_obs_elab" in existing.columns:
        record_colname = "date_obs_elab"
    elif "record_date" in existing.columns:
        record_colname = "record_date"
    else:
        raise KeyError("Missing date column: expected 'date_obs_elab' or 'record_date'")

    s = pd.to_datetime(existing[record_colname], errors="coerce")
    last_day = s.max().date()
    yesterday = date.today() - timedelta(days=1)

    # Case 2: already current
    if last_day &gt;= yesterday:
        print("\nDataset already covers yesterday or later. No update needed.")
        return False, None

    # Case 3: fetch the gap
    next_day = (last_day + pd.Timedelta(days=1))
    print(f"\nWill retrieve data starting from: {next_day}")
    return True, next_day.isoformat()
</code></pre>
<p>A few things worth a note here.</p>
<p>First, the function checks for <strong>two possible column names</strong>: <code>date_obs_elab</code>, the raw API name, or the already-renamed English name <code>record_date</code>. It doesn't assume just one. This makes the function work whether you call it on freshly-fetched raw data or an already-processed CSV loaded from disk.</p>
<p><code>errors="coerce"</code> shows up here, and it'll show up again (we'll dig into this in Part 5). Any date pandas can't parse becomes <code>NaT</code> (Not a Time) instead of raising an exception.</p>
<p>The return type is <code>Tuple[bool, Optional[str]]</code>. This <a href="https://www.w3schools.com/python/python_tuples.asp">tuple</a> bundles two related results: <em>should I update, and from when?</em> That beats returning two separate values, or worse, one ambiguous value that means different things depending on context.</p>
<p><strong>Best practice:</strong> use <code>Tuple</code> return types (or, for more fields, a small dataclass or <code>NamedTuple</code>) to bundle related results together. Document clearly what each position means. If you return different <em>types</em> from different code paths without documentation, you'll hit a common source of confusion and bugs: <em>"why is this</em> <code>None</code> <em>sometimes and a string other times?"</em>.</p>
<p>Run a quick check against three scenarios:</p>
<pre><code class="language-python"># Test 1: No existing data
should_update, start_date = determine_update_range(pd.DataFrame())
# → True, "1900-01-01"

# Test 2: Old existing data (covers only Jan 10-15)
# → True, "2026-01-16"  (the day after the last known date)

# Test 3: Recent data that already covers yesterday
# → False, None
</code></pre>
<h3 id="heading-simulate-the-api">Simulate the API</h3>
<h4 id="heading-design-logic">Design logic</h4>
<p>In production, fetching means a real HTTP call:</p>
<pre><code class="language-python">requests.get(
    "https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab",
    params={"code_entite": "F700000109", "size": 20000, ...}
)
</code></pre>
<p>Real API calls bring real challenges. Rather than fight those on your first read-through, this tutorial first builds and tests everything against a <strong>mock</strong> generator. It returns data shaped exactly like the real API. Only once the logic works does it swap in the real endpoint.</p>
<p>This technique is useful well beyond this project. Build and test your transform logic against fixtures or mocks first. That way, you're not into "is my parsing wrong?" and "is the network flaky right now?" at the same time.</p>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p><code>generate_mock_api_data()</code> generates fake sample data for a station, one record per day, starting from <code>start_date</code>.</p>
<p>How it works:</p>
<ul>
<li><p>It converts <code>start_date</code> into a real date.</p>
</li>
<li><p>It loops for <code>num_days</code>.</p>
</li>
<li><p>For each day, it creates a mock observation dictionary.</p>
</li>
<li><p>It adds a small random variation to the water level so the data looks realistic.</p>
</li>
<li><p>It randomly picks status, quality, and method labels.</p>
</li>
<li><p>It returns a list of these dictionaries.</p>
</li>
</ul>
<pre><code class="language-python">def generate_mock_api_data(station_code: str, start_date: str, num_days: int = 10) -&gt; List[Dict]:
    """Generate realistic mock API data for demonstration.

    Simulates what HubEau API would return: list of observation dicts.
    """
    start = pd.to_datetime(start_date).date()
    records = []

    validation_statuses = ["Donnée validée", "Donnée brute", "Donnée pré-validée"]
    qualities = ["Bonne", "Non qualifiée", "Douteuse"]
    methods = ["Mesurée", "Calculée", "Expertisée"]

    for i in range(num_days):
        obs_date = start + timedelta(days=i)
        base_level = 5500 + int(station_code[-2:])  # Varies by station
        noise = random.randint(-200, 200)
        water_level = base_level + noise

        record = {
            "code_site": "mock_" + station_code[1:],
            "code_station": "mock_" + station_code,
            "date_obs_elab": obs_date.isoformat(),
            "resultat_obs_elab": water_level,
            "date_prod": (obs_date + timedelta(days=1)).isoformat(),
            "code_statut": "1",
            "libelle_statut": random.choice(validation_statuses),
            "code_methode": "1",
            "libelle_methode": random.choice(methods),
            "code_qualification": "1",
            "libelle_qualification": random.choice(qualities),
            "longitude": 2.3522 + random.uniform(-0.01, 0.01),
            "latitude": 48.8566 + random.uniform(-0.01, 0.01),
            "grandeur_hydro_elab": "mock_HIXnJ",
        }
        records.append(record)

    return records
</code></pre>
<h3 id="heading-fetch-data-for-real">Fetch Data for Real</h3>
<h4 id="heading-design-logic">Design logic</h4>
<p>Two functions handle extraction. They're deliberately split according to the <a href="https://en.wikipedia.org/wiki/Single-responsibility_principle"><strong>Single Responsibility Principle</strong></a> (SRP): each function should have one reason to change.</p>
<pre><code class="language-text">fetch_all_data() (orchestrator)
    ├── fetch_single_station_data(station_1) ← handles all complexity
    ├── fetch_single_station_data(station_2) ← handles all complexity
    └── fetch_single_station_data(station_n) ← handles all complexity
</code></pre>
<ul>
<li><p><code>fetch_single_station_data()</code> owns <em>all</em> the messy per-station complexity: pagination, cursor advancement, stop conditions, and network error handling.</p>
</li>
<li><p><code>fetch_all_data()</code> owns none of that. It just loops over stations and delegates.</p>
</li>
</ul>
<p>This split has two payoffs. First, you can debug or swap out the pagination strategy for one station without touching the orchestration code at all. Second, if you ever want to parallelize fetching (with <code>concurrent.futures</code> or <code>asyncio</code>, for example), the orchestrator is the <em>only</em> place you'd need to touch.</p>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p><code>fetch_single_station_data()</code> fetches station data either from mock test data or from the real API, page by page, until it has everything it needs.</p>
<ul>
<li><p><strong>If</strong> <code>use_mock=True</code>, it uses <strong>fake data</strong> instead of calling the real API.</p>
<ol>
<li><p>It calls <code>generate_mock_api_data()</code></p>
</li>
<li><p>Turns the result into a DataFrame</p>
</li>
<li><p>Converts the date column into real pandas dates</p>
</li>
<li><p>Returns that DataFrame</p>
</li>
</ol>
</li>
<li><p><strong>If</strong> <code>use_mock=False</code>, it does the <strong>real API request</strong>:</p>
<ol>
<li><p>Creates a reusable HTTP session</p>
</li>
<li><p>Starts from <code>start_date</code></p>
</li>
<li><p>Repeatedly asks the API for a page of data</p>
</li>
<li><p>Stops when:</p>
<ul>
<li><p>the API returns no data</p>
</li>
<li><p>the latest date reaches yesterday</p>
</li>
<li><p>the page is smaller than expected</p>
</li>
<li><p>a network error happens</p>
</li>
</ul>
</li>
<li><p>Combines all pages into one DataFrame</p>
</li>
<li><p>Returns an empty DataFrame if nothing was fetched</p>
</li>
</ol>
</li>
</ul>
<pre><code class="language-python">def fetch_single_station_data(station_code: str, start_date: str, use_mock: bool = True) -&gt; pd.DataFrame:
    """Fetch all hydrometric data for a single station from mock API (default) or from the real endpoint.

    In production (cursor based pagination strategy):
    - Fetches max_per_page records per request
    - Continues until no new data or yesterday's date reached
    - Stop when: no data returned | last date &gt;= yesterday | page was not full
    - Handles network errors gracefully
    """
    if use_mock:
        data = generate_mock_api_data(station_code, start_date, num_days=7)
        page_df = pd.DataFrame(data)
        page_df["date_obs_elab"] = pd.to_datetime(
            page_df["date_obs_elab"], errors="coerce").dt.normalize()
        return page_df

    # Real implementation
    else:
        session = requests.Session()  # Reuse TCP connection across pages
        frames = []
        cursor = start_date

        while True:
            params = {
                "code_entite": station_code,
                "grandeur_hydro_elab": API_CONFIG.metric,
                "date_debut_obs_elab": cursor,
                "size": API_CONFIG.max_per_page,
            }

            try:
                response = session.get(
                    API_CONFIG.base_url,
                    params=params,
                    timeout=API_CONFIG.timeout_seconds  # Best practice, always set
                )
                response.raise_for_status()
            except requests.RequestException as e:  # Don't let one station kill the whole pipeline
                print(f"Error fetching data for station {station_code}: {e}")
                break

            data = response.json().get("data", [])
            if not data:  # Empty response: we've exhausted this station
                break

            page_df = pd.DataFrame(data)
            page_df["date_obs_elab"] = pd.to_datetime(
                page_df["date_obs_elab"], errors="coerce").dt.normalize()
            frames.append(page_df)

            last_page_date = page_df["date_obs_elab"].max()
            yesterday = date.today() - timedelta(days=1)

            # Prevent infinite loops
            if pd.isna(last_page_date) or last_page_date.date() &gt;= yesterday:
                break

            cursor = (last_page_date + pd.Timedelta(days=1)).strftime("%Y-%m-%d")

            if len(data) &lt; API_CONFIG.max_per_page:
                break

    if frames:
        return pd.concat(frames, ignore_index=True)
    return pd.DataFrame()
</code></pre>
<p><strong>How does the pagination loop actually work?</strong></p>
<p>Let's walk through it step by step. This is the densest bit of logic in the whole notebook.</p>
<ol>
<li><p>Send a request with <code>date_debut_obs_elab=cursor</code>: "give me records from this date on."</p>
</li>
<li><p>If the request fails outright (<code>requests.RequestException</code>), log it and <code>break</code>. <code>break</code> stops the loop right away and moves on. One station's network hiccup shouldn't kill the pipeline for every other station.</p>
</li>
<li><p>If the response has no data at all, you've caught up: <code>break</code>.</p>
</li>
<li><p>Otherwise, note the <em>latest</em> date seen on this page (<code>last_page_date</code>).</p>
</li>
<li><p>If that latest date is already <code>&gt;= yesterday</code>, you've caught up: <code>break</code>.</p>
</li>
<li><p>Otherwise, advance the cursor to <code>last_page_date + 1 day</code> and loop again for the next page.</p>
</li>
<li><p>As a safety net: if the page returned <em>fewer</em> records than <code>max_per_page</code>, that also means you've reached the end. The API wouldn't return a partial page unless it ran out of data, so: <code>break</code>.</p>
</li>
</ol>
<p>That last check (step 7) is a classic <strong>pagination termination heuristic</strong>. You don't always need a <code>next_page</code> token from the API. If a full page is <code>size=20000</code> and you get back only <code>4213</code> records, there's nothing left to fetch.</p>
<p><strong>Three specific, deliberate choices, called out:</strong></p>
<pre><code class="language-python">session = requests.Session()
</code></pre>
<p>A <a href="https://requests.readthedocs.io/en/latest/user/advanced/#session-objects"><code>Session</code></a> object reuses the underlying TCP connection across multiple requests to the same host. That avoids a fresh TCP/TLS handshake on every single page request. In short, it's faster for you and more polite to Hub'Eau's servers.</p>
<p><strong>Best practice:</strong> any time you call <code>requests.get()</code> more than once against the same host in a loop, reach for a <code>Session</code>.</p>
<pre><code class="language-python">except requests.RequestException as e:
</code></pre>
<p><code>RequestException</code> is the base class for <a href="https://requests.readthedocs.io/en/latest/api/#requests.RequestException">every exception</a> <code>requests</code> can raise: timeouts, connection errors, HTTP errors from <code>raise_for_status()</code> and more. Catching the base class here means <em>any</em> network hiccup gets handled the same forgiving way: log it, stop fetching this station, move on.</p>
<pre><code class="language-python">timeout=API_CONFIG.timeout_seconds
</code></pre>
<p>By default, <code>requests</code> calls <strong>never time out</strong>. Without an explicit timeout, a hung server can freeze your entire pipeline indefinitely. The <a href="https://requests.readthedocs.io/en/latest/user/advanced/#timeouts">requests advanced usage docs</a> states that requests to external servers should have a timeout attached.</p>
<p><strong>Best practice:</strong> wrap every external I/O call in <code>try/except</code>. Always fail gracefully. Log the error and let the pipeline recover or move on. This avoids one flaky request taking down an unattended weekly job.</p>
<p>Now the orchestrator, <code>fetch_all_data()</code>, is deliberately much simpler:</p>
<ol>
<li><p>Loop through all station codes.</p>
<ul>
<li><p>Fetch each station's data.</p>
</li>
<li><p>Keep the non-empty results.</p>
</li>
</ul>
</li>
<li><p>Combine them into one big DataFrame.</p>
</li>
</ol>
<pre><code class="language-python">def fetch_all_data(start_date: str, use_mock: bool = True) -&gt; pd.DataFrame:
    """Orchestrator: Fetch data for all configured stations."""
    frames = []

    for station_code in STATION_CONFIG.station_codes:
        print(f"Fetching data for station {station_code}...")
        df_station = fetch_single_station_data(station_code, start_date, use_mock=use_mock)

        if not df_station.empty:
            print(f"  Got {len(df_station)} records")
            frames.append(df_station)
        else:
            print(f"  (no data)")

    if frames:
        return pd.concat(frames, ignore_index=True)
    return pd.DataFrame()
</code></pre>
<p>That's it. A loop and a <code>pd.concat</code>. All the hard-won complexity lives one layer down, exactly where SRP says it should.</p>
<h2 id="heading-part-5-the-transform-step">Part 5: The Transform Step</h2>
<p>This is where raw, freshly-fetched data becomes something you can publish. Here's the design philosophy for this whole section: compose many small, <strong>pure functions</strong>. Each one takes a DataFrame in and returns a <em>new</em> DataFrame out without side effects. Avoid the one giant do-everything function trap.</p>
<pre><code class="language-text">    (EXTRACT)
    Raw API Data
    ↓
    (TRANSFORM)
1. Type parsing (datetime, numeric)
2. Column renaming (French → English)
3. Categorical mapping (validation status, quality)
4. Derived columns computation (flood alert flags)
5. Column reordering (logical grouping)
6. Sorting &amp; index reset
    ↓
    (LOAD)
    Publication-ready dataset
</code></pre>
<p>Why split this into six tiny steps instead of one big function? Each piece is testable and replaceable on its own. When something breaks at 3am on a scheduled run, every function is debuggable in isolation. You can pinpoint exactly which stage produced bad output. No need to pick apart one 200-line function.</p>
<h3 id="heading-type-parsing-and-graceful-coercion">Type Parsing and Graceful Coercion</h3>
<h4 id="heading-design-logic">Design logic</h4>
<p>Data from a CSV or a JSON API starts out, by default, as strings. Pandas needs real types to sort dates chronologically, do date arithmetic (<code>last_date + timedelta(days=1)</code>), or compare numeric values (<code>water_level &gt; 6000</code>). Type mismatches are a common error that break batch pipelines. One malformed row, like <code>"N/A"</code>, a truncated date, misaligned values, or stray characters and a strict parser throws an exception that kills the whole run.</p>
<p><strong>Best practice:</strong> both conversion functions below use <code>errors="coerce"</code>, so values pandas can't parse become <code>NaT</code> (Not a Time) or <code>NaN</code> (Not a Number) instead of raising an error. This is called <a href="https://stackoverflow.com/questions/36394814/what-is-the-significance-of-coerce-in-python-pandas">graceful coercion</a>.</p>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p><code>convert_to_date()</code> and <code>convert_to_numeric()</code> are small helper functions that make sure certain columns have the right type.</p>
<ul>
<li><p>Both start with <code>df.copy()</code> so they don’t change the original DataFrame.</p>
</li>
<li><p><code>errors="coerce"</code> means bad values become missing values instead of causing a crash.</p>
</li>
</ul>
<p>For <code>convert_to_date()</code>:</p>
<ol>
<li><p><code>df.copy()</code> makes a separate copy, so the original table stays unchanged</p>
</li>
<li><p>The loop goes through each column name in <code>columns</code></p>
<ul>
<li><p><code>if col in df.columns</code> checks that the column actually exists before trying to convert it</p>
</li>
<li><p><code>pd.to_datetime(...)</code> turns text like <code>"2026-07-12"</code> into real pandas date/time values</p>
</li>
<li><p><code>errors="coerce"</code> means invalid values become <code>NaT</code> (missing date) instead of raising an error</p>
</li>
<li><p><code>.dt.normalize()</code> removes the time part and keeps only the date at midnight</p>
</li>
</ul>
</li>
</ol>
<p>For <code>convert_to_numeric()</code>:</p>
<ol>
<li><p><code>df.copy()</code> is used here as well.</p>
</li>
<li><p>The loop goes through each column name in <code>columns</code></p>
<ul>
<li><p><code>if col in df.columns</code> checks that the column actually exists before trying to convert it</p>
</li>
<li><p>The loop calls <code>pd.to_numeric()</code> turns text like <code>"12.5"</code> into numbers</p>
</li>
<li><p><code>errors="coerce"</code> turns bad values into <code>NaN</code> instead of crashing</p>
</li>
</ul>
</li>
</ol>
<pre><code class="language-python">def convert_to_date(df: pd.DataFrame, columns: List[str]) -&gt; pd.DataFrame:
    """Convert specified columns to pandas datetime type."""
    df = df.copy()  # Never modify the original!
    for col in columns:
        if col in df.columns:
            df[col] = pd.to_datetime(df[col], errors="coerce").dt.normalize()
    return df


def convert_to_numeric(df: pd.DataFrame, columns: List[str]) -&gt; pd.DataFrame:
    """Convert specified columns to numeric (float) type."""
    df = df.copy()
    for col in columns:
        if col in df.columns:
            df[col] = pd.to_numeric(df[col], errors="coerce")
    return df
</code></pre>
<p>Why is this useful?</p>
<ul>
<li><p>You can aggregate the data: group it, summarize it and so on...</p>
</li>
<li><p>Dates become sortable and filterable as real dates.</p>
</li>
<li><p>Numbers work correctly in calculations like averages, sums, or comparisons.</p>
</li>
<li><p>It prevents bugs caused by mixed types, like <code>"12"</code> and <code>12</code>.</p>
</li>
</ul>
<p>Both <code>pd.to_datetime</code> and <code>pd.to_numeric</code> are official pandas functions with an <code>errors</code> parameter. By default, that parameter is set to <code>"raise"</code>, which throws on bad input. The other options are <code>"coerce"</code> (replace with null) or <code>"ignore"</code> (leave untouched). See the <a href="https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html">pandas <code>to_datetime</code> docs</a> and <a href="https://pandas.pydata.org/docs/reference/api/pandas.to_numeric.html"><code>to_numeric</code> docs</a> for the full parameter list.</p>
<p>Test it against an example of messy input:</p>
<pre><code class="language-python">messy_df = pd.DataFrame({
    "date_obs_elab": ["2026-01-15", "2026-01-16", "not a date", None],
    "resultat_obs_elab": [5800.0, "5900", "N/A", None],
})

type_safe_df = convert_to_date(messy_df, ["date_obs_elab"])
type_safe_df = convert_to_numeric(type_safe_df, ["resultat_obs_elab"])

# "not a date"  → NaT
# "N/A"         → NaN
# Pipeline continues safely — nothing crashed.
</code></pre>
<p><strong>Best practice:</strong> don't let one bad row kill an entire run. Coerce bad data to nulls rather than raising exceptions. Flag or log nulls separately if you need to investigate data quality later.</p>
<p>This is a deliberate trade-off. A choice between <em>availability</em> (the pipeline keeps running) over <em>strictness</em> (catching every bad row immediately). That's usually the right call for a scheduled, unattended job.</p>
<h4 id="heading-dfcopy-immutability-by-convention"><code>df.copy()</code>: immutability by convention</h4>
<p>Look again at the top of both functions above: <code>df = df.copy()</code>. This single line appears at the start of <strong>every transform function</strong> in the pipeline, and that's not an accident.</p>
<p>Python DataFrames are mutable objects, passed by reference. If a function modifies <code>df</code> in place without copying first, the caller's original DataFrame changes too. That's a classic <a href="https://en.wikipedia.org/wiki/Side_effect_(computer_science)">side effect</a> and it can produce genuinely confusing bugs. Call <code>.copy()</code> first means each function's output is a brand-new object. The input the caller passed in stays <em>guaranteed untouched</em>.</p>
<p><strong>Best practice:</strong> treat DataFrames as <strong>immutable inputs</strong>. Return a new DataFrame rather than modify it one in place. Even if it costs a small amount of memory or CPU, the debugging win is almost always worth it for a pipeline that isn't operating at extreme scale.</p>
<p>Finally, one auto-detecting convenience function wraps these two low-level functions. It scans text columns, guesses whether they contain dates or numbers, and calls the matching parsing function above.</p>
<p><code>auto_convert_columns()</code> tries to <strong>guess which columns are dates or numbers</strong> and then fixes these types automatically.</p>
<p>How it works:</p>
<p>It starts with two empty lists:</p>
<ul>
<li><p><code>datetime_cols</code> for date columns</p>
</li>
<li><p><code>numeric_cols</code> for number columns</p>
</li>
</ul>
<p>It goes through each column in the DataFrame. If a column is already a real datetime or numeric type, it skips it. If the column is text-like (<code>object</code> or <code>string</code>), it looks at up to 10 non-empty sample values.</p>
<p>It first tries to read those values as dates:</p>
<ul>
<li>if that works, the column is added to <code>datetime_cols</code></li>
</ul>
<p>If not, it tries to read them as numbers:</p>
<ul>
<li>if that works, the column is added to <code>numeric_cols</code></li>
</ul>
<p>At the end, it converts all date columns with <code>convert_to_date()</code>. Then it converts all numeric columns with <code>convert_to_numeric()</code></p>
<pre><code class="language-python">def auto_convert_columns(df: pd.DataFrame) -&gt; pd.DataFrame:
    """Auto-detect and convert datetime and numeric columns to the correct type."""
    datetime_cols = []
    numeric_cols = []

    for col in df.columns:
        if pd.api.types.is_datetime64_any_dtype(df[col]):
            continue
        if pd.api.types.is_numeric_dtype(df[col]):
            continue

        if pd.api.types.is_object_dtype(df[col]) or pd.api.types.is_string_dtype(df[col]):
            sample = df[col].dropna().head(10)
            if len(sample) == 0:
                continue

            try:
                pd.to_datetime(sample, errors='raise', format='mixed')
                datetime_cols.append(col)
                continue
            except (ValueError, TypeError):
                pass

            try:
                pd.to_numeric(sample, errors='raise')
                numeric_cols.append(col)
                continue
            except (ValueError, TypeError):
                pass

    df = convert_to_date(df, datetime_cols)
    df = convert_to_numeric(df, numeric_cols)
    return df
</code></pre>
<p>Notice the inner <code>try/except</code> blocks here use <code>errors='raise'</code>. That's the <em>opposite</em> of the coercion strategy above, but it only runs against a small <code>.head(10)</code> <strong>sample</strong> of each column.</p>
<p>This is a type-<em>sniffing</em> step, not the final conversion. It tests <em>"does this column look like dates, or numbers, or neither?"</em> on a cheap sample. Then, it hands off the actual, forgiving conversion of the <em>whole</em> column to <code>convert_to_date()</code> or <code>convert_to_numeric()</code>. Two different <code>errors</code> strategies, two different jobs.</p>
<h3 id="heading-schema-translation-with-bidirectional-mappings">Schema Translation with Bidirectional mappings</h3>
<h4 id="heading-design-logic">Design logic</h4>
<p>The Hub'Eau API returns French column names and French categorical values, like <code>code_station</code> or <code>"Donnée validée"</code>. A dataset meant for an international audience should ship in English. If you rename columns inline, wherever it's convenient, the mapping between French and English ends up scattered across the codebase. Then you'd have no way to reverse it if you ever needed to.</p>
<p><strong>The fix:</strong> define <strong>one authoritative mapping</strong> at the top of the module, and derive everything else from it.</p>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p><code>API_TO_EN</code> maps API field names to English translations.</p>
<pre><code class="language-python"># Primary mapping: French API columns to English column names
API_TO_EN = {
    "code_site": "location_code",
    "code_station": "station_code",
    "date_obs_elab": "record_date",
    "resultat_obs_elab": "water_level_mm",
    "date_prod": "data_production_date",
    "code_statut": "validation_status_code",
    "libelle_statut": "validation_status",
    "code_methode": "production_method_code",
    "libelle_methode": "production_method",
    "code_qualification": "quality_code",
    "libelle_qualification": "quality_assessment",
    "longitude": "longitude",
    "latitude": "latitude",
    "grandeur_hydro_elab": "hubeau_elab_code",
}

# Reverse mapping: English to French (computed automatically)
EN_TO_API = {v: k for k, v in API_TO_EN.items()}
</code></pre>
<p>The second line (<code>EN_TO_API</code>) is just a shortcut: it swaps each key and value from <code>API_TO_EN</code>. A <a href="https://docs.python.org/3/tutorial/datastructures.html#dictionaries">dict comprehension</a> builds it. <strong>Here's the important design point:</strong> <code>EN_TO_API</code> isn't hand-maintained, it's <em>derived</em>. If you add, remove, or rename an entry in <code>API_TO_EN</code>, <code>EN_TO_API</code> updates automatically the next time the module runs.</p>
<p>There's <em>exactly one place</em> in the entire codebase where a schema change needs to happen.</p>
<p>Categorical <em>values</em> (not just column names) get the same treatment:</p>
<pre><code class="language-python">CATEGORICAL_MAPPINGS = {
    "validation_status": {
        "Donnée validée": "validated",
        "Donnée brute": "raw",
        "Donnée pré-validée": "pre-validated",
    },
    "quality_assessment": {
        "Bonne": "good",
        "Non qualifiée": "unqualified",
        "Douteuse": "dubious",
    },
    "production_method": {
        "Calculée": "calculated",
        "Mesurée": "measured",
        "Expertisée": "expert-reviewed",
    },
}
</code></pre>
<p>The functions below apply these mappings, in order, to standardize column names and category values.</p>
<p><code>rename_to_english()</code>:</p>
<ol>
<li><p>If the DataFrame is empty, it returns a copy right away.</p>
</li>
<li><p>It builds a list of columns that are renamed from API names to English names.</p>
<ul>
<li><p>It only renames a column if:</p>
<ul>
<li><p>the old name exists, and</p>
</li>
<li><p>the new name does not already exist</p>
</li>
</ul>
</li>
</ul>
</li>
<li><p>Then it returns the renamed DataFrame.</p>
</li>
</ol>
<p><code>rename_to_api_schema()</code>:</p>
<ol>
<li><p>Also returns a copy for empty data.</p>
</li>
<li><p>Does the reverse: English names back to API names.</p>
</li>
<li><p>Returns the DataFrame.</p>
</li>
</ol>
<p>(Useful if you need to send data back in the API’s original format).</p>
<p><code>apply_categorical_mappings()</code>:</p>
<ol>
<li><p>Makes a copy so the original DataFrame is not changed.</p>
</li>
<li><p>For each column in <code>CATEGORICAL_MAPPINGS</code>, it replaces values using the mapping.</p>
<ul>
<li><p>Example: <code>"Donnée validée"</code> becomes <code>"validated"</code>.</p>
</li>
<li><p><code>.fillna(df[col_name])</code> keeps the original value if a value isn’t found in the mapping.</p>
</li>
</ul>
</li>
<li><p>Returns the DataFrame.</p>
</li>
</ol>
<pre><code class="language-python">def rename_to_english(df: pd.DataFrame) -&gt; pd.DataFrame:
    """Rename API column names to English schema names."""
    if df.empty:
        return df.copy()

    columns_to_rename = {}
    for src, dst in API_TO_EN.items():
        if src in df.columns and dst not in df.columns:
            columns_to_rename[src] = dst

    return df.rename(columns=columns_to_rename)

def rename_to_api_schema(df: pd.DataFrame) -&gt; pd.DataFrame:
    """Rename English column names back to API schema names (defensive/reverse operation)."""
    if df.empty:
        return df.copy()

    columns_to_rename = {k: v for k, v in EN_TO_API.items() if k in df.columns}
    return df.rename(columns=columns_to_rename)
</code></pre>
<pre><code class="language-python">def apply_categorical_mappings(df: pd.DataFrame) -&gt; pd.DataFrame:
    df = df.copy()
    for col_name, mapping in CATEGORICAL_MAPPINGS.items():
        if col_name in df.columns:
            df[col_name] = df[col_name].map(mapping).fillna(df[col_name])
    return df
</code></pre>
<p>Two defensive habits are worth a note here:</p>
<p><strong>Robustness to partial inputs:</strong> Both rename functions only rename columns that actually exist in the input (<code>if src in df.columns</code>).</p>
<p>A rename function that assumes every mapped column is always present will crash the moment it's called on a partial or differently-shaped DataFrame.</p>
<p><code>.map(mapping).fillna(df[col_name])</code>. <a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.map.html"><code>Series.map</code></a> replaces every value it finds in the mapping dict and turns any value <em>not</em> found in the dict into <code>NaN</code>. Chain <code>.fillna(df[col_name])</code> right after restores the <em>original</em> value wherever the mapping didn't apply. That way, unexpected categorical values pass through unchanged instead of silently becoming null. It's a subtle but important robustness choice.</p>
<p>See this exact <code>.map()</code>-then-<code>.fillna()</code> idiom discussed on <a href="https://stackoverflow.com/questions/19798153/difference-between-map-applymap-and-apply-methods-in-pandas">Stack Overflow: pandas map vs apply performance</a>.</p>
<p>A quick round-trip test proves the bidirectional mapping actually works:</p>
<pre><code class="language-python">sample_api_df = pd.DataFrame({"code_station": ["F700000109"], ...})
renamed_df = rename_to_english(sample_api_df)          # French → English
reversed_df = rename_to_api_schema(renamed_df)          # English → French
# reversed_df.columns.tolist() == sample_api_df.columns.tolist()  → True
</code></pre>
<h3 id="heading-compute-flood-alerts">Compute Flood Alerts</h3>
<p><code>add_derived_columns()</code> is a one-liner. It just computes <code>flood_alert</code>. If the water level is greater than the flood threshold set in the <code>Config</code>, the result is <code>True</code>. Otherwise, the result is <code>False</code>.</p>
<pre><code class="language-python">def add_derived_columns(df: pd.DataFrame) -&gt; pd.DataFrame:
    """Add a computed columns based on raw data.
    """
    df = df.copy()

    if "water_level_mm" in df.columns:
        df["flood_alert"] = df["water_level_mm"] &gt; STATION_CONFIG.flood_threshold_mm

    return df
</code></pre>
<h3 id="heading-column-ordering">Column Ordering</h3>
<p>Here's a small but user-facing detail: define a preferred column order once, as data, and reuse it everywhere. <code>COLUMN_ORDER</code> holds that preferred column sequence.</p>
<p><code>order_columns()</code> rearranges a DataFrame so those columns come first, while any extra columns stay at the end.</p>
<pre><code class="language-python">COLUMN_ORDER = [
    # Primary identifiers &amp; measurements
    "station_code", "record_date", "water_level_mm", "flood_alert",
    # Metadata about the observation
    "hubeau_elab_code", "data_production_date",
    "validation_status_code", "validation_status",
    "production_method_code", "production_method",
    "quality_code", "quality_assessment",
    # Geographic info (less important)
    "location_code", "longitude", "latitude",
]

def order_columns(df: pd.DataFrame) -&gt; pd.DataFrame:
    """Reorder columns to preferred order."""
    present_cols = [c for c in COLUMN_ORDER if c in df.columns]
    other_cols = [c for c in df.columns if c not in present_cols]
    return df[present_cols + other_cols]
</code></pre>
<p><code>other_cols</code> acts as a safety net: any column not explicitly listed in <code>COLUMN_ORDER</code> still gets included at the end. They're not silently dropped.</p>
<p>Rules, in order of priority:</p>
<ul>
<li><p>Keep identifiers and key fields up front.</p>
</li>
<li><p>Put the most important, most frequently used, and most stable columns first.</p>
</li>
<li><p>Group related fields together, so the table reads naturally.</p>
</li>
<li><p>Push optional or rarely-used fields to the end.</p>
</li>
</ul>
<h3 id="heading-deduplication">Deduplication</h3>
<h4 id="heading-design-logic">Design logic</h4>
<p>Remove duplicates as you go. In an incremental pipeline, date ranges and records can overlap:</p>
<ul>
<li><p>The same day might get re-fetched, because API data arrives late or gets finalized later.</p>
</li>
<li><p>The same observation might appear on two different pages of a paginated response.</p>
</li>
</ul>
<p>Without deduplication, these scenarios let duplicate rows pile up in the dataset over time. This is also, recall from Part 1, exactly what makes the pipeline <strong>idempotent</strong>: run it once or run it five times and the resulting dataset stays identical.</p>
<p>The fix requires defining what makes a record <strong>unique</strong>. Define a composite key:</p>
<pre><code class="language-text">key = (station_code, observation_date, water_level_value)
</code></pre>
<p>Why this specific combination? Physically, one sensor (<code>station_code</code>) reports one day's (<code>observation_date</code>) daily-maximum reading (<code>water_level_mm</code>), and that reading should be unique.</p>
<ul>
<li><p>Records from different stations obviously aren't duplicates of each other.</p>
</li>
<li><p>Two readings on different days aren't duplicates.</p>
</li>
<li><p>A subtler point: if the <em>same</em> station reports the <em>same</em> day but with a <em>different</em> value, that counts as a distinct observation, for example a corrected or revised measurement, not a duplicate to silently discard.</p>
</li>
</ul>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p><code>create_dedup_key()</code> builds a unique text ID for each row by joining three pieces together with an underscore:</p>
<ul>
<li><p>station code</p>
</li>
<li><p>date, as YYYY-MM-DD</p>
</li>
<li><p>water level value</p>
</li>
</ul>
<p>Example key:</p>
<pre><code class="language-text">"F700000109_2024-01-15_5800.0"
</code></pre>
<pre><code class="language-python">def create_dedup_key(df: pd.DataFrame) -&gt; pd.Series:
    """Create unique deduplication key from station, day, and water level value.

    Key format: "station_code_YYYY-MM-DD_value"
    Example: "F700000109_2024-01-15_5800.0"
    """
    parts = []

    if "code_station" in df.columns:
        parts.append(df["code_station"].astype(str))

    if "date_obs_elab" in df.columns:
        parts.append(df["date_obs_elab"].dt.strftime("%Y-%m-%d"))

    if "resultat_obs_elab" in df.columns:
        parts.append(df["resultat_obs_elab"].astype(str))

    if not parts:
        return pd.Series(index=df.index, dtype="object")

    return pd.Series(
        ["_".join(row) for row in zip(*parts)],
        index=df.index
    )
</code></pre>
<p><strong>How the key-building works:</strong> <code>parts</code> ends up as a list of <a href="https://www.geeksforgeeks.org/pandas/python-pandas-series/">Series</a>, one per key component (station, date, value), each the same length as the DataFrame.</p>
<p>Read the last line from the inside out:</p>
<pre><code class="language-python">    return pd.Series(["_".join(row) for row in zip(*parts)], index=df.index)
</code></pre>
<p><code>zip(*parts)</code> transposes that list of columns into row-wise tuples. It yields <code>(station_1, date_1, value_1)</code>, then <code>(station_2, date_2, value_2)</code>, and so on. The <a href="https://www.geeksforgeeks.org/python/python-list-comprehension/">list comprehension</a> then joins each row-tuple with underscores into one string key per row.</p>
<p>This <em>"list of columns → zip → row tuples"</em> idiom is a common and efficient way to combine several Series into one derived Series, without writing <code>.apply(lambda row: ..., axis=1)</code>. Row-wise <code>.apply</code> is notoriously <a href="https://stackoverflow.com/questions/54432583/when-should-i-not-want-to-use-pandas-apply-in-my-code">slow in pandas</a> compared to <a href="https://www.datacamp.com/es/tutorial/pandas-iterate-over-rows">vectorized</a> string operations.</p>
<p>The actual deduplication happens in <code>remove_duplicates()</code>. It removes rows from <em>"new"</em> (freshly fetched data) that already exist in <em>"existing"</em> (historic data).</p>
<p>Step by step:</p>
<ol>
<li><p>If one table is empty, it just returns <code>new</code>.</p>
</li>
<li><p>It enforces types in both DataFrames with <code>auto_convert_columns()</code> first, so dates and numbers compare correctly.</p>
</li>
<li><p>It creates a deduplication key for each row in both tables with <code>create_dedup_key()</code>.</p>
</li>
<li><p>It checks which keys from the fetched data don't appear in the existing historic dataset, and keeps only truly new rows.</p>
</li>
<li><p>It returns the filtered result, keeping the original columns from <code>new</code>.</p>
</li>
</ol>
<pre><code class="language-python">def remove_duplicates(existing: pd.DataFrame, new: pd.DataFrame) -&gt; pd.DataFrame:
    """Remove rows from 'new' that already exist in 'existing'."""
    # Short-circuit: if either is empty, no work to do
    if existing.empty or new.empty:
        return new.copy()

    # Parse types on both sides for a fair comparison
    existing_std = auto_convert_columns(existing)
    new_std = auto_convert_columns(new)

    # Build the keys
    existing_keys = set(create_dedup_key(existing_std).dropna())
    new_keys = create_dedup_key(new_std)

    # Boolean mask: True where the new row is genuinely new
    mask = ~new_keys.isin(existing_keys)

    # Index back into the ORIGINAL (non-standardized) 'new' to preserve all columns
    result = new.iloc[new_keys[mask].index].copy()
    return result
</code></pre>
<p>Two performance and robustness details are worth to note:</p>
<p>First, <code>existing_keys</code> <strong>is a</strong> <code>set</code><strong>, not a</strong> <code>list</code><strong>.</strong> Testing membership (<code>in</code> / <code>.isin()</code>) against a Python <code>set</code> is <a href="https://robbell.io/2009/06/a-beginners-guide-to-big-o-notation"><strong>O(1)</strong></a> on average, because it uses a fast hash lookup to answer <em>is this item already here?</em> directly. Testing against a <code>list</code> is <strong>O(n)</strong>: it has to scan item by item and it gets slower as the existing dataset grows.</p>
<p>For a dataset with tens of thousands of rows, checked on every single pipeline run, that difference matters and that's a textbook example of choosing the right data structure for the job. See the general discussion of <a href="https://stackoverflow.com/questions/513882/python-list-vs-dict-for-look-up-table">list vs. set lookup performance in Python</a> and this <a href="https://thelinuxcode.com/pandas-value-list/">guidance</a> on <a href="https://stackoverflow.com/questions/61515457/fastest-way-to-filter-a-pandas-dataframe-using-a-list"><code>.isin()</code> performance for filtering</a>.</p>
<p><strong>Simple rule:</strong> use a <code>set</code> when you care about fast membership checks. Use a <code>list</code> when you care about order or duplicates.</p>
<p>Second, <code>new.iloc[new_keys[mask].index]</code> <strong>indexes back into the <em>original</em>, non-validated</strong> <code>new</code> <strong>DataFrame</strong>, not <code>new_std</code>.</p>
<p>Why? <code>auto_convert_columns()</code> only ran to get <em>consistent types for comparison</em>. The caller still wants the <em>original</em> raw values and schema back for everything that survives deduplication.</p>
<p>Don't let a side-computation <em>accidentally</em> become your source of truth. Always modify a copy only to decide what to keep or remove. Once you've made that decision, apply it to the original data. That way you preserve the real source values for later steps.</p>
<p>In short:</p>
<ol>
<li><p>First modifications are only for comparison.</p>
</li>
<li><p>Use that comparison to filter the original data.</p>
</li>
<li><p>Return the original rows unchanged, so later stages can process them.</p>
</li>
</ol>
<p><strong>Best practice:</strong> keep your deduplication key simple and stable (immutable) and always short-circuit (with <code>if existing.empty or new.empty: return new.copy()</code>) before doing any heavier work. This cheap guard clause skips the expensive deduplication logic when there's no data to process.</p>
<h3 id="heading-put-it-all-together-in-postprocess">Put it All Together in <code>postprocess()</code></h3>
<p>All the pieces above are small and testable on their own. The last step of <strong>Transform</strong> glues them together, <em>in order</em>, into the single pipeline function below:</p>
<pre><code class="language-python">def postprocess(df: pd.DataFrame) -&gt; pd.DataFrame:
    """Apply all post-processing transformations."""
    if df.empty:
        return df

    df = df.copy()

    print("  1. Converting types...")
    df = auto_convert_columns(df)

    print("  2. Renaming columns (French → English)...")
    df = rename_to_english(df)

    print("  3. Mapping categorical values...")
    df = apply_categorical_mappings(df)

    print("  4. Adding derived columns...")
    df = add_derived_columns(df)

    print("  5. Reordering columns...")
    df = order_columns(df)

    print("  6. Sorting and resetting index...")
    df = df.sort_values(["record_date", "station_code"]).reset_index(drop=True)

    return df
</code></pre>
<p>Notice the shape of this function: it's basically a linear script with six numbered, printed steps. Each one calls a previously-defined pure function. There's no new <em>logic</em> here, only <em>sequencing</em>. That's intentional.</p>
<p><strong>Best practice:</strong> log progress clearly at each stage. When a scheduled job fails at 3am, a clear step-by-step log is the difference between a two-minute diagnosis and an hour of guessing. A good option is to use a <a href="https://www.geeksforgeeks.org/python/logging-in-python/">logger</a>. Here, this tutorial sticks with classic console printing (<code>print(f" 1. Converting types...")</code>).</p>
<h2 id="heading-part-6-the-load-step">Part 6: The Load Step</h2>
<h3 id="heading-design-logic">Design Logic</h3>
<p><strong>Load</strong> is the final stage: take the processed data and move it to its destination. Typical destinations include data warehouses, data lakes, and databases.</p>
<p>Before publishing, the pipeline prepares two things: the output folder itself, and a metadata file that describes the dataset.</p>
<h4 id="heading-code-level-walkthrough">Code-level walkthrough</h4>
<p><code>create_output_dir()</code> makes sure an output folder exists, and creates one if it doesn't:</p>
<ul>
<li><p>If <code>use_mock</code> is <code>True</code>, it uses <code>KAGGLE_CONFIG.mock_output_dir</code>, the directory for fake data.</p>
</li>
<li><p>Otherwise, it uses the real output directory set in <code>KAGGLE_CONFIG.output_dir</code>.</p>
</li>
</ul>
<pre><code class="language-python">def create_output_dir(use_mock: bool = False) -&gt; None:
    """Create output directory (and parents) if it does not already exist."""
    output_dir = KAGGLE_CONFIG.mock_output_dir if use_mock else KAGGLE_CONFIG.output_dir
    output_dir.mkdir(parents=True, exist_ok=True)
    # exist_ok=True: idempotent, safe to call multiple times
</code></pre>
<p><code>exist_ok=True</code> is a small but important detail. Without it, <a href="https://docs.python.org/3/library/pathlib.html#pathlib.Path.mkdir"><code>Path.mkdir()</code></a> raises <code>FileExistsError</code> if the directory already exists, which it will on every run after the first.</p>
<p>Setting <code>exist_ok=True</code> makes directory creation <strong>idempotent</strong>: calling it 100 times has the same effect as calling it once. On the filesystem, this is the "safe to re-run" idea behind the deduplication logic from Part 5.</p>
<p>The Kaggle API follows the <a href="https://frictionlessdata.io/specs/data-package/">Data Package specification</a>. It <a href="https://github.com/Kaggle/kaggle-cli/wiki/Dataset-Metadata/bc2684f533cd40afae28210d8f6e62b88d793d62">requires</a> a descriptive <code>dataset-metadata.json</code> file alongside the CSV. This matters for the search and discoverability of the <a href="https://www.kaggle.com/datasets/grimespoint/paris-flood-dataset">Paris flood dataset</a>:</p>
<pre><code class="language-python">def create_metadata(df: pd.DataFrame, config: KaggleConfig) -&gt; Dict:
    """Generate Kaggle dataset metadata from DataFrame and config."""
    if df.empty or "record_date" not in df.columns:
        first_date = "unknown"
        last_date = "unknown"
    else:
        first_date = df["record_date"].min().strftime("%Y-%m-%d")
        last_date = df["record_date"].max().strftime("%Y-%m-%d")

    return {
        "title": config.title,
        "id": config.dataset_slug,
        "licenses": [{"name": config.license_name}],
        "keywords": config.keywords,
        "temporalCoverage": {"startDate": first_date, "endDate": last_date},
        "geospatialCoverage": config.geospatial_coverage,
        "updateFrequency": config.update_frequency,
    }
</code></pre>
<p>Metadata comes from the <code>KaggleConfig</code> dataclass. Note that <code>temporalCoverage</code> is computed <em>from the data itself</em> (<code>df["record_date"].min()</code>/<code>.max()</code>) rather than hardcoded. Every time the pipeline runs, the metadata's date range automatically reflects reality, with zero manual bookkeeping.</p>
<p>Finally, <code>publish_to_kaggle()</code> uploads the updated dataset to Kaggle. It calls out to the <a href="https://github.com/Kaggle/kaggle-api">Kaggle CLI</a> as an external command.</p>
<p>What it does:</p>
<ol>
<li><p>Gets the current time and turns it into a text label like <code>Weekly update: 2026-07-13 14:30:00</code></p>
</li>
<li><p>Builds a Kaggle command that says:</p>
<ul>
<li><p>publish a new dataset version</p>
</li>
<li><p>use files from <code>KAGGLE_CONFIG.output_dir</code></p>
</li>
<li><p>attach the message</p>
</li>
<li><p>zip the directory contents</p>
</li>
</ul>
</li>
<li><p>Prints the command so you can see what will run</p>
</li>
<li><p>Runs the command with <code>subprocess.run(...)</code></p>
</li>
</ol>
<p>If the upload works:</p>
<ul>
<li>it prints <code>Successfully published to Kaggle.</code></li>
</ul>
<p>If it fails:</p>
<ul>
<li>it prints the error and then raises the error again so the failure isn't hidden.</li>
</ul>
<pre><code class="language-python">def publish_to_kaggle() -&gt; None:
    """Publish the updated dataset to Kaggle using the Kaggle CLI."""
    timestamp = pd.Timestamp.now().strftime("%Y-%m-%d %H:%M:%S")
    message = f"Weekly update: {timestamp}"

    cmd = [
        "kaggle", "datasets", "version",
        "-p", str(KAGGLE_CONFIG.output_dir),
        "-m", message,
        "--dir-mode", "zip",
    ]

    print("Publishing to Kaggle...")
    print("Command:", " ".join(cmd))

    try:
        subprocess.run(cmd, check=True)
        print("Successfully published to Kaggle.")
    except subprocess.CalledProcessError as e:
        print(f"Error publishing to Kaggle: {e}")
        raise
</code></pre>
<p><code>publish_to_kaggle</code> calls the Kaggle CLI (Command Line Interface) through Python's <a href="https://docs.python.org/3/library/subprocess.html"><code>subprocess</code></a> module. This is the standard way to run an external command-line tool from a Python pipeline.</p>
<p>Two choices here are worth adopting as general habits, not just for Kaggle:</p>
<ul>
<li><p><code>cmd</code> <strong>is built as a</strong> <code>list</code><strong>, never as a concatenated string.</strong> Pass a list of arguments to <code>subprocess.run</code> skips invoking a shell entirely. That sidesteps <a href="https://portswigger.net/web-security/os-command-injection">shell-injection</a> risks and correctly handles arguments containing spaces or special characters. See the <a href="https://docs.python.org/3/library/subprocess.html#security-considerations">official <code>subprocess</code> security considerations</a>.</p>
</li>
<li><p><code>check=True</code> means that if the Kaggle CLI fails, by exiting with a non-zero status, <code>subprocess.run</code> raises <code>CalledProcessError</code> instead of silently returning. Without <code>check=True</code>, a failed publish would look exactly like a successful one to the rest of the code.</p>
</li>
</ul>
<h2 id="heading-part-7-assemble-the-full-pipeline">Part 7: Assemble the Full Pipeline</h2>
<h3 id="heading-the-global-rehearsal-mock-mode">The Global Rehearsal (Mock Mode)</h3>
<p>The whole pipeline is built and tested in isolation. It's time to wire the blocks into one linear function. First, run it entirely against mock data. Nothing gets published this way. For Demonstration purposes, a CSV filled with artificial data will land in a local <code>mock_output</code> folder.</p>
<pre><code class="language-python">def run_etl_pipeline(use_mock: bool = True) -&gt; pd.DataFrame:
    """Run the complete ETL pipeline: Extract → Transform → Load to CSV."""

    # STEP 1: LOAD EXISTING DATA
    if use_mock:
        loaded_df = pd.DataFrame(mock_api_response)
        loaded_df['date_obs_elab'] = pd.to_datetime(loaded_df['date_obs_elab'])
        loaded_df = rename_to_english(loaded_df)
    else:
        loaded_df = load_csv(KAGGLE_CONFIG.input_csv)

    # STEP 2: DETERMINE UPDATE RANGE
    should_update, start_date = determine_update_range(loaded_df)
    if not should_update:
        return loaded_df   # already current: nothing more to do

    # STEP 3: EXTRACT (FETCH DATA)
    fetched_data = fetch_all_data(start_date, use_mock=use_mock)

    # STEP 4: TRANSFORM I - DEDUPLICATE
    deduped_fetched_data = remove_duplicates(loaded_df, fetched_data)

    # STEP 5: TRANSFORM II - COMBINE (MERGE WITH EXISTING)
    new_df_english = rename_to_english(deduped_fetched_data)
    merged_historical_and_new = pd.concat([loaded_df, new_df_english], ignore_index=True)

    # STEP 6: TRANSFORM III - POST-PROCESS
    processed_records = postprocess(merged_historical_and_new)

    # STEP 7: EXPORT (LOAD TO CSV)
    create_output_dir(use_mock=use_mock)
    output_path = KAGGLE_CONFIG.mock_output_filename if use_mock else KAGGLE_CONFIG.output_csv_path
    processed_records.to_csv(output_path, index=False, sep=",")

    return processed_records
</code></pre>
<p>Every one of the seven steps above corresponds to a function you already built, and tested earlier in this tutorial. <strong>To Assemble them is almost mechanical.</strong> That's always the payoff of composing small, single-responsibility functions:</p>
<pre><code class="language-python">loaded_df                 = load_csv(...)                          # 1. load existing data
should_update, start_date = determine_update_range(...)            # 2. check what's needed
fetched_data               = fetch_all_data(start_date)             # 3. fetch new records
deduped_fetched_data       = remove_duplicates(existing, new_raw)   # 4. deduplicate
merged_historical_and_new  = pd.concat([existing, new_clean])       # 5. merge
processed_records          = postprocess(merged_historical_and_new) # 6. postprocess
processed_records.to_csv(...)                                       # 7. save
write_metadata() + publish_to_kaggle()                               # 8. publish
</code></pre>
<p>Each line's intent is obvious just from reading it left to right. That is exactly the point of good decomposition.</p>
<h3 id="heading-main-pipeline-orchestration"><code>main()</code>: Pipeline Orchestration</h3>
<p><code>main()</code> is the <a href="https://docs.python.org/en/3/library/__main__.html">conductor</a>. It wires every previously-built component together in the right order, just like <code>postprocess()</code> did one level down.</p>
<pre><code class="language-python">def main() -&gt; None:
    """Execute the full Paris Flood Monitoring ETL pipeline.

    EXTRACT
    1. Load the existing dataset from the Kaggle input mount
    2. Determine whether an update is needed (and from what date)
    3. Fetch new data from the Hub'Eau API

    TRANSFORM
    4. Deduplicate against the existing dataset
    5. Combine and post-process (type parsing, translation, derived cols...)

    LOAD
    6. Write the updated CSV and metadata file
    7. Publish the new dataset version to Kaggle

    Exits early if the dataset is already up to date.
    """
    final_dataset = run_etl_pipeline(use_mock=False)
    final_dataset.to_csv(KAGGLE_CONFIG.output_csv_path, index=False)
    write_metadata(final_dataset)

    print("\n[FINAL STEP] Publishing to Kaggle...")
    publish_to_kaggle()

    print("Running post-run validation")
    validate_and_analyze(final_dataset)


if __name__ == "__main__":
    main()
</code></pre>
<h3 id="heading-the-if-name-main-guard">The <code>if __name__ == "__main__":</code> Guard</h3>
<pre><code class="language-python">if __name__ == "__main__":
    main()
</code></pre>
<p>This is one of the most common idioms in Python, and it's worth understanding exactly what it does. Per the <a href="https://docs.python.org/3/library/__main__.html">official Python documentation</a> on <code>__main__</code>:</p>
<ul>
<li><p>Run the file directly (<code>python script.py</code>)</p>
<ul>
<li><p>→ the special variable <code>__name__</code> is set to <code>"__main__"</code></p>
</li>
<li><p>→ the condition is <code>True</code></p>
</li>
<li><p>→ <code>main()</code> executes.</p>
</li>
</ul>
</li>
<li><p><strong>Import</strong> the file as a module elsewhere (<code>import script</code>)</p>
<ul>
<li><p>→ <code>__name__</code> is set to the module's name instead</p>
</li>
<li><p>→ the condition is <code>False</code></p>
</li>
<li><p>→ <code>main()</code> does <strong>not</strong> run automatically.</p>
</li>
</ul>
</li>
</ul>
<p><strong>Best practice:</strong> always guard your entry point this way. It makes the module safely <strong>importable</strong> for testing individual functions, or for reuse in another script, without triggering the full pipeline (including a real publish to Kaggle!) just by importing it.</p>
<h2 id="heading-part-8-go-live-and-switch-to-the-real-api">Part 8: Go Live and Switch to the Real API</h2>
<p>Everything above runs safely against mock data. To point the pipeline at the real Hub'Eau API instead:</p>
<ol>
<li><p>Note the <code>use_mock=False</code> setting above. It threads through <code>APIConfig.use_mock</code>, and the calls to <code>run_etl_pipeline(use_mock=False)</code> and <code>main()</code>.</p>
</li>
<li><p>Make sure <code>KAGGLE_CONFIG.dataset_slug</code> and <code>input_csv</code> point at <strong>your own</strong> Kaggle dataset copy before you use it. You can only publish new versions of a dataset you own. Fork both the <a href="https://www.kaggle.com/code/grimespoint/data-engineering-with-python-etl-pipeline">notebook</a> and the <a href="https://www.kaggle.com/datasets/grimespoint/paris-flood-dataset">dataset</a>, then update the slug in the <a href="#heading-part-3-manage-configuration-with-dataclasses"><code>KaggleConfig</code></a> section.</p>
</li>
<li><p>Install and authenticate the <a href="https://github.com/Kaggle/kaggle-api">Kaggle CLI</a> if you plan to call <code>publish_to_kaggle()</code> outside of a Google Colab or Kaggle notebook.</p>
</li>
</ol>
<p>Everything else needs <strong>zero changes</strong>.</p>
<h3 id="heading-post-run-validation">Post-Run Validation</h3>
<p><code>validate_and_analyze()</code> closes the loop, and it's more than just a nice-to-have. It's a welcome sanity check that runs <em>after</em> the pipeline finishes. The output is a human-readable report covering shape, data types, null counts, summary statistics on <code>water_level_mm</code>, how many flood-alert records turned up, the temporal coverage and per-station record counts.</p>
<p>It doesn't change any data. It exists so that whoever, or whatever monitoring system, reads the run's log output can immediately see whether this week's numbers look sane. No need to analyze the CSV by hand. Try it!</p>
<h2 id="heading-summary">Summary</h2>
<p>You've just built a <em>complete</em> ETL pipeline that you can run again and again without breaking anything. The skeleton is here, and that same pattern repeats for almost any scheduled data job: swap out the API, the field mapping, and the destination.</p>
<p>For more tutorials like this one, check out my <a href="https://github.com/hyperphantasia">GitHub</a> or <a href="https://kaggle.com/grimespoint">Kaggle profile</a>.</p>
<p>Thanks for reading!</p>
<h2 id="heading-references">References</h2>
<ul>
<li><p><a href="https://hubeau.eaufrance.fr/">Hub'Eau API</a>: France's official open water-data platform</p>
</li>
<li><p>The <a href="https://www.kaggle.com/code/grimespoint/paris-flood-dataset-weekly-updater">updater <code>.py</code> script</a> runs live once a week and updates the original <a href="https://www.kaggle.com/datasets/grimespoint/paris-flood-dataset">dataset</a>.</p>
</li>
<li><p>The source dataset generator is available on <a href="https://github.com/hyperphantasia/paris-flood-dataset">GitHub</a>.</p>
</li>
<li><p>The complete <a href="https://www.kaggle.com/code/grimespoint/data-engineering-with-python-etl-pipeline">Jupyter notebook</a> to follow this tutorial and code all along (<a href="https://github.com/hyperphantasia/kaggle/blob/main/notebook_archive/data-engineering-with-python-etl-pipeline.ipynb">backup available</a>).</p>
</li>
</ul>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How Programmatic Advertising Works ]]>
                </title>
                <description>
                    <![CDATA[ Most tutorials on programmatic advertising stop at the web banner. That's a shame, because the idea gets far more interesting once you follow it off the screen and into the physical world. If you've n ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-programmatic-advertising-works/</link>
                <guid isPermaLink="false">6a614617166ef0e401b18d80</guid>
                
                    <category>
                        <![CDATA[ Advertising ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Manish Shivanandhan ]]>
                </dc:creator>
                <pubDate>Wed, 22 Jul 2026 22:37:11 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/ead3788a-52c2-4821-b5c9-1c626c50d375.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Most tutorials on <a href="https://advertising.amazon.com/blog/programmatic-advertising#1">programmatic advertising</a> stop at the web banner. That's a shame, because the idea gets far more interesting once you follow it off the screen and into the physical world.</p>
<p>If you've never worked in advertising, don't worry. You don't need any ad industry background to follow along. If you can read basic Python, you have everything you need.</p>
<p>The advertising part is just the setting. The real subject is a skill you'll use everywhere: taking a messy slice of the real world and turning it into data that software can act on.</p>
<p>In this article, you'll learn what programmatic advertising is and why it worked so well on the web. You'll see why bringing it to a billboard is really a data modeling problem, and you'll build small Python models for each step.</p>
<h3 id="heading-what-well-cover">What We'll Cover:</h3>
<ul>
<li><p><a href="#heading-what-is-programmatic-advertising">What Is Programmatic Advertising?</a></p>
</li>
<li><p><a href="#heading-why-the-web-made-programmatic-easy">Why the Web Made Programmatic Easy</a></p>
</li>
<li><p><a href="#heading-the-billboard-problem">The Billboard Problem</a></p>
</li>
<li><p><a href="#heading-step-1-resolve-entities">Step 1: Resolve Entities</a></p>
</li>
<li><p><a href="#heading-step-2-model-location-as-computed-context">Step 2: Model Location as Computed Context</a></p>
</li>
<li><p><a href="#heading-step-3-represent-availability-as-a-schedule-not-a-boolean">Step 3: Represent Availability as a Schedule, Not a Boolean</a></p>
</li>
<li><p><a href="#heading-step-4-express-price-as-a-function-not-a-number">Step 4: Express Price as a Function, Not a Number</a></p>
</li>
<li><p><a href="#heading-putting-it-together">Putting It Together</a></p>
</li>
<li><p><a href="#heading-an-exercise-to-build-the-intuition">An Exercise to Build the Intuition</a></p>
</li>
<li><p><a href="#heading-the-takeaway">The Takeaway</a></p>
</li>
</ul>
<h2 id="heading-what-is-programmatic-advertising">What Is Programmatic Advertising?</h2>
<p>Advertising has two sides. A publisher, such as a news site, has ad space to fill. And an advertiser, such as a shoe brand, wants to fill it. For decades, connecting the two meant phone calls, emails, and paperwork.</p>
<p>Programmatic advertising replaces that manual process with software. No human negotiates the placement. A system looks at an ad slot and decides, in milliseconds, whether to buy it and at what price.</p>
<p>Here's how it works when you open a web page. The page tells an ad exchange that a slot is open, along with some context about the page and the viewer. Advertisers' systems bid on the slot in a live auction. The winning ad appears before the page finishes loading. This process is called <a href="https://en.wikipedia.org/wiki/Real-time_bidding">real-time bidding</a>, and it happens billions of times a day.</p>
<p>Two quick terms will help. Inventory means the ad space a seller has to offer. An impression means one view of an ad by one person.</p>
<p>On the web, this model arrived fast and felt almost effortless. It's worth understanding why.</p>
<h2 id="heading-why-the-web-made-programmatic-easy">Why the Web Made Programmatic Easy</h2>
<p>The web made programmatic easy by accident. Every web ad slot came pre-structured. It had an address, meaning a URL and a position on the page. It sat inside a document with a known shape, the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model">Document Object Model</a>. And it could report back whether an ad had loaded and been seen.</p>
<p>The whole environment was machine-readable from birth, because machines were already serving it. When automated buying showed up, it had a clean surface to work against. The medium had already done the hard modeling work.</p>
<p>Keep that in mind. It's the key to the whole lesson.</p>
<h2 id="heading-the-billboard-problem">The Billboard Problem</h2>
<p>Now point the same idea at a billboard. None of that convenient structure exists.</p>
<p>A physical sign doesn't announce where it is or which way it faces. It doesn't say what it costs this week or whether it's even available. For most of its history, that information lived in rate cards and salespeople's memories, not in anything a program could query.</p>
<p>Here's the insight. Out-of-home advertising, the industry term for billboards, transit posters, and public screens, didn't lag the web because it was a weaker medium. It lagged because it had no machine-readable interface. The audience was always there. The structured data was not.</p>
<p>So the real work of bringing programmatic to the physical world isn't clever bidding logic. It's data modeling.</p>
<p>Let's make that concrete. There are four steps. Each one maps to a pattern you'll see in many other engineering problems.</p>
<h2 id="heading-step-1-resolve-entities">Step 1: Resolve Entities</h2>
<p>The same billboard often shows up in several vendors' datasets. Each vendor gives it a different name and slightly different coordinates. Before you can do anything else, one physical object has to become one record. This is known as <a href="https://en.wikipedia.org/wiki/Record_linkage">record linkage</a>, here with a geographic twist.</p>
<p>A simple approach: treat two records as the same panel if they sit close together and share a similar name. To measure the distance between two coordinates, you can use the <a href="https://en.wikipedia.org/wiki/Haversine_formula">haversine formula</a>.</p>
<pre><code class="language-python">from math import radians, sin, cos, asin, sqrt

def haversine_m(lat1, lon1, lat2, lon2):
    """Distance between two coordinates in meters."""
    lat1, lon1, lat2, lon2 = map(radians, [lat1, lon1, lat2, lon2])
    a = sin((lat2 - lat1) / 2) ** 2 + \
        cos(lat1) * cos(lat2) * sin((lon2 - lon1) / 2) ** 2
    return 6371000 * 2 * asin(sqrt(a))

def same_panel(a, b, max_distance_m=25):
    close = haversine_m(a["lat"], a["lon"], b["lat"], b["lon"]) &lt;= max_distance_m
    similar_name = a["name"].lower().split()[0] == b["name"].lower().split()[0]
    return close and similar_name

vendor_a = {"name": "I-95 North Bulletin", "lat": 25.7907, "lon": -80.1300}
vendor_b = {"name": "I-95 Bulletin #4421", "lat": 25.7908, "lon": -80.1301}

print(same_panel(vendor_a, vendor_b))  # True – one billboard, one record
</code></pre>
<p>Real systems use fuzzier matching and more signals, but the principle is the same. Entity resolution comes first, or every later step counts the same billboard twice.</p>
<h2 id="heading-step-2-model-location-as-computed-context">Step 2: Model Location as Computed Context</h2>
<p>A coordinate is not an audience. To make a panel useful to a buying system, you have to represent it as a point with an orientation. Then you estimate who actually passes it by joining it against road and traffic data.</p>
<p>The useful value is computed, not given:</p>
<pre><code class="language-python">def estimate_impressions(panel, traffic_by_road):
    """Estimate daily impressions from road traffic data."""
    daily_vehicles = traffic_by_road[panel["road_id"]]
    # Only traffic moving toward the panel's face can see it
    facing_share = 0.5
    avg_occupancy = 1.4  # people per vehicle
    return int(daily_vehicles * facing_share * avg_occupancy)

panel = {"id": "P-4421", "road_id": "I-95-N", "facing": "south"}
traffic = {"I-95-N": 120000}

print(estimate_impressions(panel, traffic))  # 84000
</code></pre>
<p>Notice what happened. The raw record was just a point on a map. It got enriched into something a buyer can reason about: 84,000 estimated daily impressions. This pattern shows up everywhere in data work. A raw record plus outside data equals a useful field.</p>
<h2 id="heading-step-3-represent-availability-as-a-schedule-not-a-boolean">Step 3: Represent Availability as a Schedule, Not a Boolean</h2>
<p>A web ad slot is either open right now or it isn't. In code, that's a boolean: a value that is simply true or false. A billboard gets booked in date ranges, and its state changes underneath you. Inventory gets held, booked, and released. You need live data, not a snapshot.</p>
<pre><code class="language-python">from datetime import date

class PanelSchedule:
    def __init__(self):
        self.bookings = []  # list of (start, end) tuples

    def book(self, start, end):
        if not self.is_available(start, end):
            raise ValueError("Panel not available for that range")
        self.bookings.append((start, end))

    def is_available(self, start, end):
        return all(end &lt; b_start or start &gt; b_end
                   for b_start, b_end in self.bookings)

schedule = PanelSchedule()
schedule.book(date(2026, 8, 1), date(2026, 8, 14))

print(schedule.is_available(date(2026, 8, 10), date(2026, 8, 20)))  # False
print(schedule.is_available(date(2026, 8, 15), date(2026, 8, 31)))  # True
</code></pre>
<p>The lesson carries over. Whenever you model the real world, ask whether a value is truly fixed. Often it's state that changes over time. Availability, stock levels, and seat maps are all schedules pretending to be booleans.</p>
<h2 id="heading-step-4-express-price-as-a-function-not-a-number">Step 4: Express Price as a Function, Not a Number</h2>
<p>The rate card says one number. Reality prices by date, demand, and how much inventory is left:</p>
<pre><code class="language-python">def price_for(base_rate, start, weeks_out, occupancy):
    """Price a booking based on lead time and demand."""
    demand_multiplier = 1 + occupancy          # busier market, higher price
    urgency_discount = 0.9 if weeks_out &gt; 8 else 1.0  # reward early booking
    return round(base_rate * demand_multiplier * urgency_discount, 2)

print(price_for(base_rate=3000, start=date(2026, 12, 1),
                weeks_out=19, occupancy=0.85))  # 4995.0
print(price_for(base_rate=3000, start=date(2026, 8, 1),
                weeks_out=2, occupancy=0.40))   # 4200.0
</code></pre>
<p>Once price is a function, software can compare thousands of panels and dates instantly. That's exactly what automated buying needs.</p>
<h2 id="heading-putting-it-together">Putting It Together</h2>
<p>You now have resolved entities, computed audience, live availability, and dynamic pricing. With that in place, <a href="https://www.adquick.com/guides/programmatic-dooh">programmatic digital out of home</a> advertising works in the physical world the same way it does online. Software can plan a campaign across physical screens, buy them automatically, adjust in near real time, and measure the result.</p>
<p>It was never really about the web. It was about whether the inventory had a schema, meaning a defined structure that tells software what each piece of data is. Once you give the physical world a schema, the automation follows.</p>
<h2 id="heading-an-exercise-to-build-the-intuition">An Exercise to Build the Intuition</h2>
<p>You don't need special access to practice this. Take any messy, location-based public dataset and rehearse the four steps. A good source is <a href="https://www.openstreetmap.org/">OpenStreetMap</a>, which offers free data on millions of real-world places.</p>
<p>First, resolve duplicates by merging records that describe the same real-world thing. Second, enrich each point with computed context instead of treating the raw record as complete. Third, model one field as state that changes over time rather than a fixed value. Fourth, wrap the result in a clean interface, such as a class or a small API, that something else could query.</p>
<p>You won't have built an ad platform, but you'll have practiced the exact skills that let automated buying reach a new field.</p>
<h2 id="heading-the-takeaway">The Takeaway</h2>
<p>A good abstraction survives contact with the real world.</p>
<p>When you first learn a concept in a tidy setting, it's easy to think the tidiness is part of the concept. It rarely is. You only find out how general an idea is when you drag it somewhere messy and watch it still work.</p>
<p>Programmatic buying, moved off the screen and onto a wall by a road, is one of the cleaner proofs of that. Keep an eye out for the same pattern elsewhere: a rich domain waiting on nothing but a schema.</p>
<p>Hope you enjoyed this article. You can <a href="https://linkedin.com/in/manishmshiva">connect with me on LinkedIn</a>.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Build and Schedule Local AI Assistants for Daily Tasks ]]>
                </title>
                <description>
                    <![CDATA[ Most AI agents are reactive as they wait for us to ask something. In this tutorial, I'll show you how to build local AI assistants that run on a schedule, handle the tasks you care about, and generate ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-build-and-schedule-local-ai-assistants-for-daily-tasks/</link>
                <guid isPermaLink="false">6a555a585f978e5aa7071985</guid>
                
                    <category>
                        <![CDATA[ AI ]]>
                    </category>
                
                    <category>
                        <![CDATA[ ai-agent ]]>
                    </category>
                
                    <category>
                        <![CDATA[ ollama ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ cron ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AI assistant ]]>
                    </category>
                
                    <category>
                        <![CDATA[ #qwen ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Darsh Shah ]]>
                </dc:creator>
                <pubDate>Mon, 13 Jul 2026 21:36:24 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/67ad144a-050e-4d98-a7c3-9f0a2c9b5648.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Most AI agents are reactive as they wait for us to ask something. In this tutorial, I'll show you how to build local AI assistants that run on a schedule, handle the tasks you care about, and generate daily digests for it. Each Assistant is an AI agent and the goal is to automate repetitive work with a cron-driven setup that saves you time.</p>
<p>We'll use Python to create a simple local scheduler, a directory of agents, and Ollama running the model locally so you avoid per-call API charges and keep inference on your own machine.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-background">Background</a></p>
</li>
<li><p><a href="#heading-motivation-and-architecture">Motivation and architecture</a></p>
</li>
<li><p><a href="#heading-step-1-install-ollama-and-pull-the-model">Step 1: Install Ollama and pull the model</a></p>
</li>
<li><p><a href="#heading-step-2-install-python-dependencies">Step 2: Install Python dependencies</a></p>
</li>
<li><p><a href="#heading-step-3-define-the-agent-format">Step 3: Define the agent format</a></p>
</li>
<li><p><a href="#heading-step-4-create-the-agent-scheduler">Step 4: Create the Agent Scheduler</a></p>
</li>
<li><p><a href="#heading-step-5-add-three-real-agents">Step 5: Add three real agents</a></p>
<ul>
<li><p><a href="#heading-agent-1-googl-stock-check">Agent 1: GOOGL stock check</a></p>
</li>
<li><p><a href="#heading-agent-2-ai-news-digest">Agent 2: AI news digest</a></p>
</li>
<li><p><a href="#heading-agent-3-weather-brief">Agent 3: Weather brief</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-step-6-add-agent-scheduler-to-cron">Step 6: Add Agent Scheduler to cron</a></p>
<ul>
<li><p><a href="#heading-macos-and-linux">MacOS and Linux</a></p>
</li>
<li><p><a href="#heading-windows-with-task-scheduler">Windows with Task Scheduler</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-sample-output">Sample output</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
</ul>
<h2 id="heading-background">Background</h2>
<p>Many of us have AI agents that can perform useful tasks – but they still need to be triggered. What if you could build a system that runs every day, automatically invokes those agents, and delivers the results without any manual effort? As an example, Claude uses the <code>/loop</code> command to scheduling recurring tasks.</p>
<p>In this tutorial, we'll build a lightweight daily scheduler that does exactly that. Every day, it invokes three read-only AI agents on a schedule. The same pattern can be extended to automate virtually any recurring AI-powered workflow. The AI agent acts as your assistant to complete the task.</p>
<p>To follow this tutorial, you'll need Ollama installed on your machine. The example works on macOS, Windows, and Linux. I'm using a MacBook Pro with 32 GB of RAM, but you can run this on a lower-memory machine by choosing a smaller Qwen model from Ollama.</p>
<h2 id="heading-motivation-and-architecture">Motivation and Architecture</h2>
<p>The motivation behind this project is simple: I want AI agent workers to handle repetitive tasks for me. Instead of doing tasks manually, I can have specialized agents do the work automatically.</p>
<p>Another benefit of this approach is privacy and control. Since everything runs locally, the agents, prompts, and outputs remain on my machine. There's no need to rely on external automation platforms or send workflow data to third-party services.</p>
<p>The architecture is intentionally lightweight. A scheduler runs once a day and invokes a set of read-only AI agents.</p>
<p>Each agent is responsible for a single task: checking GOOGL stock performance, summarizing the latest AI news, and generating a weather brief. The agent scheduler executes them independently, collects their outputs, and stores the results as markdown file in outputs folder. As the needs grow, we can add more agents to the folder to create additional recurring workflows. The agent scheduler code won't change.</p>
<pre><code class="language-plaintext">project/
├── scheduler.py
├── outputs/
├── agents/
    ├── googl_stock.py
    ├── ai_news.py
    └── weather_brief.py
</code></pre>
<h2 id="heading-step-1-install-ollama-and-pull-the-model">Step 1: Install Ollama and Pull the Model</h2>
<p>First, install Ollama for your platform.</p>
<p>We'll use Qwen for the local model.</p>
<pre><code class="language-bash">ollama pull qwen3.5:4b
</code></pre>
<h2 id="heading-step-2-install-python-dependencies">Step 2: Install Python Dependencies</h2>
<p>Create a virtual environment and install the packages:</p>
<pre><code class="language-bash">python3 -m venv venv
source venv/bin/activate
pip install langchain langchain-ollama requests
</code></pre>
<p>It requires LangChain &gt;= 1.0.0</p>
<p>One of the example agents uses Ollama's hosted web search API for fresh AI news. That API requires an <a href="https://docs.ollama.com/api/authentication#api-keys">Ollama account</a> and an API key in <code>OLLAMA_API_KEY</code>.</p>
<p>Set the key like this:</p>
<pre><code class="language-bash">export OLLAMA_API_KEY="paste-key-here"
</code></pre>
<h2 id="heading-step-3-define-the-agent-format">Step 3: Define the Agent Format</h2>
<p>Every agent is a Python file in the <code>agents/</code> folder with two attributes:</p>
<ul>
<li><p><code>NAME</code></p>
</li>
<li><p><code>run()</code></p>
</li>
</ul>
<p><code>run()</code> takes no arguments and returns a string. Whatever it returns gets written to a timestamped Markdown file in <code>outputs/</code>.</p>
<p>Create the folder structure:</p>
<pre><code class="language-bash">mkdir -p agents outputs
touch agents/__init__.py
</code></pre>
<h2 id="heading-step-4-create-the-agent-scheduler">Step 4: Create the Agent Scheduler</h2>
<p>The agent scheduler does three small jobs:</p>
<ol>
<li><p>Loads every agent module from <code>agents/</code></p>
</li>
<li><p>Calls <code>run()</code> on each one</p>
</li>
<li><p>Saves the result to <code>outputs/</code></p>
</li>
</ol>
<p>That's the whole agent scheduler. There's no state file or per-agent scheduling logic. The OS scheduler decides when the agent scheduler fires, and the agent scheduler executes every agent each time and saves the output from the agents as markdown file in outputs/ folder.</p>
<p>To add more agents, simply add them to the agents/ folder. The agent scheduler doesn't need to change.</p>
<p>Save this as <code>scheduler.py</code>:</p>
<pre><code class="language-python">import importlib
from datetime import datetime
from pathlib import Path

# Folder that contains all agent files.
AGENTS_DIR = Path("agents")

# Folder where the output files will be written.
OUTPUTS_DIR = Path("outputs")


def load_agents():
    """Import every valid agent module from the agents/ folder."""
    agents = []

    # Look through all Python files in agents/
    for path in sorted(AGENTS_DIR.glob("*.py")):
        # Skip private helper files like __init__.py
        if path.name.startswith("_"):
            continue

        # Import the file as a Python module, e.g. agents.googl_stock
        module = importlib.import_module(f"agents.{path.stem}")

        # Only keep modules that define NAME and run()
        if hasattr(module, "NAME") and hasattr(module, "run"):
            agents.append(module)
        else:
            print(f"[skip] {path.name} (missing NAME or run)")

    return agents


def main():
    """Load all agents, run them, and save their outputs."""
    # Create the outputs/ folder if it doesn't exist yet.
    OUTPUTS_DIR.mkdir(exist_ok=True)

    # Run every agent we found.
    for agent in load_agents():
        print(f"[run]  {agent.NAME}")

        try:
            # Call the agent's run() function.
            output = agent.run()

            # Create a timestamped filename like:
            # outputs/weather-brief-2026-07-03_08-00-39.md
            timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
            out_path = OUTPUTS_DIR / f"{agent.NAME}-{timestamp}.md"

            # Write the returned text to disk.
            out_path.write_text(output)

            print(f"[ok]   {agent.NAME} -&gt; {out_path}")
        except Exception as e:
            # If one agent fails, log it and continue with the others.
            print(f"[fail] {agent.NAME}: {e}")


if __name__ == "__main__":
    main()
</code></pre>
<h2 id="heading-step-5-add-three-real-agents">Step 5: Add Three Real Agents</h2>
<p>Here are three simple, read-only agents.</p>
<h3 id="heading-agent-1-googl-stock-check">Agent 1: GOOGL Stock Check</h3>
<p>Save this as <code>agents/googl_stock.py</code>.</p>
<p>It fetches GOOGL's daily quote data, computes the change in Python, and asks the local model to turn that into a short summary.</p>
<pre><code class="language-python">import requests
from langchain.agents import create_agent
from langchain_ollama import ChatOllama

NAME = "googl-stock"


def fetch_googl():
    url = "https://query1.finance.yahoo.com/v8/finance/chart/GOOGL?interval=1d&amp;range=1d"
    r = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=15)
    r.raise_for_status()

    meta = r.json()["chart"]["result"][0]["meta"]
    price = meta["regularMarketPrice"]
    prev = meta["chartPreviousClose"]
    change = price - prev
    pct = (change / prev) * 100 if prev else 0

    return {
        "symbol": "GOOGL",
        "price": round(price, 2),
        "previous_close": round(prev, 2),
        "change": round(change, 2),
        "pct_change": round(pct, 2),
    }


def run():
    data = fetch_googl()

    agent = create_agent(
        model=ChatOllama(model="qwen3.5:4b", temperature=0),
        tools=[],
        system_prompt=(
            "You write short stock summaries. "
            "Given stock data, write 2 concise Markdown bullet points explaining "
            "the price move and whether it was an up or down day."
        ),
    )

    result = agent.invoke({
        "messages": [{"role": "user", "content": str(data)}]
    })

    return (
        "# GOOGL Daily Summary\n\n"
        f"{result['messages'][-1].content}\n\n"
        f"**Raw data:** `{data}`\n"
    )
</code></pre>
<h3 id="heading-agent-2-ai-news-digest">Agent 2: AI News Digest</h3>
<p>Save this as <code>agents/ai_news.py</code>.</p>
<p>This agent uses Ollama's web search API to pull recent AI news results, then asks the local model to turn them into a short digest. The <code>OLLAMA_API_KEY</code>is the same one that is used for my <a href="https://www.freecodecamp.org/news/build-a-personal-ai-web-research-agent-with-ollama-and-qwen/">Personal Web Research AI Agent</a> tutorial.</p>
<pre><code class="language-python">import os
import requests
from langchain.agents import create_agent
from langchain_ollama import ChatOllama

NAME = "ai-news"


def search_news():
    r = requests.post(
        "https://ollama.com/api/web_search",
        headers={"Authorization": f"Bearer {os.getenv('OLLAMA_API_KEY')}"},
        json={"query": "latest AI news", "max_results": 5},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["results"]


def run():
    results = search_news()

    agent = create_agent(
        model=ChatOllama(model="qwen3.5:4b", temperature=0),
        tools=[],
        system_prompt=(
            "You write short AI news digests. "
            "Given search results, produce 3-5 Markdown bullet points. "
            "Each bullet should summarize one important story and end with its source URL."
        ),
    )

    result = agent.invoke({
        "messages": [{"role": "user", "content": str(results)}]
    })

    return f"# Daily AI News Digest\n\n{result['messages'][-1].content}\n"
</code></pre>
<h3 id="heading-agent-3-weather-brief">Agent 3: Weather Brief</h3>
<p>Save this as <code>agents/weather_brief.py</code>.</p>
<pre><code class="language-python">import requests
from langchain.agents import create_agent
from langchain_ollama import ChatOllama

NAME = "weather-brief"


def fetch_weather():
    r = requests.get("https://wttr.in/New+York?format=j1", timeout=15)
    r.raise_for_status()

    current = r.json()["current_condition"][0]
    return {
        "temp_f": current["temp_F"],
        "feels_like_f": current["FeelsLikeF"],
        "humidity": current["humidity"],
        "wind_mph": current["windspeedMiles"],
        "description": current["weatherDesc"][0]["value"],
    }


def run():
    weather = fetch_weather()

    agent = create_agent(
        model=ChatOllama(model="qwen3.5:4b", temperature=0),
        tools=[],
        system_prompt=(
            "You write short weather briefs. "
            "Given current weather data, write 2 concise Markdown bullet points "
            "summarizing the conditions in plain English."
        ),
    )

    result = agent.invoke({
        "messages": [{"role": "user", "content": str(weather)}]
    })

    return f"# Daily Weather Brief\n\n{result['messages'][-1].content}\n"
</code></pre>
<h2 id="heading-step-6-add-agent-scheduler-to-cron">Step 6: Add Agent Scheduler to cron</h2>
<p>The Agent Scheduler is designed to be triggered by your OS scheduler. Every time it runs, it executes all agents in the agents/ folder.</p>
<p>We need to use the full path to Python inside the virtual environment. Schedulers usually don't inherit your shell's <code>PATH</code>, so a bare <code>python</code> often won't work the way you expect.</p>
<h3 id="heading-macos-and-linux">MacOS and Linux</h3>
<p>On macOS, you can use either <code>launchd</code> or <code>cron</code>. <code>launchd</code> is the macOS-native scheduler, but for this tutorial, I'm using <code>cron</code> as it works for Linux as well.</p>
<p>Create a run_scheduler.sh script and put it alongside your code. Paste Ollama API key in placeholder.</p>
<pre><code class="language-plaintext">#!/bin/bash

export OLLAMA_API_KEY="&lt;key&gt;"
cd /full/path/to/project
/full/path/to/project/venv/bin/python3 scheduler.py &gt;&gt; runner.log 2&gt;&amp;1
</code></pre>
<p>Make it executable by doing <code>chmod +x run_scheduler.sh</code> in the terminal. You can test it by doing <code>./run_scheduler.sh</code> in your terminal.</p>
<p>Open your crontab:</p>
<pre><code class="language-bash">crontab -e
</code></pre>
<p>Add this line:</p>
<pre><code class="language-bash">0 8 * * * /full/path/to/project/run_scheduler.sh
</code></pre>
<p>This runs the scheduler.py every day at 8:00 AM. The <code>runner.log</code> captures both normal output and errors.</p>
<p>One caveat: if your machine is asleep when the cron job is supposed to run, that invocation is usually just missed.</p>
<h3 id="heading-windows-with-task-scheduler">Windows with Task Scheduler</h3>
<p>From PowerShell:</p>
<pre><code class="language-powershell">schtasks /Create /SC DAILY /TN "AI Runner" /TR "C:\path\to\venv\Scripts\python.exe C:\path\to\scheduler.py" /ST 08:00
</code></pre>
<p>Set the working directory to your project folder in the task settings so <code>agents/</code> and <code>outputs/</code> resolve correctly.</p>
<h2 id="heading-sample-output">Sample Output</h2>
<p>Run the scheduler manually first:</p>
<pre><code class="language-bash">python scheduler.py
</code></pre>
<p>Here's what one run looks like:</p>
<pre><code class="language-text">$ python scheduler.py
[run]  ai-news
[ok]   ai-news -&gt; outputs/ai-news-2026-07-05_17-52-12.md
[run]  googl-stock
[ok]   googl-stock -&gt; outputs/googl-stock-2026-07-05_17-53-18.md
[run]  weather-brief
[ok]   weather-brief -&gt; outputs/weather-brief-2026-07-05_17-53-54.md
</code></pre>
<p>The output is stored in <code>outputs/</code> folder. The output from each agent is shown below:</p>
<pre><code class="language-plaintext">outputs % ls
ai-news-2026-07-05_17-52-12.md
googl-stock-2026-07-05_17-53-18.md	
weather-brief-2026-07-05_17-53-54.md
</code></pre>
<pre><code class="language-plaintext">$cat googl-stock-2026-07-05_17-53-18.md 
# GOOGL Daily Summary

*   GOOGL closed at $359.91, down $1.30 (0.36%) from the previous close of $361.21.
*   This marks a down day for the stock.

**Raw data:** `{'symbol': 'GOOGL', 'price': 359.91, 'previous_close': 361.21, 'change': -1.3, 'pct_change': -0.36}`
</code></pre>
<pre><code class="language-plaintext">$cat weather-brief-2026-07-05_17-53-54.md 
# Daily Weather Brief

*   It's 77°F, feeling like 80°F.
*   Partly cloudy with 9 mph winds.
</code></pre>
<pre><code class="language-plaintext">cat ai-news-2026-07-05_17-52-12.md 
# Daily AI News Digest

*   After spooking the Trump administration into safety testing, Anthropic's Fable 5 and Mythos 5 models have received global release with export curbs lifted.
    https://arstechnica.com/tech-policy/2026/07/after-spooking-trump-into-safety-testing-anthropic-ai-models-get-global-release/
*   OpenAI has previewed three GPT-5.6 models (Sol, Terra, and Luna) with limited availability restricted to U.S. government-approved organizations.
    https://www.deeplearning.ai/the-batch/gpt-5-6-lands-in-limbo
...
</code></pre>
<p>Before trusting the results, spot-check them. Smaller local models still hallucinate, and unattended agents amplify small mistakes because no one is there to catch them in real time.</p>
<p>To run it more frequently for testing, you can update the cron from <code>* 8 * * *</code> to <code>*/10 * * * *</code> so that it runs every 10 mins. Once you're satisfied with the setup and results, you can revert the cron to 8:00 AM everyday by setting it to <code>* 8 * * *</code>.</p>
<p>If you want to extend the setup, a few good next steps would be adding new agents, trying out different schedules, or setting up notifications when the agent scheduler finishes.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>In this tutorial, you built a small local AI agent scheduler that executes multiple agents from a folder. Each agent is just a Python file that calls an LLM and executes a task. The agent scheduler loads them, runs them, and writes the outputs to disk.</p>
<p>That gives you a nice workflow for lightweight local automation. Adding a new agent just involves dropping a file into <code>agents/</code>, not editing scheduler config again. The model runs locally through Ollama, the outputs stay on your machine, and there aren't LLM API costs.</p>
<p>From here, you can add your own agents. Perhaps a summary of yesterday's Git commits or a tool to watch for new releases of a repo you care about. Anything that you'd want waiting for you in the morning but that you don't want to check yourself. Happy tinkering!</p>
<p>If you enjoyed this tutorial, you can find more of my writing on my <a href="https://darshshah.org/blog/">blog</a> (recent posts include system design paper series), my work on my <a href="https://darshshah.org/">personal website</a>, and updates on <a href="https://www.linkedin.com/in/darshs">LinkedIn</a>.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Build an AI Agent That Runs its Own LLM Experiments with autoresearch ]]>
                </title>
                <description>
                    <![CDATA[ A few months ago, Andrej Karpathy released autoresearch. It's an open-source Python tool that lets an AI agent run experiments on one GPU while you sit back and wait for the results. Lately I've still ]]>
                </description>
                <link>https://www.freecodecamp.org/news/build-an-ai-agent-that-runs-its-own-llm-experiments-with-autoresearch/</link>
                <guid isPermaLink="false">6a42a24e2a8a54195ace1aab</guid>
                
                    <category>
                        <![CDATA[ ai-agent ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AI ]]>
                    </category>
                
                    <category>
                        <![CDATA[ llm ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ ishaan gupta ]]>
                </dc:creator>
                <pubDate>Mon, 29 Jun 2026 16:50:22 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/4f910471-5f78-41c0-a30e-7630737bbb74.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>A few months ago, Andrej Karpathy released <a href="https://github.com/karpathy/autoresearch"><strong>autoresearch</strong></a>. It's an open-source Python tool that lets an AI agent run experiments on one GPU while you sit back and wait for the results.</p>
<p>Lately I've still seen folks on Twitter arguing about whether AI agents can build their <em>“million dollar idea”</em> or something about <em>Openclaw</em>. But here's a repo that lets you hand an agent a real GPT training setup and ask it to do the research itself.</p>
<p>Basically it edits the code, trains, reads the loss, makes a decision about the result, and repeats this process. And all this happens while you sleep, or dig into something else. And surprisingly, it does actually work.</p>
<p>On a depth-12 nanochat baseline (more on what "depth" means later), Karpathy left it running for about two days. Over roughly 700 experiments, the agent found about 20 changes that genuinely improved the model, and those changes stacked on top of each other.</p>
<p>In this article, I'll walk through what autoresearch is, why the way it measures success is the whole trick, what each file in the repo actually does, what the agent tends to discover, and a step-by-step guide to running it yourself. By the end you should be able to point an agent at your own GPU and let it run.</p>
<h2 id="heading-table-of-contents"><strong>Table of Contents</strong></h2>
<ul>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-what-is-autoresearch">What is autoresearch?</a></p>
</li>
<li><p><a href="#heading-why-this-matters">Why This Matters</a></p>
</li>
<li><p><a href="#heading-what-exactly-is-valbpb">What Exactly is <code>val_bpb</code>?</a></p>
</li>
<li><p><a href="#heading-what-the-agent-actually-finds">What the Agent Actually&nbsp;Finds</a></p>
</li>
<li><p><a href="#heading-final-thoughts">Final Thoughts</a></p>
</li>
</ul>
<h2 id="heading-prerequisites"><strong>Prerequisites</strong></h2>
<p>This article is a complete walkthrough of this repo. The goal is that by the end, you'll understand what autoresearch is and how you can run it on your own machine.</p>
<p>No prior ML research experience required, but if you have it then the deeper sections I wrote will be more meaningful to you. Just basic knowledge of GPU, VRAM and GPUs like H100/A100/4090 would suffice, but don't worry i have quoted the text below explaining every term i think a beginner needs to understand.</p>
<h2 id="heading-what-is-autoresearch">What is autoresearch?</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6a0065c6e3eebc2e20691ad8/4d4413c5-7264-49b0-bcb0-1cf8b7e763f7.png" alt="flowchart of the autoresearch loop" style="display: block;" width="1600" height="967" loading="lazy">

<p>Simply put, autoresearch is just one specific idea executed cleanly. You take a small but real LLM training setup, put it in a single Python file, and let an AI agent edit that file.</p>
<p>The agent runs the file and reads the loss. When you train a language model, "loss" is just a single number that scores how badly the model is predicting the next chunk of text. A high number means it's guessing poorly, and a number close to zero means it's predicting almost perfectly.</p>
<p>Training is the process of nudging the model's millions of internal weights to push that number down. So when I say the agent "reads the loss," I mean it looks at that score to judge whether the change it just made helped or hurt.</p>
<p>Based on that score, the agent decides whether the change helped, and then either keeps the change or reverts it. Then it tries something else.</p>
<p>The flow runs top to bottom like this: A human (you) writes the playbook (a Markdown file called <a href="http://program.md">program.md</a>), which spells out the rules. An AI agent reads that playbook and starts an experiment loop.</p>
<p>In each pass of the loop, the agent edits the training code with a new idea, trains for five minutes, reads the resulting score, decides whether to keep or undo the change, and writes the outcome to a results file. Then it loops back and tries the next idea.</p>
<p>It does this on its own, around twelve times an hour. So a full night of sleep buys you roughly a hundred experiments and, with luck, a noticeably better model by morning.</p>
<p>The repo is laid out so the agent has exactly one knob to turn. It can't install new packages or change how the data is loaded or how the loss is measured. All of that is locked down on purpose. The only file the agent edits is <code>train.py</code> which consists of the model architecture, the optimizer, the batch size, the learning rate, and the structure of the training loop itself.</p>
<p>The reason this design works is the same reason a controlled experiment in any field works. If the data, the metric, and the budget are all fixed, then any change in the result must be coming from the change the agent made. The agent is doing science the way a careful researcher would, only it doesn't get tired and doesn't need lunch.</p>
<h2 id="heading-why-this-matters">Why This Matters</h2>
<p>It's tempting to read this as just another agent demo. But it's not, and the reason is the metric. That metric is called val_bpb, short for validation bits per byte. It's a specific way of scoring how well the model predicts text it has never seen during training (the "validation" set).</p>
<p>I'll break down exactly how it's calculated in the next section, but the one-line version is that it measures, on average, how many bits of information the model needs to encode each byte of text. Lower is better: a lower val_bpb means the model is surprised less often by real text, which is the whole goal.</p>
<p>The reason Karpathy uses bits per byte rather than the raw training loss is that bits per byte doesn't change just because you changed the vocabulary, so two very different models can still be compared fairly. The "lower is better" part and the "vocabulary-independent" part are two separate properties. The metric happens to have both.</p>
<p>When I say a baseline model from this repo "lands around 1.00 bpb," I mean that if you run the default untouched training script for its 5 minutes, the model it produces scores roughly 1.00 on this metric when measured on the held-out validation text. That's your starting line.</p>
<p>From there, an improvement of 0.005 bpb (so a score of about 0.995) is a small but real win, the kind the agent finds often. An improvement of 0.05 (a score near 0.95) would be enormous, the kind of jump you'd usually only get from a much bigger model or a much longer training run. So the numbers look tiny, but on this scale, thousandths of a bit genuinely matter.</p>
<p>Here's why optimizing this particular number is a big deal. The agent isn't chasing some artificial leaderboard that researchers spent years gaming. It's pushing down the same kind of validation loss curve that every major language model has been trained against since GPT-2 in 2019.</p>
<p>A "loss curve" is just the plot of that score dropping over the course of training, and "the wave of LLMs since GPT-2" is shorthand for the fact that essentially all of the progress, from GPT-2 to today's frontier models, came from people finding ways to make that curve drop faster or lower for the same amount of compute. The agent is working on the exact same problem, just at a small, fast cheap scale.</p>
<p>And that's what makes the next part surprising. When the agent finds an improvement "here," I mean on the small depth-12 model it's allowed to edit. "Depth" is the number of transformer layers stacked in the model. depth-12 is a small model, and depth-24 is a bigger one with twice as many layers.</p>
<p>Karpathy took the roughly 20 tweaks the agent discovered on the small depth-12 model and applied them to the bigger depth-24 model. Being stacked cleanly means two things at once: the improvements were additive (turning on all 20 together gave you the sum of their individual gains, rather than cancelling each other out), and they transferred (gains found on the small model still showed up on the big one).</p>
<p>That's the signal that the agent found real insights about training, not lucky quirks that only help at one specific size. Stacked together, they cut Karpathy's "Time to GPT-2" benchmark from 2.02 hours to 1.80 hours, which is about an 11% speedup on code he'd already hand-tuned for a long time.</p>
<p>The other thing that's significant is the budget. Each experiment runs for exactly 5 minutes of wall-clock training time, no more, no less. That gives roughly 12 experiments per hour, or about 100 in a typical 8-hour sleep cycle.</p>
<h3 id="heading-exploring-the-repo">Exploring the Repo</h3>
<p>Now if you clone the repo, you get a small handful of files. Most of them are plumbing. Three of them are the heart of the system and the difference between them is who edits what.</p>
<p>Only three files matter, and they differ by who edits them.</p>
<ol>
<li><p><a href="http://train.py">train.py</a> is the file the agent edits. it holds the GPT model, the optimizer, and the training loop, and everything in it is fair game.</p>
</li>
<li><p><a href="http://prepare.py">prepare.py</a> is the fixed foundation that nobody edits during a run: it downloads the data, trains the tokenizer, and defines the metric.</p>
</li>
<li><p><a href="http://program.md">program.md</a> is the file you, the human, edit: it's the playbook of rules the agent follows.</p>
</li>
</ol>
<p>The remaining files (README.md, pyproject.toml, uv.lock, .gitignore, .python-version, the analysis.ipynb notebook, and the progress.png image) are plumbing and documentation that neither you nor the agent needs to touch during a run.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6a0065c6e3eebc2e20691ad8/1a8acbf9-87a3-428e-9cc1-53aaee2adc91.png" alt="three main files that we need to understand" style="display: block;" width="1600" height="752" loading="lazy">

<p>There are a few other files in the repo which don't need attention from you or the agent during a run.</p>
<h2 id="heading-what-exactly-is-valbpb">What Exactly is <code>val_bpb</code>?</h2>
<p>Before going further, it helps to understand what val_bpb is. If you've read other LLM articles, you have probably seen terms like <strong>“perplexity”</strong> or <strong>“cross-entropy loss”</strong> thrown around.</p>
<p>Bits per byte is like their cousin. When a language model predicts text, it assigns probabilities to what comes next. If the model is confident and right, it gets a low loss. If it's confident and wrong, it gets a high loss, a large penalty. Add up those penalties across all the text and you get the model's total loss. Lower is better, because a lower total means the model assigned high probability to the words that actually appeared.</p>
<p>Cross-entropy loss is the standard scoring function for training language models. For each token, the model assigns a probability to every possible next token and the loss is the negative logarithm of the probability it gave to the token that actually came next. Predict the right token confidently and the loss is near zero. Assign low probability to the correct token and the loss is large. The model's total loss is the average of this across all tokens.</p>
<p>Cross-entropy loss measures this in nats. A nat is the unit you get when that logarithm is taken in base e (the natural log) instead of base 2. It measures the same quantity of "surprise" on a different scale (one nat is about 1.44 bits). Dividing the loss by the natural log of 2 is what rescales nats into bits, which is the conversion bits per byte performs.</p>
<p>Bits per byte takes that loss and divides it by the number of bytes the text actually contains, then converts to log base 2. The result is a number that tells you, on average, how many bits of information the model needs to encode each byte of text.</p>
<p>A perfect model would need close to zero, while a random model would need around 8 bits per byte (since a byte has 8 bits).</p>
<p>The reason Karpathy chose bpb instead of plain cross-entropy is that bpb is <strong>vocabulary-size-independent</strong>. If the agent decides to change the tokenizer or the vocabulary, the cross-entropy loss would be completely different even for the same model quality. Bits per byte normalizes that out, so a depth-8 model with vocab 8192 and a depth-12 model with vocab 16384 are directly comparable.</p>
<p>The function that computes this, evaluate_bpb, lives in prepare.py, which the agent is never allowed to edit. It can only touch train.py. Because the metric's definition sits in a file the agent can't modify, it can't lower its score by quietly changing how the score is calculated. The scoring rule stays identical for every experiment, which is what makes the comparison honest.</p>
<h3 id="heading-the-5-minute-rule">The 5 Minute&nbsp;Rule</h3>
<p>There's one design choice in autoresearch that deserves its own section, because it's the choice that makes the whole thing work in practice. Every experiment runs for exactly 5 minutes of wall-clock training time regardless of what the agent is doing.</p>
<p>Wall-clock time means real elapsed time: what a clock on the wall measures, and not the number of training steps or tokens processed. 5 minutes of wall-clock time is 5 literal minutes regardless, of how much the model does in them.</p>
<p>If you trained for a fixed number of steps instead, the agent could “win” by making the model so small that it ripped through more steps than the baseline. If you trained for a fixed number of tokens, the agent could win by lowering the sequence length.</p>
<p>The agent isn't competing against another agent as we might think of it. Its only objective is to push val_bpb below the previous best score on this exact setup. So "winning" means producing a lower score, and the risk is that it lowers the score through a degenerate shortcut that games whichever budget you chose rather than a real efficiency gain. If you trained until convergence, the agent’s run would take wildly different amounts of time and you would never finish 100 experiments in a night.</p>
<p>A fixed wall clock budget cuts through all of this. The agent is forced to optimize for actual training efficiency on the actual hardware in front of it. If it makes the model slightly bigger but the per-step compute drops because of a smarter attention pattern, that's a real win. If it speeds up the per-step compute but the model now learns less per step, that shows up as a worse val_bpb. The two effects get netted out automatically in the end.</p>
<p>The H100 and A100 are NVIDIA datacenter GPUs and the RTX 4090 is a high-end consumer card. They differ sharply in speed and memory, and that's the whole point: in a fixed 5 minute budget, a faster card processes more data and reaches a lower val_bpb. So a score from one GPU can't be compared head-to-head with a score from another.</p>
<p>There's a tradeoff, though. Because the budget is wall-clock, the val_bpb you get on an H100 isn't directly comparable to the val_bpb you get on a 4090 or an A100. The system is designed to find the best model <strong>for your specific compute platform</strong> in 5 minutes, not to be a global benchmark.</p>
<p>If you want to compare across hardware, you would need to fix a different budget. For the autonomous research use case, this is exactly right.</p>
<p>Let’s get into each of the files in depth now.</p>
<h3 id="heading-1-preparepy">1. <code>prepare.py</code></h3>
<p>Nobody touches this file but everything depends on it. It mainly performs three jobs.</p>
<p>The first job is downloading data. The training corpus is ClimbMix-400B, a high-quality web dataset hosted on HuggingFace and shuffled into 6,543 parquet shards. By default <code>prepare.py</code> downloads only 10 of these (about a few gigabytes), which is plenty for running thousands of 5-minute experiments.</p>
<p>The very last shard is always downloaded and pinned as the validation set. That pinning matters, since every experiment (no matter what changes) evaluates on the exact same held-out data.</p>
<p>The second job is training a tokenizer. The repo uses <strong>rustbpe,</strong> a fast Rust implementation of byte-pair encoding, to learn a vocabulary of 8,192 tokens from a sample of the training data. The result is exported as a tiktoken-compatible encoding so it integrates cleanly with PyTorch downstream. There's also a small precomputed lookup table called <code>token_bytes.pt</code> that maps each token id to its UTF-8 byte length. This is what makes the bpb calculation honest.</p>
<p>The third job is providing utilities that <code>train.py</code> imports at runtime. The dataloader is the interesting one. It does what's called <strong>best-fit packing</strong>: every row in the batch starts with a special BOS (beginning of sequence) token and the loader fills the row by greedily picking documents that fit in the remaining space. Only when no document fits does it crop the shortest available document to fill the gap.</p>
<p>The result is 100% utilization with no padding. This is meaningfully faster than the naïve approach of just truncating long documents and padding short ones. The constants at the top of <code>prepare.py</code> are deliberately simple. Three numbers and a sequence length define the entire experimental contract.</p>
<p>If you run autoresearch on different hardware and want to compare results with a friend, the only thing both of you need to share is these constants. That's the whole point of putting them here and nowhere else.</p>
<h3 id="heading-2-trainpy">2. <code>train.py</code></h3>
<p>This is the file the agent lives in. It breaks naturally into four parts: the model, the optimizer (Muon for the matrix weights, AdamW for the embeddings and scalar parameters), the hyperparameters, and the training loop. We'll walk through each one with the goal of understanding why each piece exists.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6a0065c6e3eebc2e20691ad8/a4847be2-2007-42e0-91bd-9599125b5ffc.png" alt="you can see in the image that the agent only controls the two green boxes in the middle, the model and the loop" style="display: block;" width="1600" height="644" loading="lazy">

<p>The model is a fairly modern GPT written from scratch with no library dependencies beyond PyTorch and a Flash Attention 3 kernel. If you've read other GPT implementations the high-level structure will look familiar: a token embedding, a stack of transformer blocks, a normalization layer, and a linear head that projects back to vocabulary logits.</p>
<p>The interesting parts are in the details. I don’t think explaining the architecture or code is required for this repo, so I’ll just draw out a small architecture diagram for those of you who want to visualize it. Then I'll explain how the training loop is written.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6a0065c6e3eebc2e20691ad8/42663aea-dace-4d97-8bf4-7294d61f8a0d.png" alt="simple explanation of the  model in train.py- token embedding feeding a stack of transformer blocks, then a normalization layer, then a linear head producing vocabulary logits" style="display: block;" width="1600" height="1600" loading="lazy">

<p>The loop itself is short and almost pleasant to read. The skeleton is:</p>
<pre><code class="language-python">while True:
    # accumulate gradient over micro-batches to hit TOTAL_BATCH_SIZE
    for micro_step in range(grad_accum_steps):
        with autocast_ctx:
            loss = model(x, y)
        loss = loss / grad_accum_steps
        loss.backward()
        x, y, epoch = next(train_loader)

    # update LR / momentum / weight decay based on time elapsed
    progress = min(total_training_time / TIME_BUDGET, 1.0)
    # ... set group["lr"], group["momentum"], group["weight_decay"] ...

    optimizer.step()
    model.zero_grad(set_to_none=True)

    # log step metrics
    # ...

    if step &gt; 10 and total_training_time &gt;= TIME_BUDGET:
        break
</code></pre>
<p>There are a few things worth noticing here. First, the time budget is checked after the first 10 steps. This is so the budget doesn't include the initial PyTorch compilation (which can take 30 seconds or more). Without this, fast experiments would get penalized for spending half their budget on warmup.</p>
<p>Second, the loop has a fast-fail check. If the loss explodes or hits NaN it prints “FAIL” and exits. The agent then sees a crash and logs it. This is a defense against the agent doing something that diverges spectacularly.</p>
<p>Third, after the loop ends, there's a single final call to <code>evaluate_bpb</code> and then a structured summary printed to stdout.</p>
<p>That summary is the whole API between the training script and the agent:</p>
<pre><code class="language-yaml">---
val_bpb:          0.997900
training_seconds: 300.1
total_seconds:    325.9
peak_vram_mb:     45060.2
mfu_percent:      39.80
total_tokens_M:   499.6
num_steps:        953
num_params_M:     50.3
depth:            8
</code></pre>
<p>This is what the grep extracts and the agent reads. The whole experimental contract is seven lines of this plain text.</p>
<h4 id="heading-the-hyperparameters">The Hyperparameters</h4>
<p>The hyperparameters live in their own clearly-marked section near the bottom of <code>train.py</code>, with a comment that says "edit these directly, no CLI flags needed." They look like this:</p>
<pre><code class="language-yaml"># Model architecture
ASPECT_RATIO = 64       # model_dim = depth * ASPECT_RATIO
HEAD_DIM = 128          # target head dimension for attention
WINDOW_PATTERN = "SSSL" # sliding window pattern: L=full, S=half context

# Optimization
TOTAL_BATCH_SIZE = 2**19 # ~524K tokens per optimizer step
EMBEDDING_LR = 0.6
UNEMBEDDING_LR = 0.004
MATRIX_LR = 0.04
SCALAR_LR = 0.5
WEIGHT_DECAY = 0.2
ADAM_BETAS = (0.8, 0.95)
WARMUP_RATIO = 0.0
WARMDOWN_RATIO = 0.5
FINAL_LR_FRAC = 0.0

# Model size
DEPTH = 8
DEVICE_BATCH_SIZE = 128
</code></pre>
<p>Everything here is a deliberate single point of truth. The model dimension is computed from depth (<code>depth × 64</code>, rounded to the head dimension). The number of heads is computed from model dimension. This means that the agent can change one number <code>DEPTH</code>, and the model rescales itself coherently.</p>
<p>That kind of "one knob to scale the model" parameterization is exactly what makes a search space tractable.</p>
<h3 id="heading-3-programmd">3. <code>program.md</code></h3>
<p><code>program.md</code> is the shortest of the three files and is arguably the most important. It's the file that we edit and it contains everything the agent needs to know about how to behave during a run.</p>
<p>The structure of <code>program.md</code> mirrors the lifecycle of a research session. It opens with <strong>setup,</strong> agrees on a run tag, creates a Git branch named <code>autoresearch/&lt;tag&gt;</code>, reads the in-scope files, verifies that the data exists, and initializes a results file. It then describes the experimentation rules, like what the agent can and can't modify, that VRAM is a soft constraint, and crucially a simplicity criterion that says all else being equal, simpler is better.</p>
<p>A 0.001 bpb improvement that adds 20 lines of hacky code isn't worth keeping. A 0.001 bpb improvement that <strong>removes</strong> 20 lines is definitely worth keeping.</p>
<p>Then comes the actual loop. The agent is told to run training with <code>uv run train.py &gt; run.log 2&gt;&amp;1</code> and never to use <code>tee</code> or stream the output because that would flood the agent's context window. It's also told to extract metrics with <code>grep "^val_bpb:\|^peak_vram_mb:" run.log</code>, which gives just the one or two lines that matter.</p>
<p>If the grep produces nothing, that means the run crashed and the agent is told to read the last 50 lines of the log and try to fix the issue (but it should give up after a few attempts and move on). The result of every experiment is logged to <code>results.tsv</code>.</p>
<p>The decision rule is simple: if val_bpb improved (got lower) then the agent advances the branch by keeping its commit. If it didn't improve, the agent runs <code>git reset</code> to undo the commit. If it crashed, the agent logs that and tries something else.</p>
<p>The last paragraph of <code>program.md</code> is the one that makes autoresearch what it is. It's titled <strong>NEVER STOP</strong>. The agent is explicitly told not to ask the human (you) if it should keep going, not to ask for any permissions, and not to pause for confirmation. If the agent runs out of ideas, it should think harder, look at the failures, combine near-misses, and try more radical changes.</p>
<p>The loop runs until we interrupt it. This single instruction is more interesting than any line of Python in the repo. It's the difference between an agent that does a few experiments and asks if you want to continue and an agent that genuinely does autonomous research overnight.</p>
<p>There is no contradiction with the 5 minute budget. 5 minutes governs a single experiment, one training run. The "Never stop" instruction governs the outer loop. The moment one run finishes and the agent logs the result, it launches the next one. It keeps starting fresh 5 minute experiments back-to-back until you interrupt it.</p>
<p>Nothing ever trains for more than five minutes. The agent simply never stops starting new 5 minute trainings.</p>
<p>Now that you understand how it works, let’s start using it.</p>
<h2 id="heading-setup-guide">Setup Guide</h2>
<p>I'm assuming you have a single NVIDIA GPU with enough VRAM to run these experiments. Anything with 24GB or more should work with the default settings. Smaller GPUs need some tuning, which I'll cover later on.</p>
<h3 id="heading-step-1-install-uv-the-python-project-manager-the-repo-uses">Step 1: Install uv, the Python Project Manager the Repo Uses</h3>
<p>uv is much faster than pip and handles virtual environments transparently. After you install it, then clone the repo and install dependencies:</p>
<pre><code class="language-shell">curl -LsSf https://astral.sh/uv/install.sh | sh

git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync
</code></pre>
<p>This will create a&nbsp;<code>.venv</code> and install pyTorch, Flash Attention, rustbpe, tiktoken, pyarrow, and a few other packages. It pulls PyTorch from the CUDA 12.8 wheel index, so make sure your driver supports that.</p>
<h3 id="heading-step-2-run-the-data-preparation">Step 2: Run the Data Preparation</h3>
<p>This downloads 10 ClimbMix shards plus the validation shard and then trains our tokenizer.</p>
<pre><code class="language-shell">uv run prepare.py
</code></pre>
<p>It takes about 2 minutes on a decent connection. If you have limited disk space, you can pass <code>--num-shards 4</code> for a smaller download. The data and tokenizer get cached in <code>~/.cache/autoresearch/</code>.</p>
<h3 id="heading-step-3-run-a-manual-training-experiement">Step 3: Run a Manual Training Experiement</h3>
<p>Now, you'll run a single training experiment manually, just to confirm that everything works end-to-end.</p>
<pre><code class="language-shell">uv run train.py
</code></pre>
<p>You should see the model compile (this takes 30 seconds or so the first time), then training output that looks something like this: <code>step 00050 (8.3%) | loss: 5.123456 | lrm: 1.00 | dt: 240ms | tok/sec: 2,184,533 | mfu: 39.8% | epoch: 1 | remaining: 275s</code>.</p>
<p>After about 5 minutes of training, plus an evaluation pass at the end, you'll get the summary block with <code>val_bpb</code> printed. That's your baseline.</p>
<h3 id="heading-step-4-hand-the-repo-to-an-agent">Step 4: Hand the Repo to an Agent</h3>
<p>In practice, this means opening Claude Code or your tool of choice in the repo directory, ideally with permissions disabled or scoped tightly to the repo, and prompting it with something like this:</p>
<pre><code class="language-plaintext">Have a look at program.md and let's kick off a new experiment.
Let's do the setup first.
</code></pre>
<p>The agent will read <code>program.md</code>, walk through the setup steps (creating the autoresearch branch and initializing <code>results.tsv</code>), confirm with you, and then start running. From this point on, you can leave it alone. When you come back, check <code>results.tsv</code> and the Git log on the autoresearch branch.</p>
<h3 id="heading-tuning-autoresearch-for-smaller-gpus">Tuning autoresearch for Smaller&nbsp;GPUs</h3>
<p>The default configuration assumes an H100. If you have a 4090, 3090, or anything with less than 80GB of VRAM, you'll need to dial things down.</p>
<ol>
<li><p>Lower the sequence length first: <code>MAX_SEQ_LEN = 2048</code> in <code>prepare.py</code> is the biggest VRAM lever since attention scales quadratically with it. Try 512 or even 256 on a small GPU and bump <code>DEVICE_BATCH_SIZE</code> in <code>train.py</code> slightly to compensate. The product of these two is the tokens-per-forward-pass.</p>
</li>
<li><p>Lower the depth: <code>DEPTH = 8</code> in <code>train.py</code> is the master knob for model size. Drop it to 4 on a small GPU and the model dimension automatically scales down with it.</p>
</li>
<li><p>Switch the window pattern: <code>WINDOW_PATTERN = "SSSL"</code> uses banded attention which is fast on H100 but can be slow on consumer GPUs, depending on the kernel implementation. Just <code>"L"</code> (always full attention) is simpler and often faster on smaller cards.</p>
</li>
<li><p>Lower the total batch size: <code>TOTAL_BATCH_SIZE = 2**19</code> is roughly 524K tokens per optimizer step. On a small GPU, drop it to 2^14 (~16K) to start.</p>
</li>
<li><p>Consider switching the dataset: climbMix is a hard broad web corpus. On a tiny model, the loss curve is noisy and bpb numbers are hard to interpret. Karpathy specifically recommends his own TinyStories-GPT4-Clean dataset for small-scale experimentation. The text is narrower in scope (children’s stories) so a small model can actually learn to generate something coherent in 5 minutes.</p>
</li>
</ol>
<p>There are already several community forks that have done the consumer-GPU tuning for you which you can check out in the repo's readme.md file.</p>
<h2 id="heading-what-the-agent-actually-finds">What the Agent Actually&nbsp;Finds</h2>
<p>It's one thing to describe how the loop works, and another to see what it produces. Karpathy was open about this on Twitter in his depth-12 run: the agent found about 20 changes that improved validation loss, all of which transferred to depth-24.</p>
<p>Specific examples from his post-run analysis include adding a learnable scalar to the parameterless QK-norm to sharpen attention, applying regularization to the value embeddings, widening the banded attention window, correcting the AdamW betas for certain parameter groups, tuning weight decay schedules, and adjusting initialization.</p>
<p>None of these would headline a research paper, but all of them showed up as 0.001 to 0.005 bpb improvements that stacked.</p>
<p>So it's not that an AI agent invented a new architecture. It's that the slow patient hill-climbing that real researchers spend months doing can be done by an agent in a couple of days. The result is the same boring detail-tuning that has always been where most of the actual progress in ML comes from.</p>
<h2 id="heading-final-thoughts">Final Thoughts</h2>
<p>autoresearch doesn't introduce a new model or a new optimizer or a new dataset. It just defines a kind of contract between a human researcher and an AI agent and it shows that the contract can be enough. That contract is something like <em>“here is the fixed part of reality, the metric that judges you, a budget, and within those rules, do whatever you want and tell me what worked.”</em></p>
<p>There are two questions I still ponder that are worth thinking about. One is <strong>overfitting to the validation set</strong>. If you run hundreds of experiments against the same fixed validation shard, eventually the agent will start finding tweaks that look like wins on this shard but don't transfer. Karpathy himself called the results “fragile” in some sessions.</p>
<p>There's no obvious fix here yet beyond rotating validation data which would break comparability.</p>
<p>The other question is <strong>what the human’s role becomes</strong>. If the agent does the experiments, the human’s contribution shifts to shaping the search space and the rules. That is what <code>program.md</code> is. It's a pretty good preview of what research looks like when the loop is automated.</p>
<p>Well, that’s it for today. See you folks in my next article!</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Automate PDF Data Extraction Using Python ]]>
                </title>
                <description>
                    <![CDATA[ PDFs are still one of the most widely used document formats in business. Financial reports, invoices, contracts, compliance filings, and operational documents are often shared as PDFs because they pre ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-automate-pdf-data-extraction-using-python/</link>
                <guid isPermaLink="false">6a20556a08e3e46121ab6d4e</guid>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ pdf ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Manish Shivanandhan ]]>
                </dc:creator>
                <pubDate>Wed, 03 Jun 2026 16:25:14 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/85626e6c-7433-4914-b094-19d784845d82.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>PDFs are still one of the most widely used document formats in business.</p>
<p>Financial reports, invoices, contracts, compliance filings, and operational documents are often shared as PDFs because they preserve formatting across devices and operating systems.</p>
<p>The problem is that PDFs are designed for presentation, not structured data analysis. Extracting information manually from these files is slow, repetitive, and highly prone to human error.</p>
<p>This becomes a major issue for teams that work with large volumes of documents every day.</p>
<p>Finance departments process invoices and statements, analysts review reports, and operations teams manage records that contain valuable structured data trapped inside static files.</p>
<p>Copying rows manually into spreadsheets doesn't scale, especially when organisations handle hundreds or thousands of PDFs each month.</p>
<p><a href="https://www.freecodecamp.org/learn/python-v9/">Python</a> has become one of the most effective tools for automating PDF data extraction because of its mature ecosystem of libraries and data processing frameworks.</p>
<p>Developers can build workflows that extract text, identify tables, clean inconsistent formatting, and export structured datasets into Excel or CSV files automatically.</p>
<p>In smaller workflows, some teams may simply choose to convert <a href="https://smallpdf.com/pdf-to-excel">PDF to Excel with SmallPDF</a> for quick spreadsheet conversions, while larger organizations often build fully automated extraction pipelines using Python for deeper customisation and control.</p>
<p>In this article, we'll explore how to automate PDF data extraction using Python, including how to extract text and tables from PDFs, clean and transform structured data, work with scanned documents using OCR, and export information into spreadsheet formats like Excel.</p>
<p>We'll also look at some of the most useful Python libraries for document automation and discuss the common challenges developers face when building scalable PDF processing workflows.</p>
<h3 id="heading-what-well-cover">What We'll Cover:</h3>
<ul>
<li><p><a href="#heading-understanding-pdf-structures">Understanding PDF Structures</a></p>
</li>
<li><p><a href="#heading-setting-up-the-python-environment">Setting Up the Python Environment</a></p>
</li>
<li><p><a href="#heading-extracting-text-from-pdfs">Extracting Text From&nbsp;PDFs</a></p>
</li>
<li><p><a href="#heading-extracting-tables-from-pdfs">Extracting Tables From&nbsp;PDFs</a></p>
</li>
<li><p><a href="#heading-working-with-ocr-for-scanned-pdfs">Working With OCR for Scanned&nbsp;PDFs</a></p>
</li>
<li><p><a href="#heading-building-end-to-end-automation-pipelines">Building End-to-End Automation Pipelines</a></p>
</li>
<li><p><a href="#heading-common-challenges-in-pdf-automation">Common Challenges in PDF Automation</a></p>
</li>
<li><p><a href="#heading-choosing-the-right-python-libraries">Choosing the Right Python Libraries</a></p>
</li>
<li><p><a href="#heading-the-future-of-pdf-automation">The Future of PDF Automation</a></p>
</li>
</ul>
<h2 id="heading-understanding-pdf-structures">Understanding PDF Structures</h2>
<p>One of the biggest misconceptions about PDFs is that they all behave the same way. In reality, PDFs can vary significantly depending on how they were generated.</p>
<p>Machine-readable PDFs contain embedded text that can be extracted directly using parsing libraries. These files are usually exported from software systems such as accounting tools, reporting platforms, or office applications. Since the text already exists digitally, extraction is relatively reliable.</p>
<p>Scanned PDFs are different. These documents are essentially images stored inside a PDF container. Since there's no actual text layer, extraction tools can't read the content directly. OCR software must first analyze the images and attempt to reconstruct readable text.</p>
<p>Before writing any code, you should always test whether the text inside a PDF can be selected manually. If text highlighting works normally, the file likely contains a machine-readable layer. If not, you'll probably need OCR.</p>
<h2 id="heading-setting-up-the-python-environment">Setting Up the Python Environment</h2>
<p>Python provides several excellent libraries for PDF extraction and document automation. Each library specializes in different aspects of the workflow.</p>
<p>Some tools focus on text extraction, while others are optimized for identifying tables or processing scanned documents. Commonly used libraries include pdfplumber, PyMuPDF, Camelot, tabula-py, and pytesseract.</p>
<p>You can configure the environment using pip:</p>
<p><code>pip install pdfplumber pandas openpyxl pymupdf camelot-py</code></p>
<p>If OCR support is required, you can also install some additional packages:</p>
<p><code>pip install pytesseract pillow</code></p>
<p><a href="https://www.projectpro.io/article/how-to-train-tesseract-ocr-python/561">Tesseract</a> itself must also be installed separately on the operating system because pytesseract acts only as a Python wrapper around the OCR engine.</p>
<p>Once the environment is ready, you can begin building extraction workflows tailored to specific document types.</p>
<h2 id="heading-extracting-text-from-pdfs">Extracting Text From&nbsp;PDFs</h2>
<p>The simplest PDF automation workflow involves extracting plain text from machine-readable documents.</p>
<p>Libraries such as <a href="https://ukconstructionblog.co.uk/plumbing-invoice-template/">pdfplumber</a> make this process straightforward:</p>
<pre><code class="language-plaintext">import pdfplumber

with pdfplumber.open(“report.pdf”) as pdf:

for page in pdf.pages:

text = page.extract_text()

print(text)
</code></pre>
<p>This approach works well for reports, contracts, meeting notes, and other text-heavy documents.</p>
<p>But raw text extraction often introduces formatting issues. Multi-column layouts may become scrambled, line breaks can appear unexpectedly, and tabular information may lose alignment completely.</p>
<p>While text extraction is useful for search indexing and keyword analysis, structured business workflows usually require table extraction instead.</p>
<h2 id="heading-extracting-tables-from-pdfs">Extracting Tables From&nbsp;PDFs</h2>
<p>Most business automation projects focus on extracting tables from PDFs into structured spreadsheet formats.</p>
<p><a href="https://github.com/atlanhq/camelot">Camelot</a> is one of the most widely used Python libraries for this purpose. It identifies table structures by analyzing page layouts and separating rows and columns automatically.</p>
<p>Here's a simple example:</p>
<pre><code class="language-plaintext">import camelot

tables = camelot.read_pdf(“financial_report.pdf”, pages=’1')

print(tables[0].df)
</code></pre>
<p>The extracted table is returned as a Pandas DataFrame, which makes downstream processing significantly easier.</p>
<p>Exporting the extracted data into Excel is straightforward:</p>
<pre><code class="language-plaintext">import pandas as pd

df = tables[0].df

df.to_excel(“output.xlsx”, index=False)
</code></pre>
<p>This type of workflow is extremely valuable for finance and operations teams that regularly process statements, invoices, audit reports, or procurement records.</p>
<p>Real-world PDFs, however, are rarely perfectly-structured. Tables may span multiple pages, contain merged cells, or use inconsistent spacing. You'll often need additional transformation logic to clean and standardize the extracted data before it becomes useful for analytics or reporting.</p>
<h2 id="heading-working-with-ocr-for-scanned-pdfs">Working With OCR for Scanned&nbsp;PDFs</h2>
<p>Scanned documents require OCR because there's no machine-readable text available inside the file.</p>
<p>Python devs commonly use Tesseract together with pytesseract for OCR workflows.</p>
<p>A simple example looks like this:</p>
<pre><code class="language-plaintext">from PIL import Image

import pytesseract

image = Image.open(“invoice_scan.png”)

text = pytesseract.image_to_string(image)

print(text)
</code></pre>
<p>OCR accuracy depends heavily on image quality. Low-resolution scans, skewed pages, handwritten content, and poor lighting can reduce recognition performance substantially.</p>
<p>To improve results, you can preprocess images before running OCR. Common preprocessing techniques include grayscale conversion, thresholding, sharpening, and noise reduction.</p>
<p>Even with preprocessing, OCR should generally be treated as a fallback solution rather than the primary extraction strategy whenever machine-readable PDFs are available.</p>
<h2 id="heading-building-end-to-end-automation-pipelines">Building End-to-End Automation Pipelines</h2>
<p>Single extraction scripts are useful for experimentation, but enterprise workflows usually require complete automation pipelines.</p>
<p>A production-ready document automation system may include file ingestion, document classification, extraction, transformation, validation, export, and archival stages.</p>
<p>Python works particularly well in these environments because it integrates cleanly with APIs, databases, cloud storage platforms, and workflow orchestration systems.</p>
<p>For example, an accounts payable workflow might automatically monitor an inbox for incoming invoices, extract tabular data from attached PDFs, validate totals, and push the cleaned records into an ERP platform without human intervention.</p>
<p>This type of automation can save organizations hundreds of hours of repetitive administrative work each month while improving consistency and reducing operational errors.</p>
<p>Many advanced systems also combine traditional extraction logic with AI models that automatically classify document types before routing them into specialized extraction pipelines.</p>
<h2 id="heading-common-challenges-in-pdf-automation">Common Challenges in PDF Automation</h2>
<p>PDF extraction becomes more difficult as workflows scale.</p>
<p>One major challenge is inconsistency. Documents generated from the same source system may still vary slightly in formatting, page layout, or spacing. Small formatting differences can break rigid extraction logic unexpectedly.</p>
<p>Accuracy validation is another critical issue. Extracted data should never be assumed correct automatically, especially in finance, healthcare, or compliance workflows where errors can create operational or regulatory risks.</p>
<p>Performance can also become a bottleneck when processing large volumes of files. Sequential extraction may be sufficient for small workloads, but larger systems often require parallel processing and queue-based architectures.</p>
<p>Scanned PDFs introduce even more uncertainty because OCR engines are inherently probabilistic. Many organizations use human review systems for low-confidence extractions instead of relying entirely on automation.</p>
<p>The most reliable automation systems combine structured extraction logic, validation rules, and selective manual oversight.</p>
<h2 id="heading-choosing-the-right-python-libraries">Choosing the Right Python Libraries</h2>
<p>Different libraries perform better depending on the structure and complexity of the documents being processed.</p>
<p>pdfplumber is excellent for lightweight text extraction and layout analysis. Camelot performs particularly well with clearly defined tables. <a href="https://pymupdf.readthedocs.io/en/latest/">PyMuPDF</a> offers strong performance and lower-level PDF manipulation capabilities.</p>
<p>For OCR workflows, <a href="https://pypi.org/project/pytesseract/">pytesseract</a> remains one of the most popular open-source solutions because it integrates easily into Python pipelines.</p>
<p>There's rarely a single perfect tool for every document type. Experienced developers typically combine multiple libraries within the same workflow and dynamically choose extraction strategies based on document characteristics.</p>
<p>Testing against real production data is critical because sample documents rarely capture the inconsistencies found in live operational environments.</p>
<h2 id="heading-the-future-of-pdf-automation">The Future of PDF Automation</h2>
<p>Document automation is evolving rapidly as AI systems become better at understanding unstructured information.</p>
<p>Traditional rule-based extraction workflows still dominate most enterprise systems, but AI-assisted models are increasingly capable of interpreting layouts, identifying fields, and understanding relationships between document elements more accurately than older parsing techniques.</p>
<p>Python remains central to this ecosystem because of its flexibility and extensive machine learning tooling. You can combine PDF extraction libraries with AI frameworks to build systems that continuously improve as they process more documents.</p>
<p>As organizations continue digitizing operations, automated PDF extraction will become increasingly important across finance, legal, healthcare, logistics, and compliance industries.</p>
<p>Teams that invest in document automation early can reduce manual work, improve reporting accuracy, and unlock structured business data that would otherwise remain trapped inside static PDF files.</p>
<p>Hope you enjoyed this article. You can <a href="https://linkedin.com/in/manishmshiva">connect with me on LinkedIn</a>.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Use Bash & Python for Real DevOps Automation – Full Handbook with 5 Production Use Cases ]]>
                </title>
                <description>
                    <![CDATA[ Automation scripts often validate process completion instead of system health. A Kubernetes pod can be running while the application inside it can't authenticate to the database. A Terraform deploymen ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-use-bash-python-for-real-devops-automation-handbook-with-production-use-cases/</link>
                <guid isPermaLink="false">6a171310badcd8afcb060460</guid>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Bash ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Osomudeya Zudonu ]]>
                </dc:creator>
                <pubDate>Wed, 27 May 2026 15:51:44 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/73f2a745-c1b5-4cbb-8f97-2ba6c5230592.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Automation scripts often validate process completion instead of system health.</p>
<p>A Kubernetes pod can be running while the application inside it can't authenticate to the database. A Terraform deployment can return clean while someone has manually changed infrastructure in the cloud console. A canary rollout can show zero errors while users wait five seconds for every request.</p>
<p>The problem isn't the tooling. The problem is that the system can look healthy when it really is not.</p>
<p>This handbook walks through five production-style automation scenarios using Bash and Python for:</p>
<ul>
<li><p>Detecting abnormal AWS spend before the monthly invoice arrives</p>
</li>
<li><p>Correlating logs across multiple services using trace IDs</p>
</li>
<li><p>Finding infrastructure drift outside Terraform</p>
</li>
<li><p>Validating secret rotation at the application level</p>
</li>
<li><p>Automatically rolling back slow deployments before users complain</p>
</li>
</ul>
<p>By the end of this handbook, you'll be able to build small scripts that help you notice when something is wrong in a system, even when the tools say everything is fine.</p>
<p>The scripts are intentionally small. The important part is the operational thinking behind them like what signal the script measures, what failure mode it can detect, and what assumptions the platform is making underneath.</p>
<p>Each use case includes a runnable demo environment, the complete script, a breakdown of the system behaviour involved, and an intentional failure you can trigger yourself.</p>
<p>If you're new to this workflow, start with use case 1 and work forward. The later sections build on the same pattern: automation is useful when it verifies reality, not just process completion.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>Before you start, set up the following:</p>
<ul>
<li><p><strong>Python 3.8 or higher</strong> – check with <code>python3 --version</code></p>
</li>
<li><p><strong>A Python virtual environment</strong> – create one before installing anything:</p>
</li>
</ul>
<pre><code class="language-plaintext">python3 -m venv venv
source venv/bin/activate  

 # on Windows: 

venv\Scripts\activate
</code></pre>
<p>This keeps your installed packages isolated from your system Python and prevents permission errors on shared machines.</p>
<ul>
<li><p><strong>pip</strong> – Python's package installer, included with Python</p>
</li>
<li><p><strong>AWS CLI</strong> configured with a working profile – a free-tier AWS account is enough for use cases 1, 3, and 4. Verify it's working with:</p>
<pre><code class="language-plaintext">aws sts get-caller-identity
</code></pre>
</li>
<li><p><strong>Docker and Docker Compose</strong> – needed for use cases 2, 4, and 5</p>
</li>
<li><p><strong>Kind</strong> (Kubernetes in Docker) – a way to run Kubernetes locally for use cases 4 and 5. Install with <code>brew install kind</code> on macOS, or follow the <a href="https://kind.sigs.k8s.io/docs/user/quick-start/">Kind quick start guide</a></p>
</li>
<li><p><strong>kubectl</strong> – the command-line tool for talking to a Kubernetes cluster. After installing Kind, run <code>kind create cluster</code> and kubectl is configured automatically</p>
</li>
<li><p><strong>Helm</strong> – a package manager for Kubernetes, needed for use case 5. Install with <code>brew install helm</code> or the <a href="https://helm.sh/docs/intro/install/">Helm install guide</a></p>
</li>
<li><p><strong>Terraform</strong> – needed for use case 3. Install with <code>brew install terraform</code> on macOS or follow the <a href="https://developer.hashicorp.com/terraform/install">Terraform install guide</a>. Check with <code>terraform version</code>.</p>
</li>
<li><p><strong>bc</strong> – a calculator utility used by the canary watch scripts for floating-point comparison. Install with <code>brew install bc</code> on macOS or <code>apt install bc</code> on Ubuntu. Run <code>bc --version</code> to confirm it is available before starting use case 5.</p>
</li>
</ul>
<h3 id="heading-knowledge-and-skills">Knowledge and Skills</h3>
<ul>
<li><p>You should be comfortable reading Python and Bash scripts without needing to write them from scratch.</p>
</li>
<li><p>You should have basic Linux terminal comfort – navigating directories, running scripts, reading output, and so on.</p>
</li>
<li><p>You should know what Kubernetes pods and deployments are at a basic level – you don't need deep Kubernetes expertise, as use cases 4 and 5 will introduce the Kubernetes concepts they rely on as they go.</p>
</li>
<li><p>Familiarity with AWS basics such as what EC2, IAM, and Secrets Manager will help with use cases 1, 3, and 4, while use case 2 runs entirely on your local machine and requires no AWS knowledge at all.</p>
</li>
<li><p>For use case 3, knowing what Terraform is and what a state file does will help. You don't need to write any Terraform, but understanding that Terraform tracks and what it created is the foundation of the whole use case.</p>
</li>
</ul>
<h3 id="heading-aws-iam-permissions-required">AWS IAM Permissions Required</h3>
<p>The scripts in this article make real AWS API calls. Your IAM user or role needs the following minimum permissions. (If you see an <code>AccessDenied</code> error, this is the first place to look.):</p>
<table>
<thead>
<tr>
<th>Use Case</th>
<th>Required IAM Permission</th>
</tr>
</thead>
<tbody><tr>
<td>1 - Cost Anomaly Detection</td>
<td><code>ce:GetCostAndUsage</code></td>
</tr>
<tr>
<td>3 - Drift Detection</td>
<td><code>ec2:DescribeSecurityGroups</code></td>
</tr>
<tr>
<td>4 - Secrets Rotation</td>
<td><code>secretsmanager:GetSecretValue</code>, <code>secretsmanager:PutSecretValue</code></td>
</tr>
</tbody></table>
<p>If you're using a fresh AWS free-tier account with <code>AdministratorAccess</code> attached, these permissions are already included and you can skip this step.</p>
<p>If you're on a restricted IAM user, here's how to add them. In the AWS Console, go to IAM, click Users, then click your username. Under the Permissions tab, click Add permissions, then Create inline policy.</p>
<p>Switch to the JSON tab and paste a policy document granting the permissions in the table above, then save it.</p>
<p>If your company manages AWS through an organization and you don't have permission to edit your own IAM policies, ask your administrator to add these permissions to your role.</p>
<h3 id="heading-companion-github-repository">Companion GitHub Repository</h3>
<p>All demo projects live at: <a href="https://github.com/Osomudeya/devops-scripting-labs"><strong>https://github.com/irvingtalks/devops-scripting-labs</strong></a></p>
<p>Each use case has its own numbered folder with the complete script, supporting files, a <code>setup.sh</code> to prepare the environment, and a <code>break_it.sh</code> that injects the specific failure each use case is built around.</p>
<p>Clone the repo before starting:</p>
<pre><code class="language-plaintext">git clone https://github.com/irvingtalks/devops-scripting-labs
cd devops-scripting-labs
</code></pre>
<p>Before running any use case, check that you have everything installed:</p>
<pre><code class="language-plaintext">./preflight.sh
</code></pre>
<p>This checks for every tool the lab needs like Python, AWS CLI, Docker, Kind, Helm, Terraform, and <code>bc</code> and tells you exactly what's missing with the install command for each one.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-use-case-1-cost-anomaly-detection">Use Case 1 - Cost Anomaly Detection</a></p>
</li>
<li><p><a href="#heading-use-case-2-log-correlation-across-services">Use Case 2 - Log Correlation Across Services</a></p>
</li>
<li><p><a href="#heading-use-case-3-infrastructure-drift-detection">Use Case 3 - Infrastructure Drift Detection</a></p>
</li>
<li><p><a href="#heading-use-case-4-secrets-rotation-with-zero-downtime">Use Case 4 - Secrets Rotation with Zero Downtime</a></p>
</li>
<li><p><a href="#heading-use-case-5-automated-canary-rollback-trigger">Use Case 5 - Automated Canary Rollback Trigger</a></p>
</li>
<li><p><a href="#heading-what-you-can-do-now">What You Can Do Now</a></p>
</li>
</ul>
<h2 id="heading-use-case-1-cost-anomaly-detection">Use Case 1 - Cost Anomaly Detection</h2>
<p><strong>Environment:</strong> AWS Cost Explorer API (read-only, available in all accounts) <strong>Language:</strong> Python</p>
<h3 id="heading-the-production-problem">The Production Problem</h3>
<p>A junior engineer is testing a Kubernetes configuration. They spin up a managed node group in AWS (a set of EC2 virtual machines that the Kubernetes cluster uses to run workloads) and configure the cluster autoscaler, which is the Kubernetes component responsible for adding more machines when the cluster needs more capacity. The test goes well, and on Friday afternoon, they forget to tear the environment down.</p>
<p>Over the weekend, the autoscaler keeps provisioning new nodes because the test workloads are still running and requesting resources. By Monday morning you have a node group that has been quietly growing for two and a half days, and nobody noticed until the invoice landed three weeks later.</p>
<p>The script in this use case exists because your AWS bill isn't just a monthly number. It's a time series, and you can monitor it the same way you monitor application metrics. Check it daily, know your baseline, and you catch this kind of event in hours instead of weeks.</p>
<h3 id="heading-whats-actually-happening-at-the-system-level">What's Actually Happening at the System Level</h3>
<p><strong>What this is not:</strong> This isn't a finance dashboard. It's an operational anomaly detector and the signal it monitors is cost. But the thing it's actually detecting is unexpected infrastructure behavior such as resources left running, autoscaler events, and forgotten environments.</p>
<p>AWS Cost Explorer is a service that stores your billing data and exposes it through an API, and when you call it, you're running a query against your account's billing records by specifying the time range, the granularity, and how you want results grouped.</p>
<p>One thing to know before you start investigating any flagged cost is that AWS decides which service category to put a charge under, not you. An EBS snapshot copy running across regions might appear under the EC2 line item rather than data transfer, which means a spike in EC2 spend doesn't necessarily mean something went wrong with your EC2 instances. The script flags the spike correctly, but investigating it means asking <em>"what changed in my infrastructure on this date"</em> rather than <em>"what is running in EC2 right now."</em></p>
<p>The billing label is a starting point, not a diagnosis.</p>
<h3 id="heading-set-up-the-demo-environment">Set Up the Demo Environment</h3>
<p>Navigate to <code>01-cost-anomaly/</code> in the <a href="https://github.com/Osomudeya/devops-scripting-labs">companion repo</a>. No cluster setup is needed for this use case because the script runs against your AWS account directly, and the only dependency is boto3:</p>
<pre><code class="language-plaintext">cd 01-cost-anomaly
pip install boto3
</code></pre>
<p>Before running against your real account, make sure your AWS credentials are configured. The script uses whatever credentials the AWS CLI is set up with. If you haven't done this yet:</p>
<pre><code class="language-plaintext">aws configure
</code></pre>
<p>This will ask for your AWS Access Key ID, Secret Access Key, default region (use <code>us-east-1</code> if unsure), and output format (type <code>json</code>). You can find your access keys in the AWS Console under IAM → Users → your username → Security credentials → Create access key.</p>
<p>Your account needs the <code>ce:GetCostAndUsage</code> permission also, if you're on a fresh account with AdministratorAccess that's already included.</p>
<p>If you have an AWS account with a few weeks of billing history, you can run the script directly against your real data:</p>
<pre><code class="language-plaintext">python detect_cost_anomaly.py
</code></pre>
<p>Two things to know before running against a real account. First, Cost Explorer data has a 24-hour lag. This means spend from today won't appear until tomorrow, so the script automatically excludes the most recent day to avoid incomplete results.</p>
<p>Second, the script uses unblended costs, which is what you actually pay on a single-account setup. Blended costs are a weighted average used in multi-account organisations sharing reserved capacity and will give different numbers.</p>
<p>If you have a new account or prefer not to use real billing data, the script includes a <code>--sample</code> flag that uses built-in data and calls no AWS APIs at all.<br>Run this first to see what the output looks like before reading the code:</p>
<pre><code class="language-plaintext">python detect_cost_anomaly.py --sample
</code></pre>
<h3 id="heading-the-script">The Script</h3>
<pre><code class="language-python">#!/usr/bin/env python3
# detect_cost_anomaly.py — Use Case 1: Cost Anomaly Detection
# Full explanation of every function is in the article.

import statistics
import sys
from datetime import datetime, timedelta

import boto3

def build_sample_data(days=30):
    """Synthetic Cost Explorer rows for the last `days` (ending yesterday).

    The EC2 spike is placed on yesterday (device local date) so sample output
    always matches the same window as live Cost Explorer mode.
    """
    last_day = datetime.today().date() - timedelta(days=1)
    first_day = last_day - timedelta(days=days - 1)
    anomaly_day_index = days - 1
    results = []
    for i in range(days):
        day = first_day + timedelta(days=i)
        d = i + 1
        results.append(
            {
                "TimePeriod": {
                    "Start": str(day),
                    "End": str(day + timedelta(days=1)),
                },
                "Groups": [
                    {
                        "Keys": ["Amazon EC2"],
                        "Metrics": {
                            "UnblendedCost": {
                                "Amount": str(
                                    round(
                                        18.50
                                        if i == anomaly_day_index
                                        else 1.10 + (d % 3) * 0.10,
                                        2,
                                    )
                                )
                            }
                        },
                    },
                    {
                        "Keys": ["Amazon S3"],
                        "Metrics": {
                            "UnblendedCost": {
                                "Amount": str(round(0.04 + (d % 5) * 0.01, 2))
                            }
                        },
                    },
                    {
                        "Keys": ["Amazon RDS"],
                        "Metrics": {
                            "UnblendedCost": {
                                "Amount": str(round(0.85 + (d % 4) * 0.05, 2))
                            }
                        },
                    },
                ],
            }
        )
    return results, str(last_day)


def get_daily_costs(days=30):
    ce = boto3.client("ce", region_name="us-east-1")
    end = datetime.today().date() - timedelta(days=1)
    start = end - timedelta(days=days)
    response = ce.get_cost_and_usage(
        TimePeriod={"Start": str(start), "End": str(end)},
        Granularity="DAILY",
        Metrics=["UnblendedCost"],
        GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
    )
    return response["ResultsByTime"]


def build_service_timeseries(results):
    services = {}
    for day in results:
        date_str = day["TimePeriod"]["Start"]
        for group in day["Groups"]:
            service = group["Keys"][0]
            cost = float(group["Metrics"]["UnblendedCost"]["Amount"])
            if service not in services:
                services[service] = []
            services[service].append({"date": date_str, "cost": cost})
    return services


def detect_anomalies(services, baseline_days=7, multiplier=2.0, recent_days=None):
    """Flag days where cost exceeds prior `baseline_days` average + 2σ.

    Uses a rolling baseline (each day vs the previous week). If `recent_days`
    is set, only returns anomalies on or after today - recent_days.
    """
    cutoff = None
    if recent_days is not None:
        cutoff = datetime.today().date() - timedelta(days=recent_days)

    anomalies = []
    for service, daily in services.items():
        if len(daily) &lt; baseline_days + 1:
            continue
        for i in range(baseline_days, len(daily)):
            day = daily[i]
            day_date = datetime.strptime(day["date"], "%Y-%m-%d").date()
            if cutoff is not None and day_date &lt; cutoff:
                continue
            baseline_costs = [d["cost"] for d in daily[i - baseline_days : i]]
            avg = statistics.mean(baseline_costs)
            if avg &lt; 0.01:
                continue
            try:
                std = statistics.stdev(baseline_costs)
            except statistics.StatisticsError:
                continue
            threshold = avg + (multiplier * std)
            if day["cost"] &gt; threshold:
                anomalies.append(
                    {
                        "service": service,
                        "date": day["date"],
                        "actual": round(day["cost"], 4),
                        "baseline_avg": round(avg, 4),
                        "threshold": round(threshold, 4),
                        "pct_above": round(((day["cost"] - avg) / avg) * 100, 1),
                    }
                )
    return sorted(anomalies, key=lambda x: x["date"])


def parse_args(argv):
    use_sample = "--sample" in argv
    recent_days = None
    for arg in argv[1:]:
        if arg.startswith("--recent-days="):
            recent_days = int(arg.split("=", 1)[1])
    return use_sample, recent_days


def run(use_sample=False, recent_days=None):
    if use_sample:
        results, anomaly_date = build_sample_data()
        print("Running against sample data (--sample mode).")
        print(
            f"This data represents 30 days of billing ending yesterday, "
            f"with a realistic EC2 anomaly on {anomaly_date}.\n"
        )
    else:
        print("Fetching 30 days of daily AWS costs by service...")
        print("Note: today is excluded — Cost Explorer has a 24-hour billing lag.\n")
        results = get_daily_costs(days=30)

    if recent_days is not None:
        since = datetime.today().date() - timedelta(days=recent_days)
        print(
            f"Checking for spikes in the last {recent_days} days only "
            f"(on or after {since}), each vs its prior 7-day average.\n"
        )

    services = build_service_timeseries(results)
    anomalies = detect_anomalies(services, recent_days=recent_days)

    if not anomalies:
        print("No anomalies detected.")
        print("\nNote: this script flags statistical outliers against your own baseline.")
        print("A consistently elevated spend level will not trigger — only sudden increases.")
        return

    print(f"{'=' * 60}")
    print(f"ANOMALIES DETECTED: {len(anomalies)}")
    print(f"{'=' * 60}\n")

    for a in anomalies:
        print(f"Service:      {a['service']}")
        print(f"Date:         {a['date']}")
        print(f"Actual cost:  ${a['actual']}")
        print(f"Baseline avg: ${a['baseline_avg']} (prior 7-day average)")
        print(f"Threshold:    ${a['threshold']}")
        print(f"Overage:      {a['pct_above']}% above baseline")
        print()

    print("=" * 60)
    print("A note on AWS cost attribution:")
    print("The service label in Cost Explorer is assigned by AWS, not by the resource")
    print("that caused the cost. An EC2 spike may be caused by EBS snapshot copies,")
    print("cross-region data transfer, or autoscaling events that AWS categorizes under")
    print("EC2 in billing — not a running EC2 instance you can find in the console.")
    print()
    print("Before investigating the flagged service directly, ask:")
    print("What changed in my infrastructure on or before the flagged date?")
    print("Work backward from the operational change, not forward from the billing label.")


if __name__ == "__main__":
    use_sample, recent_days = parse_args(sys.argv)
    run(use_sample=use_sample, recent_days=recent_days)
</code></pre>
<h3 id="heading-how-the-script-works">How the Script Works</h3>
<p><code>get_daily_costs</code> pulls your AWS billing data for the last 30 days.</p>
<p><code>build_service_timeseries</code> takes the raw data from AWS and reorganises it. AWS groups the data by day first, then by service. This function flips that around so each service has its own list of daily costs, which is what the detection step needs to work with.</p>
<p><code>detect_anomalies</code> is where the actual check happens. For each service, it compares each day's spend to the 7 days right before it. If yesterday cost dramatically more than the week before, the script flags it. That's all it does.</p>
<p><code>--recent-days=7</code> means <em>"only show me anomalies from the last 7 days."</em> The script still fetches 30 days of data because it needs that history to calculate the comparison, but the results are filtered to the window you care about. This is good for a quick Monday morning check.</p>
<p><code>--sample</code> runs without touching your AWS account at all. It uses built-in fake billing data with a spike baked into yesterday's date so the detection always fires. Use this first to see what the output looks like before connecting it to real data.</p>
<h3 id="heading-what-the-output-looks-like">What the Output Looks Like</h3>
<p>Running <code>--sample</code> (the spike date will show as yesterday's actual date, not a fixed value):</p>
<pre><code class="language-plaintext">Running against sample data (--sample mode).
30 days of billing ending yesterday, with an EC2 spike on 2026-05-14.

============================================================
ANOMALIES DETECTED: 1
============================================================

Service:      Amazon EC2
Date:         2026-05-14
Actual cost:  $18.5
Baseline avg: $1.2143 (prior 7-day average)
Threshold:    $1.3939
Overage:      1423.4% above baseline

============================================================
A note on AWS cost attribution:
The service label in Cost Explorer is assigned by AWS, not by the resource
that caused the cost. An EC2 spike may be caused by EBS snapshot copies,
cross-region data transfer, or autoscaling events that AWS categorizes under
EC2 in billing - not a running EC2 instance you can find in the console.

Before investigating the flagged service directly, ask:
What changed in my infrastructure on or before the flagged date?
Work backward from the operational change, not forward from the billing label.
</code></pre>
<p>Your numbers will differ slightly from the above because the sample data generates dates from today dynamically. The spike always shows up on yesterday and the surrounding baseline numbers shift depending on the day you run it.</p>
<h3 id="heading-the-decision-the-script-cant-make-for-you">The Decision the Script Can't Make for You</h3>
<p>The anomaly is on the EC2 line, and the instinct is to go look at running EC2 instances. But as the output warns, the attribution is AWS's choice, not yours.</p>
<p>Before opening the EC2 console, check your deployment history for that date. What was deployed? Was a new environment created? Did an autoscaler event run? Start from the operational change and follow the thread to the billing data, because starting from the billing label and working backward is slower and frequently misleading.</p>
<h3 id="heading-break-it-on-purpose">Break it On Purpose</h3>
<pre><code class="language-bash"># See the spike immediately with no AWS account needed
python detect_cost_anomaly.py --sample

# Run against your real account
python detect_cost_anomaly.py

# Only show anomalies from the last 7 days, good for a quick this-week check
python detect_cost_anomaly.py --recent-days=7

# Combine both flags - sample data filtered to the last 7 days
python detect_cost_anomaly.py --sample --recent-days=7
</code></pre>
<p><strong>If your real account returns "No anomalies detected" that's not a failure.</strong> It means your spend has been consistent. A clean account returns clean output. The script is doing exactly what it should.</p>
<p>When a real event happens on your account such as an autoscaler left running, a forgotten environment or an unexpected data transfer, this is what catches it before the invoice does.</p>
<h2 id="heading-use-case-2-log-correlation-across-services">Use Case 2 – Log Correlation Across Services</h2>
<p><strong>Environment:</strong> Fully local – Docker Compose, three Python services<br><strong>Language:</strong> Python</p>
<h3 id="heading-the-production-problem">The Production Problem</h3>
<p>A user reports that their payment failed. You open your logging tool and search. The auth service logged a successful authentication. The ledger service logged a successful transaction but the notification service which should have sent a payment confirmation email has logged nothing at all.</p>
<p>Two services reported success while one service stay silent. The payment still failed, and you have three logs and no clear answer about where the chain broke.</p>
<h3 id="heading-whats-actually-happening-at-the-system-level">What's Actually Happening at the System Level</h3>
<p><strong>What this is not:</strong> This isn't a guide to installing a log aggregation tool. It's about the data structure that makes log correlation possible in the first place and what happens when that structure breaks on one service's error path.</p>
<p>In a system with a single service, debugging is simple: one service, one log file, one timeline. But when a user request passes through multiple services, you need a way to link all the logs together. That link is called a trace ID.</p>
<p>Think of it like a ticket number at a government office. When you walk in, you get a number, say, A247. Every desk that handles your case writes A247 on your file. If something goes wrong, the manager pulls every record with A247 and sees exactly what happened, in order, across every desk. That is a trace ID. One number, shared across every service that touched the request.</p>
<p>In the demo, when a payment comes in, the auth service creates a unique ID for it. Every log line that auth, ledger, and notification write for that payment includes the same ID. When something breaks, you run <code>correlate.py</code> with that ID and it finds every related log line across all three services and sorts them by time:</p>
<pre><code class="language-plaintext">python correlate.py pay-abc123
</code></pre>
<p>Here's what those logs look like. Notice that every line has the same <code>trace_id</code>:</p>
<pre><code class="language-json">{"timestamp": "2026-05-01T14:23:01.234Z", "trace_id": "pay-abc123", "service": "auth", "event": "user_authenticated", "level": "INFO", "user_id": "u_789", "duration_ms": 12}
{"timestamp": "2026-05-01T14:23:01.891Z", "trace_id": "pay-abc123", "service": "ledger", "event": "transaction_recorded", "level": "INFO", "amount": 50.0, "currency": "USD"}
{"timestamp": "2026-05-01T14:23:02.103Z", "trace_id": "pay-abc123", "service": "notification", "event": "email_queued", "level": "INFO", "recipient": "user@example.com"}
</code></pre>
<p>Now here's what breaks it. The notification service hits a timeout connecting to the email provider. The developer who wrote the error handler forgot to include the trace ID, so instead of a proper log line, it writes this:</p>
<pre><code class="language-plaintext">2026-05-01T14:23:02.415Z ERROR Connection timeout to email provider smtp.example.com:587
</code></pre>
<p>The error happened, the log line exists. But because it has no <code>trace_id</code>, <code>correlate.py</code> can't find it.</p>
<p>The notification still appears in the timeline, and you can see <code>email_send_attempt</code> – but <code>email_queued</code> never follows it.</p>
<pre><code class="language-plaintext">Timeline — 5 events across 3 service(s):

  [2026-05-15T21:59:00.605307+00:00] [AUTH] [INFO] payment_request_received
  [2026-05-15T21:59:00.606008+00:00] [AUTH] [INFO] user_authenticated
  [2026-05-15T21:59:00.617331+00:00] [LEDGER] [INFO] transaction_recorded
  [2026-05-15T21:59:00.630313+00:00] [NOTIFICATION] [INFO] email_send_attempt
  [2026-05-15T21:59:00.685182+00:00] [AUTH] [INFO] payment_complete
</code></pre>
<p>The attempt is there but the failure is not. The developer just forgot one field.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/22b7d7b0-8ae5-4573-bcb0-faaf5d807e8a.png" alt="log correlation attempt terminal output - ERROR Connection timeout" style="display: block;" width="1041" height="82" loading="lazy">

<h3 id="heading-set-up-the-demo-environment">Set Up the Demo Environment</h3>
<p>Navigate to <code>02-log-correlation/</code> and start the three services:</p>
<pre><code class="language-plaintext">cd 02-log-correlation
docker compose up -d
</code></pre>
<p>This starts the auth, ledger, and notification services. Trigger a payment request to generate some logs:</p>
<pre><code class="language-plaintext">./trigger_request.sh
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/06977757-e7fd-43c3-aaca-dbc6d17c951a.png" alt="trigger_request.sh terminal output - also showing the traceid" style="display: block;" width="630" height="131" loading="lazy">

<p>The script prints the trace ID it used. Copy the ID and Run the correlation script against it now, before we break anything, to see the full working path:</p>
<pre><code class="language-plaintext">python correlate.py pay-5831e1bf
</code></pre>
<p>You should see something like this (your trace ID will be different but the structure is the same):</p>
<pre><code class="language-plaintext">Loading logs from ./logs/...
Loaded 6 structured log lines.

============================================================
Trace ID: pay-5831e1bf
============================================================

Timeline - 6 events across 3 service(s):

  [2026-05-15T21:42:28.079046+00:00] [AUTH] [INFO] payment_request_received
    service: auth
    user_id: u_789
    amount: 50.0
  [2026-05-15T21:42:28.080718+00:00] [AUTH] [INFO] user_authenticated
    service: auth
    user_id: u_789
    duration_ms: 12
  [2026-05-15T21:42:28.145528+00:00] [LEDGER] [INFO] transaction_recorded
    service: ledger
    user_id: u_789
    amount: 50.0
    currency: USD
  [2026-05-15T21:42:28.210088+00:00] [NOTIFICATION] [INFO] email_send_attempt
    service: notification
    recipient: user@example.com
  [2026-05-15T21:42:28.347893+00:00] [NOTIFICATION] [INFO] email_queued
    service: notification
    recipient: user@example.com
    amount: 50.0
  [2026-05-15T21:42:28.378402+00:00] [AUTH] [INFO] payment_complete
    service: auth
    user_id: u_789
    amount: 50.0
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/26a0b226-4e3d-4967-a7c6-6d367409fb1d.png" alt="terminal output showing the full payment journey" style="display: block;" width="1100" height="559" loading="lazy">

<p>That's the full payment journey with auth, ledger, notification in the exact order it happened. Now let's look at how the script works.</p>
<h3 id="heading-the-script">The Script</h3>
<pre><code class="language-python"># correlate.py
import json
import os
import sys

SERVICES = ["auth", "ledger", "notification"]
LOG_DIR = "./logs"


def load_logs(log_dir):
    """
    Read each service's log file and parse every line as JSON.
    Lines that fail JSON parsing are printed as warnings.
    They are not silently dropped - a plain-text error line in a service
    that should emit structured logs is itself evidence worth seeing.
    """
    all_lines = []

    for service in SERVICES:
        log_file = os.path.join(log_dir, f"{service}.log")

        if not os.path.exists(log_file):
            print(f"  WARNING: No log file for '{service}' at {log_file}")
            continue

        with open(log_file) as f:
            for line_num, line in enumerate(f, 1):
                line = line.strip()
                if not line:
                    continue
                try:
                    parsed = json.loads(line)
                    parsed["_source"] = service
                    all_lines.append(parsed)
                except json.JSONDecodeError:
                    # This line exists in the log but cannot be correlated.
                    print(f"  WARNING: {service}.log line {line_num} is not structured JSON:")
                    print(f"           {line[:100]}")
                    print(f"           This line will NOT appear in any trace-based search.")

    return all_lines


def correlate(trace_id, all_lines):
    """
    Find every log line with this trace_id and sort by timestamp.
    The sorted result is the reconstructed timeline of the request.
    """
    matched = [line for line in all_lines if line.get("trace_id") == trace_id]
    matched.sort(key=lambda x: x.get("timestamp", ""))
    return matched


def find_missing_services(matched):
    """
    Check which services produced zero trace-tagged lines for this request.
    A missing service is not just an absence - it is a signal.
    Either the request never reached that service, or an error path swallowed
    the trace ID. Both are worth investigating.
    """
    services_seen = {line["_source"] for line in matched}
    return [s for s in SERVICES if s not in services_seen]


def print_timeline(trace_id, matched, missing):
    print(f"\n{'=' * 60}")
    print(f"Trace ID: {trace_id}")
    print(f"{'=' * 60}")

    if not matched:
        print("\nNo structured log lines found with this trace ID.")
        print("Either the trace ID is wrong, or no service emitted")
        print("a structured log line for this request.")
        return

    services_count = len({line["_source"] for line in matched})
    print(f"\nTimeline - {len(matched)} events across {services_count} service(s):\n")

    for line in matched:
        ts = line.get("timestamp", "unknown")
        service = line.get("_source", "unknown").upper()
        event = line.get("event", "unknown event")
        level = line.get("level", "INFO")
        extras = {k: v for k, v in line.items()
                  if k not in ("timestamp", "trace_id", "event", "level", "_source")}

        print(f"  [{ts}] [{service}] [{level}] {event}")
        for k, v in extras.items():
            print(f"    {k}: {v}")

    if missing:
        print(f"\n{'=' * 60}")
        print("MISSING TELEMETRY")
        print(f"{'=' * 60}")
        print(f"These services produced no trace-tagged events for trace {trace_id}:\n")
        for s in missing:
            print(f"  - {s}")
        print()
        print("This means one of three things:")
        print("  1. The request never reached this service.")
        print("  2. The service received it but an error path swallowed the trace ID,")
        print("     leaving a plain-text log line that trace correlation cannot find.")
        print("  3. This service's log file was not included in this run.")
        print()
        print("Check the raw log file for a plain-text error line around the same timestamp.")
        print("If one exists, that is your root cause - and a structured logging gap to fix.")


def run(trace_id):
    print(f"Loading logs from {LOG_DIR}/...")
    all_lines = load_logs(LOG_DIR)
    print(f"Loaded {len(all_lines)} structured log lines.\n")

    matched = correlate(trace_id, all_lines)
    missing = find_missing_services(matched)
    print_timeline(trace_id, matched, missing)


if __name__ == "__main__":
    if len(sys.argv) &lt; 2:
        print("Usage: python correlate.py &lt;trace_id&gt;")
        print("Example: python correlate.py pay-abc123")
        sys.exit(1)
    run(sys.argv[1])
</code></pre>
<h3 id="heading-how-the-script-works">How the Script Works</h3>
<p><code>load_logs</code> reads log files from each service. Each line should be JSON. If a line isn't JSON, it prints a warning that usually means an error log is missing a trace ID and can't be tracked.</p>
<p><code>correlate</code> finds all logs that match the given trace ID and sorts them by time. This rebuilds the full request flow across services.</p>
<p><code>find_missing_services</code> checks which services have no logs for that trace ID. This tells you where the request stopped or where the trace ID was lost.</p>
<p><code>print_timeline</code> displays the full request timeline in order. It also shows which services are missing if something didn't log correctly.</p>
<p>One thing worth knowing for when you use this in a real Kubernetes environment:<br>in Kubernetes, <code>kubectl logs</code> only shows the current running container.<br>If a pod restarts, you can use this:</p>
<pre><code class="language-plaintext">kubectl logs &lt;pod-name&gt; --previous
</code></pre>
<p>But this only works for the last restart. Older logs are gone unless you use a logging system like Loki or CloudWatch.</p>
<h3 id="heading-what-the-output-looks-like-after-breaking-it">What the Output Looks Like After Breaking it</h3>
<p>The point of this section is to show you what happens when a service fails silently, – when the error exists in the logs but the script can't find it because the developer forgot one field.</p>
<p><code>break_it.sh</code> forces the notification service to fail when it tries to send an email, and because the error handler was written without a trace ID, the failure gets logged as plain text with no way to tie it back to the original request.</p>
<p>Run it:</p>
<pre><code class="language-plaintext">./break_it.sh
</code></pre>
<p>Then trigger a new request:</p>
<pre><code class="language-plaintext">./trigger_request.sh
</code></pre>
<p>Copy the trace ID it prints, then correlate it:</p>
<pre><code class="language-plaintext">python correlate.py pay-xxxxxxxx
</code></pre>
<p>Here is what you'll see:</p>
<pre><code class="language-plaintext">Loading logs from ./logs/...
  WARNING: notification.log line 10 is not structured JSON:
           2026-05-15T21:59:00.681583+00:00 ERROR Connection timeout to email
           provider http://mock-email:80/ after 0.001s - failed to send
           confirmation to user@example.com
           This line will NOT appear in any trace-based search.
Loaded 29 structured log lines.

============================================================
Trace ID: pay-6cf69a8c
============================================================

Timeline - 5 events across 3 service(s):

  [2026-05-15T21:59:00.605307+00:00] [AUTH] [INFO] payment_request_received
  [2026-05-15T21:59:00.606008+00:00] [AUTH] [INFO] user_authenticated
  [2026-05-15T21:59:00.617331+00:00] [LEDGER] [INFO] transaction_recorded
  [2026-05-15T21:59:00.630313+00:00] [NOTIFICATION] [INFO] email_send_attempt
  [2026-05-15T21:59:00.685182+00:00] [AUTH] [INFO] payment_complete
</code></pre>
<p>Look at this carefully. The notification is in the timeline, and it logged <code>email_send_attempt</code>. But <code>email_queued</code> is missing, which means the email never actually sent and the error that explains why isn't in the timeline at all. It's hiding in the WARNING at the very top, where the script told you it found a line it couldn't parse.</p>
<p>That's the problem: where the attempt is visible but the failure is invisible.</p>
<p>Run <code>cat logs/notification.log</code> and scroll to the bottom:</p>
<pre><code class="language-plaintext">{"timestamp": "2026-05-15T21:59:00.630313+00:00", "trace_id": "pay-6cf69a8c",
 "service": "notification", "event": "email_send_attempt", ...}
2026-05-15T21:59:00.681583+00:00 ERROR Connection timeout to email provider
http://mock-email:80/ after 0.001s - failed to send confirmation to user@example.com
</code></pre>
<p>Two lines to note: the first has a trace ID, which the script found and showed in the timeline. The second doesn't – the script flagged it as a warning and skipped it. The error happened 0.075 seconds after the attempt. The log file has both lines. The timeline only has one.</p>
<p>That is what <em>"invisible failure"</em> looks like in production. The payment went through. The confirmation email never sent. The error is sitting right there in the log file, <code>Connection timeout to email provider after 0.001s</code> but in the correlation output above, the timeline shows <code>email_send_attempt</code> and then jumps straight to <code>payment_complete</code> with nothing in between: no error, no failure, no gap. It looks like everything worked.</p>
<p>The fix is in <code>02-log-correlation/services/notification/main.py</code>. Here's the broken error handler:</p>
<pre><code class="language-python">except httpx.TimeoutException:
    emit_plain(f"Connection timeout to email provider {EMAIL_PROVIDER_URL}")
    return {"status": "ok"}
</code></pre>
<p>And here's the fixed version. The only change is passing <code>req.trace_id</code> into <code>emit</code> instead of calling <code>emit_plain</code>:</p>
<pre><code class="language-python">except httpx.TimeoutException:
    emit(req.trace_id, "email_timeout", level="ERROR",
         provider=EMAIL_PROVIDER_URL)
    return {"status": "ok"}
</code></pre>
<p>Once that change is made, the timeout error shows up in the timeline like everything else:</p>
<pre><code class="language-plaintext">  [2026-05-15T21:59:00.681583+00:00] [NOTIFICATION] [ERROR] email_timeout
    provider: http://mock-email:80/
</code></pre>
<p>One command, one trace ID, the full picture.</p>
<h3 id="heading-the-decision-the-script-cant-make-for-you">The Decision the Script Can't Make For You</h3>
<p>The correlation script identifies notification as the gap. When you check the raw <code>notification.log</code>, you find the plain-text timeout error, that the request reached the service, that authentication and transaction recording both succeeded, but that the email failed.</p>
<p>Whether a notification failure is a payment failure depends entirely on how your system was designed. If notification is a soft dependency, this error shouldn't have surfaced to the user as a payment failure, and something else in your system design is wrong. If it's a hard dependency, the transaction itself should have rolled back. The script found where things broke, but the right response depends on the design.</p>
<h3 id="heading-break-it-on-purpose">Break it On Purpose</h3>
<ol>
<li><p>Run <code>./break_it.sh</code> – this switches the notification service to a mode where its error handler drops the trace ID</p>
</li>
<li><p>Run <code>./trigger_request.sh</code> to generate a new payment request and get a new trace ID</p>
</li>
<li><p>Run <code>python correlate.py &lt;new trace ID&gt;</code> – the notification will be missing from the timeline</p>
</li>
<li><p>Run <code>cat logs/notification.log</code> – the timeout error is right there, without a trace ID, invisible to the script</p>
</li>
</ol>
<h2 id="heading-use-case-3-infrastructure-drift-detection">Use Case 3 - Infrastructure Drift Detection</h2>
<p><strong>Environment:</strong> AWS free tier (one security group) + Terraform<br><strong>Language:</strong> Python</p>
<h3 id="heading-the-production-problem">The Production Problem</h3>
<p>Your Terraform plan shows no changes. Your deployment is behaving differently than it did yesterday, and when you ask around, someone eventually remembers: a colleague made a quick manual change to a security group in the AWS console last week to unblock a staging test. They meant to go back and apply it through Terraform but they forgot.</p>
<p>Your Terraform state file and your actual AWS infrastructure have been quietly disagreeing ever since. Not that anything broke loudly or an alert fired. Terraform wouldn't even know unless someone ran <code>terraform plan</code> to check, and in this scenario, nobody did.</p>
<p>This is called infrastructure drift, and it's far more common than most teams want to admit.</p>
<h3 id="heading-whats-actually-happening-at-the-system-level">What's Actually Happening at the System Level</h3>
<p><strong>What this is not:</strong> This isn't the same as running <code>terraform plan</code>. A plan shows you what Terraform <em>would</em> change. This script shows you what has <em>already</em> changed in AWS without Terraform knowing.</p>
<p>The script itself doesn't run any Terraform commands. It reads the state file Terraform already produced. In the demo, Terraform creates that file. In a real environment, it already exists from your normal workflow.</p>
<p>Think of Terraform's state file as a receipt. When Terraform creates a security group, it writes down exactly what it created, the rules, the ports, the CIDRs. That receipt is the state file.</p>
<p>The script compares that receipt against what AWS actually has right now. If someone went into the AWS console and added a rule that isn't on the receipt, the script flags it as drift.</p>
<p>The blind spot is that, if someone creates a completely new security group in the console and never uses Terraform at all, there's no receipt for it. The script can't compare something it has never seen. It returns clean, and that group sits in your account undetected.</p>
<p>The demo shows both. First you break a known resource. Then the <code>--invisible</code> scenario creates a new one outside Terraform entirely, and the script returns clean even though your account now has an extra security group.</p>
<h3 id="heading-set-up-the-demo-environment">Set Up the Demo Environment</h3>
<p>Navigate to <code>03-drift-detection/</code> in the companion repo:</p>
<pre><code class="language-plaintext">cd 03-drift-detection
pip install -r requirements.txt
</code></pre>
<p>Run setup. This uses real Terraform, not a mock:</p>
<pre><code class="language-plaintext">./setup.sh
</code></pre>
<p>This runs <code>terraform init</code> and <code>terraform apply</code>, which creates a real AWS security group:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/d5617704-d04e-40cc-8ca9-aa9d6b806e5e.png" alt="screenshot of AWS dashboard showing security group created" style="display: block;" width="1189" height="260" loading="lazy">

<p>It also writes a genuine <code>terraform.tfstate</code> file. Open it in any text editor if you want to see what Terraform actually produces. It's JSON, it's readable, and it's the real thing.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3ea8e237-a1d7-43bc-be3d-50dcb6ff4b76.png" alt="screenshot of IDE folder structure showing terraform.tfstate file being created" style="display: block;" width="215" height="230" loading="lazy">

<p>Once setup completes, run the script:</p>
<pre><code class="language-plaintext">python detect_drift.py terraform.tfstate
</code></pre>
<p>You should see something like this, but your actual security group ID will be different:</p>
<pre><code class="language-plaintext">Loading Terraform state from: terraform.tfstate

Checking: sg-0a1b2c3d4e5f6a7b8

  OK - No drift detected.
</code></pre>
<p>The lab is alive and both sides of the contract match. Now let's look at what the script is doing.</p>
<h3 id="heading-the-script-code-files">The Script (<a href="https://github.com/Osomudeya/devops-scripting-labs">Code Files</a>)</h3>
<pre><code class="language-python"># detect_drift.py
import boto3
import json
import sys


def load_tfstate(path):
    """
    The Terraform state file is plain JSON - open it in any text editor
    and you will see a 'resources' array listing everything Terraform knows about.
    This function reads that file and returns the parsed contents.
    """
    with open(path) as f:
        return json.load(f)


def get_security_groups_from_state(tfstate):
    """
    Walk through the resources array and collect every security group entry.
    Each resource has a 'type', a 'name', and an 'instances' array holding
    the attribute values Terraform recorded when it last ran.
    We extract the resource ID and the ingress (inbound) rules.
    """
    resources = {}
    for resource in tfstate.get("resources", []):
        if resource["type"] == "aws_security_group":
            for instance in resource.get("instances", []):
                sg_id = instance["attributes"]["id"]
                resources[sg_id] = {
                    "ingress": instance["attributes"].get("ingress", [])
                }
    return resources


def get_security_group_from_aws(sg_id):
    """
    Call the AWS EC2 API to fetch the live current state of this security group.
    Under the hood, boto3 constructs an authenticated HTTPS request, signs it with
    your AWS credentials, sends it to the EC2 API endpoint in your configured region,
    and parses the response. The response contains far more data than we need -
    we extract only the inbound rules.
    """
    ec2 = boto3.client("ec2")
    response = ec2.describe_security_groups(GroupIds=[sg_id])
    sg = response["SecurityGroups"][0]
    return {"ingress": sg.get("IpPermissions", [])}


def normalize_state_rules(rules):
    """
    Terraform stores ingress rules in its own format.
    We normalize them into a set of tuples for easy comparison.
    Each tuple is: (from_port, to_port, protocol, cidr_block)
    """
    normalized = set()
    for rule in rules:
        for cidr in rule.get("cidr_blocks", []):
            normalized.add((
                rule.get("from_port", 0),
                rule.get("to_port", 0),
                rule.get("protocol", "-1"),
                cidr
            ))
    return normalized


def normalize_aws_rules(rules):
    """
    AWS returns ingress rules in a different format from Terraform's.
    We normalize them into the same tuple shape so the comparison works.
    """
    normalized = set()
    for rule in rules:
        from_port = rule.get("FromPort", 0)
        to_port = rule.get("ToPort", 0)
        protocol = rule.get("IpProtocol", "-1")
        for ip_range in rule.get("IpRanges", []):
            normalized.add((from_port, to_port, protocol, ip_range["CidrIp"]))
    return normalized


def detect_drift(tfstate_path):
    print(f"Loading Terraform state from: {tfstate_path}")
    tfstate = load_tfstate(tfstate_path)
    state_sgs = get_security_groups_from_state(tfstate)

    if not state_sgs:
        print("No security groups found in state file. Nothing to compare.")
        return

    drift_found = False

    for sg_id, state_data in state_sgs.items():
        print(f"\nChecking: {sg_id}")

        try:
            aws_data = get_security_group_from_aws(sg_id)
        except Exception as e:
            print(f"  ERROR: Could not fetch {sg_id} from AWS - {e}")
            print(f"  Check your IAM permissions: ec2:DescribeSecurityGroups is required.")
            continue

        state_rules = normalize_state_rules(state_data["ingress"])
        aws_rules = normalize_aws_rules(aws_data["ingress"])

        # Rules in AWS that Terraform does not know about (manual additions)
        added_in_aws = aws_rules - state_rules
        # Rules Terraform expects that no longer exist in AWS (manual deletions)
        removed_from_aws = state_rules - aws_rules

        if added_in_aws:
            drift_found = True
            print("  DRIFT - Rules present in AWS but missing from state file:")
            for rule in added_in_aws:
                print(f"    Port {rule[0]}-{rule[1]} | Protocol: {rule[2]} | CIDR: {rule[3]}")

        if removed_from_aws:
            drift_found = True
            print("  DRIFT - Rules in state file but removed from AWS:")
            for rule in removed_from_aws:
                print(f"    Port {rule[0]}-{rule[1]} | Protocol: {rule[2]} | CIDR: {rule[3]}")

        if not added_in_aws and not removed_from_aws:
            print("  OK - No drift detected.")

    print("\n" + "=" * 60)
    if drift_found:
        print("Drift detected. See above for details.")
    else:
        print("No drift detected in monitored resources.")

    print("\nIMPORTANT: This script only checks resources tracked in your state file.")
    print("Resources created manually in AWS without Terraform are invisible to this check.")
    print("A clean output here does not mean your AWS account is clean - it means")
    print("the resources you are watching match what Terraform last recorded.")


if __name__ == "__main__":
    tfstate_path = sys.argv[1] if len(sys.argv) &gt; 1 else "terraform.tfstate"
    detect_drift(tfstate_path)
</code></pre>
<h3 id="heading-how-the-script-works">How the Script Works</h3>
<p><code>load_tfstate</code> opens <code>terraform.tfstate</code> and reads it. Run <code>cat terraform.tfstate</code> after setup and you'll see that it's just a text file and everything Terraform knows about your infrastructure is stored in there.</p>
<p><code>get_security_groups_from_state</code> pulls out every security group from that file, the ID AWS assigned it, and the inbound rules Terraform last recorded. These are the expected values.</p>
<p><code>get_security_group_from_aws</code> calls the AWS API and fetches the same security group's current inbound rules. These are the actual values. The script now has two versions of the same thing.</p>
<p><code>normalize_state_rules</code> and <code>normalize_aws_rules</code> exist because Terraform and AWS store the same rule in slightly different formats. These two functions convert both into the same format so the comparison works.</p>
<p>The comparison is the last step. Rules in AWS but not in the state file were added manually. Rules in the state file but not in AWS were deleted manually. The script prints both.</p>
<h3 id="heading-what-the-output-looks-like">What the Output Looks Like</h3>
<p>A clean run with no drift:</p>
<pre><code class="language-plaintext">Loading Terraform state from: terraform.tfstate

Checking: sg-0a1b2c3d4e5f6a7b8

  OK - No drift detected.

============================================================
No drift detected in monitored resources.

IMPORTANT: This script only checks resources tracked in your state file.
Resources created manually in AWS without Terraform are invisible to this check.
A clean output here does not mean your AWS account is clean - it means
the resources you are watching match what Terraform last recorded.
</code></pre>
<p>After injecting drift:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/61ae6f66-36e4-4e03-8e76-f250e2489dab.png" alt="screenshot of AWS dashboard showing security group inbound rule created" style="display: block;" width="1170" height="220" loading="lazy">

<pre><code class="language-plaintext">Loading Terraform state from: terraform.tfstate

Checking: sg-0a1b2c3d4e5f6a7b8

  DRIFT - Rules present in AWS but missing from state file:
    Port 22-22 | Protocol: tcp | CIDR: 0.0.0.0/0

============================================================
Drift detected. See above for details.
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/5744ac0c-2e4f-4015-9b72-a1fa7084587e.png" alt="screenshot of terminal output after injecting drift showing &quot;drift detected&quot;" style="display: block;" width="952" height="183" loading="lazy">

<h3 id="heading-the-decision-the-script-cant-make-for-you">The Decision the Script Can't Make For You</h3>
<p>The script finds drift, an inbound rule that Terraform doesn't know about. The instinct is to revert it immediately by running <code>terraform apply</code>, but before doing that, ask one question: was this change an emergency hotfix? Someone may have manually opened a port at 2am to restore a broken service while a proper fix was being prepared. And if you revert it automatically, you might undo something that was deliberately placed there to keep a service running.</p>
<p>Drift detection tells you that things are different. It doesn't tell you which version is correct, and investigating that is the work that comes after the script runs.</p>
<h3 id="heading-break-it-on-purpose">Break it On Purpose</h3>
<ol>
<li><p>Run <code>./break_it.sh</code>. This adds an SSH inbound rule (port 22) directly via the AWS CLI, simulating a manual console change.</p>
</li>
<li><p>Run <code>python detect_drift.py terraform.tfstate</code>. The drift appears in the output.</p>
</li>
<li><p>Run <code>./break_it.sh --invisible</code> to create a brand new security group that's not in the state file at all, then run the script again. It returns clean even though a new resource exists in your account, making the coverage gap visible.</p>
</li>
<li><p>Run <code>./teardown.sh</code>. When finished, this runs <code>terraform destroy</code> to delete the security group and clean up all AWS resources. No charges will remain after this.</p>
</li>
</ol>
<h2 id="heading-use-case-4-secrets-rotation-with-zero-downtime">Use Case 4 - Secrets Rotation with Zero Downtime</h2>
<p><strong>Environment:</strong> AWS Secrets Manager + local Kind cluster<br><strong>Language:</strong> Python</p>
<h3 id="heading-the-production-problem">The Production Problem</h3>
<p><strong>The goal of this use case:</strong> Kubernetes says a pod is healthy, but your users are getting database errors. The script catches that gap before the users are affected by running one extra check that Kubernetes never runs.</p>
<p>You rotate your database credentials. The pod restarts. <code>kubectl get pods</code> shows Running. Ten minutes later, users can't log in.</p>
<p>The rotation worked, but the problem is that Kubernetes checked whether the HTTP server was alive, not whether it could authenticate with the database. Those are two different things.</p>
<h3 id="heading-whats-actually-happening">What's Actually Happening</h3>
<p><strong>What this is not:</strong> This isn't about how to store secrets in Kubernetes. It's about what happens after the secret is rotated.</p>
<p>When a pod is already running, it holds a pool of open database connections that were authenticated before the rotation happened. Those connections stay alive after the password changes because they were authenticated before the change and the database does not kick them out. But when the pool needs to open a new connection, it uses the current environment credentials, which still have the old password. That new connection fails immediately.</p>
<p>Meanwhile, Kubernetes sees the pod responding to HTTP and marks it Running, so your users are hitting the failures with no indication from the cluster that anything is wrong.</p>
<h3 id="heading-what-the-healthzdb-endpoint-does">What the <code>/healthz/db</code> Endpoint Does</h3>
<p><code>/healthz</code> returns 200 if the HTTP server is alive. That is all Kubernetes checks.</p>
<p><code>/healthz/db</code> opens a fresh database connection using the current credentials and runs <code>SELECT 1</code>. If that fails after a rotation, the pod is Running but can't serve database requests. The rotation script calls this endpoint as its final step – the check Kubernetes never runs.</p>
<p>Here's what that looks like in the demo FastAPI application (<a href="https://github.com/Osomudeya/devops-scripting-labs">code files</a>):</p>
<pre><code class="language-python"># app.py (relevant section)
import os
import asyncpg
from fastapi import FastAPI, HTTPException

app = FastAPI()

DB_HOST = os.environ.get("DB_HOST", "postgres")
DB_PORT = int(os.environ.get("DB_PORT", "5432"))
DB_NAME = os.environ.get("DB_NAME", "appdb")
DB_USERNAME = os.environ.get("DB_USERNAME", "appuser")
DB_PASSWORD = os.environ.get("DB_PASSWORD", "")

@app.get("/healthz")
async def healthz():
    # Always returns 200 if the HTTP server is alive.
    # This is all the Kubernetes readiness probe checks.
    return {"status": "ok"}

@app.get("/healthz/db")
async def healthz_db():
    # Opens a fresh connection using the current environment credentials.
    # If the password was rotated and this pod has not restarted yet,
    # the environment still has the old password - this connection fails.
    # /healthz above would still return 200. Your users would see errors.
    try:
        conn = await asyncpg.connect(
            host=DB_HOST, port=DB_PORT,
            database=DB_NAME, user=DB_USERNAME, password=DB_PASSWORD,
        )
        await conn.execute("SELECT 1")
        await conn.close()
        return {"status": "ok", "db": "authenticated"}

    except asyncpg.InvalidPasswordError:
        raise HTTPException(
            status_code=503,
            detail=(
                f"Authentication failed for '{DB_USERNAME}'. "
                "Password may have been rotated. "
                "Readiness probe does not check this."
            )
        )
    except Exception as e:
        raise HTTPException(status_code=503, detail=f"Database error: {str(e)}")
</code></pre>
<p>The difference between these two endpoints is the entire lesson of this use case.</p>
<h3 id="heading-set-up-the-demo-environment">Set Up the Demo Environment</h3>
<p>Navigate to <code>04-secrets-rotation/</code> and run the setup script:</p>
<pre><code class="language-plaintext">cd 04-secrets-rotation
./setup.sh
</code></pre>
<p>This starts a Kind cluster, deploys real PostgreSQL with the <code>appuser</code> account already created, deploys the demo FastAPI app connected to it, and creates an initial secret in AWS Secrets Manager.</p>
<p>Once setup completes, install the dependencies:</p>
<pre><code class="language-plaintext">pip install boto3 kubernetes
</code></pre>
<p>Before running the rotation, confirm everything is running:</p>
<pre><code class="language-plaintext">kubectl get pods
</code></pre>
<p>You should see <code>myapp</code> and <code>postgres</code> pods both in the Running state. If any pod shows Pending or Error, wait 30 seconds and check again. PostgreSQL takes a moment to finish initialising.</p>
<p>You can also verify that the secret was created in AWS. In the console, go to AWS Secrets Manager and look for <code>myapp/db-credentials</code>:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/c5d481f7-c938-43f8-91ec-09640c137897.png" alt="screenshot showing AWS secret created" style="display: block;" width="1414" height="337" loading="lazy">

<p>If you prefer the CLI:</p>
<pre><code class="language-plaintext">aws secretsmanager get-secret-value --secret-id myapp/db-credentials
</code></pre>
<p>Once both pods are Running and the secret exists, run the rotation to see the full path:</p>
<pre><code class="language-plaintext">python rotate_secret.py
</code></pre>
<p><strong>If Step 6 shows FAILED on this first clean run</strong>, it's almost always a timing issue: the app pod restarted successfully but <code>/healthz/db</code> ran before the new pod finished establishing its first database connection. Wait 20 seconds and run <code>python rotate_secret.py</code> again. If it fails repeatedly, run <code>kubectl logs deployment/myapp</code> to see what the app is reporting.</p>
<p>You should see all six steps complete cleanly, ending with:</p>
<pre><code class="language-plaintext">Rotation complete. Credential verified at the application level.
  AWS Secrets Manager: updated
  PostgreSQL:          updated (ALTER USER)
  Kubernetes Secret:   updated
  Application pod:     restarted, authenticated
</code></pre>
<p>The lab is alive and the full rotation chain works end to end. Now let's look at what the script is doing.</p>
<h3 id="heading-the-script-code-files">The Script (<a href="https://github.com/Osomudeya/devops-scripting-labs">Code Files</a>)</h3>
<pre><code class="language-python"># rotate_secret.py
import boto3
import base64
import json
import subprocess
import sys
from kubernetes import client, config


def get_current_secret(secret_name):
    """
    Fetch the current credential from AWS Secrets Manager.
    The secret is stored as a JSON string with 'username' and 'password' fields.
    """
    sm = boto3.client("secretsmanager")
    response = sm.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])


def rotate_in_aws(secret_name, username, new_password):
    """
    Write the new credential to AWS Secrets Manager.
    put_secret_value creates a new version - the previous version is
    not deleted immediately, giving you a short rollback window.
    """
    sm = boto3.client("secretsmanager")
    new_value = json.dumps({"username": username, "password": new_password})
    sm.put_secret_value(SecretId=secret_name, SecretString=new_value)
    print("  [AWS] Secret updated in Secrets Manager.")


def update_kubernetes_secret(namespace, k8s_secret_name, username, new_password):
    """
    Patch the Kubernetes Secret object with the new credential values.
    Kubernetes requires secret data to be base64-encoded - this is encoding,
    not encryption. Anyone with access to the Secret object can decode the values.
    Real encryption at rest requires separate etcd encryption configuration.
    """
    config.load_kube_config()
    v1 = client.CoreV1Api()

    secret_data = {
        "username": base64.b64encode(username.encode()).decode(),
        "password": base64.b64encode(new_password.encode()).decode()
    }

    v1.patch_namespaced_secret(
        name=k8s_secret_name,
        namespace=namespace,
        body={"data": secret_data}
    )
    print(f"  [K8s] Kubernetes Secret '{k8s_secret_name}' updated.")


def rolling_restart(namespace, deployment_name):
    """
    Trigger a rolling restart of the deployment.
    Rolling restart means Kubernetes creates one new pod, waits for it to pass
    its readiness probe, then terminates one old pod - and repeats until all
    pods have been replaced. Availability is preserved throughout.
    This is very different from deleting all pods at once.
    """
    result = subprocess.run(
        ["kubectl", "rollout", "restart",
         f"deployment/{deployment_name}", "-n", namespace],
        capture_output=True, text=True
    )
    if result.returncode != 0:
        raise RuntimeError(f"Rolling restart failed: {result.stderr}")
    print(f"  [K8s] Rolling restart triggered for '{deployment_name}'.")


def wait_for_rollout(namespace, deployment_name, timeout=120):
    """
    Block until the rolling restart finishes or times out.
    'Finished' means all new pods are Running and their readiness probes passed.
    This does NOT mean the application can authenticate with the new credential.
    That is what verify_credential checks next.
    """
    print(f"  [K8s] Waiting for rollout (timeout: {timeout}s)...")
    result = subprocess.run(
        ["kubectl", "rollout", "status",
         f"deployment/{deployment_name}",
         "-n", namespace,
         f"--timeout={timeout}s"],
        capture_output=True, text=True
    )
    if result.returncode != 0:
        raise RuntimeError(f"Rollout did not complete: {result.stderr}")
    print("  [K8s] Rollout complete. All pods report Ready.")


def verify_credential(namespace, deployment_name):
    """
    This is the check the readiness probe does not make.
    We exec into the running pod and call /healthz/db - an endpoint that
    makes an actual authenticated query to the database.
    If this passes: the credential is working at the application level.
    If this fails after the readiness probe passed: the contract mismatch is confirmed.
    The pod is Running. The application cannot serve database requests.
    """
    print("  [Verify] Running post-rotation credential check...")

    result = subprocess.run(
        ["kubectl", "get", "pods", "-n", namespace,
         "-l", f"app={deployment_name}",
         "-o", "jsonpath={.items[0].metadata.name}"],
        capture_output=True, text=True
    )
    pod_name = result.stdout.strip()

    if not pod_name:
        print("  [Verify] ERROR: No running pod found for this deployment.")
        return False

    verify = subprocess.run(
        ["kubectl", "exec", pod_name, "-n", namespace,
         "--", "curl", "-sf", "http://localhost:8000/healthz/db"],
        capture_output=True, text=True
    )

    if verify.returncode != 0:
        print("  [Verify] FAILED - Pod is Running but database authentication failed.")
        print("           The readiness probe validated HTTP reachability.")
        print("           The application cannot authenticate with the new credential.")
        print("           These are two different contracts. Only one was checked automatically.")
        return False

    print("  [Verify] PASSED - Application confirmed it can authenticate with the new credential.")
    return True


def rotate(secret_name, new_password, namespace, k8s_secret_name, deployment_name):
    print("\n[Step 1/6] Reading current secret from AWS Secrets Manager...")
    current = get_current_secret(secret_name)
    username = current["username"]

    print("[Step 2/6] Updating AWS Secrets Manager...")
    rotate_in_aws(secret_name, username, new_password)

    print("[Step 3/6] Rotating password at the database level (ALTER USER)...")
    rotate_postgres_password(namespace, new_password)

    print("[Step 4/6] Updating Kubernetes Secret object...")
    update_kubernetes_secret(namespace, k8s_secret_name, username, new_password)

    print("[Step 5/6] Triggering rolling restart...")
    rolling_restart(namespace, deployment_name)
    wait_for_rollout(namespace, deployment_name)

    print("[Step 6/6] Verifying the new credential works at the application level...")
    success = verify_credential(namespace, deployment_name)

    print("\n" + "=" * 60)
    if success:
        print("Rotation complete. Credential verified at the application level.")
    else:
        print("Rotation incomplete. Readiness probe passed but credential verification failed.")
        print("Recommended action: force-restart all pods to flush the connection pool,")
        print("or investigate the database session timeout configuration.")
        sys.exit(1)


if __name__ == "__main__":
    import secrets as _secrets
    rotate(
        secret_name="myapp/db-credentials",
        new_password=_secrets.token_urlsafe(16),
        namespace="default",
        k8s_secret_name="db-credentials",
        deployment_name="myapp"
    )
</code></pre>
<h3 id="heading-how-the-script-works">How the Script Works</h3>
<p><code>get_current_secret</code> reads the current credential from AWS Secrets Manager so the script knows the username before it generates a new password.</p>
<p><code>rotate_in_aws</code> writes the new credential to Secrets Manager. It creates a new version rather than overwriting the old one, so you have a short window to roll back if something goes wrong.</p>
<p><code>_pg_password_literal</code> and <code>rotate_postgres_password</code> handle the step that most rotation scripts skip, which is actually changing the password inside PostgreSQL. This is done by running <code>ALTER USER appuser PASSWORD '...'</code> directly on the live PostgreSQL pod. Before this step, the database still accepts the old password. After this step, it does not.</p>
<p><code>update_kubernetes_secret</code> writes the new password into the Kubernetes Secret so that any new pod that starts will get the new credential from the beginning.</p>
<p><code>rolling_restart</code> and <code>wait_for_rollout</code> restart the application pods one at a time so the deployment stays available throughout. When this step completes, all pods are Running and their readiness probes have passed – but keep in mind that "Running" only means <code>/healthz</code> returned 200, which is exactly the problem this use case is about.</p>
<p><code>verify_credential</code> is the extra step Kubernetes never runs. It reaches inside the new pod and calls <code>/healthz/db</code>, which opens a real database connection with the credentials in the pod's current environment. If this passes, the rotation is genuinely complete. If this fails after the readiness probe passed, you have confirmed the gap: the pod looks healthy but can't serve database requests.</p>
<h3 id="heading-what-the-output-looks-like">What the Output Looks Like</h3>
<p>Successful rotation:</p>
<pre><code class="language-plaintext">[Step 1/6] Reading current secret from AWS Secrets Manager...
[Step 2/6] Updating AWS Secrets Manager...
  [AWS] Secrets Manager updated.
[Step 3/6] Rotating password at the database level (ALTER USER)...
  [DB]  Running ALTER USER on PostgreSQL...
  [DB]  Password changed at the database level.
        New connections now require the new password.
        Existing pool connections remain valid until they close.
[Step 4/6] Updating Kubernetes Secret object...
  [K8s] Kubernetes Secret 'db-credentials' updated.
[Step 5/6] Triggering rolling restart...
  [K8s] Rolling restart triggered for 'myapp'.
  [K8s] Waiting for rollout (timeout: 120s)...
  [K8s] Rollout complete. All pods report Ready.
[Step 6/6] Verifying the new credential works at the application level...
  [Verify] Running post-rotation credential check...
  [Verify] PASSED - Application confirmed it can authenticate with the new credential.

============================================================
Rotation complete. Credential verified at the application level.
  AWS Secrets Manager: updated
  PostgreSQL:          updated (ALTER USER)
  Kubernetes Secret:   updated
  Application pod:     restarted, authenticated
</code></pre>
<p>The lab is alive and the full rotation chain works end to end.</p>
<p>Before you break anything, confirm the pod is healthy:</p>
<pre><code class="language-plaintext">kubectl get pods
</code></pre>
<p>You should see <code>myapp</code> in Running state. That is the baseline: everything working as expected. Now let's break it.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/caabd892-562a-40f4-81c1-2c59fd15240b.png" alt="terminal screenshot showing output of 'kubectl get pods&quot;" style="display: block;" width="753" height="187" loading="lazy">

<h3 id="heading-break-it-on-purpose">Break it On Purpose</h3>
<h4 id="heading-step-1-desync-the-db">Step 1: Desync the DB</h4>
<pre><code class="language-plaintext">./break_it.sh
</code></pre>
<p>This runs <code>ALTER USER</code> directly on PostgreSQL with a wrong password. The K8s Secret still has the old password, so the pod's environment and the database are now out of sync.</p>
<h4 id="heading-step-2-check-what-kubernetes-sees">Step 2: Check what Kubernetes sees</h4>
<pre><code class="language-plaintext">kubectl exec deployment/myapp -- curl -s http://localhost:8000/healthz
</code></pre>
<p>You will see <code>{"status":"ok"}</code>. The pod is still showing Ready in <code>kubectl get pods</code>. Kubernetes has no idea anything is wrong – that's the contract gap made visible in your terminal.</p>
<h4 id="heading-step-3-check-what-your-users-experience">Step 3: Check what your users experience</h4>
<pre><code class="language-plaintext">kubectl exec deployment/myapp -- curl -s http://localhost:8000/healthz/db
</code></pre>
<p>You'll see a <code>503</code> error. Fresh database connections are failing. Your users are already seeing this.</p>
<h4 id="heading-step-4-see-the-mixed-pattern-optional">Step 4: See the mixed pattern (optional)</h4>
<pre><code class="language-plaintext">./load_test.sh
</code></pre>
<p>Some requests succeed because they hit old pool connections that were authenticated before the break. Some fail because they need a fresh connection. The pod looks healthy, but half your traffic is failing.</p>
<h4 id="heading-step-5-run-the-rotation-script">Step 5: Run the rotation script</h4>
<pre><code class="language-plaintext">python rotate_secret.py
</code></pre>
<p>This time, Step 6 catches the failure. Here's what you'll see:</p>
<pre><code class="language-plaintext">[Step 5/6] Triggering rolling restart...
  [K8s] Rollout complete. All pods report Ready.
[Step 6/6] Verifying the new credential works at the application level...
  [Verify] Running post-rotation credential check...
  [Verify] FAILED - Pod is Running but database authentication failed.
           The readiness probe validated HTTP reachability.
           The application cannot authenticate with the new credential.
           These are two different contracts. Only one was checked automatically.

============================================================
Rotation incomplete. Readiness probe passed but credential verification failed.
</code></pre>
<p>The pod is Running and shows Ready in <code>kubectl get pods</code>. The rotation script says the credential is broken. That's the contract gap visible in your terminal, caught before your users hit it.</p>
<p><strong>The lesson:</strong> <code>/healthz</code> tells you the HTTP server is alive. <code>/healthz/db</code> tells you the application can actually connect to the database. Kubernetes only checks the first one unless you add a database probe. The rotation script adds that check at the end of every rotation so you catch the failure before your users do.</p>
<h3 id="heading-the-decision-the-script-cant-make-for-you">The Decision the Script Can't Make For You</h3>
<p>The verification failed, the pod is Running, and requests to the database are failing. You have two options:</p>
<ol>
<li><p>force-restart all pods at once to flush the connection pool (which is faster but causes a brief capacity reduction), or</p>
</li>
<li><p>wait for old sessions to expire naturally (which avoids downtime but leaves requests failing intermittently until the pool cycles).</p>
</li>
</ol>
<p>The script found the problem, but deciding what to do next belongs to an engineer who knows the system.</p>
<h3 id="heading-teardown">Teardown</h3>
<pre><code class="language-plaintext">./teardown.sh
</code></pre>
<h2 id="heading-use-case-5-automated-canary-rollback-trigger">Use Case 5 - Automated Canary Rollback Trigger</h2>
<p><strong>Environment:</strong> Fully local – Kind, Prometheus via Helm<br><strong>Language:</strong> Bash</p>
<h3 id="heading-what-this-use-case-does-and-why-it-matters">What This Use Case Does and Why it Matters</h3>
<p>This use case runs a script that watches your new deployment and automatically rolls it back if something goes wrong, before your users flood your support queue.</p>
<p>This matters in production because, when you ship a new version, you don't send all traffic to it immediately. You send a small slice, say 20% to the new version while 80% still goes to the old one. If the new version is broken, only 20% of users are affected and you can roll back before the damage spreads. But the rollback only works if you're watching the right things.</p>
<p><strong>The takehome:</strong> Two scripts watch the same failing canary. One reports everything is fine. The other fires the rollback. The only difference is what they measure. Your automation is only as good as what it watches.</p>
<p><strong>What to watch for:</strong> <code>canary_watch_v1.sh</code> watches errors only and stays silent while the canary is slow. <code>canary_watch_v2.sh</code> watches errors AND response time and fires the rollback. The difference between them is the lesson.</p>
<p><strong>What this is not:</strong> This isn't a guide to canary deployments. It's about what your monitoring misses when it only watches one signal.</p>
<h3 id="heading-how-it-works">How it Works</h3>
<p>Three things run in the cluster: the stable app (three pods, handles most traffic), the canary app (one pod, handles a small slice), and Prometheus (collects response times and error counts from both every 15 seconds).</p>
<p>The watch script asks Prometheus every 15 seconds: <em>"Is the canary behaving normally?"</em> If the answer is no for three checks in a row, it rolls back the canary automatically.</p>
<p>The question is that what does <em>"behaving normally"</em> mean? That is the entire use case.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/204f992e-44c2-4404-a6ff-f2279ea23aeb.png" alt="terminal screenshot showing output result of 'kubectl get pods&quot;" style="display: block;" width="753" height="187" loading="lazy">

<h3 id="heading-set-up-the-demo-environment">Set Up the Demo Environment</h3>
<p>Navigate to <code>05-canary-rollback/</code> and run:</p>
<pre><code class="language-plaintext">cd 05-canary-rollback
./setup.sh
</code></pre>
<p>Setup takes a few minutes. It installs Prometheus, deploys both versions of the demo app, and starts a load generator pod that sends continuous traffic to both so Prometheus always has data.</p>
<p>When setup finishes, confirm everything is running:</p>
<pre><code class="language-plaintext">kubectl get pods
</code></pre>
<p>You should see output like this:</p>
<pre><code class="language-plaintext">NAME                                                   READY   STATUS    RESTARTS   AGE
load-generator-68c59698b7-kws2l                        1/1     Running   0          4m54s
myapp-canary-6d6979c66f-g9lgw                          1/1     Running   0          32s
myapp-stable-6bcf994fc4-b4k9l                          1/1     Running   0          4m55s
myapp-stable-6bcf994fc4-ndhxc                          1/1     Running   0          4m55s
myapp-stable-6bcf994fc4-z97kx                          1/1     Running   0          4m55s
prometheus-kube-prometheus-operator-59b847d96c-mp72s   1/1     Running   0          5m58s
prometheus-prometheus-kube-prometheus-prometheus-0     2/2     Running   0          5m1s
</code></pre>
<p>Three stable pods, one canary pod, one load generator, Prometheus running. The lab is alive.</p>
<p><strong>Wait 60 seconds before running anything else.</strong> Prometheus needs time to scrape the first metrics from the pods. If you skip this, the watch scripts return empty data with no explanation.</p>
<h3 id="heading-three-terminal-windows">Three Terminal Windows</h3>
<p>You need three separate command prompts running at the same time.</p>
<p><strong>On macOS:</strong> open Terminal and press <code>Cmd+T</code> twice. You now have three tabs, each an independent terminal.<br><strong>On Linux:</strong> press <code>Ctrl+Shift+T</code> in most terminal apps, or right-click and choose "Open new tab."</p>
<p>Label them Terminal 1 for the watch script, Terminal 2 for injecting failures, Terminal 3 for watching latency.</p>
<h3 id="heading-the-scripts">The Scripts</h3>
<h4 id="heading-version-1-watches-errors-only-code-here">Version 1: watches errors only (<a href="https://github.com/Osomudeya/devops-scripting-labs.git">code here</a>)</h4>
<pre><code class="language-bash">#!/usr/bin/env bash
# canary_watch_v1.sh

PROMETHEUS="http://localhost:9090"
DEPLOYMENT="myapp-canary"
NAMESPACE="default"
ERROR_THRESHOLD="0.05"
CHECK_INTERVAL=15
STRIKE_LIMIT=3

strikes=0

echo "Canary monitor running (v1 - error rate only)."
echo "Rollback triggers if error rate exceeds \({ERROR_THRESHOLD} for \){STRIKE_LIMIT} checks."
echo ""

while true; do
    ts=$(date '+%Y-%m-%dT%H:%M:%S')

    error_query='sum(rate(http_requests_total{app="myapp-canary",status=~"5.."}[1m])) / sum(rate(http_requests_total{app="myapp-canary"}[1m]))'

    error_rate=\((curl -sf "\){PROMETHEUS}/api/v1/query" \
        --data-urlencode "query=${error_query}" | \
        python3 -c "
import sys, json
d = json.load(sys.stdin)
result = d['data']['result']
print(result[0]['value'][1] if result else '0')
" 2&gt;/dev/null)

    error_rate=${error_rate:-0}
    above=\((echo "\)error_rate &gt; $ERROR_THRESHOLD" | bc -l)

    echo "[\(ts] error_rate=\){error_rate} | threshold=\({ERROR_THRESHOLD} | breach=\)([ "$above" = "1" ] &amp;&amp; echo YES || echo NO)"

    if [ "$above" = "1" ]; then
        strikes=$((strikes + 1))
        echo "  Strike \({strikes}/\){STRIKE_LIMIT}"
        if [ "\(strikes" -ge "\)STRIKE_LIMIT" ]; then
            echo "  ROLLBACK TRIGGERED"
            kubectl rollout undo deployment/"\({DEPLOYMENT}" -n "\){NAMESPACE}"
            exit 0
        fi
    else
        strikes=0
    fi

    sleep "${CHECK_INTERVAL}"
done
</code></pre>
<h4 id="heading-version-2-watches-error-rate-and-response-time">Version 2: watches error rate AND response time</h4>
<pre><code class="language-bash">#!/usr/bin/env bash
# canary_watch_v2.sh

PROMETHEUS="http://localhost:9090"
DEPLOYMENT="myapp-canary"
NAMESPACE="default"
ERROR_THRESHOLD="0.05"
LATENCY_THRESHOLD="2.0"
CHECK_INTERVAL=15
STRIKE_LIMIT=3

strikes=0

echo "Canary monitor running (v2 - error rate + P99 latency)."
echo "Error threshold: \({ERROR_THRESHOLD} | Latency P99 threshold: \){LATENCY_THRESHOLD}s"
echo ""

while true; do
    ts=$(date '+%Y-%m-%dT%H:%M:%S')

    error_query='sum(rate(http_requests_total{app="myapp-canary",status=~"5.."}[1m])) / sum(rate(http_requests_total{app="myapp-canary"}[1m]))'
    error_rate=\((curl -sf "\){PROMETHEUS}/api/v1/query" \
        --data-urlencode "query=${error_query}" | \
        python3 -c "
import sys, json
d = json.load(sys.stdin)
result = d['data']['result']
print(result[0]['value'][1] if result else '0')
" 2&gt;/dev/null)

    latency_query='histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{app="myapp-canary"}[1m])) by (le))'
    latency=\((curl -sf "\){PROMETHEUS}/api/v1/query" \
        --data-urlencode "query=${latency_query}" | \
        python3 -c "
import sys, json
d = json.load(sys.stdin)
result = d['data']['result']
print(result[0]['value'][1] if result else '0')
" 2&gt;/dev/null)

    error_rate=${error_rate:-0}
    latency=${latency:-0}

    error_breach=\((echo "\)error_rate &gt; $ERROR_THRESHOLD" | bc -l)
    latency_breach=\((echo "\)latency &gt; $LATENCY_THRESHOLD" | bc -l)

    triggered_by=""
    [ "\(error_breach" = "1" ] &amp;&amp; triggered_by="error_rate(\){error_rate})"
    [ "\(latency_breach" = "1" ] &amp;&amp; triggered_by="\){triggered_by:+\({triggered_by}, }latency_p99(\){latency}s)"

    echo "[\(ts] error_rate=\){error_rate} | latency_p99=\({latency}s | breach=\){triggered_by:-none}"

    if [ "\(error_breach" = "1" ] || [ "\)latency_breach" = "1" ]; then
        strikes=$((strikes + 1))
        echo "  Strike \({strikes}/\){STRIKE_LIMIT} | Triggered by: ${triggered_by}"
        if [ "\(strikes" -ge "\)STRIKE_LIMIT" ]; then
            echo ""
            echo "  ROLLBACK TRIGGERED"
            echo "  Signal: ${triggered_by}"
            kubectl rollout undo deployment/"\({DEPLOYMENT}" -n "\){NAMESPACE}"
            exit 0
        fi
    else
        strikes=0
    fi

    sleep "${CHECK_INTERVAL}"
done
</code></pre>
<h3 id="heading-how-the-scripts-work">How the Scripts Work</h3>
<p>The <code>error rate query</code> asks Prometheus: <em>"What fraction of requests to the canary returned an error in the last minute?"</em> A result of <code>0.0</code> means no errors. A result of <code>0.06</code> means 6% of requests are failing, above the 5% threshold. You see this in the output as:</p>
<pre><code class="language-plaintext">error_rate=0.06 | threshold=0.05 | breach=YES
</code></pre>
<p>The <code>latency query</code> asks: <em>"How slow is the slowest 1% of requests to the canary right now?"</em> A result of <code>5.234</code> means 1 in every 100 requests is taking over 5 seconds. You see this as:</p>
<pre><code class="language-plaintext">latency_p99=5.234s | breach=latency_p99(5.234s)
</code></pre>
<p>V1 only runs the first query. V2 runs both. Same canary, same problem, different answers.</p>
<p>The three-strike rule means a single bad check doesn't trigger a rollback – three in a row does. The tradeoff is 45 seconds (three checks at 15 seconds each) of exposure before the rollback fires.</p>
<p>When three strikes hit, the watch script itself runs:</p>
<pre><code class="language-plaintext">kubectl rollout undo deployment/myapp-canary -n default
</code></pre>
<p>That one line is what triggers the rollback. It lives inside <code>canary_watch_v2.sh</code> and runs automatically – you don't have to do anything. The script detects, decides, and acts.</p>
<h3 id="heading-break-it-on-purpose">Break it On Purpose</h3>
<p><strong>In Terminal 1</strong>, start the v1 monitor:</p>
<pre><code class="language-plaintext">./canary_watch_v1.sh
</code></pre>
<p>You will see this repeating every 15 seconds:</p>
<pre><code class="language-plaintext">Canary monitor running (v1 - error rate only).
Rollback triggers if error rate exceeds 0.05 for 3 checks.

[2026-05-17T11:53:12] error_rate=0 | threshold=0.05 | breach=NO
[2026-05-17T11:53:27] error_rate=0 | threshold=0.05 | breach=NO
[2026-05-17T11:53:42] error_rate=0 | threshold=0.05 | breach=NO
</code></pre>
<p><code>breach=NO</code> means the canary looks healthy. Leave this running and move to Terminal 2.</p>
<p><strong>In Terminal 2</strong>, inject latency into the canary:</p>
<pre><code class="language-plaintext">./break_it.sh
</code></pre>
<p>This makes every request to the canary take 5 seconds. Requests still return 200 – no errors, just slowness. You will see:</p>
<pre><code class="language-plaintext">Injecting latency into the canary deployment...
deployment "myapp-canary" successfully rolled out
Latency injection is active.

The canary pod is Running and passing its readiness probe.
Every request to the canary now takes 5 seconds.
Error rate: 0%   |   P99 latency: ~5s
</code></pre>
<p>Now look back at Terminal 1. The v1 monitor keeps printing <code>breach=NO</code>. The canary is taking 5 seconds per request and your monitoring says everything is fine. That's the failure.</p>
<p><strong>In Terminal 3</strong>, see what your users are actually experiencing:</p>
<pre><code class="language-plaintext">./check_latency.sh
</code></pre>
<pre><code class="language-plaintext">TIMESTAMP                   STABLE (ms)   CANARY (ms)   STATUS
---------                   -----------   -----------   ------
2026-05-17T11:55:14         18ms          5008ms        CANARY DEGRADED
2026-05-17T11:55:20         7ms           5008ms        CANARY DEGRADED
2026-05-17T11:55:27         6ms           5008ms        CANARY DEGRADED
</code></pre>
<p>Stable is responding in 6–18 milliseconds. Canary is taking over 5 seconds. Users on the canary are waiting 5 seconds for every page load. The v1 monitor in Terminal 1 still says <code>breach=NO</code>.</p>
<p>This is the lesson: the monitoring and the user experience are completely disconnected. The script isn't broken. It's watching the wrong thing.</p>
<p>Now let's see the fix. Press <code>Ctrl+C</code> in Terminal 1 to stop v1. Start v2 in the same terminal:</p>
<pre><code class="language-plaintext">./canary_watch_v2.sh
</code></pre>
<p>In Terminal 2, re-inject the latency:</p>
<pre><code class="language-plaintext">./break_it.sh
</code></pre>
<p>Watch Terminal 1. V2 catches the latency and fires the rollback after three strikes:</p>
<pre><code class="language-plaintext">Canary monitor running (v2 - error rate + P99 latency).
Error threshold: 0.05 | Latency P99 threshold: 2.0s

[2026-05-15T14:30:00] error_rate=0.0 | latency_p99=0.082s | breach=none
[2026-05-15T14:30:15] error_rate=0.0 | latency_p99=5.234s | breach=latency_p99(5.234s)
  Strike 1/3 | Triggered by: latency_p99(5.234s)
[2026-05-15T14:30:30] error_rate=0.0 | latency_p99=5.891s | breach=latency_p99(5.891s)
  Strike 2/3 | Triggered by: latency_p99(5.891s)
[2026-05-15T14:30:45] error_rate=0.0 | latency_p99=6.102s | breach=latency_p99(6.102s)
  Strike 3/3 | Triggered by: latency_p99(6.102s)

  ROLLBACK TRIGGERED
  Signal: latency_p99(6.102s)

deployment.apps/myapp-canary rolled back
</code></pre>
<p>The error rate never moved from 0. V2 rolled back anyway because latency crossed the threshold. That's the difference one extra measurement makes.</p>
<p>After the rollback, confirm the canary is dormant but not deleted:</p>
<pre><code class="language-plaintext">kubectl rollout history deployment/myapp-canary -n default
</code></pre>
<pre><code class="language-plaintext">REVISION  CHANGE-CAUSE
1         &lt;none&gt;
2         &lt;none&gt;
</code></pre>
<p>Two revisions. The rollback scaled revision 2 down to zero and restored revision 1. Nothing was deleted, and you can re-deploy if you decide the rollback was a false alarm.</p>
<h3 id="heading-the-decision-the-script-cant-make-for-you">The Decision the Script Can't Make For You</h3>
<p>V2 rolled back based on latency with zero errors. Before re-deploying, ask if the latency was a real regression in the new code, or a temporary spike, like a database cache warming up on first use? Both produce the same signal. Only you know which is more likely given what changed.</p>
<p>False positive rollbacks slow down deployments and erode confidence in automation. The right thresholds depend on your users and your system.<br>What the script enforces is whatever you configure.</p>
<h3 id="heading-teardown">Teardown</h3>
<pre><code class="language-plaintext">./teardown.sh
</code></pre>
<h2 id="heading-what-you-can-do-now">What You Can Do Now</h2>
<p>Each use case in this handbook was a script solving a specific problem the standard tooling wasn't catching. Here's where you land:</p>
<p>You can catch AWS cost spikes before the invoice and you know that the service label is AWS's attribution, not a pointer to what actually caused the cost. Start from what changed operationally, not from the billing label.</p>
<p>You can reconstruct the full timeline of any failed request across multiple services from a single trace ID, and you know that a missing service in that timeline is evidence, not just an absence.</p>
<p>You can detect infrastructure drift by comparing what Terraform believes against what AWS actually contains, and you know that a clean result means the resources Terraform manages are in sync, not that your entire AWS account is clean.</p>
<p>You can validate a secret rotation at the application level, not just at the infrastructure level, and you know the difference between a readiness probe passing and the application actually being able to connect to the database.</p>
<p>You can build a canary rollback trigger that watches the right signals, and you know why watching only error rates can leave a slow, broken deployment running while users wait.</p>
<p>The pattern across all five use cases is the same: the standard tooling reported everything as fine while something was actually broken. The cost script returned clean, the pod showed Running, and the canary showed zero errors – not because the tools were wrong but because they were only checking what was easy to check. These scripts check what the standard tooling skips.</p>
<p><strong>GitHub repo:</strong> <a href="https://github.com/Osomudeya/devops-scripting-labs.git">https://github.com/Osomudeya/devops-scripting-labs</a></p>
<p>I write about DevOps weekly, covering real systems, interview, CV tips and tricks, and real incidents – <a href="https://osomudeya.kit.com/23db7ca59f"><strong>Join the newsletter</strong></a>.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Connect Your AI Coding Agent to a Browser on macOS  ]]>
                </title>
                <description>
                    <![CDATA[ AI coding agents like Claude Code, Cursor, and the rest have gotten remarkably good at reading and writing code. But the moment they need to look at something on the web, they hit a wall. They can't s ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-connect-your-ai-coding-agent-to-a-browser-on-macos/</link>
                <guid isPermaLink="false">6a1594c1da253d50d4ae1277</guid>
                
                    <category>
                        <![CDATA[ AI ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ macOS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Open Source ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Developer Tools ]]>
                    </category>
                
                    <category>
                        <![CDATA[ agentic AI ]]>
                    </category>
                
                    <category>
                        <![CDATA[ mcp ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ אחיה כהן ]]>
                </dc:creator>
                <pubDate>Tue, 26 May 2026 12:40:33 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5fc16e412cae9c5b190b6cdd/7e77f1c5-6942-4dbe-a3c6-ca74cc4354e5.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>AI coding agents like Claude Code, Cursor, and the rest have gotten remarkably good at reading and writing code. But the moment they need to <em>look at something on the web</em>, they hit a wall. They can't see your staging site. They can't read the error in your analytics dashboard. They can't check whether the form they just built actually submits.</p>
<p>The usual fix is to hand the agent a headless browser — Puppeteer or Playwright driving a fresh Chromium instance. That works, sort of. But a headless Chromium starts every session as a stranger: no logins, no cookies, no sessions. It spins up a second browser engine that pushes your CPU and spins up your fan. And a growing number of sites simply block it on sight.</p>
<p>There's another option, and on a Mac it's a good one: let the agent drive the <strong>Safari you already use</strong> — the one that's already logged into GitHub, your analytics, your staging environment. That's what Safari MCP does. It's an open-source MCP server that exposes Safari to any MCP-capable agent through around 80 tools, with no Chromium, no WebDriver, and no separate browser to babysit.</p>
<p>In this tutorial you'll connect Safari MCP to an AI agent, run your first automation, and then build something a headless browser fundamentally cannot do: an automation that works inside a page you're logged into. By the end you'll understand not just <em>how</em> to wire this up, but <em>when</em> native browser automation is the right call — and when it isn't.</p>
<p>Here's what you'll need:</p>
<ul>
<li><p>A Mac (Safari MCP is macOS-only — more on that trade-off later)</p>
</li>
<li><p>Node.js 18 or newer</p>
</li>
<li><p>An MCP-capable AI agent — this tutorial uses Claude Code and Cursor, but any MCP client works</p>
</li>
</ul>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-what-is-mcp-and-why-does-browser-automation-need-it">What is MCP, and Why Does Browser Automation Need It?</a></p>
</li>
<li><p><a href="#heading-why-safari-instead-of-chrome-or-playwright">Why Safari Instead of Chrome or Playwright?</a></p>
</li>
<li><p><a href="#heading-installing-safari-mcp">Installing Safari MCP</a></p>
</li>
<li><p><a href="#heading-your-first-automation-reading-a-page">Your First Automation: Reading a Page</a></p>
</li>
<li><p><a href="#heading-the-payoff-automating-a-logged-in-workflow">The Payoff: Automating a Logged-in Workflow</a></p>
</li>
<li><p><a href="#heading-handling-the-tricky-parts">Handling the Tricky Parts</a></p>
</li>
<li><p><a href="#heading-limitations-when-not-to-use-this">Limitations: When Not to Use This</a></p>
</li>
<li><p><a href="#heading-wrapping-up">Wrapping Up</a></p>
</li>
</ul>
<h2 id="heading-what-is-mcp-and-why-does-browser-automation-need-it">What is MCP, and Why Does Browser Automation Need It?</h2>
<p>Before wiring anything up, it helps to know what the "MCP" in Safari MCP stands for.</p>
<p><strong>MCP</strong> is the Model Context Protocol — an open standard for connecting AI agents to external tools and data. Think of it the way you'd think of a USB port. Before USB, every device needed its own connector. MCP is the equivalent of agreeing on one connector: an agent that speaks MCP can use <em>any</em> tool that speaks MCP, with no custom integration code on either side.</p>
<p>An MCP <strong>server</strong> exposes a set of tools. An MCP <strong>client</strong> — your AI agent — discovers those tools and calls them. The server describes each tool (its name, what it does, what arguments it takes) and the agent decides when to call it. When Claude Code decides it needs to read a web page, it doesn't run browser code itself. It calls a tool that some MCP server provides.</p>
<p>Browser automation is a natural fit for this model. The agent's job is reasoning — "I need to see what's on the staging site, then check the console for errors." The actual mechanics — open a tab, wait for load, read the DOM, capture console output — are well-defined operations that belong behind a stable interface. That interface is exactly what an MCP server provides.</p>
<p>Safari MCP is one such server. It runs as a local process, exposes around 80 browser tools (navigate, click, fill, read, screenshot, extract, and more), and any MCP client can drive it. The agent never touches AppleScript or WebKit internals. It just calls <code>safari_navigate</code> and gets a result.</p>
<p>The "USB port" framing matters for a practical reason: nothing in this tutorial is Claude-specific. Wire Safari MCP into Cursor, Cline, Windsurf, or your own MCP client and the tools are identical.</p>
<h2 id="heading-why-safari-instead-of-chrome-or-playwright">Why Safari Instead of Chrome or Playwright?</h2>
<p>If you've automated a browser before, you've almost certainly used Chrome through Puppeteer, Playwright, or Selenium. So why reach for Safari?</p>
<p>It comes down to three differences that matter once an <em>AI agent</em>, not a test script, is the thing driving the browser.</p>
<p><strong>1. It's your real browser, with your real sessions.</strong> A headless Chromium launched by Playwright is a clean room. It has never logged into anything. If you want your agent to read your analytics dashboard, you first have to solve authentication — store credentials somewhere, script the login, handle two-factor prompts, refresh tokens. Safari MCP skips all of that. It drives the Safari instance you use every day, which is <em>already</em> logged into your dashboards, your GitHub, your email. The agent inherits those sessions for free.</p>
<p><strong>2. It doesn't melt your laptop.</strong> A headless Chromium is a second, full browser engine running alongside the browser you already have open. On a laptop that's real CPU, real memory, and a fan you can hear. Safari MCP uses the WebKit engine that's already running on every Mac — there's no second engine to start. The project measures this at roughly 60% less CPU for the browsing work, and the automation runs with Safari in the background, so it doesn't steal your screen.</p>
<p><strong>3. Sites don't treat it as a bot.</strong> Headless browsers leak. They expose <code>navigator.webdriver</code>, they ship with telltale automation fingerprints, and bot-detection services — Cloudflare's challenge pages, reCAPTCHA, the WAFs in front of a lot of B2B sites — have gotten very good at spotting them. Your real Safari, driven through the operating system, looks like exactly what it is: a person's browser. (To be clear: this is for automating <em>your own</em> accounts and sites — not for evading access controls you don't own.)</p>
<p>The cost of all this is the obvious one: <strong>Safari MCP is macOS-only.</strong> It's built on WebKit and AppleScript, so there's no Windows or Linux story. If your agent runs on a Linux CI box, this isn't your tool. If it runs on your Mac — which, for a coding agent, it very often does — the trade is a good one. We'll come back to limitations honestly at the end.</p>
<h2 id="heading-installing-safari-mcp">Installing Safari MCP</h2>
<p>Installation is genuinely one command, but there are two Safari settings to flip first. Let's do it in order.</p>
<h3 id="heading-step-1-enable-safaris-developer-features">Step 1 — Enable Safari's developer features</h3>
<p>Safari MCP reads and controls pages by running JavaScript inside Safari. Two settings have to be on:</p>
<ol>
<li><p>Open <strong>Safari → Settings → Advanced</strong> and check <strong>"Show features for web developers."</strong> This reveals the Develop menu.</p>
</li>
<li><p>Open the new <strong>Develop</strong> menu and check <strong>"Allow JavaScript from Apple Events."</strong></p>
</li>
</ol>
<p>That second one is the important one. It's what lets an outside process — the MCP server — ask Safari to run JavaScript on a page. Without it, every tool call fails.</p>
<h3 id="heading-step-2-run-the-server">Step 2 — Run the server</h3>
<pre><code class="language-bash">npx safari-mcp
</code></pre>
<p>That's the whole install. <code>npx</code> fetches the package and runs it; there's nothing to build. The first time an agent calls a tool, macOS will pop up a permission prompt — something like <em>"Terminal wants to control Safari."</em> Click <strong>OK</strong>. That's the standard Automation permission, and you can review it later under <strong>System Settings → Privacy &amp; Security → Automation</strong>.</p>
<p>If you'd rather have it installed permanently:</p>
<pre><code class="language-bash">npm install -g safari-mcp
</code></pre>
<h3 id="heading-step-3-tell-your-agent-about-it">Step 3 — Tell your agent about it</h3>
<p>Your AI agent needs to know the server exists. For <strong>Claude Code</strong>, one command does it:</p>
<pre><code class="language-bash">claude mcp add safari -- npx safari-mcp
</code></pre>
<p>For <strong>Cursor</strong>, create <code>.cursor/mcp.json</code> in your project:</p>
<pre><code class="language-json">{
  "mcpServers": {
    "safari": {
      "command": "npx",
      "args": ["safari-mcp"]
    }
  }
}
</code></pre>
<p>The process is the same for every client — Claude Desktop, Cline, Windsurf, Continue, VS Code. You're telling the agent: "there's an MCP server named <code>safari</code>; start it by running <code>npx safari-mcp</code>."</p>
<p>Restart your agent (or reload its MCP servers) and it will connect. In Claude Code you can confirm with the <code>/mcp</code> command, which lists connected servers and their tools. You should see <code>safari</code> with around 80 tools available.</p>
<p>That's it. Your agent now has a browser.</p>
<h2 id="heading-your-first-automation-reading-a-page">Your First Automation: Reading a Page</h2>
<p>Let's prove the wiring works with the simplest possible task: have the agent read a web page.</p>
<p>In your agent, just ask in plain language:</p>
<blockquote>
<p>"Use the safari tools to open example.com and tell me what the page says."</p>
</blockquote>
<p>Behind that request, the agent makes two tool calls. First it navigates:</p>
<pre><code class="language-json">{ "tool": "safari_navigate", "arguments": { "url": "https://example.com" } }
</code></pre>
<p>Then it reads the content:</p>
<pre><code class="language-json">{ "tool": "safari_read_page", "arguments": {} }
</code></pre>
<p><code>safari_read_page</code> returns the page's title, URL, and text content with the HTML stripped out — exactly the form an LLM wants. The agent gets back something like this:</p>
<pre><code class="language-plaintext">Example Domain
https://example.com/
This domain is for use in illustrative examples in documents. You may
use this domain in literature without prior coordination or asking for
permission.
</code></pre>
<p>And it relays that to you. You just watched your agent browse.</p>
<p>A quick note on <em>how</em> the agent should look at a page, because it changes everything downstream. <code>safari_read_page</code> is great for "what does this say." But when the agent needs to <em>act</em> — click a button, fill a field — text isn't enough. It needs to know what's actually there and how to target it. For that, the better first move is <code>safari_snapshot</code>:</p>
<pre><code class="language-json">{ "tool": "safari_snapshot", "arguments": {} }
</code></pre>
<p>This returns an accessibility-tree view of the page, where every interactive element has a stable <code>ref</code> ID:</p>
<pre><code class="language-plaintext">[textbox ref=0_8] "Full Name" value=""
[combobox ref=0_10] "Subject"
[button ref=0_15] "Submit"
</code></pre>
<p>Those <code>ref</code> IDs are the agent's reliable handles. CSS selectors break when a page re-renders. A snapshot ref stays valid for the life of the page. Keep that in mind — it's the difference between an automation that works once and one that works every time.</p>
<h2 id="heading-the-payoff-automating-a-logged-in-workflow">The Payoff: Automating a Logged-in Workflow</h2>
<p>Reading example.com is a wiring test. Here's the thing a headless browser genuinely cannot do.</p>
<p>Pick a site you're logged into in Safari right now — your analytics, your project board, your CI dashboard. We'll use GitHub, because every developer has an account and the notifications page is a real, mildly annoying chore. The task: <strong>have the agent open your GitHub notifications and summarize what actually needs your attention.</strong></p>
<p>Ask the agent:</p>
<blockquote>
<p>"Open my GitHub notifications, read them, and group them into 'needs a reply' versus 'just FYI'."</p>
</blockquote>
<p>The agent navigates:</p>
<pre><code class="language-json">{ "tool": "safari_navigate", "arguments": { "url": "https://github.com/notifications" } }
</code></pre>
<p>Stop and notice what <em>didn't</em> happen. No login screen. No OAuth dance. No personal access token in an environment variable. Safari is already authenticated as you, so the agent lands directly on your real notifications. A headless Chromium would have hit a login wall here and stopped.</p>
<p>Notification lists load incrementally, so the agent should wait for content before reading. <code>safari_wait_for</code> polls the page until a selector or piece of text appears, or a timeout elapses:</p>
<pre><code class="language-json">{ "tool": "safari_wait_for", "arguments": { "text": "Inbox", "timeout": 10000 } }
</code></pre>
<p>Then it reads. <code>safari_read_page</code> scoped to the notifications region returns the list as clean text:</p>
<pre><code class="language-json">{ "tool": "safari_read_page", "arguments": { "selector": "main" } }
</code></pre>
<p>The agent reasons over that text and hands you the grouped summary. The whole loop — navigate, wait, read, summarize — is a handful of tool calls.</p>
<p>When you need data in a precise shape rather than prose — to feed another step, or to write to a file — the agent can reach for <code>safari_evaluate</code>, which runs custom JavaScript on the page and returns whatever you build:</p>
<pre><code class="language-json">{
  "tool": "safari_evaluate",
  "arguments": {
    "expression": "JSON.stringify([...document.querySelectorAll('li')].map(li =&gt; li.innerText.trim()))"
  }
}
</code></pre>
<p>The agent writes that expression itself, against the structure it just saw in the snapshot — you don't hand-author selectors.</p>
<p>You might be thinking: <em>GitHub has an API, why scrape the page?</em> Fair. For GitHub specifically, the API is excellent. But the point generalizes. Most of the dashboards you stare at every day — your billing portal, your error tracker's specific filtered view, a client's analytics, the admin panel of some tool your company pays for — either have no usable API or would cost you an afternoon of OAuth setup to reach. With Safari MCP, "the page I'm already looking at" <em>is</em> the API. The agent reads what you can see, because it's using the browser you're seeing it in.</p>
<p>That's the capability headless automation can't match. Not speed, not features — <strong>access.</strong></p>
<h2 id="heading-handling-the-tricky-parts">Handling the Tricky Parts</h2>
<p>A first automation always looks easy. Three things tend to bite on the second one.</p>
<h3 id="heading-tab-safety-the-agent-must-not-hijack-your-tabs">Tab Safety — The Agent Must not Hijack Your Tabs</h3>
<p>This is the scariest failure mode: you're typing in a tab, the agent navigates <em>that</em> tab, and your work is gone. Safari MCP guards against it by stamping each automation tab with an identity marker — it uses <code>window.name</code>, which survives page navigations — and resolving "the agent's tab" through that marker on every call. If it can't positively identify its own tab, it refuses to act and raises a re-anchor error rather than guessing.</p>
<p>The practical rule for you: let the agent open its own tab with <code>safari_new_tab</code>, and it will stay in its lane. Don't point it at "the current tab" and assume.</p>
<h3 id="heading-waiting-for-dynamic-content">Waiting for Dynamic Content</h3>
<p>Modern pages render after load. If the agent reads too early, it reads an empty shell. Don't have it guess with fixed sleeps — use <code>safari_wait_for</code>, which polls for a selector or text until it appears or the timeout elapses:</p>
<pre><code class="language-json">{ "tool": "safari_wait_for", "arguments": { "selector": ".results-list", "timeout": 8000 } }
</code></pre>
<p>This is the single most common fix for "the automation works when I step through it slowly but fails when it runs."</p>
<h3 id="heading-framework-forms">Framework Forms</h3>
<p>Set a React or Vue input's <code>.value</code> directly and the framework never notices — its internal state stays empty, and your "filled" form submits blank. Safari MCP's <code>safari_fill</code> and <code>safari_fill_form</code> use the native value setters and dispatch the <code>input</code> and <code>change</code> events the framework listens for, so React, Vue, Angular, and Svelte state all stay in sync:</p>
<pre><code class="language-json">{
  "tool": "safari_fill_form",
  "arguments": {
    "fields": [
      { "selector": "#email", "value": "jane@example.com" },
      { "selector": "#message", "value": "Looks great." }
    ]
  }
}
</code></pre>
<p>For framework-heavy pages where CSS selectors are fragile, go back to the snapshot refs from the previous section — pass <code>{ "ref": "0_9" }</code> instead of <code>{ "selector": "#email" }</code>. Refs survive re-renders; selectors don't.</p>
<p>None of these are exotic. They're just the difference between a demo and an automation you'd actually leave running.</p>
<h2 id="heading-limitations-when-not-to-use-this">Limitations: When Not to Use This</h2>
<p>A tool tutorial that only lists strengths isn't worth much. Here's where Safari MCP is the wrong choice.</p>
<p><strong>It's macOS-only, and that's structural.</strong> Safari MCP is built on WebKit and AppleScript. There's no Windows or Linux port coming, because the foundation doesn't exist on those platforms. If your agent runs in Linux CI, use Playwright.</p>
<p><strong>It drives one Safari, on one Mac.</strong> This is browser automation for <em>your</em> machine — a coding agent working alongside you. It is not a fleet. If you need 50 parallel browsers scraping in a data center, that's a headless-Chromium-in-containers job, and Safari MCP is the wrong shape for it.</p>
<p><strong>Cross-browser test suites should stay on Playwright.</strong> If you're writing end-to-end tests that must pass on Chrome, Firefox, and Safari, use the tool built for that. Safari MCP drives exactly one engine: WebKit.</p>
<p><strong>It shares a browser with you.</strong> Because it uses your real Safari, the agent and you are in the same browser. That's the entire point — but it means you should let the agent work in its own tabs and not fight it for the same window.</p>
<p>The honest summary: Safari MCP is built for one specific situation — an AI agent doing real browser work on the Mac you're sitting at, against sites you're already logged into. In that situation it's hard to beat. Outside it, reach for the headless tools. Knowing which situation you're in is the actual skill.</p>
<h2 id="heading-wrapping-up">Wrapping Up</h2>
<p>You've gone from an AI agent that could only see code to one that can see the web — the real web, behind your real logins.</p>
<p>To recap what you did: you learned what MCP is and why browser automation belongs behind that interface. You saw why a native Safari engine beats a headless Chromium for an agent working on your Mac and you installed Safari MCP with one command and two settings. You ran a first read, and then you did the thing that actually matters — an automation inside a logged-in page, with no auth code at all. Finally, you saw the edges: tab safety, waiting for dynamic content, framework forms, and the cases where you should pick a different tool.</p>
<p>The bigger idea is worth holding onto. An AI agent is only as capable as the tools you connect to it. Giving it a browser — a <em>real</em> one — turns "write me code" into "go look at the staging site, find the bug, and tell me what's wrong." That's a different kind of collaborator.</p>
<p>Safari MCP is open source under the MIT license, and it exposes around 80 tools beyond the handful you used here — screenshots, network inspection, storage, accessibility audits, multi-tab workflows. The repository and full tool reference are at <a href="https://github.com/achiya-automation/safari-mcp">github.com/achiya-automation/safari-mcp</a>. Point your agent at it and see what it does when it can finally look around.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Build a Self-Hosted WhatsApp Bot with n8n and WAHA ]]>
                </title>
                <description>
                    <![CDATA[ WhatsApp is where your many of your customers likely already are. For support tickets, order updates, booking reminders, and lead qualification, a WhatsApp channel often converts several times better  ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-build-a-self-hosted-whatsapp-bot-with-n8n-and-waha/</link>
                <guid isPermaLink="false">6a01e032fca21b0d4b2bb4c1</guid>
                
                    <category>
                        <![CDATA[ whatsapp ]]>
                    </category>
                
                    <category>
                        <![CDATA[ automation ]]>
                    </category>
                
                    <category>
                        <![CDATA[ n8n ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Docker ]]>
                    </category>
                
                    <category>
                        <![CDATA[ self-hosted ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ אחיה כהן ]]>
                </dc:creator>
                <pubDate>Mon, 11 May 2026 13:57:06 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/28affe4d-9359-4cbb-a311-a2ee9d0829c0.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>WhatsApp is where your many of your customers likely already are. For support tickets, order updates, booking reminders, and lead qualification, a WhatsApp channel often converts several times better than email.</p>
<p>But the official WhatsApp Business Cloud API can be slow to onboard, template-restricted for proactive messages, and priced per conversation — which adds up fast at scale.</p>
<p>There's another path: you can run your own WhatsApp HTTP gateway on a small server, connect it to a workflow engine, and keep every message — inbound and outbound — inside infrastructure you control. No monthly conversation fees, no template approvals for routine replies, no third-party middleman holding your customer data.</p>
<p>In this tutorial, you'll build exactly that. By the end, you'll have a WhatsApp bot that:</p>
<ul>
<li><p>Receives every incoming message through a webhook</p>
</li>
<li><p>Routes messages through an n8n workflow</p>
</li>
<li><p>Replies automatically based on keywords, AI, or any API call you want</p>
</li>
<li><p>Runs entirely on your own server, using two open-source tools</p>
</li>
</ul>
<p>You'll use <strong>WAHA</strong> (WhatsApp HTTP API) as the gateway, and <strong>n8n</strong> as the workflow engine. Both run in Docker, both are free for self-hosting, and together they cover everything from a simple auto-reply to a full CRM integration.</p>
<h2 id="heading-table-of-contents">Table of contents</h2>
<ul>
<li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-a-note-on-which-whatsapp-account-to-use">A Note on Which WhatsApp Account to Use</a></p>
</li>
<li><p><a href="#heading-waha-vs-the-official-whatsapp-business-cloud-api">WAHA vs the official WhatsApp Business Cloud API</a></p>
</li>
<li><p><a href="#heading-part-1-understanding-waha">Part 1: Understanding WAHA</a></p>
</li>
<li><p><a href="#heading-part-2-running-waha-with-docker">Part 2: Running WAHA with Docker</a></p>
</li>
<li><p><a href="#heading-part-3-starting-a-whatsapp-session">Part 3: Starting a WhatsApp session</a></p>
</li>
<li><p><a href="#heading-part-4-running-n8n">Part 4: Running n8n</a></p>
</li>
<li><p><a href="#heading-part-5-creating-the-webhook-trigger-in-n8n">Part 5: Creating the Webhook Trigger in n8n</a></p>
</li>
<li><p><a href="#heading-part-6-wiring-waha-to-n8n">Part 6: Wiring WAHA to n8n</a></p>
</li>
<li><p><a href="#heading-part-7-building-the-first-auto-reply">Part 7: Building the first auto-reply</a></p>
</li>
<li><p><a href="#heading-part-8-a-second-example-proactive-booking-confirmations">Part 8: A Second Example — Proactive Booking Confirmations</a></p>
</li>
<li><p><a href="#heading-part-9-going-to-production">Part 9: Going to Production</a></p>
</li>
<li><p><a href="#heading-common-pitfalls">Common Pitfalls</a></p>
</li>
<li><p><a href="#heading-where-to-go-next">Where to Go Next</a></p>
</li>
</ul>
<h2 id="heading-what-youll-learn">What You'll Learn</h2>
<ul>
<li><p>How WAHA works under the hood and when to use it instead of the official Cloud API</p>
</li>
<li><p>How to run WAHA and n8n side by side with Docker Compose</p>
</li>
<li><p>How to scan the QR code and bind a WhatsApp account to your gateway</p>
</li>
<li><p>How to connect WAHA's webhook to an n8n workflow</p>
</li>
<li><p>How to build a keyword-based auto-reply bot</p>
</li>
<li><p>How to send proactive confirmations from a separate workflow</p>
</li>
<li><p>How to harden the setup for production (HTTPS, API keys, rate limits, Queue Mode)</p>
</li>
</ul>
<h2 id="heading-prerequisites">Prerequisites</h2>
<ul>
<li><p>A Linux server (any VPS works — 2 GB of RAM is enough for a small bot)</p>
</li>
<li><p>Docker and Docker Compose installed</p>
</li>
<li><p>A public hostname with DNS pointing at the server, or an ngrok tunnel for local testing</p>
</li>
<li><p>A WhatsApp account you're willing to dedicate to the bot (more on that below)</p>
</li>
<li><p>Basic familiarity with JSON and HTTP requests</p>
</li>
</ul>
<p>You don't need prior n8n experience. If you can drag a box and wire it to another box, you can build the flow.</p>
<h2 id="heading-a-note-on-which-whatsapp-account-to-use">A Note on Which WhatsApp Account to Use</h2>
<p>WAHA works by running an actual WhatsApp Web session inside a headless Chromium process. It logs in as a real account — the same way you would open web.whatsapp.com in your browser. Meta doesn't officially endorse this approach for commercial use at scale, and heavy volume from a single number can lead to a ban.</p>
<p>For that reason, use a dedicated number for the bot. Don't use your personal WhatsApp. Get a second SIM, eSIM, or a VoIP number that supports WhatsApp activation. Keep outbound volume reasonable, and you'll be fine for most small-business use cases.</p>
<p>If you plan to send thousands of marketing messages per day, switch to the official WhatsApp Business Cloud API — that's what it exists for. This tutorial is aimed at the middle ground: support bots, order updates, booking confirmations, and similar conversational flows where you need real-time control without enterprise pricing.</p>
<h2 id="heading-waha-vs-the-official-whatsapp-business-cloud-api">WAHA vs the official WhatsApp Business Cloud API</h2>
<p>Before writing any code, it helps to understand when each option is the right fit.</p>
<table>
<thead>
<tr>
<th>Dimension</th>
<th>WAHA (self-hosted)</th>
<th>WhatsApp Cloud API (Meta)</th>
</tr>
</thead>
<tbody><tr>
<td>Onboarding</td>
<td>Scan a QR code — ready in minutes</td>
<td>Business verification, app review — days to weeks</td>
</tr>
<tr>
<td>Cost</td>
<td>Server cost only</td>
<td>Per-conversation pricing</td>
</tr>
<tr>
<td>Template approval</td>
<td>Not needed</td>
<td>Required for proactive messages outside the 24-hour window</td>
</tr>
<tr>
<td>Session model</td>
<td>One WhatsApp Web session per Core container</td>
<td>Native API, no web session</td>
</tr>
<tr>
<td>Risk</td>
<td>Account ban possible at high unsolicited volume</td>
<td>Rate limits but no ban for normal use</td>
</tr>
<tr>
<td>Vendor lock-in</td>
<td>None — pure open source</td>
<td>Tied to Meta's API and pricing</td>
</tr>
<tr>
<td>Best for</td>
<td>Support bots, small-team workflows, internal tools</td>
<td>High-volume marketing, regulated industries, &gt;100k monthly messages</td>
</tr>
</tbody></table>
<p>Neither is strictly better. If you run a support team for a small business, WAHA is often the pragmatic choice. If you're a bank sending millions of transactional messages, you want the Cloud API. Many teams run both — WAHA for conversational support, Cloud API for bulk transactional traffic.</p>
<h2 id="heading-part-1-understanding-waha">Part 1: Understanding WAHA</h2>
<p>WAHA is an open-source project that wraps WhatsApp Web behind a clean REST API. You <code>POST /api/sendText</code> with a chat ID and a message, and WAHA sends it. You configure a webhook URL, and WAHA <code>POST</code>s to that URL every time a message arrives.</p>
<p>Under the hood, WAHA spawns a Chromium instance, opens WhatsApp Web, and uses an engine (<code>whatsapp-web.js</code>, <code>NOWEB</code>, or <code>GOWS</code>) to automate the session. Your code doesn't see any of that complexity — you just see an HTTP API.</p>
<p>The project ships in two flavors:</p>
<ul>
<li><p><strong>WAHA Core</strong> — free, MIT licensed, one active session per container, community support.</p>
</li>
<li><p><strong>WAHA Plus</strong> — commercial license, multi-session support, priority support, and access to advanced endpoints.</p>
</li>
</ul>
<p>For most developers building a single bot, Core is enough. You can always upgrade later.</p>
<p>Official docs live at <a href="https://waha.devlike.pro/">waha.devlike.pro</a>. Keep that open in another tab — we'll reference specific endpoints as we go.</p>
<h2 id="heading-part-2-running-waha-with-docker">Part 2: Running WAHA with Docker</h2>
<p>Create a fresh directory for the project:</p>
<pre><code class="language-bash">mkdir whatsapp-bot &amp;&amp; cd whatsapp-bot
</code></pre>
<p>Create a <code>docker-compose.yml</code> file:</p>
<pre><code class="language-yaml">services:
  waha:
    image: devlikeapro/waha:latest
    container_name: waha
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - WAHA_DASHBOARD_ENABLED=true
      - WAHA_DASHBOARD_USERNAME=admin
      - WAHA_DASHBOARD_PASSWORD=change-me-now
      - WHATSAPP_API_KEY=super-secret-key-change-me
      - WHATSAPP_DEFAULT_ENGINE=WEBJS
    volumes:
      - ./waha-sessions:/app/.sessions
</code></pre>
<p>A few things to notice:</p>
<ul>
<li><p>The dashboard username and password protect the web UI at <code>http://your-server:3000</code>. Always change the defaults before you expose the port publicly.</p>
</li>
<li><p><code>WHATSAPP_API_KEY</code> is the key every HTTP request to WAHA must include in the <code>X-Api-Key</code> header. Treat it like a database password.</p>
</li>
<li><p><code>WHATSAPP_DEFAULT_ENGINE=WEBJS</code> uses the mature <code>whatsapp-web.js</code> engine. WAHA also supports <code>NOWEB</code> and <code>GOWS</code> engines with different trade-offs — WEBJS is the safest default for a first deployment.</p>
</li>
<li><p>The volume mount persists the session across restarts. Without it, every container rebuild forces you to scan the QR code again.</p>
</li>
</ul>
<p>Start the container:</p>
<pre><code class="language-bash">docker compose up -d
docker compose logs -f waha
</code></pre>
<p>Within about 20 seconds WAHA finishes booting. Visit <code>http://your-server:3000</code> and log in with the dashboard credentials.</p>
<h2 id="heading-part-3-starting-a-whatsapp-session">Part 3: Starting a WhatsApp session</h2>
<p>WAHA calls each WhatsApp account a "session." You can have one session at a time on WAHA Core.</p>
<p>From the dashboard, click <strong>Start New Session</strong> and name it <code>default</code>. WAHA displays a QR code.</p>
<p>On your phone:</p>
<ol>
<li><p>Open WhatsApp.</p>
</li>
<li><p>Tap the three-dot menu (Android) or Settings (iOS).</p>
</li>
<li><p>Tap Linked Devices → Link a Device.</p>
</li>
<li><p>Point the camera at the QR code on your screen.</p>
</li>
</ol>
<p>Within a few seconds the dashboard shows <code>WORKING</code> status. Your session is live.</p>
<p>You can also do this over the API. Start the session (<code>default</code> is the session name, encoded in the URL path):</p>
<pre><code class="language-bash">curl -X POST http://your-server:3000/api/sessions/default/start \
  -H "X-Api-Key: super-secret-key-change-me"
</code></pre>
<p>The call is idempotent — if the session is already running, nothing happens.</p>
<p>Fetch the QR as a PNG:</p>
<pre><code class="language-bash">curl http://your-server:3000/api/default/auth/qr \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Accept: image/png" \
  --output qr.png
</code></pre>
<p>Scan and you're in.</p>
<p>Test that the session works by sending a message to yourself:</p>
<pre><code class="language-bash">curl -X POST http://your-server:3000/api/sendText \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "session": "default",
    "chatId": "15555550123@c.us",
    "text": "Hello from WAHA!"
  }'
</code></pre>
<p>Replace <code>15555550123</code> with your own number (country code plus number, no <code>+</code>, no spaces, no dashes). The <code>@c.us</code> suffix marks it as an individual chat. Groups use <code>@g.us</code>.</p>
<p>If the message lands on your phone — congratulations. The gateway works.</p>
<h2 id="heading-part-4-running-n8n">Part 4: Running n8n</h2>
<p>Add an <code>n8n</code> service to your <code>docker-compose.yml</code> alongside WAHA:</p>
<pre><code class="language-yaml">services:
  waha:
    # ... existing config

  n8n:
    image: n8nio/n8n:latest
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=n8n.example.com
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - WEBHOOK_URL=https://n8n.example.com/
      - GENERIC_TIMEZONE=UTC
    volumes:
      - ./n8n-data:/home/node/.n8n
</code></pre>
<p>Replace <code>n8n.example.com</code> with your real domain. For purely local testing, set:</p>
<pre><code class="language-yaml">- N8N_HOST=localhost
- N8N_PROTOCOL=http
- WEBHOOK_URL=http://localhost:5678/
</code></pre>
<p>If you want to test webhooks from your laptop without a server, run <code>ngrok http 5678</code> in another terminal and use the ngrok HTTPS URL as <code>WEBHOOK_URL</code>. n8n uses <code>WEBHOOK_URL</code> to tell external services where to POST — get this wrong and your webhooks will 404.</p>
<p>Start the stack:</p>
<pre><code class="language-bash">docker compose up -d
</code></pre>
<p>Visit <code>http://your-server:5678</code>. On the first visit, n8n walks you through creating an owner account (email and password). Every subsequent visit requires that login. For extra safety in production, put n8n behind a reverse proxy with an allow-list or an additional auth layer — we'll set that up later.</p>
<h2 id="heading-part-5-creating-the-webhook-trigger-in-n8n">Part 5: Creating the Webhook Trigger in n8n</h2>
<p>Click Create Workflow. You'll see an empty canvas.</p>
<p>Add a Webhook node and configure it:</p>
<ul>
<li><p><strong>HTTP Method</strong>: POST</p>
</li>
<li><p><strong>Path</strong>: <code>whatsapp</code> (this becomes part of the URL)</p>
</li>
<li><p><strong>Response Mode</strong>: Respond Immediately</p>
</li>
<li><p><strong>Response Data</strong>: First Entry JSON</p>
</li>
</ul>
<p>Click Listen for Test Event. n8n shows you two URLs: a test URL and a production URL. Copy the production URL. It looks like this:</p>
<pre><code class="language-plaintext">https://n8n.example.com/webhook/whatsapp
</code></pre>
<p>Not <code>webhook-test</code> — that one only fires while the editor is open. You want <code>webhook</code>.</p>
<h2 id="heading-part-6-wiring-waha-to-n8n">Part 6: Wiring WAHA to n8n</h2>
<p>WAHA can POST to a webhook on every WhatsApp event. Tell it where to send those events.</p>
<p>In the WAHA dashboard, open your session and set the webhook URL. Or do it over the API:</p>
<pre><code class="language-bash">curl -X PUT http://your-server:3000/api/sessions/default \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "webhooks": [
        {
          "url": "https://n8n.example.com/webhook/whatsapp",
          "events": ["message", "session.status"]
        }
      ]
    }
  }'
</code></pre>
<p>The <code>message</code> event fires on every inbound message. <code>session.status</code> fires when the session connects, disconnects, or reconnects — which is useful for alerting when your bot goes down.</p>
<p>Test it. From another phone, send a WhatsApp message to your bot's number. Head back to the n8n editor. Within a second or two the webhook node lights up with the event data.</p>
<p>The payload looks roughly like this:</p>
<pre><code class="language-json">{
  "event": "message",
  "session": "default",
  "payload": {
    "id": "false_15555550123@c.us_3EB0...",
    "from": "15555550123@c.us",
    "body": "Hello",
    "timestamp": 1713801234,
    "fromMe": false
  }
}
</code></pre>
<p>Everything you need is in <code>payload</code>: who sent it (<code>from</code>), what they said (<code>body</code>), and when (<code>timestamp</code>).</p>
<h2 id="heading-part-7-building-the-first-auto-reply">Part 7: Building the first auto-reply</h2>
<p>A bot that only listens is boring. Let's make it answer.</p>
<p>You'll build a tiny keyword router: if the user sends <code>hi</code> or <code>hello</code>, the bot greets them. If they send <code>price</code>, it sends a pricing message. Anything else gets a fallback.</p>
<p>After the Webhook node, add a Switch node.</p>
<p>Configure the Switch node:</p>
<ul>
<li><p><strong>Mode</strong>: Expression</p>
</li>
<li><p><strong>Value</strong>: <code>{{ $json.payload.body.toLowerCase().trim() }}</code></p>
</li>
<li><p>Add routing rules:</p>
<ul>
<li><p>Rule 1: equals <code>hi</code> — output 0</p>
</li>
<li><p>Rule 2: equals <code>hello</code> — output 0</p>
</li>
<li><p>Rule 3: equals <code>price</code> — output 1</p>
</li>
<li><p>Fallback output: 2</p>
</li>
</ul>
</li>
</ul>
<p>After the Switch, add three HTTP Request nodes, one per output.</p>
<p>Configure each HTTP Request node identically, except for the body text:</p>
<ul>
<li><p><strong>Method</strong>: POST</p>
</li>
<li><p><strong>URL</strong>: <code>http://waha:3000/api/sendText</code> (inside the Docker network you can reach WAHA by its service name. From outside use the full public URL)</p>
</li>
<li><p><strong>Send Headers</strong>: on</p>
<ul>
<li><p><code>X-Api-Key</code>: <code>super-secret-key-change-me</code></p>
</li>
<li><p><code>Content-Type</code>: <code>application/json</code></p>
</li>
</ul>
</li>
<li><p><strong>Send Body</strong>: on</p>
<ul>
<li><p><strong>Body Content Type</strong>: JSON</p>
</li>
<li><p><strong>Specify Body</strong>: Using JSON</p>
</li>
</ul>
</li>
</ul>
<p>For the greeting node, the JSON body is:</p>
<pre><code class="language-json">{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "Hi! I'm the bot. Send 'price' to see pricing, or anything else for help."
}
</code></pre>
<p>For the pricing node:</p>
<pre><code class="language-json">{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "Our plans start at $49/month. Reply 'sales' to talk to a human."
}
</code></pre>
<p>For the fallback:</p>
<pre><code class="language-json">{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "I didn't catch that. Try 'hi' or 'price'."
}
</code></pre>
<p>The <code>={{ ... }}</code> syntax is an n8n expression — at runtime it pulls values from earlier nodes.</p>
<p>Connect the Switch outputs to their matching HTTP Request nodes. Save the workflow. Click Activate in the top-right.</p>
<p>Send <code>hi</code> to your bot from any phone. It should reply within a second.</p>
<p>Congratulations — you have a WhatsApp bot running entirely on your own infrastructure.</p>
<h2 id="heading-part-8-a-second-example-proactive-booking-confirmations">Part 8: A Second Example — Proactive Booking Confirmations</h2>
<p>Auto-reply is useful. Proactive outbound is where the value really compounds. Here's a second workflow that sends a booking confirmation whenever a new row lands in a database.</p>
<p>Create a second workflow in n8n. Use one of these triggers:</p>
<ul>
<li><p><strong>Schedule Trigger</strong> — poll a database every minute for new rows</p>
</li>
<li><p><strong>Webhook Trigger</strong> — listen for a notification from your booking system</p>
</li>
<li><p><strong>Database Trigger</strong> (Postgres, MySQL, Supabase) — react to inserts in real time</p>
</li>
</ul>
<p>For this example, use a Schedule Trigger set to every minute, followed by a Postgres <strong>Execute Query</strong> node that reads pending confirmations:</p>
<pre><code class="language-sql">SELECT id, customer_phone, service_name, booking_time
FROM bookings
WHERE confirmation_sent = false
LIMIT 20;
</code></pre>
<p>After the Postgres node, add an HTTP Request node pointing to the same WAHA <code>sendText</code> endpoint you used earlier. The body:</p>
<pre><code class="language-json">{
  "session": "default",
  "chatId": "={{ $json.customer_phone }}@c.us",
  "text": "Hi! Your booking for {{ \(json.service_name }} on {{ \)json.booking_time }} is confirmed. Reply 'change' to reschedule."
}
</code></pre>
<p>Finally, add a second Postgres node that marks the booking as sent:</p>
<pre><code class="language-sql">UPDATE bookings
SET confirmation_sent = true, confirmation_sent_at = NOW()
WHERE id = {{ $json.id }};
</code></pre>
<p>Activate the workflow. Every minute, n8n pulls pending bookings, sends a WhatsApp confirmation, and marks them done.</p>
<p>This pattern generalizes. Replace the SQL with a call to Shopify for order confirmations, Stripe for receipt messages, or Calendly for appointment reminders. The WhatsApp layer stays the same — only the source of truth changes.</p>
<h2 id="heading-part-9-going-to-production">Part 9: Going to Production</h2>
<p>The setup above works, but it's not yet production-ready. Here's what to harden before you point real customers at it.</p>
<h3 id="heading-1-put-everything-behind-https">1. Put Everything Behind HTTPS</h3>
<p>Never expose n8n or WAHA directly on plain HTTP. Put a reverse proxy in front. Caddy is the easiest choice because it handles Let's Encrypt automatically.</p>
<p>A minimal <code>Caddyfile</code>:</p>
<pre><code class="language-plaintext">n8n.example.com {
    reverse_proxy n8n:5678
}

waha.example.com {
    reverse_proxy waha:3000
}
</code></pre>
<p>Run Caddy as another service in the same Docker Compose. TLS certificates are issued and renewed automatically.</p>
<h3 id="heading-2-rotate-the-api-keys">2. Rotate the API Keys</h3>
<p>Don't ship <code>super-secret-key-change-me</code> to production. Generate a real key:</p>
<pre><code class="language-bash">openssl rand -hex 32
</code></pre>
<p>Put it in a <code>.env</code> file, reference it as <code>${WHATSAPP_API_KEY}</code> in <code>docker-compose.yml</code>, and add <code>.env</code> to your <code>.gitignore</code>.</p>
<h3 id="heading-3-rate-limit-outbound-messages">3. Rate-limit Outbound Messages</h3>
<p>WhatsApp bans accounts that send too many messages too fast. A safe outbound rate for a fresh number is well under 20 messages per minute. For bursty replies, add an n8n Wait node between sends, or queue outgoing messages through a small custom function node that sleeps between requests.</p>
<h3 id="heading-4-scale-n8n-with-queue-mode">4. Scale n8n with Queue Mode</h3>
<p>By default, n8n runs everything in a single process. That's fine for low volume. For higher throughput, switch to Queue Mode:</p>
<ul>
<li><p>Add a Redis container.</p>
</li>
<li><p>Run one <code>n8n</code> main container (the web UI and webhook receiver).</p>
</li>
<li><p>Run one or more <code>n8n-worker</code> containers that pull jobs from the queue.</p>
</li>
</ul>
<p>Queue Mode is documented at <a href="https://docs.n8n.io/hosting/scaling/queue-mode/">docs.n8n.io/hosting/scaling/queue-mode/</a>. Setup adds two environment variables (<code>EXECUTIONS_MODE=queue</code>, <code>QUEUE_BULL_REDIS_HOST=redis</code>) and decouples incoming webhooks from workflow execution. The webhook responds in milliseconds while workers chew through the queue in the background.</p>
<h3 id="heading-5-monitor-the-session">5. Monitor the Session</h3>
<p>WhatsApp Web sessions drop. The phone loses connection, WhatsApp rotates security tokens, or your server reboots. Catch those drops early.</p>
<p>Subscribe to the <code>session.status</code> webhook event in WAHA. When status becomes <code>FAILED</code> or <code>STOPPED</code>, route it to an n8n workflow that posts to Slack, sends an email, or pages you. The faster you know, the faster you recover.</p>
<p>For overall uptime, point something like Uptime Kuma at <code>GET /api/sessions/default</code> on WAHA. If WAHA reports <code>WORKING</code>, you're fine. Anything else triggers an alert.</p>
<h3 id="heading-6-back-up-the-sessions-volume">6. Back Up the Sessions Volume</h3>
<p>The <code>waha-sessions</code> directory contains the logged-in state. If you lose it, you have to scan the QR code again — possibly from a phone that's no longer handy. Back it up nightly. A simple cron job with <code>tar</code> and <code>rclone</code> to S3-compatible storage is plenty.</p>
<h3 id="heading-7-add-a-live-agent-handoff">7. Add a Live-Agent Handoff</h3>
<p>Not every conversation should stay with the bot. When a user types <code>human</code> — or when your intent classifier can't answer confidently — hand off to a real agent.</p>
<p>Chatwoot is a solid open-source option: it has a dedicated WhatsApp channel, agent inbox, team assignment, and conversation history. The handoff is an n8n branch that stops processing bot replies and forwards the message stream to Chatwoot's API.</p>
<h2 id="heading-common-pitfalls">Common Pitfalls</h2>
<p>A few issues catch almost everyone on their first production deploy.</p>
<h3 id="heading-webhooks-timing-out">Webhooks Timing Out</h3>
<p>WAHA gives your webhook a few seconds to respond. If your n8n workflow is slow (calling an LLM, hitting a remote API), the webhook times out and WAHA retries, potentially causing duplicate replies.</p>
<p>Fix: make the webhook return <code>200</code> immediately and offload the slow work. In n8n, set the Webhook node's Response Mode to <em>Using Respond to Webhook Node</em>, add a Respond to Webhook node as the first step with a <code>200</code> and empty body, then do the heavy lifting after that.</p>
<h3 id="heading-duplicate-messages">Duplicate Messages</h3>
<p>WAHA delivers the same <code>message</code> event more than once in edge cases (phone comes back online, session reconnects). Store the <code>payload.id</code> somewhere — Redis, a database, or n8n's static data store — and drop any ID you've already processed.</p>
<h3 id="heading-messages-arriving-out-of-order">Messages Arriving Out of Order</h3>
<p>The webhook is async, and n8n may parallelize executions. If ordering matters — for example, in a multi-step conversation — key a queue by the sender's <code>chatId</code> and process each sender serially.</p>
<h3 id="heading-sessions-disconnecting-after-a-phone-restart">Sessions Disconnecting After a Phone Restart</h3>
<p>Normal WhatsApp Web behavior. WAHA auto-reconnects, but occasionally the linked-devices list needs a manual refresh. If a session refuses to come back, stop the WAHA container, delete that session's folder under <code>waha-sessions/</code>, start the container again, and rescan the QR.</p>
<h3 id="heading-your-number-gets-banned">Your Number Gets Banned</h3>
<p>The single biggest cause is rate: a new number blasting hundreds of messages an hour gets flagged fast. Warm up a number slowly — send a normal, human-like volume for the first week. Don't send to strangers unsolicited. Prefer inbound-driven replies over outbound pushes wherever you can.</p>
<h3 id="heading-the-wrong-chat-id-format">The Wrong Chat ID Format</h3>
<p>WhatsApp individual chats use <code>&lt;number&gt;@c.us</code> and groups use <code>&lt;groupId&gt;@g.us</code>. Don't include the <code>+</code> or spaces in the number. If WAHA returns a 404 when sending, the chat ID is almost always the problem.</p>
<h2 id="heading-where-to-go-next">Where to Go Next</h2>
<p>You now have the foundation. The same two-service stack supports almost any bot you can imagine — you're only limited by what you can build in an n8n workflow.</p>
<p>Some natural next steps:</p>
<ul>
<li><p><strong>Plug in AI replies:</strong> Add an OpenAI or Anthropic node after the Webhook, pass the user's message through it with a short system prompt, and send the response back through WAHA. Cap conversation length to prevent runaway token usage.</p>
</li>
<li><p><strong>Integrate a CRM:</strong> Look up the caller's <code>chatId</code> in HubSpot, Pipedrive, or your own database before deciding how to reply. Segment responses by customer tier.</p>
</li>
<li><p><strong>Send proactive notifications:</strong> Appointment reminders, shipping updates, payment receipts, abandoned-cart nudges. Keep the content transactional and expected — unsolicited marketing blasts are the fastest way to a ban.</p>
</li>
<li><p><strong>Log every conversation:</strong> Add a Postgres or Supabase node after the Webhook to persist messages for analytics and customer history. Your future self (and your support team) will thank you.</p>
</li>
<li><p><strong>Add media handling:</strong> WAHA exposes <code>sendImage</code>, <code>sendFile</code>, and <code>sendVoice</code> endpoints. Teach the bot to accept photos for support tickets, or send invoices as PDFs directly inside the chat.</p>
</li>
</ul>
<p>The WhatsApp layer stays the same. Everything interesting happens upstream in the workflow.</p>
<p><em>If you want to see production examples of n8n and WAHA running at scale — or you need a similar automation built for your business — I'm the founder of Achiya Automation, where we ship WhatsApp, n8n, and Chatwoot integrations. You can find more at</em> <a href="https://achiya-automation.com"><em>achiya-automation.com</em></a><em>.</em></p>
 ]]>
                </content:encoded>
            </item>
        
    </channel>
</rss>
