In this tutorial, you'll build a data pipeline yourself. You'll create a Python ETL (Extract, Transform, Load) pipeline that pulls daily water-level readings from Hub'Eau, France's official open water-data API. Then, you'll clean that data and publish it as a public dataset, just like the live version does.

This tutorial is based on a real pipeline that runs once a week, on a schedule, and it keeps the Paris Flood Dataset updated automatically.

You won't just copy and paste code, though. The real goal is to understand why the pipeline works the way it does. You'll walk through the design decisions that separate a script meant to run once from a script that keeps working, unattended, for years.

You can code along with this notebook. For most of the tutorial, the pipeline runs on simulated (mock) API data. This lets you run every cell safely, without hammering a real server. A later section shows how to switch to the live API.

By the end, you'll be able to:

• Explain and implement the Extract, Transform, Load pattern

• Manage configuration with Python @dataclass instead of scattering constants everywhere

• Write API-fetching code that survives network failures and paginated responses

• Apply robust type-coercion so one bad row can't crash a whole pipeline run

• Deduplicate and merge incremental data safely

• Wire everything into a single, idempotent, schedulable main() entry point

Table of Contents:

• Prerequisites

• Part 1: The Design Logic

  • What is an ETL pipeline?

  • The architecture, at a glance

  • Two patterns that make it production-grade

• Part 2: Setup and Dependencies

• Part 3: Manage Configuration with Dataclasses

  • Why bother with a config layer at all?

  • Code-level walkthrough

• Part 4: The Extraction Step

  • Graceful file loading

  • Incremental update logic

  • Simulate the API

  • Fetch data for real

• Part 5: The Transform Step

  • Type parsing and graceful coercion

  • Schema translation with bidirectional mappings

  • Compute flood alerts

  • Column ordering

  • Deduplication

  • Put it all together inpostprocess()

• Part 6: The Load Step

  • Design logic

  • Code-level walkthrough

• Part 7: Assemble the Full Pipeline

  • The global rehearsal (mock mode)

  • main(): pipeline orchestration

  • Theif name == "main":guard

• Part 8: Go Live and Switch to the Real API

  • Post-run validation

• Summary

• References

Prerequisites

• Python 3.10+. The code uses type hints and dataclasses. These work from Python 3.7 onward, but 3.10+ is best.

• Knowledge of pandas DataFrames: read data, filter rows, and basic column operations.

• Comfort with functions and basic OOP (Object-Oriented programming) in Python. Don't worry, this guide explains every non-obvious piece as you go.

• Optional: a free Kaggle account and the Kaggle CLI, only if you want to run the final publishing step for real.

Install the dependencies:

pip install requests pandas numpy ipykernel

You'll do the work inside a Jupyter notebook. Download the .ipynb file from Kaggle, or click "Copy and Edit" to work directly on Kaggle. You'll need a Kaggle account for that.

Optional: If you plan to run the notebook locally, install the notebook package too:

pip install notebook

Part 1: The Design Logic

Before you touch a single line of code, we'll spend three minutes on why the pipeline is shaped this way.

This is the big-picture view. Every code-level decision later traces back to one of these ideas. Read this section even if you skim everything else.

What is an ETL Pipeline?

ETL stands for Extract, Transform, Load. It's the standard pattern for moving data from a source to a destination in a reliable, repeatable way.

• Extract: pull data from a source, like an API, a database, or files.

• Transform: clean, standardize, enrich, and validate the data.

• Load: write the result to a destination, like a warehouse, a CSV, or a public platform.

Here's how those three stages map onto this project:

The Architecture, at a Glance

Notice that "Extract" already touches two different sources: the existing dataset (what you already have) and the new data from the API. That distinction is the seed of the next idea.

Two Patterns that Make it Production-Grade

There two patterns that make this pipeline safe to run unattended, scheduled, for years.

1. Idempotency

Idempotence means when the same operation runs twice it gives the same result as if it runs once. Deduplication is what makes this pipeline idempotent. If the scheduler accidentally triggers twice, or a network retry fetches the same day again, the second run won't create duplicate rows. This matters a lot for anything that runs on a schedule with no one watching it.

2. Incremental loading

A naïve pipeline would re-download the entire history on every run. That's slow, it wastes your API quota, and it's fragile: the more data you transfer, the more chances something fails.

An incremental pipeline avoids this. Instead, it:

• Checks the most recent date already in the dataset

• Requests only the data from that date onward

• Merges the new records into the existing dataset

You'll find this logic built explicitly in determine_update_range.

Always ask yourself: "Do I actually need to do this?" That one habit separates a fragile script from a production pipeline. For example, every "expensive" or "external" step in this pipeline (network calls, disk writes, publishing) is guarded by a cheap, local check first.

Part 2: Setup and Dependencies

Here's the import block. It looks unremarkable, but how it's organized is itself a best practice worth calling out.

# Standard library imports
import json
import os
import random
import subprocess
from dataclasses import dataclass, field
from datetime import date, timedelta
from pathlib import Path
from typing import Dict, List, Optional, Set, Tuple

# Third-party libraries
import numpy as np
import pandas as pd
import requests

# Display config for notebooks
pd.set_option('display.max_columns', None)   # All columns will show
pd.set_option('display.max_colwidth', None)  # Prevents cutting long column text with ...

Code-level notes:

• Imports fall into three blocks: standard library, then third-party, then local, with blank lines between them. This follows PEP 8's import-ordering convention. Most auto-formatters (isort, ruff) enforce this same grouping.

• Nothing is imported with from module import *. That syntax pollutes the current namespace. It makes it hard to trace where a name came from when someone debugs the code six months later. Python style guides echo this advice, including Google's Python Style Guide.

• The pd.set_option(...) calls exist purely for notebook readability, so wide DataFrames don't get truncated with .... They have zero effect on the pipeline's logic. You'd typically remove or scope them differently in a .py script.

Part 3: Manage Configuration with Dataclasses

Why Bother with a Config Layer at All?

Every pipeline has knobs: which stations to monitor, what the flood threshold is, and where to publish. The bad approach sprinkles these values as literals throughout the code as you write it. You end up with an if level > 6000: buried three functions deep. Then changing any setting means hunting through the whole file, and it's easy to update one spot and miss another.

The fix: centralize all settings in one place. Python's @dataclass decorator is a natural fit for that.

Compare that to a configuration written as a plain dict:

config = {
    "stations": ["STN001", "STN002"],
    "flood_threshold": 6000,
    "publish_url": "https://example.com/alerts",
    "retry_count": 2,
    "timeout_seconds": 5,
}

A plain dict gives you none of that: no immutability, no type checking, and no autocomplete.

Code-Level Walkthrough

The pipeline defines three configuration classes. Each one has a single responsibility: API details, station rules, and publishing destination. Each class becomes one module-level singleton instance. Every other function in the pipeline reads from these singletons.

@dataclass
class APIConfig:
    """API configuration for HubEau data fetching.

    Think of this as the "address book" for the API.
    """
    use_mock: bool = True
    base_url: str = "https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab"
    metric: str = "HIXnJ"  # Daily max water level (elaborated observations)
    # Pagination: fetch 20k records per request (API Limit)
    max_per_page: int = 20000
    timeout_seconds: int = 60  # Network timeout

    def __post_init__(self):
        """Validate configuration after initialization."""
        if self.max_per_page <= 0:
            raise ValueError("max_per_page must be positive")
        if self.timeout_seconds <= 0:
            raise ValueError("timeout_seconds must be positive")

Notice the validation inside __post_init__. It runs right after the auto-generated __init__. A misconfigured APIConfig(max_per_page=-1) fails loudly and immediately at startup, instead of surfacing as a bug later during an actual pipeline run.

@dataclass
class StationConfig:
    """Station monitoring configuration.

    The 'what' of data collection: which stations, what's a flood?
    """
    station_codes: List[str] = field(default_factory=lambda: [
        "F700000109", "F700000110", "F700000111",
        "F700000102", "F700000103",
    ])
    flood_threshold_mm: int = 6000  # Flood alert threshold
    earliest_date: str = "1900-01-01"  # How far back to go

The mutable-default trap: Look closely at station_codes. It isn't written as station_codes: List[str] = [...]. That's deliberate, and it dodges one of the most common gotchas in Python. If you use a plain mutable object (a list, dict, or set) as a default argument or dataclass field, every instance shares the same underlying object. Mutate it on one instance, and you silently mutate it everywhere else too.

Stack Overflow covers this at length in its "least astonishment" mutable-default-argument discussion, as does Real Python's guide to optional arguments. The fix is field(default_factory=...). It calls a fresh factory function (here, a lambda) for every new instance, so each one gets its own independent list.

Explanation:

# Bad: shared mutable default
@dataclass
class BadConfig:
    station_codes: list[str] = []

a = BadConfig()
b = BadConfig()

a.station_codes.append("ALERT")
print(a.station_codes)  # ['ALERT']
print(b.station_codes)  # ['ALERT']  <-- same list!!


# Good: fresh list per instance
@dataclass
class GoodConfig:
    station_codes: list[str] = field(default_factory=list)

x = GoodConfig()
y = GoodConfig()

x.station_codes.append("ALERT")
print(x.station_codes)  # ['ALERT']
print(y.station_codes)  # []  <-- independent list

And the last Config singleton, the Kaggle settings:

@dataclass
class KaggleConfig:
    """Kaggle dataset publishing configuration."""
    dataset_slug: str = "grimespoint/paris-flood-dataset"
    input_csv: str = "kaggle/input/datasets/{slug}/paris_flood_dataset.csv"
    output_dir: Path = field(default_factory=lambda: Path("kaggle/working/kaggle_dataset"))
    mock_output_dir: Path = field(default_factory=lambda: Path("mock_output"))
    output_filename: str = "paris_flood_dataset.csv"
    mock_output_filename: str = "mock_flood_dataset.csv"
    metadata_filename: str = "dataset-metadata.json"

    # Metadata
    title: str = "Paris flood dataset"
    keywords: list = field(default_factory=lambda: [
        "tabular", "weather and climate", "environment", "europe", "time series analysis"
    ])
    geospatial_coverage: str = "Paris, France"
    update_frequency: str = "Weekly"
    license_name: str = "CC0-1.0"

    # Computed fields (set in __post_init__)
    output_csv_path: Path = field(init=False)
    metadata_path: Path = field(init=False)

    def __post_init__(self):
        """Compute derived paths after initialization."""
        self.input_csv = self.input_csv.format(slug=self.dataset_slug)
        self.output_csv_path = self.output_dir / self.output_filename
        self.metadata_path = self.output_dir / self.metadata_filename
        self.mock_output_filename = self.mock_output_dir / self.mock_output_filename

# Initialize configs - module-level singletons
API_CONFIG = APIConfig()
STATION_CONFIG = StationConfig()
KAGGLE_CONFIG = KaggleConfig()

This is the most common use of __post_init__. output_csv_path and metadata_path are marked field(init=False), so you can't set them directly through the constructor. Instead, __post_init__ computes them from other fields (output_dir and output_filename).

Use this pattern for derived, computed values: compute them once, in one place, instead of recomputing output_dir / output_filename every time you need the path elsewhere in the codebase.

See Real Python's data classes guide or the data classes chapter of O'Reilly's Fluent Python for more on this pattern.

Tip: the auto-generated __repr__ gives you a readable printout for free. Call print(API_CONFIG): it shows every field and value without a single line of formatting code. It's handy for quick sanity checks when you're debugging a pipeline run.

print(APIConfig)
# prints APIConfig(use_mock=True, base_url='https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab', ...)

Part 4: The Extraction Step

The Extract phase pulls data from source systems and reads it into memory. Here, you genuinely have two sources to extract from: the existing dataset (what you already published last time) and the new (recent) data from the Hub'Eau API.

Graceful File Loading

Design logic

What should happen the very first time this pipeline runs (and there's no existing dataset yet)? A naïve implementation would crash with a FileNotFoundError.

Here's the trick: load_csv() follows the Null Object pattern. Instead of raising an error, it returns an empty DataFrame. Every downstream function can then treat "no existing data" and "some existing data" the same way, with no special-casing needed.

Code-level walkthrough

This function loads a CSV file into a DataFrame. If the file doesn't exist, it returns an empty DataFrame instead of crashing.

low_memory=False tells pandas to read the file carefully, so it avoids mixed-type guesses. parse_dates=True tries to automatically convert date-like columns into dates. delimiter="," tells pandas the file is comma-separated.

def load_csv(path: str) -> pd.DataFrame:
    """Load CSV file or return empty DataFrame if file does not exist.

    Args:
        path (str): Full path to the CSV file.

    Returns:
        pd.DataFrame: Loaded data, or empty DataFrame if file not found.

    Raises:
        pd.errors.ParserError: If the CSV is malformed.
    """
    if os.path.exists(path):
        return pd.read_csv(path, low_memory=False, parse_dates=True, delimiter=",")
    return pd.DataFrame()   # Null Object: consistent return type

To test, try it against a path that doesn't exist:

df_missing = load_csv("/tmp/does_not_exist.csv")
print(df_missing.empty)  # True: no crash

# Callers can always do this, instead of an `is None` check:
if df_missing.empty:
    print("No existing data. Will run a full fetch from earliest date.")

Best practice: return a consistent type from every code path in a function. A function that sometimes returns a DataFrame and sometimes None forces every caller to add a None check before using the result.

Return a DataFrame, empty or not, which keeps things simpler. You never have to ask "did I get a real result, or None?" before using it.

Incremental Update Logic

Design logic

This is the "incremental loading" pattern from Part 1 in action. Before you fetch anything, ask: "What's the most recent record I already have, and do I actually need more?"

Strategy: check whether the existing data already covers yesterday.

• Yes: skip the update entirely. Nothing to do, the dataset is current.

• No: fetch starting from the day after the last known date.

Existing data: Jan. 1 – Jan. 15
Yesterday: Jan. 19

Decision: fetch from Jan. 16 onwards (not from Jan. 1)

Why yesterday and not today? Today's measurement might not be finalized on the source system yet. Hub'Eau's "elaborated observations" are a processed daily aggregate, so the safest check is against the last fully completed day.

Code-level walkthrough

determine_update_range() checks the newest saved date. It tells you either "you're already up to date" or "start downloading from this next date."

Step by step:

1. The function starts with existing data

   • existing is a pandas DataFrame that already has some rows of data.

2. If there is no data at all

   • if existing.empty:

   • If the DataFrame has zero rows, it says: "Nothing is saved yet, so fetch everything."

   • It returns:

     • True = update needed

     • STATION_CONFIG.earliest_date = start from the earliest allowed date

3. Find the date column

   • The code checks which column contains dates:

     • first tries "date_obs_elab"

     • then "record_date"

   • If neither exists, it raises an error because it doesn’t know which column to use.

4. Convert the date column into real dates

   • pd.to_datetime(...) turns the column into date objects pandas can work with.

   • errors="coerce" means bad date values become missing values instead of crashing.

5. Find the latest date in the data

   • last_day = s.max().date()

   • This gets the newest date already in the dataset.

6. Compare it with yesterday

   • yesterday = date.today() - timedelta(days=1)

   • The function checks whether data already includes yesterday.

7. If data is already up to date

   • If last_day >= yesterday, it returns:

     • False = no update needed

     • None = no start date needed

8. If data is behind

   • It sets next_day to the day after the last saved date.

   • Then it returns:

     • True = update needed

     • that next day as a string like "2026-07-12"

def determine_update_range(existing: pd.DataFrame) -> Tuple[bool, Optional[str]]:
    """Determine whether an update is needed and from what date.

    Logic:
    1. Check if existing data covers yesterday's date
    2. If yes → no update needed
    3. If no → start fetching from day after last data

    Returns:
        Tuple[should_update, start_date]
    """
    # Case 1: nothing on disk yet
    if existing.empty:
        print("No existing data found. Will fetch all data from earliest date.")
        return True, STATION_CONFIG.earliest_date

    if "date_obs_elab" in existing.columns:
        record_colname = "date_obs_elab"
    elif "record_date" in existing.columns:
        record_colname = "record_date"
    else:
        raise KeyError("Missing date column: expected 'date_obs_elab' or 'record_date'")

    s = pd.to_datetime(existing[record_colname], errors="coerce")
    last_day = s.max().date()
    yesterday = date.today() - timedelta(days=1)

    # Case 2: already current
    if last_day >= yesterday:
        print("\nDataset already covers yesterday or later. No update needed.")
        return False, None

    # Case 3: fetch the gap
    next_day = (last_day + pd.Timedelta(days=1))
    print(f"\nWill retrieve data starting from: {next_day}")
    return True, next_day.isoformat()

A few things worth a note here.

First, the function checks for two possible column names: date_obs_elab, the raw API name, or the already-renamed English name record_date. It doesn't assume just one. This makes the function work whether you call it on freshly-fetched raw data or an already-processed CSV loaded from disk.

errors="coerce" shows up here, and it'll show up again (we'll dig into this in Part 5). Any date pandas can't parse becomes NaT (Not a Time) instead of raising an exception.

The return type is Tuple[bool, Optional[str]]. This tuple bundles two related results: should I update, and from when? That beats returning two separate values, or worse, one ambiguous value that means different things depending on context.

Best practice: use Tuple return types (or, for more fields, a small dataclass or NamedTuple) to bundle related results together. Document clearly what each position means. If you return different types from different code paths without documentation, you'll hit a common source of confusion and bugs: "why is this None sometimes and a string other times?".

Run a quick check against three scenarios:

# Test 1: No existing data
should_update, start_date = determine_update_range(pd.DataFrame())
# → True, "1900-01-01"

# Test 2: Old existing data (covers only Jan 10-15)
# → True, "2026-01-16"  (the day after the last known date)

# Test 3: Recent data that already covers yesterday
# → False, None

Simulate the API

Design logic

In production, fetching means a real HTTP call:

requests.get(
    "https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab",
    params={"code_entite": "F700000109", "size": 20000, ...}
)

Real API calls bring real challenges. Rather than fight those on your first read-through, this tutorial first builds and tests everything against a mock generator. It returns data shaped exactly like the real API. Only once the logic works does it swap in the real endpoint.

This technique is useful well beyond this project. Build and test your transform logic against fixtures or mocks first. That way, you're not into "is my parsing wrong?" and "is the network flaky right now?" at the same time.

Code-level walkthrough

generate_mock_api_data() generates fake sample data for a station, one record per day, starting from start_date.

How it works:

• It converts start_date into a real date.

• It loops for num_days.

• For each day, it creates a mock observation dictionary.

• It adds a small random variation to the water level so the data looks realistic.

• It randomly picks status, quality, and method labels.

• It returns a list of these dictionaries.

def generate_mock_api_data(station_code: str, start_date: str, num_days: int = 10) -> List[Dict]:
    """Generate realistic mock API data for demonstration.

    Simulates what HubEau API would return: list of observation dicts.
    """
    start = pd.to_datetime(start_date).date()
    records = []

    validation_statuses = ["Donnée validée", "Donnée brute", "Donnée pré-validée"]
    qualities = ["Bonne", "Non qualifiée", "Douteuse"]
    methods = ["Mesurée", "Calculée", "Expertisée"]

    for i in range(num_days):
        obs_date = start + timedelta(days=i)
        base_level = 5500 + int(station_code[-2:])  # Varies by station
        noise = random.randint(-200, 200)
        water_level = base_level + noise

        record = {
            "code_site": "mock_" + station_code[1:],
            "code_station": "mock_" + station_code,
            "date_obs_elab": obs_date.isoformat(),
            "resultat_obs_elab": water_level,
            "date_prod": (obs_date + timedelta(days=1)).isoformat(),
            "code_statut": "1",
            "libelle_statut": random.choice(validation_statuses),
            "code_methode": "1",
            "libelle_methode": random.choice(methods),
            "code_qualification": "1",
            "libelle_qualification": random.choice(qualities),
            "longitude": 2.3522 + random.uniform(-0.01, 0.01),
            "latitude": 48.8566 + random.uniform(-0.01, 0.01),
            "grandeur_hydro_elab": "mock_HIXnJ",
        }
        records.append(record)

    return records

Fetch Data for Real

Design logic

Two functions handle extraction. They're deliberately split according to the Single Responsibility Principle (SRP): each function should have one reason to change.

fetch_all_data() (orchestrator)
    ├── fetch_single_station_data(station_1) ← handles all complexity
    ├── fetch_single_station_data(station_2) ← handles all complexity
    └── fetch_single_station_data(station_n) ← handles all complexity

• fetch_single_station_data() owns all the messy per-station complexity: pagination, cursor advancement, stop conditions, and network error handling.

• fetch_all_data() owns none of that. It just loops over stations and delegates.

This split has two payoffs. First, you can debug or swap out the pagination strategy for one station without touching the orchestration code at all. Second, if you ever want to parallelize fetching (with concurrent.futures or asyncio, for example), the orchestrator is the only place you'd need to touch.

Code-level walkthrough

fetch_single_station_data() fetches station data either from mock test data or from the real API, page by page, until it has everything it needs.

• If use_mock=True, it uses fake data instead of calling the real API.

  1. It calls generate_mock_api_data()

  2. Turns the result into a DataFrame

  3. Converts the date column into real pandas dates

  4. Returns that DataFrame

• If use_mock=False, it does the real API request:

  1. Creates a reusable HTTP session

  2. Starts from start_date

  3. Repeatedly asks the API for a page of data

  4. Stops when:

     • the API returns no data

     • the latest date reaches yesterday

     • the page is smaller than expected

     • a network error happens

  5. Combines all pages into one DataFrame

  6. Returns an empty DataFrame if nothing was fetched

def fetch_single_station_data(station_code: str, start_date: str, use_mock: bool = True) -> pd.DataFrame:
    """Fetch all hydrometric data for a single station from mock API (default) or from the real endpoint.

    In production (cursor based pagination strategy):
    - Fetches max_per_page records per request
    - Continues until no new data or yesterday's date reached
    - Stop when: no data returned | last date >= yesterday | page was not full
    - Handles network errors gracefully
    """
    if use_mock:
        data = generate_mock_api_data(station_code, start_date, num_days=7)
        page_df = pd.DataFrame(data)
        page_df["date_obs_elab"] = pd.to_datetime(
            page_df["date_obs_elab"], errors="coerce").dt.normalize()
        return page_df

    # Real implementation
    else:
        session = requests.Session()  # Reuse TCP connection across pages
        frames = []
        cursor = start_date

        while True:
            params = {
                "code_entite": station_code,
                "grandeur_hydro_elab": API_CONFIG.metric,
                "date_debut_obs_elab": cursor,
                "size": API_CONFIG.max_per_page,
            }

            try:
                response = session.get(
                    API_CONFIG.base_url,
                    params=params,
                    timeout=API_CONFIG.timeout_seconds  # Best practice, always set
                )
                response.raise_for_status()
            except requests.RequestException as e:  # Don't let one station kill the whole pipeline
                print(f"Error fetching data for station {station_code}: {e}")
                break

            data = response.json().get("data", [])
            if not data:  # Empty response: we've exhausted this station
                break

            page_df = pd.DataFrame(data)
            page_df["date_obs_elab"] = pd.to_datetime(
                page_df["date_obs_elab"], errors="coerce").dt.normalize()
            frames.append(page_df)

            last_page_date = page_df["date_obs_elab"].max()
            yesterday = date.today() - timedelta(days=1)

            # Prevent infinite loops
            if pd.isna(last_page_date) or last_page_date.date() >= yesterday:
                break

            cursor = (last_page_date + pd.Timedelta(days=1)).strftime("%Y-%m-%d")

            if len(data) < API_CONFIG.max_per_page:
                break

    if frames:
        return pd.concat(frames, ignore_index=True)
    return pd.DataFrame()

How does the pagination loop actually work?

Let's walk through it step by step. This is the densest bit of logic in the whole notebook.

1. Send a request with date_debut_obs_elab=cursor: "give me records from this date on."

2. If the request fails outright (requests.RequestException), log it and break. break stops the loop right away and moves on. One station's network hiccup shouldn't kill the pipeline for every other station.

3. If the response has no data at all, you've caught up: break.

4. Otherwise, note the latest date seen on this page (last_page_date).

5. If that latest date is already >= yesterday, you've caught up: break.

6. Otherwise, advance the cursor to last_page_date + 1 day and loop again for the next page.

7. As a safety net: if the page returned fewer records than max_per_page, that also means you've reached the end. The API wouldn't return a partial page unless it ran out of data, so: break.

That last check (step 7) is a classic pagination termination heuristic. You don't always need a next_page token from the API. If a full page is size=20000 and you get back only 4213 records, there's nothing left to fetch.

Three specific, deliberate choices, called out:

session = requests.Session()

A Session object reuses the underlying TCP connection across multiple requests to the same host. That avoids a fresh TCP/TLS handshake on every single page request. In short, it's faster for you and more polite to Hub'Eau's servers.

Best practice: any time you call requests.get() more than once against the same host in a loop, reach for a Session.

except requests.RequestException as e:

RequestException is the base class for every exception requests can raise: timeouts, connection errors, HTTP errors from raise_for_status() and more. Catching the base class here means any network hiccup gets handled the same forgiving way: log it, stop fetching this station, move on.

timeout=API_CONFIG.timeout_seconds

By default, requests calls never time out. Without an explicit timeout, a hung server can freeze your entire pipeline indefinitely. The requests advanced usage docs states that requests to external servers should have a timeout attached.

Best practice: wrap every external I/O call in try/except. Always fail gracefully. Log the error and let the pipeline recover or move on. This avoids one flaky request taking down an unattended weekly job.

Now the orchestrator, fetch_all_data(), is deliberately much simpler:

1. Loop through all station codes.

   • Fetch each station's data.

   • Keep the non-empty results.

2. Combine them into one big DataFrame.

def fetch_all_data(start_date: str, use_mock: bool = True) -> pd.DataFrame:
    """Orchestrator: Fetch data for all configured stations."""
    frames = []

    for station_code in STATION_CONFIG.station_codes:
        print(f"Fetching data for station {station_code}...")
        df_station = fetch_single_station_data(station_code, start_date, use_mock=use_mock)

        if not df_station.empty:
            print(f"  Got {len(df_station)} records")
            frames.append(df_station)
        else:
            print(f"  (no data)")

    if frames:
        return pd.concat(frames, ignore_index=True)
    return pd.DataFrame()

That's it. A loop and a pd.concat. All the hard-won complexity lives one layer down, exactly where SRP says it should.

Part 5: The Transform Step

This is where raw, freshly-fetched data becomes something you can publish. Here's the design philosophy for this whole section: compose many small, pure functions. Each one takes a DataFrame in and returns a new DataFrame out without side effects. Avoid the one giant do-everything function trap.

    (EXTRACT)
    Raw API Data
    ↓
    (TRANSFORM)
1. Type parsing (datetime, numeric)
2. Column renaming (French → English)
3. Categorical mapping (validation status, quality)
4. Derived columns computation (flood alert flags)
5. Column reordering (logical grouping)
6. Sorting & index reset
    ↓
    (LOAD)
    Publication-ready dataset

Why split this into six tiny steps instead of one big function? Each piece is testable and replaceable on its own. When something breaks at 3am on a scheduled run, every function is debuggable in isolation. You can pinpoint exactly which stage produced bad output. No need to pick apart one 200-line function.

Type Parsing and Graceful Coercion

Design logic

Data from a CSV or a JSON API starts out, by default, as strings. Pandas needs real types to sort dates chronologically, do date arithmetic (last_date + timedelta(days=1)), or compare numeric values (water_level > 6000). Type mismatches are a common error that break batch pipelines. One malformed row, like "N/A", a truncated date, misaligned values, or stray characters and a strict parser throws an exception that kills the whole run.

Best practice: both conversion functions below use errors="coerce", so values pandas can't parse become NaT (Not a Time) or NaN (Not a Number) instead of raising an error. This is called graceful coercion.

Code-level walkthrough

convert_to_date() and convert_to_numeric() are small helper functions that make sure certain columns have the right type.

• Both start with df.copy() so they don’t change the original DataFrame.

• errors="coerce" means bad values become missing values instead of causing a crash.

For convert_to_date():

1. df.copy() makes a separate copy, so the original table stays unchanged

2. The loop goes through each column name in columns

   • if col in df.columns checks that the column actually exists before trying to convert it

   • pd.to_datetime(...) turns text like "2026-07-12" into real pandas date/time values

   • errors="coerce" means invalid values become NaT (missing date) instead of raising an error

   • .dt.normalize() removes the time part and keeps only the date at midnight

For convert_to_numeric():

1. df.copy() is used here as well.

2. The loop goes through each column name in columns

   • if col in df.columns checks that the column actually exists before trying to convert it

   • The loop calls pd.to_numeric() turns text like "12.5" into numbers

   • errors="coerce" turns bad values into NaN instead of crashing

def convert_to_date(df: pd.DataFrame, columns: List[str]) -> pd.DataFrame:
    """Convert specified columns to pandas datetime type."""
    df = df.copy()  # Never modify the original!
    for col in columns:
        if col in df.columns:
            df[col] = pd.to_datetime(df[col], errors="coerce").dt.normalize()
    return df


def convert_to_numeric(df: pd.DataFrame, columns: List[str]) -> pd.DataFrame:
    """Convert specified columns to numeric (float) type."""
    df = df.copy()
    for col in columns:
        if col in df.columns:
            df[col] = pd.to_numeric(df[col], errors="coerce")
    return df

Why is this useful?

• You can aggregate the data: group it, summarize it and so on...

• Dates become sortable and filterable as real dates.

• Numbers work correctly in calculations like averages, sums, or comparisons.

• It prevents bugs caused by mixed types, like "12" and 12.

Both pd.to_datetime and pd.to_numeric are official pandas functions with an errors parameter. By default, that parameter is set to "raise", which throws on bad input. The other options are "coerce" (replace with null) or "ignore" (leave untouched). See the pandas to_datetime docs and to_numeric docs for the full parameter list.

Test it against an example of messy input:

messy_df = pd.DataFrame({
    "date_obs_elab": ["2026-01-15", "2026-01-16", "not a date", None],
    "resultat_obs_elab": [5800.0, "5900", "N/A", None],
})

type_safe_df = convert_to_date(messy_df, ["date_obs_elab"])
type_safe_df = convert_to_numeric(type_safe_df, ["resultat_obs_elab"])

# "not a date"  → NaT
# "N/A"         → NaN
# Pipeline continues safely — nothing crashed.

Best practice: don't let one bad row kill an entire run. Coerce bad data to nulls rather than raising exceptions. Flag or log nulls separately if you need to investigate data quality later.

This is a deliberate trade-off. A choice between availability (the pipeline keeps running) over strictness (catching every bad row immediately). That's usually the right call for a scheduled, unattended job.

df.copy(): immutability by convention

Look again at the top of both functions above: df = df.copy(). This single line appears at the start of every transform function in the pipeline, and that's not an accident.

Python DataFrames are mutable objects, passed by reference. If a function modifies df in place without copying first, the caller's original DataFrame changes too. That's a classic side effect and it can produce genuinely confusing bugs. Call .copy() first means each function's output is a brand-new object. The input the caller passed in stays guaranteed untouched.

Best practice: treat DataFrames as immutable inputs. Return a new DataFrame rather than modify it one in place. Even if it costs a small amount of memory or CPU, the debugging win is almost always worth it for a pipeline that isn't operating at extreme scale.

Finally, one auto-detecting convenience function wraps these two low-level functions. It scans text columns, guesses whether they contain dates or numbers, and calls the matching parsing function above.

auto_convert_columns() tries to guess which columns are dates or numbers and then fixes these types automatically.

How it works:

It starts with two empty lists:

• datetime_cols for date columns

• numeric_cols for number columns

It goes through each column in the DataFrame. If a column is already a real datetime or numeric type, it skips it. If the column is text-like (object or string), it looks at up to 10 non-empty sample values.

It first tries to read those values as dates:

• if that works, the column is added to datetime_cols

If not, it tries to read them as numbers:

• if that works, the column is added to numeric_cols

At the end, it converts all date columns with convert_to_date(). Then it converts all numeric columns with convert_to_numeric()

def auto_convert_columns(df: pd.DataFrame) -> pd.DataFrame:
    """Auto-detect and convert datetime and numeric columns to the correct type."""
    datetime_cols = []
    numeric_cols = []

    for col in df.columns:
        if pd.api.types.is_datetime64_any_dtype(df[col]):
            continue
        if pd.api.types.is_numeric_dtype(df[col]):
            continue

        if pd.api.types.is_object_dtype(df[col]) or pd.api.types.is_string_dtype(df[col]):
            sample = df[col].dropna().head(10)
            if len(sample) == 0:
                continue

            try:
                pd.to_datetime(sample, errors='raise', format='mixed')
                datetime_cols.append(col)
                continue
            except (ValueError, TypeError):
                pass

            try:
                pd.to_numeric(sample, errors='raise')
                numeric_cols.append(col)
                continue
            except (ValueError, TypeError):
                pass

    df = convert_to_date(df, datetime_cols)
    df = convert_to_numeric(df, numeric_cols)
    return df

Notice the inner try/except blocks here use errors='raise'. That's the opposite of the coercion strategy above, but it only runs against a small .head(10) sample of each column.

This is a type-sniffing step, not the final conversion. It tests "does this column look like dates, or numbers, or neither?" on a cheap sample. Then, it hands off the actual, forgiving conversion of the whole column to convert_to_date() or convert_to_numeric(). Two different errors strategies, two different jobs.

Schema Translation with Bidirectional mappings

Design logic

The Hub'Eau API returns French column names and French categorical values, like code_station or "Donnée validée". A dataset meant for an international audience should ship in English. If you rename columns inline, wherever it's convenient, the mapping between French and English ends up scattered across the codebase. Then you'd have no way to reverse it if you ever needed to.

The fix: define one authoritative mapping at the top of the module, and derive everything else from it.

Code-level walkthrough

API_TO_EN maps API field names to English translations.

# Primary mapping: French API columns to English column names
API_TO_EN = {
    "code_site": "location_code",
    "code_station": "station_code",
    "date_obs_elab": "record_date",
    "resultat_obs_elab": "water_level_mm",
    "date_prod": "data_production_date",
    "code_statut": "validation_status_code",
    "libelle_statut": "validation_status",
    "code_methode": "production_method_code",
    "libelle_methode": "production_method",
    "code_qualification": "quality_code",
    "libelle_qualification": "quality_assessment",
    "longitude": "longitude",
    "latitude": "latitude",
    "grandeur_hydro_elab": "hubeau_elab_code",
}

# Reverse mapping: English to French (computed automatically)
EN_TO_API = {v: k for k, v in API_TO_EN.items()}

The second line (EN_TO_API) is just a shortcut: it swaps each key and value from API_TO_EN. A dict comprehension builds it. Here's the important design point: EN_TO_API isn't hand-maintained, it's derived. If you add, remove, or rename an entry in API_TO_EN, EN_TO_API updates automatically the next time the module runs.

There's exactly one place in the entire codebase where a schema change needs to happen.

Categorical values (not just column names) get the same treatment:

CATEGORICAL_MAPPINGS = {
    "validation_status": {
        "Donnée validée": "validated",
        "Donnée brute": "raw",
        "Donnée pré-validée": "pre-validated",
    },
    "quality_assessment": {
        "Bonne": "good",
        "Non qualifiée": "unqualified",
        "Douteuse": "dubious",
    },
    "production_method": {
        "Calculée": "calculated",
        "Mesurée": "measured",
        "Expertisée": "expert-reviewed",
    },
}

The functions below apply these mappings, in order, to standardize column names and category values.

rename_to_english():

1. If the DataFrame is empty, it returns a copy right away.

2. It builds a list of columns that are renamed from API names to English names.

   • It only renames a column if:

     • the old name exists, and

     • the new name does not already exist

3. Then it returns the renamed DataFrame.

rename_to_api_schema():

1. Also returns a copy for empty data.

2. Does the reverse: English names back to API names.

3. Returns the DataFrame.

(Useful if you need to send data back in the API’s original format).

apply_categorical_mappings():

1. Makes a copy so the original DataFrame is not changed.

2. For each column in CATEGORICAL_MAPPINGS, it replaces values using the mapping.

   • Example: "Donnée validée" becomes "validated".

   • .fillna(df[col_name]) keeps the original value if a value isn’t found in the mapping.

3. Returns the DataFrame.

def rename_to_english(df: pd.DataFrame) -> pd.DataFrame:
    """Rename API column names to English schema names."""
    if df.empty:
        return df.copy()

    columns_to_rename = {}
    for src, dst in API_TO_EN.items():
        if src in df.columns and dst not in df.columns:
            columns_to_rename[src] = dst

    return df.rename(columns=columns_to_rename)

def rename_to_api_schema(df: pd.DataFrame) -> pd.DataFrame:
    """Rename English column names back to API schema names (defensive/reverse operation)."""
    if df.empty:
        return df.copy()

    columns_to_rename = {k: v for k, v in EN_TO_API.items() if k in df.columns}
    return df.rename(columns=columns_to_rename)

def apply_categorical_mappings(df: pd.DataFrame) -> pd.DataFrame:
    df = df.copy()
    for col_name, mapping in CATEGORICAL_MAPPINGS.items():
        if col_name in df.columns:
            df[col_name] = df[col_name].map(mapping).fillna(df[col_name])
    return df

Two defensive habits are worth a note here:

Robustness to partial inputs: Both rename functions only rename columns that actually exist in the input (if src in df.columns).

A rename function that assumes every mapped column is always present will crash the moment it's called on a partial or differently-shaped DataFrame.

.map(mapping).fillna(df[col_name]). Series.map replaces every value it finds in the mapping dict and turns any value not found in the dict into NaN. Chain .fillna(df[col_name]) right after restores the original value wherever the mapping didn't apply. That way, unexpected categorical values pass through unchanged instead of silently becoming null. It's a subtle but important robustness choice.

See this exact .map()-then-.fillna() idiom discussed on Stack Overflow: pandas map vs apply performance.

A quick round-trip test proves the bidirectional mapping actually works:

sample_api_df = pd.DataFrame({"code_station": ["F700000109"], ...})
renamed_df = rename_to_english(sample_api_df)          # French → English
reversed_df = rename_to_api_schema(renamed_df)          # English → French
# reversed_df.columns.tolist() == sample_api_df.columns.tolist()  → True

Compute Flood Alerts

add_derived_columns() is a one-liner. It just computes flood_alert. If the water level is greater than the flood threshold set in the Config, the result is True. Otherwise, the result is False.

def add_derived_columns(df: pd.DataFrame) -> pd.DataFrame:
    """Add a computed columns based on raw data.
    """
    df = df.copy()

    if "water_level_mm" in df.columns:
        df["flood_alert"] = df["water_level_mm"] > STATION_CONFIG.flood_threshold_mm

    return df

Column Ordering

Here's a small but user-facing detail: define a preferred column order once, as data, and reuse it everywhere. COLUMN_ORDER holds that preferred column sequence.

order_columns() rearranges a DataFrame so those columns come first, while any extra columns stay at the end.

COLUMN_ORDER = [
    # Primary identifiers & measurements
    "station_code", "record_date", "water_level_mm", "flood_alert",
    # Metadata about the observation
    "hubeau_elab_code", "data_production_date",
    "validation_status_code", "validation_status",
    "production_method_code", "production_method",
    "quality_code", "quality_assessment",
    # Geographic info (less important)
    "location_code", "longitude", "latitude",
]

def order_columns(df: pd.DataFrame) -> pd.DataFrame:
    """Reorder columns to preferred order."""
    present_cols = [c for c in COLUMN_ORDER if c in df.columns]
    other_cols = [c for c in df.columns if c not in present_cols]
    return df[present_cols + other_cols]

other_cols acts as a safety net: any column not explicitly listed in COLUMN_ORDER still gets included at the end. They're not silently dropped.

Rules, in order of priority:

• Keep identifiers and key fields up front.

• Put the most important, most frequently used, and most stable columns first.

• Group related fields together, so the table reads naturally.

• Push optional or rarely-used fields to the end.

Deduplication

Design logic

Remove duplicates as you go. In an incremental pipeline, date ranges and records can overlap:

• The same day might get re-fetched, because API data arrives late or gets finalized later.

• The same observation might appear on two different pages of a paginated response.

Without deduplication, these scenarios let duplicate rows pile up in the dataset over time. This is also, recall from Part 1, exactly what makes the pipeline idempotent: run it once or run it five times and the resulting dataset stays identical.

The fix requires defining what makes a record unique. Define a composite key:

key = (station_code, observation_date, water_level_value)

Why this specific combination? Physically, one sensor (station_code) reports one day's (observation_date) daily-maximum reading (water_level_mm), and that reading should be unique.

• Records from different stations obviously aren't duplicates of each other.

• Two readings on different days aren't duplicates.

• A subtler point: if the same station reports the same day but with a different value, that counts as a distinct observation, for example a corrected or revised measurement, not a duplicate to silently discard.

Code-level walkthrough

create_dedup_key() builds a unique text ID for each row by joining three pieces together with an underscore:

• station code

• date, as YYYY-MM-DD

• water level value

Example key:

"F700000109_2024-01-15_5800.0"

def create_dedup_key(df: pd.DataFrame) -> pd.Series:
    """Create unique deduplication key from station, day, and water level value.

    Key format: "station_code_YYYY-MM-DD_value"
    Example: "F700000109_2024-01-15_5800.0"
    """
    parts = []

    if "code_station" in df.columns:
        parts.append(df["code_station"].astype(str))

    if "date_obs_elab" in df.columns:
        parts.append(df["date_obs_elab"].dt.strftime("%Y-%m-%d"))

    if "resultat_obs_elab" in df.columns:
        parts.append(df["resultat_obs_elab"].astype(str))

    if not parts:
        return pd.Series(index=df.index, dtype="object")

    return pd.Series(
        ["_".join(row) for row in zip(*parts)],
        index=df.index
    )

How the key-building works: parts ends up as a list of Series, one per key component (station, date, value), each the same length as the DataFrame.

Read the last line from the inside out:

    return pd.Series(["_".join(row) for row in zip(*parts)], index=df.index)

zip(*parts) transposes that list of columns into row-wise tuples. It yields (station_1, date_1, value_1), then (station_2, date_2, value_2), and so on. The list comprehension then joins each row-tuple with underscores into one string key per row.

This "list of columns → zip → row tuples" idiom is a common and efficient way to combine several Series into one derived Series, without writing .apply(lambda row: ..., axis=1). Row-wise .apply is notoriously slow in pandas compared to vectorized string operations.

The actual deduplication happens in remove_duplicates(). It removes rows from "new" (freshly fetched data) that already exist in "existing" (historic data).

Step by step:

1. If one table is empty, it just returns new.

2. It enforces types in both DataFrames with auto_convert_columns() first, so dates and numbers compare correctly.

3. It creates a deduplication key for each row in both tables with create_dedup_key().

4. It checks which keys from the fetched data don't appear in the existing historic dataset, and keeps only truly new rows.

5. It returns the filtered result, keeping the original columns from new.

def remove_duplicates(existing: pd.DataFrame, new: pd.DataFrame) -> pd.DataFrame:
    """Remove rows from 'new' that already exist in 'existing'."""
    # Short-circuit: if either is empty, no work to do
    if existing.empty or new.empty:
        return new.copy()

    # Parse types on both sides for a fair comparison
    existing_std = auto_convert_columns(existing)
    new_std = auto_convert_columns(new)

    # Build the keys
    existing_keys = set(create_dedup_key(existing_std).dropna())
    new_keys = create_dedup_key(new_std)

    # Boolean mask: True where the new row is genuinely new
    mask = ~new_keys.isin(existing_keys)

    # Index back into the ORIGINAL (non-standardized) 'new' to preserve all columns
    result = new.iloc[new_keys[mask].index].copy()
    return result

Two performance and robustness details are worth to note:

First, existing_keys is a set, not a list. Testing membership (in / .isin()) against a Python set is O(1) on average, because it uses a fast hash lookup to answer is this item already here? directly. Testing against a list is O(n): it has to scan item by item and it gets slower as the existing dataset grows.

For a dataset with tens of thousands of rows, checked on every single pipeline run, that difference matters and that's a textbook example of choosing the right data structure for the job. See the general discussion of list vs. set lookup performance in Python and this guidance on .isin() performance for filtering.

Simple rule: use a set when you care about fast membership checks. Use a list when you care about order or duplicates.

Second, new.iloc[new_keys[mask].index] indexes back into the original, non-validated new DataFrame, not new_std.

Why? auto_convert_columns() only ran to get consistent types for comparison. The caller still wants the original raw values and schema back for everything that survives deduplication.

Don't let a side-computation accidentally become your source of truth. Always modify a copy only to decide what to keep or remove. Once you've made that decision, apply it to the original data. That way you preserve the real source values for later steps.

In short:

1. First modifications are only for comparison.

2. Use that comparison to filter the original data.

3. Return the original rows unchanged, so later stages can process them.

Best practice: keep your deduplication key simple and stable (immutable) and always short-circuit (with if existing.empty or new.empty: return new.copy()) before doing any heavier work. This cheap guard clause skips the expensive deduplication logic when there's no data to process.

Put it All Together in postprocess()

All the pieces above are small and testable on their own. The last step of Transform glues them together, in order, into the single pipeline function below:

def postprocess(df: pd.DataFrame) -> pd.DataFrame:
    """Apply all post-processing transformations."""
    if df.empty:
        return df

    df = df.copy()

    print("  1. Converting types...")
    df = auto_convert_columns(df)

    print("  2. Renaming columns (French → English)...")
    df = rename_to_english(df)

    print("  3. Mapping categorical values...")
    df = apply_categorical_mappings(df)

    print("  4. Adding derived columns...")
    df = add_derived_columns(df)

    print("  5. Reordering columns...")
    df = order_columns(df)

    print("  6. Sorting and resetting index...")
    df = df.sort_values(["record_date", "station_code"]).reset_index(drop=True)

    return df

Notice the shape of this function: it's basically a linear script with six numbered, printed steps. Each one calls a previously-defined pure function. There's no new logic here, only sequencing. That's intentional.

Best practice: log progress clearly at each stage. When a scheduled job fails at 3am, a clear step-by-step log is the difference between a two-minute diagnosis and an hour of guessing. A good option is to use a logger. Here, this tutorial sticks with classic console printing (print(f" 1. Converting types...")).

Part 6: The Load Step

Design Logic

Load is the final stage: take the processed data and move it to its destination. Typical destinations include data warehouses, data lakes, and databases.

Before publishing, the pipeline prepares two things: the output folder itself, and a metadata file that describes the dataset.

Code-level walkthrough

create_output_dir() makes sure an output folder exists, and creates one if it doesn't:

• If use_mock is True, it uses KAGGLE_CONFIG.mock_output_dir, the directory for fake data.

• Otherwise, it uses the real output directory set in KAGGLE_CONFIG.output_dir.

def create_output_dir(use_mock: bool = False) -> None:
    """Create output directory (and parents) if it does not already exist."""
    output_dir = KAGGLE_CONFIG.mock_output_dir if use_mock else KAGGLE_CONFIG.output_dir
    output_dir.mkdir(parents=True, exist_ok=True)
    # exist_ok=True: idempotent, safe to call multiple times

exist_ok=True is a small but important detail. Without it, Path.mkdir() raises FileExistsError if the directory already exists, which it will on every run after the first.

Setting exist_ok=True makes directory creation idempotent: calling it 100 times has the same effect as calling it once. On the filesystem, this is the "safe to re-run" idea behind the deduplication logic from Part 5.

The Kaggle API follows the Data Package specification. It requires a descriptive dataset-metadata.json file alongside the CSV. This matters for the search and discoverability of the Paris flood dataset:

def create_metadata(df: pd.DataFrame, config: KaggleConfig) -> Dict:
    """Generate Kaggle dataset metadata from DataFrame and config."""
    if df.empty or "record_date" not in df.columns:
        first_date = "unknown"
        last_date = "unknown"
    else:
        first_date = df["record_date"].min().strftime("%Y-%m-%d")
        last_date = df["record_date"].max().strftime("%Y-%m-%d")

    return {
        "title": config.title,
        "id": config.dataset_slug,
        "licenses": [{"name": config.license_name}],
        "keywords": config.keywords,
        "temporalCoverage": {"startDate": first_date, "endDate": last_date},
        "geospatialCoverage": config.geospatial_coverage,
        "updateFrequency": config.update_frequency,
    }

Metadata comes from the KaggleConfig dataclass. Note that temporalCoverage is computed from the data itself (df["record_date"].min()/.max()) rather than hardcoded. Every time the pipeline runs, the metadata's date range automatically reflects reality, with zero manual bookkeeping.

Finally, publish_to_kaggle() uploads the updated dataset to Kaggle. It calls out to the Kaggle CLI as an external command.

What it does:

1. Gets the current time and turns it into a text label like Weekly update: 2026-07-13 14:30:00

2. Builds a Kaggle command that says:

   • publish a new dataset version

   • use files from KAGGLE_CONFIG.output_dir

   • attach the message

   • zip the directory contents

3. Prints the command so you can see what will run

4. Runs the command with subprocess.run(...)

If the upload works:

• it prints Successfully published to Kaggle.

If it fails:

• it prints the error and then raises the error again so the failure isn't hidden.

def publish_to_kaggle() -> None:
    """Publish the updated dataset to Kaggle using the Kaggle CLI."""
    timestamp = pd.Timestamp.now().strftime("%Y-%m-%d %H:%M:%S")
    message = f"Weekly update: {timestamp}"

    cmd = [
        "kaggle", "datasets", "version",
        "-p", str(KAGGLE_CONFIG.output_dir),
        "-m", message,
        "--dir-mode", "zip",
    ]

    print("Publishing to Kaggle...")
    print("Command:", " ".join(cmd))

    try:
        subprocess.run(cmd, check=True)
        print("Successfully published to Kaggle.")
    except subprocess.CalledProcessError as e:
        print(f"Error publishing to Kaggle: {e}")
        raise

publish_to_kaggle calls the Kaggle CLI (Command Line Interface) through Python's subprocess module. This is the standard way to run an external command-line tool from a Python pipeline.

Two choices here are worth adopting as general habits, not just for Kaggle:

• cmd is built as a list, never as a concatenated string. Pass a list of arguments to subprocess.run skips invoking a shell entirely. That sidesteps shell-injection risks and correctly handles arguments containing spaces or special characters. See the official subprocess security considerations.

• check=True means that if the Kaggle CLI fails, by exiting with a non-zero status, subprocess.run raises CalledProcessError instead of silently returning. Without check=True, a failed publish would look exactly like a successful one to the rest of the code.

Part 7: Assemble the Full Pipeline

The Global Rehearsal (Mock Mode)

The whole pipeline is built and tested in isolation. It's time to wire the blocks into one linear function. First, run it entirely against mock data. Nothing gets published this way. For Demonstration purposes, a CSV filled with artificial data will land in a local mock_output folder.

def run_etl_pipeline(use_mock: bool = True) -> pd.DataFrame:
    """Run the complete ETL pipeline: Extract → Transform → Load to CSV."""

    # STEP 1: LOAD EXISTING DATA
    if use_mock:
        loaded_df = pd.DataFrame(mock_api_response)
        loaded_df['date_obs_elab'] = pd.to_datetime(loaded_df['date_obs_elab'])
        loaded_df = rename_to_english(loaded_df)
    else:
        loaded_df = load_csv(KAGGLE_CONFIG.input_csv)

    # STEP 2: DETERMINE UPDATE RANGE
    should_update, start_date = determine_update_range(loaded_df)
    if not should_update:
        return loaded_df   # already current: nothing more to do

    # STEP 3: EXTRACT (FETCH DATA)
    fetched_data = fetch_all_data(start_date, use_mock=use_mock)

    # STEP 4: TRANSFORM I - DEDUPLICATE
    deduped_fetched_data = remove_duplicates(loaded_df, fetched_data)

    # STEP 5: TRANSFORM II - COMBINE (MERGE WITH EXISTING)
    new_df_english = rename_to_english(deduped_fetched_data)
    merged_historical_and_new = pd.concat([loaded_df, new_df_english], ignore_index=True)

    # STEP 6: TRANSFORM III - POST-PROCESS
    processed_records = postprocess(merged_historical_and_new)

    # STEP 7: EXPORT (LOAD TO CSV)
    create_output_dir(use_mock=use_mock)
    output_path = KAGGLE_CONFIG.mock_output_filename if use_mock else KAGGLE_CONFIG.output_csv_path
    processed_records.to_csv(output_path, index=False, sep=",")

    return processed_records

Every one of the seven steps above corresponds to a function you already built, and tested earlier in this tutorial. To Assemble them is almost mechanical. That's always the payoff of composing small, single-responsibility functions:

loaded_df                 = load_csv(...)                          # 1. load existing data
should_update, start_date = determine_update_range(...)            # 2. check what's needed
fetched_data               = fetch_all_data(start_date)             # 3. fetch new records
deduped_fetched_data       = remove_duplicates(existing, new_raw)   # 4. deduplicate
merged_historical_and_new  = pd.concat([existing, new_clean])       # 5. merge
processed_records          = postprocess(merged_historical_and_new) # 6. postprocess
processed_records.to_csv(...)                                       # 7. save
write_metadata() + publish_to_kaggle()                               # 8. publish

Each line's intent is obvious just from reading it left to right. That is exactly the point of good decomposition.

main(): Pipeline Orchestration

main() is the conductor. It wires every previously-built component together in the right order, just like postprocess() did one level down.

def main() -> None:
    """Execute the full Paris Flood Monitoring ETL pipeline.

    EXTRACT
    1. Load the existing dataset from the Kaggle input mount
    2. Determine whether an update is needed (and from what date)
    3. Fetch new data from the Hub'Eau API

    TRANSFORM
    4. Deduplicate against the existing dataset
    5. Combine and post-process (type parsing, translation, derived cols...)

    LOAD
    6. Write the updated CSV and metadata file
    7. Publish the new dataset version to Kaggle

    Exits early if the dataset is already up to date.
    """
    final_dataset = run_etl_pipeline(use_mock=False)
    final_dataset.to_csv(KAGGLE_CONFIG.output_csv_path, index=False)
    write_metadata(final_dataset)

    print("\n[FINAL STEP] Publishing to Kaggle...")
    publish_to_kaggle()

    print("Running post-run validation")
    validate_and_analyze(final_dataset)


if __name__ == "__main__":
    main()

The if __name__ == "__main__": Guard

if __name__ == "__main__":
    main()

This is one of the most common idioms in Python, and it's worth understanding exactly what it does. Per the official Python documentation on __main__:

• Run the file directly (python script.py)

  • → the special variable __name__ is set to "__main__"

  • → the condition is True

  • → main() executes.

• Import the file as a module elsewhere (import script)

  • → __name__ is set to the module's name instead

  • → the condition is False

  • → main() does not run automatically.

Best practice: always guard your entry point this way. It makes the module safely importable for testing individual functions, or for reuse in another script, without triggering the full pipeline (including a real publish to Kaggle!) just by importing it.

Part 8: Go Live and Switch to the Real API

Everything above runs safely against mock data. To point the pipeline at the real Hub'Eau API instead:

1. Note the use_mock=False setting above. It threads through APIConfig.use_mock, and the calls to run_etl_pipeline(use_mock=False) and main().

2. Make sure KAGGLE_CONFIG.dataset_slug and input_csv point at your own Kaggle dataset copy before you use it. You can only publish new versions of a dataset you own. Fork both the notebook and the dataset, then update the slug in the KaggleConfig section.

3. Install and authenticate the Kaggle CLI if you plan to call publish_to_kaggle() outside of a Google Colab or Kaggle notebook.

Everything else needs zero changes.

Post-Run Validation

validate_and_analyze() closes the loop, and it's more than just a nice-to-have. It's a welcome sanity check that runs after the pipeline finishes. The output is a human-readable report covering shape, data types, null counts, summary statistics on water_level_mm, how many flood-alert records turned up, the temporal coverage and per-station record counts.

It doesn't change any data. It exists so that whoever, or whatever monitoring system, reads the run's log output can immediately see whether this week's numbers look sane. No need to analyze the CSV by hand. Try it!

Summary

You've just built a complete ETL pipeline that you can run again and again without breaking anything. The skeleton is here, and that same pattern repeats for almost any scheduled data job: swap out the API, the field mapping, and the destination.

For more tutorials like this one, check out my GitHub or Kaggle profile.

Thanks for reading!

References

• Hub'Eau API: France's official open water-data platform

• The updater .py script runs live once a week and updates the original dataset.

• The source dataset generator is available on GitHub.

• The complete Jupyter notebook to follow this tutorial and code all along (backup available).

This tutorial demonstrates how to clean messy JSON and export the results into a new file, based on a predefined schema. The JSON file we’ll be cleaning contains a dataset of 200 synthetic customer records.

In this tutorial, we’ll apply two methods for cleaning the input data:

• With pure Python

• With pandas

You can apply either of these in your code. But the pandas method is better for large, complex data sets. Let’s jump right into the process.

Here’s what we’ll cover:

• Prerequisites

• Add and Inspect the JSON File

• Define the Target Schema

• How to Clean JSON Data with Pure Python

• How to Clean JSON Data with Pandas

• How to Validate the Cleaned JSON

• Pandas vs Pure Python for Data Cleaning

Prerequisites

To follow along with this tutorial, you should have a basic understanding of:

• Python dictionaries, lists, and loops

• JSON data structure (keys, values, and nesting)

• How to read and write JSON files with Python’s json module

Add and Inspect the JSON File

Before you begin writing any code, make sure that the .json file you intend to clean is in your project directory. This makes it easy to load in your script using the file name alone.

You can now inspect the data structure by viewing the file locally or loading it in your script, with Python’s built-in json module.

Here’s how (assuming the file name is “old_customers.json”):

This shows you whether the JSON file is structured as a dictionary or a list. It also prints out the entire file in your terminal. Mine is a dictionary that maps to a list of 200 customer entries. You should always open up the raw JSON file in your IDE to get a closer look at its structure and schema.

Define the Target Schema

If someone asks for JSON data to be cleaned, it probably means that the current schema is unsuitable for its intended purpose. At this point, you want to be clear on what the final JSON export should look like.

JSON schema is essentially a blueprint that describes:

• required fields

• field names

• data type for each field

• standardized formats (for example, lowercase emails, trimmed whitespace, etc.)

Here’s what the old schema versus the target schema looks like:

As you can see, the goal is to delete the ”customer_id” and ”address” fields in each entry and rename the rest from:

• ”name” to ”full_name”

• ”email” to ”email_address”

• ”phone” to ”mobile”

• ”membership_level” to ”tier”

The output should contain 4 response fields instead of 6, all renamed to fit the project requirements.

How to Clean JSON Data with Pure Python

Let’s explore using Python’s built-in json module to align the raw data with the predefined schema.

Step 1: Import json and time modules

Importing json is necessary because we’re working with JSON files. But we’ll use the time module to track how long the data cleaning process takes.

import json
import time

Step 2: Load the file with json.load()

start_time = time.time()
with open('old_customers.json') as file:
    crm_data = json.load(file)

Step 3: Write a function to loop through and clean each customer entry in the dictionary

def clean_data(records):
    transformed_records = []
    for customer in records["customers"]:
        transformed_records.append({
                "full_name": customer["name"],
                "email_address": customer["email"],
                "mobile": customer["phone"],
                "tier": customer["membership_level"],

                })
    return {"customers": transformed_records}

new_data = clean_data(crm_data)

clean_data() takes in the original data (temporarily) stored in the records variable, transforming it to match our target schema.

Since the JSON file we loaded is a dictionary containing a ”customers” key, which maps to a list of customer entries, we access this key and loop through each entry in the list.

In the for loop, we rename the relevant fields and store the cleaned entries in a new list called ”transformed_records”.

Then, we return the dictionary, with the ”customers” key intact.

Step 4: Save the output in a .json file

Decide on a name for your cleaned JSON data and assign that to an output_file variable, like so:

output_file = "transformed_data.json"
with open(output_file, "w") as f:
    json.dump(new_data, f, indent=4)

You can also add a print() statement below this block to confirm that the file has been saved in your project directory.

Step 5: Time the data cleaning process

At the beginning of this process, we imported the time module to measure how long it takes to clean up JSON data using pure Python. To track the runtime, we stored the current time in a start_time variable before the cleaning function, and we’ll now include an end_time variable at the end of the script.

The difference between the end_time and start_time values gives you the total runtime in seconds.

end_time = time.time()
elapsed_time = end_time - start_time

print(f"Transformed data saved to {output_file}")
print(f"Processing data took {elapsed_time:.2f} seconds")

Here’s how long the data cleaning process took with the pure Python approach:

How to Clean JSON Data with Pandas

Now we’re going to try achieving the same results as above, using Python and a third-party library called pandas. Pandas is an open-source library used for data manipulation and analysis in Python.

To get started, you need to have the Pandas library installed in your directory. In your terminal, run:

pip install pandas

Then follow these steps:

Step 1: Import the relevant libraries

import json
import time
import pandas as pd

Step 2: Load file and extract customer entries

Unlike the pure Python method, where we simply indexed the key name ”customers” to access the list of customer data, working with pandas requires a slightly different approach.

We must extract the list before loading it into a DataFrame because pandas expects structured data. Extracting the list of customer dictionaries upfront ensures that we isolate and clean the relevant records alone, preventing errors caused by nested or unrelated JSON data.

start_time = time.time()
with open('old_customers.json', 'r') as f:
    crm_data = json.load(f)

#Extract the list of customer entries
clients = crm_data.get("customers", [])

Step 3: Load customer entries into a DataFrame

Once you’ve got a clean list of customer dictionaries, load the list into a DataFrame and assign said list to a variable, like so:

#Load into a dataframe
df = pd.DataFrame(clients)

This creates a tabular or spreadsheet-like structure, where each row represents a customer. Loading the list into a DataFrame also allows you to access pandas’ powerful data cleaning methods like:

• drop_duplicate(): removes duplicate rows or entries from a DataFrame

• dropna(): drops rows with any missing or null data

• fillna(value): replaces all missing or null data with a specified value

• drop(columns): drops unused columns explicitly

Step 4: Write a custom function to rename relevant fields

At this point, we need a function that takes in a single customer entry – a row – and returns a cleaned version that fits the target schema (“full_name”, “email_address”, “mobile” and “tier”).

The function should also handle missing data by setting default values like ”Unknown” or ”N/A” when a field is absent.

P.S: At first, I used drop(columns) to explicitly remove the “address” and “customer_id” fields. But it’s not needed in this case, as the transform_fields() function only selects and renames the required fields. Any extra columns are automatically excluded from the cleaned data.

Step 5: Apply schema transformation to all rows

We’ll use pandas' apply() method to apply our custom function to each row in the DataFrame. This will creates a Series (for example, 0 → {...}, 1 → {...}, 2 → {...}), which is not JSON-friendly.

As json.dump() expects a list, not a Pandas Series, we’ll apply tolist(), converting the Series to a list of dictionaries.

#Apply schema transformation to all rows
transformed_df = df.apply(transform_fields, axis=1)

#Convert series to list of dicts
transformed_data = transformed_df.tolist()

Another way to approach this is with list comprehension. Instead of using apply() at all, you can write:

transformed_data = [transform_fields(row) for row in df.to_dict(orient="records")]

orient=”records” is an argument for df.to_dict that tells pandas to convert the DataFrame to a list of dictionaries, where each dictionary represents a single customer record (that is, one row).

Then the for loop iterates through every customer record on the list, calling the custom function on each row. Finally, the list comprehension ([...]) collects the cleaned rows into a new list.

Step 6: Save the output in a .json file

#Save the cleaned data
output_data = {"customers": transformed_data}
output_file = "applypandas_customer.json"
with open(output_file, "w") as f:
    json.dump(output_data, f, indent=4)

I recommend picking a different file name for your pandas output. You can inspect both files side by side to see if this output matches the result you got from cleaning with pure Python.

Step 7: Track runtime

Once again, check for the difference between start time and end time to determine the program’s execution time.

end_time = time.time()
elapsed_time = end_time - start_time

#print(f"Transformed data saved to {output_file}")
print(f"Transformed data saved to {output_file}")
print(f"Processing data took {elapsed_time:.2f} seconds")

When I used list comprehension to apply the custom function, my script’s runtime was 0.03 seconds, but with pandas’ apply() function, the total runtime dropped to 0.01 seconds.

Final output preview:

If you followed this tutorial closely, your JSON output should look like this – whether you used the pandas method or the pure Python approach:

How to Validate the Cleaned JSON

Validating your output ensures that the cleaned data follows the expected structure before being used or shared. This step helps to catch formatting errors, missing fields, and wrong data types early.

Below are the steps for validating your cleaned JSON file:

Step 1: Install and import jsonschema

jsonschema is a third-party validation library for Python. It helps you define the expected structure of your JSON data and automatically check if your output matches that structure.

In your terminal, run:

pip install jsonschema

Import the required libraries:

import json
from jsonschema import validate, ValidationError

validate() checks whether your JSON data matches the rules defined in your schema. If the data is valid, nothing happens. But if there’s an error – like a missing field or wrong data type – it raises a ValidationError.

Step 2: Define a schema

As you know, JSON schema changes with each file structure. If your JSON data differs from what we’ve been working with so far, learn how to create a schema here. Otherwise, the schema below defines the structure we expect for our cleaned JSON:

schema = {
    "type": "object",
    "properties": {
        "customers": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "full_name": {"type": "string"},
                    "email_address": {"type": "string"},
                    "mobile": {"type": "string"},
                    "tier": {"type": "string"}
                },
                "required": ["full_name", "email_address", "mobile", "tier"]
            }
        }
    },
    "required": ["customers"]
}

• The data is an object that must contain a key: "customers".

• "customers" must be an array (a list), with each object representing one customer entry.

• Each customer entry must have four fields–all strings:

  • "full_name"

  • "email_address"

  • "mobile"

  • "tier"

• The "required" fields ensure that none of the relevant fields are missing in any customer record.

Step 3: Load the cleaned JSON file

with open("transformed_data.json") as f:
    data = json.load(f)

Step 4: Validate the data

For this step, we’ll use a try. . . except block to end the process safely, and display a helpful message if the code raises a ValidationError.

try:
    validate(instance=data, schema=schema)
    print("JSON is valid.")
except ValidationError as e:
    print("JSON is invalid:", e.message)

Pandas vs Pure Python for Data Cleaning

From this tutorial, you can probably tell that using pure Python to clean and restructure JSON is the more straightforward approach. It is fast and ideal for handling small datasets or simple transformations.

But as data grows and becomes more complex, you might need advanced data cleaning methods that Python alone does not provide. In such cases, pandas becomes the better choice. It handles large, complex datasets effectively, providing built-in functions for handling missing data and removing duplicates.

You can study the Pandas cheatsheet to learn more data manipulation methods.

This statement is 100% accurate, as this encompasses a series of steps that ensure data used for data science, machine learning and analysis projects are complete, accurate, unbiased and reliable.

The quality of your dataset plays a pivotal role in the success of your analysis or model. As the saying goes, “garbage in, garbage out”, the quality and reliability of your model and analysis heavily depends on the quality of your data.

Raw data, collected from various sources, are often messy, contain errors, inconsistencies, missing values and outliers. Data cleaning and preprocessing aims to identify and rectify these issues to ensure accurate, reliable and meaningful results during model building and data analysis as wrong conclusions could be costly.

This is where Pandas comes into play, it is a wonderful tool used in the data world to do both data cleaning and preprocessing. In this article, we'll delve into the essential concepts of data cleaning and preprocessing using the powerful Python library, Pandas.

Table of Contents

• Prerequisites

• Introduction

• What is Data Cleaning?

• What is Data Processing?

• How to Import the Necessary Libraries

• How to Load the Dataset

• Exploratory Data Analysis (EDA)

• How to Handle Missing Values

• How to Remove Duplicate Records

• Data Types and Conversion

• How to Encode Categorical Variables

• How to Handle Outliers

• Conclusion

Prerequisites

• A basic understanding of Python.

• Basic understanding of data cleaning.

Introduction

Pandas is a popular open-source data manipulation and analysis library for Python. It provides easy-to-use functions needed to work with structured data seamlessly.

Pandas also integrates seamlessly with other popular Python libraries, such as NumPy for numerical computing and Matplotlib for data visualization. This makes it a powerful asset for data driven tasks.

Pandas excels in handling missing data, reshaping datasets, merging and joining multiple datasets, and performing complex operations on data, making it exceptionally useful for data cleaning and manipulation.

At its core, Pandas introduces two key data structures: Series and DataFrame. A Series is a one-dimensional array-like object that can hold any data type, while a DataFrame is a two-dimensional table with labeled axes (rows and columns). These structures allow users to manipulate, clean, and analyze datasets efficiently.

What is Data Cleaning?

Before we embark on our data adventure with Pandas, let's take a moment to explain the term "data cleaning." Think of it as the digital detox for your dataset, where we tidy up, and and prioritize accuracy above all else.

Data cleaning involves identifying and rectifying errors, inconsistencies, and missing values within a dataset. It's like preparing your ingredients before cooking; you want everything in order to get the perfect analysis or visualization.

Why bother with data cleaning? Well, imagine trying to analyze sales trends when some entries are missing, or working with a dataset that has duplicate records throwing off your calculations. Not ideal, right?

In this digital detox, we use tools like Pandas to get rid of inconsistencies, straighten out errors, and let the true clarity of your data shine through.

What is Data Processing?

You may be wondering, "Does data cleaning and data preprocessing mean the same thing?" The answer is no – they do not.

Picture this: you stumble upon an ancient treasure chest buried in the digital sands of your dataset. Data cleaning is like carefully unearthing that chest, dusting off the cobwebs, and ensuring that what's inside is authentic and reliable.

As for data preprocessing, you can think of it as taking that discovered treasure and preparing its contents for public display. It goes beyond cleaning; it's about transforming and optimizing the data for specific analyses or tasks.

Data cleaning is the initial phase of refining your dataset, making it readable and usable with techniques like removing duplicates, handling missing values and data type conversion while data preprocessing is similar to taking this refined data and scaling with more advanced techniques such as feature engineering, encoding categorical variables and and handling outliers to achieve better and more advanced results.

The goal is to turn your dataset into a refined masterpiece, ready for analysis or modeling.

How to Import the Necessary Libraries

Before we embark on data cleaning and preprocessing, let's import the Pandas library.

To save time and typing, we often import Pandas as pd. This lets us use the shorter pd.read_csv() instead of pandas.read_csv() for reading CSV files, making our code more efficient and readable.

import pandas as pd

How to Load the Dataset

Start by loading your dataset into a Pandas DataFrame.

In this example, we'll use a hypothetical dataset named your_dataset.csv. We will load the dataset into a variable called df.

#Replace 'your_dataset.csv' with the actual dataset name or file path
df = pd.read_csv('your_dataset.csv')

Exploratory Data Analysis (EDA)

EDA helps you understand the structure and characteristics of your dataset. Some Pandas functions help us gain insights into our dataset. We call these functions by calling the dataset variable plus the function.

For example:

• df.head() will call the first 5 rows of the dataset. You can specify the number of rows to be displayed in the parentheses.

• df.describe() gives some statistical data like percentile, mean and standard deviation of the numerical values of the Series or DataFrame.

• df.info() gives the number of columns, column labels, column data types, memory usage, range index, and the number of cells in each column (non-null values).

Here's a code example below:

#Display the first few rows of the dataset
print(df.head())

#Summary statistics
print(df.describe())

#Information about the dataset
print(df.info())

How to Handle Missing Values

As a newbie in this field, missing values pose a significant stress as they come in different formats and can adversely impact your analysis or model.

Machine learning models cannot be trained with data that has missing or "NAN" values as they can alter your end result during analysis. But do not fret, Pandas provides methods to handle this problem.

One way to do this is by removing the missing values altogether. Code snippet below:

#Check for missing values
print(df.isnull().sum())

#Drop rows with missing valiues and place it in a new variable "df_cleaned"
df_cleaned = df.dropna()

#Fill missing values with mean for numerical data and place it ina new variable called df_filled
df_filled = df.fillna(df.mean())

But if the number of rows that have missing values is large, then this method will be inadequate.

For numerical data, you can simply compute the mean and input it into the rows that have missing values. Code snippet below:

#Replace missing values with the mean of each column
df.fillna(df.mean(), inplace=True)

#If you want to replace missing values in a specific column, you can do it this way:
#Replace 'column_name' with the actual column name
df['column_name'].fillna(df['column_name'].mean(), inplace=True)

#Now, df contains no missing values, and NaNs have been replaced with column mean

How to Remove Duplicate Records

Duplicate records can distort your analysis by influencing the results in ways that do not accurately show trends and underlying patterns (by producing outliers).

Pandas helps to identify and remove the duplicate values in an easy way by placing them in new variables.

Code snippet below:

#Identify duplicates
print(df.duplicated().sum())

#Remove duplicates
df_no_duplicates = df.drop_duplicates()

Data Types and Conversion

Data type conversion in Pandas is a crucial aspect of data preprocessing, allowing you to ensure that your data is in the appropriate format for analysis or modeling.

Data from various sources are usually messy and the data types of some values may be in the wrong format, for example some numerical values may come in 'float' or 'string' format instead of 'integer' format and a mix up of these formats leads to errors and wrong results.

You can convert a Column of type int to float with the following code:

#Convert 'Column1' to float
df['Column1'] = df['Column1'].astype(float)

#Display updated data types
print(df.dtypes)

You can use df.dtypes to print column data types.

How to Encode Categorical Variables

For machine learning algorithms, having categorical values in your dataset (non-numerical values) is crucial in ensuring the best model as they are equally as important.

These could be car brand names in a cars dataset for predicting car prices. But machine learning algorithms cannot processes this datatype, therefore it must be converted to numerical data before it can be used.

Pandas provides the get_dummies function which converts categorical values into numerical format(Binary format) such that it is recognized by the algorithm as a placeholder for values and not hierarchical data that can undergo numerical analysis. this just means that the numbers the brand name is converted to is not interpreted as 1 is greater than 0, but it tells the algorithm that both 1 and 0 are placeholders for categorical data. Code snippet is shown below:

#To convert categorical data from the column "Car_Brand" to numerical data
df_encode = pd.get_dummies(df, columns=[Car_Brand])

#The categorical data is converted to binary format of Numerical data

How to Handle Outliers

Outliers are data points significantly different from the majority of the data, they can distort statistical measures and adversely affect the performance of machine learning models.

They may be caused by human error, missing NaN values, or could be accurate data that does not correlate with the rest of the data.

There are several methods to identify and remove outliers, they are:

• Remove NaN values.

• Visualize the data before and after removal.

• Z-score method (for normally distributed data).

• IQR (Interquartile range) method for more robust data.

The IQR is useful for identifying outliers in a dataset. According to the IQR method, values that fall below Q1−1.5× IQR or above Q3+1.5×IQR are considered outliers.

This rule is based on the assumption that most of the data in a normal distribution should fall within this range.

Here's a code snippet for the IQR method:

#Using median calculations and IQR, outliers are identified and these data points should be removed
Q1 = df["column_name"].quantile(0.25)
Q3 = df["column_name"].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
df = df[df["column_name"].between(lower_bound, upper_bound)]

Conclusion

Data cleaning and preprocessing are integral components of any data analysis, science or machine learning project. Pandas, with its versatile functions, facilitates these processes efficiently.

By following the concepts outlined in this article, you can ensure that your data is well-prepared for analysis and modeling, ultimately leading to more accurate and reliable results.

We just published a course on the freeCodeCamp.org YouTube channel that will teach you how to use Pandas through interactive projects. You will develop 7 projects ranging from the basics of Pandas for Data Analysis, to Data Cleaning and Data Wrangling.

This course targets everyone, from data science enthusiasts to professionals, aiming to refine their skills in data analysis, data cleaning, and data wrangling using Pandas and Python.

Santiago Basulto developed this course. His is knowledgeable and has an engaging teaching style, guarantees an enriching learning experience. Santiago is also the creator of datawars.io, a platform that offers a bunch of interactive data science projects.

A Sneak Peek into the Course

The course is designed to provide you with hands-on experience through real-life projects. The projects vary in complexity, catering to learners with different skill levels. It’s advisable to try resolving the projects independently and then compare your solutions with Santiago’s.

Below are the projects you will build.

For Beginners:

DataFrames Practice: Working with English Words: This project is an excellent entry point for beginners. You will get acquainted with the basics of Pandas DataFrames, focusing on understanding and manipulating their structures, all while working with an extensive dictionary of English words.

Filtering and Sorting with Pokemon Data: Dive into the captivating world of Pokemon as you perform fundamental data analysis tasks like filtering and sorting. This project is not only educational but also fun, making it perfect for those just starting.

Intermediate Level:

The Birthday Paradox in the NBA: Have you heard of the Birthday Paradox? Discover the answer to an intriguing question: How many people need to be in a room to have a 50% probability that at least two people share a birthday? Apply these insights to explore shared player birthdays within NBA teams.

Matching Strings by Similarity using Levenshtein Distance: String handling is an integral aspect of data cleaning. This project introduces advanced techniques such as Combinatorics and the Levenshtein distance to detect irregularities in company names.

Data Cleaning with Google Playstore Dataset: This project is a comprehensive guide to data cleaning. Learn how to identify and rectify null values, duplicate values, outliers, and more, using a dataset scraped from the Google Playstore, which is replete with irregularities.

Advanced:

Premier League Match Analysis: This project is designed for advanced learners. It combines data cleaning with analysis based on grouping operations, using data from the Premier League, the top-tier football league in England.

NBA 2017 Season Analysis: Joining and Groupby Practice: This project is a test of your data wrangling skills. Learn how to merge different dataframes, clean them, and perform analysis and question/answering tasks using the 2017 NBA statistics dataset.

Conclusion

Data is the new oil, and learning how to refine it is a skill that is in high demand. The Pandas course on freeCodeCamp.org, developed by Santiago Basulto, is a great opportunity to enhance your data science career. You can watch the full course on the freeCodeCamp.org YouTube channel (5-hour watch).

A dataframe is two-dimensional data structure in Pandas that organizes data in a tabular format with rows and columns.

In this article, you'll learn how to get the number of rows in a dataframe using the following:

• The len() function.
• The shape attribute.
• The index attribute.
• The axes attribuite.

How to Get the Number of Rows in a Dataframe Using the len() Function

You can use the len() function to return the length of an object. With a dataframe, the function returns the number of rows.

Consider the dataframe below:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

In the example above, we created a dataframe with three rows — row 0, 1, and 2.

You can use the len() function to verify the number of rows:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = len(df)

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

In the code above, we passed the dataframe as a parameter to the len() function and stored it in a variable called num_of_rows:

num_of_rows = len(df)

When num_of_rows was printed, we got a value of 3 (the number of rows).

How to Get the Number of Rows in a Dataframe Using the shape Attribute

The shape attribute returns a tuple with the number of rows and columns in a dataframe.

Here's an example using the same dataframe as in the last section:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.shape

print(num_of_rows)
# (3, 2)

In the code above, a tuple — (3, 2) — was returned when we used the shape attribute on the dataframe: df.shape.

The first value, 3, is the number of rows in the dataframe while the second value, 2, is the number of columns.

Since we're only interested in the number of rows, we can extract just that value using its index in the tuple (remember that index numbers start at 0). That is:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.shape[0]

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

Now we're getting just the number of rows using its index in the tuple: df.shape[0].

How to Get the Number of Rows in a Dataframe Using the index Attribute

You can use the index attribute to access the number of elements in a dataframe, which corresponds with the number of rows.

You can do this in two different ways:

• Using the index attribute's size property.
• Passing the index property as a parameter to the len() function.

Here are examples to explain the methods above:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.index.size

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

In the example above, we accessed the number of rows in the dataframe using df.index.size.

Without the size property, you'd get a result like this: RangeIndex(start=0, stop=3, step=1).

• start denotes the first index number.
• stop denotes the number of rows in the dataframe.
• step denotes the way indexes are incremented (indexes are increased by 1 in our case).

So the size property is way of specifying that you're only interested in the number of elements in the dataframe.

Here's another example that uses the len() function:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = len(df.index)

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

In the code above, we passed df.index as a parameter to the len() function. This returns the number of rows in the dataframe.

The difference between this example and the previous one is that we're not attaching the size property to df.index. Instead, we're using df.index as the len() function's parameter.

How to Get the Number of Rows in a Dataframe Using the axes Attribute

The axes attribute returns the value as the index attribute: RangeIndex(start=0, stop=3, step=1).

Similarly, you can return the number of rows using either the size property or the len() function:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.axes[0].size

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = len(df.axes[0])

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

The logic in the two code blocks above is the same as those in the last section:

• df.index.size returns the number of elements/rows in the dataframe.
• len(df.index) returns the number of rows in the dataframe.

Summary

In this article, we talked about dataframes in Pandas. They are two-dimensional data structures that organize data in rows and columns.

We saw different methods for getting the number of rows in a dataframe. We discussed the following methods along with code examples to show their application:

• The len() function.
• The shape attribute.
• The index attribute.
• The axes attribuite.

Happy coding! You can learn more about Python on my blog.

In this article, you'll learn about the differences between Python and Pandas, and what they are used for.

You'll start by learning what each technology is used for, then you'll see their differences in tabular format.

Let's get started!

What Is Python Used For?

Python is a popular high-level, general-purpose programming language. Python has a simple syntax that is easy to read, write and understand.

It has an active open source community.

Python has a variety of libraries and frameworks that can be used in developing different applications and products.

Here are some of the use cases for Python:

• Web development: Python can be used to create full-stack web applications using frameworks Django, and Flask.
• Machine Learning: You can use Python libraries like TensorFlow, PyTorch, and so on to build machine learning models.
• Game Development: Ren'Py, Pygame, and Panda3D are some of the Python frameworks that can be used to build cross-platform games.
• Data Science and Analysis: There are numerous libraries and frameworks that can be used in data science and analysis like Matplotlib, NumPy, Pandas, and so on. They can also be used for scientific computing.

From the use cases above, you should have an idea of the first difference between Python and Pandas — Python is a programming language while Pandas is a Python library.

What Is Pandas Used For?

Pandas is an open source Python library used for manipulating and analyzing data.

Here are some of the use cases:

• Data manipulation.
• Data analysis.
• Data visualization.
• Machine learning, and so on.

What Are the Differences Between Python and Pandas?

Here are some of the differences between Python and Pandas:

Summary

In this article, we talked about the differences between Python and Pandas. We saw what each of them is used for, and their differences in tabular format.

Python is a general-purpose programming language used in different fields like web development, machine learning, and so on.

Pandas is a Python library used mainly for data manipulation and analysis.

Happy coding!

This article provides a comprehensive guide on how to loop through a Pandas DataFrame in Python.

I'll start by introducing the Pandas library and DataFrame data structure. I'll explain the essential characteristics of Pandas, how to loop through rows in a dataframe, and finally how to loop through columns in a dataframe.

What is Pandas?

Pandas is a popular open-source Python library that's used for data cleaning, analysis, and manipulation. In addition to functions for carrying out operations on those datasets, it offers data structures for effectively storing and handling large and complex datasets.

Some of the essential characteristics of Pandas are:

• DataFrame and Series Objects: Pandas provides two primary data structures, DataFrames and Series. They allow users to store and manipulate tabular data and time series data, respectively. These data structures are highly efficient and can handle large datasets with ease.
• Data Cleaning and Preparation: Pandas provides a wide range of functions and methods for cleaning and preparing data, including handling missing values, removing duplicates, and transforming data.
• Data Analysis and Visualization: Pandas provides powerful functions for performing data analysis, including statistical functions and grouping and aggregation functions. It also integrates well with other data analysis and visualization libraries in Python, such as Matplotlib and Seaborn.
• Data Input and Output: Pandas provides functions for reading and writing data in a variety of formats, including CSV, Excel, SQL databases, and more.

What is a Pandas Dataframe?

In Pandas, a dataframe is a two-dimensional labeled data structure. It is comparable to a spreadsheet or a SQL table, where data is arranged in rows and columns with a variety of data types in each column.

Since dataframes offer an easy way to store, manipulate, and analyze data, they are frequently used in data science and data analysis applications. Dataframes provide a number of features, including pivoting, grouping, indexing, and filtering, that make it simple to carry out complex operations on data.

How to Loop Through Rows in a Dataframe

You can loop through rows in a dataframe using the iterrows() method in Pandas. This method allows us to iterate over each row in a dataframe and access its values.

Here's an example:

import pandas as pd

# create a dataframe
data = {'name': ['Mike', 'Doe', 'James'], 'age': [18, 19, 29]}
df = pd.DataFrame(data)

# loop through the rows using iterrows()
for index, row in df.iterrows():
    print(row['name'], row['age'])

Output:

Mike 18
Doe 19
James 29

In this example, we first create a dataframe with two columns, name and age. We then loop through each row in the dataframe using iterrows(), which returns a tuple containing the index of the row and a Series object that contains the values for that row.

Within the loop, we can access the values for each column by using the column name as an index on the row object. For example, to access the value for the name column, we use row['name'].

How to Loop Through Columns in a Dataframe

Looping through columns in a dataframe is a common task in data analysis and manipulation. It's different from the way we loop through rows, though.

Here's an example:

import pandas as pd

# Create a sample dataframe
df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6],
    'C': [7, 8, 9]
})

# Loop through columns using a for loop
for col in df.columns:
    print(col)

Output:

A
B
C

First, we import the Pandas library using the import pandas as pd statement.

Then, we create a sample dataframe using the pd.DataFrame() function, which takes a dictionary of column names and values as an input.

Next, we loop through the columns of the dataframe using a for loop and the df.columns attribute, which returns a list of column names.

Inside the loop, we simply print the name of each column using the print() function.

Use Cases for Looping Through a Dataframe

Looping through a dataframe is an important technique in data analysis and manipulation, as it allows us to perform operations on each row or column of the dataframe.

You'll loop through dataframes in the following activities:

• Data Cleaning and Transformation.
• Data Analysis.
• Data Visualization.
• Feature Engineering.

Conclusion

By looping through the rows in a dataframe, we can perform operations on each row, such as filtering or transforming the data.

But it's important to note that looping through rows in a dataframe can be slow and inefficient for large datasets. In general, it's often better to use vectorized operations or apply() functions to perform operations on dataframes, as these methods are optimized for performance.

Let's connect on Twitter and on LinkedIn. You can also subscribe to my YouTube channel.

Happy Coding!

In Pandas, sometimes you'll need to remove columns from a DataFrame for various reasons, such as cleaning data, reducing memory usage, or simplifying analysis. And in this article, I'll show you how to do it.

I'll start by introducing the .drop() method, which is the primary method for removing columns in Pandas. We'll go through the syntax and parameters of the .drop() method, including how to specify columns to remove and how to control whether the original DataFrame is modified in place or a new DataFrame is returned.

Next, I'll provide an example of how to use the .drop() method to remove columns from a DataFrame.

How to Use the .drop() Method in Pandas

The .drop() method is a built-in function in Pandas that allows you to remove one or more rows or columns from a DataFrame. It returns a new DataFrame with the specified rows or columns removed and does not modify the original DataFrame in place, unless you set the inplace parameter to True.

The syntax for using the .drop() method is as follows:

DataFrame.drop(labels=None, axis=0, index=None, columns=None, level=None, inplace=False, errors='raise')

Here, DataFrame refers to the Pandas DataFrame that you want to remove rows or columns from. The parameters you can use with the .drop() method include:

• labels: This parameter specifies the labels or indices of the rows or columns to be removed. You can pass either a single label or index or a list of labels or indices.
• axis: This parameter specifies whether to remove rows or columns. By default, it is set to 0, which means rows are removed. If you want to remove columns, set it to 1.
• index and columns: These parameters are alternative to the labels parameter and specify the labels or indices of rows or columns to be removed, respectively.
• level: This parameter is used to remove a specific level of a hierarchical index.
• inplace: This parameter is a boolean value that determines whether to modify the original DataFrame in place. By default, it is set to False.
• errors: This parameter specifies how to handle errors if the specified label(s) or index(es) are not found in the DataFrame. By default, it is set to raise, which means that a KeyError is raised. Other options are ignore and warn, which will respectively ignore or display a warning when the label/index is not found.

How to Remove a Single Column from a Dataframe in Pandas

Let's ease into it by first learning how to remove a single column from a Dataframe before we remove multiple columns.

Code sample:

import pandas as pd

# create a sample dataframe
data = {'name': ['Alice', 'Bob', 'Charlie'],
        'age': [25, 30, 35],
        'gender': ['F', 'M', 'M']
        }
df = pd.DataFrame(data)

# display the original dataframe
print('Original DataFrame:\n', df)

# drop the 'gender' column
df = df.drop(columns=['gender'])

# display the modified dataframe
print('Modified DataFrame:\n', df)

Output:

Original DataFrame:
       name  age gender
0    Alice   25      F
1      Bob   30      M
2  Charlie   35      M

Modified DataFrame:
       name  age
0    Alice   25
1      Bob   30
2  Charlie   35

Code explanation:

In the example above, we first created a sample DataFrame with three columns – name, age, and gender. We then used the .drop() method with the columns parameter to remove the gender column. The resulting DataFrame only contains the name and age columns.

It's important to note that the .drop() method does not modify the original DataFrame in place. Instead, it returns a new DataFrame with the specified column(s) removed. If you want to modify the original DataFrame, you need to assign the result of the .drop() method back to the original variable, as we did in the example above.

In addition to the columns parameter, the .drop() method also has a number of other optional parameters you can use to control how columns are removed.

For example, you can use the inplace parameter to modify the original DataFrame in place instead of returning a new DataFrame. You can also use the axis parameter to remove columns by index instead of name.

How to Remove Multiple Columns from a Dataframe in Pandas

In this section we will remove multiple columns from our dataframe. This approach is similar to removing a single column from the dataframe.

To remove two or more columns from a DataFrame using the .drop() method in Pandas, we can pass a list of column names to the columns parameter of the method.

Code sample:

import pandas as pd

# create a sample dataframe
data = {'name': ['John', 'Mary', 'Peter'],
        'age': [30, 25, 35],
        'gender': ['Male', 'Female', 'Male'],
        'city': ['New York', 'London', 'Paris']}
df = pd.DataFrame(data)

# remove the 'gender' and 'city' columns
df.drop(columns=['gender', 'city'], inplace=True)

# print the modified dataframe
print(df)

Output:

    name  age
0   John   30
1   Mary   25
2  Peter   35

Code explanation:

In this example, we first create a sample DataFrame with four columns – name, age, gender, and city. Then, we use the .drop() method to remove the gender and city columns by passing a list of their names to the columns parameter. Finally, we set the inplace parameter to True to modify the original DataFrame and print the modified DataFrame.

Note that you can also remove columns by their indices by passing a list of indices to the columns parameter. For example, to remove the second and third columns, you can use:

df.drop(columns=df.columns[1:3], inplace=True)

This will remove the columns with indices 1 and 2 (that is the age and gender columns in this example).

Conclusion

I hope this article is a useful resource for anyone working with Pandas DataFrames who needs to remove columns efficiently and effectively.

Let's connect on Twitter and on LinkedIn. You can also subscribe to my YouTube channel.

Happy Coding!

Pandas is a widely used open-source library in Python for data manipulation and analysis. It provides a range of data structures and functions for working with data, one of which is the DataFrame.

DataFrames are a powerful tool for storing and analyzing large sets of data, but they can be challenging to work with if they are not saved or exported correctly.

It is common practice in data analysis to export data from Pandas DataFrames into CSV files because it can help conserve time and resources. Due to their portability and ability to be easily read by numerous applications, CSV files are a common file format for storing and distributing tabular data.

Regardless of whether you are a novice or an expert data analyst, this article will walk you through the process of saving Pandas DataFrames into CSV files and give you useful tips on how to do so.

How to Save Pandas DataFrames Using the .to_csv() Method

The .to_csv() method is a built-in function in Pandas that allows you to save a Pandas DataFrame as a CSV file. This method exports the DataFrame into a comma-separated values (CSV) file, which is a simple and widely used format for storing tabular data.

The syntax for using the .to_csv() method is as follows:

DataFrame.to_csv(filename, sep=',', index=False, encoding='utf-8')

Here, DataFrame refers to the Pandas DataFrame that we want to export, and filename refers to the name of the file that you want to save your data to.

The sep parameter specifies the separator that should be used to separate values in the CSV file. By default, it is set to , for comma-separated values. We can also set it to a different separator like \t for tab-separated values.

The index parameter is a boolean value that determines whether to include the index of the DataFrame in the CSV file. By default, it is set to False, which means the index is not included.

The encoding parameter specifies the character encoding to be used for the CSV file. By default, it is set to utf-8, which is a standard encoding for text files.

Code example

import pandas as pd

# Create a sample dataframe
Biodata = {'Name': ['John', 'Emily', 'Mike', 'Lisa'],
        'Age': [28, 23, 35, 31],
        'Gender': ['M', 'F', 'M', 'F']
        }
df = pd.DataFrame(Biodata)

# Save the dataframe to a CSV file
df.to_csv('Biodata.csv', index=False)

Code explanation

Let's break down what each part of this code does:

• import pandas as pd: This imports the Pandas library and assigns it the alias pd, which is a commonly used convention.
• Biodata = {'Name': ['John', 'Emily', 'Mike', 'Lisa'], 'Age': [28, 23, 35, 31], 'Gender': ['M', 'F', 'M', 'F']}: This creates a Python dictionary with the data we want to store in the DataFrame. Each key represents a column in the DataFrame, and its corresponding value is a list of values for that column.
• df = pd.DataFrame(Biodata): This creates a Pandas DataFrame from the Biodata dictionary.
• df.to_csv('Biodata.csv', index=False): This saves the DataFrame to a CSV file named Biodata.csv.

Other Ways to Save Pandas DataFrames

There are several alternative methods to .to_csv() for saving Pandas DataFrames into various file formats, including:

1. to_excel(): This method is used to save a DataFrame as an Excel file.
2. to_json(): This method is used to save a DataFrame as a JSON file.
3. to_hdf(): This method is used to save a DataFrame as an HDF5 file, which is a hierarchical data format commonly used in scientific computing.
4. to_sql(): This method is used to save a DataFrame to a SQL database.
5. to_pickle(): This method is used to save a DataFrame as a pickled object, which is a serialized representation of the DataFrame.

These alternative methods provide flexibility in choosing the file format that best suits your use case and can be particularly useful for advanced data analysis and sharing.

Conclusion

Thanks for reading! I hope you now understand how you can easily convert your Pandas Dataframes by exporting into a CSV file using the build-in to_csv() method.

Let's connect on Twitter and on LinkedIn. You can also subscribe to my YouTube channel.

Happy Coding!

In this article, you'll learn how to sort data in ascending and descending order using Pandas' sort_values() method.

How To Sort Values in Pandas

You can use the sort_values() method to sort values in a data set. By default, the method sorts values in ascending order.

In this section, you'll learn how to sort data in ascending and descending order using the sort_values() method.

How To Sort Values in Ascending Order Using Pandas sort_values() Method

The sort_values() method takes in multiple parameters, as can be seen in the Pandas documentation.

We'll focus on the by and ascending parameters. That is:

Dataframe.sort_values(by, ascending)

• The by parameter denotes the column or index to sort.
• ascending is used to specify what other the values should be sorted in. By default, it is set to True.

Here's an example:

import pandas as pd

# create a sample dataframe
data = {'cost': [50000, 30000, 70000, 60000]}

df = pd.DataFrame(data)

df

In the table above, we have different items along with the cost of each item. To sort the items in ascending order using their cost, you can do this:

import pandas as pd

data = {'item': ['laptop', 'monitor', 'HDMI', 'speaker'],
        'cost': [500, 300, 700, 600]
       }

df = pd.DataFrame(data)

sorted_data = df.sort_values(by='cost', ascending=True)

sorted_data

In the code above, the sort_values() method was used to sort the cost column.

• Using the by parameter, we specified which column was to be sorted: by='cost'
• Using the ascending parameter, we set the order of the data to be sorted: ascending=True.

Note that the default order of the sort_values() method is ascending=True. So if you remove ascending parameter, you'd still have the values sorted in ascending order.

How To Sort Values in Descending Order Using Pandas sort_values() Method

You can sort values in descending order by simply setting the ascending parameter to False.

We'll work with the same code in the last section:

import pandas as pd

# create a sample dataframe
data = {'cost': [50000, 30000, 70000, 60000]}

df = pd.DataFrame(data)

df

Here's the code for sorting the cost column in descending order:

import pandas as pd

data = {'item': ['laptop', 'monitor', 'HDMI', 'speaker'],
        'cost': [500, 300, 700, 600]
       }

df = pd.DataFrame(data)

sorted_data = df.sort_values(by='cost', ascending=False)

sorted_data

By setting the value of the ascending parameter to False, we've sorted the data by cost in descending order.

Summary

In this article, we learned about sorting values in Pandas using the sort_values() method.

We saw two code examples on how to sort data in Pandas in ascending or descending order.

You can use the sort_values() method's ascending parameter to sort data in ascending or descending order.

Happy coding!

In this article, you'll learn how to use the Pandas round() method to round a float value to a specified number of decimal places.

We'll begin by looking the method's syntax, and then see some practical code applications.

Pandas round() Method Example

Here's what the syntax for the round() method looks like:

DataFrame.round(decimals)

The decimals parameter represents the number of decimal places a number should be rounded to.

The number of decimal places to be returned is passed in as a parameter. round(2) return rounds a number to two decimal places.

Here's an example to demonstrate:

import pandas as pd

data = {'cost':[20.5550, 21.03535, 19.67373, 18.233233]}

df = pd.DataFrame(data)

df['rounded_cost'] = df['cost'].round(2)
print(df)

In the code above, we have a list of numbers that fall under the cost column. The column had these values: [20.5550, 21.03535, 19.67373, 18.233233].

Using the round() method, we rounded the values to 2 decimal places: df['cost'].round(2).

The return values were stored in a column called rounded_cost.

Here's the output of the code:

From the table above, you can see that the values in the cost column have been rounded to 2 decimal places in the rounded_cost columns.

Summary

In this article, we have learned about rounding float values with Pandas using the round() method.

We started looking at the syntax for the round() method looks like. We then saw an example using the method to round float values to a specified number of decimal places.

Happy coding!

Pandas is a fast and approachable open-source library in Python built for analyzing and manipulating data.

This library has a lot of functions and methods to expedite the data analysis process. One of my favorites is the groupby method, mainly because it lets you get quick insights into your data by transforming, aggregating, and splitting data into various categories.

In this article, you will learn about the Pandas groupby function, how to aggregate data, and group Pandas DataFrames with multiple columns using the groupby method.

What do I need to install on my computer to follow this article?

For this article, I'll be using a Jupyter notebook. You can install Jupyter notebook and get it up and running on your computer via the official website.

After installing Juypter, create a new notebook and run Import pandas as pd to import pandas and Import numpy as np to import NumPy.

NumPy will let us work with multi-dimensional arrays and high-level mathematical functions. On the other hand, Pandas will allow us to manipulate our data and access the df.groupby(), the groupby method.

Let's get started.

Importing the required libraries

What is groupby in Pandas?

If you're familiar with SQL and its GROUP BY syntax, you already know how powerful it is in summarizing and categorizing data.

The Pandas groupby method in Python does the same thing and is great when splitting and categorizing data into groups to analyze your data better.

Here is the syntax for Pandas groupby:

python DataFrame.groupby(by=None, axis=0, level=None, as_index=True, sort=True, group_keys=_NoDefault.no_default, squeeze=_NoDefault.no_default, observed=False, dropna=True)

Each attribute has a meaning:

• by – List of the columns you want to group by.
• axis – Defaults to 0. It takes 0 or 'index', 1 or 'columns'.
• level – Used with MultiIndex.
• as_index – SQL style grouped otput.
• sort – Defaults to True. Specify whether to sort after grouping.
• group_keys – add group keys or not.
• squeeze – deprecated in new versions.
• observed – Only use if any of the groupers are Categoricals.
• dropna – Defaults to False. Use True to drop None/Nan.

Now let's see how this function works in action.

How to Load the Dataset

For this tutorial, we'll use the supermarket sales dataset from Kaggle, which you can access and download here.

After downloading the dataset, load the data into a pandas dataframe.

A DataFrame is a 2-dimensional data structure made up of rows and columns. This is very similar to your spreadsheet.

You can do that by running this code:

df = pd.read_csv(r"C:\Users\Double Arkad\Downloads\archive\supermarket_sales - Sheet1.csv")

After that, use the df.head() method to show the first few rows of your dataset. After running df.head(), you should get the result below. This indicates that the dataset got loaded successfully.

Dataset has loaded correctly

How to Use the groupby Method in Pandas

Assume your employer asked you to total the number of items ordered and categorize them according to the different payment options. This will let you determine which payment method generates the most revenue.

You can answer this question with the groupby function by simply grouping the data based on the 'Payment'.

df.groupby('Payment')['Quantity'].sum()

Using the sum function with groupby

The first column, 'Payments', is the column you want to group by. The second column, 'Quantity' is the column you'll perform an aggregate function on. Lastly, you have the aggregate function .sum().

The Sum() is one of many functions you can use in a groupby. You could also use other aggregate functions like the Min(), Mean(), Median(), Count(), and Average() to find the minimum, mean, median, count, and average value in a group within your dataset.

But by using the agg() function, you can perform two or more aggregations simultaneously.

Let's see how that works.

How to Aggregate Data Using groupby in Pandas

Pandas groupby and Agg()

Here's how to use agg() in a groupby function to find this supermarket's most used payment method.

df.groupby('Payment')['Quantity'].agg([np.sum, np.mean])

Using the agg function with groupby

There are more cash transactions done. Ewallets and credit card transactions follow in level of use.

Notice here we created a dictionary and passed the aggregate functions to be performed. This simultaneously performed two statistical computations on our data! Of course, you can add more aggregate functions in the dictionary depending on the insights you want to get.

Here is what I mean:

df.groupby(['Payment', 'Customer type'])['Quantity'].agg([np.sum, np.mean, np.max, np.min])

Adding more aggregate functions

In the groupby function, we added more aggregate functions to our statistical computation to gain insight into the maximum and the minimum number of goods ordered in each payment group.

Pandas groupby and count()

Here's how it works:

df.groupby('Payment')['Quantity'].count()

And here's the result you get:

Using the count function with groupby

From the output, we're counting the total number of orders placed in the store and grouping the results by each payment method.

How to Group Pandas DataFrames by Multiple Columns

You can also group multiple columns in the groupby function. For example, we included a column below to our groupby function called 'Customer type'.

df.groupby(['Payment', 'Customer type'])['Quantity'].sum()

Grouping multiple columns

Our output shows that the data was split and categorized into two groups based on the Customer type column. The output is becoming easier to analyze.

How to Aggregate Multiple Columns Using Pandas groupby

You can also perform statistical computations on multiple columns with the groupby function. For example, let's look at the total sales generated and quantity ordered and group our results by the "Payment" and "Customer type" columns.

Run the code:

df.groupby(['Payment', 'Customer type']) [['Quantity','Unit price']].sum()

Using the sum function with multiple columns grouping

We can see from the output that the Payment type "Ewallet" generated the most revenue, and you can move on to determine which type of Customers contributed the most revenue for the Store.

Summary

In this article, you learned about the importance of the Pandas groupby method. You saw how the groupby function allows you to do a lot of operations on your data, from splitting the data to applying a function like Sum() to get more insight and add more functionality.

To learn more about Python and how you can use it for data analysis, I'll recommend this Python for data analysis course on the freeCodeCamp YouTube channel.

If you enjoyed reading this article and/or have questions and want to connect, you can find me on LinkedIn or Twitter.

In this article, you'll learn how to rename columns in a Pandas Dataframe by using:

• The rename() function.
• A List.
• The set_axis() function.

How to Rename a Column in Pandas Using the rename() Function

In this section, you'll see a practical example of renaming a Pandas Dataframe using the rename() function.

Let's begin by passing data into a Dataframe object:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)

print(df)

# Output
  firstname lastname
0      John      Doe
1      Jane     Done
2      Jade       Do

In the example above, we created a Python dictionary which we used to store the firstname and lastname of students.

We then converted the dictionary to a Dataframe by passing it as a parameter to the Pandas Dataframe object: pd.DataFrame(students).

When printed to the console, we had this table printed out:

  firstname lastname
0      John      Doe
1      Jane     Done
2      Jade       Do

The goal here is to rename the columns. We can do that using the rename() function.

Here's what the syntax looks like:

df.rename(columns={"OLD_COLUMN_VALUE": "NEW_COLUMN_VALUE"})

Let's go ahead and change the column names (firstname and lastname) in the table from lowercase to uppercase (FIRSTNAME and LASTNAME).

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)

df.rename(columns={"firstname": "FIRSTNAME", "lastname": "LASTNAME"}, inplace=True)

print(df)

# Output
  FIRSTNAME LASTNAME
0      John      Doe
1      Jane     Done
2      Jade       Do

In the code above, we specified that the columns firstname and lastname should be renamed to FIRSTNAME and LASTNAME, respectively: df.rename(columns={"firstname": "FIRSTNAME", "lastname": "LASTNAME"}, inplace=True)

You'll notice that we added the inplace=True parameter. This helps in persisting the new changes in the Dataframe. Delete the parameter and see what happens ;)

You can rename the columns to whatever you want. For instance, we can use SURNAME instead of lastname by doing this:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)
df.rename(columns={"firstname": "FIRSTNAME", "lastname": "SURNAME"}, inplace=True)

print(df)

# Output
  FIRSTNAME SURNAME
0      John     Doe
1      Jane    Done
2      Jade      Do

You can change just one column name, too. You are not required to change all the column names at the same time.

How to Rename a Column in Pandas Using a List

You can access the column names of a Dataframe using df.columns. Consider the table below:

  firstname lastname
0      John      Doe
1      Jane     Done
2      Jade       Do

We can print out the column names with the code below:

print(df.columns)

# Index(['firstname', 'lastname'], dtype='object')

Using that, we can rename the column of a Dataframe. Here's an example:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)
df.columns = ["FIRSTNAME", "SURNAME"]

print(df)

# Output
  FIRSTNAME SURNAME
0      John     Doe
1      Jane    Done
2      Jade      Do

In the example above, we put the new column names in a List and assigned it to the Dataframe columns: df.columns = ["FIRSTNAME", "SURNAME"].

This will override the previous column names.

How to Rename a Column in Pandas Using the set_axis() Function

The syntax for renaming a column with the set_axis() function looks like this:

df.set_axis([NEW_COLUMN_NAME,...], axis="columns")

Here's a code example:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)

df.set_axis(["FIRSTNAME", "SURNAME"], axis="columns", inplace=True) 

print(df)

# Output
  FIRSTNAME SURNAME
0      John     Doe
1      Jane    Done
2      Jade      Do

Note that the inplace=True parameter might raise a warning because it's deprecated for the set_axis() function and will be replaced in the future.

Summary

In this article, we talked about renaming a column in Pandas.

We saw different methods that can be used to rename a Pandas Dataframe column with code examples.

Happy coding!

Data Analysis is the process of exploring, investigating, and gathering insights from data using statistical measures and visualizations.

The objective of data analysis is to develop an understanding of data by uncovering trends, relationships, and patterns.

Data analysis is both a science and an art. On the one hand it requires that you know statistics, visualization techniques, and data analysis tools like Numpy, Pandas, and Seaborn.

On the other hand, it requires that you ask interesting questions to guide the investigation, and then interpret the numbers and figures to generate useful insights.

This tutorial on data analysis covers the following topics:

1. What is Numerical Computation? Python and Numpy for Beginners
2. How to Analyze Tabular Data using Python and Pandas
3. Data Visualization using Python, Matplotlib, and Seaborn

What is Numerical Computation? Python and Numpy for Beginners

_Source: Elegant Scipy_

You can follow along with the tutorial and run the code here: https://jovian.ai/aakashns/python-numerical-computing-with-numpy

This section covers the following topics:

• How to work with numerical data in Python
• How to turn Python lists into Numpy arrays
• Multi-dimensional Numpy arrays and their benefits
• Array operations, broadcasting, indexing, and slicing
• How to work with CSV data files using Numpy

How to Work with Numerical Data in Python

The "data" in Data Analysis typically refers to numerical data, like stock prices, sales figures, sensor measurements, sports scores, database tables, and so on.

The Numpy library provides specialized data structures, functions, and other tools for numerical computing in Python. Let's work through an example to see why and how to use Numpy to work with numerical data.

Suppose we want to use climate data like the temperature, rainfall, and humidity to determine if a region is well suited for growing apples.

A simple approach to do this would be to formulate the relationship between the annual yield of apples (tons per hectare) and the climatic conditions like the average temperature (in degrees Fahrenheit), rainfall (in millimeters), and average relative humidity (in percentage) as a linear equation.

yield_of_apples = w1 * temperature + w2 * rainfall + w3 * humidity

We're expressing the yield of apples as a weighted sum of the temperature, rainfall, and humidity.

This equation is an approximation, since the actual relationship may not necessarily be linear, and there may be other factors involved. But a simple linear model like this often works well in practice.

Based on some statistical analysis of historical data, we might come up with reasonable values for the weights w1, w2, and w3. Here's an example set of values:

w1, w2, w3 = 0.3, 0.2, 0.5

Given some climate data for a region, we can now predict the yield of apples. Here's some sample data:

To begin, we can define some variables to record climate data for a region.

kanto_temp = 73
kanto_rainfall = 67
kanto_humidity = 43

We can now substitute these variables into the linear equation to predict the yield of apples.

kanto_yield_apples = kanto_temp * w1 + kanto_rainfall * w2 + kanto_humidity * w3
kanto_yield_apples
# 56.8

print("The expected yield of apples in Kanto region is {} tons per hectare.".format(kanto_yield_apples))
# The expected yield of apples in Kanto region is 56.8 tons per hectare.

To make it slightly easier to perform the above computation for multiple regions, we can represent the climate data for each region as a vector, that is a list of numbers.

kanto = [73, 67, 43]
johto = [91, 88, 64]
hoenn = [87, 134, 58]
sinnoh = [102, 43, 37]
unova = [69, 96, 70]

The three numbers in each vector represent the temperature, rainfall, and humidity data, respectively.

We can also represent the set of weights used in the formula as a vector.

weights = [w1, w2, w3]

We can now write a function crop_yield to calculate the yield of apples (or any other crop) given the climate data and the respective weights.

def crop_yield(region, weights):
    result = 0
    for x, w in zip(region, weights):
        result += x * w
    return result

crop_yield(kanto, weights)
# 56.8

crop_yield(johto, weights)
# 76.9

crop_yield(unova, weights)
# 74.9

How to Turn Python Lists into Numpy Arrays

The calculation performed by the crop_yield (element-wise multiplication of two vectors and taking a sum of the results) is also called the dot product. Learn more about dot products here.

The Numpy library provides a built-in function to compute the dot product of two vectors. However, we must first convert the lists into Numpy arrays.

Let's install the Numpy library using the pip package manager.

!pip install numpy --upgrade --quiet

Next, let's import the numpy module. It's common practice to import numpy with the alias np.

import numpy as np

We can now use the np.array function to create Numpy arrays.

kanto = np.array([73, 67, 43])

kanto
# array([73, 67, 43])

weights = np.array([w1, w2, w3])

weights
# array([0.3, 0.2, 0.5])

Numpy arrays have the type ndarray.

type(kanto)
# numpy.ndarray

type(weights)
# numpy.ndarray

Just like lists, Numpy arrays support the indexing notation [].

weights[0]
# 0.3

kanto[2]
#43

How to Operate on Numpy arrays

We can now compute the dot product of the two vectors using the np.dot function.

np.dot(kanto, weights)
# 56.8

We can achieve the same result with low-level operations supported by Numpy arrays: performing an element-wise multiplication and calculating the resulting numbers' sum.

(kanto * weights).sum()
# 56.8

The * operator performs an element-wise multiplication of two arrays if they have the same size. The sum method calculates the sum of numbers in an array.

arr1 = np.array([1, 2, 3])
arr2 = np.array([4, 5, 6])

arr1 * arr2
# array([ 4, 10, 18])

arr2.sum()
# 15

What are the Benefits of Using Numpy Arrays?

Numpy arrays offer the following benefits over Python lists for operating on numerical data:

• They're easy to use: You can write small, concise, and intuitive mathematical expressions like (kanto * weights).sum() rather than using loops and custom functions like crop_yield.
• Performance: Numpy operations and functions are implemented internally in C++, which makes them much faster than using Python statements and loops that are interpreted at runtime

Here's a comparison of dot products performed using Python loops vs. Numpy arrays on two vectors with a million elements each.

# Python lists
arr1 = list(range(1000000))
arr2 = list(range(1000000, 2000000))

# Numpy arrays
arr1_np = np.array(arr1)
arr2_np = np.array(arr2)

%%time
result = 0
for x1, x2 in zip(arr1, arr2):
    result += x1*x2
result

# CPU times: user 300 ms, sys: 3.26 ms, total: 303 ms
# Wall time: 302 ms
# 833332333333500000

%%time
np.dot(arr1_np, arr2_np)

# CPU times: user 2.11 ms, sys: 951 µs, total: 3.07 ms
# Wall time: 1.58 ms
# 833332333333500000

As you can see, using np.dot is 100 times faster than using a for loop. This makes Numpy especially useful while working with really large datasets with tens of thousands or millions of data points.

Multi-Dimensional Numpy Arrays

We can now go one step further and represent the climate data for all the regions using a single 2-dimensional Numpy array.

climate_data = np.array([[73, 67, 43],
                         [91, 88, 64],
                         [87, 134, 58],
                         [102, 43, 37],
                         [69, 96, 70]])

climate_data
# array([[ 73,  67,  43],
#        [ 91,  88,  64],
#        [ 87, 134,  58],
#        [102,  43,  37],
#        [ 69,  96,  70]])

If you've taken a linear algebra class in high school, you may recognize the above 2-d array as a matrix with five rows and three columns. Each row represents one region, and the columns represent temperature, rainfall, and humidity, respectively.

Numpy arrays can have any number of dimensions and different lengths along each dimension. We can inspect the length along each dimension using the .shape property of an array.

_Source: Elegant Scipy_

# 2D array (matrix)
climate_data.shape
# (5, 3)

weights
# array([0.3, 0.2, 0.5])

# 1D array (vector)
weights.shape
# (3,)

# 3D array 
arr3 = np.array([
    [[11, 12, 13], 
     [13, 14, 15]], 
    [[15, 16, 17], 
     [17, 18, 19.5]]])

arr3.shape
# (2, 2, 3)

All the elements in a numpy array have the same data type. You can check the data type of an array using the .dtype property.

weights.dtype
# dtype('float64')

climate_data.dtype
# dtype('int64')

If an array contains even a single floating point number, all the other elements are also converted to floats.

arr3.dtype
# dtype('float64')

We can now compute the predicted yields of apples in all the regions, using a single matrix multiplication between climate_data (a 5x3 matrix) and weights (a vector of length 3). Here's what it looks like visually:

You can learn about matrices and matrix multiplication by watching the first 3-4 videos of this YouTube playlist.

We can use the np.matmul function or the @ operator to perform matrix multiplication.

np.matmul(climate_data, weights)
# array([56.8, 76.9, 81.9, 57.7, 74.9])

climate_data @ weights
# array([56.8, 76.9, 81.9, 57.7, 74.9])

How to Work with CSV Data Files

Numpy also provides helper functions reading from and writing to files. Let's download a file climate.txt, which contains 10,000 climate measurements (temperature, rainfall, and humidity) in the following format:

temperature,rainfall,humidity
25.00,76.00,99.00
39.00,65.00,70.00
59.00,45.00,77.00
84.00,63.00,38.00
66.00,50.00,52.00
41.00,94.00,77.00
91.00,57.00,96.00
49.00,96.00,99.00
67.00,20.00,28.00
...

This format of storing data is known as comma-separated values or CSV.

CSVs: A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. (Wikipedia)

To read this file into a numpy array, we can use the genfromtxt function.

import urllib.request

urllib.request.urlretrieve(
    'https://hub.jovian.ml/wp-content/uploads/2020/08/climate.csv', 
    'climate.txt')

climate_data = np.genfromtxt('climate.txt', delimiter=',', skip_header=1)

climate_data
# array([[25., 76., 99.],
#        [39., 65., 70.],
#        [59., 45., 77.],
#        ...,
#        [99., 62., 58.],
#        [70., 71., 91.],
#        [92., 39., 76.]])

climate_data.shape
# (10000, 3)

We can now perform a matrix multiplication using the @ operator to predict the yield of apples for the entire dataset using a given set of weights.

weights = np.array([0.3, 0.2, 0.5])

yields = climate_data @ weights
yields
# array([72.2, 59.7, 65.2, ..., 71.1, 80.7, 73.4])

yields.shape
# (10000,)

Let's add the yields to climate_data as a fourth column using the np.concatenate function.

climate_results = np.concatenate((climate_data, yields.reshape(10000, 1)), axis=1)

climate_results
# array([[25. , 76. , 99. , 72.2],
#        [39. , 65. , 70. , 59.7],
#        [59. , 45. , 77. , 65.2],
#        ...,
#        [99. , 62. , 58. , 71.1],
#        [70. , 71. , 91. , 80.7],
#        [92. , 39. , 76. , 73.4]])

There are a couple of subtleties here:

• Since we wish to add new columns, we pass the argument axis=1 to np.concatenate. The axis argument specifies the dimension for concatenation.
• The arrays should have the same number of dimensions, and the same length along each except the dimension used for concatenation. We use the np.reshape function to change the shape of yields from (10000,) to (10000,1).

Here's a visual explanation of np.concatenate along axis=1 (can you guess what axis=0 results in?):

Source: w3resource.com

The best way to understand what a Numpy function does is to experiment with it and read the documentation to learn about its arguments and return values. Use the cells below to experiment with np.concatenate and np.reshape.

Let's write the final results from our computation above back to a file using the np.savetxt function.

np.savetxt('climate_results.txt', 
           climate_results, 
           fmt='%.2f', 
           delimiter=',',
           header='temperature,rainfall,humidity,yeild_apples', 
           comments='')

The results are written back in the CSV format to the file climate_results.txt.

temperature,rainfall,humidity,yeild_apples
25.00,76.00,99.00,72.20
39.00,65.00,70.00,59.70
59.00,45.00,77.00,65.20
84.00,63.00,38.00,56.80
...

Numpy provides hundreds of functions for performing operations on arrays. Here are some commonly used functions:

• Mathematics: np.sum, np.exp, np.round, arithmetic operators
• Array manipulation: np.reshape, np.stack, np.concatenate, np.split
• Linear Algebra: np.matmul, np.dot, np.transpose, np.eigvals
• Statistics: np.mean, np.median, np.std, np.max

So how do you find the function you need?** The easiest way to find the right function for a specific operation or use-case is to do a web search. For instance, searching for "How to join numpy arrays" leads to this tutorial on array concatenation.

You can find a full list of array functions here.

Numpy Arithmetic Operations, Broadcasting, and Comparison

Numpy arrays support arithmetic operators like +, -, *, etc. You can perform an arithmetic operation with a single number (also called a scalar) or with another array of the same shape.

Operators make it easy to write mathematical expressions with multi-dimensional arrays.

arr2 = np.array([[1, 2, 3, 4], 
                 [5, 6, 7, 8], 
                 [9, 1, 2, 3]])

arr3 = np.array([[11, 12, 13, 14], 
                 [15, 16, 17, 18], 
                 [19, 11, 12, 13]])

# Adding a scalar
arr2 + 3

# array([[ 4,  5,  6,  7],
#        [ 8,  9, 10, 11],
#        [12,  4,  5,  6]])

# Element-wise subtraction
arr3 - arr2

# array([[10, 10, 10, 10],
#        [10, 10, 10, 10],
#        [10, 10, 10, 10]])

# Division by scalar
arr2 / 2

# array([[0.5, 1. , 1.5, 2. ],
#        [2.5, 3. , 3.5, 4. ],
#        [4.5, 0.5, 1. , 1.5]])

# Element-wise multiplication
arr2 * arr3

# array([[ 11,  24,  39,  56],
#        [ 75,  96, 119, 144],
#        [171,  11,  24,  39]])

# Modulus with scalar
arr2 % 4

# array([[1, 2, 3, 0],
#        [1, 2, 3, 0],
#        [1, 1, 2, 3]])

Numpy Array Broadcasting

Numpy arrays also support broadcasting, allowing arithmetic operations between two arrays with different numbers of dimensions but compatible shapes. Let's look at an example to see how it works.

arr2 = np.array([[1, 2, 3, 4], 
                 [5, 6, 7, 8], 
                 [9, 1, 2, 3]])               
arr2.shape
# (3, 4)

arr4 = np.array([4, 5, 6, 7])
arr4.shape
# (4,)

arr2 + arr4
# array([[ 5,  7,  9, 11],
#        [ 9, 11, 13, 15],
#        [13,  6,  8, 10]])

When the expression arr2 + arr4 is evaluated, arr4 (which has the shape (4,)) is replicated three times to match the shape (3, 4) of arr2. Numpy performs the replication without actually creating three copies of the smaller dimension array, thus improving performance and using lower memory.

Source: Python Data Science Handbook

Broadcasting only works if one of the arrays can be replicated to match the other array's shape.

arr5 = np.array([7, 8])
arr5.shape
# (2,)

arr2 + arr5
# ValueError: operands could not be broadcast together with shapes (3,4) (2,)

In the above example, even if arr5 is replicated three times, it will not match the shape of arr2. So arr2 + arr5 cannot be evaluated successfully. Learn more about broadcasting here.

Numpy Array Comparison

Numpy arrays also support comparison operations like ==, !=, > and so on. The result is an array of booleans.

arr1 = np.array([[1, 2, 3], [3, 4, 5]])
arr2 = np.array([[2, 2, 3], [1, 2, 5]])

arr1 == arr2
# array([[False,  True,  True],
#        [False, False,  True]])

arr1 != arr2
# array([[ True, False, False],
#        [ True,  True, False]])

arr1 >= arr2
# array([[False,  True,  True],
#        [ True,  True,  True]])

arr1 < arr2
# array([[ True, False, False],
#        [False, False, False]])

Array comparison is frequently used to count the number of equal elements in two arrays using the sum method. Remember that True evaluates to 1 and False evaluates to 0 when you use booleans in arithmetic operations.

(arr1 == arr2).sum()
# 3

Numpy Array Indexing and Slicing

Numpy extends Python's list indexing notation using [] to multiple dimensions in an intuitive fashion. You can provide a comma-separated list of indices or ranges to select a specific element or a subarray (also called a slice) from a Numpy array.

arr3 = np.array([
    [[11, 12, 13, 14], 
     [13, 14, 15, 19]], 

    [[15, 16, 17, 21], 
     [63, 92, 36, 18]], 

    [[98, 32, 81, 23],      
     [17, 18, 19.5, 43]]])

arr3.shape
# (3, 2, 4)

# Single element
arr3[1, 1, 2]

# 36.0

# Subarray using ranges
arr3[1:, 0:1, :2]

# array([[[15., 16.]],
# 
#        [[98., 32.]]])

# Mixing indices and ranges
arr3[1:, 1, 3]

# array([18., 43.])

arr3[1:, 1, :3]
# array([[63. , 92. , 36. ],
#        [17. , 18. , 19.5]])

# Using fewer indices
arr3[1]

# array([[15., 16., 17., 21.],
#        [63., 92., 36., 18.]])

arr3[:2, 1]
# array([[13., 14., 15., 19.],
#        [63., 92., 36., 18.]])

# Using too many indices
arr3[1,3,2,1]

# IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

The notation and its results can seem confusing at first, so take your time to experiment and become comfortable with it.

Use the cells below to try out some examples of array indexing and slicing, with different combinations of indices and ranges. Here are some more examples demonstrated visually:

_Source: Scipy Lectures_

How to Create Numpy Arrays – Other Methods

Numpy also provides some handy functions to create arrays of desired shapes with fixed or random values. Check out the official documentation or use the help function to learn more.

# All zeros
np.zeros((3, 2))

# array([[0., 0.],
#        [0., 0.],
#        [0., 0.]])

# All ones
np.ones([2, 2, 3])

# array([[[1., 1., 1.],
#         [1., 1., 1.]],
#
#        [[1., 1., 1.],
#         [1., 1., 1.]]])

# Identity matrix
np.eye(3)

# array([[1., 0., 0.],
#        [0., 1., 0.],
#        [0., 0., 1.]])

# Random vector
np.random.rand(5)

# array([0.92929562, 0.11301864, 0.64213555, 0.8600434 , 0.53738656])

# Random matrix
np.random.randn(2, 3) # rand vs. randn - what's the difference?

# array([[ 0.09906435, -1.64668094,  0.08073528],
#        [ 0.1437016 ,  0.80715712,  1.27285476]])

# Fixed value
np.full([2, 3], 42)

# array([[42, 42, 42],
#        [42, 42, 42]])

# Range with start, end and step
np.arange(10, 90, 3)

# array([10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58,
#        61, 64, 67, 70, 73, 76, 79, 82, 85, 88])

# Equally spaced numbers in a range
np.linspace(3, 27, 9)

# array([ 3.,  6.,  9., 12., 15., 18., 21., 24., 27.])

Exercises

Try the following exercises to become familiar with Numpy arrays and practice your skills:

• Assignment on Numpy array functions: https://jovian.ml/aakashns/numpy-array-operations
• (Optional) 100 numpy exercises: https://jovian.ml/aakashns/100-numpy-exercises

Summary and Further Reading

With this, we complete our discussion of numerical computing with Numpy. We've covered the following topics in this part of the tutorial:

• How to go from Python lists to Numpy arrays
• How to operate on Numpy arrays
• The benefits of using Numpy arrays over lists
• Multi-dimensional Numpy arrays
• How to work with CSV data files
• Arithmetic operations and broadcasting
• Array indexing and slicing
• Other ways of creating Numpy arrays

Check out the following resources for learning more about Numpy:

• Official tutorial
• Numpy course on freeCodeCamp
• Advanced Numpy (exploring the internals)

Review Questions to Check Your Comprehension

Try answering the following questions to test your understanding of the topics covered in this notebook:

1. What is a vector?
2. How do you represent vectors using a Python list? Give an example.
3. What is a dot product of two vectors?
4. Write a function to compute the dot product of two vectors.
5. What is Numpy?
6. How do you install Numpy?
7. How do you import the numpy module?
8. What does it mean to import a module with an alias? Give an example.
9. What is the commonly used alias for numpy?
10. What is a Numpy array?
11. How do you create a Numpy array? Give an example.
12. What is the type of Numpy arrays?
13. How do you access the elements of a Numpy array?
14. How do you compute the dot product of two vectors using Numpy?
15. What happens if you try to compute the dot product of two vectors which have different sizes?
16. How do you compute the element-wise product of two Numpy arrays?
17. How do you compute the sum of all the elements in a Numpy array?
18. What are the benefits of using Numpy arrays over Python lists for operating on numerical data?
19. Why do Numpy array operations have better performance compared to Python functions and loops?
20. Illustrate the performance difference between Numpy array operations and Python loops using an example.
21. What are multi-dimensional Numpy arrays?
22. Illustrate how you'd create Numpy arrays with 2, 3, and 4 dimensions.
23. How do you inspect the number of dimensions and the length along each dimension in a Numpy array?
24. Can the elements of a Numpy array have different data types?
25. How do you check the data types of the elements of a Numpy array?
26. What is the data type of a Numpy array?
27. What is the difference between a matrix and a 2D Numpy array?
28. How do you perform matrix multiplication using Numpy?
29. What is the @ operator used for in Numpy?
30. What is the CSV file format?
31. How do you read data from a CSV file using Numpy?
32. How do you concatenate two Numpy arrays?
33. What is the purpose of the axis argument of np.concatenate?
34. When are two Numpy arrays compatible for concatenation?
35. Give an example of two Numpy arrays that can be concatenated.
36. Give an example of two Numpy arrays that cannot be concatenated.
37. What is the purpose of the np.reshape function?
38. What does it mean to “reshape” a Numpy array?
39. How do you write a numpy array into a CSV file?
40. Give some examples of Numpy functions for performing mathematical operations.
41. Give some examples of Numpy functions for performing array manipulation.
42. Give some examples of Numpy functions for performing linear algebra.
43. Give some examples of Numpy functions for performing statistical operations.
44. How do you find the right Numpy function for a specific operation or use case?
45. Where can you see a list of all the Numpy array functions and operations?
46. What are the arithmetic operators supported by Numpy arrays? Illustrate with examples.
47. What is array broadcasting? How is it useful? Illustrate with an example.
48. Give some examples of arrays that are compatible for broadcasting.
49. Give some examples of arrays that are not compatible for broadcasting.
50. What are the comparison operators supported by Numpy arrays? Illustrate with examples.
51. How do you access a specific subarray or slice from a Numpy array?
52. Illustrate array indexing and slicing in multi-dimensional Numpy arrays with some examples.
53. How do you create a Numpy array with a given shape containing all zeros?
54. How do you create a Numpy array with a given shape containing all ones?
55. How do you create an identity matrix of a given shape?
56. How do you create a random vector of a given length?
57. How do you create a Numpy array with a given shape with a fixed value for each element?
58. How do you create a Numpy array with a given shape containing randomly initialized elements?
59. What is the difference between np.random.rand and np.random.randn? Illustrate with examples.
60. What is the difference between np.arange and np.linspace? Illustrate with examples.

You are ready to move on to the next section of this tutorial.

How to Analyze Tabular Data using Python and Pandas

Follow along and run the code here: https://jovian.ai/aakashns/python-pandas-data-analysis.

This section covers the following topics:

• How to read a CSV file into a Pandas data frame
• How to retrieve data from Pandas data frames
• How to query, sort, and analyze data
• How to merge, group, and aggregate data
• How to extract useful information from dates
• Basic plotting using line and bar charts
• How to write data frames to CSV files

How to Read a CSV File Using Pandas

Pandas is a popular Python library used for working in tabular data (similar to the data stored in a spreadsheet). It provides helper functions to read data from various file formats like CSV, Excel spreadsheets, HTML tables, JSON, SQL, and more.

Let's download a file italy-covid-daywise.txt which contains day-wise Covid-19 data for Italy in the following format:

date,new_cases,new_deaths,new_tests
2020-04-21,2256.0,454.0,28095.0
2020-04-22,2729.0,534.0,44248.0
2020-04-23,3370.0,437.0,37083.0
2020-04-24,2646.0,464.0,95273.0
2020-04-25,3021.0,420.0,38676.0
2020-04-26,2357.0,415.0,24113.0
2020-04-27,2324.0,260.0,26678.0
2020-04-28,1739.0,333.0,37554.0
...

This format of storing data is known as comma-separated values or CSV. Here's a reminder in case you need a definition of what the CSV format is:

CSVs: A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. (Wikipedia)

We'll download this file using the urlretrieve function from the urllib.request module.

from urllib.request import urlretrieve

urlretrieve('https://hub.jovian.ml/wp-content/uploads/2020/09/italy-covid-daywise.csv', 'italy-covid-daywise.csv')

To read the file, we can use the read_csv method from Pandas. First, let's install the Pandas library.

!pip install pandas --upgrade --quiet

We can now import the pandas module. As a convention, it is imported with the alias pd.

import pandas as pd

covid_df = pd.read_csv('italy-covid-daywise.csv')

Data from the file is read and stored in a DataFrame object – one of the core data structures in Pandas for storing and working with tabular data. We typically use the _df suffix in the variable names for dataframes.

type(covid_df)
# pandas.core.frame.DataFrame

covid_df

Here's what we can tell by looking at the dataframe:

• The file provides four day-wise counts for COVID-19 in Italy
• The metrics reported are new cases, deaths, and tests
• Data is provided for 248 days: from Dec 12, 2019, to Sep 3, 2020

Keep in mind that these are officially reported numbers. The actual number of cases and deaths may be higher, as not all cases are diagnosed.

We can view some basic information about the data frame using the .info method.

covid_df.info()

It appears that each column contains values of a specific data type. You can view statistical information for numerical columns (mean, standard deviation, minimum/maximum values, and the number of non-empty values) using the .describe method.

covid_df.describe()

The columns property contains the list of columns within the data frame.

covid_df.columns
# Index(['date', 'new_cases', 'new_deaths', 'new_tests'], dtype='object')

You can also retrieve the number of rows and columns in the data frame using the .shape method.

covid_df.shape
# (248, 4)

Here's a summary of the functions and methods we've looked at so far:

• pd.read_csv – Read data from a CSV file into a Pandas DataFrame object
• .info() – View basic information about rows, columns, and data types
• .describe() – View statistical information about numeric columns
• .columns – Get the list of column names
• .shape – Get the number of rows and columns as a tuple

How to Retrieve Data from a Data Frame in Pandas

The first thing you might want to do is retrieve data from this data frame, like the counts of a specific day or the list of values in a particular column.

To do this, you should understand the internal representation of data in a data frame. Conceptually, you can think of a dataframe as a dictionary of lists: keys are column names, and values are lists/arrays containing data for the respective columns.

# Pandas format is simliar to this
covid_data_dict = {
    'date':       ['2020-08-30', '2020-08-31', '2020-09-01', '2020-09-02', '2020-09-03'],
    'new_cases':  [1444, 1365, 996, 975, 1326],
    'new_deaths': [1, 4, 6, 8, 6],
    'new_tests': [53541, 42583, 54395, None, None]
}

Representing data in the above format has a few benefits:

• All values in a column typically have the same type of value, so it's more efficient to store them in a single array.
• Retrieving the values for a particular row simply requires extracting the elements at a given index from each column array.
• The representation is more compact (column names are recorded only once) compared to other formats that use a dictionary for each row of data (see the example below).

# Pandas format is not similar to this
covid_data_list = [
    {'date': '2020-08-30', 'new_cases': 1444, 'new_deaths': 1, 'new_tests': 53541},
    {'date': '2020-08-31', 'new_cases': 1365, 'new_deaths': 4, 'new_tests': 42583},
    {'date': '2020-09-01', 'new_cases': 996, 'new_deaths': 6, 'new_tests': 54395},
    {'date': '2020-09-02', 'new_cases': 975, 'new_deaths': 8 },
    {'date': '2020-09-03', 'new_cases': 1326, 'new_deaths': 6},
]

With the dictionary of lists analogy in mind, you can now guess how to retrieve data from a data frame. For example, we can get a list of values from a specific column using the [] indexing notation.

covid_data_dict['new_cases']
# [1444, 1365, 996, 975, 1326]

covid_df['new_cases']
# 0         0.0
# 1         0.0
# 2         0.0
# 3         0.0
# 4         0.0
#         ...  
# 243    1444.0
# 244    1365.0
# 245     996.0
# 246     975.0
# 247    1326.0
# Name: new_cases, Length: 248, dtype: float64

Each column is represented using a data structure called Series, which is essentially a numpy array with some extra methods and properties.

type(covid_df['new_cases'])
# pandas.core.series.Series

Like arrays, you can retrieve a specific value with a series using the indexing notation [].

covid_df['new_cases'][246]
# 975.0

covid_df['new_tests'][240]
57640.0

Pandas also provides the .at method to retrieve the element at a specific row & column directly.

covid_df.at[246, 'new_cases']
# 975.0

covid_df.at[240, 'new_tests']
# 57640.0

Instead of using the indexing notation [], Pandas also allows accessing columns as properties of the dataframe using the . notation. However, this method only works for columns whose names do not contain spaces or special characters.

covid_df.new_cases
# 0         0.0
# 1         0.0
# 2         0.0
# 3         0.0
# 4         0.0
#         ...  
# 243    1444.0
# 244    1365.0
# 245     996.0
# 246     975.0
# 247    1326.0
# Name: new_cases, Length: 248, dtype: float64

Further, you can also pass a list of columns within the indexing notation [] to access a subset of the data frame with just the given columns.

cases_df = covid_df[['date', 'new_cases']]
cases_df

The new data frame cases_df is simply a "view" of the original data frame covid_df. Both point to the same data in the computer's memory. Changing any values inside one of them will also change the respective values in the other.

Sharing data between data frames makes data manipulation in Pandas blazing fast. You needn't worry about the overhead of copying thousands or millions of rows every time you want to create a new data frame by operating on an existing one.

Sometimes you might need a full copy of the data frame, in which case you can use the copy method.

covid_df_copy = covid_df.copy()

The data within covid_df_copy is completely separate from covid_df, and changing values inside one of them will not affect the other.

To access a specific row of data, Pandas provides the .loc method.

covid_df

covid_df.loc[243]
# date          2020-08-30
# new_cases         1444.0
# new_deaths           1.0
# new_tests        53541.0
# Name: 243, dtype: object

Each retrieved row is also a Series object.

type(covid_df.loc[243])
# pandas.core.series.Series

We can use the .head and .tail methods to view the first or last few rows of data.

covid_df.head(5)

covid_df.tail(4)

Notice above that while the first few values in the new_cases and new_deaths columns are 0, the corresponding values within the new_tests column are NaN. That is because the CSV file does not contain any data for the new_tests column for specific dates (you can verify this by looking into the file). These values may be missing or unknown.

covid_df.at[0, 'new_tests']
# nan

type(covid_df.at[0, 'new_tests'])
# numpy.float64

The distinction between 0 and NaN is subtle but important. In this dataset, it represents that daily test numbers were not reported on specific dates. Italy started reporting daily tests on Apr 19, 2020. They'd already conducted 935,310 tests before Apr 19.

We can find the first index that doesn't contain a NaN value using a column's first_valid_index method.

covid_df.new_tests.first_valid_index()
# 111

Let's look at a few rows before and after this index to verify that the values change from NaN to actual numbers. We can do this by passing a range to loc.

covid_df.loc[108:113]

We can use the .sample method to retrieve a random sample of rows from the data frame.

covid_df.sample(10)

Notice that even though we have taken a random sample, each row's original index is preserved. This is a useful property of data frames.

Here's a summary of the functions and methods we looked at in this section:

• covid_df['new_cases'] – Retrieving columns as a Series using the column name
• new_cases[243] – Retrieving values from a Series using an index
• covid_df.at[243, 'new_cases'] – Retrieving a single value from a data frame
• covid_df.copy() – Creating a deep copy of a data frame
• covid_df.loc[243] - Retrieving a row or range of rows of data from the data frame
• head, tail, and sample – Retrieving multiple rows of data from the data frame
• covid_df.new_tests.first_valid_index – Finding the first non-empty index in a series

How to Analyze Data from Data Frames in Pandas

Let's try to answer some questions about our data.

Q: What are the total number of reported cases and deaths related to Covid-19 in Italy?

Similar to Numpy arrays, a Pandas series supports the sum method to answer these questions.

total_cases = covid_df.new_cases.sum()
total_deaths = covid_df.new_deaths.sum()

print('The number of reported cases is {} and the number of reported deaths is {}.'.format(int(total_cases), int(total_deaths)))
# The number of reported cases is 271515 and the number of reported deaths is 35497.

Q: What is the overall death rate (ratio of reported deaths to reported cases)?

death_rate = covid_df.new_deaths.sum() / covid_df.new_cases.sum()

print("The overall reported death rate in Italy is {:.2f} %.".format(death_rate*100))
# The overall reported death rate in Italy is 13.07 %.

Q: What is the overall number of tests conducted? A total of 935,310 tests were conducted before daily test numbers were reported.

initial_tests = 935310
total_tests = initial_tests + covid_df.new_tests.sum()

total_tests
# 5214766.0

Q: What fraction of tests returned a positive result?

positive_rate = total_cases / total_tests

print('{:.2f}% of tests in Italy led to a positive diagnosis.'.format(positive_rate*100))
# 5.21% of tests in Italy led to a positive diagnosis.

Try asking and answering some more questions about the data.

How to Query and Sort Rows in Pandas

Let's say we only want to look at the days which had more than 1,000 reported cases. We can use a boolean expression to check which rows satisfy this criterion.

high_new_cases = covid_df.new_cases > 1000

high_new_cases
# 0      False
# 1      False
# 2      False
# 3      False
# 4      False
#        ...  
# 243     True
# 244     True
# 245    False
# 246    False
# 247     True
# Name: new_cases, Length: 248, dtype: bool

The boolean expression returns a series containing True and False boolean values. You can use this series to select a subset of rows from the original dataframe, corresponding to the True values in the series.

covid_df[high_new_cases]

The data frame contains 72 rows, but only the first and last five rows are displayed by default with Jupyter for brevity. We can change some display options to view all the rows.

high_cases_df = covid_df[covid_df.new_cases > 1000]

high_cases_df

The data frame contains 72 rows, but only the first & last five rows are displayed by default with Jupyter for brevity. We can change some display options to view all the rows.

from IPython.display import display
with pd.option_context('display.max_rows', 100):
    display(covid_df[covid_df.new_cases > 1000])

This is just part of the data frame. Check out the rest here.

We can also formulate more complex queries that involve multiple columns. As an example, let's try to determine the days when the ratio of cases reported to tests conducted is higher than the overall positive_rate.

positive_rate
# 0.05206657403227681

high_ratio_df = covid_df[covid_df.new_cases / covid_df.new_tests > positive_rate]

high_ratio_df

The result of performing an operation on two columns is a new series.

covid_df.new_cases / covid_df.new_tests
# 0           NaN
# 1           NaN
# 2           NaN
# 3           NaN
# 4           NaN
#          ...   
# 243    0.026970
# 244    0.032055
# 245    0.018311
# 246         NaN
# 247         NaN
# Length: 248, dtype: float64

We can use this series to add a new column to the data frame.

covid_df['positive_rate'] = covid_df.new_cases / covid_df.new_tests

covid_df

However, keep in mind that sometimes it takes a few days to get the results for a test, so we can't compare the number of new cases with the number of tests conducted on the same day. Any inference based on this positive_rate column is likely to be incorrect.

It's essential to watch out for such subtle relationships that are often not conveyed within the CSV file and require some external context. It's always a good idea to read through the documentation provided with the dataset or ask for more information.

For now, let's remove the positive_rate column using the drop method.

covid_df.drop(columns=['positive_rate'], inplace=True)

Can you figure the purpose of the inplace argument?

How to Sort Rows Using Column Values in Pandas

You can also sort the rows by a specific column using .sort_values. Let's sort to identify the days with the highest number of cases, then chain it with the head method to list just the first ten results.

covid_df.sort_values('new_cases', ascending=False).head(10)

It looks like the last two weeks of March had the highest number of daily cases. Let's compare this to the days where the highest number of deaths were recorded.

covid_df.sort_values('new_deaths', ascending=False).head(10)

It appears that daily deaths hit a peak just about a week after the peak in daily new cases.

Let's also look at the days with the smallest number of cases. We might expect to see the first few days of the year on this list.

covid_df.sort_values('new_cases').head(10)

It seems like the count of new cases on Jun 20, 2020, was -148, a negative number! Not something we might have expected, but that's the nature of real-world data. It could be a data entry error, or the government may have issued a correction to account for miscounting in the past.

Can you dig through news articles online and figure out why the number was negative?

Let's look at some days before and after Jun 20, 2020.

covid_df.loc[169:175]

For now, let's assume this was indeed a data entry error. We can use one of the following approaches for dealing with the missing or faulty value:

1. Replace it with 0.
2. Replace it with the average of the entire column
3. Replace it with the average of the values on the previous and next date
4. Discard the row entirely

Which approach you pick requires some context about the data and the problem. In this case, since we are dealing with data ordered by date, we can go ahead with the third approach.

You can use the .at method to modify a specific value within the dataframe.

covid_df.at[172, 'new_cases'] = (covid_df.at[171, 'new_cases'] + covid_df.at[173, 'new_cases'])/2

Here's a summary of the functions and methods we looked at in this section:

• covid_df.new_cases.sum() – Computing the sum of values in a column or series
• covid_df[covid_df.new_cases > 1000] – Querying a subset of rows satisfying the chosen criteria using boolean expressions
• df['pos_rate'] = df.new_cases/df.new_tests – Adding new columns by combining data from existing columns
• covid_df.drop('positive_rate') – Removing one or more columns from the data frame
• sort_values – Sorting the rows of a data frame using column values
• covid_df.at[172, 'new_cases'] = ... – Replacing a value within the data frame

How to Work with Dates in Pandas

While we've looked at overall numbers for the cases, tests, positive rate, and more, it would also be useful to study these numbers on a month-by-month basis.

The date column might come in handy here, as Pandas provides many utilities for working with dates.

covid_df.date
# 0      2019-12-31
# 1      2020-01-01
# 2      2020-01-02
# 3      2020-01-03
# 4      2020-01-04
#           ...    
# 243    2020-08-30
# 244    2020-08-31
# 245    2020-09-01
# 246    2020-09-02
# 247    2020-09-03
# Name: date, Length: 248, dtype: object

The data type of date is currently object, so Pandas does not know that this column is a date. We can convert it into a datetime column using the pd.to_datetime method.

covid_df['date'] = pd.to_datetime(covid_df.date)

covid_df['date']
# 0     2019-12-31
# 1     2020-01-01
# 2     2020-01-02
# 3     2020-01-03
# 4     2020-01-04
#          ...    
# 243   2020-08-30
# 244   2020-08-31
# 245   2020-09-01
# 246   2020-09-02
# 247   2020-09-03
# Name: date, Length: 248, dtype: datetime64[ns]

You can see that it now has the datatype datetime64. We can now extract different parts of the data into separate columns, using the DatetimeIndex class (view docs).

covid_df['year'] = pd.DatetimeIndex(covid_df.date).year
covid_df['month'] = pd.DatetimeIndex(covid_df.date).month
covid_df['day'] = pd.DatetimeIndex(covid_df.date).day
covid_df['weekday'] = pd.DatetimeIndex(covid_df.date).weekday

covid_df

Let's check the overall metrics for May. We can query the rows for May, choose a subset of columns, and use the sum method to aggregate each selected column's values.

# Query the rows for May
covid_df_may = covid_df[covid_df.month == 5]

# Extract the subset of columns to be aggregated
covid_df_may_metrics = covid_df_may[['new_cases', 'new_deaths', 'new_tests']]

# Get the column-wise sum
covid_may_totals = covid_df_may_metrics.sum()

covid_may_totals
# new_cases       29073.0
# new_deaths       5658.0
# new_tests     1078720.0
# dtype: float64

type(covid_may_totals)
# pandas.core.series.Series

We can also combine the above operations into a single statement.

covid_df[covid_df.month == 5][['new_cases', 'new_deaths', 'new_tests']].sum()
# new_cases       29073.0
# new_deaths       5658.0
# new_tests     1078720.0
# dtype: float64

As another example, let's check if the number of cases reported on Sundays is higher than the average number of cases reported every day. This time, we might want to aggregate columns using the .mean method.

# Overall average
covid_df.new_cases.mean()

# 1096.6149193548388

# Average for Sundays
covid_df[covid_df.weekday == 6].new_cases.mean()

# 1247.2571428571428

It seems like more cases were reported on Sundays compared to other days.

Try asking and answering some more date-related questions about the data.

How to Group and Aggregate Data in Pandas

As a next step, we might want to summarize the day-wise data and create a new dataframe with month-wise data. We can use the groupby function to create a group for each month, select the columns we wish to aggregate, and aggregate them using the sum method.

covid_month_df = covid_df.groupby('month')[['new_cases', 'new_deaths', 'new_tests']].sum()

covid_month_df

The result is a new data frame that uses unique values from the column passed to groupby as the index. Grouping and aggregation is a powerful method for progressively summarizing data into smaller data frames.

Instead of aggregating by sum, you can also aggregate by other measures like mean. Let's compute the average number of daily new cases, deaths, and tests for each month.

covid_month_mean_df = covid_df.groupby('month')[['new_cases', 'new_deaths', 'new_tests']].mean()

covid_month_mean_df

Apart from grouping, another form of aggregation is the running or cumulative sum of cases, tests, or deaths up to each row's date. We can use the cumsum method to compute the cumulative sum of a column as a new series.

Let's add three new columns: total_cases, total_deaths, and total_tests.

covid_df['total_cases'] = covid_df.new_cases.cumsum()
covid_df['total_deaths'] = covid_df.new_deaths.cumsum()
covid_df['total_tests'] = covid_df.new_tests.cumsum() + initial_tests

We've also included the initial test count in total_test to account for tests conducted before daily reporting was started.

covid_df

Notice how the NaN values in the total_tests column remain unaffected.

How to Merge Data from Multiple Sources in Pandas

To determine other metrics like test per million, cases per million, and so on, we require some more information about the country, namely its population.

Let's download another file locations.csv that contains health-related information for many countries, including Italy.

urlretrieve('https://gist.githubusercontent.com/aakashns/8684589ef4f266116cdce023377fc9c8/raw/99ce3826b2a9d1e6d0bde7e9e559fc8b6e9ac88b/locations.csv', 'locations.csv')

locations_df = pd.read_csv('locations.csv')
locations_df

locations_df[locations_df.location == "Italy"]

We can merge this data into our existing data frame by adding more columns. However, to merge two data frames, we need at least one common column. Let's insert a location column in the covid_df dataframe with all values set to "Italy".

covid_df['location'] = "Italy"

covid_df

We can now add the columns from locations_df into covid_df using the .merge method.

merged_df = covid_df.merge(locations_df, on="location")

merged_df

Check out the full data frame here.

The location data for Italy is appended to each row within covid_df. If the covid_df data frame contained data for multiple locations, then the respective country's location data would be appended for each row.

We can now calculate metrics like cases per million, deaths per million, and tests per million.

merged_df['cases_per_million'] = merged_df.total_cases * 1e6 / merged_df.population
merged_df['deaths_per_million'] = merged_df.total_deaths * 1e6 / merged_df.population
merged_df['tests_per_million'] = merged_df.total_tests * 1e6 / merged_df.population

merged_df

Check out the full data frame here.

How to Write Data Back to Files in Pandas

After completing your analysis and adding new columns, you should write the results back to a file. Otherwise, the data will be lost when the Jupyter notebook shuts down.

Before writing to file, let's first create a data frame containing just the columns we wish to record.

result_df = merged_df[['date',
                       'new_cases', 
                       'total_cases', 
                       'new_deaths', 
                       'total_deaths', 
                       'new_tests', 
                       'total_tests', 
                       'cases_per_million', 
                       'deaths_per_million', 
                       'tests_per_million']]

result_df

To write the data from the data frame into a file, we can use the to_csv function.

result_df.to_csv('results.csv', index=None)

The to_csv function also includes an additional column for storing the index of the dataframe by default. We pass index=None to turn off this behavior. You can now verify that the results.csv is created and contains data from the data frame in CSV format:

date,new_cases,total_cases,new_deaths,total_deaths,new_tests,total_tests,cases_per_million,deaths_per_million,tests_per_million
2020-02-27,78.0,400.0,1.0,12.0,,,6.61574439992122,0.1984723319976366,
2020-02-28,250.0,650.0,5.0,17.0,,,10.750584649871982,0.28116913699665186,
2020-02-29,238.0,888.0,4.0,21.0,,,14.686952567825108,0.34732658099586405,
2020-03-01,240.0,1128.0,8.0,29.0,,,18.656399207777838,0.47964146899428844,
2020-03-02,561.0,1689.0,6.0,35.0,,,27.93498072866735,0.5788776349931067,
2020-03-03,347.0,2036.0,17.0,52.0,,,33.67413899559901,0.8600467719897585,
...

Bonus: Basic Plotting with Pandas

We generally use a library like matplotlib or seaborn to plot graphs within a Jupyter notebook. However, Pandas dataframes and series provide a handy .plot method for quick and easy plotting.

Let's plot a line graph showing how the number of daily cases varies over time.

result_df.new_cases.plot();

While this plot shows the overall trend, it's hard to tell where the peak occurred, as there are no dates on the X-axis. We can use the date column as the index for the data frame to address this issue.

result_df.set_index('date', inplace=True)

result_df

Notice that the index of a data frame doesn't have to be numeric. Using the date as the index also allows us to get the data for a specific data using .loc.

result_df.loc['2020-09-01']
# new_cases             9.960000e+02
# total_cases           2.696595e+05
# new_deaths            6.000000e+00
# total_deaths          3.548300e+04
# new_tests             5.439500e+04
# total_tests           5.214766e+06
# cases_per_million     4.459996e+03
# deaths_per_million    5.868661e+02
# tests_per_million     8.624890e+04
# Name: 2020-09-01 00:00:00, dtype: float64

Let's plot the new cases and new deaths per day as line graphs.

result_df.new_cases.plot()
result_df.new_deaths.plot();

We can also compare the total cases vs. total deaths.

result_df.total_cases.plot()
result_df.total_deaths.plot();

Let's see how the death rate and positive testing rates vary over time.

death_rate = result_df.total_deaths / result_df.total_cases

death_rate.plot(title='Death Rate');

positive_rates = result_df.total_cases / result_df.total_tests

positive_rates.plot(title='Positive Rate');

Finally, let's plot some month-wise data using a bar chart to visualize the trend at a higher level.

covid_month_df.new_cases.plot(kind='bar');

covid_month_df.new_tests.plot(kind='bar')

Pandas Exercises

Try the following exercises to become familiar with Pandas dataframes and practice your skills:

• Assignment on Pandas dataframes
• Additional exercises on Pandas
• Try downloading and analyzing some data from Kaggle

Summary and Further Reading

We've covered the following topics in this tutorial:

• How to read a CSV file into a Pandas data frame
• How to retrieve data from Pandas data frames
• How to query, sort, and analyze data
• How to merge, group, and aggregate data
• How to extract useful information from dates
• Basic plotting using line and bar charts
• How to write data frames to CSV files

Check out the following resources to learn more about Pandas:

• User guide for Pandas
• Python for Data Analysis (book by Wes McKinney - creator of Pandas)

Review Questions to Check Your Comprehension

Try answering the following questions to test your understanding of the topics covered in this notebook:

1. What is Pandas? What makes it useful?
2. How do you install the Pandas library?
3. How do you import the pandas module?
4. What is the common alias used while importing the pandas module?
5. How do you read a CSV file using Pandas? Give an example.
6. What are some other file formats you can read using Pandas? Illustrate with examples.
7. What are Pandas dataframes?
8. How are Pandas dataframes different from Numpy arrays?
9. How do you find the number of rows and columns in a dataframe?
10. How do you get the list of columns in a dataframe?
11. What is the purpose of the describe method of a dataframe?
12. How are the info and describe dataframe methods different?
13. Is a Pandas dataframe conceptually similar to a list of dictionaries or a dictionary of lists? Explain with an example.
14. What is a Pandas Series? How is it different from a Numpy array?
15. How do you access a column from a dataframe?
16. How do you access a row from a dataframe?
17. How do you access an element at a specific row and column of a dataframe?
18. How do you create a subset of a dataframe with a specific set of columns?
19. How do you create a subset of a dataframe with a specific range of rows?
20. Does changing a value within a dataframe affect other dataframes created using a subset of the rows or columns? Why is it so?
21. How do you create a copy of a dataframe?
22. Why should you avoid creating too many copies of a dataframe?
23. How do you view the first few rows of a dataframe?
24. How do you view the last few rows of a dataframe?
25. How do you view a random selection of rows of a dataframe?
26. What is the "index" in a dataframe? How is it useful?
27. What does a NaN value in a Pandas dataframe represent?
28. How is Nan different from 0?
29. How do you identify the first non-empty row in a Pandas series or column?
30. What is the difference between df.loc and df.at?
31. Where can you find a full list of methods supported by Pandas DataFrame and Series objects?
32. How do you find the sum of numbers in a column of a dataframe?
33. How do you find the mean of numbers in a column of a dataframe?
34. How do you find the number of non-empty numbers in a column of a dataframe?
35. What is the result obtained by using a Pandas column in a boolean expression? Illustrate with an example.
36. How do you select a subset of rows where a specific column's value meets a given condition? Illustrate with an example.
37. What is the result of the expression df[df.new_cases > 100] ?
38. How do you display all the rows of a pandas dataframe in a Jupyter cell output?
39. What is the result obtained when you perform an arithmetic operation between two columns of a dataframe? Illustrate with an example.
40. How do you add a new column to a dataframe by combining values from two existing columns? Illustrate with an example.
41. How do you remove a column from a dataframe? Illustrate with an example.
42. What is the purpose of the inplace argument in dataframe methods?
43. How do you sort the rows of a dataframe based on the values in a particular column?
44. How do you sort a pandas dataframe using values from multiple columns?
45. How do you specify whether to sort by ascending or descending order while sorting a Pandas dataframe?
46. How do you change a specific value within a dataframe?
47. How do you convert a dataframe column to the datetime data type?
48. What are the benefits of using the datetime data type instead of object?
49. How do you extract different parts of a date column like the month, year, month, weekday, and so on into separate columns? Illustrate with an example.
50. How do you aggregate multiple columns of a dataframe together?
51. What is the purpose of the groupby method of a dataframe? Illustrate with an example.
52. What are the different ways in which you can aggregate the groups created by groupby?
53. What do you mean by a running or cumulative sum?
54. How do you create a new column containing the running or cumulative sum of another column?
55. What are other cumulative measures supported by Pandas dataframes?
56. What does it mean to merge two dataframes? Give an example.
57. How do you specify the columns that should be used for merging two dataframes?
58. How do you write data from a Pandas dataframe into a CSV file? Give an example.
59. What are some other file formats you can write to from a Pandas dataframe? Illustrate with examples.
60. How do you create a line plot showing the values within a column of a dataframe?
61. How do you convert a column of a dataframe into its index?
62. Can the index of a dataframe be non-numeric?
63. What are the benefits of using a non-numeric dataframe? Illustrate with an example.
64. How you create a bar plot showing the values within a column of a dataframe?
65. What are some other types of plots supported by Pandas dataframes and series?

You are ready to move on to the next section of the tutorial.

Data Visualization using Python, Matplotlib, and Seaborn

Notebook link: https://jovian.ai/aakashns/python-matplotlib-data-visualization

Data visualization is the graphic representation of data. It involves producing images that communicate relationships among the represented data to viewers.

Visualizing data is an essential part of data analysis and machine learning. We'll use Python libraries Matplotlib and Seaborn to learn and apply some popular data visualization techniques. We'll use the words chart, plot, and graph interchangeably in this tutorial.

To begin, let's install and import the libraries. We'll use the matplotlib.pyplot module for basic plots like line and bar charts. It is often imported with the alias plt. We'll use the seaborn module for more advanced plots. It is commonly imported with the alias sns.

!pip install matplotlib seaborn --upgrade --quiet

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

Notice this we also include the special command %matplotlib inline to ensure that our plots are shown and embedded within the Jupyter notebook itself. Without this command, sometimes plots may show up in pop-up windows.

How to Create a Line Chart in Python

The line chart is one of the simplest and most widely used data visualization techniques. A line chart displays information as a series of data points or markers connected by straight lines.

You can customize the shape, size, color, and other aesthetic elements of the lines and markers for better visual clarity.

Here's a Python list showing the yield of apples (tons per hectare) over six years in an imaginary country called Kanto.

yield_apples = [0.895, 0.91, 0.919, 0.926, 0.929, 0.931]

We can visualize how the yield of apples changes over time using a line chart. To draw a line chart, we can use the plt.plot function.

plt.plot(yield_apples)

Calling the plt.plot function draws the line chart as expected. It also returns a list of plots drawn [<matplotlib.lines.Line2D at 0x7ff70aa20760>], shown within the output. We can include a semicolon (;) at the end of the last statement in the cell to avoiding showing the output and display just the graph.

plt.plot(yield_apples);

Let's enhance this plot step-by-step to make it more informative and beautiful.

How to Customize the X-axis in MatPlotLib

The X-axis of the plot currently shows list element indices 0 to 5. The plot would be more informative if we could display the year for which we're plotting the data. We can do this by two arguments plt.plot.

years = [2010, 2011, 2012, 2013, 2014, 2015]
yield_apples = [0.895, 0.91, 0.919, 0.926, 0.929, 0.931]

plt.plot(years, yield_apples)

Axis Labels in MatPlotLib

We can add labels to the axes to show what each axis represents using the plt.xlabel and plt.ylabel methods.

plt.plot(years, yield_apples)
plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)');

How to Plot Multiple Lines in MatPlotLib

You can invoke the plt.plot function once for each line to plot multiple lines in the same graph. Let's compare the yields of apples vs. oranges in Kanto.

years = range(2000, 2012)
apples = [0.895, 0.91, 0.919, 0.926, 0.929, 0.931, 0.934, 0.936, 0.937, 0.9375, 0.9372, 0.939]
oranges = [0.962, 0.941, 0.930, 0.923, 0.918, 0.908, 0.907, 0.904, 0.901, 0.898, 0.9, 0.896, ]

plt.plot(years, apples)
plt.plot(years, oranges)
plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)');

Chart Title and Legend in MatPlotLib

To differentiate between multiple lines, we can include a legend within the graph using the plt.legend function. We can also set a title for the chart using the plt.title function.

plt.plot(years, apples)
plt.plot(years, oranges)

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

How to Use Line Markers in MatPlotLib

We can also show markers for the data points on each line using the marker argument of plt.plot.

Matplotlib provides many different markers like a circle, cross, square, diamond, and more. You can find the full list of marker types here: https://matplotlib.org/3.1.1/api/markers_api.html .

plt.plot(years, apples, marker='o')
plt.plot(years, oranges, marker='x')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

How to Style Lines and Markers in MatPlotLib

The plt.plot function supports many arguments for styling lines and markers:

• color or c – Set the color of the line (supported colors)
• linestyle or ls – Choose between a solid or dashed line
• linewidth or lw – Set the width of a line
• markersize or ms – Set the size of markers
• markeredgecolor or mec – Set the edge color for markers
• markeredgewidth or mew – Set the edge width for markers
• markerfacecolor or mfc – Set the fill color for markers
• alpha – Opacity of the plot

Check out the documentation for plt.plot to learn more: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html#matplotlib.pyplot.plot .

plt.plot(years, apples, marker='s', c='b', ls='-', lw=2, ms=8, mew=2, mec='navy')
plt.plot(years, oranges, marker='o', c='r', ls='--', lw=3, ms=10, alpha=.5)

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

The fmt argument provides a shorthand for specifying the marker shape, line style, and line color. You can provide it as the third argument to plt.plot.

fmt = '[marker][line][color]'

plt.plot(years, apples, 's-b')
plt.plot(years, oranges, 'o--r')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

You can use the plt.figure function to change the size of the figure.

plt.plot(years, oranges, 'or')
plt.title("Yield of Oranges (tons per hectare)");

How to Change the Figure Size in MatPlotLib

You can use the plt.figure function to change the size of the figure.

plt.figure(figsize=(12, 6))

plt.plot(years, oranges, 'or')
plt.title("Yield of Oranges (tons per hectare)");

How to Improve Default Styles using Seaborn

An easy way to make your charts look beautiful is to use some default styles from the Seaborn library. You can apply them globally using the sns.set_style function. You can see a full list of predefined styles here: https://seaborn.pydata.org/generated/seaborn.set_style.html .

sns.set_style("whitegrid")
plt.plot(years, apples, 's-b')
plt.plot(years, oranges, 'o--r')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

sns.set_style("darkgrid")

plt.plot(years, apples, 's-b')
plt.plot(years, oranges, 'o--r')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

plt.plot(years, oranges, 'or')
plt.title("Yield of Oranges (tons per hectare)");

You can also edit default styles directly by modifying the matplotlib.rcParams dictionary. Learn more: https://matplotlib.org/3.2.1/tutorials/introductory/customizing.html#matplotlib-rcparams .

import matplotlib

matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'

Scatter Plots in MatPlotLib

In a scatter plot, the values of 2 variables are plotted as points on a 2-dimensional grid. Additionally, you can also use a third variable to determine the size or color of the points. Let's try out an example.

The Iris flower dataset provides sample measurements of sepals and petals for three species of flowers. The Iris dataset is included with the Seaborn library and you can load it as a Pandas data frame.

# Load data into a Pandas dataframe
flowers_df = sns.load_dataset("iris")

flowers_df

flowers_df.species.unique()
# array(['setosa', 'versicolor', 'virginica'], dtype=object)

Let's try to visualize the relationship between sepal length and sepal width. Our first instinct might be to create a line chart using plt.plot.

plt.plot(flowers_df.sepal_length, flowers_df.sepal_width);

The output is not very informative as there are too many combinations of the two properties within the dataset. There doesn't seem to be simple relationship between them.

We can use a scatter plot to visualize how sepal length and sepal width vary using the scatterplot function from the seaborn module (imported as sns).

sns.scatterplot(x=flowers_df.sepal_length, y=flowers_df.sepal_width);

How to Add Hues in MatPlotLib

Notice how the points in the above plot seem to form distinct clusters with some outliers. We can color the dots using the flower species as a hue. We can also make the points larger using the s argument.

sns.scatterplot(x=flowers_df.sepal_length, y=flowers_df.sepal_width, hue=flowers_df.species, s=100);

Adding hues makes the plot more informative. We can immediately tell that Setosa irises have a smaller sepal length but higher sepal widths. In contrast, the opposite is true for Virginica irises.

How to Customize Seaborn Figures**

Since Seaborn uses Matplotlib's plotting functions internally, we can use functions like plt.figure and plt.title to modify the figure.

plt.figure(figsize=(12, 6))
plt.title('Sepal Dimensions')

sns.scatterplot(x=flowers_df.sepal_length, 
                y=flowers_df.sepal_width, 
                hue=flowers_df.species,
                s=100);

How to Plot Data using Pandas Data Frames with Seaborn

Seaborn has built-in support for Pandas data frames. Instead of passing each column as a series, you can provide column names and use the data argument to specify a data frame.

plt.title('Sepal Dimensions')
sns.scatterplot(x='sepal_length', 
                y='sepal_width', 
                hue='species',
                s=100,
                data=flowers_df);

Histograms in MatPlotLib

A histogram represents the distribution of a variable by creating bins (intervals) along the range of values and showing vertical bars to indicate the number of observations in each bin.

For example, let's visualize the distribution of values of sepal width in the Iris dataset. We can use the plt.hist function to create a histogram.

# Load data into a Pandas dataframe
flowers_df = sns.load_dataset("iris")

flowers_df.sepal_width
# 0      3.5
# 1      3.0
# 2      3.2
# 3      3.1
# 4      3.6
#       ... 
# 145    3.0
# 146    2.5
# 147    3.0
# 148    3.4
# 149    3.0
# Name: sepal_width, Length: 150, dtype: float64

plt.title("Distribution of Sepal Width")
plt.hist(flowers_df.sepal_width);

We can immediately see that the sepal widths lie in the range 2.0 - 4.5, and around 35 values are in the range 2.9 - 3.1, which seems to be the most populous bin.

How to Control the Size and Number of Bins**

We can control the number of bins or the size of each one using the bins argument.

# Specifying the number of bins
plt.hist(flowers_df.sepal_width, bins=5);

import numpy as np

# Specifying the boundaries of each bin
plt.hist(flowers_df.sepal_width, bins=np.arange(2, 5, 0.25));

# Bins of unequal sizes
plt.hist(flowers_df.sepal_width, bins=[1, 3, 4, 4.5]);

How to Manage Multiple Histograms in MatPlotLib

Similar to line charts, we can draw multiple histograms in a single chart. We can reduce each histogram's opacity so that one histogram's bars don't hide the others'.

Let's draw separate histograms for each species of flowers.

setosa_df = flowers_df[flowers_df.species == 'setosa']
versicolor_df = flowers_df[flowers_df.species == 'versicolor']
virginica_df = flowers_df[flowers_df.species == 'virginica']

plt.hist(setosa_df.sepal_width, alpha=0.4, bins=np.arange(2, 5, 0.25));
plt.hist(versicolor_df.sepal_width, alpha=0.4, bins=np.arange(2, 5, 0.25));

We can also stack multiple histograms on top of one another.

plt.title('Distribution of Sepal Width')

plt.hist([setosa_df.sepal_width, versicolor_df.sepal_width, virginica_df.sepal_width], 
         bins=np.arange(2, 5, 0.25), 
         stacked=True);

plt.legend(['Setosa', 'Versicolor', 'Virginica']);

Bar Charts in MatPlotLib

Bar charts are quite similar to line charts, that is they show a sequence of values. However, a bar is shown for each value, rather than points connected by lines. We can use the plt.bar function to draw a bar chart.

years = range(2000, 2006)
apples = [0.35, 0.6, 0.9, 0.8, 0.65, 0.8]
oranges = [0.4, 0.8, 0.9, 0.7, 0.6, 0.8]

plt.bar(years, oranges);

Like histograms, we can stack bars on top of one another. We use the bottom argument of plt.bar to achieve this.

plt.bar(years, apples)
plt.bar(years, oranges, bottom=apples);

Bar Plots with Averages in Seaborn

Let's look at another sample dataset included with Seaborn called tips. The dataset contains information about the sex, time of day, total bill, and tip amount for customers visiting a restaurant over a week.

tips_df = sns.load_dataset("tips");

tips_df

We might want to draw a bar chart to visualize how the average bill amount varies across different days of the week. One way to do this would be to compute the day-wise averages and then use plt.bar (try it as an exercise).

However, since this is a very common use case, the Seaborn library provides a barplot function which can automatically compute averages.

sns.barplot(x='day', y='total_bill', data=tips_df);

The lines cutting each bar represent the amount of variation in the values. For instance, it seems like the variation in the total bill is relatively high on Fridays and low on Saturdays.

We can also specify a hue argument to compare bar plots side-by-side based on a third feature, for example sex.

sns.barplot(x='day', y='total_bill', hue='sex', data=tips_df);

You can make the bars horizontal simply by switching the axes.

sns.barplot(x='total_bill', y='day', hue='sex', data=tips_df);

Heatmaps in Seaborn

A heatmap is used to visualize 2-dimensional data like a matrix or a table using colors. The best way to understand it is by looking at an example.

We'll use another sample dataset from Seaborn, called flights, to visualize monthly passenger footfall at an airport over 12 years.

flights_df = sns.load_dataset("flights").pivot("month", "year", "passengers")

flights_df

flights_df is a matrix with one row for each month and one column for each year. The values show the number of passengers (in thousands) that visited the airport in a specific month of a year. We can use the sns.heatmap function to visualize the footfall at the airport.

plt.title("No. of Passengers (1000s)")
sns.heatmap(flights_df);

The brighter colors indicate a higher footfall at the airport. By looking at the graph, we can infer two things:

• The footfall at the airport in any given year tends to be the highest around July and August.
• The footfall at the airport in any given month tends to grow year by year.

We can also display the actual values in each block by specifying annot=True and using the cmap argument to change the color palette.

plt.title("No. of Passengers (1000s)")
sns.heatmap(flights_df, fmt="d", annot=True, cmap='Blues');

Images in MatPlotLib

We can also use Matplotlib to display images. Let's download an image from the internet.

from urllib.request import urlretrieve

urlretrieve('https://i.imgur.com/SkPbq.jpg', 'chart.jpg');

Before displaying an image, it has to be read into memory using the PIL module.

from PIL import Image

img = Image.open('chart.jpg')

An image loaded using PIL is simply a 3-dimensional numpy array containing pixel intensities for the red, green & blue (RGB) channels of the image. We can convert the image into an array using np.array.

img_array = np.array(img)

img_array.shape
# (481, 640, 3)

We can display the PIL image using plt.imshow.

plt.imshow(img);

We can turn off the axes & grid lines and show a title using the relevant functions.

plt.grid(False)
plt.title('A data science meme')
plt.axis('off')
plt.imshow(img);

To display a part of the image, we can simply select a slice from the numpy array.

plt.grid(False)
plt.axis('off')
plt.imshow(img_array[125:325,105:305]);

How to Plot Multiple Charts in a Grid in MatPlotLib and Seaborn

Matplotlib and Seaborn also support plotting multiple charts in a grid, using plt.subplots, which returns a set of axes for plotting.

Here's a single grid showing the different types of charts we've covered in this tutorial.

fig, axes = plt.subplots(2, 3, figsize=(16, 8))

# Use the axes for plotting
axes[0,0].plot(years, apples, 's-b')
axes[0,0].plot(years, oranges, 'o--r')
axes[0,0].set_xlabel('Year')
axes[0,0].set_ylabel('Yield (tons per hectare)')
axes[0,0].legend(['Apples', 'Oranges']);
axes[0,0].set_title('Crop Yields in Kanto')


# Pass the axes into seaborn
axes[0,1].set_title('Sepal Length vs. Sepal Width')
sns.scatterplot(x=flowers_df.sepal_length, 
                y=flowers_df.sepal_width, 
                hue=flowers_df.species, 
                s=100, 
                ax=axes[0,1]);

# Use the axes for plotting
axes[0,2].set_title('Distribution of Sepal Width')
axes[0,2].hist([setosa_df.sepal_width, versicolor_df.sepal_width, virginica_df.sepal_width], 
         bins=np.arange(2, 5, 0.25), 
         stacked=True);

axes[0,2].legend(['Setosa', 'Versicolor', 'Virginica']);

# Pass the axes into seaborn
axes[1,0].set_title('Restaurant bills')
sns.barplot(x='day', y='total_bill', hue='sex', data=tips_df, ax=axes[1,0]);

# Pass the axes into seaborn
axes[1,1].set_title('Flight traffic')
sns.heatmap(flights_df, cmap='Blues', ax=axes[1,1]);

# Plot an image using the axes
axes[1,2].set_title('Data Science Meme')
axes[1,2].imshow(img)
axes[1,2].grid(False)
axes[1,2].set_xticks([])
axes[1,2].set_yticks([])

plt.tight_layout(pad=2);

See this page for a full list of supported functions: https://matplotlib.org/3.3.1/api/axes_api.html#the-axes-class .

Pair Plots with Seaborn

Seaborn also provides a helper function sns.pairplot to automatically plot several different charts for pairs of features within a dataframe.

sns.pairplot(flowers_df, hue='species');

See the full output here.

sns.pairplot(tips_df, hue='sex');

Summary and Further Reading

We have covered the following topics in this tutorial:

• How to create and customize line charts using Matplotlib
• How to visualize relationships between two or more variables using scatter plots
• How to study distributions of variables using histograms and bar charts
• How to visualize two-dimensional data using heatmaps
• How to display images using Matplotlib's plt.imshow
• How to plot multiple Matplotlib and Seaborn charts in a grid

In this tutorial we've covered some of the fundamental concepts and popular techniques for data visualization using Matplotlib and Seaborn. Data visualization is a vast field and we've barely scratched the surface here. Check out these references to learn and discover more:

• Data Visualization cheat sheet: https://jovian.ml/aakashns/dataviz-cheatsheet
• Seaborn gallery: https://seaborn.pydata.org/examples/index.html
• Matplotlib gallery: https://matplotlib.org/3.1.1/gallery/index.html
• Matplotlib tutorial: https://github.com/rougier/matplotlib-tutorial

Review Questions to Check Your Comprehension

Try answering the following questions to test your understanding of the topics covered in this notebook:

1. What is data visualization?
2. What is Matplotlib?
3. What is Seaborn?
4. How do you install Matplotlib and Seaborn?
5. How you import Matplotlib and Seaborn? What are the common aliases used while importing these modules?
6. What is the purpose of the magic command %matplotlib inline?
7. What is a line chart?
8. How do you plot a line chart in Python? Illustrate with an example.
9. How do you specify values for the X-axis of a line chart?
10. How do you specify labels for the axes of a chart?
11. How do you plot multiple line charts on the same axes?
12. How do you show a legend for a line chart with multiple lines?
13. How you set a title for a chart?
14. How do you show markers on a line chart?
15. What are the different options for styling lines and markers in line charts? Illustrate with examples.
16. What is the purpose of the fmt argument to plt.plot?
17. Where can you see a list of all the arguments accepted by plt.plot?
18. How do you change the size of the figure using Matplotlib?
19. How do you apply the default styles from Seaborn globally for all charts?
20. What are the predefined styles available in Seaborn? Illustrate with examples.
21. What is a scatter plot?
22. How is a scatter plot different from a line chart?
23. How do you draw a scatter plot using Seaborn? Illustrate with an example.
24. How do you decide when to use a scatter plot vs a line chart?
25. How do you specify the colors for dots on a scatter plot using a categorical variable?
26. How do you customize the title, figure size, legend, and son on for Seaborn plots?
27. How do you use a Pandas dataframe with sns.scatterplot?
28. What is a histogram?
29. When should you use a histogram vs a line chart?
30. How do you draw a histogram using Matplotlib? Illustrate with an example.
31. What are "bins" in a histogram?
32. How do you change the sizes of bins in a histogram?
33. How do you change the number of bins in a histogram?
34. How do you show multiple histograms on the same axes?
35. How do you stack multiple histograms on top of one another?
36. What is a bar chart?
37. How do you draw a bar chart using Matplotlib? Illustrate with an example.
38. What is the difference between a bar chart and a histogram?
39. What is the difference between a bar chart and a line chart?
40. How do you stack bars on top of one another?
41. What is the difference between plt.bar and sns.barplot?
42. What do the lines cutting the bars in a Seaborn bar plot represent?
43. How do you show bar plots side-by-side?
44. How do you draw a horizontal bar plot?
45. What is a heat map?
46. What type of data is best visualized with a heat map?
47. What does the pivot method of a Pandas dataframe do?
48. How do you draw a heat map using Seaborn? Illustrate with an example.
49. How do you change the color scheme of a heat map?
50. How do you show the original values from the dataset on a heat map?
51. How do you download images from a URL in Python?
52. How do you open an image for processing in Python?
53. What is the purpose of the PIL module in Python?
54. How do you convert an image loaded using PIL into a Numpy array?
55. How many dimensions does a Numpy array for an image have? What does each dimension represent?
56. What are "color channels" in an image?
57. What is RGB?
58. How do you display an image using Matplotlib?
59. How do you turn off the axes and gridlines in a chart?
60. How do you display a portion of an image using Matplotlib?
61. How do you plot multiple charts in a grid using Matplotlib and Seaborn? Illustrate with examples.
62. What is the purpose of the plt.subplots function?
63. What are pair plots in Seaborn? Illustrate with an example.
64. How do you export a plot into a PNG image file using Matplotlib?
65. Where can you learn about the different types of charts you can create using Matplotlib and Seaborn?

Congratulations on making it to the end of this tutorial! You can now apply these skills to analyze real world datasets from sources like Kaggle.

If you're pursuing a career in data science and machine learning, consider joining the Zero to Data Science Bootcamp by Jovian. It's a 20-week part-time program where you'll complete 7 courses, 12 coding assignments and 4-real world projects. You will also receive 6 months of career support to help you find your first data science job.

When you have a list of data records in a dataframe, you may need to drop a specific list of rows depending on the needs of your model and your goals when studying your analytics.

In this tutorial, you'll learn how to drop a list of rows from a Pandas dataframe.

To learn how to drop columns, you can read here about How to Drop Columns in Pandas.

How to Drop a Row or Column in a Pandas Dataframe

To drop a row or column in a dataframe, you need to use the drop() method available in the dataframe. You can read more about the drop() method in the docs here.

Dataframe Axis

• Rows are denoted using axis=0
• Columns are denoted using axis=1

Dataframe Labels

• Rows are labelled using the index number starting with 0, by default.
• Columns are labelled using names.

Drop() Method Parameters

• index - the list of rows to be deleted
• axis=0 - Marks the rows in the dataframe to be deleted
• inplace=True - Performs the drop operation in the same dataframe, rather than creating a new dataframe object during the delete operation.

Sample Pandas DataFrame

Our sample dataframe contains the columns product_name, Unit_Price, No_Of_Units, Available_Quantity, and Available_Since_Date columns. It also has rows with NaN values which are used to denote missing values.

import pandas as pd

data = {"product_name":["Keyboard","Mouse", "Monitor", "CPU","CPU", "Speakers",pd.NaT],
        "Unit_Price":[500,200, 5000.235, 10000.550, 10000.550, 250.50,None],
        "No_Of_Units":[5,5, 10, 20, 20, 8,pd.NaT],
        "Available_Quantity":[5,6,10,"Not Available","Not Available", pd.NaT,pd.NaT],
        "Available_Since_Date":['11/5/2021', '4/23/2021', '08/21/2021','09/18/2021','09/18/2021','01/05/2021',pd.NaT]
       }

df = pd.DataFrame(data)

df

The dataframe will look like this:

And just like that we've created our sample dataframe.

After each drop operation, you'll print the dataframe by using df which will print the dataframe in a regular HTML table format.

You can read here about how to Pretty Print a Dataframe to print the dataframe in different visual formats.

Next, you'll learn how to drop a list of rows in different use cases.

How to Drop a List of Rows by Index in Pandas

You can delete a list of rows from Pandas by passing the list of indices to the drop() method.

df.drop([5,6], axis=0, inplace=True)

df

In this code,

• [5,6] is the index of the rows you want to delete
• axis=0 denotes that rows should be deleted from the dataframe
• inplace=True performs the drop operation in the same dataframe

After dropping rows with the index 5 and 6, you'll have the below data in the dataframe:

This is how you can delete rows with a specific index.

Next, you'll learn about dropping a range of indices.

How to Drop Rows by Index Range in Pandas

You can also drop a list of rows within a specific range.

A range is a set of values with a lower limit and an upper limit.

This may be useful in cases where you want to create a sample dataset exlcuding specific ranges of data.

You can create a range of rows in a dataframe by using the df.index() method. Then you can pass this range to the drop() method to drop the rows as shown below.

df.drop(df.index[2:4], inplace=True)

df

Here's what this code is doing:

• df.index[2:4] generates a range of rows from 2 to 4. The lower limit of the range is inclusive and the upper limit of the range is exclusive. This means that rows 2 and 3 will be deleted and row 4 will not be deleted.
• inplace=True performs the drop operation in the same dataframe

After dropping rows within the range 2-4, you'll have the below data in the dataframe:

This is how you can drop the list of rows in the dataframe using its range.

How to Drop All Rows after an Index in Pandas

You can drop all rows after a specific index by using iloc[].

You can use iloc[] to select rows by using its position index. You can specify the start and end position separated by a :. For example, you'd use 2:3 to select rows from 2 to 3. If you want to select all the rows, you can just use : in iloc[].

This may be useful in cases where you want to split the dataset for training and testing purposes.

Use the below snippet to select rows from 0 to the index 2. This results in dropping the rows after the index 2.

df = df.iloc[:2]

df

In this code, :2 selects the rows until the index 2.

This is how you can drop all rows after a specific index.

After dropping rows after the index 2, you'll have the below data in the dataframe:

This is how you can drop rows after a specific index.

Next, you'll learn how to drop rows with conditions.

How to Drop Rows with Multiple Conditions in Pandas

You can drop rows in the dataframe based on specific conditions.

For example, you can drop rows where the column value is greater than X and less than Y.

This may be useful in cases where you want to create a dataset that ignores columns with specific values.

To drop rows based on certain conditions, select the index of the rows which pass the specific condition and pass that index to the drop() method.

df.drop(df[(df['Unit_Price'] >400) & (df['Unit_Price'] < 600)].index, inplace=True)

df

In this code,

• (df['Unit_Price'] >400) & (df['Unit_Price'] < 600) is the condition to drop the rows.
• df[].index selects the index of rows which passes the condition.
• inplace=True performs the drop operation in the same dataframe rather than creating a new one.

After dropping the rows with the condition which has the unit_price greater than 400 and less than 600, you'll have the below data in the dataframe:

This is how you can drop rows in the dataframe using certain conditions.

Conclusion

To summarize, in this article you've learnt what the drop() method is in a Pandas dataframe. You've also seen how dataframe rows and columns are labelled. And finally you've learnt how to drop rows using indices, a range of indices, and based on conditions.

If you liked this article, feel free to share it.

You May Also Like

• How to Add a Column to a Dataframe in Pandas
  • How to Rename a Column in Pandas