Ashutosh Krishna - freeCodeCamp.org

Database Version Control with Liquibase and Spring Boot

Ashutosh Krishna — Tue, 09 Jun 2026 02:29:12 +0000

Picture this familiar scenario: you're working on a new feature that requires a new database column. You open your local database client, write an ALTER TABLE statement, and execute it. Your code works perfectly. You commit the Java code, push it to the repository, and go grab a coffee.

A few hours later, a teammate pulls your branch, runs the application, and everything crashes.

"Hey," they ask across the room (or in a Slack channel), "did you change the database?"

You quickly realize you forgot to share the SQL script. You paste it into the chat. They run it. Everything works. Then, a week later, the deployment to the staging environment fails for the exact same reason. By the time this code reaches production, everyone is asking a variation of the same terrified question: "Which SQL script should I run?"

This situation is called schema drift. It happens when the state of your database diverges across different environments. Staging has one schema, production has another, and every developer's local machine is a unique snowflake of untested database modifications.

Managing database changes manually is a recipe for deployment headaches and team collaboration challenges. Application code is stateless and easy to replace. Databases are stateful. Databases have surprisingly good memories, and they rarely forget a bad migration.

Liquibase solves this problem by bringing version-control discipline to your database changes. Instead of passing around SQL files and hoping people remember to run them, you define your database changes in code. These changes travel with your application repository and execute automatically.

Here is a high-level look at how this architecture works:

Think about the journey of a single database change. A developer commits their database migration alongside their Java code into Git. When the CI/CD pipeline (or a teammate) pulls that code, the Spring Boot application starts. But before the app fully boots up and accepts web traffic, Liquibase intercepts the process. It acts as a gatekeeper, connecting to the database and applying the required schema changes. This ensures the database exactly matches the code's expectations before a single user makes a request.

Why Database Version Control Matters

If you've spent any time working on team-based applications, you've probably seen a folder structure that looks exactly like this:

project-sql-scripts/
├── create_employee_table.sql
├── create_employee_table_final.sql
├── create_employee_table_final_v2.sql
├── add_email_column.sql
├── latest.sql
└── definitely_latest_use_this_one.sql

The phrase "just run this SQL script manually" has launched many memorable incidents.

When you rely on manual database updates, you guarantee failure at scale. Onboarding a new developer becomes an archeological expedition to figure out how to build the local schema. Deployments become stressful events requiring a checklist of manual queries that must be run in a highly specific order.

Version-controlled database changes treat your schema as code. When your database changes live alongside your application logic, you gain several immediate benefits:

Consistency: Every environment (local, staging, production) applies the exact same changes in the exact same order.
Safety: You eliminate the human error of skipping a script or running an outdated query.
Visibility: You can look at a Git commit and see exactly how the Java code and the database schema changed together to support a new feature.

Git solved version control for code. Liquibase helps prevent databases from becoming the rebellious sibling.

What is Liquibase?

At its core, Liquibase is a database migration tool that tracks and applies schema changes in a predictable and repeatable way.

Instead of writing loose SQL scripts, you write "migrations" (also called changeSets). Liquibase reads these files, compares them against a tracking table inside your actual database, and figures out exactly what needs to be executed to bring the database up to date.

To use Liquibase effectively, you only need to understand a few conceptual terms:

changeLog: The master file. This is essentially a list that tells Liquibase which migration files to execute and in what order.
changeSet: A single, atomic change to your database. Creating a table is one changeSet. Adding a column is another.
Migration History: A table Liquibase automatically creates in your database (called DATABASECHANGELOG) to remember which changeSets have already been executed.
Checksums: A unique hash generated for every changeSet. Liquibase uses this to detect if someone secretly modified a file after it was already executed.

When you integrate Liquibase with Spring Boot, the migration process happens completely automatically during the application startup phase.

During startup, Liquibase takes control before your web server is allowed to receive HTTP traffic. It reaches into the database and checks the tracking table to see which migrations have already run. If it finds new migrations in your local files, it locks the database to prevent concurrent updates, executes the changes, records the new history, and finally releases the lock. Only after this entire process completes does Spring Boot finish booting up.

Because Liquibase runs before Spring Boot fully initializes the web server, your application will never serve traffic with an outdated database schema. If a migration fails, the application fails to start, protecting your system from entering a broken state.

Project Setup

Now that you understand the theory, let's build something real. We're going to build the database layer for an Employee Management API.

For this project we'll use:

Java 17+
Spring Boot 3.x
Maven
Liquibase
H2 Database

We're using H2 because it's an in-memory database that requires zero installation. You can run this project immediately without configuring Docker containers or installing database servers. But everything you learn here applies exactly the same way to PostgreSQL, MySQL, SQL Server, or Oracle.

If you're generating this project via Spring Initializr, select the following dependencies: Spring Web, Spring Data JPA, Liquibase Migration, and H2 Database.

In your pom.xml, you'll see the critical dependencies that make this work:


    
        org.springframework.boot
        spring-boot-starter-web
    
    
        org.springframework.boot
        spring-boot-starter-data-jpa
    

    
        com.h2database
        h2
        runtime
    

    
        org.liquibase
        liquibase-core

Next, configure Spring Boot to talk to H2 and find your Liquibase files. Open your src/main/resources/application.properties file and add the following:

# H2 Database Configuration
spring.datasource.url=jdbc:h2:file:./data/employeedb;DB_CLOSE_DELAY=-1
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=

# Enable H2 Console to inspect the database in your browser
spring.h2.console.enabled=true
spring.h2.console.path=/h2-console

# Liquibase Configuration
spring.liquibase.change-log=classpath:db/changelog/db.changelog-master.xml

That last line is the most important. It tells Spring Boot exactly where to find the "master list" of your database changes.

Note: We're using a file-based H2 database instead of an in-memory database. The problem with an in-memory database is that it completely wipes itself clean every time you restart Spring Boot.

While Liquibase will happily rebuild the schema from scratch on every boot, a file-based database is much better for this tutorial (and for real-world local development). With a file-based database, your data, and more importantly, your Liquibase history, will actually persist between application restarts.

Understanding Core Liquibase Concepts

Before we write our first table, we need to understand how Liquibase organizes files. Liquibase uses a hierarchical structure.

Think of it like a book. The changeLog is the table of contents, and the changeSets are the actual chapters.

The Master ChangeLog: This is the entry point. It rarely contains actual database changes. Instead, its only job is to include other files in a specific order.
Child ChangeLogs: These group related changes together.
ChangeSets: These are the actual, atomic database commands (like creating a table or adding a column).

Here's a visual breakdown of how this hierarchy works in a real Spring Boot project:

Liquibase organizes migrations hierarchically. You maintain a single master file that acts as a table of contents. This master file rarely holds actual SQL commands. Instead, it explicitly includes child XML files in a strict execution order. Each of those child files (like 01-create-employees.xml) contains one or more individual database commands, which Liquibase calls changeSets.

A changeSet is uniquely identified by three things:

id: A unique string (often a number or a Jira ticket ID).
author: The person who wrote the migration.
file path: Where the file is located.

When Liquibase runs, it looks at a changeSet, calculates a cryptographic hash of its contents (a checksum), and records the id, author, and checksum in the database. If it sees that exact combination of id, author, and file path in the database again on the next startup, it skips it.

Create the Initial Employee Schema (Version 1)

Let's write our first version. We need a table to store employees.

First, create the master file at src/main/resources/db/changelog/db.changelog-master.xml:

Next, create the actual migration file at src/main/resources/db/changelog/changes/01-create-employees.xml:

Let's look at what we just did. We defined a changeSet with an id of "1" and an author of "ashutoshkrris". Inside, we used Liquibase's XML syntax to define a table.

Why use XML instead of plain SQL? Because Liquibase is database-agnostic. This exact XML will generate the correct auto-increment syntax for PostgreSQL (SERIAL), MySQL (AUTO_INCREMENT), or Oracle (IDENTITY). You define the structure, and Liquibase translates it to the specific database dialect.

Now, run your Spring Boot application. Watch your terminal output. You'll see logs similar to this:

Liquibase realized the database was empty. It automatically created its tracking table (DATABASECHANGELOG), read our changeSet, executed the table creation, and recorded the event.

If you restart the application right now, Liquibase will run again. But this time, it'll check the DATABASECHANGELOG table, see that id="1" and author="ashutoshkrris" has already been executed, and silently skip it. Your database is now safely version-controlled.

What Just Happened?

Up to this point, Liquibase might feel a bit like magic. You dropped an XML file into a folder, started Spring Boot, and your database schema transformed.

But understanding how Liquibase actually works under the hood is critical. If you understand the startup sequence, you'll know exactly how to debug deployments when things eventually go wrong.

When your Spring Boot application starts, it doesn't immediately begin accepting web requests. First, it initializes its internal components. When it creates the Liquibase component, the migration process begins.

Here's exactly what happens during that startup phase:

Let's trace the exact sequence. When Spring Boot initializes Liquibase, the very first thing the tool does is query the lock table to ensure no other application instance is currently migrating the database. If the coast is clear, it claims the lock. It then calculates cryptographic checksums for your local XML files, compares them against the database history, executes any missing changes, and logs them. Finally, it releases the lock so the Tomcat web server can safely start.

This sequence guarantees that your application will never serve a user request before the database schema is completely ready to handle it.

Inspecting the Database: Liquibase Metadata Tables

Let's look at what this history and locking actually looks like inside the database itself. Since we configured the H2 Console earlier, we can inspect the raw tables.

While your Spring Boot application is running, open your browser and navigate to http://localhost:8080/h2-console. Connect using the JDBC URL jdbc:h2:file:./data/employeedb with the username sa and a blank password.

Inside, you'll see your employees table. You'll also see two extra tables created automatically by Liquibase: DATABASECHANGELOG and DATABASECHANGELOGLOCK.

The `DATABASECHANGELOG` Table

This table is the brain of your migration strategy. It acts as the permanent ledger of every database change ever applied to this environment.

If you run SELECT * FROM DATABASECHANGELOG;, you'll see output that looks like this:

ID	AUTHOR	FILENAME	DATEEXECUTED	ORDEREXECUTED	EXECTYPE	MD5SUM	DESCRIPTION	COMMENTS	TAG	LIQUIBASE	CONTEXTS	LABELS	DEPLOYMENT_ID
1	ashutoshkrris	db/changelog/changes/01-create-employees.xml	2026-05-30 13:11:35.937919	1	EXECUTED	9:66e7dcffb2b1902a4e9f01670cb5f192	createTable tableName=employees		null	4.31.1	null	null	0126894849

Let's break down the most important columns:

ID, AUTHOR, FILENAME: These three columns form a composite key. Together, they uniquely identify a single migration.
DATEEXECUTED & ORDEREXECUTED: Tells you exactly when a script ran and in what sequence.
MD5SUM: This is the cryptographic hash of your XML file. When Liquibase starts, it hashes your local XML file and compares it to this column. If you secretly edit a file after it's been executed, this hash won't match, and Liquibase will crash the startup to protect your database.
EXECTYPE: Most of the time, this simply says EXECUTED. But it provides a crucial audit trail: if you use Liquibase commands to intentionally skip a migration but record it as finished, you'll see MARK_RAN. If a migration was skipped because its preconditions failed, you'll see SKIPPED.
TAG: Think of this as a Git tag for your database schema. Before a major, high-risk deployment, you can configure Liquibase to "tag" the current state of the database (for example, v1.4.0). If the deployment fails catastrophically, you can trigger a rollback command telling Liquibase to undo every change applied after the v1.4.0 tag.
CONTEXTS: This is how you manage environment-specific changes. By adding a context attribute to your changeSet (for example, ), that migration will only execute if Spring Boot passes "dev" or "qa" to Liquibase on startup. Production will safely ignore it.
LABELS: While Contexts target environments, Labels target categories of work. You can label a changeSet with a Jira ticket number (issue-842) or a release train (Q3-release). This allows advanced teams to selectively execute or roll back specific subsets of features without affecting the rest of the database.

The `DATABASECHANGELOGLOCK` Table

This table is tiny, but it plays a massive role in modern deployments.

If you run SELECT * FROM DATABASECHANGELOGLOCK;, you'll see a single row:

ID	LOCKED	LOCKGRANTED	LOCKEDBY
1	FALSE	null	null

Imagine you're deploying your Spring Boot application to a Kubernetes cluster. You tell Kubernetes to spin up three identical instances simultaneously. All three instances connect to the exact same database.

If all three instances try to run the CREATE TABLE migration at the exact same millisecond, your database will throw concurrency errors. The lock table prevents this. The very first instance to reach the database sets LOCKED to TRUE. The other two instances check the table, see the lock, and politely wait.

Practical Troubleshooting Tip: Sometimes, a deployment fails catastrophically mid-migration (perhaps the server lost power). When this happens, Liquibase might die before it can set LOCKED back to FALSE.

The next time you start the application, the logs will hang indefinitely, repeating: Waiting for changelog lock....

If you're absolutely certain no other applications are currently running migrations, you can manually fix this by running a simple SQL command in your database client:

UPDATE DATABASECHANGELOGLOCK SET LOCKED = FALSE;

This forces the lock open, allowing your application to resume.

Evolving the Employee API

Software is never finished. Two weeks after your successful Version 1 deployment, the business team comes back with new requirements.

Because you now understand how Liquibase tracks history, evolving the database is simple. You just append new files to your master list.

Version 2: Adding an Email Field

The HR team needs to contact employees. You need an email column.

Create a new file at src/main/resources/db/changelog/changes/02-add-employee-email.xml:

Add this to your db.changelog-master.xml file immediately below your first include:

When you restart the application, Liquibase checks the DATABASECHANGELOG table. It sees that id="1" is already there, so it skips it. It sees id="2" is missing, so it executes it and adds a new row to the tracking table.

Version 3: Adding Departments Support

The company is growing. Employees now belong to departments. You need a departments table and a foreign key constraint linking the two.

Create 03-add-departments.xml:

Notice that we used two separate changeSets in one file. This is a best practice. Each changeSet represents one logical operation. If the foreign key creation (id="4") fails, the department table creation (id="3") will still be recorded as successful, and only id="4" will roll back.

Version 4 & 5: Employee Status and Performance Indexes

Finally, HR wants to track active versus inactive staff, and the database team noticed that searching by last name is getting slow.

Create 04-status-and-indexes.xml:

Remember to add all new files to your db.changelog-master.xml. The order of your include statements is the exact order Liquibase will execute them.

The Golden Rule: Never Modify Executed ChangeSets

Eventually, a developer on your team will look at your 01-create-employees.xml file and notice a mistake. Perhaps they spot a typo in a column name, or perhaps they realize a column is missing a strict non-null constraint.

Their instinct, based on years of writing standard Java code, will be to open that XML file, fix the mistake, save the file, and restart the application.

Let's actually do this and see what happens.

Open your src/main/resources/db/changelog/changes/01-create-employees.xml file. Change the first_name column to given_name:

Save the file and restart your Spring Boot application.

Instead of a smooth startup, your application will instantly crash, and your terminal will vomit a massive stack trace. Look closely at the top of the error logs. You should see this exact message:

Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
     1 changesets check sum
          db/changelog/changes/01-create-employees.xml::1::ashutoshkrris was: 9:66e7dcffb2b1902a4e9f01670cb5f192 but is now: 9:2bd3ef21343d3b5c9448cc50bc35deef

Here's why this happens. Once a changeSet runs against an environment, it becomes immutable history. You can't change the past.

When Liquibase starts up, it calculates a cryptographic hash (an MD5 checksum) of your local XML file. It then queries the DATABASECHANGELOG table and compares the freshly calculated hash against the hash that was recorded when the file originally executed.

If you change even a single character in a file that has already been executed, the hash changes. Liquibase detects the tampering and refuses to start. It does this to protect your data. If your XML code says a column is named first_name but the database was originally built using fist_name, your Spring Data JPA repositories are going to fail anyway.

How to Fix It (The Right Way)

If you made this mistake locally, you might be tempted to go into your database, delete the row from the DATABASECHANGELOG table, and try again. Don't do this. If this code reaches staging or production, you can't manually delete rows on production servers.

The correct way to fix a schema mistake is to roll forward.

First, undo your change in 01-create-employees.xml so the hash matches the database again. Then, write a brand new changeSet to apply the fix:

Include it in your master changelog, restart the application, and the database will safely evolve to the correct state.

Working with Seed Data

Sometimes, a schema change requires initial data to be useful.

For example, in Version 3, we created a departments table. Right now, that table is completely empty. When a new developer clones the repository and spins up the project locally, they have to manually write SQL INSERT statements just to test the API.

We can automate this by making baseline data insertion part of our migration strategy.

Create a new file at src/main/resources/db/changelog/changes/05-seed-departments.xml:

Add the include statement to your db.changelog-master.xml file. When you restart the application, Liquibase will insert these rows. Your API is now instantly usable out of the box.

The Danger of Data Migrations

While seeding data is powerful, it requires discipline. Here is a practical engineering rule of thumb:

Do use Liquibase for:

Static lookup tables (status codes, country lists, default departments).
System configuration flags required for the application to boot.

Do NOT use Liquibase for:

Generating thousands of fake users for testing.
Migrating massive amounts of transactional data (for example, moving 5 million records from one table to another).

Large data migrations can lock up database tables for hours. If you lock a core table during a deployment, your application will experience a massive outage. Keep your changeSets focused on schema structure and essential baseline data. Use dedicated scripts or background jobs for heavy data manipulation.

Rollbacks

In a perfect world, code always works. In reality, you'll eventually deploy a database change that breaks a critical production query or corrupts data. When this happens, you need a way to hit the undo button.

Liquibase supports rollbacks, but you have to understand how it interprets them.

Automatic vs. Explicit Rollbacks

Many Liquibase commands are automatically reversible. For example, if you write a changeSet to or , Liquibase implicitly knows that the opposite of adding a column is dropping a column. You don't have to tell it how to undo these actions.

But some operations are inherently destructive or ambiguous. If you use custom tags, or if you use , Liquibase has no idea how to put the data back. In these cases, you must provide explicit rollback instructions.

Let's simulate a scenario where we add a temporary access code column, but we want to ensure we know exactly how to remove it safely.

Create 06-temporary-access.xml:

Add this to your master file and run the application. The column is added.

If you were deploying this via a CI/CD pipeline and the deployment failed, you could trigger a Liquibase Maven command to roll back by a specific number of steps (for example, mvn liquibase:rollback -Dliquibase.rollbackCount=1), or roll back to a specific tag we discussed earlier.

The Reality Check on Rollbacks

While it's important to know how rollbacks work, here's a practical reality from the trenches of backend engineering: Rollbacks are often discussed but rarely executed cleanly in production.

Dropping a column is mathematically easy. Recovering the customer data that was written to that column during the 15 minutes the bad code was live is incredibly difficult.

Because of this, modern engineering teams often prefer a "roll forward" strategy. If a migration causes an issue, instead of running a scary database rollback command, they quickly write a new changeSet that fixes the issue (for example, adding a missing index or relaxing a constraint) and deploy the application again.

It's highly recommended to design your database changes to be additive and non-destructive to avoid needing complex rollbacks in the first place.

Common Beginner Mistakes

Adopting database version control is a massive step forward for any engineering team, but it comes with a learning curve. When developers transition from writing loose SQL scripts to using Liquibase, they tend to fall into a few predictable traps.

Here are the most common beginner mistakes and exactly how to avoid them.

1. The "Mega" ChangeSet

When starting out, it's tempting to dump your entire initial schema into a single XML file under a single changeSet. You might put 15 createTable statements and 20 addForeignKeyConstraint statements into id="1".

This is a terrible idea for one simple reason: transaction failure.

If your database engine fails on table number 14 (perhaps due to a syntax error), what happens to the first 13 tables? Some database engines support transactional DDL (Data Definition Language), meaning it will roll back all 13 tables automatically. But many databases do not.

If it fails halfway through, your database is now in a fractured state. Liquibase didn't record id="1" as successful, so the next time you start the app, it will try to create all 15 tables again. It will immediately crash because table 1 already exists.

The Fix: Stick to the rule of "one logical operation per changeSet." If you're creating three tables, write three separate changeSets. If one fails, the successful ones are permanently recorded, and you only have to fix the broken one.

2. Manual Database Tweaking (The Phantom Menace)

This is the most dangerous habit to break. A developer spots a missing index in production. Instead of writing a Liquibase migration, going through code review, and deploying, they log directly into the production database and run CREATE INDEX manually to save time.

A week later, another developer writes a proper Liquibase migration to create that exact same index and deploys it. The application crashes on startup. Liquibase tries to execute the CREATE INDEX command, but the database throws an error saying the index already exists.

When you adopt Liquibase, you must accept a fundamental rule: Liquibase is the absolute source of truth for your schema. Human hands should never touch the database structure directly.

The Fix: If someone accidentally does this, you have two options to fix the deployment pipeline. You can manually drop the index from the database so Liquibase can recreate it properly, or you can use the tag in Liquibase to check if the index exists before trying to create it.

3. Ignoring the "From Scratch" Build

When you work on a project for months, your local database accumulates a lot of history. You write migrations assuming certain tables or test data already exist.

Then, a new developer joins the team. They pull the code, spin up an empty database, start Spring Boot, and the migrations crash halfway through.

This happens because the migrations rely on an assumed state (like expecting a specific row to exist before creating a foreign key) rather than a guaranteed state.

The Fix: You should regularly test your migrations against a completely blank database. If you're using Docker, tear down your database container and rebuild it. If you're using a file-based H2 database like we set up earlier, simply delete the ./data/employeedb.mv.db file from your project folder and restart Spring Boot. If the application can't boot successfully from a completely empty state, your migration history is broken.

4. Hardcoding Environment Details

Beginners sometimes hardcode environment-specific details directly into their XML files. For example, they might hardcode a specific schema name (schemaName="dev_schema") or grant permissions to a specific local user (GRANT ALL ON employees TO my_local_user).

When this code goes to staging, the staging database uses a different schema name, and the deployment fails.

The Fix: Keep your migrations abstract. Let Spring Boot handle the connection details via application.properties. If you absolutely must use dynamic values inside your Liquibase files, use property substitution. You can define variables in Liquibase and pass them in from Spring Boot during startup.

5. Messing Up Migration Ordering

Liquibase executes files in the exact order they're listed in your db.changelog-master.xml file.

If developer A creates the departments table in a branch, and developer B creates a foreign key linking to departments in another branch, whoever merges their code first dictates the order. If developer B's code gets included in the master file before developer A's code, Liquibase will try to create the foreign key before the target table exists.

The Fix: The master changelog is the ultimate chokepoint for database changes. During code reviews, always verify that the statements are ordered chronologically and that dependencies make sense.

Liquibase vs Flyway vs Manual SQL Scripts

When you decide to implement database version control, you'll immediately face a choice. Liquibase isn't the only tool in the Java ecosystem. The three most common approaches to managing schema evolution are Liquibase, Flyway, and manual SQL scripts.

You should understand the practical tradeoffs of each so you can choose the right tool for your specific team and project.

1. Manual SQL Scripts (The Baseline)

This is the default approach for most beginners. You write a script.sql file and execute it directly against the database using a tool like DBeaver, pgAdmin, or DataGrip.

Strengths: There is zero setup required. You have total control over the exact syntax, and every backend developer already knows how to write SQL.
Weaknesses: There's absolutely no execution tracking. This approach practically guarantees schema drift across environments. Deployments become stressful because they rely on humans remembering to execute the right scripts in the exact right order.
The Verdict: Manual scripts are perfectly fine for solo weekend projects or rapid prototyping where you don't care if the database gets destroyed. But they become a massive liability the moment a second developer joins the team or a staging environment is created.

2. Flyway (The SQL Purist)

Flyway is the most popular alternative to Liquibase. Instead of using XML or YAML abstractions, Flyway embraces raw SQL. You write pure SQL files with a strict naming convention (for example, V1__Create_employee_table.sql).

Strengths: There's no new syntax to learn. If you know SQL, you already know how to use Flyway. It's incredibly fast to set up, highly opinionated, and integrates flawlessly with Spring Boot.
Weaknesses: Because you write raw SQL, your migrations are intimately tied to your specific database dialect. If you write Flyway scripts for MySQL and later decide to migrate the project to PostgreSQL, you have to manually rewrite your migration history. Furthermore, seamless automated rollbacks are a paid feature in Flyway's commercial tier.
The Verdict: Flyway is excellent for teams that are highly skilled in SQL, are permanently committed to a single database vendor, and prefer strict conventions over flexible configurations.

3. Liquibase (The Abstraction Layer)

As we have seen throughout this tutorial, Liquibase takes a different approach by abstracting database changes into XML, YAML, or JSON.

Strengths: It's truly database-agnostic. You define the logical structure, and Liquibase automatically translates that into the correct SQL dialect for H2, PostgreSQL, or Oracle. It supports powerful automatic rollbacks, preconditions, contexts, and deployment labels out of the box for free.
Weaknesses: It has a steeper learning curve than Flyway. The XML syntax is undeniably verbose and can feel heavy for very simple, single-table applications.
The Verdict: Liquibase shines in complex applications, multi-tenant systems, projects that support multiple database vendors, and enterprise environments that require fine-grained control over CI/CD deployment pipelines.

Liquibase Best Practices

Now that you understand the mechanics of Liquibase, you need to know how to use it in a professional environment. Writing a migration that works on your local machine is only half the battle. Writing a migration that your entire team can safely deploy to production requires discipline.

Here are the engineering best practices you should adopt when managing database changes.

1. One Logical Change Per ChangeSet (The Atomic Rule)

We discussed this in the common mistakes section, but it's important enough to repeat. Never bundle a table creation, an index creation, and a data insertion into a single changeSet.

If you're adding a salary column and an idx_employee_salary index, put them in two separate changeSets within the same file. This ensures that if the index creation fails, the column creation is still safely recorded, and you don't end up in a fractured database state.

2. Meaningful File Organization and Naming

Don't name your files update1.xml or new_changes.xml. Your file names should tell a story about how your database evolved.

Adopt a strict prefix system. In our project, we used 01-create-employees.xml and 02-add-employee-email.xml. In a real team, you might use Jira ticket numbers or release versions (for example, v1.2.0_ticket-482_add_email.xml). Whatever convention you choose, enforce it rigorously during code reviews.

3. Treat Database Changes Like Application Code

Database migrations belong in source control right next to your Java code. They should be reviewed with the exact same level of scrutiny.

When reviewing a pull request that includes a Liquibase file, engineers should ask:

Does this column need an index?
Is this a destructive change (like renaming a column) that will break the currently running application?
Did the author include explicit rollback instructions for custom SQL?

4. Integrate Migrations into CI/CD

Human hands should never run database migrations against a production server. Your deployment pipeline should handle this automatically.

When you merge code into your main branch, your CI/CD pipeline (like GitHub Actions or GitLab CI) should build your Spring Boot application and deploy it. Because we bundled Liquibase into our Spring Boot startup sequence, the application will automatically migrate the production database before it starts accepting web traffic.

Here's what a safe, automated deployment pipeline looks like:

In a mature deployment pipeline, human hands never touch the production database. When you merge a pull request, the CI/CD pipeline builds the code and runs unit tests. It deploys the Spring Boot application to a staging environment, where Liquibase automatically acquires a lock and runs the migrations during startup. Once validated, that exact same artifact is promoted to production, triggering the identical automated migration process.

5. Never Fix Forward by Deleting History

If a migration fails in an upper environment (like staging or production), never log into the database to delete the DATABASECHANGELOG row so you can try again.

You must respect the immutability of the changelog. If you made a mistake, write a new changeSet that drops the broken table or fixes the data type, and push it through your Git workflow just like you would a Java bug fix.

Final Thoughts

Managing database schema changes doesn't have to be a source of anxiety.

By treating your database schema as code, you eliminate the chaos of manual SQL scripts. You prevent the dreaded "schema drift" where every developer's local machine behaves differently. Most importantly, you make your deployments predictable and boring (which is exactly what you want deployments to be).

In this tutorial, you built a practical Spring Boot application from scratch. You learned how Liquibase intercepts the application startup, locks the database, calculates cryptographic checksums, and safely applies incremental changes. You evolved a single table into a relational schema, added seed data, and learned how to avoid the most common traps beginners fall into.

The next time you start a Spring Boot project, don't reach for a manual SQL client. Add the Liquibase dependency, create your master changelog, and start version controlling your database from day one. Your future self (and your team) will thank you.

RAG Explained Simply with a Real Project

Ashutosh Krishna — Thu, 28 May 2026 16:17:22 +0000

If you have used ChatGPT, you know how magical it feels. You ask a question, and it instantly generates a highly articulate answer.

But you also probably know its biggest flaw. If you ask it about your company's internal code, your private Notion workspace, or an event that happened yesterday, it fails.

Usually, it does one of two things. It either apologizes and says it doesn't have access to that information, or worse, it confidently makes something up entirely.

This happens because Large Language Models (LLMs) are like extremely smart students who are locked in a room without internet access. They only know what they memorized before they were locked inside. If you ask them a question outside of their memorized knowledge, they have to guess.

So, how do we fix this? How do we get an AI to answer questions about our private data without retraining the entire model from scratch?

The answer is RAG, which stands for Retrieval-Augmented Generation.

RAG is the architecture behind nearly every modern AI application that interacts with private data. If you have ever used a "chat with PDF" app or a customer support bot that actually knows company policies, you have interacted with RAG.

In this article, we'll break down exactly how RAG works from first principles. Then, we'll build a working RAG application from scratch using Python.

What is RAG?

RAG stands for Retrieval-Augmented Generation. Let's break down what those three words actually mean.

Retrieval: Finding relevant information from a database.
Augmented: Adding that information to the user's original question.
Generation: Asking the LLM to write an answer using only the added information.

The Open-Book Test Analogy

To build a mental model, think of a traditional LLM as a student taking a closed-book exam. The student has read billions of books in the past, but right now, they have to answer questions purely from memory. Sometimes they forget facts, and sometimes they make up answers to avoid leaving the page blank. Not gonna lie, I pulled the same move in quite a few university exams.

RAG turns this into an open-book exam.

When you ask a question, the system first runs to a massive library (your database), finds the exact pages that contain the answer, and hands those pages to the student. The student then reads those specific pages and writes a perfect answer.

Instead of relying on the AI's memory, we're only relying on its reading comprehension skills.

Why Traditional LLMs Fail

Before we dive into how to build RAG, we need to understand exactly why prompting an LLM on its own isn't enough.

Training cutoffs: Training an LLM takes months and costs millions of dollars. Because of this, models are trained on data up to a specific date. If an LLM was trained in 2025, it has absolutely no idea what happened in 2026.
No access to private data: Your company's Jira tickets, internal wikis, and Slack messages are private. OpenAI, Google, and Anthropic don't have them in their training datasets.
Hallucinations: LLMs are essentially advanced autocomplete engines. They predict the next most likely word based on patterns. If they don't know a fact, they'll string together words that sound highly plausible but may be completely incorrect. We call this hallucinating.
Context window limitations: You might be thinking, "Why not just copy and paste my entire company wiki into the ChatGPT prompt?" Well, every LLM has a "context window", which is the maximum amount of text it can process at once. Even with modern models that have massive context windows, pasting thousands of documents into a prompt is incredibly slow and expensive. Also, models tend to lose track of information when you overwhelm them with too much text.
The high cost of retraining: You could theoretically fine-tune an LLM on your private data. But fine-tuning is complicated and expensive. More importantly, knowledge changes constantly. If you update a company policy, you would have to fine-tune the model all over again to teach it the new rule.

RAG solves all of these problems. It gives the LLM access to real-time, private data without needing to retrain the model.

How RAG Works Internally

To make RAG work, we need a specific pipeline of technologies. Let's explore every major concept in the RAG architecture.

Documents

Everything starts with your raw data. These are your PDFs, database records, text files, or scraped websites. In the AI world, we refer to all of these source materials generally as "documents".

Chunking

You can't feed a 500-page book into an AI all at once for a simple question. It's inefficient. Instead, we break the documents down into smaller, manageable pieces called "chunks". A chunk might be a single paragraph or a few sentences.

This matters because when a user asks a question, we only want to retrieve the specific paragraphs that contain the answer, not the entire book. If we skipped chunking, the system would retrieve massive walls of text, which would crash the LLM's context window.

Embeddings

This is the most intimidating term for beginners, but the concept is brilliant. Computers don't understand words, but they're great at math. Embeddings are a way to translate human language into lists of numbers (vectors) that capture the actual meaning of the text.

Imagine a 2D map. We can plot the word "Dog" at coordinates [2, 3] and the word "Puppy" at [2.1, 3.1]. Even though they're different words, the computer knows they mean similar things because their coordinates are physically close together on the map. The word "Car" might be way over at [10, 10].

In a real AI system, an embedding model doesn't use just 2 dimensions. It maps sentences across thousands of dimensions to capture deep semantic meaning.

Vector Databases

Once we convert all of our text chunks into number coordinates (embeddings), we need a place to store them. Traditional SQL databases are great at finding exact keyword matches. But they're terrible at finding "similar meanings".

A vector database is specifically designed to store lists of numbers and quickly calculate the distance between them. Popular vector databases include ChromaDB, Pinecone, Weaviate, FAISS, and Milvus.

Semantic Search and Similarity Matching

When a user types a question into our chatbot, we run the question through the exact same embedding model. The question becomes a list of numbers.

We then ask the vector database to perform a similarity search. The database looks at the coordinates of the user's question and finds the stored chunks that are located closest to it in mathematical space. Because distance equals meaning, the closest chunks will contain the most relevant information to answer the question.

Prompt Augmentation

Now we have the user's original question and the text chunks we retrieved from the database. We "augment" (add to) the prompt. We create a hidden template behind the scenes that looks like this:

"You are a helpful assistant. Use ONLY the following context to answer the user's question.

Context:

[Insert retrieved chunks here]

Question:

[Insert user question here]"

Final LLM Response

We send this giant, augmented prompt to the LLM. The LLM reads the context, processes the question, and generates a factual response based entirely on the provided data.

Quick Recap

A RAG pipeline usually looks like this:

How to Build a Real RAG Project

Let's build a real-world RAG application. We'll build an AI chatbot that reads and understands a PDF document.

To make this completely free to build, we'll use Python, LangChain (a popular AI framework), Google's Gemini API (which has a generous free tier for developers), and ChromaDB (a local vector database).

Note: We'll be using the free Gemini tier here for illustration purposes so you can learn without spending money. Because LangChain is modular, you can easily swap this out for any other production-grade model later just by changing one line (or a few lines) of code.

Project Setup

First, open your terminal or command prompt, create a new directory for your project, and navigate into it:

mkdir my-rag-project
cd my-rag-project

Next, it's a best practice to create an isolated virtual environment. This ensures that the packages we install for this project don't conflict with other Python projects on your computer.

To create and activate a virtual environment, run the commands for your specific operating system:

For macOS and Linux:

python3 -m venv venv
source venv/bin/activate

For Windows (Command Prompt):

python -m venv venv
venv\Scripts\activate

For Windows (PowerShell):

python -m venv venv
.\venv\Scripts\Activate.ps1

Once activated, you'll see (venv) appear at the beginning of your terminal line. Now, go ahead and install the required libraries inside your fresh environment:

python -m pip install --upgrade pip
pip install langchain langchain-google-genai langchain-community chromadb python-dotenv pypdf

You'll also need a Google Gemini API key. You can get one for free from Google AI Studio.

Instead of running messy terminal configuration commands for different operating systems, create a new file named .env in the root of your project folder and add your key like this:

GOOGLE_API_KEY=your_actual_api_key_here

Preparing the PDF

Since this is a "Chat with PDF" project, you’ll need a sample PDF document to work with. To keep things simple, download this ready-made sample document below and place it inside your project folder.

You can then use this PDF throughout the tutorial for testing uploads, parsing, embeddings, and chat functionality.

Writing the RAG Code Step-by-Step

Create a Python file named rag_app.py in your project folder. Instead of copying a massive block of code, we'll build this application block by block so we can understand exactly how data flows through our pipeline.

Step 1: Imports and Environment Setup

At the very top of your file, add the necessary library imports and initialize your environment configuration:

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_google_genai import GoogleGenerativeAIEmbeddings, ChatGoogleGenerativeAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnablePassthrough

# Load environment variables from the .env file
load_dotenv()

We're bringing in LangChain modules to handle loading, splitting, embedding, storing, and prompting. The load_dotenv() function is mandatory because it scans our .env file and loads the GOOGLE_API_KEY into our system's background environment variables, ensuring our AI models can authenticate seamlessly without hardcoding passwords.

Step 2: Loading the PDF Document

Next, let's point our script to the PDF document we downloaded earlier:

print("Loading PDF document...")
loader = PyPDFLoader("TechCorp_Official_Employee_Handbook.pdf")
document = loader.load()

print(document[0].page_content)

Computers can't read a PDF like a standard text file because PDFs contain complex layout streams. PyPDFLoader handles the heavy lifting of opening the file, stripping away visual layout formatting, and extracting the raw text characters into a clean format that LangChain can work with.

At this point, when you run the script, you should see the text content from the first page of the PDF printed in the terminal. This is a quick way to verify that the PDF was loaded successfully and that PyPDFLoader was able to extract readable text from the document correctly.

Step 3: Chunking the Text

Now that the raw text is in memory, we need to chop it up into smaller pieces:

print("Chunking text...")
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(document)

print(chunks[0].page_content)

If a user asks a simple question, sending an entire 100-page document to the LLM is incredibly slow and expensive. RecursiveCharacterTextSplitter cuts the text into segments of roughly 500 characters.

The chunk_overlap=50 parameter tells the text splitter to repeat the last 50 characters of one chunk at the beginning of the next. This helps preserve context between chunks so that sentences or ideas are not abruptly cut off.

Without overlap, important information near chunk boundaries could be separated, making retrieval less accurate. By maintaining a small shared section between neighboring chunks, the model can better understand continuity in the document, resulting in more reliable search results and higher-quality responses.

When you run the script, you should now see the contents of the first text chunk printed in the terminal.

Step 4: Creating Embeddings and Initializing the Vector DB

With our chunks ready, we'll convert them into vector coordinates and save them locally:

print("Creating vector database...")
embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
vector_db = Chroma.from_documents(
    documents=chunks, 
    embedding=embeddings, 
    persist_directory="./chroma_db"
)

This is the mathematical core of RAG. GoogleGenerativeAIEmbeddings takes a raw text chunk and turns it into a list of numbers representing its conceptual meaning. We then hand those chunks and numbers to Chroma, which maps them into a local database directory named chroma_db on your hard drive, allowing for lightning-fast mathematical lookups later.

Step 5: Setting Up the Retriever and Prompt Template

Now we need a mechanism to query our database and a structure to house our instructions:

# Configure the database to act as a document retriever
retriever = vector_db.as_retriever(search_kwargs={"k": 2})

# Define the hidden prompt structure for the LLM
template = """
Use the following pieces of retrieved context to answer the question. 
If you don't know the answer, just say that you don't know. 
Use three sentences maximum and keep the answer concise.

Context: {context}

Question: {question}

Answer:
"""
prompt = PromptTemplate.from_template(template)

vector_db.as_retriever() converts the vector database into a retriever object that can search through stored document embeddings and return the most relevant chunks for a user’s question. Setting k=2 on our retriever tells the database to only pull the top two most relevant chunks for any given question, which keeps things clean and efficient.

The prompt template acts as hidden instructions for the model. When a user asks a question, LangChain automatically replaces {context} with the retrieved document chunks and {question} with the user’s actual query. The template also acts as a safety guardrail. By explicitly telling the model to say "I don't know" if the context lacks information, we heavily suppress the model's tendency to hallucinate fake answers.

Step 6: Initializing the LLM and Constructing the RAG Chain

Next, we hook up our language model and construct our execution pipeline:

# Initialize the free Gemini model tier
llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", temperature=0)

# Helper function to stitch retrieved chunks into a single text block
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# Connect everything together using LangChain Expression Language (LCEL)
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
)

We use gemini-3.5-flash with a temperature=0 setting to force the model to be completely factual and analytical rather than creative.

The retriever returns multiple document chunks as structured objects. The format_docs function converts those chunks into a single continuous text block by joining their page_content. This step is necessary because the prompt expects a clean, readable context string rather than a list of document objects.

Finally, we connect everything using LangChain Expression Language (LCEL). When a question comes in, it passes it to the retriever, formats the resulting text documents, passes the filled template to the prompt handler, and pushes the final product straight to the LLM.

Step 7: Invoking the Chain with a Question

Finally, let's execute the pipeline and print the result out to the console:

user_question = "What days can I work from home?"
print(f"\nQuestion: {user_question}")

response = rag_chain.invoke(user_question)
print(f"Answer: {response.content}")

This is where the magic happens. The invoke command sets off the entire chain reaction we just built. When you run this, the console will output:

Loading PDF document...
Chunking text...
Creating vector database...

Question: What days can I work from home?
Answer: [{'type': 'text', 'text': 'You are permitted to work from home on Tuesdays and Thursdays. Additional remote flexibility may also be approved by your department manager.', 'extras': {'signature': 'Eo0JCooJAQw51seue7vZT7Vby90GMDLhtOBWLKm5UjfEro7f8dRoKC0KAIHxSqQSLXq0s3kf6yfzTsgaUMFiNd0fnwtNSNoApzcZ7huRD8iq+f+xomoXGhmFYClnLApHUKtOLykICluJnM1j6DfYGaVHKLqU0MF4+Fng9CdqXVqPgN9HcfJEvSpeMAc9vTYENj07s8N6MidlMvMt1w0fl4GCjxAZXyEngdU4kGfjUqaKyjjCQ9yLFeoXrV55pqZdkElLxXEK4ZWNnMGh5NDqGmt2b0kMG4KoCdunUltBr1ctV15rZ+724T0qnjDvI+pIgp/ZtKa423gaVXSkSmdvSePEog38blJ2dgjtZg72XF5xlh45Yv06fZVu7e60ZB1sTn4W8iWuYGQ61i/xCN6xCX/e3SuitjwQoHSlEe/iuoaNf5BXhdp87TUyQTawiY+qIZjgWz2AMLUbMcOvns/0iFt6jpUkXr/dO4eYF39UCosrbWC5TZQp2gllNQ6mlrczTAKqe8mPZwmBVuTJ3kx3q+SsVROln584EdD94IxXrgLXhuLkbR9ub0qyvjBfAmIfvUEK5pcaBCGydQvheH9wsIvAOG1kspMb/wqjAv/mpmii8J9vztSvM9PR9v7L3YLu8vcANol80w2PfeHhyWUJWit8R58kKd7HHor5GJhA436x+tCukIlBq2oTcob+ydxVJydA12pRsiuw4kYkEIU8nr5yCiIwjYCDtVm6Ws0RUnhyk5u+dRONPZ6g+mfBShKCnahcIMzzJpXznmPXvmP2C96uD64SGTI6L86EMlLEz06/cTJTabgqAYqe2AhERgnYc/4d0XabQOkzvDmBKMr5/LOAt3ZZg7X4PIuefEwxx0eB60gLROefcbbu8k+KPazqFsDP/YA/aPyAxyss/6V43EID0amJcDA81LKJzazL9KnclefQZrN9viIwteMaV04IIlx+Ynk1vZi/LVgWiFuDVWF3Ql2luY4KwFpfFDxQ728gkrhvUdTBrfUeKRSLV1W4ox6I7ogo0e9i7db2lkOQljctGs3Km3hWu4JOkH+YzLNmcDHMF3imfgQH5Ml99H9PXh1ScBjq47MXKzJPdHijkY5ZRSjceEIlKEGv8afQO60NB8lk1MQAGwd+CxqIwVg11N8q9EFSwdJmVVmoyM1nINGJERSKhKOrkqBsOELfpKDjv14tuNgDUy4wdtuxn8C4tJBKvN8t/hrW/Z65VoBGdMwA08sRSV6Fp5l/gSdYeB9yA/Lx/VGkgVqaP5tU73XrE/XO8ysJ/kgRDXiTvsg+2uayU1Q9PfKFAawopslwybCHtdOwaVgsRdA5R4f1NIkPoP/sX+iBxyR0kKg6v4RRAj851WifM2fQ8Vsw5dtFSeh/4TfYg1GCCCDNT4JwrtI8fqcF+qMQqUb+oUqoyzjzFqqSRxXcyqHXOLV9V9C6yWYmZ3TSY043WL9L4kGGJGxFHD5VWG77Quiy+rHWGO13LOc5EBKIO05sg1xnI88QQTUgkxwJeuntytIy3f3pfMVrFYFkvi8w5LzL4RK68+4HMg=='}}]

Modern LLMs like Google's Gemini are multimodal. This means they're designed to read and generate not just plain text, but images, video, and audio simultaneously. Because of this, the LangChain Google integration doesn't always return a simple text string. Instead, it returns a list of content blocks.

In your output, the AI successfully returned your text, but it also included an extras dictionary containing a signature. This signature is a behind-the-scenes data point used by Google for AI safety tracking, grounding metadata, and thought-process verification.

To get a clean, human-readable string, you simply need to extract the text value from that list. You can update your final print statement to check if the response is a list and extract the text automatically:

# Clean up the output if Gemini returns a list of content blocks
if isinstance(response.content, list):
    clean_answer = response.content[0]['text']
else:
    clean_answer = response.content

print(f"Answer: {clean_answer}")

Now, your output will look like this:

Question: What days can I work from home?
Answer: You are permitted to work from home on Tuesdays and Thursdays. Additional remote flexibility may also be approved by your department manager.

Step 8: Making it Conversational

Right now, our script hardcodes a single question, prints the answer, and immediately exits. In the real world, you want to chat with your documents naturally. Let's upgrade our script to run continuously in your terminal so you can ask as many questions as you want without restarting the program.

Replace the bottom section of your code with a simple while loop:

# Chat with your PDF in a continuous loop
print("\n--- PDF Chatbot Initialized ---")
print("Type 'exit' or 'quit' to stop.")

while True:
    # 1. Wait for the user to type a question
    user_question = input("\nYour Question: ")

    # 2. Allow the user to break the loop and close the program
    if user_question.lower() in ['exit', 'quit']:
        print("Shutting down chatbot. Goodbye!")
        break

    # 3. Send the question through our RAG chain
    response = rag_chain.invoke(user_question)

    # 4. Clean up the output format
    if isinstance(response.content, list):
        clean_answer = response.content[0]['text']
    else:
        clean_answer = response.content

    # 5. Print the final answer to the console
    print(f"Answer: {clean_answer}")

By using Python's input() function wrapped inside an infinite while True loop, we keep the Python script alive. The PDF chunks and vector database stay loaded in your computer's memory, allowing you to fire off consecutive questions instantly. This transforms your script from a static demonstration into a fully interactive AI tool!

Here's a sample run:

Full Code

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_google_genai import GoogleGenerativeAIEmbeddings, ChatGoogleGenerativeAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnablePassthrough

# Load environment variables from the .env file
load_dotenv()

print("Loading PDF document...")
loader = PyPDFLoader("TechCorp_Official_Employee_Handbook.pdf")
document = loader.load()
# print(document[0].page_content)

print("Chunking text...")
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(document)
# print(chunks[0].page_content)

print("Creating vector database...")
embeddings = GoogleGenerativeAIEmbeddings(model="gemini-embedding-001")
vector_db = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# Configure the database to act as a document retriever
retriever = vector_db.as_retriever(search_kwargs={"k": 2})

# Define the hidden prompt structure for the LLM
template = """
Use the following pieces of retrieved context to answer the question. 
If you don't know the answer, just say that you don't know. 
Use three sentences maximum and keep the answer concise.

Context: {context}

Question: {question}

Answer:
"""
prompt = PromptTemplate.from_template(template)

# Initialize the free Gemini model tier
llm = ChatGoogleGenerativeAI(model="gemini-3.5-flash", temperature=0)

# Helper function to stitch retrieved chunks into a single text block
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


# Connect everything together using LangChain Expression Language (LCEL)
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
)
"""
user_question = "What days can I work from home?"
print(f"\nQuestion: {user_question}")

response = rag_chain.invoke(user_question)
# print(f"Answer: {response.content}")

# Clean up the output if Gemini returns a list of content blocks
if isinstance(response.content, list):
    clean_answer = response.content[0]['text']
else:
    clean_answer = response.content

print(f"Answer: {clean_answer}")
"""

# Chat with your PDF in a continuous loop
print("\n--- PDF Chatbot Initialized ---")
print("Type 'exit' or 'quit' to stop.")

while True:
    # 1. Wait for the user to type a question
    user_question = input("\nYour Question: ")

    # 2. Allow the user to break the loop and close the program
    if user_question.lower() in ['exit', 'quit']:
        print("Shutting down chatbot. Goodbye!")
        break

    # 3. Send the question through our RAG chain
    response = rag_chain.invoke(user_question)

    # 4. Clean up the output format
    if isinstance(response.content, list):
        clean_answer = response.content[0]['text']
    else:
        clean_answer = response.content

    # 5. Print the final answer to the console
    print(f"Answer: {clean_answer}")

Taking it out of the terminal

Once you have your terminal chatbot working, you probably want to give it a proper visual interface. The easiest way to do this in Python is using an open-source library called Gradio. Gradio has a built-in ChatInterface feature that can wrap your existing RAG code and automatically generate a beautiful, ChatGPT-style web UI in your browser with just three extra lines of code. It's highly recommended as your next mini-project.

The Full Data Flow

To truly solidify your understanding, let's map out the exact lifecycle of a single user question in our system:

Breaking Down the Execution Timeline

The request begins: The user interfaces with our console and asks a text-based question: "How much vacation do I get?" At this exact moment, our application code takes control of the program flow.
The text-to-vector translation: Computers can't compute similarity using raw text characters. Our app makes a fast network call to the Google Embedding Model, handing over the raw question. The model converts the text into a massive array of numbers that mathematically represents the user's intent.
The database distance calculation: Our application script takes those coordinate numbers and passes them directly to ChromaDB. ChromaDB scans the local hard drive, running a similarity math function against the numbers stored for each of our PDF chunks. It locates the text chunk mentioning "20 days of paid time off" because its coordinates are physically closest to the query coordinates.
The prompt augmentation: ChromaDB hands the raw text strings of those relevant pieces back to our script. The code automatically unrolls our prompt template, plugging the raw chunks into the {context} slot and the user's original text into the {question} slot.
The final generation: Our application drops this combined package into the final network call, pushing it directly to the Gemini LLM. Because temperature=0 is configured, the model acts strictly as a reading comprehension engine. It reads the custom context, formats a clean sentence, and sends it back to our terminal to be printed out beautifully for the user.

Common RAG Problems

Building a simple RAG app is easy. Building a RAG app that works perfectly in production is very difficult. Here are the most common problems engineers face and how they fix them.

1. Bad Chunking

If your chunks are too large, they include irrelevant information that confuses the LLM. If they're too small, they lose vital context. Engineers can solve this by experimenting with different chunk sizes or using semantic chunking (splitting by whole sentences or paragraphs rather than strict character counts).

2. Irrelevant Retrieval

Sometimes semantic search fails. If a user searches for "Apple" expecting information about fruit, but the database only has data about the tech company, the system will confidently return tech company documents. Engineers can fix this by adjusting the embedding models or adding keyword search rules.

3. Hallucinations

Even with RAG, an LLM might ignore the retrieved context and rely on its training memory. Engineers mitigate this by heavily engineering the prompt template with strict rules like "ONLY use the provided text."

4. Latency

RAG requires an embedding network call, a database search, and an LLM network call. This takes time. Engineers can optimize this by using faster, locally hosted embedding models or caching common questions.

5. Stale Data

If HR updates the company policy PDF, the vector database still holds the old numbers. The AI will give outdated answers. Engineers build update pipelines that automatically delete old vectors and embed new ones whenever a source file changes.

Advanced RAG Concepts

Once you master basic RAG, the AI engineering world opens up to highly advanced techniques.

Hybrid Search

Vector databases are great at understanding meaning, but bad at finding exact ID numbers or specific names. Hybrid search combines traditional keyword search (like searching a SQL database) with semantic vector search to get the best of both worlds.

Reranking

Sometimes the vector database returns 10 chunks, but the best answer is accidentally placed at the bottom of the list. Reranking uses a second, specialized AI model to read the retrieved chunks and sort them strictly by relevance before sending them to the LLM.

Agentic RAG

Instead of forcing the system to retrieve documents every single time, Agentic RAG uses an AI "Agent" to decide if it even needs to search. If you say "Hello", the agent skips the database and just says "Hi". If you ask a hard question, it decides to query the database.

Graph RAG

Instead of breaking text into isolated chunks, Graph RAG extracts entities (people, places, concepts) and maps how they relate to each other in a Knowledge Graph. This is incredibly powerful for complex datasets with deep relationships.

Traditional RAG only reads text. Multi-modal RAG processes images, charts, and audio files, allowing users to ask questions like, "What does the graph on page 4 indicate?"

Final Thoughts

Retrieval-Augmented Generation is the bridge between incredible reasoning engines (LLMs) and reliable factual knowledge (your data).

Understanding RAG is no longer optional for software engineers. Nearly every enterprise software product being built today involves some form of it. By learning how chunking, embeddings, vector databases, and prompt augmentation work together, you have demystified the magic behind modern AI.

Your next step is to build on the code we wrote today. Try pointing the PDF loader to your résumé, a school textbook, or a financial report. Once you experience your own code answering questions about your personal data, you'll start to truly understand the power of AI engineering.

From Symptoms to Root Cause: How to Use the 5 Whys Technique

Ashutosh Krishna — Thu, 23 Apr 2026 16:48:41 +0000

Most teams don't struggle because they can't fix problems. They struggle because they fix the wrong thing.

An API fails in production. You restart the service, errors go away, and it feels resolved. Until it happens again. And again. What's happening here is simple: you're treating symptoms, not the underlying cause.

The 5 Whys technique is a straightforward way to deal with this. It comes from the Toyota Production System and was designed to help teams dig deeper into problems instead of settling for quick fixes.

The idea is simple. Ask "why" repeatedly until you reach the real cause.

But in practice, this is where things go wrong.

Teams often:

Stop too early
Assume answers without checking data
Focus on people instead of systems
Treat "five" as a rule instead of a guideline

So even though the process looks structured, the outcome is still shallow.

In this article, we'll focus on how to actually use the 5 Whys in real situations. Not just the theory, but what it looks like when you apply it to an engineering problem.

Here's What We'll Cover:

What is the 5 Whys Technique?
Origins of the 5 Whys Method
How to Conduct an Effective 5 Whys Analysis
Real-World Example: Applying 5 Whys in an Engineering Scenario
When to Use (and When Not to Use) 5 Whys
Benefits of the 5 Whys Technique
Common Pitfalls and Limitations
Tips for Using 5 Whys Effectively
Summary

What is the 5 Whys Technique?

The 5 Whys technique is a way to break down a problem by repeatedly asking why it happened, with the goal of reaching a cause that actually explains the issue and can be addressed.

At its core, it's not about the number five. The name can be misleading. What matters is the process of following a chain of cause and effect until the explanation stops being superficial and starts becoming useful.

Each answer you uncover should move you one level deeper. You start with what went wrong, then explore what led to it, and continue until you reach something that is both believable and actionable. In most real situations, that final answer is not a single event but a gap in a system, a missing check, or an assumption that was never validated.

The technique became widely known through the Toyota Production System, where it was used to improve processes by focusing on causes rather than quick fixes.

That context is important because it highlights the original intent. The goal was not just to explain problems, but to prevent them from happening again.

A simple example makes this clearer. Imagine a mobile app suddenly starts crashing after a release. Asking "Why?" might look like this:

Why is the app crashing? → Because a null value is being accessed in the code.
Why is there a null value? → Because the API response is missing a required field
Why is the field missing? → Because a recent backend change made the field optional.
Why was this change not handled in the app? → Because the app assumes the field is always present.
Why was this assumption not caught earlier? → Because there are no contract tests validating API responses.

At this point, the issue is no longer just "fix the null check". The deeper problem is the lack of validation between systems, which allows breaking changes to slip through.

A useful way to think about the 5 Whys is that it forces you to stay with the problem a little longer than you normally would. Most of the time, the first explanation feels sufficient, so it's easy to stop there. This method pushes you to go one step further, and then another, until the explanation holds up under scrutiny.

At the same time, it's not a rigid formula. You might reach a solid root cause in three steps, or it might take more than five. The quality of the reasoning matters more than the count.

Origins of the 5 Whys Method

The 5 Whys method comes from the Toyota Production System, a manufacturing approach focused on continuous improvement and problem solving at the source.

It's often associated with Sakichi Toyoda, whose philosophy was simple: don’t just fix a problem. Understand why it happened so it doesn't happen again.

Inside Toyota, this wasn't treated as a formal tool or checklist. It was part of the day-to-day way of working. When something went wrong on the production line, the goal wasn't to get things running quickly and move on. The goal was to stop, investigate, and make sure the same issue wouldn't repeat.

That mindset is important to understand. The 5 Whys was never meant to be a rigid exercise where you ask five questions and stop. It was a way to encourage deeper thinking and accountability in processes.

Another key idea in the Toyota system is that problems are usually caused by processes, not people. Instead of asking "who made the mistake", the focus is on "what allowed this mistake to happen". The 5 Whys fits naturally into this approach because it pushes you toward system level causes rather than individual blame.

Over time, the method spread beyond manufacturing and is now used in software engineering, product teams, operations, and many other fields. The context has changed, but the core idea remains the same: if you don't understand the cause, you're likely to see the same problem again.

This origin story is useful not just as background, but as a reminder of intent. The value of the 5 Whys doesn't come from the questions themselves. It comes from the discipline of not settling for the first answer.

How to Conduct an Effective 5 Whys Analysis

A 5 Whys analysis works best when it is treated as a structured way of thinking, not a checklist to rush through. The quality of the outcome depends less on how many times you ask "why" and more on how carefully you reason through each step.

It helps to approach it in stages, each with a clear purpose.

Step 1: Define the Problem Clearly

Start with a problem statement that is specific and observable. Avoid vague descriptions like "the system is slow" or "things are failing". Instead, describe what actually happened in a way that can be verified.

For example, "API response time exceeded 5 seconds for 30 percent of requests between 2 PM and 3 PM" is much more useful than "API is slow".

A clear problem statement keeps the analysis grounded. If the starting point is fuzzy, the entire chain of reasoning will drift.

Step 2: Ask "Why" Iteratively

Once the problem is defined, begin asking why it happened. Each answer should directly address the question before it and naturally lead to the next one.

The key here is continuity. Every step should feel like a logical extension of the previous one. If you find yourself jumping topics or introducing unrelated explanations, it's a sign that the chain is breaking.

Keep going until the answers stop being immediate symptoms and start pointing toward underlying conditions or decisions.

Also, don't force the process to stop at five. Some problems may need fewer steps, while others may need more. What matters is reaching a point where the explanation is meaningful and actionable.

Step 3: Validate Each Answer with Evidence

This is where many analyses go wrong. It's easy to come up with plausible answers, but plausibility is not enough.

Each "why" should be backed by some form of evidence. This could be logs, metrics, recent changes, or direct observation. If an answer can't be verified, treat it as a hypothesis and confirm it before moving forward.

Without validation, the entire analysis becomes a chain of assumptions. Even if the final answer sounds reasonable, it may not reflect reality.

Step 4: Identify the Root Cause

A good root cause is one that explains the sequence of events and can be acted upon to prevent the issue in the future.

In many cases, this turns out to be a gap in a process rather than a single technical failure. It could be a missing validation step, an incomplete test, or an assumption that was never challenged.

If the final answer still feels like a symptom, you probably need to go one level deeper. On the other hand, if the answer points to something you can change in your system or workflow, you are likely in the right place.

Step 5: Define Corrective Actions

The analysis is only useful if it leads to meaningful action.

Once you've identified the root cause, the next step is to define changes that prevent the problem from happening again. These should go beyond quick fixes and address the underlying issue.

For example, instead of just fixing a bug, you might introduce better testing, add monitoring, or improve review processes.

Good corrective actions share a few traits: they're specific, practical to implement, and they directly address the root cause identified in the analysis.

Real-World Example: Applying 5 Whys in an Engineering Scenario

To see how this works in practice, let’s walk through a realistic backend issue. The goal here is not just to reach an answer, but to show how each step builds on evidence and leads to something actionable.

The Problem:

Users report intermittent failures while fetching order details:

GET /api/orders/{id}
→ HTTP 500 Internal Server Error

Application logs show:

// Java 21 example (Spring Boot style logging)
logger.error("Database connection timeout while fetching order", ex);

At this point, it's tempting to conclude that the database is the problem. But that's only what we can see on the surface.

Applying the 5 Whys

1. Why did the API return a 500 error?

Because the database query timed out.

This is directly supported by the error logs, so we can treat it as a confirmed fact.

2. Why did the query time out?

Because the database connection pool was exhausted.

Metrics show that all available connections were in use during peak traffic.

3. Why was the connection pool exhausted?

Because some requests were holding database connections for too long.

Slow query logs confirm that a subset of queries had unusually high execution times.

4. Why were some queries slow?

Because a recently introduced feature added a query on a non-indexed column.

Looking at recent deployments reveals a change that introduced filtering without proper indexing.

5. Why was an unoptimized query deployed to production?

Because there is no performance validation step in the development or release process.

There are no checks in code review or CI/CD to catch inefficient database queries before deployment.

Root Cause

The issue is not the timeout itself.

It's this:

The system allows inefficient database queries to reach production without any safeguards.

What a Shallow Fix Would Look Like

If we stopped early, we might:

Increase the database timeout
Increase the connection pool size

These might reduce the frequency of failures, but they don't solve the underlying problem.

What a Strong Fix Looks Like

A proper 5 Whys analysis leads to changes that improve the system:

Add appropriate indexing for frequently queried fields
Introduce query performance checks in CI/CD pipelines
Add monitoring and alerts for slow queries
Include database considerations in code reviews

Why This Example Matters

The difference between a shallow fix and a real solution is depth.

The first explanation often feels sufficient, especially under pressure. But stopping there means the issue is likely to return in a different form.

The value of the 5 Whys comes from following the chain all the way to something you can change in your system.

When to Use (and When Not to Use) 5 Whys

Like any problem-solving method, the 5 Whys is useful in the right context and less effective in others. Knowing when to apply it is just as important as knowing how to use it.

If used appropriately, it can uncover meaningful insights. If used in the wrong situation, it can lead to oversimplified or misleading conclusions

When to Use 5 Whys

The 5 Whys is most useful when your goal is to understand why something happened, not just to fix it and move on.

It works well in situations where problems are recurring or not fully explained by the first answer. For example, production incidents, repeated bugs, or issues that reappear after a quick fix are strong signals that you need deeper analysis. In these cases, the technique helps uncover what is happening beneath the surface.

It's also effective during retrospectives and postmortems. When a release doesn't go as expected or a sprint runs into issues, the 5 Whys helps teams move beyond observations like "this failed" and get to "why did this fail in the first place".

In general, use it when:

The problem is not obvious
The issue has occurred more than once
You want to prevent recurrence, not just resolve the current instance

When Not to Use 5 Whys

The 5 Whys has its limits, and using it in the wrong context can lead to oversimplified conclusions.

If a problem involves multiple interacting factors, a single chain of "why" questions may not capture the full picture. Complex systems often have several contributing causes, and forcing them into one linear explanation can hide important details. In such cases, the 5 Whys should be combined with other approaches.

It's also less effective when there's not enough data. If each answer is based on assumptions rather than evidence, the analysis quickly becomes unreliable. The method depends on validation at every step.

Another limitation is in time-critical situations. During an active incident, the priority is to restore the system. The deeper analysis should happen later, once things are stable.

Finally, if your goal is quantitative analysis or optimization, the 5 Whys alone isn't enough. You'll need more data-driven methods to support decision making.

A simple rule of thumb is this. If you are trying to learn from a problem, use the 5 Whys. If you are trying to fix something immediately or analyze complex data, use it carefully or alongside other techniques.

Benefits of the 5 Whys Technique

The 5 Whys technique is simple, but it offers several powerful benefits that can help you solve problems more effectively and make lasting improvements. Here are the key advantages:

Simple and Easy to Apply

One of the biggest strengths of the 5 Whys is how easy it is to start using. You don't need special tools, training, or complex frameworks. It can be applied in a quick discussion, during debugging, or as part of a formal postmortem.

This low barrier makes it accessible across teams, regardless of experience level.

Encourages Deeper Thinking

The method naturally pushes you to go beyond the first explanation. Instead of reacting to what's visible, it encourages you to question why the problem occurred in the first place.

This shift from surface-level fixes to deeper understanding often leads to better decisions.

Promotes System-Level Improvements

When used correctly, the focus moves away from individual people and toward systems. Instead of asking who made a mistake, the analysis asks what allowed the mistake to happen.

This leads to improvements in processes, safeguards, and overall system design rather than one-off fixes.

Works Well in Team Settings

Because the approach is simple, it's easy for multiple people to contribute. Different perspectives help uncover gaps that might otherwise be missed.

It also creates a shared understanding of the problem, which is valuable during retrospectives and incident reviews.

Helps Prevent Recurring Issues

Quick fixes often solve the immediate problem but don't stop it from happening again. The 5 Whys helps identify underlying causes, which makes it easier to prevent similar issues in the future.

Over time, this leads to more stable systems and fewer repeated incidents.

Common Pitfalls and Limitations

While the 5 Whys technique is useful, it’s not always perfect. There are some limitations to keep in mind, so you can use it effectively and know when it might not be enough.

Stopping Too Early

One of the most common mistakes is ending the analysis after the first or second answer. These early answers usually describe symptoms, not causes.

Stopping too soon leads to fixes that address the surface but leave the underlying issue unresolved.

Treating Assumptions as Facts

It's easy to come up with explanations that sound reasonable. But without evidence, they're just assumptions.

If each step isn't validated with logs, metrics, or observations, the entire analysis can drift away from reality.

Focusing on Individuals Instead of Systems

Answers like "someone made a mistake" don't add much value. While they may be true, they don't explain why the system allowed that mistake to have an impact.

Focusing on processes and safeguards leads to more meaningful improvements.

Oversimplifying Complex Problems

The 5 Whys follows a linear chain of reasoning, but real-world systems often have multiple contributing factors.

Relying on a single chain can hide important interactions. In such cases, the method should be combined with other approaches.

Treating It as a Rigid Formula

The name suggests asking "why" five times, but this shouldn't be taken literally. Some problems require fewer steps, while others need more.

Forcing the structure can lead to artificial or weak conclusions.

Not a Replacement for Deeper Analysis

The 5 Whys isn't designed for every type of problem. For complex system failures, performance optimization, or data-heavy investigations, additional tools and methods are often required.

It works best as a starting point or a complement to other techniques, not a complete solution on its own.

Tips for Using 5 Whys Effectively

To get the most out of the 5 Whys technique, there are a few tips that can help you use it effectively. These will guide you to ask the right questions and reach useful, actionable insights.

Start with a Clear, Specific Problem

A vague problem leads to vague answers. Spend a little extra time making sure the problem statement is precise and based on observable facts. This keeps the analysis grounded and avoids unnecessary detours.

Base Every Step on Evidence

Treat each answer as something that needs to be verified. Use logs, metrics, recent changes, or direct observations to support your reasoning. If something can't be validated, call it out as a hypothesis and confirm it before moving forward.

Keep the Chain Logical and Connected

Each "why" should naturally follow from the previous answer. If the reasoning starts to jump between unrelated ideas, pause and re-evaluate. A clean, logical chain is a strong indicator that you're on the right track.

Focus on Systems, Not Individuals

Avoid stopping at explanations that point to human error. Instead, ask what allowed that error to have an impact. This shift in thinking leads to improvements that actually reduce the chances of similar issues in the future.

Do Not Force Exactly Five Steps

The number five is a guideline, not a rule. Some problems become clear in three steps, while others need more exploration. Stop when you reach a cause that's both convincing and actionable.

Involve the Right People

If possible, do the analysis as a group. People from different parts of the system bring different perspectives, which helps uncover details that might otherwise be missed. It also creates shared ownership of both the problem and the solution.

Turn Insights into Actions

The analysis only matters if it leads to change. Make sure the final outcome includes clear, practical steps that address the root cause. Without this, even a well-done analysis has limited impact.

Summary

The 5 Whys is a simple technique, but using it well takes some discipline.

At its core, it's about resisting the urge to stop at the first explanation. By following the chain of cause and effect, you move from symptoms to something you can actually fix. In many cases, that turns out to be a gap in a process rather than a one-off failure.

When applied thoughtfully, it helps teams learn from problems instead of just reacting to them. Over time, this leads to better systems, fewer recurring issues, and more confidence in how problems are handled.

The key is to treat it as a way of thinking, not just a set of steps.

Build a Website Screenshot Generator with Python and Flask

Ashutosh Krishna — Wed, 29 Oct 2025 16:50:16 +0000

Have you ever needed to take screenshots of websites automatically – maybe to track visual changes, include them in reports, or generate previews? Doing this manually can be time-consuming, especially if you need to capture multiple pages regularly.

In this tutorial, you’ll learn how to build a simple website screenshot generator using Python and Flask. The app will let users enter any website URL and instantly get a screenshot of that page – all powered by a screenshot API.

You’ll use Flask, a lightweight web framework, to create a simple web interface and handle requests. Then, you’ll integrate an external API to capture website screenshots programmatically.

By the end of this tutorial, you’ll have learned how to:

Build a basic Flask web app
Accept user input through an HTML form
Make HTTP requests to an external API
Display images dynamically on a web page

This project is a great way to learn how APIs can extend the capabilities of your web applications, and how Python can easily handle tasks like image generation and display.

What we’ll cover:

Prerequisites
Setting Up the Project
Integrating the ScreenshotBase API
Adding Customization Options
Wrapping Up

Prerequisites

Before we start building the app, make sure you have a few basics covered:

1. Python Installed

You’ll need Python 3.9 or higher installed on your machine. You can check your version by running:

python --version

If you don’t have it, you can download it from python.org/downloads.

2. Basic Knowledge of Python and Flask

You don’t need to be a Flask expert. Just have some familiarity with how to create routes and templates, and how to handle form data in Flask.

If you’re new to Flask, don’t worry – we’ll go step-by-step.

3. Screenshot API Key

We’ll be using the ScreenshotBase API to capture website screenshots. It provides a simple REST endpoint that takes a URL and returns a screenshot image.

You can get a free API key by signing up on their website. Once you have the key, keep it handy, as we’ll use it when we integrate the API in the later steps.

4. A Code Editor and Terminal

You can use any editor you prefer, like VS Code, PyCharm, or even a simple text editor. Make sure your terminal or command prompt can run Python commands and install packages using pip.

Setting Up the Project

Let’s start by setting up a new Flask project from scratch.

Step 1: Create a New Folder

Create a new folder for your project. You can name it screenshot-generator:

mkdir screenshot-generator
cd screenshot-generator

Step 2: Create a Virtual Environment

A virtual environment helps keep your project dependencies isolated from other Python projects.

Run the following commands:

python -m venv venv

Then activate it:

On macOS/Linux:
```
  source venv/bin/activate
```
On Windows:
```
  venv\Scripts\activate
```

Once activated, you should see (venv) at the beginning of your terminal prompt.

Step 3: Install Dependencies

We’ll need two packages for this project:

Flask for building the web app
Requests for making API calls to ScreenshotBase

Install them using pip:

pip install flask requests

Step 4: Set Up the Project Structure

Inside your project folder, create the following files and folders:

screenshot-generator/
├── app.py
├── templates/
│   └── index.html
└── static/

Here’s what each part does:

app.py: main Python file that runs your Flask app
templates/: stores HTML templates
static/: stores images, CSS, and JavaScript files

Step 5: Add a Basic Flask App

Open app.py and add the following starter code:

from flask import Flask, render_template, request
import requests
import os

app = Flask(__name__)
API_KEY = os.getenv("SCREENSHOTBASE_API_KEY")
SCREENSHOTBASE_BASE_ENDPOINT = "https://api.screenshotbase.com/v1/take"

@app.route('/', methods=['GET', 'POST'])
def home():
    if request.method == 'POST':
        url = request.form.get('url')
        # Placeholder: We’ll add the API call here in the next section
        return render_template('index.html', url=url)
    return render_template('index.html')

if __name__ == '__main__':
    app.run(debug=True)

This sets up a minimal Flask application with one route (/). We’ll add the ScreenshotBase API call in the next section.

Step 6: Create a Simple HTML Template

Now that we’ve written the Flask backend to handle the screenshot request, let’s create the frontend for our app.

Inside your project folder, create a new directory named templates, and within it, add a file called index.html. Flask will automatically look for templates in this folder.

Here’s how the template should look:

html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Website Screenshot Generatortitle>
    <link
      href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.8/dist/css/bootstrap.min.css"
      rel="stylesheet"
      integrity="sha384-sRIl4kxILFvY47J16cr9ZwB07vP4J8+LH7qKQnuqkuIAvNWLzeN8tE5YBujZqJLB"
      crossorigin="anonymous"
    />
    <style>
        body {
            padding-top: 2rem;
        }
        .screenshot-container {
            max-height: 80vh;
            overflow-y: auto;
            border: 1px solid #ddd;
            border-radius: 8px;
            padding: 1rem;
            background: #f8f9fa;
        }
    style>
head>
<body>
    <div class="container text-center">
        <h1 class="mb-4">Website Screenshot Generatorh1>
        <form method="POST" class="d-flex justify-content-center mb-4">
            <input 
                type="text" 
                name="url" 
                placeholder="Enter website URL" 
                class="form-control w-50 me-2"
                required
            >
            <button type="submit" class="btn btn-primary">Capture Screenshotbutton>
        form>

        {% if screenshot %}
            <h4>Screenshot Preview:h4>
            <div class="screenshot-container mx-auto">
                <img src="{{ screenshot }}" alt="Website Screenshot" class="img-fluid rounded shadow">
            div>
        {% elif request.method == 'POST' %}
            <div class="alert alert-danger mt-3">
                Sorry, something went wrong while capturing the screenshot. Please try again.
            div>
        {% endif %}
    div>
body>
html>

This HTML template uses Bootstrap to keep the design clean and responsive without much custom styling. The form at the top allows users to enter any website URL and submit it to the Flask app using the POST method. Once the app retrieves the screenshot URL from the API, it dynamically renders the image on the page.

The img-fluid class from Bootstrap ensures the screenshot scales properly across all screen sizes while maintaining its aspect ratio. Also, the .screenshot-container provides a scrollable area, which helps display full-page screenshots without shrinking them too much or breaking the layout.

Now your project is set up and ready to capture real screenshots using the ScreenshotBase API.

In the next section, we’ll write the logic to call the ScreenshotBase API and display the resulting screenshot dynamically in the browser.

Integrating the ScreenshotBase API

Now that our Flask app is set up, let’s connect it to the ScreenshotBase API so we can generate real screenshots from any URL.

Step 1: Understanding the API Endpoint

The ScreenshotBase API provides a simple endpoint that takes a website URL and returns a screenshot image.

A typical API call looks like this:

GET https://api.screenshotbase.com/v1/take

You send the target website URL as a query parameter (url) and include your API key in the request header as apikey.

Here’s the cURL example from their documentation:

curl -G https://api.screenshotbase.com/v1/take?url=https%3A%2F%2Fbbc.com \
    -H "apikey: YOUR-API-KEY"

This request captures a screenshot of https://bbc.com and returns the screenshot image as the response.

Step 2: Add the API Call in Flask

Let’s update our home route in app.py to send a request to the ScreenshotBase API when a user submits a URL.

Here’s the updated version of app.py:

from flask import Flask, render_template, request
import requests
import os

app = Flask(__name__)
API_KEY = os.getenv("SCREENSHOTBASE_API_KEY")
SCREENSHOTBASE_BASE_ENDPOINT = "https://api.screenshotbase.com/v1/take"

@app.route('/', methods=['GET', 'POST'])
def home():
    screenshot_url = None

    if request.method == 'POST':
        target_url = request.form.get('url')

        params = {"url": target_url}
        headers = {"apikey": API_KEY}

        try:
            # Send GET request to ScreenshotBase API
            response = requests.get(SCREENSHOTBASE_BASE_ENDPOINT, params=params, headers=headers, timeout=30)
            response.raise_for_status()

            # Save the returned image
            image_path = os.path.join('static', 'screenshot.png')
            with open(image_path, 'wb') as f:
                f.write(response.content)

            screenshot_url = image_path

        except requests.exceptions.RequestException as e:
            print(f"Error capturing screenshot: {e}")

    return render_template('index.html', screenshot=screenshot_url)

if __name__ == '__main__':
    app.run(debug=True)

Here’s what this code does:

Captures the user input (URL).
Sends a GET request to the /v1/take endpoint.
Passes your API key in the request header (apikey).
Saves the returned image to the static folder.
Displays the screenshot on the page.

Step 3: Test Your App

Run the app again:

python app.py

Then open http://127.0.0.1:5000 in your browser.

Enter a URL like:

https://github.com/ashutoshkrris

After submitting, you should see the screenshot of that website displayed below the form.

Your Flask app is now fully functional! It takes a URL, sends it to the ScreenshotBase API, and displays the resulting screenshot.

In the next section, we’ll enhance the app by adding customization options like full-page screenshots, viewport settings, and delays. Later, we’ll explore the ScreenshotBase SDKs for popular languages like Python, Node.js, and PHP.

Adding Customization Options

Right now, our app simply takes a screenshot of the given website URL using the default settings. However, most screenshot APIs (including the one we’re using) allow you to customize the output by passing additional parameters in the request.

The ScreenshotBase API offers several powerful customization options to help you tailor each screenshot to your needs:

Image format: Choose from png, jpg, gif, or webp, depending on your image quality or compression requirements.
Full page capture: Capture the entire scrollable webpage (full_page=1) or just the visible viewport (full_page=0).
Viewport dimensions: Set the browser window size with viewport_width and viewport_height to simulate screenshots from desktop, tablet, or mobile screens.

These options make ScreenshotBase ideal for building automation tools, thumbnail generators, testing dashboards, and visual documentation systems. Let’s add these options to make our screenshot generator more flexible.

Updating the Flask Code

Open your app.py file and update the route that handles form submissions to include these optional parameters:

from flask import Flask, render_template, request
import requests
import os

app = Flask(__name__)
API_KEY = os.getenv("SCREENSHOTBASE_API_KEY", "scr_live_9Qkn1gs01rivZqrFk7lusXPiqAUg85J86aU6bHvG")
SCREENSHOTBASE_BASE_ENDPOINT = "https://api.screenshotbase.com/v1/take"


@app.route('/', methods=['GET', 'POST'])
def home():
    screenshot_url = None

    if request.method == 'POST':
        target_url = request.form.get('url')
        format_ = request.form.get('format', 'png')
        full_page = request.form.get('full_page') == 'on'

        params = {
            "url": target_url,
            "format": format_,
            "full_page": int(full_page)
        }
        headers = {"apikey": API_KEY}

        try:
            # Send GET request to ScreenshotBase API
            response = requests.get(SCREENSHOTBASE_BASE_ENDPOINT, params=params, headers=headers, timeout=30)
            response.raise_for_status()

            # Save the returned image
            image_extension = format_ if format_ != 'jpeg' else 'jpg'
            image_path = os.path.join('static', f'screenshot.{image_extension}')
            with open(image_path, 'wb') as f:
                f.write(response.content)

            screenshot_url = image_path

        except requests.exceptions.RequestException as e:
            print(f"Error capturing screenshot: {e}")

    return render_template('index.html', screenshot=screenshot_url)


if __name__ == '__main__':
    app.run(debug=True)

Updating the HTML Template

Next, let’s modify the form in our index.html to include the customization options:

<form method="POST" class="d-flex flex-column justify-content-center mb-4">
  
  <div class="input-group mb-4">
    <input
      type="text"
      name="url"
      placeholder="Enter website URL"
      class="form-control w-50 me-2"
      required
    />
    <button type="submit" class="btn btn-primary">Capture Screenshotbutton>
  div>

  
  <div class="row g-3">
    
    <div class="col-md-6">
      <div class="form-check">
        <input
          type="checkbox"
          name="full_page"
          id="full_page"
          class="form-check-input"
        />
        <label class="form-check-label" for="full_page">
          Capture Full Page Screenshot
        label>
      div>
    div>

    
    <div class="col-md-6">
      <label for="format" class="form-label">Screenshot Formatlabel>
      <select class="form-select" name="format" id="format" required>
        <option value="png">PNGoption>
        <option value="jpg">JPGoption>
        <option value="gif">GIFoption>
        <option value="webp">WEBPoption>
      select>
    div>

    
    <div class="col-md-6">
      <label for="viewport_width" class="form-label">Viewport Widthlabel>
      <input
        type="number"
        name="viewport_width"
        id="viewport_width"
        class="form-control"
        value="1280"
        min="320"
      />
    div>

    
    <div class="col-md-6">
      <label for="viewport_height" class="form-label">Viewport Heightlabel>
      <input
        type="number"
        name="viewport_height"
        id="viewport_height"
        class="form-control"
        value="720"
        min="320"
      />
    div>
  div>
form>

The updated form gives users control over how the screenshot is generated. The format dropdown lets them choose between PNG, JPG, GIF, or WEBP. The Full Page checkbox toggles whether the API captures the entire scrollable webpage or just the visible viewport. The viewport width and height fields define the browser window dimensions, which is useful if you want to simulate different device sizes or responsive layouts.

When the form is submitted, Flask reads these values and sends them as query parameters in the API request.

For example, a request to capture a full-page, 1920×1080 PNG screenshot of your site would look like this:

https://api.screenshotbase.com/v1/take?url=https%3A%2F%2Fgithub.com%2Fashutoshkrris&format=png&full_page=1&viewport_width=1920&viewport_height=1080

This flexibility makes it easy to fine-tune screenshots for different use cases – whether you’re generating thumbnails, testing responsiveness, or automating visual reports.

Note: If you prefer working with SDKs instead of direct API calls, ScreenshotBase also offers official SDKs for popular languages like Python, JavaScript, Ruby, PHP, and Go.

These SDKs provide a simpler and more convenient way to interact with the API, handling authentication and request formatting behind the scenes.

You can explore them in the ScreenshotBase SDK documentation.

Wrapping Up

In this tutorial, you learned how to build a simple Flask application that captures website screenshots using a third-party API. We explored how to send requests, handle image responses, and add customization options like image format, full-page capture, and viewport size – all while keeping the project lightweight and easy to extend.

This small project demonstrates how web automation tasks, such as generating previews or visual reports, can be simplified using modern APIs. You can build upon this foundation to create batch screenshot tools, visual monitoring systems, or even integrate screenshots into larger web applications.

The key takeaway is understanding how to interact with external APIs, process responses, and design a clean interface for users. These are skills that are essential for backend and full-stack developers alike.

How to Work with SQLite in Python – A Handbook for Beginners

Ashutosh Krishna — Wed, 02 Oct 2024 09:44:37 +0000

SQLite is one of the most popular relational database management systems (RDBMS). It’s lightweight, meaning that it doesn’t take up much space on your system. One of its best features is that it’s serverless, so you don’t need to install or manage a separate server to use it.

Instead, it stores everything in a simple file on your computer. It also requires zero configuration, so there’s no complicated setup process, making it perfect for beginners and small projects.

SQLite is a great choice for small to medium applications because it’s easy to use, fast, and can handle most tasks that bigger databases can do, but without the hassle of managing extra software. Whether you're building a personal project or prototyping a new app, SQLite is a solid option to get things up and running quickly.

In this tutorial, you'll learn how to work with SQLite using Python. Here’s what we’re going to cover in this tutorial:

How to Set Up Your Python Environment
How to Create an SQLite Database
How to Create Database Tables
How to Insert Data into a Table
How to Query Data
How to Update and Delete Data
How to Use Transactions
How to Optimize SQLite Query Performance with Indexing
How to Handle Errors and Exceptions
How to Export and Import Data [Bonus Section]
Wrapping Up

This tutorial is perfect for anyone who wants to get started with databases without diving into complex setups.

How to Set Up Your Python Environment

Before working with SQLite, let’s ensure your Python environment is ready. Here’s how to set everything up.

Installing Python

If you don’t have Python installed on your system yet, you can download it from the official Python website. Follow the installation instructions for your operating system (Windows, macOS, or Linux).

To check if Python is installed, open your terminal (or command prompt) and type:

python --version

This should show the current version of Python installed. If it’s not installed, follow the instructions on the Python website.

Installing SQLite3 Module

The good news is that SQLite3 comes built-in with Python! You don’t need to install it separately because it’s included in the standard Python library. This means you can start using it right away without any additional setup.

How to Create a Virtual Environment (Optional but Recommended)

It’s a good idea to create a virtual environment for each project to keep your dependencies organized. A virtual environment is like a clean slate where you can install packages without affecting your global Python installation.

To create a virtual environment, follow these steps:

First, open your terminal or command prompt and navigate to the directory where you want to create your project.
Run the following command to create a virtual environment:

python -m venv env

Here, env is the name of the virtual environment. You can name it anything you like.

Activate the virtual environment:

# Use the command for Windows
env\Scripts\activate

# Use the command for macOS/Linux:
env/bin/activate

After activating the virtual environment, you’ll notice that your terminal prompt changes, showing the name of the virtual environment. This means you’re now working inside it.

Installing Necessary Libraries

We’ll need a few additional libraries for this project. Specifically, we’ll use:

pandas: This is an optional library for handling and displaying data in tabular format, useful for advanced use cases.
faker: This library will help us generate fake data, like random names and addresses, which we can insert into our database for testing.

To install pandas and faker, simply run the following commands:

pip install pandas faker

This installs both pandas and faker into your virtual environment. With this, your environment is set up, and you’re ready to start creating and managing your SQLite database in Python!

How to Create an SQLite Database

A database is a structured way to store and manage data so that it can be easily accessed, updated, and organized. It’s like a digital filing system that allows you to efficiently store large amounts of data, whether it’s for a simple app or a more complex system. Databases use tables to organize data, with rows and columns representing individual records and their attributes.

How SQLite Databases Work

Unlike most other database systems, SQLite is a serverless database. This means that it doesn’t require setting up or managing a server, making it lightweight and easy to use. All the data is stored in a single file on your computer, which you can easily move, share, or back up. Despite its simplicity, SQLite is powerful enough to handle many common database tasks and is widely used in mobile apps, embedded systems, and small to medium-sized projects.

How to Create a New SQLite Database

Let’s create a new SQLite database and learn how to interact with it using Python’s sqlite3 library.

Connecting to the Database

Since sqlite3 is pre-installed, you just need to import it in your Python script. To create a new database or connect to an existing one, we use the sqlite3.connect() method. This method takes the name of the database file as an argument. If the file doesn’t exist, SQLite will automatically create it.

import sqlite3

# Connect to the SQLite database (or create it if it doesn't exist)
connection = sqlite3.connect('my_database.db')

In this example, a file named my_database.db is created in the same directory as your script. If the file already exists, SQLite will just open the connection to it.

Creating a Cursor

Once you have a connection, the next step is to create a cursor object. The cursor is responsible for executing SQL commands and queries on the database.

# Create a cursor object
cursor = connection.cursor()

Closing the Connection

After you’ve finished working with the database, it’s important to close the connection to free up any resources. You can close the connection with the following command:

# Close the database connection
connection.close()

However, you should only close the connection once you’re done with all your operations.

When you run your Python script, a file named my_database.db will be created in your current working directory. You’ve now successfully created your first SQLite database!

How to Use Context Manager to Open and Close Connections

Python provides a more efficient and cleaner way to handle database connections using the with statement, also known as a context manager. The with statement automatically opens and closes the connection, ensuring that the connection is properly closed even if an error occurs during the database operations. This eliminates the need to manually call connection.close().

Here’s how you can use the with statement to handle database connections:

import sqlite3

# Step 1: Use 'with' to connect to the database (or create one) and automatically close it when done
with sqlite3.connect('my_database.db') as connection:

    # Step 2: Create a cursor object to interact with the database
    cursor = connection.cursor()

    print("Database created and connected successfully!")

# No need to call connection.close(); it's done automatically!

From now on, we’ll use the with statement in our upcoming code examples to manage database connections efficiently. This will make the code more concise and easier to maintain.

How to Create Database Tables

Now that we’ve created an SQLite database and connected to it, the next step is to create tables inside the database. A table is where we’ll store our data, organized in rows (records) and columns (attributes). For this example, we’ll create a table called Students to store information about students, which we’ll reuse in upcoming sections.

To create a table, we use SQL's CREATE TABLE statement. This command defines the table structure, including the column names and the data types for each column.

Here’s a simple SQL command to create a Students table with the following fields:

id: A unique identifier for each student (an integer).
name: The student's name (text).
age: The student's age (an integer).
email: The student's email address (text).

The SQL command to create this table would look like this:

CREATE TABLE Students (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    age INTEGER,
    email TEXT
);

We can execute this CREATE TABLE SQL command in Python using the sqlite3 library. Let’s see how to do that.

import sqlite3

# Use 'with' to connect to the SQLite database and automatically close the connection when done
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to create the Students table
    create_table_query = '''
    CREATE TABLE IF NOT EXISTS Students (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT NOT NULL,
        age INTEGER,
        email TEXT
    );
    '''

    # Execute the SQL command
    cursor.execute(create_table_query)

    # Commit the changes
    connection.commit()

    # Print a confirmation message
    print("Table 'Students' created successfully!")

IF NOT EXISTS: This ensures that the table is only created if it doesn’t already exist, preventing errors if the table has been created before.
connection.commit(): This saves (commits) the changes to the database.

When you run the Python code above, it will create the Students table in the my_database.db database file. You’ll also see a message in the terminal confirming that the table has been created successfully.

If you’re using Visual Studio Code, you can install the SQLite Viewer extension to view SQLite databases.

Data Types in SQLite and Their Mapping to Python

SQLite supports several data types, which we need to understand when defining our tables. Here’s a quick overview of common SQLite data types and how they map to Python types:

SQLite Data Type	Description	Python Equivalent
INTEGER	Whole numbers	`int`
TEXT	Text strings	`str`
REAL	Floating-point numbers	`float`
BLOB	Binary data (e.g., images, files)	`bytes`
NULL	Represents no value or missing data	`None`

In our Students table:

id is of type INTEGER, which maps to Python’s int.
name and email are of type TEXT, which map to Python’s str.
age is also of type INTEGER, mapping to Python’s int.

How to Insert Data into a Table

Now that we have our Students table created, it’s time to start inserting data into the database. In this section, we’ll cover how to insert both single and multiple records using Python and SQLite, and how to avoid common security issues like SQL injection by using parameterized queries.

How to Insert a Single Record

To insert data into the database, we use the INSERT INTO SQL command. Let’s start by inserting a single record into our Students table.

Here’s the basic SQL syntax for inserting a single record:

INSERT INTO Students (name, age, email) 
VALUES ('John Doe', 20, 'johndoe@example.com');

However, instead of writing SQL directly in our Python script with hardcoded values, we’ll use parameterized queries to make our code more secure and flexible. Parameterized queries help prevent SQL injection, a common attack where malicious users can manipulate the SQL query by passing harmful input.

Here’s how we can insert a single record into the Students table using a parameterized query:

import sqlite3

# Use 'with' to open and close the connection automatically
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # Insert a record into the Students table
    insert_query = '''
    INSERT INTO Students (name, age, email) 
    VALUES (?, ?, ?);
    '''
    student_data = ('Jane Doe', 23, 'jane@example.com')

    cursor.execute(insert_query, student_data)

    # Commit the changes automatically
    connection.commit()

    # No need to call connection.close(); it's done automatically!
    print("Record inserted successfully!")

The ? placeholders represent the values to be inserted into the table. The actual values are passed as a tuple (student_data) in the cursor.execute() method.

How to Insert Multiple Records

If you want to insert multiple records at once, you can use the executemany() method in Python. This method takes a list of tuples, where each tuple represents one record.

To make our example more dynamic, we can use the Faker library to generate random student data. This is useful for testing and simulating real-world scenarios.

from faker import Faker
import sqlite3

# Initialize Faker
fake = Faker(['en_IN'])

# Use 'with' to open and close the connection automatically
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # Insert a record into the Students table
    insert_query = '''
    INSERT INTO Students (name, age, email) 
    VALUES (?, ?, ?);
    '''
    students_data = [(fake.name(), fake.random_int(
        min=18, max=25), fake.email()) for _ in range(5)]

    # Execute the query for multiple records
    cursor.executemany(insert_query, students_data)

    # Commit the changes
    connection.commit()

    # Print confirmation message
    print("Fake student records inserted successfully!")

In this code:

Faker() generates random names, ages, and emails for students. Passing the locale([‘en_IN’]) is optional.
cursor.executemany(): This method allows us to insert multiple records at once, making the code more efficient.
students_data: A list of tuples where each tuple represents one student’s data.

How to Handle Common Issues: SQL Injection

SQL injection is a security vulnerability where attackers can insert or manipulate SQL queries by providing harmful input. For example, an attacker might try to inject code like '; DROP TABLE Students; -- to delete the table.

By using parameterized queries (as demonstrated above), we avoid this issue. The ? placeholders in parameterized queries ensure that input values are treated as data, not as part of the SQL command. This makes it impossible for malicious code to be executed.

How to Query Data

Now that we’ve inserted some data into our Students table, let’s learn how to retrieve the data from the table. We'll explore different methods for fetching data in Python, including fetchone(), fetchall(), and fetchmany().

To query data from a table, we use the SELECT statement. Here’s a simple SQL command to select all columns from the Students table:

SELECT * FROM Students;

This command retrieves all records and columns from the Students table. We can execute this SELECT query in Python and fetch the results.

How to Fetch All Records

Here’s how we can fetch all records from the Students table:

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Execute the SQL command
    cursor.execute(select_query)

    # Fetch all records
    all_students = cursor.fetchall()

    # Display results in the terminal
    print("All Students:")
    for student in all_students:
        print(student)

In this example, the fetchall() method retrieves all rows returned by the query as a list of tuples.

All Students:
(1, 'Jane Doe', 23, 'jane@example.com')
(2, 'Bahadurjit Sabharwal', 18, 'tristanupadhyay@example.net')
(3, 'Zayyan Arya', 20, 'yashawinibhakta@example.org')
(4, 'Hemani Shukla', 18, 'gaurikanarula@example.com')
(5, 'Warda Kara', 20, 'npatil@example.net')
(6, 'Mitali Nazareth', 19, 'sparekh@example.org')

How to Fetch a Single Record

If you want to retrieve only one record, you can use the fetchone() method:

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Execute the SQL command
    cursor.execute(select_query)

    # Fetch one record
    student = cursor.fetchone()

    # Display the result
    print("First Student:")
    print(student)

Output:

First Student:
(1, 'Jane Doe', 23, 'jane@example.com')

How to Fetch Multiple Records

To fetch a specific number of records, you can use fetchmany(size):

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Execute the SQL command
    cursor.execute(select_query)

    # Fetch three records
    three_students = cursor.fetchmany(3)

    # Display results
    print("Three Students:")
    for student in three_students:
        print(student)

Output:

Three Students:
(1, 'Jane Doe', 23, 'jane@example.com')
(2, 'Bahadurjit Sabharwal', 18, 'tristanupadhyay@example.net')
(3, 'Zayyan Arya', 20, 'yashawinibhakta@example.org')

How to Use `pandas` for Better Data Presentation

For better data presentation, we can use the pandas library to create a DataFrame from our query results. This makes it easier to manipulate and visualize the data.

Here’s how to fetch all records and display them as a pandas DataFrame:

import sqlite3
import pandas as pd

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:
    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Use pandas to read SQL query directly into a DataFrame
    df = pd.read_sql_query(select_query, connection)

# Display the DataFrame
print("All Students as DataFrame:")
print(df)

Output:

All Students as DataFrame:
   id                  name  age                        email
0   1              Jane Doe   23             jane@example.com
1   2  Bahadurjit Sabharwal   18  tristanupadhyay@example.net
2   3           Zayyan Arya   20  yashawinibhakta@example.org
3   4         Hemani Shukla   18    gaurikanarula@example.com
4   5            Warda Kara   20           npatil@example.net
5   6       Mitali Nazareth   19          sparekh@example.org

The pd.read_sql_query() function executes the SQL query and directly returns the results as a pandas DataFrame.

How to Update and Delete Data

In this section, we’ll learn how to update existing records and delete records from our Students table using SQL commands in Python. This is essential for managing and maintaining your data effectively.

Updating Existing Records

To modify existing records in a database, we use the SQL UPDATE command. This command allows us to change the values of specific columns in one or more rows based on a specified condition.

For example, if we want to update a student's age, the SQL command would look like this:

UPDATE Students 
SET age = 21 
WHERE name = 'Jane Doe';

Now, let’s write Python code to update a specific student's age in our Students table.

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # SQL command to update a student's age
    update_query = '''
    UPDATE Students 
    SET age = ? 
    WHERE name = ?;
    '''

    # Data for the update
    new_age = 21
    student_name = 'Jane Doe'

    # Execute the SQL command with the data
    cursor.execute(update_query, (new_age, student_name))

    # Commit the changes to save the update
    connection.commit()

    # Print a confirmation message
    print(f"Updated age for {student_name} to {new_age}.")

In this example, we used parameterized queries to prevent SQL injection.

How to Delete Records from the Table

To remove records from a database, we use the SQL DELETE command. This command allows us to delete one or more rows based on a specified condition.

For example, if we want to delete a student named 'Jane Doe', the SQL command would look like this:

DELETE FROM Students 
WHERE name = 'Jane Doe';

Let’s write Python code to delete a specific student from our Students table using the with statement.

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # SQL command to delete a student
    delete_query = '''
    DELETE FROM Students 
    WHERE name = ?;
    '''

    # Name of the student to be deleted
    student_name = 'Jane Doe'

    # Execute the SQL command with the data
    cursor.execute(delete_query, (student_name,))

    # Commit the changes to save the deletion
    connection.commit()

    # Print a confirmation message
    print(f"Deleted student record for {student_name}.")

Important Considerations

Conditions: Always use the WHERE clause when updating or deleting records to avoid modifying or removing all rows in the table. Without a WHERE clause, the command affects every row in the table.
Backup: It’s good practice to back up your database before performing updates or deletions, especially in production environments.

How to Use Transactions

A transaction is a sequence of one or more SQL operations that are treated as a single unit of work. In the context of a database, a transaction allows you to perform multiple operations that either all succeed or none at all. This ensures that your database remains in a consistent state, even in the face of errors or unexpected issues.

For example, if you are transferring money between two bank accounts, you would want both the debit from one account and the credit to the other to succeed or fail together. If one operation fails, the other should not be executed to maintain consistency.

Why Use Transactions?

Atomicity: Transactions ensure that a series of operations are treated as a single unit. If one operation fails, none of the operations will be applied to the database.
Consistency: Transactions help maintain the integrity of the database by ensuring that all rules and constraints are followed.
Isolation: Each transaction operates independently of others, preventing unintended interference.
Durability: Once a transaction is committed, the changes are permanent, even in the event of a system failure.

When to Use Transactions?

You should use transactions when:

Performing multiple related operations that must succeed or fail together.
Modifying critical data that requires consistency and integrity.
Working with operations that can potentially fail, such as financial transactions or data migrations.

How to Manage Transactions in Python

In SQLite, transactions are managed using the BEGIN, COMMIT, and ROLLBACK commands. However, when using the sqlite3 module in Python, you typically manage transactions through the connection object.

Starting a Transaction

A transaction begins implicitly when you execute any SQL statement. To start a transaction explicitly, you can use the BEGIN command:

cursor.execute("BEGIN;")

However, it’s usually unnecessary to start a transaction manually, as SQLite starts a transaction automatically when you execute an SQL statement.

How to Commit a Transaction

To save all changes made during a transaction, you use the commit() method. This makes all modifications permanent in the database.

connection.commit()

We have already used the commit() method in the above provided examples.

Rolling Back a Transaction

If something goes wrong and you want to revert the changes made during a transaction, you can use the rollback() method. This will undo all changes made since the transaction started.

connection.rollback()

Example of Using Transactions in Python

To illustrate the use of transactions in a real-world scenario, we’ll create a new table called Customers to manage customer accounts. In this example, we’ll assume each customer has a balance. We will add two customers to this table and perform a funds transfer operation between them.

First, let's create the Customers table and insert two customers:

import sqlite3

# Create the Customers table and add two customers
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # Create Customers table
    create_customers_table = '''
    CREATE TABLE IF NOT EXISTS Customers (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT NOT NULL UNIQUE,
        balance REAL NOT NULL
    );
    '''
    cursor.execute(create_customers_table)

    # Insert two customers
    cursor.execute(
        "INSERT INTO Customers (name, balance) VALUES (?, ?);", ('Ashutosh', 100.0))
    cursor.execute(
        "INSERT INTO Customers (name, balance) VALUES (?, ?);", ('Krishna', 50.0))

    connection.commit()

Now, let’s perform the funds transfer operation between Ashutosh and Krishna:

import sqlite3


def transfer_funds(from_customer, to_customer, amount):
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        try:
            # Start a transaction
            cursor.execute("BEGIN;")

            # Deduct amount from the sender
            cursor.execute(
                "UPDATE Customers SET balance = balance - ? WHERE name = ?;", (amount, from_customer))
            # Add amount to the receiver
            cursor.execute(
                "UPDATE Customers SET balance = balance + ? WHERE name = ?;", (amount, to_customer))

            # Commit the changes
            connection.commit()
            print(
                f"Transferred {amount} from {from_customer} to {to_customer}.")

        except Exception as e:
            # If an error occurs, rollback the transaction
            connection.rollback()
            print(f"Transaction failed: {e}")


# Example usage
transfer_funds('Ashutosh', 'Krishna', 80.0)

In this example, we first created a Customers table and inserted two customers, Ashutosh with a balance of ₹100, and Krishna with a balance of ₹50. We then performed a funds transfer of ₹80 from Ashutosh to Krishna. By using transactions, we ensure that both the debit from Ashutosh's account and the credit to Krishna's account are executed as a single atomic operation, maintaining data integrity in the event of any errors. If the transfer fails (for example, due to insufficient funds), the transaction will roll back, leaving both accounts unchanged.

How to Optimize SQLite Query Performance with Indexing

Indexing is a powerful technique used in databases to improve query performance. An index is essentially a data structure that stores the location of rows based on specific column values, much like an index at the back of a book helps you quickly locate a topic.

Without an index, SQLite has to scan the entire table row by row to find the relevant data, which becomes inefficient as the dataset grows. By using an index, SQLite can jump directly to the rows you need, significantly speeding up query execution.

How to Populate the Database with Fake Data

To effectively test the impact of indexing, we need a sizable dataset. Instead of manually adding records, we can use the faker library to quickly generate fake data. In this section, we’ll generate 10,000 fake records and insert them into our Students table. This will simulate a real-world scenario where databases grow large, and query performance becomes important.

We will use the executemany() method to insert the records as below:

import sqlite3
from faker import Faker

# Initialize the Faker library
fake = Faker(['en_IN'])


def insert_fake_students(num_records):
    """Generate and insert fake student data into the Students table."""
    fake_data = [(fake.name(), fake.random_int(min=18, max=25),
                  fake.email()) for _ in range(num_records)]

    # Use 'with' to handle the database connection
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Insert fake data into the Students table
        cursor.executemany('''
        INSERT INTO Students (name, age, email) 
        VALUES (?, ?, ?);
        ''', fake_data)

        connection.commit()

    print(f"{num_records} fake student records inserted successfully.")


# Insert 10,000 fake records into the Students table
insert_fake_students(10000)

By running this script, 10,000 fake student records will be added to the Students table. In the next section, we'll query the database and compare the performance of queries with and without indexing.

How to Query Without Indexes

In this section, we’ll query the Students table without any indexes to observe how SQLite performs when there are no optimizations in place. This will serve as a baseline to compare the performance when we add indexes later.

Without indexes, SQLite performs a full table scan, which means that it must check every row in the table to find matching results. For small datasets, this is manageable, but as the number of records grows, the time taken to search increases dramatically. Let’s see this in action by running a basic SELECT query to search for a specific student by name and measure how long it takes.

First, we’ll query the Students table by looking for a student with a specific name. We’ll log the time taken to execute the query using Python’s time module to measure the performance.

import sqlite3
import time


def query_without_index(search_name):
    """Query the Students table by name without an index and measure the time taken."""

    # Connect to the database using 'with'
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Measure the start time
        start_time = time.perf_counter_ns()

        # Perform a SELECT query to find a student by name
        cursor.execute('''
        SELECT * FROM Students WHERE name = ?;
        ''', (search_name,))

        # Fetch all results (there should be only one or a few in practice)
        results = cursor.fetchall()

        # Measure the end time
        end_time = time.perf_counter_ns()

        # Calculate the total time taken
        elapsed_time = (end_time - start_time) / 1000

        # Display the results and the time taken
        print(f"Query completed in {elapsed_time:.5f} microseconds.")
        print("Results:", results)


# Example: Searching for a student by name
query_without_index('Ojasvi Dhawan')

Here’s the output:

Query completed in 1578.10000 microseconds.
Results: [(104, 'Ojasvi Dhawan', 21, 'lavanya26@example.com')]

By running the above script, you'll see how long it takes to search the Students table without any indexes. For example, if there are 10,000 records in the table, the query might take 1000-2000 microseconds depending on the size of the table and your hardware. This may not seem too slow for a small dataset, but the performance will degrade as more records are added.

We use time.perf_counter_ns() to measure the time taken for the query execution in nanoseconds. This method is highly accurate for benchmarking small time intervals. We convert the time to microseconds(us) for easier readability.

Introducing the Query Plan

When working with databases, understanding how queries are executed can help you identify performance bottlenecks and optimize your code. SQLite provides a helpful tool for this called EXPLAIN QUERY PLAN, which allows you to analyze the steps SQLite takes to retrieve data.

In this section, we’ll introduce how to use EXPLAIN QUERY PLAN to visualize and understand the inner workings of a query—specifically, how SQLite performs a full table scan when no index is present.

Let’s use EXPLAIN QUERY PLAN to see how SQLite retrieves data from the Students table without any indexes. We’ll search for a student by name, and the query plan will reveal the steps SQLite takes to find the matching rows.

import sqlite3


def explain_query(search_name):
    """Explain the query execution plan for a SELECT query without an index."""

    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Use EXPLAIN QUERY PLAN to analyze how the query is executed
        cursor.execute('''
        EXPLAIN QUERY PLAN
        SELECT * FROM Students WHERE name = ?;
        ''', (search_name,))

        # Fetch and display the query plan
        query_plan = cursor.fetchall()

        print("Query Plan:")
        for step in query_plan:
            print(step)


# Example: Analyzing the query plan for searching by name
explain_query('Ojasvi Dhawan')

When you run this code, SQLite will return a breakdown of how it plans to execute the query. Here’s an example of what the output might look like:

Query Plan:
(2, 0, 0, 'SCAN Students')

This indicates that SQLite is scanning the entire Students table (a full table scan) to find the rows where the name column matches the provided value (Ojasvi Dhawan). Since there is no index on the name column, SQLite must examine each row in the table.

How to Create an Index

Creating an index on a column allows SQLite to find rows more quickly during query operations. Instead of scanning the entire table, SQLite can use the index to jump directly to the relevant rows, significantly speeding up queries—especially those involving large datasets.

To create an index, use the following SQL command:

CREATE INDEX IF NOT EXISTS index-name ON table (column(s));

In this example, we will create an index on the name column of the Students table. Here’s how you can do it using Python:

import sqlite3
import time


def create_index():
    """Create an index on the name column of the Students table."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # SQL command to create an index on the name column
        create_index_query = '''
        CREATE INDEX IF NOT EXISTS idx_name ON Students (name);
        '''

        # Measure the start time
        start_time = time.perf_counter_ns()

        # Execute the SQL command to create the index
        cursor.execute(create_index_query)

        # Measure the start time
        end_time = time.perf_counter_ns()

        # Commit the changes
        connection.commit()

        print("Index on 'name' column created successfully!")

        # Calculate the total time taken
        elapsed_time = (end_time - start_time) / 1000

        # Display the results and the time taken
        print(f"Query completed in {elapsed_time:.5f} microseconds.")


# Call the function to create the index
create_index()

Output:

Index on 'name' column created successfully!
Query completed in 102768.60000 microseconds.

Even though creating the index takes this long (102768.6 microseconds), it's a one-time operation. You will still get substantial speed-up when running multiple queries. In the following sections, we will query the database again to observe the performance improvements made possible by this index.

How to Query with Indexes

In this section, we will perform the same SELECT query we executed earlier, but this time we will take advantage of the index we created on the name column of the Students table. We'll measure and log the execution time to observe the performance improvements provided by the index.

import sqlite3
import time


def query_with_index(student_name):
    """Query the Students table using an index on the name column."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # SQL command to select a student by name
        select_query = 'SELECT * FROM Students WHERE name = ?;'

        # Measure the execution time
        start_time = time.perf_counter_ns()  # Start the timer

        # Execute the query with the provided student name
        cursor.execute(select_query, (student_name,))
        result = cursor.fetchall()  # Fetch all results

        end_time = time.perf_counter_ns()  # End the timer

        # Calculate the elapsed time in microseconds
        execution_time = (end_time - start_time) / 1000

        # Display results and execution time
        print(f"Query result: {result}")
        print(f"Execution time with index: {execution_time:.5f} microseconds")


# Example: Searching for a student by name
query_with_index('Ojasvi Dhawan')

Here’s what we get in the output:

Query result: [(104, 'Ojasvi Dhawan', 21, 'lavanya26@example.com')]
Execution time with index: 390.70000 microseconds

We can observe a significant reduction in execution time compared to when the query was performed without an index.

Let’s analyze the query execution plan for the query with the index on the name column of the Students table. If you execute the same script again to explain the query, you’ll get the below output:

Query Plan:
(3, 0, 0, 'SEARCH Students USING INDEX idx_name (name=?)')

The plan now shows that the query uses the index idx_name, significantly reducing the number of rows that need to be scanned, which leads to faster query execution.

Comparing Performance Results

Now, let's summarize the performance results we obtained when querying with and without indexes.

Execution Time Comparison

Query Type	Execution Time (microseconds)
Without Index	1578.1
With Index	390.7

Performance Improvement Summary

The query with the index is approximately 4.04 times faster than the query without the index.
The execution time improved by about 75.24% after adding the index.

Best Practices for Using Indexes

Indexes can significantly enhance the performance of your SQLite database, but they should be used judiciously. Here are some best practices to consider when working with indexes:

When and Why to Use Indexes

Frequent Query Columns: Use indexes on columns that are frequently used in SELECT queries, especially those used in WHERE, JOIN, and ORDER BY clauses. This is because indexing these columns can drastically reduce query execution time.
Uniqueness Constraints: When you have columns that must hold unique values (like usernames or email addresses), creating an index can enforce this constraint efficiently.
Large Datasets: For tables with a large number of records, indexes become increasingly beneficial. They enable quick lookups, which is essential for maintaining performance as your data grows.
Composite Indexes: Consider creating composite indexes for queries that filter or sort by multiple columns. For example, if you often search for students by both name and age, an index on both columns can optimize such queries.

Potential Downsides of Indexes

While indexes provide significant advantages, there are some potential downsides:

Slower Insert/Update Operations: When you insert or update records in a table with indexes, SQLite must also update the index, which can slow down these operations. This is because each insert or update requires additional overhead to maintain the index structure.
Increased Storage Requirements: Indexes consume additional disk space. For large tables, the storage cost can be substantial. Consider this when designing your database schema, especially for systems with limited storage resources.
Complex Index Management: Having too many indexes can complicate database management. It may lead to situations where you have redundant indexes, which can degrade performance rather than enhance it. Regularly reviewing and optimizing your indexes is a good practice.

Indexes are powerful tools for optimizing database queries, but they require careful consideration. Striking a balance between improved read performance and the potential overhead on write operations is key. Here are some strategies for achieving this balance:

Monitor Query Performance: Use SQLite’s EXPLAIN QUERY PLAN to analyze how your queries perform with and without indexes. This can help identify which indexes are beneficial and which may be unnecessary.
Regular Maintenance: Periodically review your indexes and assess whether they are still needed. Remove redundant or rarely used indexes to streamline your database operations.
Test and Evaluate: Before implementing indexes in a production environment, conduct thorough testing to understand their impact on both read and write operations.

By following these best practices, you can leverage the benefits of indexing while minimizing potential drawbacks, ultimately enhancing the performance and efficiency of your SQLite database.

How to Handle Errors and Exceptions

In this section, we’ll discuss how to handle errors and exceptions when working with SQLite in Python. Proper error handling is crucial for maintaining the integrity of your database and ensuring that your application behaves predictably.

Common Errors in SQLite Operations

When interacting with an SQLite database, several common errors may arise:

Constraint Violations: This occurs when you try to insert or update data that violates a database constraint, such as primary key uniqueness or foreign key constraints. For example, trying to insert a duplicate primary key will trigger an error.
Data Type Mismatches: Attempting to insert data of the wrong type (for example, inserting a string where a number is expected) can lead to an error.
Database Locked Errors: If a database is being written to by another process or connection, trying to access it can result in a "database is locked" error.
Syntax Errors: Mistakes in your SQL syntax will result in errors when you try to execute your commands.

How to Use Python's Exception Handling

Python’s built-in exception handling mechanisms (try and except) are essential for managing errors in SQLite operations. By using these constructs, you can catch exceptions and respond appropriately without crashing your program.

Here’s a basic example of how to handle errors when inserting data into the database:

import sqlite3


def add_customer_with_error_handling(name, balance):
    """Add a new customer with error handling."""
    try:
        with sqlite3.connect('my_database.db') as connection:
            cursor = connection.cursor()
            cursor.execute(
                "INSERT INTO Customers (name, balance) VALUES (?, ?);", (name, balance))
            connection.commit()
            print(f"Added customer: {name} with balance: {balance}")

    except sqlite3.IntegrityError as e:
        print(f"Error: Integrity constraint violated - {e}")

    except sqlite3.OperationalError as e:
        print(f"Error: Operational issue - {e}")

    except Exception as e:
        print(f"An unexpected error occurred: {e}")


# Example usage
add_customer_with_error_handling('Vishakha', 100.0)  # Valid
add_customer_with_error_handling('Vishakha', 150.0)  # Duplicate entry

In this example:

We catch IntegrityError, which is raised for violations like unique constraints.
We catch OperationalError for general database-related issues (like database locked errors).
We also have a generic except block to handle any unexpected exceptions.

Output:

Added customer: Vishakha with balance: 100.0
Error: Integrity constraint violated - UNIQUE constraint failed: Customers.name

Best Practices for Ensuring Database Integrity

Use Transactions: Always use transactions (as discussed in the previous section) when performing multiple related operations. This helps ensure that either all operations succeed or none do, maintaining consistency.
Validate Input Data: Before executing SQL commands, validate the input data to ensure it meets the expected criteria (for example, correct types, within allowable ranges).
Catch Specific Exceptions: Always catch specific exceptions to handle different types of errors appropriately. This allows for clearer error handling and debugging.
Log Errors: Instead of just printing errors to the console, consider logging them to a file or monitoring system. This will help you track issues in production.
Graceful Degradation: Design your application to handle errors gracefully. If an operation fails, provide meaningful feedback to the user rather than crashing the application.
Regularly Backup Data: Regularly back up your database to prevent data loss in case of critical failures or corruption.
Use Prepared Statements: Prepared statements help prevent SQL injection attacks and can also provide better performance for repeated queries.

How to Export and Import Data [Bonus Section]

In this section, we will learn how to export data from an SQLite database to common formats like CSV and JSON, as well as how to import data into SQLite from these formats using Python. This is useful for data sharing, backup, and integration with other applications.

Exporting Data from SQLite to CSV

Exporting data to a CSV (Comma-Separated Values) file is straightforward with Python’s built-in libraries. CSV files are widely used for data storage and exchange, making them a convenient format for exporting data.

Here’s how to export data from an SQLite table to a CSV file:

import sqlite3
import csv

def export_to_csv(file_name):
    """Export data from the Customers table to a CSV file."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Execute a query to fetch all customer data
        cursor.execute("SELECT * FROM Customers;")
        customers = cursor.fetchall()

        # Write data to CSV
        with open(file_name, 'w', newline='') as csv_file:
            csv_writer = csv.writer(csv_file)
            csv_writer.writerow(['ID', 'Name', 'Balance'])  # Writing header
            csv_writer.writerows(customers)  # Writing data rows

        print(f"Data exported successfully to {file_name}.")

# Example usage
export_to_csv('customers.csv')

How to Export Data to JSON

Similarly, you can export data to a JSON (JavaScript Object Notation) file, which is a popular format for data interchange, especially in web applications.

Here’s an example of how to export data to JSON:

import json
import sqlite3


def export_to_json(file_name):
    """Export data from the Customers table to a JSON file."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Execute a query to fetch all customer data
        cursor.execute("SELECT * FROM Customers;")
        customers = cursor.fetchall()

        # Convert data to a list of dictionaries
        customers_list = [{'ID': customer[0], 'Name': customer[1],
                           'Balance': customer[2]} for customer in customers]

        # Write data to JSON
        with open(file_name, 'w') as json_file:
            json.dump(customers_list, json_file, indent=4)

        print(f"Data exported successfully to {file_name}.")


# Example usage
export_to_json('customers.json')

How to Import Data into SQLite from CSV

You can also import data from a CSV file into an SQLite database. This is useful for populating your database with existing datasets.

Here's how to import data from a CSV file:

import csv
import sqlite3


def import_from_csv(file_name):
    """Import data from a CSV file into the Customers table."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Open the CSV file for reading
        with open(file_name, 'r') as csv_file:
            csv_reader = csv.reader(csv_file)
            next(csv_reader)  # Skip the header row

            # Insert each row into the Customers table
            for row in csv_reader:
                cursor.execute(
                    "INSERT INTO Customers (name, balance) VALUES (?, ?);", (row[1], row[2]))

        connection.commit()
        print(f"Data imported successfully from {file_name}.")


# Example usage
import_from_csv('customer_data.csv')

How to Import Data into SQLite from JSON

Similarly, importing data from a JSON file is simple. You can read the JSON file and insert the data into your SQLite table.

Here's how to do it:

import json
import sqlite3


def import_from_json(file_name):
    """Import data from a JSON file into the Customers table."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Open the JSON file for reading
        with open(file_name, 'r') as json_file:
            customers_list = json.load(json_file)

            # Insert each customer into the Customers table
            for customer in customers_list:
                cursor.execute("INSERT INTO Customers (name, balance) VALUES (?, ?);", (customer['Name'], customer['Balance']))

        connection.commit()
        print(f"Data imported successfully from {file_name}.")


# Example usage
import_from_json('customer_data.json')

Wrapping Up

And that’s a wrap! This guide has introduced you to the fundamentals of working with SQLite in Python, covering everything from setting up your environment to querying and manipulating data, as well as exporting and importing information. I hope you found it helpful and that it has sparked your interest in using SQLite for your projects.

Now it's time to put your newfound knowledge into practice! I encourage you to create your project using SQLite and Python. Whether it’s a simple application for managing your library, a budgeting tool, or something unique, the possibilities are endless.

Once you’ve completed your project, share it on Twitter and tag me! I’d love to see what you’ve created and celebrate your accomplishments.

You can find all the code from this tutorial on GitHub. Thank you for following along, and happy coding!

Generate Table of Contents for your freeCodeCamp articles for free using the TOC Generator tool.

How to Read and Write Files with Node.js

Ashutosh Krishna — Mon, 19 Aug 2024 13:41:36 +0000

Node.js is a powerful JavaScript runtime environment that lets you run JS code outside the browser. And a fundamental part of many Node.js applications involves reading and writing files – whether that's text, JSON, HTML, or other file formats. So you should understand how to read and write files.

Files are the backbone of data storage. Node.js provides a powerful 'fs' (file system) module to interact with these files seamlessly. Let's assume I want to read a JSON file in Node.js. The fs module can help me with that.

In this tutorial, I'll explain the core functionalities of this module, explore various techniques to read different file types, and discover some best practices to make your file-handling operations smoother and more efficient.

Throughout this tutorial, we'll cover everything from importing the package to using it to work with files asynchronously. Let's get started on this journey of learning file operations with Node.js!

Node.js fs Module
Prerequisites
How to Read And Write Files With Node.js
Ways to Read Files in Node.js
Wrapping Up

Node.js `fs` Module

The Node.js File System (fs) module is an essential component of the Node.js runtime environment. It provides a variety of features for interacting with your computer's file system.

The fs module allows you to read, write, update, delete, and manage files and directories. This module is especially useful for handling file-related operations in both synchronous and asynchronous modes.

Let’s break down the key aspects of the module:

The fs module, at its core, provides a collection of APIs for interacting with the file system. It provides ways to perform basic activities such as reading file contents, writing data to files, creating directories, deleting files, and so on.
The module includes both synchronous and asynchronous methods for interacting with files. The synchronous methods block the execution of the program until the operation completes. But the asynchronous methods are ideal for scenarios where we need to perform concurrent tasks without halting the execution of the entire program.
The module also supports handling directories, such as creating directories, removing directories, and listing directory contents.
The module also supports working with file streams, allowing efficient handling of large files by reading or writing data in chunks without loading the entire content into memory. It also facilitates the use of buffers for handling binary data, which helps with activities like data transformation and manipulation.

Prerequisites

To continue with the tutorial, I recommend having the following prerequisites:

Basic Understanding of JavaScript: It is essential to be familiar with JavaScript, as Node.js uses JavaScript.
Node.js Installed: Ensure that Node.js is installed on your system. You can download and install Node.js from its official website.
Text Editor/IDE: Have a text editor or an Integrated Development Environment (IDE) installed and ready to use.

How to Read And Write Files With Node.js

Let's look at an example to understand the process of reading and writing files in Node.js. We'll assume a scenario where we have two files – name.json and address.json.

The content inside the name.json looks like this:

[
  { "id": 1, "name": "Alice" },
  { "id": 2, "name": "Bob" },
  { "id": 3, "name": "Charlie" }
]

The content inside address.json looks like this:

[
  { "id": 1, "address": "123 Main St" },
  { "id": 2, "address": "456 Elm St" },
  { "id": 3, "address": "789 Oak St" }
]

Our objective is to create a bio.json file that merges the id, name, and address information, creating a structure as follows:

[
  {
    "id": 1,
    "name": "Alice",
    "address": "123 Main St"
  },
  {
    "id": 2,
    "name": "Bob",
    "address": "456 Elm St"
  },
  {
    "id": 3,
    "name": "Charlie",
    "address": "789 Oak St"
  }
]

Let's build the application!

Step 1: Import Node.js Packages `path` and `fs`

Let us start with creating an app.js file. The first thing we will do is import the fs library:

const fs = require("fs");

Step 2: Read From Files

Next, let's read data from the two files using Node.js. We'll make a utility function that helps us read these files easily in our Node.js environment.

async function readJSONFile(filename) {
  try {
    const data = await fs.readFile(filename, "utf8");
    return JSON.parse(data);
  } catch (error) {
    console.error(`Error reading ${filename}: ${error}`);
    return [];
  }
}

Since we are using JSON files in our example, we have defined a readJSONFile method in our code. It's an asynchronous JavaScript function that takes a filename as input and aims to return the parsed JSON contents of that file.

Inside a try block, we attempt to read the file using fs.readFile in Node with the specified filename and "utf8" encoding. If successful, the function then parses the file content as JSON using JSON.parse and returns it.

If any error occurs during reading or parsing, the catch block takes over. It logs the error with the filename and details and then returns an empty array instead of the expected JSON object.

Step 3: Implement the Main Function

The next step is to create a main function where we make use of the above-defined method and combine the data of the two files to create a bio.json file.

async function main() {
  try {
    const names = await readJSONFile("names.json");
    const addresses = await readJSONFile("address.json");

    const bioData = names.map((name) => {
      const matchingAddress = addresses.find(
        (address) => address.id === name.id
      );
      return { ...name, ...matchingAddress };
    });

    await fs.writeFile("bio.json", JSON.stringify(bioData, null, 2));
    console.log("bio.json created successfully!");
  } catch (error) {
    console.error("Error combining data:", error);
  }
}

In the main function, we first read the two JSON files named names.json and address.json using the readJSONFile function. Both readJSONFile calls use await, so the function waits for both files to be read before moving on.

Next, we iterate through each name using a map, creating a new bioData for each. Inside the loop, it searches for a matching address from the addresses collection based on the index using find.

The search compares name.id with each address.id until it finds a match. If a match is found, the function combines the information from both files. It uses the spread operator (...) to merge all properties from both objects into a single new bioData object. If no matching address is found, the bioData object will only have the name information.

Once all bioData objects are prepared, the function writes them as a new JSON file named bio.json using fs.writeFile. This writing process also uses await, ensuring the file is created before moving on.

The try block ensures smooth execution, while the catch block takes care of any errors like missing files or incorrect data. If any error occurs, a generic error message and the specific error details are logged for debugging.

Complete Code

Our completed code looks like below:

const fs = require("fs").promises;

async function readJSONFile(filename) {
  try {
    const data = await fs.readFile(filename, "utf8");
    return JSON.parse(data);
  } catch (error) {
    console.error(`Error reading ${filename}: ${error}`);
    return [];
  }
}

async function main() {
  try {
    const names = await readJSONFile("names.json");
    const addresses = await readJSONFile("address.json");

    const bioData = names.map((name) => {
      const matchingAddress = addresses.find(
        (address) => address.id === name.id
      );
      return { ...name, ...matchingAddress };
    });

    await fs.writeFile("bio.json", JSON.stringify(bioData, null, 2));
    console.log("bio.json created successfully!");
  } catch (error) {
    console.error("Error combining data:", error);
  }
}

// Execute the main method
main();

We can run the application using the following command:

node app.js

Once the application runs, we get to see the following logs in the terminal if everything goes well:

bio.json created successfully!

This indicates that the bio.json file was created successfully. The content inside the file should look like below:

[
  {
    "id": 1,
    "name": "Alice",
    "address": "123 Main St"
  },
  {
    "id": 2,
    "name": "Bob",
    "address": "456 Elm St"
  },
  {
    "id": 3,
    "name": "Charlie",
    "address": "789 Oak St"
  }
]

Ways to Read Files in Node.js

To read files in Node.js, we mostly use two primary methods: fs.readFile() and fs.readFileSync(). The difference lies in their synchronous and asynchronous nature.

`fs.readFile()` Method

The fs.readFile() method in Node.js is asynchronous. It reads the content of the entire file without blocking other operations. This makes it suitable for scenarios where non-blocking I/O operations are essential.

In simple terms, the function allows other operations to continue while the reading takes place. It accepts the following three parameters:

path: the path to the file to be read.
encoding: optional, specifies the encoding of the file (for example, "utf8"). Defaults to "utf8" if not provided.
callback function: optional, a function to be called when the file is read. The function receives three arguments: error, data, and buffer.

Let’s look at an example:

const fs = require("fs");

fs.readFile("data.txt", "utf8", (err, data) => {
  if (err) {
    console.error(err);
  } else {
    console.log(data); // data will be a string containing the content of the file
  }
});

`fs.readFileSync()` Method

The fs.readFileSync() method is synchronous. It reads the file content synchronously, stopping further execution until the file is completely read. This method is beneficial in scenarios where we require synchronous processing.

It takes only two parameters:

path: the path to the file to be read
encoding: optional, specifies the encoding of the file (for example, "utf8"). Defaults to "utf8" if not provided.

Let’s look at an example:

const fs = require("fs");

try {
  const data = fs.readFileSync("data.txt", "utf8");
  console.log(data); // data will be a string containing the content of the file
} catch (err) {
  console.error(err);
}

`fs.readFile()` vs `fs.readFileSync()`

To understand the difference between the Nodejs readFile and readFileSync methods, we will write two programs and monitor their execution flow.

Let's start with the fs.readFile() method.

const fs = require("fs");

fs.readFile("example.txt", "utf8", (err, data) => {
  console.log("Content from readFile:", data);
});

console.log("Completed reading file content asynchronously");

Output:

Completed reading file content asynchronously
Content from readFile: freeCodeCamp is awesome!

Because of fs.readFile()'s asynchronous nature, the code after fs.readFile() doesn't wait for the file reading operation to finish. So the "Completed reading file content synchronously" message is immediately logged to the console, showing that the subsequent code continues executing without waiting for the file read to complete.

Eventually, when the file reading operation finishes, the callback function executes and logs the file content.

Next, let's see the execution of the fs.readFileSync() method.

const fs = require("fs");

const data = fs.readFileSync("data.txt", "utf8");
console.log("Content from readFileSync:", data);

console.log("Completed reading file content synchronously");

Output:

Content from readFileSync: freeCodeCamp is awesome!
Completed reading file content synchronously

In contrast, fs.readFileSync() reads the data.txt file synchronously, blocking further code execution until the file is completely read. Consequently, the code continues execution only after the file reading operation is finished.

Because of this, the message "Completed reading file content synchronously" is logged after successfully reading the file content.

Now, we know the difference between the two methods. Understanding this difference is important as it impacts the program flow, especially in scenarios where timing and blocking operations are critical considerations in Node.js applications.

How to Read a Text File

It's pretty straightforward to read a text file in Node.js, and we have been doing this throughout the tutorial. Let’s consider that we have a file called message.txt with the following content:

Learn Node.js with freeCodeCamp

Now, we want to read the contents of this file. We can do it like this:

const fs = require("fs");

fs.readFile("message.txt", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    console.log(data);
  }
});

The callback function returns the content of the file in a data variable. Since we set the encoding to “utf8”, the value of data is a string. So we can perform string operations on the data variable.

const fs = require("fs");

fs.readFile("message.txt", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
      let splittedWords = data.split(" ");
      console.log(splittedWords);
  }
});

In the above code, we split the data variable using a space. So the splittedWords will be a string array containing the following value:

[ 'Learn', 'Node.js', 'with', 'freeCodeCamp' ]

How to Read HTML Files

Reading HTML files follows a similar approach to reading text files in Node.js. We can use the fs module to read HTML files:

const fs = require("fs");

fs.readFile("index.html", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    console.log(data);
  }
});

We can then use the HTML content for further processing, such as rendering it with the Node.js http package.

How to Read Files by URL

Reading files by URL in Node.js involves additional steps beyond the native fs module. Typically, we’ll need to use additional modules like http or axios to fetch file content from a URL.

const fs = require("fs");
const https = require("https");

const file = fs.createWriteStream("data.txt");

https.get(
  "https://example-files.online-convert.com/document/txt/example.txt",
  (response) => {
    var stream = response.pipe(file);

    stream.on("finish", function () {
      console.log("done");
    });
  }
);

First, we set up a writable stream named file, associated with the local file data.txt. We then use Node.js's https module to perform an HTTP GET request to the specified URL. The get method triggers a callback function when we receive a response from the server.

Inside the callback, the script pipes the response directly into the writable stream. This operation efficiently directs the data received from the remote server to the local file data.txt, essentially downloading and writing the content concurrently.

Finally, we set up an event listener for the "finish" event on the stream. This event fires when all the data has been successfully written to the file. Upon completion, the script logs "done" to the console, indicating the successful download and writing of the file.

How to Read a JSON File

We have already seen how we can read a JSON file using the fs module. Let's say that we want to read our previously created bio.json file. Its data looks like the below:

[
  {
    "id": 1,
    "name": "Alice",
    "address": "123 Main St"
  },
  {
    "id": 2,
    "name": "Bob",
    "address": "456 Elm St"
  },
  {
    "id": 3,
    "name": "Charlie",
    "address": "789 Oak St"
  }
]

In Node.js, we read JSON like this:

const fs = require("fs");

fs.readFile("bio.json", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    console.log(data);
  }
});

With this, our JSON data is stored within the data variable as a string. If we want, we can use it for further processing. Let’s say, we want to print the details of the user:

const fs = require("fs");

fs.readFile("bio.json", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    const users = JSON.parse(data);
    users.forEach((user) => {
      console.log(`${user.name} with ID ${user.id} lives at ${user.address}`);
    });
  }
});

Output:

Alice with ID 1 lives at 123 Main St
Bob with ID 2 lives at 456 Elm St
Charlie with ID 3 lives at 789 Oak St

In the above code, we first parse the string data variable to JSON and store it in a users variable. We then loop over the users variable to log the required message.

`fs.promises`

fs.promises provides a set of asynchronous functions for interacting with the file system in Node.js. These functions are based on promises and offer a more readable and efficient way to handle asynchronous operations compared to callbacks.

With fs.promises, we don't need to add nested callbacks – which means that we can avoid callback hell.

A basic readFile operation with fs.promises looks like this:

const fs = require("fs").promises;

async function readTextFile() {
  try {
    const data = await fs.readFile("data.txt", "utf8");
    console.log(data);
  } catch (err) {
    console.error(err);
  }
}

readTextFile();

Wrapping Up

In this tutorial, we've explored the essential techniques for handling files in Node.js with the help of the fs module.

From understanding synchronous and asynchronous file reading with methods like fs.readFile() and fs.readFileSync() to processing various file formats such as text, HTML, JSON, and even reading files from URLs, we've covered a bunch of functionalities. We also learned about fs.promises, which is a more elegant way to handle file operations using asynchronous functions.

Frequently Asked Questions (FAQs)

1. How do I read a file asynchronously in Node.js?

There are mainly two ways to read a file asynchronously in Node.js: using fs.readFile() and using fs.promises.

The fs.readFile() method uses callbacks to handle the operation. We provide a callback function that will be called with the file data (or an error) when the reading is complete.

But fs.promises offers a promise-based approach. We can use the readFile function with await to wait for the reading to finish and then access the data directly.

2. How can I check if a file exists before reading or writing in Node.js?

There are two methods to check if a file exist: using fs.stat and using fs.promises.access.

The fs.stat method synchronously checks if a file exists and returns information about it like size and access time. The fs.promises.access method asynchronously checks if a file exists and returns a promise that resolves or rejects based on the existence of the file.

3. How can I handle errors when reading or writing files in Node.js?

To handle errors while reading or writing files in Node.js, we can utilize error-first callbacks or promises along with try-catch blocks.

Comparable vs Comparator Interfaces in Java – Which Should You Use and When?

Ashutosh Krishna — Tue, 23 Jul 2024 13:13:33 +0000

Sorting is a fundamental operation in programming, essential for organizing data in a specific order. In Java, built-in sorting methods provide efficient ways to sort primitive data types and arrays, making it easy to manage and manipulate collections of data. For instance, you can quickly sort an array of integers or a list of strings using methods like Arrays.sort() and Collections.sort().

However, when it comes to sorting custom objects, such as instances of user-defined classes, the built-in sorting methods fall short. These methods don't know how to order objects based on custom criteria. This is where Java's Comparable and Comparator interfaces come into play, allowing developers to define and implement custom sorting logic tailored to specific requirements.

In this blog post, we'll explore how to use the Comparable and Comparator interfaces to sort custom objects in Java. I'll provide examples to illustrate the differences and use cases for each approach, helping you master custom sorting in your Java applications.

Sorting Methods for Primitive Types
How to Use the Comparable Interface
How to Use the Comparator Interface
Comparable vs Comparator
Wrapping Up

Sorting Methods for Primitive Types

Java provides a variety of built-in sorting methods that make it easy to sort primitive data types. These methods are highly optimized and efficient, allowing you to sort arrays and collections with minimal code. For primitive types, such as integers, floating-point numbers, and characters, the Arrays.sort() method is commonly used.

How to Use the Arrays.sort() Method

The Arrays.sort() method sorts the specified array into ascending numerical order. This method uses a dual-pivot quicksort algorithm, which is faster and more efficient for most data sets.

Let's look at an example of sorting an array of integers and characters using Arrays.sort():

package tutorial;

import java.util.Arrays;

public class PrimitiveSorting {
    public static void main(String[] args) {
        int[] numbers = { 5, 3, 8, 2, 1 };
        System.out.println("Original array: " + Arrays.toString(numbers));

        Arrays.sort(numbers);
        System.out.println("Sorted array: " + Arrays.toString(numbers));

        char[] characters = { 'o', 'i', 'e', 'u', 'a' };
        System.out.println("Original array: " + Arrays.toString(characters));

        Arrays.sort(characters);
        System.out.println("Sorted array: " + Arrays.toString(characters));
    }
}

Output:

Original array: [5, 3, 8, 2, 1]
Sorted array: [1, 2, 3, 5, 8]
Original array: [o, i, e, u, a]
Sorted array: [a, e, i, o, u]

How to Use the Collections.sort() Method

The Collections.sort() method is used to sort collections such as ArrayList. This method is also based on the natural ordering of the elements or a custom comparator.

package tutorial;

import java.util.ArrayList;
import java.util.Collections;

public class CollectionsSorting {
    public static void main(String[] args) {
        ArrayList wordsList = new ArrayList<>();
        wordsList.add("banana");
        wordsList.add("apple");
        wordsList.add("cherry");
        wordsList.add("date");
        System.out.println("Original list: " + wordsList);

        Collections.sort(wordsList);
        System.out.println("Sorted list: " + wordsList);
    }
}

Output:

Original list: [banana, apple, cherry, date]
Sorted list: [apple, banana, cherry, date]

Limitations with Custom Classes

While Java's built-in sorting methods, such as Arrays.sort() and Collections.sort(), are powerful and efficient for sorting primitive types and objects with natural ordering (like String), they fall short when it comes to sorting custom objects. These methods do not inherently know how to order user-defined objects because there is no natural way for them to compare these objects.

For example, consider a simple Person class that has name, age, and weight attributes:

package tutorial;

public class Person {
    String name;
    int age;
    double weight;

    public Person(String name, int age, double weight) {
        this.name = name;
        this.age = age;
        this.weight = weight;
    }

    @Override
    public String toString() {
        return "Person [name=" + name + ", age=" + age + ", weight=" + weight + " kgs]";
    }
}

If we try to sort a list of Person objects using Arrays.sort() or Collections.sort(), we will encounter a compilation error because these methods do not know how to compare Person objects:

package tutorial;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class CustomClassSorting {
    public static void main(String[] args) {
        List people = new ArrayList<>(Arrays.asList(
                new Person("Alice", 30, 65.5),
                new Person("Bob", 25, 75.0),
                new Person("Charlie", 35, 80.0)
        ));
        System.out.println("Original people list: " + people);

        Collections.sort(people);
        System.out.println("Sorted people list: " + people);
    }
}

Compilation Error:

java: no suitable method found for sort(java.util.List)
    method java.util.Collections.sort(java.util.List) is not applicable
      (inference variable T has incompatible bounds
        equality constraints: tutorial.Person
        lower bounds: java.lang.Comparable)
    method java.util.Collections.sort(java.util.List,java.util.Comparator) is not applicable
      (cannot infer type-variable(s) T
        (actual and formal argument lists differ in length))

The error occurs because the Person class does not implement the Comparable interface, and there is no way for the sorting method to know how to compare two Person objects.

To sort custom objects like Person, we need to provide a way to compare these objects. Java offers two main approaches to achieve this:

Implementing the Comparable Interface: This allows a class to define its natural ordering by implementing the compareTo method.
Using the Comparator Interface: This allows us to create separate classes or lambda expressions to define multiple ways of comparing objects.

We will explore both approaches in the upcoming sections, starting with the Comparable interface.

How to Use the Comparable Interface

Java provides a Comparable interface to define a natural ordering for objects of a user-defined class. By implementing the Comparable interface, a class can provide a single natural ordering that can be used to sort its instances. This is particularly useful when you need a default way to compare and sort objects.

Overview

The Comparable interface contains a single method, compareTo(), which compares the current object with the specified object for order. The method returns:

A negative integer if the current object is less than the specified object.
Zero if the current object is equal to the specified object.
A positive integer if the current object is greater than the specified object.

How Comparable Allows for a Single Natural Ordering of Objects

By implementing the Comparable interface, a class can ensure that its objects have a natural ordering. This allows the objects to be sorted using methods like Arrays.sort() or Collections.sort() without the need for a separate comparator.

Let's implement the Comparable interface in a new PersonV2 class, comparing by age.

package tutorial;

public class PersonV2 implements Comparable<PersonV2> {
    String name;
    int age;
    double weight;

    public PersonV2(String name, int age, double weight) {
        this.name = name;
        this.age = age;
        this.weight = weight;
    }

    @Override
    public String toString() {
        return "PersonV2 [name=" + name + ", age=" + age + ", weight=" + weight + " kgs]";
    }

    @Override
    public int compareTo(PersonV2 other) {
        return this.age - other.age;
    }
}

In this implementation, the compareTo() method compares the age attribute of the current PersonV2 object with the age attribute of the specified PersonV2 object by subtracting one age from the other. By using the expression this.age - other.age, we’re effectively implementing this logic as follows:

If this.age is less than other.age, the result will be negative.
If this.age is equal to other.age, the result will be zero.
If this.age is greater than other.age, the result will be positive.

Note: We can also use Integer.compare(this.age, other.age) instead of performing the arithmetic operation manually.

Now that the PersonV2 class implements the Comparable interface, we can sort a list of PersonV2 objects using Collections.sort():

package tutorial;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class CustomClassSortingV2 {
    public static void main(String[] args) {
        List people = new ArrayList<>(Arrays.asList(
                new PersonV2("Alice", 30, 65.5),
                new PersonV2("Bob", 25, 75.0),
                new PersonV2("Charlie", 35, 80.0)
        ));
        System.out.println("Original people list: " + people);

        Collections.sort(people);
        System.out.println("Sorted people list: " + people);
    }
}

Output:

Original people list: [PersonV2 [name=Alice, age=30, weight=65.5 kgs], PersonV2 [name=Bob, age=25, weight=75.0 kgs], PersonV2 [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list: [PersonV2 [name=Bob, age=25, weight=75.0 kgs], PersonV2 [name=Alice, age=30, weight=65.5 kgs], PersonV2 [name=Charlie, age=35, weight=80.0 kgs]]

In this example, the PersonV2 objects are sorted in ascending order of age using the Collections.sort() method, which relies on the natural ordering defined by the compareTo() method in the PersonV2 class.

Limitations of Comparable

While the Comparable interface provides a way to define a natural ordering for objects, it has several limitations that can restrict its use in practical applications. Understanding these limitations can help us determine when to use other mechanisms, such as the Comparator interface, to achieve more flexible sorting.

Single Natural Ordering: The primary limitation of Comparable is that it allows only one natural ordering for the objects of a class. When you implement Comparable, you define a single way to compare objects, which is used whenever the objects are sorted or compared. This can be restrictive if you need to sort objects in multiple ways.
Inflexibility: If you need to sort objects by different attributes or in different orders, you will have to modify the class or create new implementations of Comparable. This inflexibility can lead to a proliferation of comparison methods and can make the code harder to maintain.
Non-Adaptable: Once a class implements Comparable, the natural ordering is fixed and cannot be easily changed. For instance, if your PersonV2 class initially sorts by age but later you need to sort by weight or name, you have to either change the compareTo() method or create a new version of the class.

This is where the Comparator interface comes into play. To define multiple ways of comparing objects, we can use the Comparator interface, which we will explore in the next section.

How to Use the Comparator Interface

The Comparator interface in Java provides a way to define multiple ways of comparing and sorting objects. Unlike the Comparable interface, which allows only a single natural ordering, Comparator is designed to offer flexibility by allowing multiple sorting strategies. This makes it particularly useful for scenarios where objects need to be sorted in different ways.

Overview

The Comparator interface defines a single method, compare(), which compares two objects and returns:

A negative integer if the first object is less than the second object.
Zero if the first object is equal to the second object.
A positive integer if the first object is greater than the second object.

This method provides a way to define custom ordering for objects without modifying the class itself.

How Comparator Allows for Multiple Ways of Ordering Objects

The Comparator interface allows you to create multiple Comparator instances, each defining a different ordering for objects. This flexibility means that you can sort objects by various attributes or in different orders without altering the object's class.

Let's implement multiple Comparator instances for the Person class. We'll define comparators for sorting by name, by age, and by weight. First, we need to update the Person class to include getters and ensure that attributes are accessible.

package tutorial;

public class Person {
    String name;
    int age;
    double weight;

    public Person(String name, int age, double weight) {
        this.name = name;
        this.age = age;
        this.weight = weight;
    }

    public String getName() {
        return name;
    }

    public int getAge() {
        return age;
    }

    public double getWeight() {
        return weight;
    }

    @Override
    public String toString() {
        return "Person [name=" + name + ", age=" + age + ", weight=" + weight + " kgs]";
    }
}

Comparator by Name

This comparator sorts Person objects alphabetically by their name.

package tutorial.comparator;

import tutorial.Person;

import java.util.Comparator;

public class PersonNameComparator implements Comparator<Person> {

    @Override
    public int compare(Person p1, Person p2) {
        return p1.getName().compareTo(p2.getName());
    }
}

Comparator by Age

This comparator sorts Person objects by their age, in ascending order.

package tutorial.comparator;

import tutorial.Person;

import java.util.Comparator;

public class PersonAgeComparator implements Comparator<Person> {

    @Override
    public int compare(Person p1, Person p2) {
        return p1.getAge() - p2.getAge();
    }
}

Comparator by Weight

This comparator sorts Person objects by their weight, in ascending order.

package tutorial.comparator;

import tutorial.Person;

import java.util.Comparator;

public class PersonWeightComparator implements Comparator<Person> {

    @Override
    public int compare(Person p1, Person p2) {
        return (int) (p1.getWeight() - p2.getWeight());
    }
}

Now, here’s how you can use these Comparator instances to sort a list of Person objects:

package tutorial;

import tutorial.comparator.PersonAgeComparator;
import tutorial.comparator.PersonNameComparator;
import tutorial.comparator.PersonWeightComparator;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class CustomClassSortingV3 {
    public static void main(String[] args) {
        List people = new ArrayList<>(Arrays.asList(
                new Person("Alice", 30, 65.5),
                new Person("Bob", 25, 75.0),
                new Person("Charlie", 35, 80.0)
        ));
        System.out.println("Original people list: " + people);

        Collections.sort(people, new PersonNameComparator());
        System.out.println("Sorted people list by name: " + people);

        Collections.sort(people, new PersonAgeComparator());
        System.out.println("Sorted people list by age: " + people);

        Collections.sort(people, new PersonWeightComparator());
        System.out.println("Sorted people list by weight: " + people);
    }
}

Output:

Original people list: [Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list by name: [Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list by age: [Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list by weight: [Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]

In this example, the Comparator instances allow sorting the Person objects by different attributes: name, age, and weight. This demonstrates how the Comparator interface enables flexible and versatile sorting strategies for a class.

Comparable vs Comparator

When sorting objects in Java, you have two primary options: the Comparable and Comparator interfaces. Understanding the differences between these two interfaces can help you choose the right approach for your needs. Please note that this is also a very important interview question.

Comparison

Here’s a table comparing and contrasting the Comparable and Comparator interfaces in Java:

Feature	Comparable	Comparator
Definition	Provides a single, natural ordering for objects	Provides multiple ways to compare objects
Method	compareTo(T o)	compare(T o1, T o2)
Implementation	Implemented within the class itself	Implemented outside the class
Sorting Criteria	One default natural ordering	Multiple sorting criteria
Flexibility	Limited to one way of comparing objects	Flexible; multiple comparators can be defined
Class Modification	Requires modifying the class to implement `Comparable`	Does not require modifying the class
Use Case	Use when there is a clear, natural ordering (e.g., sorting employees by ID)	Use when different sorting orders are needed or when you cannot modify the class

Benefits and Drawbacks of Each Approach

Comparable Operator

Benefits:

Simplicity: Provides a default sorting order that is easy to implement and use.
Built-in: The natural ordering is part of the class itself, so it is always available and used by default in sorting methods.

Drawbacks:

Single Ordering: Can only define one way to compare objects. If different sorting orders are needed, the class must be modified or additional Comparator instances must be used.
Class Modification: Requires altering the class to implement Comparable, which might not be feasible if the class is part of a library or if its natural ordering is not clear.

Comparator

Benefits:

Flexibility: Allows for multiple sorting orders and criteria, which can be defined externally and used as needed.
Non-invasive: Does not require modification of the class itself, making it suitable for classes you do not control or when you need different sorting options.

Drawbacks:

Complexity: Requires creating and managing multiple Comparator instances, which can add complexity to the code.
Overhead: Might introduce additional overhead if many comparators are used, especially if they are created on the fly.

In summary, Comparable is best used when a class has a natural ordering that makes sense for most use cases.

Comparator, on the other hand, provides flexibility for sorting by multiple criteria and is useful when the class does not have a natural ordering or when different sorting orders are needed.

Choosing between Comparable and Comparator depends on your specific sorting needs and whether you need a single default order or multiple flexible sorting options.

Wrapping Up

Understanding and utilizing both Comparable and Comparator can significantly enhance your ability to manage and manipulate object collections in Java. By applying these concepts, you can create more flexible and powerful sorting mechanisms.

To solidify your understanding, try implementing both Comparable and Comparator in real-world scenarios. Experiment with different classes and sorting criteria to see how each approach works in practice.

Links to Official Java Documentation:

How To Implement Instant Search with Flask and HTMX

Ashutosh Krishna — Mon, 22 Jul 2024 11:36:53 +0000

Instant search is a feature that shows search results as users type their query. Instead of waiting for a full page reload or submitting a form, results appear instantly, allowing users to find what they are looking for quickly. For example, when you start typing in a search box, suggestions or matching items will appear immediately, making the process smoother and more efficient.

In this tutorial, you'll learn how to create a simple instant search feature using Flask and HTMX. This will help you build interactive web applications with better user experience.

Flask: Flask is a popular web framework for Python. It is simple and lightweight, making it easy to set up and start building web applications quickly. Flask lets you to create routes, handle requests, and serve HTML templates with minimal setup.
HTMX: This is a powerful JavaScript library that lets you to create dynamic web pages without having to write a lot of JavaScript code. With HTMX, you can update parts of a page based on user actions, like typing in a search box. It makes it easy to load data from the server and display it on the page without a full reload.

How to Set Up the Environment

In this section, we'll set up the environment for our Flask project, including installing the necessary packages and organizing the project structure.

1. How to Install Flask and HTMX

First, you need to install Flask, Flask-SQLAlchemy, and Flask-Migrate. You can do this using pip. Open your terminal and run:

pip install Flask Flask-SQLAlchemy Flask-Migrate

For HTMX, we'll include it in our HTML template directly from a CDN.

2. How to Create a Virtual Environment

It's a good practice to create a virtual environment for your projects to manage dependencies. Here's how to create one:

python -m venv venv

Next, activate the environment:

# On Windows
venv\Scripts\activate

# On macOS/Linux
source venv/bin/activate

3. How to Set Up the Project Structure

Now, set up your project structure as follows:

my_flask_app/
├── core/
│   ├── __init__.py
│   ├── models.py
│   └── routes.py
├── config.py
└── main.py

Let us start with creating the first file: core/init.py. This file is the initialization script for the core module of our Flask application. It sets up the Flask app instance and configures it using the settings from the DevelopmentConfig class, and initializes the database and migration system.

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from config import DevelopmentConfig

# Create the Flask app instance
app = Flask(__name__)

# Load configuration from DevelopmentConfig
app.config.from_object(DevelopmentConfig)

# Initialize SQLAlchemy with the app instance
db = SQLAlchemy(app)

# Initialize Flask-Migrate with the app instance and database
migrate = Migrate(app, db)

# Import routes to register them with the app
from core import routes

Next, we will create the config.py file from where we'll import the DevelopmentConfig class. This file contains configuration settings for different environments (development, testing, production). These settings help manage different behaviors and configurations based on where your app is running.

class Config(object):
    DEBUG = False
    TESTING = False
    CSRF_ENABLED = True
    SECRET_KEY = "guess-me"
    SQLALCHEMY_DATABASE_URI = "sqlite:///db.sqlite"
    SQLALCHEMY_TRACK_MODIFICATIONS = False
    BCRYPT_LOG_ROUNDS = 13
    WTF_CSRF_ENABLED = True
    DEBUG_TB_ENABLED = False
    DEBUG_TB_INTERCEPT_REDIRECTS = False

class DevelopmentConfig(Config):
    DEVELOPMENT = True
    DEBUG = True
    WTF_CSRF_ENABLED = False
    DEBUG_TB_ENABLED = True

class TestingConfig(Config):
    TESTING = True
    DEBUG = True
    SQLALCHEMY_DATABASE_URI = "sqlite:///testdb.sqlite"
    BCRYPT_LOG_ROUNDS = 1
    WTF_CSRF_ENABLED = False

class ProductionConfig(Config):
    DEBUG = False
    DEBUG_TB_ENABLED = False

Config: The base configuration class with default settings.
DevelopmentConfig: Inherits from Config and overrides development settings.
TestingConfig: Inherits from Config and overrides settings for testing.
ProductionConfig: Inherits from Config and overrides production settings.

Finally, we'll create the main.py file. This is the entry point of our application. When we run this file, it starts the Flask web server.

from core import app

# Start the Flask app
if __name__ == '__main__':
    app.run(debug=True)

if __name__ == '__main__': This ensures that the Flask app runs only if the script is executed directly (not imported as a module).
app.run(debug=True): Starts the Flask development server with debug mode enabled, which provides detailed error messages and auto-reloading.

Now that you understand the project files, we can proceed with implementing the instant search functionality. This will involve creating the models and search route, setting up the HTMX-powered front-end, and connecting everything to fetch and display search results dynamically.

How to Set up the Database

In this section, we will set up the database for our Flask application. We will use SQLite for simplicity. We will create a model for the data we want to search and seed the database with sample data.

SQLite is a lightweight, disk-based database that doesn’t require a separate server process. It's an excellent choice for development and small projects because it is easy to set up and use.

How to Create a Model for the Data to Be Searched

We will create a Book model to represent the data in our database. This model will include fields like the book title and author.

Let's create the core/models.py file and add the model there:

from core import db

class Book(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(100), nullable=False)
    author = db.Column(db.String(100), nullable=False)

How to Apply Migrations Using Flask-Migrate

Before we can seed our database, we need to set up database migrations using Flask-Migrate. This tool helps us manage database changes, such as creating tables and altering schemas, systematically.

Initialize the migrations folder by running the following command in your project directory:

flask db init

This command creates a migrations directory in our project, which will store migration scripts.

Generate a migration script that creates the necessary database tables based on your models:

flask db migrate -m "Initial migration"

This command scans your models and generates a new migration script in the migrations folder.

Apply the migration to create the tables in your database:

flask db upgrade

This command executes the migration script, creating the tables defined by your models in the database. Post this step, you will see an instance/db.sqlite file created.

How to Seed Data Into Your Database

Now that we have set up the database and applied the migration, we can proceed with seeding the database. Create a file named seeder.py with the following content:

import csv
from sqlalchemy.exc import IntegrityError

from core import db, app
from core.models import Book


def seed_data():
    with app.app_context():
        # Open the CSV file
        with open("data.csv", newline='', encoding='utf-8') as csvfile:
            reader = csv.DictReader(csvfile)

            # Iterate over the rows in the CSV file
            for row in reader:
                # Create a new Book instance
                book = Book(
                    title=row['Book Name'],
                    author=row['Author Name']
                )

                # Add the book to the session
                db.session.add(book)

            try:
                # Commit the session to write the books to the database
                db.session.commit()
                print("Books added successfully.")
            except IntegrityError as e:
                db.session.rollback()
                print(f"Error occurred: {e}")


if __name__ == "__main__":
    seed_data()

The seeder script is responsible for populating the database with initial data. This is useful for testing and development purposes, allowing you to work with a set of sample data. This script reads data from data.csv, and processes it to insert it into the database.

Note: You can download the data.csv file from here.

To use this script, ensure your data.csv file exists in the same directory as seeder.py. Run the script using Python:

python seeder.py

How to Set Up Basic Routing and HTML

In this section, we'll set up a basic route in Flask to serve an index page (index.html) where users can search and display books.

How to Set Up Flask Route

Let's set up a Flask route (/) to render an index.html template and display books. For that, create a core/routes.py file and add the following route:

from flask import render_template
from core import app
from core.models import Book

@app.route('/')
def index():
    # Fetch the first 20 books to display by default
    books = Book.query.limit(20).all()
    return render_template("index.html", books=books)

The Flask application handles routing through the @app.route('/') decorator, which directs requests to the root URL (/). When a user visits the homepage, the index() function is invoked.

Inside this function, we query the Book model using SQLAlchemy to fetch the first 20 books from the database. These books are then passed as a parameter (books) to the render_template function, which renders the index.html template.

How to Creating the index.html Template

Create a file named index.html inside a templates directory in your project. The templates directory will lie in the core package. This file will contain the HTML structure for our book search page.

html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Book Searchtitle>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css" />
head>
<body>
  <section class="section">
    <div class="columns">
      <div class="column is-one-third is-offset-one-third">
        <input type="text" class="input" placeholder="Search" name="query" />
      div>
    div>
    <table class="table is-fullwidth">
      <thead>
        <tr>
          <th>IDth>
          <th>Book Titleth>
          <th>Book Authorth>
        tr>
      thead>
      <tbody id="results">
        {% for book in books %}
        <tr>
          <td>{{ book.id }}td>
          <td>{{ book.title }}td>
          <td>{{ book.author }}td>
        tr>
        {% endfor %}
      tbody>
    table>
  section>
body>
html>

This HTML file uses the Bulma CSS framework for styling and includes elements such as an input field for user searches and a table to display book details fetched from the database.

The index.html template utilizes Jinja2 templating to dynamically populate the table rows () with book data retrieved from the Flask backend. Each book's ID, title, and author are displayed in the table rows using {{book.id}}, {{ book.title }}, and {{book.author}} respectively.

How to Run the Application

Let's run the application using the following command:

flask run

Once your application is up and running, this what how it should look like:

web page with ID, book titles, and book authors

How to Add HTMX for Instant Search

Finally, we'll add HTMX to enhance our Flask application with dynamic search capabilities. For this, we'll introduce a new route and modify existing HTML template.

How to Create the Search Route

First, create a new route /search in your Flask application to handle book searches based on user input:

from flask import render_template, request
from core import app
from core.models import Book

@app.route('/search')
def search():
    query = request.args.get("query")
    if query:
        results = Book.query.filter(Book.title.ilike(f"%{query}%") | Book.author.ilike(f"%{query}%")).limit(10).all()
    else:
        results = Book.query.limit(20).all()
    return render_template("search_results.html", results=results)

This route listens for GET requests to /search. It retrieves the search query from the URL parameter using request.args.get("query").

If a query parameter is present, it uses SQLAlchemy's ilike method to perform a case-insensitive search across the title and author columns of the Book table, fetching up to 10 results.

If no query parameter is provided, it defaults to fetching the first 20 books from the database. The results are passed to a new search_results.html template for rendering.

How to Modify index.html to Add HTMX

html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Book Searchtitle>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css" />
  
  <script src="https://cdn.jsdelivr.net/npm/htmx.org/dist/htmx.min.js">script>
head>
<body>
  <section class="section">
    <div class="columns">
      <div class="column is-one-third is-offset-one-third">
        
        <input
            type="text"
            class="input"
            placeholder="Search"
            name="query"
            hx-get="/search"
            hx-trigger="keyup changed delay:500ms"
            hx-target="#results"
          />
      div>
    div>
    <table class="table is-fullwidth">
      <thead>
        <tr>
          <th>IDth>
          <th>Book Titleth>
          <th>Book Authorth>
        tr>
      thead>
      <tbody id="results">
        {% for book in books %}
          <tr>
            <td>{{ book.id }}td>
            <td>{{ book.title }}td>
            <td>{{ book.author }}td>
          tr>
        {% endfor %}
      tbody>
    table>
  section>
body>
html>

The

Ashutosh Krishna - freeCodeCamp.org

Database Version Control with Liquibase and Spring Boot

Why Database Version Control Matters

What is Liquibase?

Project Setup

Understanding Core Liquibase Concepts

Create the Initial Employee Schema (Version 1)

What Just Happened?

Inspecting the Database: Liquibase Metadata Tables

The DATABASECHANGELOG Table

The DATABASECHANGELOGLOCK Table

Evolving the Employee API

Version 2: Adding an Email Field

Version 3: Adding Departments Support

Version 4 & 5: Employee Status and Performance Indexes

The Golden Rule: Never Modify Executed ChangeSets

How to Fix It (The Right Way)

Working with Seed Data

The Danger of Data Migrations

Rollbacks

Automatic vs. Explicit Rollbacks

The Reality Check on Rollbacks

Common Beginner Mistakes

1. The "Mega" ChangeSet

2. Manual Database Tweaking (The Phantom Menace)

3. Ignoring the "From Scratch" Build

4. Hardcoding Environment Details

5. Messing Up Migration Ordering

Liquibase vs Flyway vs Manual SQL Scripts

1. Manual SQL Scripts (The Baseline)

2. Flyway (The SQL Purist)

3. Liquibase (The Abstraction Layer)

Liquibase Best Practices

1. One Logical Change Per ChangeSet (The Atomic Rule)

2. Meaningful File Organization and Naming

3. Treat Database Changes Like Application Code

4. Integrate Migrations into CI/CD

5. Never Fix Forward by Deleting History

Final Thoughts

RAG Explained Simply with a Real Project

Here's what we'll cover:

What is RAG?

The Open-Book Test Analogy

Why Traditional LLMs Fail

How RAG Works Internally

Documents

Chunking

Embeddings

Vector Databases

Semantic Search and Similarity Matching

Prompt Augmentation

Final LLM Response

Quick Recap

How to Build a Real RAG Project

Project Setup

Preparing the PDF

Writing the RAG Code Step-by-Step

Step 1: Imports and Environment Setup

Step 2: Loading the PDF Document

Step 3: Chunking the Text

Step 4: Creating Embeddings and Initializing the Vector DB

Step 5: Setting Up the Retriever and Prompt Template

Step 6: Initializing the LLM and Constructing the RAG Chain

Step 7: Invoking the Chain with a Question

Step 8: Making it Conversational

Full Code

Taking it out of the terminal

The Full Data Flow

Breaking Down the Execution Timeline

Common RAG Problems

1. Bad Chunking

2. Irrelevant Retrieval

3. Hallucinations

4. Latency

5. Stale Data

Advanced RAG Concepts

Hybrid Search

Reranking

Agentic RAG

Graph RAG

The `DATABASECHANGELOG` Table

The `DATABASECHANGELOGLOCK` Table