Data processing lets teams automate processes. It also supports decision-making, offers a significantly more personalized experience to the end user, and detects patterns in many areas such as banking fraud or risk mitigation. Companies need to know how to capture and use data effectively, safely, and legally.

You likely are or have been a user of various products and services. And you know that processes involving data are fundamental to almost everything around us. You're likely also already familiar with terms like Big Data, Data Analytics, Artificial Intelligence, and Machine Learning.

But unless you're an expert in one of these fields, some of these concepts might seem overwhelming. These are large areas of study, after all.

And even if you're trained in one of these areas, it's difficult to know all the details about each field, as the data world is vast.

One way to understand this world of data a bit better is by dividing it, and establishing a distinction between the areas of Artificial Intelligence and Data Management. This isn't the only way to proceed, but I've found it helpful to separate the set of disciplines and techniques for information processing into these two blocks.

On one side is Data Management, which encompasses everything related to the capture, storage, protection, and analysis of data.

Meanwhile, on the other side is Artificial Intelligence, which focuses on developing techniques that allow a machine to emulate human capabilities like reasoning or learning to solve a problem, whether interacting with data or not.

Here, interaction refers to an algorithm acquiring "knowledge" from data, but not all artificial intelligence functions.

In any case, this book offers a comprehensive overview of Data Management, helping you understand all the terms and related concepts involved in using, processing, and analyzing data.

It won't just provide an abstract explanation of the field and its contents. It'll instead help you understand it holistically and offer a more practical and realistic view. We'll also study a use case to put into practice everything we discuss.

Table Of Contents

• Our Case Study

• Data Management Fundamentals

  • Data as an Asset

  • Data, Information, Knowledge, and Value

  • The Data Lifecycle

  • Data Management Principles

  • Data Management Capabilities

• Data Governance

  • Data Ownership

  • Data Stewardship

  • Decision Rights

  • Data Policies

  • Data Standards

  • Data Accountability

• Data Ethics

  • Ethical Data Use

  • Consent and Transparency

  • Fairness and Non-Discrimination

  • Responsible Data Sharing

  • Ethical Risk Management

• Data Security and Privacy

  • Data Classification

  • Identity and Access Management

  • Encryption

  • Data Masking

  • Privacy Controls

  • Audit and Compliance

  • Security Operations (SecOps)

• Data Architecture

  • Enterprise Data Architecture

  • Data Domains

  • Data Flows

  • Operational Data Architecture

  • Analytical Data Architecture

  • Cloud and Hybrid Data Architectures

• Data Modeling and Design

  • Conceptual Data Models

  • Logical Data Models

  • Physical Data Models

  • Entity-Relationship Modeling

  • Dimensional Modeling

  • Data Model Governance

• Data Storage and Operations

  • Databases

  • File and Object Storage

  • Data Warehouses

  • Data Lakes and Lakehouses

  • Backup and Recovery

  • Retention and Archiving

  • Performance and Availability

• Document and Content Management

  • Unstructured Data

  • Document Capture

  • Document Classification

  • Content Storage

  • Search and Retrieval

  • Records Management

• Reference and Master Data Management

  • Master Data

  • Reference Data

  • Golden Records

  • Entity Resolution

  • Deduplication

  • Survivorship Rules

• Metadata Management

  • Business Metadata

  • Technical Metadata

  • Operational Metadata

  • Data Catalogs

  • Business Glossaries

  • Data Lineage

  • Metadata Standards

  • Metadata Quality

  • Metadata Governance

• Data Integration and Interoperability

  • Data Ingestion

  • Batch Integration

  • Streaming Integration

  • API-Based Integration

  • ETL and ELT

  • Data Exchange Standards

  • Schema Management

• Data Quality

  • Data Quality Dimensions

  • Data Profiling

  • Data Quality Rules

  • Data Validation

  • Data Cleansing

  • Data Quality Monitoring

  • Issue Management

• Data Engineering

  • Data Pipelines

  • Pipeline Orchestration

  • Data Transformation

  • Workflow Automation

  • Data Testing

  • Data Versioning

  • Data Platform Operations

  • Data Observability

  • Data Contracts

  • DataOps

  • DevOps

• Data Warehousing and Business Intelligence

  • Analytical Data Stores

  • Facts and Dimensions

  • Metrics and KPIs

  • Semantic Layers

  • Reports and Dashboards

  • Self-Service Analytics

• Big Data

  • The 3Vs: Volume, Velocity, and Variety

  • Big Data Architectures

  • Big Data Storage and Processing

  • Big Data Analytics

• Analytics and Data Science

  • Analytical Datasets

  • Exploratory Data Analysis

  • Feature Engineering

  • Experimentation

  • Model-Ready Data

  • Analytical Product Delivery

• Data Products

  • Product Characteristics

  • Ownership and Lifecycle

• Data Management Organization

  • Operating Model

  • Roles and Collaboration

• Data Management Maturity

  • Maturity Levels

  • Assessment and Roadmap

• Conclusions

Our Case Study

Our use case involves a fictional university offering international master's programs in Artificial Intelligence and Data Management. It's a public-private institution providing various training programs for different end users, such as recent graduates looking to specialize in this area, working professionals, or international students.

This use case lets us analyze the entire data lifecycle, from student admission to graduation. Also, in a university setting, we can use data alongside artificial intelligence to automate enrollment processes, enhance the student's experience when accessing educational resources, optimize organizational operations, and ultimately help the university differentiate itself from other institutions offering similar programs.

The data lifecycle begins before enrollment in a master's program, as a candidate might discover the program through an advertising campaign, visit the institution's website, or complete an application form. They can then enroll and attend classes, using digital platforms and participating in various educational activities. Finally, they'll complete the program and become part of the alumni community.

Each of these interactions generates different types of data, such as personal, academic, administrative, and financial data. There are also more complex types of data, like activity and digital behavior data, which can include records of access to the virtual campus or consulted resources, among others.

This journey allows us to see how data goes through different phases. We'll see how it's captured, validated, stored, integrated with other systems, protected, analyzed, and finally retained or deleted according to the organization's policies.

As you can imagine, Data Management isn't just about storing data in a database. It's also about ensuring that, throughout its lifecycle, the data is accurate, secure, understandable, accessible to those who need it, and used legitimately.

Also, the university, like any other entity, uses data to identify the needs or problems of its users in order to propose solutions. One such issue could be commuting, as some students in the master's programs live far from campus, others might work, and still others may have poor public transportation options. In these cases, distance or travel time becomes a decisive factor for those students.

Faced with this seemingly complex issue, the university can use data and artificial intelligence techniques to plan and offer suitable transportation services to certain interested students. This means, based on eligibility criteria such as the distance from campus or enrollment in mandatory in-person classes, the university can plan to offer free taxi/VTC services to certain students.

But the idea wouldn't be to provide unlimited taxi services to all students – just to design a controlled, measurable, and sustainable benefit based on clear business rules.

Processing this data effectively would allow the university to offer a more precise service than other competitors, who might offer generic public transportation discounts or fixed bus routes. And while these solutions might be very useful, they don't always adequately meet the needs of all students.

In this scenario, it's clear that a wide variety of data is generated, including data on students, faculty, courses, schedules, attendance records, trips taken, and so on. Using and analyzing this data, we'll be able to learn many Data Management principles. We'll also demonstrate how data pipelines are built, how data is transformed into useful analytical products, and what techniques are involved.

To make these ideas easier to follow in practice, this book is accompanied by a hands-on Jupyter notebook. It uses a compact sample of real taxi-trip data and treats it as a provider feed for the university's transportation service.

Some of the examples discussed throughout the book are reproduced in the notebook with the same dataset, so as you move through the chapters, you can see selected concepts in action, including data profiling, quality rules, integration, transformation, privacy protection, dimensional modeling, SQL analysis, and visualization. It's a focused demonstration rather than a complete implementation of every capability discussed here.

You'll also learn how the university might use artificial intelligence to predict which candidates are most likely to enroll, recommend master's programs, estimate future demand for mobility services, detect unusual patterns in taxi usage, and create conversational assistants to help candidates and students resolve their questions.

This case study will also highlight the university's need to make decisions about privacy, consent, transparency, and security. For example, personal data must be protected, eligibility rules should not unfairly discriminate, and human oversight should be established for decisions that could significantly impact a candidate or student.

Data Management Fundamentals

Data Management is the discipline responsible for capturing, storing, protecting, integrating, understanding, maintaining, and correctly using data throughout its lifecycle. At first glance, management and processing might seem to involve only storage and perhaps later analysis, but nothing could be further from the truth.

There are many more requirements like security (as managing large volumes of information quickly is useless if security is compromised) as well as data integrity and organization.

While researching for this book, I studied the very useful book Data Management Body of Knowledge (DAMA-DMBOK). It's one of the most comprehensive and reputable guides on the world of data. And I highly recommend it if you want to dive even deeper here.

According to the book, Data Management involves the development, execution, and supervision of plans, policies, programs, and practices that enable the delivery, control, protection, and enhancement of the value of data and information assets throughout their lifecycle.

This definition is especially relevant because it highlights two fundamental ideas. One is that data has intrinsic value, allowing it to be treated as an asset. The other is that this value doesn't appear directly in all cases but depends on how the data is managed.

In other words, data alone has no value, but if you process it properly, it has the potential to become usable information and subsequently knowledge.

To achieve this goal, you can think about Data Management as a set of operational capabilities, meaning the various actions a team or organization must undertake regarding its data.

Among the most fundamental are the following:

• Data Governance: deciding who has access to each piece of data and who sets the access rules.

  • Example: University faculty may have access to certain data about students in their courses, but not about any student in the organization.

• Data Architecture: designing the processes that data will follow throughout its lifecycle.

  • Example: A data architect defines how data travels from the moment a user enters it into the system, such as during an enrollment form, to where it's stored and processed internally on the university server.

• Data Storage and Operations: deciding how and where the data is stored.

  • Example: The decision is made to store students' personal data in an internal database, as opposed to alternatives like storing it in an external cloud service. Meanwhile, other data, such as educational materials, are more likely to end up stored in the cloud, although it ultimately depends on the organization's policies.

• Data Integration: gathering information from different sources to provide a unified view or access to all of them.

  • Example: A data engineer integrates information from different sources about taxi routes, as each company will have its own source with unique characteristics, making it necessary to standardize the data into an intermediate schema.

• Data Quality: ensuring that the information is accurate, complete, consistent, up-to-date, and reliable.

  • Example: A quality analyst defines the rules that the virtual campus frontend must follow to prevent end users from entering incorrect data into the system, ensuring its quality. They also impose rules on the various internal systems where the information is stored to avoid inconsistencies.

• Data Security and Privacy: protecting information against unauthorized access and other threats.

  • Example: User access passwords are stored as hashed values, not in plain text, to prevent easy access in case of a potential vulnerability.

• Metadata Management: specifically managing the data that determines the meaning of other data.

  • Example: A glossary is created with terms that define the meaning of each concept represented in the data. One of them could be "distance to campus in meters." In this case, the meaning is clear, and its inclusion in the glossary allows it to be used in the implementation of storage systems and data processing, facilitating development.

• Analytics and Business Intelligence: transforming data into reports and visual indicators that facilitate strategic decision-making within the organization.

  • Example: A data analyst creates an interactive dashboard for the administration, displaying graphs of monthly taxi expenses, the number of students benefiting, and how this service has improved the percentage of attendance in in-person classes.

So as you can see, Data Management isn't a specific activity but a collection of many different tasks and processes. When coordinated, these allow data to be transformed into strategic value.

In the university use case, it's clear that the personal data of applicants and students must be protected. Also, to help implement the free taxi service, the data sources from different transportation companies must be well-integrated and of high quality.

Data as an Asset

Data can be defined as a symbolic representation of a quantitative or qualitative attribute or variable. In other words, data are representations of facts, observations, events, or characteristics occurring in an environment, which can later be stored and processed.

This definition of data relates more to its types, such as numbers, dates, text, or images. In our use case, data might include a student's name, address, or the distance from their home to the campus. Each of these, in isolation, is a simple record, but when contextualized and analyzed together, they have the potential to become an asset.

For instance, an isolated piece of data like "18 kilometers" isn't very relevant by itself. But if it's interpreted as the characteristic "distance to campus", it becomes useful for understanding a student's situation and making a decision.

In this context, an asset is any resource expected to yield a return in the future, like buildings, patents, or other elements. Here, we're also including data because of its potential to generate value within the organization.

But this doesn't mean that just any piece of data is an asset. Data can be incorrect, duplicated, or incomplete. So its value mainly depends on how it's managed. For example, at the university, "distance to campus" becomes an asset when it's not used as an isolated number but rather for decision-making.

In our example, the distance from campus along with other student and organizational data can help us decide which students are eligible for this taxi service or how much budget should be allocated for it.

Data that's considered an asset can help drive these decisions only when the quality is adequate, because incomplete, inconsistent, or erroneous data can affect this process negatively or not contribute to the decision.

Ultimately, considering data as assets means treating it as a resource that requires specific management. And this can lead to benefits you wouldn't be able to achieve otherwise, whether it's improved end-user satisfaction or cost optimization.

Data, Information, Knowledge, and Value

From this idea arises the distinction between data, information, knowledge, and value. We'll study the progressive transformation that turns data into useful knowledge and ultimately into value for an organization.

Data

First, data is the most basic unit dealt with in Data Management, and its main function is to represent an aspect of reality. That is, data is what we imagine when we think of something like a number, some text, a date, and so on. Data has types (because of its variety), and also has a basic meaning associated, generally called semantics.

• Example: "18 kilometers" is a piece of data of the integer type, and its semantics indicate that it represents a quantity of kilometers. Here, it's important to realize that the quantity alone might be considered data, but its semantics allow for interpretation.

Information

Once we have isolated data, we can relate and contextualize it to create a more abstract meaning, which is considered information.

• Example: To better understand this concept, the previous data "18 kilometers" can be contextualized with other information like a student's name or address, allowing us to infer that the student lives that far from the campus. This is considered information, as its semantics go beyond that of a simple piece of data.

Knowledge

After obtaining information, we can then analyze and interpret it to identify patterns, trends, or cause-and-effect relationships. We do this by integrating the information and observing the prior experience of the organization or similar ones, creating an even more abstract contextualization.

• Example: If the university observes that students living more than 15 kilometers away and having in-person classes miss more classes, it can conclude that distance and schedule influence attendance. This requires information such as the students' distance from campus or their attendance records and schedules.

Value

Finally, we use knowledge in decision-making and taking actions that can generate a benefit, which is the value derived from the data.

• Example: The university can offer free taxi services only to specific students who meet certain criteria, improving attendance and user satisfaction while minimizing the impact on the budget. Here, the value lies in the benefit gained from these decisions, which may or may not be easily measurable.

In summary, success doesn't lie solely in storing large volumes of data or processing them at high speed, but in advancing them through this sequence of transformations to turn them into value. This process requires an infrastructure suited to these needs, as well as qualified people who are capable of applying the appropriate Data Management techniques.

The Data Lifecycle

Now let's look at the stages data goes through. Its lifecycle starts when the organization identifies a need for it and ends when the data is no longer useful. Between those points, teams capture, store, maintain, use, and eventually retain or delete the data. The lifecycle describes the phases that keep this journey controlled.

As the diagram shows, the lifecycle starts with business needs, not technology choices. Because data is an organizational asset, each phase should help protect it, maintain it, or turn it into value.

The lifecycle consists of the following phases (as in the graphic above):

• Planning: The organization decides what data it needs, why it needs it, who will be responsible for it, and how it could create value. Before capturing anything, the team should know which data is truly necessary and what they expect to do with it.

  • Example: The university decides it needs to know the distance between a student's home and the campus to evaluate whether it can offer free taxi service, explaining why it's necessary and what decision it will allow later.

• Design and Enablement: Once the need is clear, the team designs the infrastructure, data flows, and policies that will support it. This work draws on capabilities such as data architecture, modeling, security, quality, and governance, which we'll discuss later.

  • Example: The university defines that the distance to the campus will be calculated from the address provided by the student, that the data will be stored in a specific system, that only certain departments will have access to it, and that it must be updated if the student changes their address.

• Creation or Acquisition: At this stage, the data enters the organization for the first time. A user might create it through an interaction, or the organization might obtain it from an external source through an API, exchange, purchase, or integration.

  • Example: The data is created when the applicant completes the admission form indicating their address. External data such as geographic information or estimates of distance and travel time from a geographic API could also be obtained.

• Storage and Maintenance: Once captured, the data must be stored in an appropriate environment and kept ready for later use. Teams may store it in databases, Data Warehouses, Data Lakes, or other systems. They can then clean, integrate, update, document, and protect it as needed.

  • Example: A student's name is stored in a university server database, while the calculated distance to the campus can be saved in a cloud-based analytical database. Additionally, rules are applied to avoid duplicates, incomplete data, or inconsistent formats, ensuring data quality.

• Use: The organization uses the data for the purpose defined during planning. It might query or analyze the data, generate reports and dashboards, or use it to train AI models.

  • Example: The university uses IP addresses, response times, and virtual campus activity logs in a predictive Machine Learning algorithm to detect behavioral anomalies that indicate potential fraud. This use allows for the detection of identity theft, security issues, and the prevention of fraud in online educational activities.

• Enrichment: In this phase, teams connect and transform data to add context and uncover patterns or trends that were previously hard to see. This is one way data becomes information and knowledge.

  • Example: A student's access log to the virtual campus can be enriched with data about the time they spent using online resources, the number of material downloads they made, their interaction counts, and their historical statistics. This gives the university more context for studying engagement and its possible relationship with academic progress, without assuming that digital activity alone explains a student's results.

• Dispose: When the data is no longer needed for its original purpose, the organization decides whether to retain, archive, anonymize, or delete it. Retention policies, business needs, and legal requirements guide that decision. This phase prevents the organization from accumulating unnecessary data, which raises costs and creates extra risk when the data is personal or sensitive.

  • Example: When a student completes their master's program, the university may need to retain grades and other academic records for legal or administrative reasons. Some banking details or operational payment data may no longer be necessary once financial and legal obligations end. The retention policy should identify which fields to keep, delete securely, or anonymize for approved statistical use rather than preserving the full record indefinitely.

Although these phases appear in sequence, real data rarely moves through them only once. Teams may enrich it several times or integrate it with new sources, such as public APIs or partner systems. Think of the lifecycle as a continuous process whose phases can repeat whenever the need changes. It helps keep Data Management consistent, secure, and useful.

Data Management Principles

Now that you understand the lifecycle, you can use a few key principles to guide decisions at every stage. They give teams a shared reference instead of letting each system or department manage data in isolation.

The first fundamental principle already covered is considering data as an asset. From there, another relevant principle emerges: the value of data depends on its quality and context. Incorrect, incomplete, or misinterpreted data can lead to wrong decisions. For example, if a student's name contains a typo, it might not match what's stored in government databases, complicating certain processes. It's also crucial to understand that data needs metadata to be used correctly.

As I explained before, an isolated number like "18" has little value if it's unclear what it represents, in what unit it's expressed, how it was calculated, or when it was updated. Metadata documents this meaning and prevents ambiguities.

Another important principle is the need for planning. As seen in the lifecycle, the first step should be planning which data is expected to be used, among other things. In the case of enrollment, the university shouldn't collect just any student data, but only the relevant information required for the necessary processes.

Another essential principle is to use technology for a clear purpose. A team shouldn't choose a database or a new tool simply because it's the current popular tool. It should choose technology that addresses a real need. At the university, the decision to use a relational database, a Data Warehouse, a geographic API, or a dashboard should depend on the goal of the use case.

Data Management Capabilities

These principles become practical through a set of Data Management capabilities. The capabilities describe what an organization must be able to do with its data throughout the lifecycle.

The principles guide the work, while capabilities such as Data Governance and Data Modeling put that guidance into practice. The diagram above shows the main capabilities we'll cover in the coming sections.

Some Data Management roles work across several capabilities. One is the Chief Data Officer (CDO), who defines the organization's data strategy and helps ensure that teams manage data as an asset. In our use case, the CDO would help set goals for using data, such as improving attendance, enrollment, or student satisfaction.

Another relevant role is the Data Steward, who helps maintain data definitions, quality, and proper handling within a domain. At the university, they might verify the completeness and consistency of student location and enrollment data. A Chief Privacy Officer (CPO) may also be involved whenever a use of data affects privacy.

Data Governance

Let's start with Data Governance. Data Governance defines how an organization makes decisions about data, who may access or change it, and what responsibilities come with each role. It also helps the organization meet its legal and regulatory obligations.

You can see why this matters at the university: admissions staff, the academic office, faculty, and even AI systems may use student data. Without clear rules, people can gain inappropriate access or make decisions without enough justification.

Not everyone should be able to perform every action on every piece of data. Data Governance provides an organizational control layer across the lifecycle so people use data in an orderly, secure, and legitimate way.

Organizations assign this work to roles such as the CDO and Data Owners. Data Owners usually work within a business area and have authority to make important decisions about the data in their domain.

For example, the director of mobility at a university would be the Data Owner of all data related to the transportation service offered by the university. There may also be other Data Owners like the financial director for all billing and tuition payment information.

Governance tools help teams control, document, and review data use. Their main purpose isn't programming or technical processing.

A CDO or Data Owner might use data catalogs and business glossaries to understand what information exists and what it means. They may also use policy-management platforms, dashboards, and lineage tools that track data from its source to its destination.

Data Ownership

One of the key governance concepts is Data Ownership, which assigns responsibility for different data domains. Ownership doesn't mean that a person literally owns the data. It means that someone has the authority and accountability to make decisions about it.

The main role here is the Data Owner. This is usually a business leader who makes lifecycle and usage decisions for a domain rather than an end user who simply works with the data.

For example, the university may need a student's address to calculate the distance to campus. But the Data Owner of that domain should determine if that address can be accessed by their teachers or shared with an external transportation company, among other decisions.

The Data Owner usually doesn't implement the technical solution. Instead, they use tools such as data catalogs to find and understand the assets in their domain. A catalog organizes those assets through metadata and makes them easier to govern.

Data Stewardship

The Data Owner sets direction for a domain, while a Data Steward supports its day-to-day management. Data Stewardship includes maintaining definitions, monitoring quality, and helping ensure that data is accurate, complete, and handled according to agreed-upon standards.

In practice, a Data Steward might focus on verifying that students' dates and addresses are in a valid and consistent format, ensuring their names are complete, free of illegible characters, and without other issues. Also, this role emphasizes metadata to interpret data and allow other team members to do so without conflicts.

Data Stewards often work with data catalogs and business glossaries. A business glossary standardizes key organizational terms. For example, it might define "distance to campus" as the route distance in meters along public streets rather than a straight-line measurement.

Decision Rights

Another governance concept is Decision Rights: the formal definition of who can make which decisions about data in a given context.

Decision Rights form part of the foundation of governance. Organizations often classify decisions by their scope. Strategic decisions happen at the highest level, for example, when the university decides whether to use mobility data to offer a transportation service.

Then there are tactical decisions, which bridge the gap between the organization's overall strategy and day-to-day operations, such as defining eligibility criteria for candidates for the transportation service.

Finally, there are operational decisions, which are closest to the end users, like accepting or rejecting an enrollment application.

Decision Rights formally assign these choices to specific roles and data domains. The Data Owner and Data Governance Council are especially important here, with the council usually setting the broader decision framework.

A Data Protection Officer (DPO) may advise on a decision and escalate concerns when access would conflict with data-protection requirements. The DPO's exact authority depends on the applicable law and the organization's governance model. Teams often implement Decision Rights through workflow tools and Identity and Access Management (IAM) systems that manage digital identities and permissions.

For example, a university administrator shouldn't have unrestricted database access. They might open a ticket in a workflow tool like Jira to request a specific permission. The appropriate Data Owner reviews the request, and an IAM system such as Microsoft Entra ID grants the approved access to the administrator's verified identity.

Data Policies

While Decision Rights say who can make a decision, Data Policies state how people must manage and use data. They set the limits, principles, and obligations everyone must follow.

At the university, there might be a policy stating that user geolocation data can only be used to calculate eligibility for transportation services and not for other decisions unrelated to academic activities. This is an example of a policy related to privacy, data retention, or its use in AI models.

The Data Governance Council often formalizes these policies, the CDO sponsors them, and Data Stewards help teams apply them. A data catalog can publish the rules and connect them to the affected data assets, while technical systems enforce the controls.

Data Standards

Data Standards are more specific than policies. A standard might define a format, naming convention, or validation rule so teams follow a policy consistently across the organization.

For example, the university might establish that all dates be stored in the same ISO-8601 format or that the distance to the campus is always stored in meters. To better understand, a well-known case in computer science is the storage of decimal numbers, where the IEEE-754 standard is commonly used for binary representation.

Shared standards let systems exchange data with fewer unnecessary transformations. Data Architects and Data Modelers help select and define the standards, while Data Engineers apply them in the implementation. Data Owners and Stewards oversee their use within each domain.

Data Accountability

Data Accountability means that people who have authority over data must also answer for how it's used. Teams need enough monitoring and evidence to trace important actions and understand what happened over time.

If a problem occurs, the organization should be able to establish who accessed the data, when they accessed it, what they did, and whether the action followed policy. Evidence, traceability, and clear responsibilities make governance demonstrable.

At the university, a faculty member may have a legitimate reason to access part of a student's record, but the system should log the access when appropriate. If a privacy issue arises later, audit records can help investigators understand what happened.

The Data Owner is accountable for proper use within the domain, while security, compliance, and platform teams provide controls such as access logs, audit trails, and lineage where relevant.

Data Ethics

Governance alone isn't enough. An organization also needs to ask whether a use of data is fair, proportionate, and justifiable. That's where Data Ethics comes in.

In a data context, ethics applies principles such as transparency, responsibility, privacy, and non-discrimination throughout the lifecycle. This becomes especially important with personal or sensitive data because poor decisions can limit opportunities or deny people services.

For example, in a university, data handling during admission processes can result in discriminatory biases in many ways, some possibly unknown or unexpected. Notable among these are biases based on income, ethnicity, or disability.

Data can introduce these biases in numerous ways, which is why it's important to consider ethics and question whether data should be collected or used and what biases they might introduce.

Ethical Data Use

Ethical data use starts with a clear, legitimate, and proportionate purpose. An organization should know why it needs each piece of data, what value it expects, and what risks the proposed use creates.

Laws such as the General Data Protection Regulation establish legal requirements that overlap with some ethical principles, but legal compliance and ethical judgment aren't identical. The GDPR applies in the European context, and organizations must identify the rules that apply in every region where they operate.

An example of unethical use is when personal data from candidates entered into a form is sold to marketing companies without the candidates' explicit consent. Here, it's evident that personal data can be used to make decisions and improve a service or be used without consent for other purposes unrelated to the user's benefit.

Roles involved in ethical data use can include the CPO, DPO, a Chief Data Ethics Officer or ethics committee, and Data Stewards. Their exact responsibilities vary by organization. Consent management platforms (CMPs) can record and manage the permissions users grant, but consent is only one possible legal basis for processing and one part of ethical review.

Consent and Transparency

Consent and transparency are two important principles. Users should be able to understand what data is collected, why it's needed, how long it will be kept, who can access it, and whether it will be shared. These explanations should use plain language that a non-expert can follow.

In the case of a university, when a candidate applies for enrollment, the form shouldn't just request information and acceptance of terms. Instead, it should provide explanations about why each piece of data is requested. Clear explanations about how the data will be used and whether it will be shared with third parties should be given whenever possible.

Transparency doesn't end when a user submits a form. People should also be able to learn about their rights and use the processes available to request access or corrections when the applicable law provides them.

Fairness and Non-Discrimination

Fairness aims to prevent discrimination and harmful bias in the use of data. It matters especially in AI systems, where complex models and historical data can make bias difficult to detect or explain.

For example, a university might decide to award scholarships based on a candidate's zip code or area of residence. At first glance, this may not seem unjust, but in reality, people with very different incomes or academic records may live within the same zip code, and excluding entire areas could deprive qualified people of scholarship opportunities.

Data ethics requires teams to review their decision criteria. In practice, they may analyze bias, examine sensitive variables and their proxies, validate data quality and representativeness, and monitor outcomes over time. For consequential decisions, the organization should also provide suitable human oversight and a way to challenge errors.

Responsible Data Sharing

Organizations often need to share some data with service providers because they can't deliver every part of a service alone.

Sharing increases risk and needs an appropriate legal basis. That basis isn't always consent. For example, the university may need to share limited data with a taxi/VTC company to provide the service, but the company shouldn't receive the student's full record.

Whenever the use allows it, the organization should share anonymous or pseudonymous data instead of direct identifiers. Properly anonymized data can no longer be linked to a person by reasonably likely means. Pseudonymization replaces identifiers with codes or references, but an authorized party can still reconnect the data to the person using information kept separately, so the data remains personal and protected.

Ethical Risk Management

One practical way to support ethical data use is to assess and manage risk before a new use begins. The review should consider the expected benefits alongside possible harms, bias, privacy effects, and impacts on different groups.

For example, when designing the enrollment application form, before including a field to collect specific data like gender, income, or any other information, it's essential for an ethics committee to evaluate their usefulness, the problems that having this data might cause for students, and whether biases or discrimination could arise.

Data Ethics helps the university improve its services without losing sight of the fact that the data represents real people.

Data Security and Privacy

So far, we've looked at the rules and ethical choices that shape data use. We also need to protect data throughout its lifecycle. Security and privacy work together here, but they solve different problems.

Data Security uses policies, processes, and controls to prevent unauthorized access, alteration, disclosure, or loss. Privacy focuses on whether personal data is collected and used for legitimate purposes, with appropriate transparency and respect for people's rights.

Security commonly aims to preserve the confidentiality, integrity, and availability of data. Only authorized people should access or change it, and it should be available when needed. Those properties alone don't guarantee privacy. An address might be strongly secured, for example, but using or selling it for an unauthorized purpose would still violate privacy.

The university therefore has to address security and privacy together. It handles personal data whose exposure or misuse could cause real harm to students.

A Chief Information Security Officer (CISO) usually leads the security strategy and coordinates technical and defensive policies. The security team uses controls such as Identity and Access Management platforms to centralize identities, authentication, and permissions.

On the privacy side, the previously mentioned CPO helps oversee the organization's privacy program and compliance obligations.

Data Classification

We won't cover every part of security here. But a useful starting point is to identify what information exists and classify it by sensitivity.

Different data can cause very different levels of harm if exposed. A common classification scheme uses public, internal use, confidential, and restricted levels.

The first can be accessed by anyone, while internal use data is intended for organization members, though exposure wouldn't have a particularly severe impact. In contrast, confidential data requires authorization to be accessed, and restricted data needs the highest level of protection.

At the university, schedules published on the website would be public. Faculty work procedures might be for internal use. A student's academic record or travel history could be confidential, while banking information or credentials could be restricted.

When a dataset combines several categories, the organization should classify and protect the result according to the risk of the combined data, which may be as high as or higher than its most sensitive field.

The organization should record the classification as metadata in the data catalog so teams can use it throughout the lifecycle. When a Data Engineer integrates a source or an analyst creates a dashboard, they can see which precautions apply. Data Stewards often help classify the data, Data Owners approve the business decision, and security and privacy teams define the required controls.

Identity and Access Management

Once data is classified, the organization must control who can access it. Identity and Access Management (IAM) covers the processes and technologies used to manage digital identities and grant, review, or revoke permissions. Authentication verifies an identity, while authorization determines what that identity may do.

The fundamental principle guiding data access management is the principle of least privilege, according to which each identity receives only the permissions necessary to perform their job.

For example, an instructor can view the contact information of students enrolled in their courses but shouldn't access the information of unenrolled students. In other words, they have the minimum necessary permissions to perform their duties.

If the number of users to manage is high, it's most common to use Role-Based Access Control (RBAC)BAC), where permissions are associated with roles like instructor, administrative staff, or student, and then each user has a specific role.

As for the professionals responsible for these tasks, the Data Owners decide which roles need access to the data in their domain, while the IAM administrators implement the roles and their permissions with software like Microsoft Entra ID, an IAM technology that centralizes the management of identities, groups, and access policies.

Encryption

Access controls can fail, so organizations also use cryptography. Data encryption transforms readable information into ciphertext that an authorized system can reverse with the correct key.

This encryption should be applied both at rest and in transit, meaning when data is stored and when it is transmitted from one system to another over the network.

For example, the university should encrypt sensitive student data at rest so stolen storage doesn't reveal it in plain text without the required keys. Communications between a student and the university server should also use TLS through HTTPS to protect data in transit. Encryption is effective only when the algorithms, implementation, and key management are sound. Examples include:

Common approaches use symmetric and asymmetric cryptography, and both depend on strong key management. Keys shouldn't be embedded in source code or stored unprotected beside the data they secure. A Key Management System (KMS) or Hardware Security Module (HSM) can help generate, protect, rotate, and control access to them.

Security Architects and security specialists help select approved encryption standards, protocols, and key-management patterns, while Data Engineers and other developers apply them in each system. Encryption doesn't solve every security problem, so teams combine it with access controls, monitoring, secure development, and usage policies.

Data Masking

Many processes don't need to reveal a complete value. Data Masking transforms or partially hides data to reduce exposure while preserving enough utility for a specific task.

There are mainly two forms of masking. Dynamic Data Masking partially hides the information presented to the user without altering the original stored data. Thus, an authorized person can see the full value, while someone with fewer privileges sees a partial version like **1234.

On the other hand, Persistent Data Masking creates a permanently transformed copy, allowing systems to be tested without using real data.

For example, if the developers of the virtual campus need to test that the application works with thousands of students, subjects, and trips, they don't need to use real data. Instead, they can replace it with fictitious data, shifting dates, changing names to fictitious ones, and so on.

To better understand its purpose, here are some specific examples:

Masking, pseudonymization, and anonymization overlap in some implementations, but they aren't interchangeable. Masking alone doesn't guarantee that a dataset is anonymous. Pseudonymization replaces identifiers with codes while keeping the information needed to reconnect those codes to people separately. Because re-identification remains possible, pseudonymized data is still personal data and needs protection. Anonymization requires reducing identification risk to the point that people are no longer identifiable by reasonably likely means.

In this case, Data Stewards determine which data should be concealed and why, while security and Data Engineering teams implement these decisions at a low level.

Privacy Controls

The previous techniques help prevent unauthorized access. Privacy Controls address a different question: whether the organization has a valid purpose and appropriate rules for processing personal data.

The principles of Privacy by Design and Privacy by Default make privacy part of a system from the start and set privacy-protective defaults. One fundamental control is data minimization, which means collecting only what the stated purpose requires. An enrollment form, for example, shouldn't request a complete medical history unless a specific service and lawful purpose justify it.

Other controls apply to the purpose of the data and its retention. So in use cases, students' personal data shouldn't be kept longer than necessary or reused for other purposes like personalized marketing campaigns without authorization.

In Europe, the GDPR establishes principles and requirements that guide these controls. The organization must also identify the rules that apply in every region where it operates. The DPO monitors and advises on compliance where that role applies, while Data Owners, privacy specialists, security teams, and system designers turn the requirements into practical controls.

Audit and Compliance

The organization must be able to show that its controls and policies work. Auditing independently reviews the available evidence and tests whether controls operate as expected. Compliance covers the ongoing work of meeting internal policies, standards, contractual duties, and applicable regulations.

Logs are one important source of audit evidence. They can record who accessed data, when, from which system, and what action they took. Teams protect these records against tampering and retain them for a defined period based on risk, legal needs, and cost. Security Information and Event Management (SIEM) platforms centralize events from different systems and can generate alerts for unusual behavior.

For example, if a teacher occasionally checks the record of a student enrolled in their course, the behavior may be legitimate. But if they download hundreds of student records with whom they have no connection during the night and from another country, an alert should be generated for the security team to investigate the incident.

An audit might analyze logs, test whether identities have excessive privileges, and review how teams apply encryption and other controls. Independent reviewers and separation of duties help prevent the same administrator from controlling a system and the evidence used to assess their actions.

Roles involved include the CISO, the DPO, the Data Owners, the Data Stewards, and the compliance and audit teams. In summary, security and privacy require knowing what data exists, limiting who can use it, protecting it through controls, and preserving evidence that all of this is correctly followed.

Security Operations (SecOps)

Data security is ongoing work. Beyond policies and encryption mechanisms, SecOps (Security Operations) brings people, processes, and technology together for continuous defense.

SecOps teams monitor systems, detect threats, investigate alerts, and respond to incidents. They try to reduce risk early while staying ready to contain and recover from events that still occur.

In the university context, the SecOps team is responsible for overseeing the digital ecosystem in real time. For example, if a SIEM generates an alert because a teacher has downloaded hundreds of academic records at night or engages in any similar suspicious activity, the SecOps analyst receives the notification, assesses the risk, and takes action, such as temporarily blocking access as a preventive measure.

SecOps teams may also coordinate vulnerability scanning and remediation for the virtual campus and other systems so weaknesses are addressed before attackers exploit them.

In SecOps, key roles include SecOps engineers and security analysts, who work with the CISO to define and implement a defense strategy. These professionals rely on SIEM platforms to centralize event information and SOAR (Security Orchestration, Automation, and Response) tools to automate responses to common threats.

Data Architecture

Once you know who makes decisions about data and how to protect it, you still need to organize the systems that store, move, and process it.

Data Architecture designs the structure that meets those needs. Once the organization defines what it wants to achieve with data, the architecture shows how systems will store, transport, protect, and analyze it.

This capability connects business goals with technical implementation. It goes beyond choosing a database or sketching a pipeline: the design identifies which data the organization needs, where it lives, how it relates, and how it moves. The work can produce data models, flow diagrams, standards, and other architecture decisions.

In this use case, a candidate might enter their home address in a web form during enrollment. This data could then be sent to an admissions system and used in a query to a geographic API to calculate the distance to the campus, for example. It could also be used along with other data present in other systems, like class schedules, to verify eligibility if transportation service is requested.

Here, Data Architecture is responsible for designing how this complete data journey is carried out.

A poorly designed architecture can fail in several ways. Systems may exchange data incorrectly or stop communicating, interrupting a service for users. Even if nothing breaks outright, teams may duplicate data unnecessarily, raising costs and making integration harder. Good architecture reduces these risks and makes tradeoffs explicit.

The Enterprise Data Architect maintains the organization-wide view, while Data Architects and Solution Architects adapt it to particular solutions. Data Modelers, Data Engineers, Data Stewards, Data Owners, and security specialists contribute the design details and help put the architecture into practice.

In simple terms, architects design and document the solution, engineers and developers implement it, and Data Owners and Data Stewards clarify the meaning, rules, and responsibilities of the data.

Enterprise Data Architecture

The broadest level of data architecture is Enterprise Data Architecture, the organization-wide view of how data should be organized, connected, and governed.

At a university, Enterprise Architecture provides a comprehensive view of how systems should be coordinated, what each should do, and how information is exchanged between them.

For example, the web application through which a candidate completes a process must be properly connected with an admissions system or a database where that information is stored. This database or system can also support the operation of other internal systems dedicated to analyzing that data, or parts of it, according to privacy policies.

This work is led by the Enterprise Data Architect with support from the CDO, who aligns the architecture with the data strategy, and other roles like Application Architects or Security Architects. Additionally, Data Owners validate that the architecture meets the needs of their domains.

Data Domains

Data domains are an important part of an organization's architecture. Not all data describes the same part of the business, so teams group related concepts to make the data easier to organize, understand, and govern.

A Data Domain is a logical area containing related organizational concepts and data. A university might define domains for students, faculty, finance, and mobility. Grouping data this way makes its meaning clearer and helps the organization assign a Data Owner to each domain.

Additionally, a domain isn't isolated from others, as data often needs to be contextualized, even if it belongs to different domains. For example, the transportation service may require data from the mobility domain, as well as the schedule of its courses present in another domain.

Each governed domain should have a Data Owner with suitable decision authority. A Data Architect helps design the domain boundaries and relationships, which teams can represent in a conceptual model and document in a data catalog.

Data Flows

Once the domains and systems are clear, the team designs how data moves between them. Data Flows document the source, the systems and processes involved, the transformations applied, and the final storage or consumption point.

You can describe a flow at several levels. A high-level diagram may show data moving from one domain to another. An implementation view names the systems involved, while a more detailed design can show the fields, interfaces, and transformations that each consumer requires.

In the process of enrolling a candidate at the university, the main flow could be as follows:

1. The candidate accesses the enrollment portal and completes the form with their personal, academic, and contact information.

2. The enrollment portal validates the required fields and data format. Then, it sends the application to the admissions system via an API.

3. The admissions system creates the candidate's file and stores documents like the ID, academic degree, and certificates in a document database.

4. When the application is approved, the admissions system generates an offer that the candidate views and accepts through the enrollment portal.

5. The portal consults the academic management system to display courses, schedules, and available slots, allowing the candidate to select their options and confirm enrollment.

6. The payment system sends the transaction to an external payment gateway. The gateway returns the payment status, such as authorized, rejected, or pending. The university stores only a reference to the transaction and its result.

7. If the payment is successful, the academic management system creates the final enrollment and converts the candidate's file into a student file.

8. Next, the system updates the identity platform, virtual campus, and billing system. The student receives their credentials, payment receipt, and enrollment confirmation.

9. Finally, the necessary data can be pseudonymized and sent via a data pipeline to an analytics platform, where statistics on applications, admissions, payments, and enrollments are calculated and displayed on a dashboard.

Some data movements need near-real-time responses, especially in the transportation service, while others can run later in a batch. The flow should state those timing requirements.

The main role that designs the flow and determines which components participate is the Data Architect, while the Data Engineer implements it. But Security Architects also participate, reviewing data protection during the flow, and Data Owners authorize exchanges between domains. Finally, it's important to highlight the significance of data lineage tools for maintaining, monitoring, and auditing the flows.

Operational Data Architecture

The systems in an architecture serve different purposes. It's useful to distinguish between systems that run day-to-day processes and systems designed mainly for analysis.

The first group forms the Operational Data Architecture. This area covers the systems that keep an organization running each day. Online Transactional Processing (OLTP) systems handle frequent operational transactions and use controls that help preserve data integrity and consistency.

The university's operational architecture could include the virtual campus, application services, and a database. The portal would normally use an application or service layer rather than giving the user's browser direct database access. These components support the daily capture and management of data rather than long-running historical analysis.

That is, the operational database can serve as an authorized source to know the current status of enrollments, for example. However, it is not the most suitable place to continuously run complex queries over several years of activity to build statistics, as they could consume the resources needed for daily operations. Therefore, the data required to study trends, compare programs, or create dashboards is handled in another part of the architecture explained later.

For this type of information, it's common to use relational databases like PostgreSQL or MySQL. But you should choose the specific technology based on the volume of your operations, expected availability, existing infrastructure, and other requirements such as maximum response latency.

A Solution Architect or Data Architect designs the operational architecture, Software Engineers build the application components, and Data Engineers help define and implement the data exchanges between them.

Analytical Data Architecture

While operational architecture handles day-to-day activity, Analytical Data Architecture supports the integration, aggregation, and study of historical data. Its systems help teams create reports, discover patterns, and prepare data for AI models without placing unnecessary analytical load on operational services.

At a university, this architecture would be used to combine data on schedules, attendance, and budgets so an analyst can calculate the monthly expenses per master's program or the variation in student attendance over different periods. Similarly, a Data Scientist could use historical data to estimate future demand for transportation services, for example.

A typical analytical flow uses ETL or ELT (which we'll discuss more below) to obtain data from several sources. Teams then transform it before or after loading it into a specialized system such as a Data Warehouse. The result gives Business Intelligence tools and Machine Learning workflows suitable data without competing directly with the virtual campus for the same operational resources.

In this area, the Data Architect or Analytics Architect designs the analytical components of an architecture. Meanwhile, Analytics Engineers and Data Engineers design the processes that prepare data for analysis by Data Analysts or Data Scientists.

Cloud and Hybrid Data Architectures

Architecture also determines where components run: in the cloud, on premises, or across both. Cloud Data Architecture uses cloud computing, storage, database, and analytics services. These services can simplify scaling and reduce the need to manage physical hardware, but the organization still has to configure security, control costs, and govern its data.

On the other hand, a Hybrid Data Architecture combines on-premises systems with cloud services. This approach is common when an organization retains existing applications in its own data center but wants to use the cloud's elasticity or analytical services.

To understand the motivation for a hybrid architecture, in the case of the university, the academic system and the database with records and payments might initially remain in internal infrastructure to prevent third-party access to those data. But some pseudonymized data could be sent to cloud analytics platforms to obtain certain statistics on virtual campus usage or academic metrics.

Nevertheless, keeping certain data on-premises doesn't automatically guarantee greater security, just as using the cloud doesn't automatically mean a loss of control. The decision should consider data sensitivity, latency, availability, scalability, and the total cost of each solution.

In this design, the Enterprise Data Architect and the Data Architect participate, along with the Cloud Architect, who specializes in understanding cloud services to use them correctly in an architecture.

Network Engineers, Cloud Engineers, and Data Engineers also participate in its implementation, while the DPO and Data Owners must review issues like which data can leave the internal infrastructure and for what purpose.

Data Modeling and Design

Data architecture defines which systems manage data and how they exchange it. Data Modeling and Design specifies how those systems represent the information. It identifies the concepts that matter to the organization and describes their attributes, relationships, and rules.

A data model is a simplified representation of part of reality. It gives people a shared structure they can understand and later implement. Before creating the university's database, for example, the team needs to define what a candidate, student, master's program, and enrollment mean, which information each one needs, and how they relate.

Teams commonly describe a design at three levels:

1. a conceptual model with the main business concepts,

2. a logical model that adds detail without depending on a particular technology,

3. and a physical model that maps the design to structures in a specific platform.

Each model can evolve as the team learns more about the requirements.

The Data Modeler leads the design and works with the Data Architect to fit it into the wider architecture. Data Owners, Data Stewards, Business Analysts, and domain experts clarify meaning and rules. Database Administrators (DBAs), Data Engineers, and Software Engineers contribute to the physical design and implementation.

Conceptual Data Models

A conceptual data model gives you a high-level view of an organization's data. It shows the main business concepts and their relationships without technical details about storage or format.

For example, as shown in the diagram above, in a university, a conceptual model would include concepts like candidate, student, course, or enrollment. (Keep in mind that this is a sketch to help you better understand the concept of a conceptual model, not a diagram used in production.)

At this level, it's sufficient to indicate what each of these concepts is and what they can do in relation to others, such as a student requesting enrollment or an enrollment containing a set of courses. The goal is for both technical teams and academic leaders to understand the same reality before designing a specific solution.

This model is usually developed through interviews or workshops with Data Owners, Data Stewards, Business Analysts, and domain experts, who are generally not very technical given the nature of the task. In this process, the Data Modeler or Data Architect creates diagrams with the model and validates that the concepts match the business glossary.

Logical Data Models

A logical model develops the conceptual model in more detail while remaining independent of a specific technology. It defines entities, attributes, identifiers, relationships, cardinalities, and other business constraints.

For example, the Student entity might have attributes like ID, name, email, and address. A student can enroll in several courses, and a course can have many students. This many-to-many relationship could be represented at the logical level with an intermediate entity called Enrollment, which might include attributes like date, status, or academic year.

People often associate logical models with relational databases, but a logical model doesn't have to use that paradigm. Think of it as a technology-independent specification of the information and its connections, even though different paradigms represent entities and relationships in different ways.

These relational models can be refined. For example, in a relational database, its logical model can be normalized to reduce duplications and incorrect dependencies. But in other paradigms or solutions, there will be very different procedures. And the design of this model is led by a Data Modeler, in collaboration with a Data Architect, as mentioned earlier.

Physical Data Models

The physical data model maps the logical design to a specific technology. In a relational database, for example, it turns logical entities and relationships into tables, columns, keys, constraints, partitions, and lower-level structures such as indexes, which often use a B-tree.

At the university, student records could live in a relational table. The DBMS decides how to store the table itself, while the team can create indexes, often B-tree indexes, on selected columns to speed up common queries.

As you can imagine, the same logical model can generate different physical models. For instance, the academic system could be implemented in PostgreSQL or MySQL. So the physical design must consider the DBMS intended for use, data volume, query patterns, security, availability, and operational cost to provide an effective solution.

In this design phase, the Data Modeler or Database Designer, the Data Architect, and the Data Engineers primarily work together with the Software Engineers to implement the solution.

Entity-Relationship Modeling

Entity-relationship diagrams are a common way to represent relational concepts. Depending on how much detail they contain, they can support conceptual or logical modeling. Entities are typically shown as rectangles, while lines represent relationships, cardinality, and optionality.

For example, a student can have many enrollments, and each enrollment belongs to a single student. In contrast, a relationship between Student and Course would be many-to-many because a student can be enrolled in many courses at once.

Keys are also identified to distinguish each instance of an entity and maintain the integrity of their relationships, among other details that aren't as relevant here.

If you're curious, you can read more about database design in my previous book here.

Dimensional Modeling

Another useful approach, especially in Data Warehouses and analytical systems, is dimensional models. These models organize data around facts and dimensions. Facts record measurable events, while dimensions provide the context used to analyze them.

For example, in a transportation service, you might have a fact table called Trip, containing a row for each completed journey, recording measures such as cost, distance, and duration. But instead of storing the traveler's data in the same table, it relates to others representing dimensions like Student, Date, or Transportation Provider. Thus, the fact table models the existence of trips, while other dimensional tables contain specific data for each trip, such as the person or transportation provider, resulting in a structure known as a star schema.

This type of model is primarily used because it simplifies analytical queries and allows studying the same fact from different "perspectives." For example, the university could calculate the total cost of trips by month, student, or provider without having to construct excessively complex queries.

Data Model Governance

Data models also need governance so they stay consistent, current, and aligned with the implementation. Teams should maintain the connection between conceptual, logical, and physical designs as each one changes.

Once teams approve a model, the implementation should follow it or update it through a controlled change. Unexpected differences between an expected and an actual schema are commonly called schema drift.

For example, the university's model might define a numeric age field while the implementation stores it as text. That difference may look small, but downstream systems can fail if they rely on the agreed type. Teams should detect and control schema changes so models, contracts, and implementations stay aligned.

A Data Governance Council or Architecture Review Board may review significant model changes. Data Owners confirm that the design reflects business rules, while database and engineering teams implement approved changes through a controlled process.

Data Storage and Operations

Data models guide the implementation of systems that store data persistently and make it available to applications and other systems. The team has to choose an appropriate storage technology, keep the data accessible when needed, and operate the system at an acceptable level of performance.

Data Storage and Operations covers the design, implementation, and operation of storage systems throughout their lifecycle. This includes choosing databases, file systems, and object stores, then maintaining, monitoring, and optimizing them. As you'll see, a database isn't the right home for every type of data.

Data architecture determines which systems the organization needs and how they communicate. Data modeling specifies how they represent information. Data Storage and Operations turns those designs into working storage systems. A physical model might say that the Student entity maps to a PostgreSQL table with a B-tree index on student_id. This section focuses on implementing and operating that kind of design.

The main objectives of data storage are to maintain availability, integrity, and ensure good performance of the underlying system. To achieve these, you shouldn't always use one technology for all the data in an organization, as the data for an enrollment or a class video, for example, has very different structures, uses, and requirements. So the same organization often combines different storage systems.

The team should choose the technology based on its expected volume, access patterns, sensitivity, availability, cost, and other requirements. Every additional technology increases operational complexity, so each one should solve a real problem.

A Database Administrator creates, configures, secures, tunes, and maintains databases. Storage Administrators manage the underlying storage, while Site Reliability Engineers and platform teams monitor services and respond to reliability incidents. The exact division of work depends on the platform and organization.

Databases

A database is an organized collection of data that applications can store, change, and query. A Database Management System (DBMS) is the software that manages databases and provides services for querying, concurrency, security, recovery, and administration. PostgreSQL is a DBMS. The university's academic database would be a particular database managed by a PostgreSQL server or service.

Operational systems often need transactional support, especially for workflows such as enrollment and payment. A transaction groups related operations into one logical unit. The ACID properties (Atomicity, Consistency, Isolation, and Durability) describe guarantees that help applications preserve valid state despite failures and concurrent access.

For example, when making a payment, values must be modified in multiple places corresponding to the users exchanging money. Thus, the atomicity of a transaction allows confirming all these modifications together, and if any fail, reverting them to maintain the previous state.

Database designs make different tradeoffs among data model, scale, consistency, latency, and access patterns. That's why several database paradigms exist:

These are only a few database paradigms. A university could use PostgreSQL for an academic system that manages Student, Enrollment, and Course records through tables and relationships. For a specialized route or network analysis, a graph-oriented database could represent locations as nodes and connections as edges. The operational taxi service itself might still use a relational or other transactional store, depending on its access patterns.

The Data Architect and Data Modeler select the database paradigm and design with input from the engineers who will build and operate the solution.

Once operational, the database is maintained by a Database Administrator. Before this, a Database Engineer will have implemented the physical model, created instances, schemas, tables, and other necessary elements to subsequently operate the environment. Software Engineers develop the applications that access these databases and perform queries.

File and Object Storage

Not all data fits naturally in a database. Universities manage diplomas, identity documents, and large files such as class recordings. A DBMS can store binary content, but file or object storage often provides more suitable access, scale, and cost characteristics for these assets.

File Storage organizes files into directories and exposes them through paths and protocols such as NFS or SMB. Teams can implement it with a Network Attached Storage (NAS) system or a cloud service such as Amazon EFS or Azure Files.

Object Storage stores content as objects with identifiers and metadata, usually inside buckets or containers. Its namespace and access model differ from a mounted hierarchical file system, even when tools display folder-like prefixes. Services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage can hold large collections of documents, images, and videos.

The main difference is the access model. File Storage behaves like a shared file system, while applications usually access Object Storage through an API using an object key and metadata.

For example, the university could use File Storage to save administrative documents for each student, like registrations and certificates, in a shared folder. This way, authorized staff could manage them as if they were in a traditional file system.

On the other hand, it could use Object Storage to store a large number of class recordings, images, and multimedia materials in a bucket. Instead of locating a video by navigating folders, the system could retrieve it directly using its identifier or by filtering through its metadata.

The roles responsible for configuring and operating these systems are mainly Storage Administrators, Cloud Engineers, and Platform Engineers, while Software Engineers implement access to these systems from other applications.

Data Warehouses

Operational databases are usually optimized for current transactions and application queries rather than repeated analysis across years of integrated history. Complex analytical workloads can also compete with the applications using the same resources. Organizations therefore often copy suitable data into a separate Data Warehouse.

A Data Warehouse is an analytical repository that integrates data from multiple sources and organizes it for repeatable analysis, reports, and dashboards. These systems support Online Analytical Processing (OLAP) workloads that scan and aggregate many records, in contrast with the Online Transactional Processing (OLTP) workloads common in operational applications.

This difference often affects storage design. Many Data Warehouses use columnar storage because an analytical query may scan a few columns across a large number of rows. To calculate the total cost of taxi rides by date, for example, the engine may only need the cost and date columns.

Many operational relational databases use row-oriented storage because it efficiently retrieves or changes complete records. These are common patterns rather than universal rules. Specific products can support several storage formats.

In practice, the university could have a database and a pipeline where data is periodically extracted to be inserted into a Data Warehouse. There, a dimensional data model could be applied as seen earlier to analyze the data and allow an analyst to answer questions like:

• What's the average monthly cost of a certain course per student?

• How has in-person attendance changed over a specific period?

• How many students enrolled last month?

It's important to understand that a Data Warehouse doesn't replace a database. Rather, it's an auxiliary system focused on data analysis. Among the technologies available for these types of systems are cloud platforms like Snowflake, Google BigQuery, or Amazon Redshift.

The roles that work with them include Data Architects or Analytics Architects, who design the analytical platform, while Data Engineers design the pipelines to extract and load the data.

Data Warehouse Administrators or Platform Engineers manage performance, permissions, reliability, and cost. Data Analysts and Business Intelligence professionals query the governed analytical data without changing the operational source records.

Data Lakes and Lakehouses

A traditional Data Warehouse applies defined schemas and organizes data for known or anticipated analytical needs.

But this isn't always the case, as an organization might also need to retain original files, semi-structured data, logs, images, or events whose future use isn't yet fully defined.

For these situations, we can use a Data Lake, which is a repository designed to store large amounts of data in their original formats or with minimal transformations.

A Data Lake also supports analytical and data-processing needs, but it can retain structured, semi-structured, and unstructured data with fewer transformations at ingestion. It's often associated with schema-on-read, where a query or processing job applies part of the structure, while a traditional Data Warehouse commonly uses schema-on-write before loading curated data.

Schema-on-read doesn't remove the need for metadata, security, quality, and governance. Without them, the lake can become a data swamp.

To understand how information is organized in a Data Lake, in the university's use case, the data could be processed in layers according to their readiness for consumption.

1. In a specific area of the system, data could be kept in their original formats without modification, such as CSV or JSON files. This would allow for reprocessing the information if an error in a transformation is detected later or if another type of analysis is needed.

2. In another area, the data could be in a different format, or the same format but with certain transformations applied to remove invalid records or standardize units of measure, for example.

3. In a curated area, teams could apply further quality checks and transformations until the data meets the requirements for dashboards, with selected statistics pre-calculated.

This separation doesn't imply that all original data is always retained indefinitely, as privacy, security, and retention policies must be followed.

For example, the university may temporarily store documents submitted by a candidate during the admission process. But if the candidate is rejected and enough time has passed, the university must delete those documents, even if derived and anonymized data have been generated to compile statistics on the admission process.

Lakehouses add capabilities such as transactions, schema enforcement, and table management to the flexible storage commonly used for a Data Lake. They can let several analytical workloads share one data foundation, although they don't eliminate every reason to use specialized systems.

Among the technologies used to build a Lakehouse are Delta Lake, Apache Iceberg, and Apache Hudi. They define the data format usually stored on services like Amazon S3, Azure Blob Storage, or Google Cloud Storage and processed using tools like Apache Spark, Databricks, or Trino.

In the case of the university, a Lakehouse could be used to store student data, enrollments, attendance, and taxi rides in one place. This way, the university could securely update this data and use it directly to create reports, such as monthly transportation expenses or the number of students attending classes, without needing separate systems.

Finally, those responsible for designing and implementing data ingestion from different sources in these systems are the Data Engineers. On the other hand, Platform Engineers manage the infrastructure, and Analytics Engineers, along with Data Scientists, consume the data to conduct relevant analyses and research.

Backup and Recovery

Even a well-designed storage system can suffer hardware failures, software defects, corruption, mistakes, or attacks that cause data loss. That's why Backup and Recovery is essential in production.

A backup is a recoverable copy of data kept for loss or corruption scenarios. A backup is useful only if the organization protects it, verifies it, and tests the recovery process. Common mechanisms include:

• Full backup: Copies the entire dataset. For example, the university could perform a complete weekly copy of the enrollment database. It simplifies restoration, though it requires more time and storage.

• Incremental backup: Saves only the changes made since a previous copy. After a monthly full backup, only the modified enrollments could be copied daily. It reduces volume, but recovery may require several linked copies.

• Snapshot: Captures the state of a storage system at a point in time. Depending on the technology, it may share underlying storage and may not be an independent copy. The university could take one before a major academic-system change, while still keeping separate backups for stronger protection.

• Log backup: A backup that relies on a change log, allowing recovery of the database to a previous point in time if data is accidentally deleted. It's more precise but requires maintaining the entire log sequence.

• Replication: Maintains a replica of an entire system that can take over if the main system fails. For example, a secondary database could continue serving the enrollment portal, improving availability. But it can also replicate deletions or errors, so it doesn't replace a backup.

A recovery strategy uses two common objectives. The Recovery Point Objective (RPO) expresses the maximum tolerable data loss in time, while the Recovery Time Objective (RTO) states how long service restoration may take before the impact becomes unacceptable.

For example, the university might hypothetically set an RPO of five minutes and an RTO of one hour for the enrollment database during the registration period. This would mean that, in the event of a serious failure, they aim to lose a maximum of five minutes of operations and restore service within an hour. In contrast, a collection of already published videos might allow for a slower recovery if durable copies exist elsewhere.

A well-known practice in designing backup solutions is the 3-2-1 rule, which involves maintaining three copies of important information, using at least two storage media or technologies, and keeping one copy offsite. But you should tailor your solution to the requirements of your organization.

The Data Owners and business leaders are responsible for identifying critical processes and determining acceptable loss or interruption. On a technical level, a DBA implements and validates the database recovery mechanisms. Additionally, Storage Administrators and Cloud or Platform Engineers manage storage and automate backups, while Site Reliability Engineers monitor and conduct tests to ensure recovery functions as expected.

Retention and Archiving

An organization shouldn't keep every piece of data indefinitely. Doing so raises costs, complicates discovery, and increases the impact of a breach.

A retention policy should state how long data stays active, when it moves to an archive, and when it is deleted or anonymized. The policy should reflect business needs, contractual duties, legal requirements, and applicable holds.

In this context, it's important to distinguish between two concepts:

• Archive: Stores information that's no longer regularly used but must remain accessible. For example, a former student's record might be moved to an archive with lower storage and retrieval costs, in case it's needed to verify their existence when requesting a certificate.

• Retention: Defines how long data is kept and what happens when that period ends. For example, the personal and academic documentation of a rejected applicant might be retained until the admission process and the appeal period are over. Afterward, those documents would be deleted, although the university might keep anonymous statistics on the number of applications received.

Data Owners, Records Managers, legal counsel, and privacy specialists help establish retention periods. A legal hold can temporarily suspend normal disposal for information related to an investigation or proceeding. The organization therefore needs a documented reason to keep or delete data rather than deciding only by whether it seems useful.

Afterward, Data Stewards classify the data, and DBAs, Storage Administrators, or Cloud Engineers implement the policies. As an interesting technology, Write Once Read Many (WORM) storage is often used for records that must remain unalterable.

Performance and Availability

Stored and protected information must be available when the service needs it and perform within its agreed targets. Performance describes qualities such as response time and throughput, while availability measures whether the expected service can be used.

A system can be technically running yet unusable if it responds too slowly. It can also be fast when online but fail its availability target because of frequent outages. Teams need to manage both qualities.

Some techniques that can improve the performance of a storage system include:

• Create indexes on frequently queried fields, after ensuring they justify the space cost of the index itself.

• Analyze the most frequent queries or workloads to try to optimize the query plans generated by the DBMS.

• Introduce caches whenever possible, especially when results will be needed multiple times.

Performance work depends on the system and workload. Adding hardware won't fix every problem, as software design matters just as much. Unnecessary pipeline transformations, for instance, increase execution time and cost even when they don't cause an outage.

On the other hand, redundancy is often used to improve availability. Essentially, if there are replicas of the same server or system, it's less likely that all will fail simultaneously, leaving end users without service.

You can manage the existence of replicas with failover mechanisms, so if a PostgreSQL instance, for example, stops working, you can redirect traffic to another replica automatically and transparently for the end user.

In the university example, during the last days of the enrollment period, thousands of students might access the portal simultaneously. To maintain good performance, requests would be distributed among several servers, preventing any single one from becoming overloaded and reducing wait times. Also, the database could have replicas so that if one instance fails, another can automatically take over.

This way, the system would remain fast during high demand and stay available even in the event of an unexpected failure.

To measure an organization's performance and availability objectives, observability is especially important. This involves generating metrics, logs, and statistics, and managing them with tools like Prometheus and Grafana to monitor the system and check its availability and performance at any given time.

This analysis and optimization of a storage system is usually performed by the DBA, although certain Software Engineers and Data Engineers may also be involved, optimizing the data pipelines through which various systems exchange information. Regarding availability, SREs, Platform Engineers, and Cloud Engineers automate deployments, monitoring, scaling, and implement failover mechanisms.

Document and Content Management

So far, we've worked with several kinds of data: structured records in tables, semi-structured data such as JSON, and unstructured content such as scans, images, videos, and free-form text.

Documents can contain a mix of structured metadata and unstructured content, so they need their own management practices.

A document usually doesn't follow a rigid row-and-column structure, but it can still have metadata such as a title, author, type, date, or tags. Some digital formats also contain an internal hierarchy. A JSON document, for example, uses named fields and nested objects:

{
  "student_id": "ALU-2026-8942",
  "full_name": "Amélie Dubois",
  "master_program": "Master in Artificial Intelligence",
  "campus_distance_km": 18.2,
  "rideshare_benefit_approved": true,
  "last_trip": {
    "date": "2026-03-09",
    "cost_euros": 24.50
  }
}

Many digital files combine content with descriptive metadata such as a title, author, or creation date. That metadata makes the content easier to identify, organize, secure, and retrieve. Document and Content Management provides the processes and systems for doing this consistently.

Simply placing files in folders isn't enough at organizational scale. Teams need ways to classify documents, describe their content, control access, track versions and retention, and find them later. A basic file system or database can be part of the solution, but a document or content platform adds the management features the organization needs.

For example, the journey of a document in the university systems might be:

1. The candidate's academic record is captured from a form, an email, or any equivalent means.

2. It's indexed and metadata is added to provide context.

3. It's stored in an appropriate repository.

4. Authorized users and systems can access or share it under the applicable controls. For example, an admissions analyst might query approved extracted fields to count candidates with prior study in a subject area without opening every certificate manually.

5. Finally, it's deleted or retained according to applicable policies.

Unstructured Data

An important part of the data managed by an organization contains unstructured information. This means that, as mentioned above, its content isn't rigidly structured in clearly identifiable and directly queryable fields. For example, a motivation letter in PDF, a scanned image of a diploma, or a contract may contain information that's difficult to structure.

Documents may have format-specific metadata such as a title or creation date. This helps identify the file but rarely describes everything inside it. The body may contain free-form text, images, tables, or other content that the system must extract or index before it can answer detailed queries.

To perform queries on this information, the system indexes this content or applies techniques like Optical Character Recognition (OCR), Natural Language Processing, or Intelligent Document Processing.

For example, if the university wants to know how many candidates have taken math-related courses before entering the master's program, it must first extract that information from academic certificates, normalize it, and store it in queryable fields. When extracting data from a document, you should maintain a link to the original document to verify its source later.

After extracting useful content, the system can index it in a structure optimized for search. The index may represent a document with fields or key-value pairs such as the candidate identifier, document type, courses taken, and subject area.

{
  "index_id": "idx_cert_2026_0042",
  "student_id": "ALU-2026-8942",
  "student_name": "Amélie Dubois",
  "document_type": "Academic Transcript",
  "extracted_subjects": [
    {
      "original_name": "Algèbre Linéaire",
      "normalized_area": "Mathematics",
      "score": "18/20"
    },
    {
      "original_name": "Introduction à Python",
      "normalized_area": "Computer Science",
      "score": "16/20"
    }
  ],
  "metadata": {
    "issuing_country": "France",
    "language": "fr",
    "confidence_score_ocr": 0.98
  },
  "original_file_url": "https://s3.uni.edu/bucket-cert/2026/8942_transcript.pdf"
}

For example, above you can see what an indexed document might look like. Originally, it could be an academic certificate of a candidate, but for the system, it's a JSON dictionary with this information, meaning the internal content of the document is organized hierarchically.

Representing it this way makes it much easier to perform queries, as you can navigate and access fields like score to see each candidate's grades in the various subjects they've taken at another university.

Document Capture

The first operational step is Document Capture, the controlled process for accepting a document into the organization's systems.

In these processes, it's important to consider the format of the document to be captured, as they're not always digital files. Often, they can be physical documents delivered to an administrative body, which then needs to digitize and upload them to the system.

In any case, assuming a digitized document reaches the data management systems, an adequate capture should perform at least the following actions:

• Validate the file format and size, and ensure it doesn't contain malicious software.

• Assign it an identifier and basic metadata, such as its origin and date of receipt, along with a digital fingerprint like a hash to detect changes in the file.

• Preserve the original and, when necessary, extract a usable representation of its content.

If a document is scanned, its text appears as pixels rather than directly searchable characters. OCR converts visible text into machine-readable text. More advanced Intelligent Document Processing (IDP) systems can also classify documents and extract fields, tables, and layout using rules and Machine Learning models.

For example, a candidate might upload a photo of a diploma issued in another language from their phone. The capture process would detect the language, extract all the corresponding text using OCR, and associate the file with their application so that the document's content can later be reviewed, knowing to whom it belongs.

Document Classification

After capture, the system may need to classify the document so it knows what it is and which workflow, access rules, and retention policy apply. People can do this manually, or software can assist with rules and Machine Learning.

In some workflows, the university may let users attach certificates, reports, and other supporting files. The system can't trust the filename or assume that every upload is safe. It must validate the file, scan it according to security policy, and identify the document type before further processing.

A filename alone isn't reliable: A.pdf could contain almost anything. Classification assigns one of the organization's defined document types and determines the next processing steps. Teams may automate low-risk cases and route uncertain or consequential cases to a person for review.

Content Storage

After capture and classification, the organization stores the original document and its metadata. Object or file storage often holds the binary file, while a document database such as MongoDB, Couchbase, or Amazon DocumentDB may hold flexible metadata or extracted content. The right combination depends on access, retention, search, and scale requirements.

Other alternatives include using a Document Management System (DMS) or a platform with Enterprise Content Management (ECM) capabilities. These document repositories are based on File or Object Storage internally, with additional capabilities that a bucket or folder alone cannot provide, such as advanced metadata management. Lastly, it's worth mentioning the existence of Content Management Systems (CMS), which are designed for creating and publishing content on websites.

Search and Retrieval

A document is useful only if authorized users and systems can find it when needed. After storage and indexing, the platform may support several search methods:

• Metadata search: Filters by fields such as document_type = Academic Certificate.

• Full-text search: Finds words or phrases in extracted text and ranks the matching documents.

• Semantic search: Retrieves documents by meaning, even when they don't contain the exact words in the query.

For example, an authorized employee could search for a certain teacher's employment contract using keywords like "contract" or the person's name, even if they don't remember the exact file name. Alternatively, with a semantic search like the one we can perform on Google, they can also locate that document or any other based on the meaning of its content.

Records Management

Not every document has the same value or lifecycle. Teams may discard drafts quickly, while official evidence of an activity or decision must be preserved as a record. Records Management controls those records throughout their required lifecycle.

Unlike a draft, a record is an official document that must be preserved and kept authentic, complete, and protected. For example, a draft of an admission offer would be disposable, while the accepted and signed offer by the student becomes a record.

Each type of record has an associated retention period that determines how long it must be kept and what should be done afterward. If there's an investigation or legal proceeding, a legal hold may be applied, temporarily suspending its disposal. At a university, official course records or final academic transcripts might be considered records.

Overall, the most common technologies and roles in document management can be summarized as:

Reference and Master Data Management

Organizations reuse some data across many processes and systems. The same student may appear in the admissions platform, virtual campus, and billing platform. If each system represents that person differently, duplicates and contradictions quickly appear.

Reference and Master Data Management coordinates this shared data so systems can use consistent, trusted values.

First, you need to distinguish between:

• Master Data: This describes an entity that is relevant and shared by several processes. For example, the record of the student Amélie Dubois.

• Reference Data: These are allowed values within a classification or organization of the master data. For example, APPROVED can represent the status of an accepted enrollment application, with the candidate's record considered master data.

The goal isn't to force every piece of data into one database. It's to identify trusted values and systems of record, define who maintains them, and distribute the right representation to each consumer.

Master Data

Master Data represents core entities such as people, organizations, places, or products. At a university, it might include students, faculty, and courses. A trusted student record could contain a global identifier, name, and selected contact attributes, while sensitive payment details remain in the systems that need them.

But payment information won't be used in all processes involving these data. This is why authorized data needs to be distributed to each system so that the entire organization has a consistent view of the data, even if it's used differently.

Not all attributes of a record have to come from the same place. A payment platform may maintain its fiscal information, while the student portal keeps the most recent contact email. Then, a Master Data Management (MDM) platform would integrate these sources to provide a reliable view to other systems.

Platforms used for this purpose include Reltio, SAP Master Data Governance, and IBM InfoSphere MDM. The role that operates them is the MDM or Data Architect, who defines the data model and the architecture used for deployment, while the MDM Engineer configures the platform. Data Engineers and Integration Engineers need to be aware of these authorized sources of truth.

Reference Data

Reference Data supplies controlled values used to classify or organize other data. The university might allow a transportation request to have the status PENDING, APPROVED, or REJECTED. If applications use different terms for the same state, integration and reporting become unreliable. These approved status values are Reference Data.

These values usually change infrequently but aren't immutable. This can happen because new values need to be added to the classification, like CANCELLED.

To make this modification, a Data Steward would document its meaning, while the Data Owner of the corresponding data domain approves the change. Subsequently, the Integration Engineers are responsible for distributing the new value to the systems that consume it.

Golden Records

Information about one entity often appears in several systems, with each system storing what it needs. An MDM platform can combine selected trusted attributes into a unified view called a Golden Record. The goal is a governed, useful representation, not a copy of every piece of information the organization holds.

For example, the university might have an admissions system where a student's personal data, like the name Amelie Dubois, is stored, while their payment information is in a system specialized for processing payments. After verifying they belong to the correct person, they can be linked to provide a single view of the student.

A Golden Record isn't automatically perfect or permanently definitive. It's the best trusted view available under the current matching and survivorship rules.

Entity Resolution

To build that view, the platform must decide which records refer to the same real-world entity. This task is called Entity Resolution.

For example, records named Amélie Dubois and A. Dubois might refer to the same person, or to different people. A resolution process can compare authorized attributes such as email, phone number, or date of birth and apply deterministic rules or probabilistic matching. Because false matches and missed matches can cause harm, teams should review uncertain cases and provide a way to correct decisions.

This is assigned to the MDM Engineer, while the Data Quality Analyst analyzes and supervises the results along with a Data Steward. It's implemented through the functionalities incorporated in MDM platforms, services like AWS Entity Resolution, or record linkage libraries like Splink.

Deduplication

Another issue that drives the need for Entity Resolution is the presence of duplicate data. For example, a candidate might register on the virtual campus with one email and later apply for admission using another. If it's confirmed that both records belong to the same person, they should be handled appropriately in each specific scenario.

This process is called Deduplication and involves using Entity Resolution to detect and manage repeated records, aiming to prevent them from being treated as independent entities. Common approaches include linking, which retains the records in their original systems and creates a correspondence between their identifiers. Alternatively, merging generates a consolidated record, similar to the Golden Record.

Here, responsibilities are divided among several roles. The Data Owner sets the criteria guiding the Deduplication process, the MDM Engineer implements these criteria on the platform, and the Data Quality Analyst, along with the Data Steward, supervises the outcome of the process.

Survivorship Rules

When several source records refer to the same entity, the MDM process must decide which value to use for each attribute in the Golden Record.

Previously, we saw this with the example of the student name Amélie Dubois and A. Dubois, values that may appear in several records. Thus, when creating a Golden Record, it will be necessary to decide which one to keep.

For this, there are Survivorship Rules, which, as their name suggests, are rules that determine the resolution of these situations based on the data involved.

These criteria are designed by a Data Owner, while a Data Steward supervises the process and its application, and an MDM Engineer implements these rules on a platform.

Metadata Management

In the previous section, document metadata helped identify us a file and describe details such as its type or creation date. But metadata applies far beyond documents.

Metadata is data that describes other data. The number 42 is ambiguous by itself. A column name such as age, a unit, a definition, and a timestamp can tell you what it represents and how to interpret it.

At organizational scale, metadata needs deliberate management of its own. Metadata Management collects, connects, maintains, and publishes metadata so people and systems can find and use data correctly.

The goal is to make data understandable and support governance, quality, security, and discovery. People create some metadata manually, while scanners and integrations can collect technical or operational metadata from systems and files. A metadata repository connects these descriptions, and a data catalog makes them available to users.

The CDO and Data Governance Council can set the metadata strategy and governance model. A Metadata Manager or Metadata Engineer operates the platform, while Data Owners and Data Stewards maintain definitions, ownership, and other domain metadata.

Business Metadata

Metadata includes more than column names and file properties. Business Metadata explains data in the language and rules of the organization.

It includes documented definitions, business rules, ownership, and usage constraints. The university might define an "Enrolled Student" as a student with at least one active course enrollment, then specify what "active" means. That definition is business metadata.

The knowledge used to generate the definition is provided by a Business Analyst, who, together with a Data Steward, turns it into a clear and consistent definition.

Technical Metadata

Technical Metadata describes how systems represent data and where it's located. It includes schemas, data types, table and column names, paths, file formats, keys, and interfaces.

For example, in a catalog, it might indicate that a student's address data is located in a certain table attribute, is textual, and doesn't allow null values. All this information is considered metadata because it describes where the data is and how it's represented.

At this level, Data Architects or Data Modelers typically define the data representation so that Data Engineers, Analytics Engineers, and Database Administrators can handle its implementation.

Operational Metadata

Operational Metadata records what happens when systems process or use data. It can include job start and end times, row counts, query activity, freshness, status, and failures.

For example, at the university, it might be recorded that the enrollment request pipeline ran at 6:00 AM, processed 543 students, and completed successfully in 20 seconds.

This metadata is often obtained from orchestrators like Apache Airflow, application logs, and cloud platforms, which are operated by Data Engineers and DataOps or platform professionals who monitor these executions.

Data Catalogs

A Data Catalog is one of the main systems used to bring these metadata types together.

A Data Catalog is a searchable inventory of the organization's data assets. It usually stores metadata and references to source systems rather than copying all the underlying data. Its main purpose is discovery and understanding, although some catalogs also support access-request and governance workflows.

For example, if an analyst is looking for enrollment records from the past 6 months, the catalog should indicate which database or storage system holds that information, who's responsible for it, other metadata like the name of the system or table where it is located, and the access rules.

Among the most well-known commercial solutions are Collibra, Alation, and Microsoft Purview, often deployed on cloud ecosystems like AWS Glue Data Catalog and Google Cloud Knowledge Catalog. Management is handled by the Metadata Manager or Metadata Engineer, who administers this platform.

Business Glossaries

A Business Glossary is a controlled vocabulary that establishes the official meaning of the organization's concepts. It shouldn't be confused with a data dictionary: the dictionary describes tables and columns of a specific system, while the glossary defines business concepts that may be implemented in many systems.

For example, the term Completed Trip might mean a trip that has reached its destination and whose billing has been validated. This definition prevents the mobility area from considering a trip complete when the journey ends, while finance only does so when the invoice is received. The term should include its definition, synonyms, rules, related concepts, owner, steward, and approval status.

A business expert or Business Analyst proposes the term, the Data Steward reviews its clarity and potential conflicts, and the Data Owner approves its use. The glossary can start as a simple document, but as it grows, you should manage it within the data catalog to link each term with its columns, rules, reports, and policies.

Data Lineage

Data Lineage describes where data came from, how it moved, which transformations changed it, and where it's consumed.

At the university, lineage could show that an address enters through an application, passes to a geographic API, produces a route distance, and contributes to a mobility-eligibility decision. A separate operational flow may then share only the minimum trip details with the transportation provider. This metadata helps teams assess the impact of changes, investigate errors, and demonstrate how a result was produced.

Data Engineers, Analytics Engineers, and Metadata Engineers help capture lineage through tools such as dbt, OpenLineage, or Apache Atlas. Automation can collect lineage from supported systems and generate visual paths from sources to dashboards, but teams still need to validate gaps, semantics, and manually implemented processes.

Metadata Standards

Metadata also needs standards, quality controls, and governance. Metadata Standards define how teams document, represent, and exchange it.

The goal is to help people and systems locate, understand, integrate, and exchange data consistently. ISO-8601 is a data representation standard for dates and times. Within an organization, snake_case might be a metadata naming convention, while a defined JSON schema could standardize how a tool exchanges metadata.

Among the most notable external standards are the ISO/IEC 11179 family, used in metadata registries, and the Dublin Core for describing all types of digital resources. The responsibility for applying these standards falls on the Data Architect and the Metadata Manager, who select the standards.

Metadata Quality

Like other data, metadata should meet defined criteria for accuracy, completeness, consistency, and freshness.

Poor metadata can undermine governance and processing because users may interpret otherwise correct data incorrectly. If a catalog says that distance is measured in kilometers while a system stores meters, for example, downstream calculations can be wrong.

Teams can measure metadata quality through checks for completeness, validity, consistency, and freshness. Lineage then helps them see which downstream assets a bad definition or missing field could affect. A Metadata Manager, Data Steward, and Data Quality Analyst may share this work.

Metadata Governance

Metadata Governance defines who can create, approve, change, and retire metadata. Metadata has its own lifecycle, and a controlled process keeps definitions from changing in production without the right review.

For example, if a data analyst proposes changing the description of the concept "distance to campus" to specify that it will now be measured in meters instead of kilometers, they can't modify that definition directly. Governance requires that this proposal first go through the Data Steward to ensure the new wording is clear and consistent with the rest of the glossary, and then be validated by the corresponding Data Owner.

Only after this approval process is the metadata officially updated in production, preventing uncontrolled changes from causing unnecessary failures.

Although the responsibility usually falls on the Data Steward and the Data Owners, this assignment isn't universal. At the executive level, the CDO and the Data Governance Council establish the general policies that guide how governance should be conducted in the organization.

Data Integration and Interoperability

Most organizations don't keep all their data in one system. They use several systems for different jobs, so those systems need reliable ways to exchange and combine information.

Data Integration and Interoperability addresses that need. Interoperability means systems can exchange data and interpret it consistently, while integration combines or connects data for a particular use.

Because each system holds only part of the picture, data integration gathers or virtually connects information from different sources to provide the view a consumer needs.

The goal is to make the right data available in the right place, format, and time. One requirement is latency: the delay between data being created or requested and becoming available to the consumer. The portal may need current taxi availability within seconds, while a monthly cost dashboard can refresh overnight. Integration must also be secure, observable, and auditable.

For example, university systems must agree on the meaning and unit of "distance to campus" or declare a reliable conversion. Without that shared contract, a value in kilometers can be mistaken for meters and cause serious errors.

Once interoperability is ensured, the data can be integrated to generate, for example, dashboards. At the university, data can be obtained from different systems, such as a database with transportation service records and a payment platform, to ultimately generate a dashboard that shows statistics of the cost of that service over a period of time.

Data Architects define interoperability principles and shared patterns. Data Engineers and Integration Engineers design and build ingestion, mappings, and exchanges. Platform Engineering, DataOps, and SRE teams help deploy, monitor, and recover the supporting services.

Data Ingestion

Data Ingestion moves data from a source into a target environment for storage or processing. The target may keep the data temporarily or persistently.

Sources can include databases, APIs, files, applications, and event streams. Destinations can include operational systems, queues, Data Warehouses, Data Lakes, and other platforms. In a push pattern, the source sends data, while in a pull pattern, the destination or connector requests it.

It's also important to mention that there's a distinction in different types of integration depending on whether the data is inserted into a system or queried "directly" from its sources.

One type is physical integration, where data is extracted and stored in a common destination using ETL or ELT processes. For example, the university could load travel and payment records into a Data Warehouse every night using Apache Airflow, Apache Spark, or Azure Data Factory to later generate a cost dashboard.

On the other hand, virtual integration allows querying different sources without having to store their information in a destination environment, as if the sources formed a single system for querying. In this way, the university could combine the travel database and the payment platform in a single query using technologies like Denodo, obtaining integrated data.

Virtual integration doesn't normally persist a separate consolidated copy, although query engines may cache or process data temporarily. Ingestion, by contrast, deliberately moves data into another environment, where further transformations may follow.

For example, the university might want to analyze whether the free taxi service is actually improving attendance at in-person classes. To do this, it integrates data from sources that record travel logs and student attendance, which are likely in different systems. In this process, the sources are queried, and the data is ingested into a Data Warehouse where it's analyzed.

Technologies used for ingestion include Apache NiFi and Kafka Connect, as well as tools like AWS Database Migration Service or Azure Data Factory. The choice depends on the source, destination, volume, frequency, security, and interoperability requirements. A Data Engineer usually designs and implements the ingestion process with the relevant source and platform teams.

Batch Integration

After defining the sources and destination, the team decides when ingestion and processing should run. The answer depends on how fresh the consumer needs the data to be.

In Batch Integration, the system collects and processes groups of records on a schedule or trigger. This approach is often simpler and more cost-efficient when consumers don't need real-time results, although teams still need to manage the concentrated load that a batch can place on source and destination systems.

For example, the university might load completed trips and payments into a Data Warehouse each night to update the transportation service cost dashboard. The process would extract the data, temporarily store it in a staging area, apply the necessary transformations, and load it into the destination. If the frequency is somewhat higher, the batches are called micro-batches, as they contain less data, though the process is exactly the same.

This type of integration is implemented with technologies like Apache Airflow, Apache Spark, AWS Glue, or Azure Data Factory, primarily used by Data Engineers. Additionally, the integration's operation is supervised and monitored by DataOps or Platform Engineering professionals.

Streaming Integration

When consumers need lower latency, Streaming Integration processes events continuously or soon after sources produce them. Instead of waiting for a large scheduled batch, producers publish events that enter ingestion and processing as they arrive.

For example, a transportation company might publish real-time events indicating that a trip has been requested, accepted, started, completed, or canceled, allowing the student portal to be updated immediately.

These events are typically distributed through platforms like Apache Kafka, Apache Pulsar, or Amazon Kinesis, while Apache Flink or Spark Structured Streaming enable filtering, transforming, aggregating, and finally integrating them.

Here, the most important role remains the Data Engineer, although the more specialized role of Streaming Engineer emerges, capable of ensuring these processes run with the necessary low latency.

https://docs.databricks.com/aws/en/data-engineering/batch-vs-streaming

API-Based Integration

Internal and external systems often expose data or operations through an API instead of direct database access.

An Application Programming Interface (API) is a contract through which one system exposes selected data or operations without revealing its internal implementation. You can think of it as a defined set of calls or resources that other software may use.

The university might send a text address to a geographic API and receive coordinates. When a student requests transportation, an internal API could accept an authenticated student identifier and return an eligibility result without exposing the underlying academic record.

Some data platforms expose controlled query APIs, but public services should avoid accepting unrestricted SQL from clients. The API contract should expose only the operations and data that the consumer is authorized to use.

Technologically, the most common practice is to use an API via the HTTP protocol, exchanging data in JSON format and following a REST style, although there are alternatives like gRPC, GraphQL, or SOAP. Regardless of the implementation technology, the API must clearly define its contract, which can be documented using OpenAPI or AsyncAPI.

APIs are usually designed and implemented by a Backend Engineer or API Engineer, while an integration is designed by an Integration Architect, regardless of whether the sources are accessed through an API or not.

ETL and ELT

If we focus on the ingestion process, data must be extracted from a source and inserted into another system. But the target system usually has a different schema than the sources. Each source stores data in a specific organization to solve a problem, while the target system structures data differently, mainly because it integrates information from multiple sources.

For example, a data source might store records with some student information (name, date of birth, email), while the target system where integration is intended stores records with that information along with each student's payment data, possibly changing some fields (name, age, card number). This means student records need to be transformed, such as calculating age from the date of birth.

Real integrations usually need more transformations because source and target structures differ. The boundary isn't always strict: teams may transform data for compatibility, quality, privacy, enrichment, or later analysis at several stages of the flow.

In summary, the transformations referred to here constitute what's known as Extract, Transform, and Load (ETL). Basically, it's a process consisting of a series of steps where data is selected and extracted from a source, transformed to fit the target data model, and loaded.

In the previous example, the only step needed would be converting the date of birth into an age, assuming the data types of the other fields match.

An ETL is suitable when you need to strictly control the information before it enters the destination. But there's also Extract, Load, and Transform (ELT), which first loads the data into the target system and then transforms it once loaded. This approach is common in cloud Data Warehouses and Lakehouses because it allows for preserving an original version and reusing it for various purposes.

For example, with ELT, the university could load authorized student records and raw provider transaction references into a protected Data Lake before applying analytical transformations. It shouldn't copy full card details or bypass security checks simply because the layer is "raw." Keeping source-like data can support reprocessing, but retention, minimization, and access policies still apply.

Teams can implement these processes with Apache Spark, AWS Glue, and Azure Data Factory. Data Engineers usually design the end-to-end flow, while Analytics Engineers often define transformations inside the analytical platform.

Data Exchange Standards

As you've just seen, the differences between source and destination models require transformations.

To reduce the number of transformations needed for integration, there are Data Exchange Standards, which are common rules about the structure, format, and meaning of the data. Their goal is to encourage, whenever possible, the use of a "unique" or common structure so that all systems structure the data as similarly as possible, avoiding transformations when exchanged.

For example, the university could define an exchange model with fields such as (student_id, name, date_of_birth, email), along with their formats and semantics. If a consumer needs age, the contract should define the date on which it's calculated so the value doesn't become ambiguous. Data Exchange Standards don't have to dictate internal storage. They define the representation used at the boundary.

These rules can be grouped into what's known as a Canonical Data Model, documented with OpenAPI or AsyncAPI, among other tools. The responsibility for their definition falls on a Data Architect or Data Modeler, while a Data Engineer or Integration Engineer is the one who ultimately implements the application of these rules in various systems.

Schema Management

Many systems use a schema that defines field names, types, and constraints. A student record might begin as (name, date_of_birth, email) and later gain a phone field. Schemas therefore evolve as requirements change.

Schema Management versions and governs those changes so producers and consumers can coordinate safely. Compatibility policies state which changes a system can accept without breaking existing data or consumers.

Here, we can make a distinction between backward compatibility and forward compatibility. Backward compatibility refers to the ability of a system using a new schema to correctly read or process data saved or emitted with an old schema. Forward compatibility refers to the ability of a system to use an old schema to read, process (or at least safely ignore) data saved or emitted with a new schema without causing errors. In this context, the ideal is to achieve complete compatibility in both directions.

Teams can express schemas with JSON Schema, Apache Avro, Protocol Buffers, and similar technologies, then version compatible formats in Confluent Schema Registry or AWS Glue Schema Registry. Data Architects and Data Modelers define the shared approach with the engineers who produce and consume the data.

Data Quality

Integration can combine data from several sources, but a technically successful integration doesn't guarantee useful results. The output may still contain missing values, incomplete records, contradictions, or duplicates that affect its intended use.

Data Quality is the capability that measures and improves whether data is fit for purpose, in other words, suitable for its intended use.

Quality isn't an absolute label that makes data perfect for every situation. It depends on the intended use. A city of residence may be enough for aggregate demographic statistics but not enough to arrange a pickup. Data should meet measurable requirements for the task at hand.

Generally, the responsibility for maintaining data quality doesn't fall on a single person. Typically, a Data Quality Manager, along with Data Owners, evaluates which data is most critical for an organization, the impact of potential errors, and what level of quality is acceptable.

Then, a Data Quality Analyst analyzes and monitors data practically to ensure its quality, while Data Engineers and development teams implement necessary processes to achieve the required quality. These people don't use specific technologies to manage data quality but rely on other technologies like SQL.

Data Quality Dimensions

Data quality is a measurable property through Data Quality Dimensions, which are observable characteristics of the data. Each one addresses a different question about the data and can apply to a single piece of data or an entire record:

• Accuracy: Checks if the data correctly represents reality.

  • Example: A student's address is accurate if it matches their real address. Otherwise, it doesn't correctly reflect reality.

• Completeness: Checks if all necessary data for a specific use is present.

  • Example: Imagine a registration form requires a name, surname, and phone number, and the user doesn't provide their phone number, or that data is lost. The registration record would be incomplete if finalized, as the phone field would be null.

• Uniqueness: Ensures a piece of data or record doesn't appear more than once.

  • Example: When a student enrolls in a university, the database should have one record with their data, not a duplicate, unless design reasons require it.

• Consistency: Ensures different representations of data don't contradict each other.

  • Example: If a student's email or phone number must be present in multiple places across one or more systems, its value must be the same. It can't appear as one email in one place and a different email elsewhere for the same student. That wouldn't be consistent.

• Timeliness: Checks if the data is updated and available when needed.

  • Example: When a student requests a taxi, they should be able to get their real-time location data, available and updated with low latency for use.

• Validity: Ensures the data respects defined type, format, range, and constraints.

  • Example: If a registration request status can be ACCEPTED or REJECTED, those field values can't be different and must be stored in the defined format. Otherwise, they wouldn't be valid according to defined constraints and business rules.

These dimensions are interrelated, and in practice, some may be more critical for data use. For example, timeliness is crucial when a student requests a taxi, as they expect to see their real-time location immediately. Meanwhile, uniqueness is key for financial data, as a payment record can't exist multiple times, which would be a particularly severe error.

Data Profiling

Data Profiling helps a team understand the current state of a dataset. It inspects structure and content, calculates statistics, and looks for patterns or anomalies. A profile might report null percentages, distinct counts, minimum and maximum values, type patterns, and relationships between fields.

For example, if the university keeps a table with students' personal data, it could be checked that names are stored in a text field, not numeric, or that no record has null values, among other more complex checks.

Relationships between columns and tables can also be analyzed in a relational database, allowing verification that all enrollments are associated with an existing person and subject, as otherwise there would be incomplete and inconsistent data.

Profiling alone can't tell you whether the data is fit for a purpose. A null may be a defect in one field and valid in another. A Data Quality Analyst therefore interprets the profile with Data Stewards and domain experts, using tools such as SQL, pandas, or Apache Spark according to the platform and volume.

Data Quality Rules

Data Quality Rules turn requirements into specific, measurable conditions. They help a team detect when data is unsuitable for an intended use and decide what should happen next.

Profiling discovers what the data looks like, while rules state what acceptable data must look like. Examples include:

• The student's contact email can't be empty and must match the organization's accepted email format.

• The distance to the campus must be a decimal number greater than zero.

• The same taxi ride can't be recorded twice. The student's charge may be zero, while the provider cost must be recorded in the authorized finance system so the university can manage its budget.

The rules are actually treated as a type of metadata, so they must be documented and versioned accordingly. The Data Steward and Data Owners design and validate them based on their business sense, while the Data Quality Analyst and Data Engineer turn them into executable checks. Finally, the rules are expressed in the appropriate technology, such as a query language (SQL, Cypher, and so on).

Data Validation

Data Validation executes rules to decide whether data meets established requirements. Unlike profiling, which explores the data's current state, validation compares values and records with explicit conditions.

The enrollment form may require a student's name, but the API and database should still validate it because client-side checks can be bypassed and data can fail in transit. A relational database can enforce conditions with NOT NULL, UNIQUE, CHECK, foreign keys, and other controls. Application and pipeline checks can handle rules that span systems or require more context.

Data Quality Analysts help define and evaluate these checks, while Data Engineers, Software Engineers, Analytics Engineers, and database specialists implement them at the right layers.

Data Cleansing

Validation may show that all records meet the rules. When some fail, the team needs a defined response: reject, quarantine, correct, enrich, or accept the record with a documented exception.

Data Cleansing detects and corrects known defects so data can meet its requirements. The right transformation depends on the field, the rule, and whether the team can determine the correct value safely. For example:

• To avoid inconsistencies, a rule might specify that names shouldn't contain spaces at the beginning or end. So, if a name like ' Chloé Moreau ' appears, the rule would determine that the data isn't suitable, and it could be transformed by removing the extra spaces to restore its quality.

• Another rule might require that all dates use the format YYYY-MM-DD. Thus, if a date like '15/09/2025' appears, the data wouldn't comply with the rule, but it could be transformed to '2025-09-15' to fit the defined format.

Depending on the data, the rule, and the problem it presents, some transformations can be performed automatically, while others may require more supervision to be done correctly. For instance, spaces in a name can be easily detected and removed, but other issues may be more complex and require manual transformation.

Data Engineers, Analytics Engineers, application teams, or operational staff may perform cleansing, while the Data Quality Analyst and Data Steward validate the approach. The process should preserve enough traceability to explain what changed and why. Cleaning a symptom doesn't replace fixing the source of the defect.

Data Quality Monitoring

Validation shouldn't happen only when data first enters a system. Data Quality Monitoring runs relevant rules and measurements over time, stores the results, and alerts teams when quality degrades.

For example, the university can schedule the automatic execution of quality rules on student data every night. The system would check conditions such as complete addresses, non-negative distances to the campus, and valid date formats. These results can be stored and displayed on a dashboard, allowing for the detection of trends like a sudden increase in negative distance values after an update. This way, the team responsible for the change can quickly identify and correct the problem's origin.

These periodic evaluations are carried out with AWS Glue Data Quality or Microsoft Purview, among other technologies maintained by Data Engineers and DataOps teams.

Issue Management

When a quality problem appears, Issue Management records, prioritizes, investigates, and resolves it. Priority depends on the impact on people, decisions, compliance, and business processes, not only on the number of bad rows.

For example, if due to some error, all distances start showing as negative and students are denied access to transportation services, it impacts the user experience and could have more serious consequences if a student can't attend an important exam. So issues must be managed as quickly as possible.

Generally, this management follows these phases:

1. Registration and classification: When a rule is violated, the incident is documented, including its severity and who is responsible for the affected rule or data domain.

2. Containment: Depending on the severity or impact of the quality loss, measures are taken to prevent that impact from materializing. For example, if a rule states that payment records must not be duplicated and duplications are detected, the measure might be to temporarily block all payments until the issue is resolved.

3. Analysis: Data lineage is used to debug processes and locate the cause of the problem.

4. Correction: Once the cause is identified, the problem is corrected, and the rules are re-executed, validating and documenting the resolution.

If duplicate payment records appear, a Data Quality Analyst may detect and coordinate the issue, the Data Owner sets the business priority, and a Data Engineer or application team fixes the technical cause. Finance and compliance teams may also need to verify the correction.

In short, quality dimensions define what matters for a use case. Profiling shows the current state, rules formalize expectations, validation tests them, cleansing handles suitable corrections, and monitoring detects changes. Issue Management then coordinates the response when a problem reaches production.

Data Engineering

We've discussed systems that store, exchange, protect, and validate data. Now we can look at how teams build the ingestion processes, pipelines, and transformations that connect those systems in practice.

Data Engineering designs, builds, and operates the processes and components that collect and prepare data. It moves data from one or more sources into the systems where people and applications need it, including platforms such as Data Warehouses and Data Lakes.

Data Engineering works across architecture, storage, integration, and quality, although it doesn't replace those disciplines. That overlap is why Data Engineers have appeared in many earlier sections.

The implementation may be as small as a scheduled SQL transformation or as large as a distributed streaming pipeline. In either case, Data Engineering manages dependencies, automates repeatable work, tests changes, and monitors execution.

The goal is to let other professionals use trustworthy data without rebuilding the whole path back to every source.

For example, imagine the university wants to create a dashboard for the management team to analyze the monthly cost of the transportation service. To do this, it's not enough to query a single database, as travel data might be in one database while cost or payment information might be with the transportation company.

Additionally, each source updates at a different frequency and uses its own schema, so Data Engineering here would serve to build a process that performs steps such as:

1. Extract data from each source.

2. Validate its quality through rules.

3. Apply the required transformations, including cleansing defects and standardizing dates, units, and identifiers.

4. Insert them into a target system, such as a Data Warehouse, Data Lake, or similar.

5. Once inserted, they may need to be aggregated or processed as required for later use.

The Data Engineer designs and implements these processes with Data Architects, Data Stewards, and Data Quality Analysts. Together, they make sure the solution meets its technical and organizational requirements. Analytics Engineers, Data Analysts, Data Scientists, applications, and other consumers use the results.

And if the infrastructure is large enough, other professionals like Data Platform Engineers, DevOps Engineers, and Site Reliability Engineers (SRE) may be involved to assist in its operation.

Data Pipelines

A Data Pipeline is a sequence of automated tasks that moves and processes data from one or more sources to one or more targets. A task may read, validate, transform, route, or write data, then pass a result to another task.

At the university, a pipeline might extract authorized transaction references, trip records, and enrollment data, transform them into a common target schema, and load them into a Data Warehouse. Analysts can then use the curated result for reports and dashboards.

A pipeline can run in batch or streaming mode. A full load reads the complete selected dataset, while an incremental load processes records that are new or changed since a known point. One valuable design property is idempotence: safely repeating the same input or run shouldn't create unintended duplicates or inconsistent results.

Other significant properties include scalability, so a large volume of data doesn't compromise execution viability, and traceability to know when it's executed and the results it produces.

Pipelines are usually designed and implemented by a Data Engineer, but sometimes Integration Engineers or Analytics Engineers assist, depending on the final use of the data.

The technologies used for implementation vary greatly depending on the infrastructure. A pipeline may include queries in SPARQL, SQL, transformations done in Python, Apache Spark, or Apache Flink, and even use cloud services like Google Cloud Dataflow.

Pipeline Orchestration

After defining a pipeline's tasks, inputs, outputs, sources, and targets, you need to coordinate their dependencies. That coordination is orchestration.

For example, imagine a pipeline where student and travel data is obtained first, followed by payment data, and these are to be inserted into a Data Warehouse that only accepts records with both payment information and personal data of a student. With these requirements, data from all sources must be obtained before insertion, as they need to be combined. This might not be the case in other pipelines where information from each source can be inserted as it's obtained.

These dependencies in a pipeline are commonly represented with a Directed Acyclic Graph (DAG) where each node is a task and each connection indicates a dependency. It can also serve as an internal data structure for orchestration software to precisely decide when a task is ready to execute and what should happen based on its result.

Among the most commonly used technologies for orchestration are Apache Airflow, Dagster, and Prefect, as well as cloud services like Azure Data Factory, AWS Step Functions, or Google Cloud Composer.

Data Transformation

Many pipeline tasks transform the structure, representation, or content of data so a later consumer can use it.

Transformations can be simple, like converting kilometers to meters, normalizing a date to a common format, or renaming a field. Others are more complex or follow more abstract business rules, such as linking taxi routes with academic schedules to automatically validate if a trip coincides with a mandatory in-person class, thus detecting improper use of the service or any issues. Some transformations may also involve filtering, removing duplicates, or aggregating data.

When data transformations are performed, the data transitions from being newly obtained from a source to being ready for use. Here, we can establish a classification based on the level of transformation the data has undergone:

• Raw: Data kept close to the source representation. For example, a provider supplies the date string 05/03/2026, whose intended day/month order must be documented.

• Staging: Data is validated and standardized for further processing. Once the source meaning is known, the date could become the unambiguous ISO value 2026-03-05.

• Curated: At this level, the data is enriched, combined with other data, and considered ready for final use. For example, assuming the previous date corresponds to a trip, it can be combined with other data to create a record of that trip enriched with payment information.

Transformations focus on converting raw data into staging and curated data. Technically, implementation can be done using various technologies depending on the systems involved and company decisions. Primarily, you'll use languages like Python, R, SQL, or frameworks like Apache Spark.

Workflow Automation

A pipeline may also check source availability, validate quality, manage approvals, and send notifications. Workflow Automation coordinates these actions in the required order so repeatable work doesn't depend on someone running every step by hand.

It's important to differentiate between the pipeline and the workflow. The pipeline describes the path of the data and its transformations. On the other hand, the workflow includes tasks that don't directly transform the data but are essential for the execution of a pipeline.

For example, when the university receives a file from the transportation company, the workflow can validate its format, monitor the pipeline execution, and update data lineage tools.

But automating a workflow doesn't always mean eliminating human intervention. For instance, a rule might be set to detect if personal data appears in a source when it shouldn't. If this rule detects personal data, a Data Steward intervenes to approve the change or reject it and take appropriate action.

Finally, workflows are implemented using orchestrators like Apache Airflow, Dagster, or Prefect, along with CI/CD systems and incident management tools.

Data Testing

When automating the execution of a pipeline, even if manual oversight isn't completely eliminated, much of the process will run with the possibility of errors in its implementation. Even with a perfect implementation, errors can occur that affect the data and cause failures in the pipeline tasks.

Data Testing checks both transformation code and the data moving through the pipeline so teams can catch defects before they affect consumers.

The test suite should cover realistic ways that code, schemas, data, dependencies, and infrastructure can fail. Data tests and Data Quality rules overlap, but teams may apply them for different reasons.

A quality rule expresses a business or fitness requirement, while a pipeline test may verify a technical precondition or expected transformation. The same check can serve both purposes.

Common test types include:

• Unit tests: These verify that the code for a transformation is correct given certain inputs and the respective outputs it should produce. For example, if a transformation converts a distance from kilometers to meters, it could be tested with inputs 18, 4, 6 and outputs 18000, 4000, 6000.

• Schema tests: These are performed on the data to ensure its structure and format are suitable for a specific task. For instance, when receiving a student's age stored as the number 42, a schema test would verify that this data is of integer type.

• Integration tests: These check that various components of an architecture or system can interact as expected. For example, an integration test might verify that a university's Data Warehouse can receive data from an academic database.

• End-to-end tests: These involve executing the entire pipeline to ensure the result is correct given initial data.

• Reconciliation tests: Compare counts, totals, or control values across stages. If a documented filter should retain 50 of 100 input records, the test verifies both the output count and the reason for the exclusions.

• Performance tests: Given the complexity of some pipelines, performance tests are conducted to evaluate if their execution is feasible within a certain time and with available resources.

At the university, before deploying a pipeline, datasets with fictional information, also known as synthetic datasets, could be constructed for use in testing. This way, all these types of tests could be executed to verify that tasks are performed correctly, data has the expected properties after each transformation, and the process is completed within a specified time.

The test technology follows the pipeline. A Python transformation could use pytest, while SQL can support reconciliation and schema checks. Data Engineers own most pipeline tests, and Platform or DevOps Engineers help integrate them into automated delivery and runtime environments.

Data Versioning

Data pipelines generally undergo changes due to modifications in business requirements, changes in sources, or other reasons. So it's essential to maintain a history of what has happened with a pipeline over time, allowing you to track its evolution up to a specific point, primarily to facilitate error debugging.

Data Versioning keeps a history of the assets needed to reproduce a result. Depending on the use case, this can include transformation code, schemas, configuration, reference data, model inputs, and snapshots or versions of the dataset itself.

For example, imagine a report states that $10,000 was spent on taxis in a month, but upon checking later, the system says the amount was $8,000 for the same month. This discrepancy could be due to an error or a change in the policies used to calculate that cost, such as no longer counting canceled trips.

To determine if this situation is an error, versioning allows access to previous versions of the pipelines involved in that calculation to see how the figure was obtained.

Teams commonly use Git for code, configuration, and text-based schemas. Table formats such as Apache Iceberg, Delta Lake, and Apache Hudi can preserve data snapshots and change history for supported tables. Reproducibility may require both.

Data Platform Operations

Once implemented and versioned, a pipeline needs an infrastructure to run on, which refers to hardware that can be on university servers or in the cloud. It may require storage for data, computing capacity for transformations, an orchestrator to coordinate tasks, and specialized systems to ensure data and process security. These components together form a Data Platform, which is the technological environment where pipelines and other processes are executed.

The platform itself must be managed and maintained, as it's not a system that operates completely autonomously but requires supervision. This management process is known as Data Platform Operations and encompasses a series of tasks aimed at ensuring the platform is ready to execute pipelines securely, stably, and efficiently.

Some of the most fundamental tasks are:

• Provisioning and scaling of resources: The number of machines needed by databases and platform components at any given time is configured.

• Environment management and isolation: Reserved environments are created for testing, development, and production, with the latter providing services to the end user.

• Permission management: Permissions are determined for each professional to perform their tasks, preventing security breaches.

• Cost control and optimization: Resource consumption is monitored to avoid overspending, aiming to provide the service with minimal consumption.

For example, a pipeline that calculates the monthly cost of taxi usage might need to connect to a transportation company's API, transform the data, and store it in a Data Warehouse.

To achieve this, the platform must provide the necessary computing resources to perform the transformations, store the data, and allow a secure connection with the API. Thus, proper platform management is critical to ensure the pipeline runs correctly.

A Data Platform Engineer commonly leads this work and understands the services on which the platform runs, such as AWS, Azure, Google Cloud, Databricks, or Snowflake. Docker packages suitable workloads, Kubernetes can orchestrate containers when the complexity justifies it, and Terraform defines infrastructure as code. Infrastructure as code improves repeatability, but it doesn't make services automatically portable between cloud providers.

Data Observability

Data platforms can fail in subtle ways even when every job reports success. Data Observability helps teams understand the health of data and the systems that produce it so they can detect, investigate, and reduce the impact of failures.

Observability lets you infer a system's state from the signals it produces. In a data context, those signals include freshness, volume, schema, distribution, quality results, lineage, job status, logs, metrics, and traces.

Monitoring checks known conditions, such as whether a job completed and whether freshness or volume stayed within expected limits. Infrastructure signals such as CPU and memory can help explain failures, while data-level signals show whether consumers received the right output.

For example, if a data pipeline produces dozens of records when it should produce hundreds, monitoring allows you to detect these changes in results,. It can also show other relevant metrics obtained at those same moments, such as the CPU usage of each task involved in the pipeline, helping you detect if any tasks are failing and preventing data from propagating to the end.

For observability to guide action, teams can define Service Level Indicators (SLIs) for relevant properties and Service Level Objectives (SLOs) for the expected level. An SLI might measure the age of the latest attendance data, while the SLO could state that 99% of daily updates must be available by 7:00 AM. An alert tells the team when the pipeline risks missing that commitment.

The most well-known technologies in observability are Prometheus and Grafana, frequently used to collect and visualize metrics. There are also OpenTelemetry for managing telemetry data and logs, and OpenLineage for monitoring data lineage in real time.

Here, a Data Engineer might be responsible for implementing the appropriate observability mechanisms. But they don't always do it alone, as an SRE, Platform Engineer, or DataOps team may collaborate in maintaining these mechanisms.

Data Contracts

Observability helps detect errors such as failed jobs, stale data, abnormal volumes, and unexpected schema changes. If a taxi provider changes geographic coordinates from numbers to text without notice, for example, downstream processes may fail even though the network connection still works.

Data Contracts reduce this risk by making expectations between producers and consumers explicit. They define the structure and characteristics of the data, along with how teams communicate and version changes. Observability still verifies the contract in operation.

More specifically, a Data Contract can define schema, types, formats, semantics, quality rules, ownership, delivery frequency, latency, and change-management expectations.

For example, the transportation company might agree that each trip event includes (trip_id, student_reference, provider_vehicle_id, price, origin, destination). The contract could define price in euros and coordinates as numeric latitude/longitude pairs, set privacy limits on student_reference, and require a new contract version for an incompatible change.

Also, the contract isn't just documentation. Checks are implemented to verify compliance so that any change, for safety, doesn't affect data pipelines, as changes can impact both availability and security.

To define a Data Contract, data schemas are often represented in JSON Schema, Apache Avro, Protocol Buffers, or similar technologies, although standards like the Open Data Contract Standard (ODCS) are also used.

The contract is developed and reviewed by Data Engineers and Analytics Engineers within the organization, who coordinate with professionals from other companies, such as Software Engineers who know what data their source produces. At a higher level, Data Owners and Data Stewards are involved to validate the semantics, quality, and usage conditions of the data.

DataOps

Data Engineering involves many people and components. Even a pipeline that works today can become unreliable if teams don't coordinate changes to sources, contracts, code, infrastructure, and quality rules.

DataOps is an approach to improving that collaboration and delivery process. It aims to shorten the path from a business need to trustworthy data while maintaining quality, security, and traceability.

DataOps isn't a specific technology. It's a set of practices such as versioning code, automating tests, reviewing and deploying changes through controlled environments, and monitoring production pipelines. It adapts ideas from agile delivery and software operations to data-specific concerns.

For example, imagine the university starts working with a new taxi company. The first step could be creating a Data Contract with the conditions for data delivery. Then, a Data Engineer would implement all the necessary software for obtaining it through a connector and store it in Git.

Also, before deploying it in production, you should conduct data and code tests to ensure functionality. Finally, after deployment, it would be monitored through metrics like the volume of data extracted, its quality, and latency.

Data Engineers, Analytics Engineers, Data Stewards, Data Owners, Platform Engineers, SREs, and consumers all contribute to DataOps. The practices work only when the people who produce, operate, and use data share responsibility for reliable delivery.

Technologically, DataOps relies on tools we've already discussed, like Git for versioning and CI/CD tools for automating tests and deployments, among others. But its value doesn't come from a specific tool. It comes from adopting best practices in their use.

Many of these ideas come from DevOps. Nonetheless, DataOps adapts them to data work, incorporating specific aspects like quality, semantics, lineage, and the relationship between producers and consumers.

DevOps

As I just mentioned, DataOps adopts ideas from DevOps. DevOps refers to a set of best practices that help coordinate software development and the deployment of systems, all with the goal of ensuring that changes can be tested, deployed, and maintained in an automated and reliable manner.

Among its main practices is Continuous Integration (CI), which involves integrating each code change into a repository so tests are automatically conducted. Then there's Continuous Delivery or Continuous Deployment (CD), allowing changes to be deployed automatically in a controlled manner across different environments. Finally we have Infrastructure as Code (IaC), which lets you define infrastructure components programmatically, facilitating their versioning and deployment across various cloud platforms or servers.

For example, when a Data Engineer modifies the connector that extracts data from the taxi company, the change is saved in Git and a CI system automatically runs its tests. If it passes, a new version of the software is built and deployed autonomously in a test environment to continue verifying its functionality until it's deployed in the final production environment.

Common technologies include GitHub Actions, GitLab CI/CD, or Jenkins for automating tests and deployments. Docker is also commonly used for packaging software along with Terraform or OpenTofu for defining infrastructure. Kubernetes can also be used to manage containers when the system's scale and complexity require it.

DevOps Engineers, Platform Engineers, and SREs implement and maintain these mechanisms, while Data Engineers use them to deploy their pipelines. The main difference is that DevOps focuses on software and infrastructure delivery and operation, while DataOps also checks data-specific aspects like quality, semantics, lineage, and availability.

Data Warehousing and Business Intelligence

Organizations capture, integrate, and transform data through pipelines, then store it in systems chosen for particular workloads. Operational databases support the applications and transactions that keep day-to-day services running.

Analysis often needs integrated history, stable definitions, and queries that scan many records. Specialized platforms such as Data Warehouses and Data Lakes support that work. Data Warehousing and Business Intelligence makes governed analytical data available to people who explore it and use the results in decisions.

These are two related concepts. Data Warehousing covers the design and use of a Data Warehouse, which integrates historical data from several sources for repeatable analytical workloads.

Operational and analytical workloads have different priorities and access patterns. Some platforms support both, but teams still face tradeoffs in isolation, performance, freshness, consistency, and cost. Separating the workloads often protects daily operations and gives analysts a model designed for their queries.

Business Intelligence (BI) covers the practices and technologies used to query, analyze, and present data for decision-making. A Data Warehouse often provides the governed analytical foundation for BI, although BI tools can use other sources too.

For example, a university might integrate trip and finance data in a Data Warehouse. Analysts could compare provider costs, usage, attendance, and budget to assess whether the transportation benefit is sustainable and estimate short-term spending.

Also, in order to conduct these data analyses, build dashboards, and ultimately make decisions, the data needs to be of high quality, protected, and maintained with proper lineage. Any issues in these aspects can influence decision-making.

Analytical Data Stores

Analytical workloads often scan long time periods, join several sources, and aggregate large numbers of records. Storage designed mainly for operational transactions may not be the best place to run them repeatedly.

Analytical Data Stores are designed for analytical queries, transformations, and aggregations. They still need security and consistency controls, but their performance priorities usually favor scans and calculations across large datasets rather than high-frequency row-level transactions.

The most representative example of an Analytical Data Store is a Data Warehouse, which stores data in a stable and scalable way so that the same analysis process can be repeated over time with an ever-increasing volume of data.

But this is not the only option, as Data Lakes are also oriented toward this type of use, and Data Marts offer a smaller-scale analytical environment (usually being subsets of data from a Warehouse) specifically designed to meet the needs of a particular department or business area.

For example, the university could create a Data Mart containing mobility measures and the limited financial context needed to analyze service cost, without exposing irrelevant student details. The team should connect the Mart to lineage, security, quality, and audit controls just as it would any other analytical asset.

Among the most used platforms to implement these systems are Snowflake, Google BigQuery, Amazon Redshift, Microsoft Fabric Data Warehouse, and Databricks SQL. Their design and implementation are the responsibility of an Analytics Architect or Data Architect, while Data Engineers maintain the data pipelines that supply them with information, and Analytics Engineers handle the transformations required after ingestion to facilitate subsequent analysis.

At the administration and maintenance level, there are Data Warehouse Administrators or Platform Engineers, who monitor performance, manage permissions, and platform costs.

Facts and Dimensions

An Analytical Data Store may preserve source-like data or organize it into a model, depending on the platform and layer. A Data Lake commonly retains source formats in an early zone, while curated layers and Data Warehouses apply more explicit schemas.

One common analytical approach is the dimensional modeling we talked about earlier. It organizes information into facts and dimensions. A fact records an event such as a trip, while dimensions provide context for filtering, grouping, and comparison.

A particularly important design choice is granularity, or grain: exactly what one row of a fact table represents. The team should define it before choosing dimensions and measures so later aggregations remain valid.

For example, the Trip fact table might have a grain of "one completed trip." If a student takes two trips on the same day, the table stores two rows, each with its cost, distance, duration, and date key. The university can sum those rows by month. It shouldn't add monthly-total rows to the same fact table because they have a different granularity and would cause double counting.

Once the granularity is defined, dimensions should be chosen based on the context describing the fact and the analytical queries expected to be performed. A practical way to identify them is by asking who, what, when, where, and how each fact was involved. For example, if each row represents a trip, dimensions like Student, Date, Provider, Origin, and Destination could be used, each with a unique value for that trip.

These dimensions would allow analysis of the geographical areas where trips occur, which transportation company makes more or fewer trips, and so on. This way, dimensions are incorporated that provide a useful perspective for analyzing the facts.

This data modeling is done by an Analytics Engineer or Data Modeler, along with domain experts like Data Stewards. Then, Data Engineers implement the data ingestion and transformations required to adapt the data to the specific final model of each system.

Metrics and KPIs

In a dimensional model, facts can be seen as rows composed of values, called measures. These measures can help understand what happened during an event over time, but data analysis generally aims to answer questions involving all events over a certain period.

Teams combine measures into repeatable metrics, such as totals, rates, averages, and percentiles. A metric becomes a Key Performance Indicator when it's tied to an important objective and helps show whether the organization is meeting it. Here are some examples:

As is evident, not every metric is always a KPI. For example, a metric that represents the total number of trips made in a month can be useful for describing transportation service usage, but it will only be a KPI when there's a business objective that involves quantifying that number of trips.

KPIs are often used in dashboards and visualizations, although they generally don't appear in isolation. In this regard, when several KPIs with their current values are gathered and compared with established goals, this gathering is called a scorecard.

Despite both concepts being related, a scorecard and a dashboard have different purposes. A scorecard aims to determine if goals are being met, while a dashboard helps understand what's currently happening in the organization and why.

The same metric may appear in dashboards, scorecards, reports, and APIs, so teams need a reusable definition. Its documentation should include:

• The name, purpose, and business owner.

• The formula that calculates the resulting value of the metric, the sources of the data, and its granularity.

• The unit, time period, time zone, and frequency of metric value updates.

• The filters and inclusion rules, such as excluding canceled trips from the calculation.

• In the case of a KPI, the objective that originates it is documented.

Here, metrics and KPIs are primarily defined by roles like Business Owners, Data Owners, and Data Stewards. On the other hand, their practical implementation is carried out by Analytics Engineers and BI Developers, and finally, their results are used by Data Analysts, among other professionals.

Semantic Layers

As I mentioned before, metrics are documented to ensure their meaning and calculation method are well understood. But this doesn't guarantee that all systems adhere perfectly to this documentation.

For instance, monthly cost might be calculated excluding canceled trips, while another system might accidentally include them. In both cases, the same "name" is used for a metric that produces different results.

A Semantic Layer addresses this problem by centralizing reusable business definitions between stored data and consumption tools. It presents concepts such as Trip, Student, or Course instead of requiring every consumer to rebuild logic directly from tables and joins.

In this way, the formulas and filtering rules that make up each metric are implemented on the semantic layer, rather than each analyst writing their own code on a database, Data Warehouse, or corresponding system. This layer acts as an intermediary that translates the calculation of a metric expressed in a business-friendly language into the necessary code for specific systems to perform that calculation, facilitating future metric modifications and portability between different systems.

For example, in the Data Warehouse, there might be a Trip fact table, a Date dimension, and a cost measure in each fact. Here, the semantic layer would define the existence of certain concepts like trip and cost, whose calculations are "mapped" in some way onto the technology used to implement each system.

In this case, the calculation of a "Total Cost per Month" metric could be defined on the semantic layer, which would internally translate this into SQL operations, or the corresponding technology, to group trips by month and sum the cost measure of the grouped facts.

The main difference between the documentation of a metric and its implementation in a semantic layer is that the documentation specifies what the metric is and how it is formally calculated, while in the semantic layer this specification is translated into operations in a specific technology that allows the calculation.

Thus, multiple dashboards or reports can reuse the same logic defined on a semantic layer, as sometimes calculations need to be performed on data in different systems.

Technologies used to implement semantic layers include Power BI Semantic Models, LookML, dbt Semantic Layer, and Cube. Analytics Engineers and BI Developers commonly build and maintain these definitions with input from business owners and analysts.

Reports and Dashboards

After implementing the Analytical Data Stores systems in production and defining some metrics or KPIs, the next step is to create Business Intelligence products that present the analysis results to end users, professionals, or executives.

The most common products are reports and dashboards, though they aren't the only ones, as the analysis results can also lead to a visualization or documentation of a decision-making process, for example.

Let's better understand what each one is and their differences:

A report is a document that presents detailed and structured information on a specific topic and time period. It may include graphs, metrics, and explanations. Reports can be generated periodically in static formats, like PDF, or be interactive, allowing users to filter or manipulate the presented information.

For example, a university might prepare a monthly report with the transportation service cost broken down by provider, showing canceled trips, the number of students who used it, and so on.

A dashboard, the other hand, is a view that brings together the most relevant metrics and KPIs to monitor a situation. It typically contains graphs and visual elements that update more frequently than a report.

For example, a dashboard for the administration could show the consumed budget, the number of enrolled students, and the attendance trend, also allowing results to be filtered by training program if it is interactive.

There are some best practices to follow when you're creating dashboards to make sure they're useful. For example, you should display only a few indicators and only those truly relevant to the dashboard's purpose. Also, choosing a visualization isn't merely decorative, as the charts should help people understand the information presented, and they should follow best practices in their design.

In general, you'll use a dashboard when it's necessary to periodically monitor a small set of indicators and quickly detect changes or deviations. You'll use a report when you need a deeper exploration of a topic, although both products can complement each other.

For example, the administration might use a dashboard to detect an increase in transportation expenses and then consult a monthly report to find out which providers, routes, or periods caused it.

For creating these products, the most commonly used technologies are Microsoft Power BI, Tableau, Looker, Apache Superset, and Metabase. These are primarily used by BI Developers, although BI Administrators also collaborate in managing the workspace where the products are built. Finally, the results can be interpreted by a BI Analyst, who also has the knowledge to develop reports or dashboards in certain situations alongside the BI Developers.

Self-Service Analytics

The data analysis process generates products like dashboards or reports, which present specific information structured for a purpose. But sometimes it may be necessary to modify that purpose.

For example, the finance department might have a dashboard designed exclusively to monitor the overall budget the university allocates to taxi services. Yet, the director of a specific master's program might need to cross-reference that transportation data with attendance records from their training program to see if the service provides any benefit, which is a very specific need not addressed by the original dashboard.

The coordinator could ask the technical team to change the dashboard, but every small question would then enter a development queue. Self-Service Analytics lets authorized users explore governed data and create suitable analyses without depending on a technical specialist for every step.

This approach relies on elements we covered earlier, such as semantic layers where metrics are maintained, data catalogs that allow you to quickly locate available information, and business glossaries that standardize the meaning of business concepts. These elements are used by team members who independently build their own visualizations and reports, although they may not have access to all types of information due to existing privacy policies. This is why the process is called managed self-service.

For example, if a dashboard shows an increase in transportation expenses, a Master's coordinator could use a semantic layer to define a filter for their program's data. Thus, the semantic layer would ensure the official cost definition is used, while permissions would prevent access to data from other programs or unnecessary personal information.

Finally, it's worth noting that the original dashboard isn't always modified. Instead, the coordinator creates a new one with their changes.

In practice, the viability of this approach is the result of coordinated work by Analytics Engineers, BI Developers, and BI Administrators, primarily. The end users who consume and leverage this capability are Data Analysts, Business Analysts, and business managers.

Big Data

The data lifecycle runs across an infrastructure of systems and pipelines. Data enters, moves, gets stored and processed, and eventually reaches operational or analytical consumers.

For a moderate workload, a relatively simple architecture may meet the required performance, reliability, and cost targets. As the organization grows, however, it may need to store more data, process events more often, and support more varied formats and use cases.

A database that began on one machine might first scale vertically by gaining more CPU, memory, or storage. At some point, the workload or resilience requirements may justify horizontal scaling across several machines, but that added complexity should solve a measured need.

Big Data deals with datasets and flows whose volume, velocity, variety, or combination pushes beyond the practical limits of conventional tools for a particular organization. The challenge is not simply "a lot of rows". It's meeting the required processing time, reliability, and cost at that scale.

When thinking about Big Data, you might imagine a well-defined threshold beyond which a data set is considered Big Data. But this isn't the case, as the threshold depends on the current infrastructure, the target speed, the cost thr team willing to incur for its management, and the variety in the structure of the information.

A team should adopt a Big Data solution only after assessing whether the current infrastructure misses its performance, reliability, or cost requirements. Distribution may help, but it also adds operational complexity, so the benefits need to justify it.

For example, a university could grow from having 1,000 students to 100,000 due to an expansion of its faculties or the introduction of online classes. If this happens, the databases must support storing all their personal data, as well as the data generated when interacting with various services and platforms like the virtual campus, all at a speed that doesn't compromise service availability or quality.

Big Data draws on many Data Management capabilities at a larger scale. A Big Data Engineer is often a Data Engineer who specializes in distributed storage and processing. They work with Data Architects who design the solution and Data Platform Engineers who operate it.

The 3Vs: Volume, Velocity, and Variety

There's no universal threshold for Big Data, but the 3Vs – Volume, Velocity, and Variety – provide a useful guide. They aren't three boxes every project must check. They describe pressures that can make a workload harder to manage with the current infrastructure.

Volume refers to the total amount of data that must be stored and processed. The first challenge here is that data takes up space, so in a large enough volume, some systems may not be able to handle it all. Also, various management processes slow down as the volume increases because all data must go through pipelines or similar processes.

• Example: Volume can be associated with the amount of data produced by students, meaning the more students there are, the more data volume needs to be supported. Each student generates data like login events, which must be stored and processed, taking up space and consuming significant computing resources if the volume is high.

Velocity refers to how quickly data arrives, changes, and must become available to consumers.

• Example: Transportation service taxis must communicate their position and status every few seconds so a student can have a real-time view of available taxis and whether they are near their location. So it's crucial that data is available as quickly as possible to ensure a good user experience.

Variety, as previously mentioned, describes the nature or diversity of data, such as structures, formats, and meanings that data presents.

• Example: An academic database can store enrollments and students in tables using a relational paradigm, while the virtual campus produces logs in semi-structured JSON documents, or a graph-oriented database represents information about students, drivers, and locations with graphs to optimize transportation routes.

Volume affects storage, transfer, and processing costs. A team may optimize the data model, partitioning, queries, or retention before distributing the workload. When one machine can no longer meet the requirements economically or reliably, horizontal scaling becomes one option.

Not all data needs real-time processing. A live trip-status update may need seconds, while a historical tuition-payment report can refresh on a daily schedule. The required latency should come from the user and business need, not from a desire to make every pipeline real time.

Finally, variety is one of the most significant properties of data because it determines the heterogeneity of the dataset within the organization. With such diverse data stored in different structures, formats, and representations, it becomes necessary to adopt specific techniques for each variety to ensure efficient and viable management.

These are the properties typically attributed to Big Data. But it's also important to highlight other significant properties, such as veracity, which refers to the reliability of the data or value, among others.

Big Data Architectures

When the 3Vs exceed the capacity of a "conventional" solution, there are several ways to increase the capacity of an infrastructure to meet these needs. But first, it's useful to define what infrastructure is.

Infrastructure is the set of computing, storage, networking, and foundational software resources on which the organization's applications and data systems run.

Architecture describes how components use that infrastructure to meet requirements. It defines where systems run, how storage and processing are distributed, and which path data follows from source to consumer.

So if the 3Vs compromise the viability of an existing solution, it may be necessary to modify its architecture. One way to address an increase in volume or velocity, as mentioned before, is vertical scaling. This involves improving the hardware, giving each machine more resources. But this can't scale infinitely, which is why horizontal scaling exists. More machines are added, and storage and processing are distributed.

Another way to increase speed could be the parallel execution of processes across multiple machines, known as Massively Parallel Processing (MPP).

There are many ways to improve the capabilities of an infrastructure, especially when it comes to processing more data at higher speeds. Managing a greater variety of data, though, is often a challenge without established general techniques, although distribution can help.

To better understand what architecture consists of, think of it as a set of layers where each encompasses certain components that together constitute the path data takes throughout its lifecycle within the organization.

Another important aspect of any architecture is that its layers must implement security, lineage, and observability mechanisms, also ensuring data privacy.

Imagine a student requests a taxi through the virtual campus. The architecture must protect and trace the event. A streaming flow can update trip status on the portal within seconds, while a later batch process consolidates the relevant records for cost analysis.

This difference in speeds is another way to adjust the architecture so that certain critical functionalities have the required speed or so that analysis processes that don't need to be performed in real time can handle a larger volume of data.

Finally, the architecture is designed by a Data Architect or Big Data Architect and implemented by Data Engineers, Streaming Engineers, or Software Engineers. Its maintenance is the responsibility of Data Platform Engineers, Cloud Engineers, and SREs.

Big Data Storage and Processing

After designing the architecture, its components are implemented, with some dedicated to storing and processing data at the required scale. On one hand, storage is responsible for keeping data persistent, secure, and accessible. On the other, processing uses computing resources to transform and analyze them, primarily.

The university might retain authorized virtual-campus events, attendance records, and trip information for several years, creating a large storage need. Its processing demand may be more variable, with peaks during reporting periods or major academic events.

By separating storage from processing, if we focus on systems that can serve to store data in an infrastructure, we might encounter:

• Distributed databases: These are databases deployed to operate across multiple machines, using technologies like Cassandra or DynamoDB.

• Object Storage: These systems are dedicated to storing large volumes of data in independent objects, utilizing Amazon S3, Azure Blob Storage, Google Cloud Storage, or MinIO.

• Search engines: These systems specialize in quickly indexing and querying logs, texts, and other types of semi-structured information with technologies like Elasticsearch or OpenSearch.

• Distributed file systems: These store and distribute files across multiple machines using HDFS or CephFS.

On the other hand, data processing in an infrastructure can be distinguished based on the approach taken, which depends on volume and speed:

• Batch processing: Here, data is accumulated over time and periodically processed in batches. This can be implemented with Apache Spark, for example, which allows tasks like transformation and cleaning to be distributed across multiple machines.

• Streaming processing: Here, all data generated or arriving at the start of a pipeline is processed continuously, making it suitable when real-time results are needed. Technologies used in this case can be Apache Flink or Spark Structured Streaming.

• Distributed query and processing: This allows for the analysis of large volumes of data by executing operations in parallel across multiple machines. One of the most common interfaces is SQL, used by tools like Trino or Spark SQL. But in addition to SQL, these systems often offer APIs in languages like Python, Java, or Scala and abstractions like DataFrames, providing greater flexibility for implementing complex transformations or custom logic.

As an example of architecture, the university could use Kafka to receive events generated by the virtual campus or the transportation company, while Flink could process them to keep the status of each journey updated in real time in the application consulted by the end user. Then, with Spark, they would be transformed to be integrated into a Data Warehouse and queried using SQL.

In practice, the central role that implements and optimizes these storage and processing systems is the Big Data Engineer or specialized Data Engineer. For this, they use technologies like Cassandra, Amazon S3, or HDFS, decide how to implement jobs using Spark, and ensure adequate performance.

On the other hand, Data Platform Engineers, Cloud Engineers, and SREs handle the base infrastructure, ensuring its stability, availability, and resilience.

Big Data Analytics

In Big Data, besides storing a large volume of diverse data and processing it at a speed that often needs to be high and in real-time, it must be converted into information, knowledge, and ultimately value. This means that processing refers to the transformations performed on the data to enable storage, clean it, or maintain its quality, primarily.

But processing is also applied after storage to calculate statistics and generally analyze the data. This is the role of Big Data Analytics, an area dedicated to converting data into information, knowledge, and value through analytical processes applied to large volumes of data.

An analysis belongs in a Big Data context when the workload's scale or flow characteristics require distributed or otherwise specialized infrastructure to meet its targets. It doesn't need advanced Machine Learning, and using a scalable cloud platform by itself doesn't make a small analysis "Big Data."

Based on this technological foundation, there are several fundamental analytical approaches you can use, depending on the analysis you need to perform:

• Descriptive Analytics: Focuses on applying techniques that explore data to understand what has happened. For example, it allows calculating how many trips have been made, how much they have cost, and how many students have used the service each month.

• Diagnostic Analytics: Here, the analyses aim to understand why a result has occurred. At the university, it could be used to study which supplier time slots are related to an increase in transportation service costs.

• Predictive Analytics: Uses historical data to make inferences and try to predict what will happen in the future. For example, it could predict how many enrollment applications will be received next term.

• Prescriptive Analytics: Turns the results of analyses into recommendations. For instance, in this case, it could suggest how to optimize the distribution of taxi fleets and reallocate the monthly budget to ensure service coverage for the maximum number of students.

In big data environments, analysis can be executed in batch or streaming, depending on each process's requirements. For instance, with Apache Spark, you could periodically calculate the evolution of taxi trip costs and class attendance, while with Flink, real-time trips could be analyzed to generate alerts if demand exceeds a certain amount.

For analysis processes to be truly useful, they begin by defining the question to be answered with the obtained knowledge and the value expected to be added, meaning the decision to be made with the result. Then, the necessary data is selected and prepared, ensuring its quality is adequate for analysis. After execution, the result is published via a dashboard, report, alert, API, or predictive model.

It's also important to note that having a larger volume of data doesn't always guarantee "better" conclusions or more value. For example, if students using the taxi service have higher attendance, you can't directly conclude that transportation is the cause, as those students might be taking more in-person classes or have other differences.

So besides handling a large volume of information, it's crucial to interpret results correctly. In this specific case, the problem is that correlation doesn't always imply causation in the analyzed facts, but this isn't the only issue that can arise in an analysis.

Data Engineers build and maintain pipelines and analytical environments. Data Analysts use SQL, Trino, Spark SQL, Power BI, Tableau, and similar tools for a range of analyses, often descriptive and diagnostic. Data Scientists use Python, R, Jupyter, Spark, or MLlib for statistical modeling, experimentation, prediction, and optimization.

BI Developers turn governed metrics and analyses into reports and dashboards. Machine Learning Engineers help train, deploy, and operate models. Domain experts, Data Owners, and Data Stewards help teams interpret and use the results responsibly.

Analytics and Data Science

Organizations analyze data to understand what's happening, support decisions, test ideas, and build models. This is one of the main ways they turn data into knowledge and value.

Analytics and Data Science are overlapping, complementary fields. Analytics often focuses on answering defined questions with descriptive, diagnostic, predictive, or prescriptive methods. At the university, an analyst might study attendance over the past month and investigate which changes coincide with a decline.

Data Science often tackles less-defined or model-heavy questions through statistics, Machine Learning, computation, and domain knowledge. It may explain patterns, estimate effects, segment observations, or make predictions. The university could use it to forecast transportation demand over the next six months.

In practice, both use data to achieve a goal, and the exact boundary varies by organization. Both need governed, suitable, high-quality data and a clear understanding of the decision their result will support.

An analysis should start with a clear question. The university might ask whether the transportation benefit improves class attendance or how many rides students will request next week. The first needs a careful causal design, while the second calls for a forecasting or predictive model.

After formulating the question, a process is established that covers everything from the question to a final analytical product like a dashboard, report, or simply the knowledge produced that contributes to decision-making.

In this process, an analytical dataset is generally built to serve as a source for subsequent analysis. Then, this dataset is explored to understand the data, model it mathematically, or perform transformations on it. In other words, the analysis process begins by applying techniques suited to the business question's needs.

Finally, if you need to train a machine learning model, you'll make certain transformations to prepare the dataset for training, so it's considered model-ready. After training, results are delivered through a report, API, or by deploying the model in the infrastructure to make predictions, for example.

In this process, various roles collaborate, such as Data Analysts, who answer business questions related to Analytics, while Data Scientists formulate hypotheses and develop models to describe data or make predictions. Analytics Engineers focus on building analytical datasets, and Data Engineers construct the pipelines and infrastructure that supply them.

Also, when a machine learning model needs to be integrated into an application, Machine Learning Engineers are involved.

Analytical Datasets

An analytical dataset is prepared for a defined analysis. It isn't a random collection of files: it has a known schema, grain, population, time period, quality criteria, and lineage. The team selects data because it is relevant to the question rather than including every available field.

In the university use case, to study if taxi service improves attendance, a dataset could be built with records of trips and class attendance of students who have or haven't traveled, allowing for a comparison of their attendance statistics.

On the other hand, to predict transportation demand, it would be more appropriate to build another dataset that integrates travel history with class schedules, the academic calendar, or weather conditions. Thus, although both sets may reuse some data sources, their structure, granularity, and quality rules would differ, as each must be designed to address the specific business question.

The design of how a dataset should be is the responsibility of a Data Analyst or Data Scientist, while the implementation of transformations and other processes necessary for its construction is carried out by Analytics Engineers. But if data from multiple sources need to be integrated, a Data Engineer handles this task, as we have seen.

These datasets are usually materialized in the form of tables in a Data Warehouse, Data Lake, or as column-oriented files like Apache Parquet.

Exploratory Data Analysis

Most analyses include Exploratory Data Analysis (EDA) because you rarely understand a new dataset perfectly at the start.

EDA examines the dataset's distributions, patterns, relationships, and unusual values before the team draws conclusions or builds a model. It also reviews types, missing values, duplicates, quality limitations, and possible sources of bias.

Regarding exploration techniques, descriptive statistics and the creation of visualizations are usually key. For example, a Data Analyst might represent the number of enrollments paid per day, compare the payment methods used, and analyze when more incidents occur. This way, they could discover if any of the payment platforms or banks involved in the transactions have caused problems with enrollment payments at any point.

They might also observe phenomena such as students who pay earlier achieving better academic results, but that correlation wouldn't prove that paying in advance is the main cause. Still, exploration serves to generate this hypothesis and detect possible alternative explanations, but not to confirm a causal relationship on its own.

EDA is performed by both Data Analysts and Data Scientists, though in different ways, as analysts seek to make diagnoses, while scientists explore the data to decide how to model it.

The technologies they use for exploration are very diverse, from SQL for querying the dataset, Jupyter notebooks for more easily documenting Python code, to Python libraries like pandas, NumPy, SciPy, Matplotlib, and Seaborn. Other languages that also allow data exploration include R, Julia, or Scala.

Feature Engineering

After exploring the data, transformations are often applied to make them more useful depending on the intended purpose. If we view the data as a set of records where each takes values in a series of attributes called features, sometimes these features may be more or less useful for training a machine learning model or simply for understanding the data.

For example, if we have student records in the form (name, email, 1), having a feature with a fixed value of 1 doesn't contribute to an analysis unless it's a relevant feature that always takes the value 1 for some realistic reason. In this case, it would be ideal to remove the feature and keep only the most useful ones.

Feature Engineering transforms or derives model inputs so they represent the problem usefully. Techniques include normalization or standardization for scale-sensitive algorithms, data imputation for suitable missing values, encoding categories, and discretization. Each choice should follow the business meaning, model type, and evaluation plan rather than a fixed recipe.

For example, imagine the university wants to predict whether a student will finish the master's program. To do this, they have an analytical dataset with records whose features include class attendance, grades, and the number of accesses to the virtual campus, which will later be used to train a machine learning model for prediction.

For a model that is sensitive to feature scale, normalization may help because grades range from 0 to 10 while portal-access counts can reach thousands. Min-max scaling can map them to [0, 1], although other algorithms or scaling methods may be more suitable.

The team must also exclude information that wouldn't be available at prediction time. If a feature reveals the outcome directly or indirectly, data leakage can make evaluation look unrealistically good.

These transformations are usually performed by a Data Scientist, Analytics Engineers, Data Engineers, or a Machine Learning Engineer, primarily. All these roles use technologies like SQL, Apache Spark, or Python to perform them, though these aren't the only ones.

Experimentation

Many analyses test a hypothesis. If the team believes a feature doesn't improve a model, it can state that idea clearly and use experiments to compare a model trained with and without the feature.

Experimentation changes controlled parts of a dataset, method, or training process to test a hypothesis. The work is iterative: one result can reject the original idea or suggest a better question for the next experiment.

In this field, it's important to distinguish between two types of experimentation with different purposes. First, there's analytical experimentation, which is conducted on already collected data and focuses on comparing features, types of models, and training techniques to determine which combination of these elements best answers the business question.

For example, to predict if a student will complete their master's program, the university might start with a simple model using only grades and attendance. Then, they could run another experiment incorporating the number of virtual campus logins or try a different algorithm.

This way, they could determine if the change truly enhances predictive capability or merely increases model complexity.

Also, this model should be evaluated with data not used in its training. Otherwise, it might "cheat," performing well with training data but failing to "generalize" and achieve the same performance with real data.

Controlled experiments introduce a change and compare outcomes between a treatment group that receives it and a control group that doesn't. Random assignment, when feasible and ethical, helps make the groups comparable.

For instance, to see if a taxi service improves attendance, the university could gradually introduce it to a small group of students, provided it's ethically and legally appropriate. Here, the hypothesis would be that the service improves attendance, tested by analyzing treatment data from students who received the service against control data from those who didn't, using metrics like the percentage of classes attended.

Experiments should be reproducible. Teams can version code and configuration with Git, while platforms like MLflow record runs, parameters, metrics, and artifacts.

Data Scientists usually formulate hypotheses and design model experiments with Data Analysts and domain experts. Machine Learning Engineers may help make the training and evaluation workflow reliable at production scale.

Model-Ready Data

Analytical datasets are often ready for analysis but this isn't always the case. If your goal is to train a machine learning model to make predictions, then the dataset must meet additional conditions.

To train a model, the data needs to be model-ready: prepared for the selected algorithm, evaluation design, and production use. A supervised-learning dataset needs a target variable that records the outcome to learn. Unsupervised methods can work without labels, so model-ready requirements depend on the task.

For example, to predict whether a student will leave a master's program, historical training records need an outcome label such as (student_reference, enrolled_subjects, withdrew). The team should exclude direct identifiers such as names from model features unless there's a justified need, and it must review whether the proposed prediction is fair and appropriate to use.

The team also separates data for training, validation, and final testing as the evaluation design requires. It develops the model without using the held-out test data for decisions, then uses that test set for an honest estimate of performance on unseen cases. For time-based predictions, the split should also respect chronology.

Finally, model-ready also implies that the data is representative of the target concept we want the model to "learn." For example, if we train a model to predict master's program dropout using only data from those who have dropped out, it likely won't learn the patterns indicating when someone doesn't drop out, making the dataset unrepresentative.

Thus, ensuring datasets are model-ready is the responsibility of Data Engineers, Data Scientists, and Machine Learning Engineers who may use them.

Analytical Product Delivery

Analysis creates value only when its results reach the right people or systems in a usable form. If the university uses a model to identify unusual exam activity, for example, it should treat the output as a signal for authorized human review rather than proof of misconduct.

Analytical Product Delivery provides the right consumption channel for each result. That channel might be a report, dashboard, alert, file, API, or prediction embedded in an application.

For instance, the university could deliver attendance analysis through a report or dashboard. A carefully governed model that estimates withdrawal risk might provide limited alerts through an internal API to an authorized support team, which would review the context before offering help. The channel and controls should match the intended use and potential impact.

Relevant practices in Analytical Product Delivery include defining the consumers of the results, their update frequency, and quality metrics. All this is documented along with data sources and other aspects, and the delivery mechanisms are monitored.

In the example, a dashboard with attendance analysis would have the rectorate and master's coordinators as consumers, updating with new data monthly. Meanwhile, the dropout prediction model would deliver its alerts to an academic officer via an API, even if this officer accesses it with an application.

The delivery is coordinated by the Data Product Owner or Product Manager, while technical teams with professionals like Analytics Engineers, Software Engineers, or BI Developers are responsible for implementing all the result delivery mechanisms.

Data Products

An analytical result isn't automatically a product. A Data Product packages governed data with a way for defined consumers to use it and an operating model that keeps it useful over time.

It may take the form of a dataset, API, dashboard, or another interface. A dashboard or file alone isn't necessarily a Data Product: it needs a clear purpose, known consumers, ownership, documentation, and defined quality and service expectations.

In our focused use case, the university could create a Mobility Eligibility Data Product. It would combine only the approved enrollment, in-person schedule, distance, and eligibility attributes needed for the transportation benefit. An API could return an eligibility decision and its effective date to the student portal, while a separate governed dataset could provide aggregated service metrics.

Keeping this product narrow avoids exposing a complete student profile to consumers that don't need it.

Product Characteristics

In this context, managing a Data Product should be done just like a commercial product, hence the need to define its consumers and those responsible, and to ensure its quality and availability.

But in the realm of data, there are certain fundamental characteristics for any product:

• Discoverable: It must be accessible through a data catalog or the appropriate tool.

• Understandable: The data schema, its semantics, and all aspects that facilitate its comprehension and traceability, such as lineage, must be documented.

• Reliable: Quality and availability are measured against clear expectations, with monitoring and a response process when the product misses them.

• Secure: Access controls are implemented, and the exposure of personal data is minimized.

• Interoperable: The data should be able to be integrated and function correctly in other systems.

• Stable: This means the data shouldn't undergo frequent changes in its schema, properties, or consumption methods.

A Data Contract can formalize important parts of the product interface, such as schema, semantics, quality rules, and update frequency. The product also needs documentation for ownership, access, support, lifecycle, and consumer expectations.

Ownership and Lifecycle

No single role builds a Data Product alone. The Data Product Owner works with consumers, defines requirements, and sets objectives based on expected value.

On a technical level, there are Data Engineers, Analytics Engineers, or Platform Engineers, among others, who operate the infrastructure for storing and analyzing data, generating the results that become a product.

The lifecycle includes identifying consumer needs, defining the product and its contract, building and releasing it, monitoring service and data quality, improving it, and eventually retiring it.

Adoption is one sign of success, but it isn't enough by itself. The product should help consumers achieve a valuable outcome while maintaining quality, availability, security, and sustainable operating cost.

Data Management Organization

We've covered many capabilities, technologies, and roles. The Data Management Organization defines how these people work together, make decisions, and resolve issues across the lifecycle.

Its operating model assigns authority and responsibility, sets forums and workflows, and gives teams a consistent way to resolve problems and deliver value.

Operating Model

An operating model organizes decision-making and delivery. In a centralized model, one data team handles most of the work. This can improve consistency, but the team may become distant from domain knowledge or turn into a bottleneck.

Another type of operating model is decentralized, where each department or area of the organization manages the data within its domain, increasing autonomy but at the cost of a higher risk of inconsistencies and data silos, making global decision-making more difficult.

Data silos refer to sets of information isolated within an area or system, making them inaccessible or very difficult to reach for the rest of the organization.

Many organizations use a hybrid or federated model, which seeks to combine the advantages of both approaches. Here, each domain maintains a certain degree of autonomy over its data and is responsible for its quality, documentation, and use, while a central unit establishes governance principles, standards, and policies that must be respected throughout the organization.

For example, a hybrid organizational model at a university could have a central Data Management Office led by the CDO, while different domains like Academic Activity, Finance, or Mobility would have their own Data Owners, Data Stewards, and technical teams. If multiple domains need to collaborate, a Data Governance Council could assist in decision-making related to this collaboration.

Roles and Collaboration

The main roles in this context have already been mentioned. But regarding collaboration among them, it's crucial that their responsibilities are clearly defined and documented. This can be formalized through documentation, tools like a RACI matrix, Data Contracts, Governance Charters, or by setting up workflows.

For proper coordination, technologies like Git repositories are used to collaboratively version their work, data catalogs, platforms similar to Jira for communication, and observability tools. But technology doesn't replace the need for authority, communication, and clear responsibilities.

Data Management Maturity

Organizations differ in how consistently they apply these capabilities. Data Management Maturity describes how well practices are embedded, measured, governed, and aligned with organizational goals.

For example, an organization with low maturity would manage data with isolated and ad-hoc actions based on arising needs. As maturity increases, processes and management practices begin to be documented to become standardized, governed, and properly automated. At the highest levels of maturity, a managed approach is adopted, where the management strategy is controlled through quality metrics, audits, and formal risk management.

Maturity focuses not only on the technical aspect but also on the ability to coordinate personnel, their responsibilities, and the tools they use to achieve sustainable results aligned with the organization's strategy.

Maturity Levels

One illustrative maturity model uses the following levels:

• Level 0 – No Capability: There are no organized practices for managing data. Actions are taken as deemed appropriate at the moment.

• Level 1 – Initial: Management is assigned to specific professionals, but there's no control over individual actions or collaboration methods.

• Level 2 – Managed: Processes, roles, and tools begin to be documented to facilitate the replication and automation of management tasks.

• Level 3 – Defined: Policies and standards are formalized and unified across the organization, ensuring all teams work in a coordinated and scalable manner.

• Level 4 – Measured: Management is controlled more deeply through audits and metrics to evaluate performance and actively mitigate risks.

• Level 5 – Optimized: Teams use measurements, feedback, and appropriate automation to improve management continuously and reduce problems before they affect consumers.

Assessment and Roadmap

To determine the maturity level and enhance it within your organization, your team can use a Data Management Maturity Assessment.

This process begins by defining which data domains and management capabilities are to be evaluated. Evidence is then gathered to analyze the maturity level achieved with these capabilities, examining what is documented, which policies are followed, and so on.

By comparing with a target maturity level, a roadmap is developed to reach it, with steps that can vary significantly depending on the specific organization and its current level.

This process is led by the CDO or the Data Governance Office, with participation from Data Owners, Data Stewards, and technical teams.

For example, at the university, the Mobility domain would be at level 1 if student eligibility for the service were reviewed manually and depended on specific individuals' knowledge. At level 2, responsibilities would be assigned, documentation on the concept of eligibility would begin, and basic validations would be automated.

At level 3, Data Products could unify access to selected mobility information under shared rules. At level 4, dashboards could track quality, availability, usage, cost, fairness, and incidents. At level 5, teams would automate low-risk work where appropriate, keep human review and appeal paths for consequential eligibility decisions, and improve the service continuously through metrics and user feedback.

But the goal doesn't have to be reaching level 5 in all capabilities. The university might require high maturity in security and quality capabilities that protect personal data, while a more experimental analysis of classroom usage that doesn't involve personal data might have a lower target.

Conclusions

Throughout this book, we've treated Data Management as a coordinated set of capabilities that helps an organization capture, integrate, protect, understand, and use data throughout its lifecycle.

The wider university ecosystem shows the scale of a real organization, while our admissions, academic-activity, and transportation examples make the connections concrete. Even a controlled transportation benefit requires much more than a database: it needs governance, quality, privacy, integration, reliable operations, and careful analysis.

Data doesn't generate value automatically. It becomes useful when people give it context, protect it, make it available to the right consumers, and connect it to a real goal. Technology is the means, not the objective. Databases, pipelines, dashboards, models, and Data Products matter only when they solve a genuine need.

The result? Complex queries, skyrocketing read costs, and a database structure that becomes a nightmare to maintain after just a few features.

To understand how Firestore works, we first need to look at our point of comparison: the relational model. Once we map out how SQL does things, we can see exactly where Firestore diverges, and how to structure NoSQL data correctly.

In this guide, we'll cover NoSQL design principles, embedding vs. referencing, and relationship modeling (1-1, 1-N, N-N). We'll also walk through a concrete blog case study.

Table of Contents

1. Prerequisites

2. The Relational Mindset: How SQL Handles Data

3. The Firestore Paradigm: NoSQL with Relationships

4. The Core Building Blocks: Documents and Collections

5. The Golden Rule: Model for Reads, Not Writes

6. Embedding vs. Referencing (Denormalization)

7. How to Model Relationships (1-1, 1-N, N-N)

8. Best Practices and Pitfalls to Avoid

9. Case Study: Designing a Scalable Blog Database

Prerequisites

This guide is conceptual, so you don't need a running Firestore project to follow along. A little context is enough. You will need:

• Basic JavaScript syntax, since every code example uses the modular Firebase JS SDK (v9+)

• Basic familiarity with JSON objects (keys, values, nested objects, arrays)

• Some exposure to SQL or relational databases helps, since the guide leans on that comparison throughout (but it's not required)

• (Optional) A free Firebase project, if you want to try the examples yourself. The Firestore quickstart walks you through setting one up.

No prior NoSQL or Firestore experience is needed.

The Relational Mindset: How SQL Handles Data

In a relational database, data is organized into tables linked by explicit relationships. This approach relies on normalization to eliminate data redundancy.

For example, to store users and their respective countries, we split the data into two tables:

• Users: columns id (PK), last_name, first_name, #country_id (FK a foreign key)

• Countries: columns country_id (PK), country_name

With a row like 1, MINTOUMBA, Caleb, 1 in Users and 1, Canada in Countries, we automatically know that Caleb belongs to Canada through the foreign key #country_id. We never had to write the word "Canada" inside the Users table itself.

The SQL trade-off: writes are lightweight (you only update data in one place), but reads are heavier, because you have to perform a database join (JOIN) every time you want to display a user's country name.

That's exactly the opposite of how Firestore works, as we'll see next.

The Firestore Paradigm: NoSQL with Relationships

Firestore is a NoSQL document database – literally Not Only SQL. It stores JSON-like documents grouped into collections, with no enforced schema.

For most of Firestore's history, that also meant no native joins and no GROUP BY. The standard query engine simply didn't support them, and any aggregation beyond count(), sum(), and average() had to happen in your application code.

That's still true today for Standard edition, which remains the default and the one most mobile/web apps run on and the one this guide focuses on.

Google has since introduced Firestore Enterprise edition, built around a new Pipeline query engine that reached general availability in April 2026. Pipelines add a multi-stage query syntax and hundreds of new functions, including relational-style joins through correlated subqueries and a real aggregate(...) step with grouping Firestore's equivalent of SQL's GROUP BY.

Does this mean data modeling doesn't matter anymore? Not for most apps. Pipeline queries run within a 60-second timeout and a 128 MiB working-memory limit, can fall back to full collection scans when no index exists, and critically, Enterprise edition drops real-time listeners and offline support (which most Firestore client apps depend on).

Pipelines are a genuine escape hatch for analytical, admin, or reporting queries. They're not a drop-in replacement for the read-optimized structure your app's everyday screens still need.

If you're building a typical client-facing app on Standard edition, the embedding and denormalization strategies below are still how you model relationships.

But NoSQL doesn't mean "no relationships" even on Standard edition. You can and should build robust relationships between your collections. The difference is that Firestore won't enforce or resolve them for you the way a JOIN does by default. It's up to you, the developer, to build and query those relationships explicitly, and to maintain data integrity through your application code or Cloud Functions unless you've specifically opted into Enterprise edition for Pipeline-powered joins.

The Core Building Blocks: Documents and Collections

Before designing any schema, let's define Firestore's two core building blocks:

• Document: the basic unit of storage. It's a JSON-like object, identified by a unique ID, containing typed fields (strings, numbers, booleans, timestamps, geopoints, or references to other documents).

• Collection: a container for documents. Unlike SQL tables, documents in the same collection don't need to share the same structure.

What makes Firestore unique is its hierarchical nature: a document can contain sub-collections, which contain more documents, which can themselves contain more sub-collections, and so on.

In the diagram above, the root posts collection contains the document post_001, which itself hosts a comments sub-collection containing the individual comment documents comment_001 and comment_002. You can nest collections and documents several levels deep, but as we'll see later, it's best to do so sparingly.

Crucial rule: sub-collections are never retrieved automatically when you read a parent document. Unlike a SQL JOIN, you must always perform a separate, explicit query to read a sub-collection.

The Golden Rule: Model for Reads, Not Writes

This is the single most important concept in NoSQL modeling, and the one developers coming from SQL forget most often: structure your data based on how your app queries it, not on how it gets written.

Before writing any database code, ask yourself:

• Which screens in my app will display this data?

• Do I need this piece of data on its own, or always alongside another one?

• Do I read this information significantly more often than I write or update it?

If your users view a writer's profile 10,000 times for every single time that writer updates their username, optimize for the reads: duplicate the username directly inside each post. That's the exact opposite of the SQL instinct we saw earlier, where you normalize first to avoid redundancy, even if it makes reads heavier.

Embedding vs. Referencing (Denormalization)

There are two primary strategies for representing a relationship in Firestore.

Option A: Embedding (Nesting)

You store the related data directly inside the parent document, as an array or a map (object).

// A post with its comments embedded
{
  title: "Introduction to Firestore",
  author: "Caleb",
  comments: [
    { user: "Ama", text: "Great post!" },
    { user: "Kofi", text: "Thanks for the examples" }
  ]
}

• Pros: a single read retrieves everything, and consistency is guaranteed.

• Cons: Firestore documents have a hard 1 MB size limit. If the nested list grows indefinitely (comments on a viral post, for instance), your writes will start failing once you hit that limit and every write to the parent document also re-sends the whole document to any client listening in real time.

• Best for: small, bounded lists (tags on an article, a user's settings, a short list of favorites).

Option B: Referencing (Denormalization)

You split the entities into separate collections or sub-collections, and deliberately duplicate a few fields to avoid a second read.

// posts/post_001
{
  title: "Introduction to Firestore",
  authorId: "uid_123",
  authorName: "Caleb",      // denormalized: avoids a second read to "users"
  authorAvatar: "https://...",
  commentCount: 12          // denormalized counter
}

// posts/post_001/comments/comment_001
{
  userId: "uid_456",
  userName: "Ama",
  text: "Great post!",
  createdAt: Timestamp
}

Here, we duplicate the author's name and avatar into every post so we don't need an extra read to users every time the post list is displayed.

That's denormalization: we accept controlled redundancy in exchange for faster reads the exact opposite of SQL normalization. The cost is that these copies need updating if the user changes their name (usually handled by a Cloud Function triggered when the users document is updated).

• Pros: no document size limits, and entities can be queried independently.

• Cons: requires multiple reads if you didn't denormalize enough. If a duplicated value changes, you need code (often a Cloud Function) to propagate the update everywhere it's copied.

• Best for: dynamic, fast-growing data (comments, order history, activity logs).

A more precise rule of thumb: whether to reference instead of embed depends on volume. Sub-collections handle unbounded growth (comments, order history) better than arrays.

Whether to denormalize a given field depends on the cost of keeping it in sync, not how often it changes: a counter you update in place with an atomic increment (commentCount, likeCount) has no other copy to synchronize, so it's cheap to denormalize regardless of frequency.

A copied value like authorName, on the other hand, is duplicated across every document that references it. It's safe to denormalize only if it changes rarely, since any change means propagating the update everywhere it's been copied.

How to Model Relationships (1-1, 1-N, N-N)

One-to-One (1-1)

Either embed the fields in the same document, or store them in a separate collection using the exact same document ID, for example users/uid_123 and privateProfiles/uid_123. This is perfect for separating public data from sensitive data that needs different security rules.

One-to-Many (1-N)

There are three main options, depending on volume and query direction:

1. A sub-collection (posts/post_001/comments/*) is ideal when you almost always query comments through their parent post, and volume can be large.

2. A root collection with a reference (comments with a postId field) is useful if you also need to query all comments by a given user, independently of the post (where("userId", "==", uid)).

3. Use an embedded array only if the volume stays small and bounded (see Option A above).

Many-to-Many (N-N)

This is the trickiest one in NoSQL, since there's no automatic join table like in SQL. There are three common patterns:

(1). Junction collection the equivalent of a SQL pivot table:

// memberships/{membershipId}
{
  userId: "uid_123",
  groupId: "group_789",
  role: "admin",
  joinedAt: Timestamp
}

You can then query .where("userId", "==", uid) to find all groups a user belongs to, or .where("groupId", "==", gid) to find all members of a group.

(2). ID arrays on both sides (cross-denormalization):

// users/uid_123      -> groupIds: ["group_789", "group_456"]
// groups/group_789   -> memberIds: ["uid_123", "uid_456"]

Fast to read from either side, but reserve this for lists that stay small the 1 MB document limit and the cost of atomically updating long arrays both work against you at scale.

(3). Hybrid approach, which is the most common pattern in practice: an array for a lightweight relationship rarely queried from the other side (a user's favorite posts), and a junction collection for a relationship queried frequently in both directions and prone to frequent changes (team memberships).

Best Practices and Pitfalls to Avoid

• Limit nesting depth: Firestore allows sub-collections to be nested indefinitely, but beyond two or three levels, your queries and security rules become genuinely hard to maintain. Prefer flattening the structure with references when you can.

• Avoid auto-incremented document IDs: Sequential IDs (user_1, user_2, user_3...) can cause hotspotting: writes pile up on a narrow range of the index, which degrades performance at scale. Let Firestore generate random, evenly distributed IDs unless you have a specific reason not to.

• Watch out for composite indexes: Any query combining multiple .where() filters, or a .where() with an .orderBy() on a different field, requires a composite index. Plan for these during design rather than discovering them in production (Firestore's error messages include a direct link to auto-generate the missing index).

• Mind the write rate on "hot" documents: The recommended maximum sustained write rate to a single document is about 1 write per second. A document updated very frequently by many different users a global like counter, for example becomes a bottleneck well before that. Firestore can absorb short bursts (5, 10, even 50 writes in one second) by queuing them, but sustained traffic above ~1 write/sec will start producing contention errors. The standard fix is a sharded counter: split the count across several sub-documents and sum them at read time.

• Use sub-collections deliberately: They're convenient, but they always require a separate query. If you almost always need the data together, embedding or denormalization will perform better.

• Design security rules alongside your data model: Firestore's security rules (firestore.rules) should be designed at the same time as your schema a poorly thought-out structure usually makes precise rules much harder to write.

Case Study: Designing a Scalable Blog Database

Let's bring every principle from this guide together with a concrete example: a blog with posts, comments, and likes.

// posts/{postId}
{
  title: "Modeling Firestore",
  slug: "modeling-firestore",
  authorId: "uid_123",
  authorName: "Caleb",         // denormalized: avoids a second read to "users"
  content: "...",
  tags: ["firebase", "nosql"], // embedded: small, bounded list
  commentCount: 3,             // denormalized counter
  likeCount: 47,               // denormalized counter (shard it if traffic is high)
  createdAt: Timestamp
}

// posts/{postId}/comments/{commentId}  → sub-collection: read together with the post
{
  userId: "uid_456",
  userName: "Ama",
  text: "Excellent article",
  createdAt: Timestamp
}

// likes/{likeId}  → root collection + reference
{                    // lets you quickly check if ONE user liked ONE post
  postId: "post_001",
  userId: "uid_456"
}

Each choice here answers a specific read pattern. Tags are always displayed alongside the post, so they're embedded. Comments can grow large in number and are almost always fetched together with their parent post, so they live in a sub-collection. Likes need to be queried both by post and by user to check whether this user already liked this post so they sit in a root collection with two indexable fields.

Conclusion

In SQL, you normalize to eliminate redundancy, and you pay for that choice at read time, via joins. In Firestore, it's the opposite: you accept controlled redundancy (denormalization) to make reads instant and cheap, at the cost of slightly heavier writes.

Modeling data in Firestore isn't about applying relational habits with a different syntax. It's a genuinely different way of thinking, centered on your app's read patterns.

Always ask "how will I read this data, and how often?" before choosing between embedding, referencing, or a sub-collection. Also, keep Firestore's concrete limits in mind (1 MB per document, composite indexes, hotspotting) from the design phase rather than discovering them in production.

That balance between read simplicity and write cost is what separates a Firestore database that scales gracefully from one you'll be rewriting six months from now.

Slow SQL queries are one of the biggest bottlenecks in enterprise applications. This guide demonstrates how to analyze execution plans, design effective indexes, rewrite inefficient T-SQL queries, optimize joins and aggregations, and monitor performance using SQL Server tools.

By working through several practical examples, you'll learn how to build faster, scalable, and more maintainable SQL Server workloads.

Table of Contents

• Introduction

• Prerequisites

• Why Query Performance Matters in Enterprise Applications

• How SQL Server Executes Queries

• Understanding Execution Plans

• Common Execution Plan Operators

• Finding Slow Queries

• Writing Efficient WHERE Clauses

• Optimizing JOIN Operations

• Optimizing Aggregations

• Common Table Expressions vs. Temporary Tables

• Avoiding Common T-SQL Performance Anti-Patterns

• Measuring Before and After Optimization

• Monitoring Query Performance

• Real-World Example: Optimizing a Reporting Query

• When NOT to Optimize Prematurely

• Best Practices for Enterprise T-SQL Optimization

• Future Trends in SQL Performance Optimization

• Conclusion

Introduction

Enterprise application performance often depends more on the database than the application itself. Whether you're building with ASP.NET Core, Java Spring Boot, or Node.js, inefficient database queries can lead to slow API responses, page load delays, timeout errors, and increased infrastructure costs.

While adding CPU, memory, or database replicas may temporarily improve performance, the root cause is often inefficient T-SQL queries, poorly designed indexes, outdated statistics, or suboptimal execution plans. Since the same queries may execute thousands of times per minute, even small optimizations can significantly reduce latency and resource consumption.

In enterprise environments, where databases often contain millions of records and support highly concurrent workloads, query tuning becomes essential for maintaining scalability and responsiveness.

In this article, you'll learn how SQL Server executes queries, how to analyze execution plans, optimize T-SQL, design effective indexing strategies, and apply practical techniques to improve database performance in real-world applications.

Prerequisites

To get the most from this tutorial, you should be familiar with:

• Basic SQL and T-SQL syntax

• Microsoft SQL Server fundamentals

• Primary keys and foreign keys

• Basic understanding of indexes

• SQL Server Management Studio (SSMS) or Azure Data Studio

• Basic knowledge of relational database concepts

Why Query Performance Matters in Enterprise Applications

Database performance directly affects every layer of an enterprise application. Even if the frontend is highly optimized and the application servers are properly scaled, slow database operations quickly become the limiting factor.

Consider a typical enterprise architecture:

Figure 1. High-Level Architecture Showing Client, ASP.NET Core API, Business Services, and SQL Server Database

Figure 1 illustrates the high-level architecture of a typical ASP.NET Core application. Client requests are received by the ASP.NET Core API, which serves as the application's entry point. The API forwards these requests to the Business Services layer, where the core business logic is executed. The Business Services layer then interacts with the SQL Server Database to retrieve or persist data. The sequential flow of requests through these components is represented by the arrows in the diagram.

Although this architecture separates responsibilities and improves maintainability, its overall performance is often constrained by the database. Every request that requires data eventually reaches the SQL Server database. If the database responds slowly, every upstream component (including the Business Services layer, the API, and ultimately the client) must wait for the query to complete.

Consider an order management system in which a dashboard displays customer information, recent orders, invoices, inventory levels, and shipment status. Loading this dashboard may require several independent database queries. While these queries may execute concurrently, the user perceives the combined response time. Consequently, even a small number of poorly optimized queries can significantly increase page load times and degrade the overall user experience.

As database size and application usage grow, performance issues often become increasingly apparent.

Common symptoms include:

• APIs that gradually become slower as data grows

• High CPU utilization on the SQL Server

• Excessive disk I/O

• Blocking between concurrent transactions

• Deadlocks during peak usage

• Timeout exceptions in application logs

Many of these problems originate from inefficient SQL rather than insufficient hardware.

For example, suppose a customer table contains ten million records. Searching for customers by email without an appropriate index forces SQL Server to examine every row.

SELECT *
FROM Customers
WHERE Email = 'john@example.com';

Without an index on the Email column, SQL Server performs a table scan, reading every page before locating the desired row.

Adding a properly designed index transforms the same query into an index seek, allowing SQL Server to locate the record almost immediately.

As enterprise datasets continue growing, these differences become increasingly significant.

How SQL Server Executes Queries

Understanding SQL Server's execution process is essential before attempting optimization.

Every query passes through several stages before data is returned.

Step 1: Parsing

SQL Server first validates the syntax.

SELECT Name
FROM Customers;

If the statement contains syntax errors, execution stops immediately.

Step 2: Binding

Next, SQL Server verifies that referenced tables, columns, functions, and objects exist.

For example,

SELECT CustomerName
FROM Customers;

If CustomerName doesn't exist, SQL Server reports an error before optimization begins.

Step 3: Query Optimization

The SQL Server Query Optimizer evaluates multiple possible execution strategies.

It estimates the cost of various approaches, including table scans, index seeks, different join algorithms, parallel execution, and sorting methods.

The optimizer chooses the plan with the lowest estimated cost based on available statistics.

Importantly, developers don't tell SQL Server how to execute a query. They specify what data they need.

Step 4: Execution Plan Generation

The optimizer then generates an execution plan.

The execution plan acts as a blueprint describing every operation required to satisfy the query.

For example:

SELECT *
FROM Orders
WHERE CustomerID = 1250;

Depending on available indexes, SQL Server may choose either clustered Index Seek, Nonclustered Index Seek, Index Scan, or Table Scan. Understanding these operators is the foundation of effective tuning.

Understanding Execution Plans

Execution plans reveal how SQL Server actually processes a query. Rather than guessing why a query performs poorly, execution plans identify the most expensive operations directly.

SQL Server provides two primary plan types:

1. Estimated Execution Plan: Generated without executing the query. It predicts the optimizer's chosen strategy using available statistics.

2. Actual Execution Plan: Generated after the query runs, showing the real execution path along with runtime statistics such as row counts and operator costs.

For performance tuning, the actual execution plan is generally more valuable because it exposes differences between estimated and actual behavior.

In SQL Server Management Studio, you can enable the actual execution plan by selecting Include Actual Execution Plan before running your query.

Common Execution Plan Operators

Understanding a handful of common operators makes execution plans much easier to interpret.

Table Scan

A table scan reads every row in a table.

Customers ──► Table Scan

This is acceptable for small lookup tables but becomes increasingly expensive as tables grow.

Index Scan

An index scan reads every entry within an index.

Although better than scanning the full table, it still processes every index page.

Index Seek

An index seek navigates directly to matching rows.

CustomerID Index
        │
        ▼
  Index Seek

This is generally the most efficient access method for selective queries.

Nested Loop Join

Nested Loop joins perform well when one input contains relatively few rows.

Customers
  │
  ▼
Nested Loop
  ▲

│
Orders

They're commonly used for OLTP workloads.

Hash Match

Hash joins excel when processing large datasets with no useful indexes. But keep in mind that they consume more memory and may spill to disk if insufficient memory is available.

Merge Join

Merge joins require sorted inputs but can process large result sets efficiently. They're often selected when both datasets are already indexed appropriately.

Key Lookup

One operator that frequently surprises developers is the Key Lookup.

Suppose an index contains only the CustomerID column, but the query also requests Address and PhoneNumber.

SQL Server first performs an Index Seek to locate matching rows, then executes additional lookups against the clustered index to retrieve missing columns.

Although acceptable for a few rows, thousands of key lookups can significantly degrade performance.

In many cases, creating a covering index eliminates these extra lookups entirely. This is a topic we'll explore later in the article.

Finding Slow Queries

SQL Server provides several built-in tools that help locate performance bottlenecks in production environments.

Using Query Store

Query Store records query history, execution plans, runtime statistics, and performance trends over time. Rather than relying on temporary monitoring sessions, it continuously captures valuable performance information, making it one of the most useful features for enterprise SQL Server deployments.

For example, if an application suddenly becomes slower after a deployment, Query Store can compare execution plans before and after the change to determine whether the optimizer selected a less efficient plan.

Typical metrics available include:

• Average execution time

• CPU consumption

• Logical reads

• Execution count

• Query plan history

This historical view helps identify regressions that may otherwise be difficult to reproduce.

Using Dynamic Management Views (DMVs)

Dynamic Management Views expose internal SQL Server performance information while the server is running.

One commonly used DMV is:

SELECT TOP 10
    qs.execution_count,
    qs.total_worker_time,
    qs.total_elapsed_time,
    SUBSTRING(
        qt.text,
        qs.statement_start_offset / 2,
        (
            CASE
                WHEN qs.statement_end_offset = -1
                THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
                ELSE qs.statement_end_offset
            END - qs.statement_start_offset
        ) / 2
    ) AS QueryText
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) qt
ORDER BY qs.total_worker_time DESC;

This query identifies statements consuming the most CPU time, helping prioritize optimization efforts.

Measuring I/O and Execution Time

SQL Server also provides lightweight commands for measuring query performance.

SET STATISTICS IO ON;
SET STATISTICS TIME ON;

After enabling these options, executing a query displays additional information such as:

• Logical reads

• Physical reads

• CPU time

• Total elapsed time

Consider the following query:

SELECT *
FROM Orders
WHERE CustomerID = 1025;

The output might resemble:

Table 'Orders'.

Logical reads: 4832

SQL Server Execution Times:
CPU time = 215 ms

Elapsed time = 287 ms

After adding an appropriate index, the same query could produce:

Logical reads: 6

CPU time = 3 ms

Elapsed time = 5 ms

These measurements provide objective evidence that an optimization has improved performance.

Writing Efficient WHERE Clauses

One of the simplest ways to improve query performance is to write SARGable predicates. A query is considered SARGable (Search ARGument Able) when SQL Server can efficiently use an index to locate matching rows.

Many developers unintentionally prevent index usage by applying functions directly to indexed columns.

Consider this example:

SELECT *
FROM Orders
WHERE YEAR(OrderDate) = 2025;

Although the logic is correct, SQL Server must evaluate the YEAR() function for every row before performing the comparison. As a result, it can't efficiently seek into an index on OrderDate.

A better approach is to compare the column directly.

SELECT *
FROM Orders
WHERE OrderDate >= '2025-01-01'
AND OrderDate < '2026-01-01';

This version allows SQL Server to perform an index seek rather than scanning the entire table.

Similarly, avoid implicit data type conversions.

Instead of:

WHERE CustomerID = '100'

prefer:

WHERE CustomerID = 100

Matching the column's data type eliminates unnecessary conversions during query execution.

Optimizing JOIN Operations

Enterprise applications rarely query a single table. Most business operations involve combining data from multiple related tables, making joins one of the most important optimization areas.

Consider an order management system:

SELECT
    c.Name,
    o.OrderDate,
    o.TotalAmount
FROM Customers c
INNER JOIN Orders o
    ON c.CustomerID = o.CustomerID;

When both CustomerID columns are indexed, SQL Server can efficiently join the tables.

But poor indexing often forces SQL Server to scan one or both tables, dramatically increasing execution time.

EXISTS vs. IN

Another common optimization involves replacing IN with EXISTS for large subqueries.

Less efficient:

SELECT *
FROM Customers
WHERE CustomerID IN (
    SELECT CustomerID
    FROM Orders
);

Better:

SELECT *
FROM Customers c
WHERE EXISTS (
    SELECT 1
    FROM Orders o
    WHERE o.CustomerID = c.CustomerID
);

For correlated lookups involving large datasets, EXISTS often enables more efficient execution plans.

Eliminate Unnecessary Joins

Sometimes queries include tables whose data is never used.

For example:

SELECT
    o.OrderID,
    c.Name
FROM Orders o
INNER JOIN Customers c
    ON o.CustomerID = c.CustomerID
INNER JOIN Regions r
    ON c.RegionID = r.RegionID;

If no columns from Regions are selected or filtered, removing the join reduces unnecessary work and simplifies the execution plan.

Optimizing Aggregations

Aggregations become increasingly expensive as datasets grow. Reporting systems frequently summarize millions of rows using functions such as SUM(), COUNT(), AVG(), and MAX().

A straightforward aggregation might look like this:

SELECT
    CustomerID,
    SUM(TotalAmount)
FROM Orders
GROUP BY CustomerID;

Although simple, performance depends heavily on indexing and data distribution.

If the query repeatedly scans millions of rows, consider creating an index on CustomerID.

Window functions often provide a cleaner alternative to complex subqueries.

For example, identifying each customer's most recent order:

SELECT
    CustomerID,
    OrderDate,
    ROW_NUMBER() OVER (
        PARTITION BY CustomerID
        ORDER BY OrderDate DESC
    ) AS RowNum
FROM Orders;

Window functions allow SQL Server to calculate rankings and running totals without complicated self-joins.

Whenever possible, avoid unnecessary sorting operations, since sorting large result sets consumes considerable CPU and memory.

Common Table Expressions vs. Temporary Tables

Both Common Table Expressions (CTEs) and temporary tables help simplify complex queries, but they serve different purposes.

A CTE provides a readable way to structure intermediate query logic.

WITH RecentOrders AS
(
    SELECT *
    FROM Orders
    WHERE OrderDate >= DATEADD(DAY, -30, GETDATE())
)
SELECT *
FROM RecentOrders;

CTEs improve readability and maintainability but aren't materialized automatically. SQL Server may execute the underlying logic multiple times depending on the execution plan.

Temporary tables, on the other hand, physically store intermediate results.

SELECT *
INTO #RecentOrders
FROM Orders
WHERE OrderDate >= DATEADD(DAY, -30, GETDATE());

SELECT *
FROM #RecentOrders;

Temporary tables become particularly useful when:

• Intermediate results are reused multiple times

• Large datasets need additional indexing

• Complex joins benefit from breaking queries into stages

Choosing between the two depends on workload characteristics rather than personal preference.

Avoiding Common T-SQL Performance Anti-Patterns

Many performance issues stem from common coding habits rather than complex database problems.

Avoid SELECT *

Fetching every column increases network traffic, memory consumption, and I/O.

Instead of this:

SELECT *
FROM Customers;

Retrieve only the required columns:

SELECT
    CustomerID,
    Name,
    Email
FROM Customers;

This reduces both data transfer and execution costs.

Avoid Scalar Functions in WHERE Clauses

Scalar functions execute once per row, preventing efficient index usage.

Instead of:

WHERE UPPER(Name) = 'JOHN'

store normalized values or use case-insensitive collations where appropriate.

Avoid Cursors for Row-by-Row Processing

Cursors process records sequentially.

DECLARE CustomerCursor CURSOR
FOR
SELECT CustomerID
FROM Customers;

Although sometimes necessary, cursor-based solutions rarely scale well for enterprise workloads.

Most cursor logic can be rewritten using set-based operations.

For example:

Instead of updating rows individually:

UPDATE Customers
SET Status = 'Active'
WHERE LastLogin >= DATEADD(DAY, -30, GETDATE());

SQL Server processes the entire set efficiently rather than iterating row by row.

Reduce Correlated Subqueries

Correlated subqueries execute repeatedly for each outer row.

For example:

SELECT
    CustomerID,
    (
        SELECT COUNT(*)
        FROM Orders o
        WHERE o.CustomerID = c.CustomerID
    ) AS OrderCount
FROM Customers c;

Rewriting this using joins and aggregation often produces more efficient execution plans.

SELECT
    c.CustomerID,
    COUNT(o.OrderID) AS OrderCount
FROM Customers c
LEFT JOIN Orders o
    ON c.CustomerID = o.CustomerID
GROUP BY c.CustomerID;

The rewritten version allows SQL Server to process the data in a single pass rather than executing thousands of nested queries.

Measuring Before and After Optimization

Effective tuning always follows the same cycle:

1. Measure the original query using Query Store or SET STATISTICS.

2. Analyze the execution plan.

3. Identify expensive operators such as scans, sorts, or key lookups.

4. Apply one targeted optimization, such as rewriting the query or adding an index.

5. Measure again using the same workload.

This iterative approach ensures that every optimization is evidence-based rather than relying on assumptions. In enterprise environments, even small improvements to frequently executed queries can significantly reduce CPU usage, disk I/O, and response times.

Monitoring Query Performance

Query tuning is an ongoing process rather than a one-time optimization effort. As enterprise databases grow, data distributions change, index fragment, and application workloads evolve. Queries that once performed well may gradually become inefficient.

SQL Server provides several built-in tools for identifying performance issues.

Query Store

Query Store records query history, execution statistics, execution plans, and runtime information.

It helps answer questions such as:

• Which queries consume the most CPU?

• Which execution plans changed recently?

• Which query became slower after deployment?

• Which indexes are no longer being used?

Enable Query Store:

ALTER DATABASE SalesDB
SET QUERY_STORE = ON;

View top resource-consuming queries:

SELECT
    qt.query_sql_text,
    rs.avg_duration,
    rs.avg_cpu_time
FROM sys.query_store_query_text qt
JOIN sys.query_store_query q
    ON qt.query_text_id = q.query_text_id
JOIN sys.query_store_plan p
    ON q.query_id = p.query_id
JOIN sys.query_store_runtime_stats rs
    ON p.plan_id = rs.plan_id
ORDER BY rs.avg_duration DESC;

Instead of relying on user complaints, administrators can proactively detect regressions before they affect production workloads.

Dynamic Management Views (DMVs)

SQL Server exposes runtime statistics through Dynamic Management Views.

Example:

SELECT TOP 10
    qs.execution_count,
    qs.total_elapsed_time / qs.execution_count AS AvgTime,
    st.text
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) st
ORDER BY AvgTime DESC;

This query highlights expensive SQL statements currently cached by SQL Server.

Actual Execution Plans

Execution plans remain one of the most valuable tuning tools.

When reviewing plans, look for:

• Table scans

• Index scans on large tables

• Key lookups

• Sort operators

• Hash Match operations

• Missing index recommendations

• Large memory grants

The graphical execution plan often pinpoints the exact operator responsible for poor performance.

Real-World Example: Optimizing a Reporting Query

Consider an enterprise reporting system that generates monthly sales summaries.

Original query:

SELECT
    CustomerName,
    SUM(TotalAmount)
FROM Orders
WHERE YEAR(OrderDate) = 2025
GROUP BY CustomerName;

Although simple, this query performs poorly because YEAR() prevents index seeks and the entire Orders table must be scanned.

Rewrite it like this:

SELECT
    CustomerName,
    SUM(TotalAmount)
FROM Orders
WHERE OrderDate >= '2025-01-01'
AND OrderDate < '2026-01-01'
GROUP BY CustomerName;

Then create an appropriate index:

CREATE INDEX IX_Orders_OrderDate
ON Orders(OrderDate)
INCLUDE (CustomerName, TotalAmount);

Performance improvements may include:

• Index Seek instead of Table Scan

• Lower logical reads

• Reduced CPU utilization

• Faster execution time

• Better scalability under concurrent reporting workloads

This illustrates that query rewriting and indexing typically produce much larger gains than simply adding hardware.

When NOT to Optimize Prematurely

Performance optimization should be driven by evidence, not assumptions. Premature or unnecessary tuning can increase complexity, make queries harder to maintain, and sometimes even reduce overall system performance.

Before making changes, use tools such as Query Store, execution plans, and SQL Server DMVs to identify the actual bottlenecks.

Avoid Optimizing Without Profiling

Don't rewrite queries simply because they look inefficient. Measure execution time, logical reads, CPU usage, and execution plans first so that optimization efforts target real performance problems rather than perceived ones.

Don't Create Indexes for Every Query

While indexes can dramatically improve read performance, every additional index increases storage requirements and slows INSERT, UPDATE, and DELETE operations. Create indexes only for frequently executed queries that demonstrate a measurable benefit.

Don't Force Query Hints Unnecessarily

Query hints such as OPTION (FORCE ORDER) or OPTION (RECOMPILE) can override SQL Server's optimizer. They should be used only after careful testing, as they may solve one problem while causing performance regressions elsewhere.

SELECT *
FROM Orders
WHERE CustomerID = @CustomerID
OPTION (RECOMPILE);

Don't Over-Normalize or De-Normalize Without Evidence

Highly normalized schemas may require expensive joins, while excessive denormalization can introduce redundant data and update anomalies. Choose the appropriate design based on actual workload characteristics rather than assumptions.

Balance Read and Write Performance

An optimization that accelerates reporting queries may slow transactional workloads due to additional index maintenance. Always evaluate how tuning changes affect both read-heavy and write-heavy operations before deploying them to production.

Best Practices for Enterprise T-SQL Optimization

Successful tuning is about applying consistent engineering practices rather than isolated optimizations. We've already discussed some of these best practices, but I'll list them all here for review and completeness (and as a quick reference):

Design Indexes Around Queries

Indexes should reflect actual application workloads.

Instead of indexing every column, identify common WHERE clauses, frequently joined columns, ORDER BY columns, and GROUP BY columns.

Build indexes that support these operations efficiently.

Avoid Over-Indexing

More indexes are not always better.

Every INSERT, UPDATE, and DELETE operation must maintain every index.

Too many indexes increase storage, write latency, maintenance time, and fragmentation. Keep only indexes that provide measurable value.

Keep Statistics Updated

Outdated statistics lead to poor execution plans.

Update statistics regularly:

UPDATE STATISTICS Orders;

Or update the entire database:

EXEC sp_updatestats;

Many performance issues disappear after SQL Server receives accurate distribution statistics.

Monitor Index Fragmentation

Indexes become fragmented as data changes.

Check fragmentation:

SELECT
    avg_fragmentation_in_percent,
    page_count
FROM sys.dm_db_index_physical_stats
(
    DB_ID(),
    OBJECT_ID('Orders'),
    NULL,
    NULL,
    'LIMITED'
);

Avoid SELECT *

Retrieve only required columns.

Instead of:

SELECT *
FROM Customers;

Use this:

SELECT CustomerID,
       CustomerName,
       Email
FROM Customers;

Benefits include smaller network payloads, better covering index usage, lower memory consumption, and reduced I/O.

Test with Production-Like Data

Queries that perform well on development databases containing thousands of rows may behave very differently against production systems with hundreds of millions of records.

Always validate execution plans, memory grants, CPU usage, parallelism, and logical reads using realistic datasets.

Future Trends in SQL Performance Optimization

1. Intelligent Query Processing

Modern versions of SQL Server include features such as adaptive query processing, memory grant feedback, and automatic plan correction. These capabilities allow the query optimizer to adjust execution strategies based on actual workload patterns, improving performance without manual tuning.

2. Cloud-Native Database Optimization

Cloud database platforms provide built-in capabilities such as automatic indexing recommendations, continuous performance monitoring, and self-tuning features. These services reduce administrative overhead while helping maintain consistent query performance as workloads grow.

3. AI-Assisted Performance Tuning

Artificial intelligence is becoming a valuable assistant for database optimization. AI-powered tools can analyze execution plans, recommend indexes, identify inefficient queries, and even suggest T-SQL rewrites, enabling developers to resolve performance issues earlier in the development lifecycle.

4. Performance Engineering by Default

Database optimization is shifting from reactive troubleshooting to proactive performance engineering. By incorporating query analysis, indexing reviews, and performance testing into CI/CD pipelines, teams can detect regressions before they reach production.

Strong Fundamentals Still Matter

Despite advances in automation, understanding execution plans, indexing strategies, query design, and statistics remains essential. Automated tools provide recommendations, but experienced developers and DBAs are still needed to validate trade-offs and ensure optimizations align with business requirements.

Conclusion

Effective T-SQL performance optimization isn't about applying isolated tricks or adding indexes indiscriminately. It requires understanding how SQL Server executes queries, accesses data, and chooses execution plans.

By combining efficient query design with well-planned indexing strategies, accurate statistics, and continuous monitoring, you can dramatically reduce latency, lower resource consumption, and improve scalability across enterprise applications.

Rather than waiting for performance problems to appear in production, teams should make query tuning a routine part of the development lifecycle. Regularly reviewing execution plans, monitoring workload patterns through Query Store, validating indexes against real application behavior, and testing with production-scale datasets creates a foundation for predictable and reliable database performance.

As enterprise systems continue to grow in complexity and data volume, organizations that treat performance optimization as an ongoing engineering discipline will be better equipped to deliver responsive, scalable, and cost-effective applications.

A few hours later, a teammate pulls your branch, runs the application, and everything crashes.

"Hey," they ask across the room (or in a Slack channel), "did you change the database?"

You quickly realize you forgot to share the SQL script. You paste it into the chat. They run it. Everything works. Then, a week later, the deployment to the staging environment fails for the exact same reason. By the time this code reaches production, everyone is asking a variation of the same terrified question: "Which SQL script should I run?"

This situation is called schema drift. It happens when the state of your database diverges across different environments. Staging has one schema, production has another, and every developer's local machine is a unique snowflake of untested database modifications.

Managing database changes manually is a recipe for deployment headaches and team collaboration challenges. Application code is stateless and easy to replace. Databases are stateful. Databases have surprisingly good memories, and they rarely forget a bad migration.

Liquibase solves this problem by bringing version-control discipline to your database changes. Instead of passing around SQL files and hoping people remember to run them, you define your database changes in code. These changes travel with your application repository and execute automatically.

Here is a high-level look at how this architecture works:

Think about the journey of a single database change. A developer commits their database migration alongside their Java code into Git. When the CI/CD pipeline (or a teammate) pulls that code, the Spring Boot application starts. But before the app fully boots up and accepts web traffic, Liquibase intercepts the process. It acts as a gatekeeper, connecting to the database and applying the required schema changes. This ensures the database exactly matches the code's expectations before a single user makes a request.

Why Database Version Control Matters

If you've spent any time working on team-based applications, you've probably seen a folder structure that looks exactly like this:

project-sql-scripts/
├── create_employee_table.sql
├── create_employee_table_final.sql
├── create_employee_table_final_v2.sql
├── add_email_column.sql
├── latest.sql
└── definitely_latest_use_this_one.sql

The phrase "just run this SQL script manually" has launched many memorable incidents.

When you rely on manual database updates, you guarantee failure at scale. Onboarding a new developer becomes an archeological expedition to figure out how to build the local schema. Deployments become stressful events requiring a checklist of manual queries that must be run in a highly specific order.

Version-controlled database changes treat your schema as code. When your database changes live alongside your application logic, you gain several immediate benefits:

• Consistency: Every environment (local, staging, production) applies the exact same changes in the exact same order.

• Safety: You eliminate the human error of skipping a script or running an outdated query.

• Visibility: You can look at a Git commit and see exactly how the Java code and the database schema changed together to support a new feature.

Git solved version control for code. Liquibase helps prevent databases from becoming the rebellious sibling.

What is Liquibase?

At its core, Liquibase is a database migration tool that tracks and applies schema changes in a predictable and repeatable way.

Instead of writing loose SQL scripts, you write "migrations" (also called changeSets). Liquibase reads these files, compares them against a tracking table inside your actual database, and figures out exactly what needs to be executed to bring the database up to date.

To use Liquibase effectively, you only need to understand a few conceptual terms:

• changeLog: The master file. This is essentially a list that tells Liquibase which migration files to execute and in what order.

• changeSet: A single, atomic change to your database. Creating a table is one changeSet. Adding a column is another.

• Migration History: A table Liquibase automatically creates in your database (called DATABASECHANGELOG) to remember which changeSets have already been executed.

• Checksums: A unique hash generated for every changeSet. Liquibase uses this to detect if someone secretly modified a file after it was already executed.

When you integrate Liquibase with Spring Boot, the migration process happens completely automatically during the application startup phase.

During startup, Liquibase takes control before your web server is allowed to receive HTTP traffic. It reaches into the database and checks the tracking table to see which migrations have already run. If it finds new migrations in your local files, it locks the database to prevent concurrent updates, executes the changes, records the new history, and finally releases the lock. Only after this entire process completes does Spring Boot finish booting up.

Because Liquibase runs before Spring Boot fully initializes the web server, your application will never serve traffic with an outdated database schema. If a migration fails, the application fails to start, protecting your system from entering a broken state.

Project Setup

Now that you understand the theory, let's build something real. We're going to build the database layer for an Employee Management API.

For this project we'll use:

• Java 17+

• Spring Boot 3.x

• Maven

• Liquibase

• H2 Database

We're using H2 because it's an in-memory database that requires zero installation. You can run this project immediately without configuring Docker containers or installing database servers. But everything you learn here applies exactly the same way to PostgreSQL, MySQL, SQL Server, or Oracle.

If you're generating this project via Spring Initializr, select the following dependencies: Spring Web, Spring Data JPA, Liquibase Migration, and H2 Database.

In your pom.xml, you'll see the critical dependencies that make this work:

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>

    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.liquibase</groupId>
        <artifactId>liquibase-core</artifactId>
    </dependency>
</dependencies>

Next, configure Spring Boot to talk to H2 and find your Liquibase files. Open your src/main/resources/application.properties file and add the following:

# H2 Database Configuration
spring.datasource.url=jdbc:h2:file:./data/employeedb;DB_CLOSE_DELAY=-1
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=

# Enable H2 Console to inspect the database in your browser
spring.h2.console.enabled=true
spring.h2.console.path=/h2-console

# Liquibase Configuration
spring.liquibase.change-log=classpath:db/changelog/db.changelog-master.xml

That last line is the most important. It tells Spring Boot exactly where to find the "master list" of your database changes.

Note: We're using a file-based H2 database instead of an in-memory database. The problem with an in-memory database is that it completely wipes itself clean every time you restart Spring Boot.

While Liquibase will happily rebuild the schema from scratch on every boot, a file-based database is much better for this tutorial (and for real-world local development). With a file-based database, your data, and more importantly, your Liquibase history, will actually persist between application restarts.

Understanding Core Liquibase Concepts

Before we write our first table, we need to understand how Liquibase organizes files. Liquibase uses a hierarchical structure.

Think of it like a book. The changeLog is the table of contents, and the changeSets are the actual chapters.

1. The Master ChangeLog: This is the entry point. It rarely contains actual database changes. Instead, its only job is to include other files in a specific order.

2. Child ChangeLogs: These group related changes together.

3. ChangeSets: These are the actual, atomic database commands (like creating a table or adding a column).

Here's a visual breakdown of how this hierarchy works in a real Spring Boot project:

Liquibase organizes migrations hierarchically. You maintain a single master file that acts as a table of contents. This master file rarely holds actual SQL commands. Instead, it explicitly includes child XML files in a strict execution order. Each of those child files (like 01-create-employees.xml) contains one or more individual database commands, which Liquibase calls changeSets.

A changeSet is uniquely identified by three things:

• id: A unique string (often a number or a Jira ticket ID).

• author: The person who wrote the migration.

• file path: Where the file is located.

When Liquibase runs, it looks at a changeSet, calculates a cryptographic hash of its contents (a checksum), and records the id, author, and checksum in the database. If it sees that exact combination of id, author, and file path in the database again on the next startup, it skips it.

Create the Initial Employee Schema (Version 1)

Let's write our first version. We need a table to store employees.

First, create the master file at src/main/resources/db/changelog/db.changelog-master.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <include file="db/changelog/changes/01-create-employees.xml"/>

</databaseChangeLog>

Next, create the actual migration file at src/main/resources/db/changelog/changes/01-create-employees.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="1" author="ashutoshkrris">
        <createTable tableName="employees">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="first_name" type="VARCHAR(50)">
                <constraints nullable="false"/>
            </column>
            <column name="last_name" type="VARCHAR(50)">
                <constraints nullable="false"/>
            </column>
        </createTable>
    </changeSet>

</databaseChangeLog>

Let's look at what we just did. We defined a changeSet with an id of "1" and an author of "ashutoshkrris". Inside, we used Liquibase's XML syntax to define a table.

Why use XML instead of plain SQL? Because Liquibase is database-agnostic. This exact XML will generate the correct auto-increment syntax for PostgreSQL (SERIAL), MySQL (AUTO_INCREMENT), or Oracle (IDENTITY). You define the structure, and Liquibase translates it to the specific database dialect.

Now, run your Spring Boot application. Watch your terminal output. You'll see logs similar to this:

Liquibase realized the database was empty. It automatically created its tracking table (DATABASECHANGELOG), read our changeSet, executed the table creation, and recorded the event.

If you restart the application right now, Liquibase will run again. But this time, it'll check the DATABASECHANGELOG table, see that id="1" and author="ashutoshkrris" has already been executed, and silently skip it. Your database is now safely version-controlled.

What Just Happened?

Up to this point, Liquibase might feel a bit like magic. You dropped an XML file into a folder, started Spring Boot, and your database schema transformed.

But understanding how Liquibase actually works under the hood is critical. If you understand the startup sequence, you'll know exactly how to debug deployments when things eventually go wrong.

When your Spring Boot application starts, it doesn't immediately begin accepting web requests. First, it initializes its internal components. When it creates the Liquibase component, the migration process begins.

Here's exactly what happens during that startup phase:

Let's trace the exact sequence. When Spring Boot initializes Liquibase, the very first thing the tool does is query the lock table to ensure no other application instance is currently migrating the database. If the coast is clear, it claims the lock. It then calculates cryptographic checksums for your local XML files, compares them against the database history, executes any missing changes, and logs them. Finally, it releases the lock so the Tomcat web server can safely start.

This sequence guarantees that your application will never serve a user request before the database schema is completely ready to handle it.

Inspecting the Database: Liquibase Metadata Tables

Let's look at what this history and locking actually looks like inside the database itself. Since we configured the H2 Console earlier, we can inspect the raw tables.

While your Spring Boot application is running, open your browser and navigate to http://localhost:8080/h2-console. Connect using the JDBC URL jdbc:h2:file:./data/employeedb with the username sa and a blank password.

Inside, you'll see your employees table. You'll also see two extra tables created automatically by Liquibase: DATABASECHANGELOG and DATABASECHANGELOGLOCK.

The DATABASECHANGELOG Table

This table is the brain of your migration strategy. It acts as the permanent ledger of every database change ever applied to this environment.

If you run SELECT * FROM DATABASECHANGELOG;, you'll see output that looks like this:

Let's break down the most important columns:

• ID, AUTHOR, FILENAME: These three columns form a composite key. Together, they uniquely identify a single migration.

• DATEEXECUTED & ORDEREXECUTED: Tells you exactly when a script ran and in what sequence.

• MD5SUM: This is the cryptographic hash of your XML file. When Liquibase starts, it hashes your local XML file and compares it to this column. If you secretly edit a file after it's been executed, this hash won't match, and Liquibase will crash the startup to protect your database.

• EXECTYPE: Most of the time, this simply says EXECUTED. But it provides a crucial audit trail: if you use Liquibase commands to intentionally skip a migration but record it as finished, you'll see MARK_RAN. If a migration was skipped because its preconditions failed, you'll see SKIPPED.

• TAG: Think of this as a Git tag for your database schema. Before a major, high-risk deployment, you can configure Liquibase to "tag" the current state of the database (for example, v1.4.0). If the deployment fails catastrophically, you can trigger a rollback command telling Liquibase to undo every change applied after the v1.4.0 tag.

• CONTEXTS: This is how you manage environment-specific changes. By adding a context attribute to your changeSet (for example, <changeSet id="7" author="ashutoshkrris" context="dev, qa">), that migration will only execute if Spring Boot passes "dev" or "qa" to Liquibase on startup. Production will safely ignore it.

• LABELS: While Contexts target environments, Labels target categories of work. You can label a changeSet with a Jira ticket number (issue-842) or a release train (Q3-release). This allows advanced teams to selectively execute or roll back specific subsets of features without affecting the rest of the database.

The DATABASECHANGELOGLOCK Table

This table is tiny, but it plays a massive role in modern deployments.

If you run SELECT * FROM DATABASECHANGELOGLOCK;, you'll see a single row:

Imagine you're deploying your Spring Boot application to a Kubernetes cluster. You tell Kubernetes to spin up three identical instances simultaneously. All three instances connect to the exact same database.

If all three instances try to run the CREATE TABLE migration at the exact same millisecond, your database will throw concurrency errors. The lock table prevents this. The very first instance to reach the database sets LOCKED to TRUE. The other two instances check the table, see the lock, and politely wait.

Practical Troubleshooting Tip: Sometimes, a deployment fails catastrophically mid-migration (perhaps the server lost power). When this happens, Liquibase might die before it can set LOCKED back to FALSE.

The next time you start the application, the logs will hang indefinitely, repeating: Waiting for changelog lock....

If you're absolutely certain no other applications are currently running migrations, you can manually fix this by running a simple SQL command in your database client:

UPDATE DATABASECHANGELOGLOCK SET LOCKED = FALSE;

This forces the lock open, allowing your application to resume.

Evolving the Employee API

Software is never finished. Two weeks after your successful Version 1 deployment, the business team comes back with new requirements.

Because you now understand how Liquibase tracks history, evolving the database is simple. You just append new files to your master list.

Version 2: Adding an Email Field

The HR team needs to contact employees. You need an email column.

Create a new file at src/main/resources/db/changelog/changes/02-add-employee-email.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="2" author="ashutoshkrris">
        <addColumn tableName="employees">
            <column name="email" type="VARCHAR(100)">
                <constraints nullable="false" unique="true"/>
            </column>
        </addColumn>
    </changeSet>

</databaseChangeLog>

Add this to your db.changelog-master.xml file immediately below your first include:

<include file="db/changelog/changes/02-add-employee-email.xml"/>

When you restart the application, Liquibase checks the DATABASECHANGELOG table. It sees that id="1" is already there, so it skips it. It sees id="2" is missing, so it executes it and adds a new row to the tracking table.

Version 3: Adding Departments Support

The company is growing. Employees now belong to departments. You need a departments table and a foreign key constraint linking the two.

Create 03-add-departments.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="3" author="ashutoshkrris">
        <createTable tableName="departments">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="name" type="VARCHAR(50)">
                <constraints nullable="false" unique="true"/>
            </column>
        </createTable>
    </changeSet>

    <changeSet id="4" author="ashutoshkrris">
        <addColumn tableName="employees">
            <column name="department_id" type="BIGINT"/>
        </addColumn>
        <addForeignKeyConstraint baseTableName="employees"
                                 baseColumnNames="department_id"
                                 constraintName="fk_employee_department"
                                 referencedTableName="departments"
                                 referencedColumnNames="id"/>
    </changeSet>

</databaseChangeLog>

Notice that we used two separate changeSets in one file. This is a best practice. Each changeSet represents one logical operation. If the foreign key creation (id="4") fails, the department table creation (id="3") will still be recorded as successful, and only id="4" will roll back.

Version 4 & 5: Employee Status and Performance Indexes

Finally, HR wants to track active versus inactive staff, and the database team noticed that searching by last name is getting slow.

Create 04-status-and-indexes.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="3" author="ashutoshkrris">
        <createTable tableName="departments">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="name" type="VARCHAR(50)">
                <constraints nullable="false" unique="true"/>
            </column>
        </createTable>
    </changeSet>

    <changeSet id="4" author="ashutoshkrris">
        <addColumn tableName="employees">
            <column name="department_id" type="BIGINT"/>
        </addColumn>
        <addForeignKeyConstraint baseTableName="employees"
                                 baseColumnNames="department_id"
                                 constraintName="fk_employee_department"
                                 referencedTableName="departments"
                                 referencedColumnNames="id"/>
    </changeSet>

</databaseChangeLog>

Remember to add all new files to your db.changelog-master.xml. The order of your include statements is the exact order Liquibase will execute them.

The Golden Rule: Never Modify Executed ChangeSets

Eventually, a developer on your team will look at your 01-create-employees.xml file and notice a mistake. Perhaps they spot a typo in a column name, or perhaps they realize a column is missing a strict non-null constraint.

Their instinct, based on years of writing standard Java code, will be to open that XML file, fix the mistake, save the file, and restart the application.

Let's actually do this and see what happens.

Open your src/main/resources/db/changelog/changes/01-create-employees.xml file. Change the first_name column to given_name:

<column name="given_name" type="VARCHAR(50)">
    <constraints nullable="false"/>
</column>

Save the file and restart your Spring Boot application.

Instead of a smooth startup, your application will instantly crash, and your terminal will vomit a massive stack trace. Look closely at the top of the error logs. You should see this exact message:

Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
     1 changesets check sum
          db/changelog/changes/01-create-employees.xml::1::ashutoshkrris was: 9:66e7dcffb2b1902a4e9f01670cb5f192 but is now: 9:2bd3ef21343d3b5c9448cc50bc35deef

Here's why this happens. Once a changeSet runs against an environment, it becomes immutable history. You can't change the past.

When Liquibase starts up, it calculates a cryptographic hash (an MD5 checksum) of your local XML file. It then queries the DATABASECHANGELOG table and compares the freshly calculated hash against the hash that was recorded when the file originally executed.

If you change even a single character in a file that has already been executed, the hash changes. Liquibase detects the tampering and refuses to start. It does this to protect your data. If your XML code says a column is named first_name but the database was originally built using fist_name, your Spring Data JPA repositories are going to fail anyway.

How to Fix It (The Right Way)

If you made this mistake locally, you might be tempted to go into your database, delete the row from the DATABASECHANGELOG table, and try again. Don't do this. If this code reaches staging or production, you can't manually delete rows on production servers.

The correct way to fix a schema mistake is to roll forward.

First, undo your change in 01-create-employees.xml so the hash matches the database again. Then, write a brand new changeSet to apply the fix:

<changeSet id="7" author="ashutosh">
    <renameColumn tableName="employees" 
                  oldColumnName="first_name" 
                  newColumnName="given_name" 
                  columnDataType="VARCHAR(50)"/>
</changeSet>

Include it in your master changelog, restart the application, and the database will safely evolve to the correct state.

Working with Seed Data

Sometimes, a schema change requires initial data to be useful.

For example, in Version 3, we created a departments table. Right now, that table is completely empty. When a new developer clones the repository and spins up the project locally, they have to manually write SQL INSERT statements just to test the API.

We can automate this by making baseline data insertion part of our migration strategy.

Create a new file at src/main/resources/db/changelog/changes/05-seed-departments.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="8" author="ashutoshkrris">
        <insert tableName="departments">
            <column name="name" value="Engineering"/>
        </insert>
        <insert tableName="departments">
            <column name="name" value="Human Resources"/>
        </insert>
        <insert tableName="departments">
            <column name="name" value="Finance"/>
        </insert>
    </changeSet>

</databaseChangeLog>

Add the include statement to your db.changelog-master.xml file. When you restart the application, Liquibase will insert these rows. Your API is now instantly usable out of the box.

The Danger of Data Migrations

While seeding data is powerful, it requires discipline. Here is a practical engineering rule of thumb:

Do use Liquibase for:

• Static lookup tables (status codes, country lists, default departments).

• System configuration flags required for the application to boot.

Do NOT use Liquibase for:

• Generating thousands of fake users for testing.

• Migrating massive amounts of transactional data (for example, moving 5 million records from one table to another).

Large data migrations can lock up database tables for hours. If you lock a core table during a deployment, your application will experience a massive outage. Keep your changeSets focused on schema structure and essential baseline data. Use dedicated scripts or background jobs for heavy data manipulation.

Rollbacks

In a perfect world, code always works. In reality, you'll eventually deploy a database change that breaks a critical production query or corrupts data. When this happens, you need a way to hit the undo button.

Liquibase supports rollbacks, but you have to understand how it interprets them.

Automatic vs. Explicit Rollbacks

Many Liquibase commands are automatically reversible. For example, if you write a changeSet to <createTable> or <addColumn>, Liquibase implicitly knows that the opposite of adding a column is dropping a column. You don't have to tell it how to undo these actions.

But some operations are inherently destructive or ambiguous. If you use custom <sql> tags, or if you use <dropTable>, Liquibase has no idea how to put the data back. In these cases, you must provide explicit rollback instructions.

Let's simulate a scenario where we add a temporary access code column, but we want to ensure we know exactly how to remove it safely.

Create 06-temporary-access.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="9" author="ashutosh">
        <addColumn tableName="employees">
            <column name="temp_access_code" type="VARCHAR(10)"/>
        </addColumn>
        
        <rollback>
            <dropColumn tableName="employees" columnName="temp_access_code"/>
        </rollback>
    </changeSet>

</databaseChangeLog>

Add this to your master file and run the application. The column is added.

If you were deploying this via a CI/CD pipeline and the deployment failed, you could trigger a Liquibase Maven command to roll back by a specific number of steps (for example, mvn liquibase:rollback -Dliquibase.rollbackCount=1), or roll back to a specific tag we discussed earlier.

The Reality Check on Rollbacks

While it's important to know how rollbacks work, here's a practical reality from the trenches of backend engineering: Rollbacks are often discussed but rarely executed cleanly in production.

Dropping a column is mathematically easy. Recovering the customer data that was written to that column during the 15 minutes the bad code was live is incredibly difficult.

Because of this, modern engineering teams often prefer a "roll forward" strategy. If a migration causes an issue, instead of running a scary database rollback command, they quickly write a new changeSet that fixes the issue (for example, adding a missing index or relaxing a constraint) and deploy the application again.

It's highly recommended to design your database changes to be additive and non-destructive to avoid needing complex rollbacks in the first place.

Common Beginner Mistakes

Adopting database version control is a massive step forward for any engineering team, but it comes with a learning curve. When developers transition from writing loose SQL scripts to using Liquibase, they tend to fall into a few predictable traps.

Here are the most common beginner mistakes and exactly how to avoid them.

1. The "Mega" ChangeSet

When starting out, it's tempting to dump your entire initial schema into a single XML file under a single changeSet. You might put 15 createTable statements and 20 addForeignKeyConstraint statements into id="1".

This is a terrible idea for one simple reason: transaction failure.

If your database engine fails on table number 14 (perhaps due to a syntax error), what happens to the first 13 tables? Some database engines support transactional DDL (Data Definition Language), meaning it will roll back all 13 tables automatically. But many databases do not.

If it fails halfway through, your database is now in a fractured state. Liquibase didn't record id="1" as successful, so the next time you start the app, it will try to create all 15 tables again. It will immediately crash because table 1 already exists.

The Fix: Stick to the rule of "one logical operation per changeSet." If you're creating three tables, write three separate changeSets. If one fails, the successful ones are permanently recorded, and you only have to fix the broken one.

2. Manual Database Tweaking (The Phantom Menace)

This is the most dangerous habit to break. A developer spots a missing index in production. Instead of writing a Liquibase migration, going through code review, and deploying, they log directly into the production database and run CREATE INDEX manually to save time.

A week later, another developer writes a proper Liquibase migration to create that exact same index and deploys it. The application crashes on startup. Liquibase tries to execute the CREATE INDEX command, but the database throws an error saying the index already exists.

When you adopt Liquibase, you must accept a fundamental rule: Liquibase is the absolute source of truth for your schema. Human hands should never touch the database structure directly.

The Fix: If someone accidentally does this, you have two options to fix the deployment pipeline. You can manually drop the index from the database so Liquibase can recreate it properly, or you can use the <preConditions> tag in Liquibase to check if the index exists before trying to create it.

3. Ignoring the "From Scratch" Build

When you work on a project for months, your local database accumulates a lot of history. You write migrations assuming certain tables or test data already exist.

Then, a new developer joins the team. They pull the code, spin up an empty database, start Spring Boot, and the migrations crash halfway through.

This happens because the migrations rely on an assumed state (like expecting a specific row to exist before creating a foreign key) rather than a guaranteed state.

The Fix: You should regularly test your migrations against a completely blank database. If you're using Docker, tear down your database container and rebuild it. If you're using a file-based H2 database like we set up earlier, simply delete the ./data/employeedb.mv.db file from your project folder and restart Spring Boot. If the application can't boot successfully from a completely empty state, your migration history is broken.

4. Hardcoding Environment Details

Beginners sometimes hardcode environment-specific details directly into their XML files. For example, they might hardcode a specific schema name (schemaName="dev_schema") or grant permissions to a specific local user (GRANT ALL ON employees TO my_local_user).

When this code goes to staging, the staging database uses a different schema name, and the deployment fails.

The Fix: Keep your migrations abstract. Let Spring Boot handle the connection details via application.properties. If you absolutely must use dynamic values inside your Liquibase files, use property substitution. You can define variables in Liquibase and pass them in from Spring Boot during startup.

5. Messing Up Migration Ordering

Liquibase executes files in the exact order they're listed in your db.changelog-master.xml file.

If developer A creates the departments table in a branch, and developer B creates a foreign key linking to departments in another branch, whoever merges their code first dictates the order. If developer B's code gets included in the master file before developer A's code, Liquibase will try to create the foreign key before the target table exists.

The Fix: The master changelog is the ultimate chokepoint for database changes. During code reviews, always verify that the <include> statements are ordered chronologically and that dependencies make sense.

Liquibase vs Flyway vs Manual SQL Scripts

When you decide to implement database version control, you'll immediately face a choice. Liquibase isn't the only tool in the Java ecosystem. The three most common approaches to managing schema evolution are Liquibase, Flyway, and manual SQL scripts.

You should understand the practical tradeoffs of each so you can choose the right tool for your specific team and project.

1. Manual SQL Scripts (The Baseline)

This is the default approach for most beginners. You write a script.sql file and execute it directly against the database using a tool like DBeaver, pgAdmin, or DataGrip.

• Strengths: There is zero setup required. You have total control over the exact syntax, and every backend developer already knows how to write SQL.

• Weaknesses: There's absolutely no execution tracking. This approach practically guarantees schema drift across environments. Deployments become stressful because they rely on humans remembering to execute the right scripts in the exact right order.

• The Verdict: Manual scripts are perfectly fine for solo weekend projects or rapid prototyping where you don't care if the database gets destroyed. But they become a massive liability the moment a second developer joins the team or a staging environment is created.

2. Flyway (The SQL Purist)

Flyway is the most popular alternative to Liquibase. Instead of using XML or YAML abstractions, Flyway embraces raw SQL. You write pure SQL files with a strict naming convention (for example, V1__Create_employee_table.sql).

• Strengths: There's no new syntax to learn. If you know SQL, you already know how to use Flyway. It's incredibly fast to set up, highly opinionated, and integrates flawlessly with Spring Boot.

• Weaknesses: Because you write raw SQL, your migrations are intimately tied to your specific database dialect. If you write Flyway scripts for MySQL and later decide to migrate the project to PostgreSQL, you have to manually rewrite your migration history. Furthermore, seamless automated rollbacks are a paid feature in Flyway's commercial tier.

• The Verdict: Flyway is excellent for teams that are highly skilled in SQL, are permanently committed to a single database vendor, and prefer strict conventions over flexible configurations.

3. Liquibase (The Abstraction Layer)

As we have seen throughout this tutorial, Liquibase takes a different approach by abstracting database changes into XML, YAML, or JSON.

1. Strengths: It's truly database-agnostic. You define the logical structure, and Liquibase automatically translates that into the correct SQL dialect for H2, PostgreSQL, or Oracle. It supports powerful automatic rollbacks, preconditions, contexts, and deployment labels out of the box for free.

2. Weaknesses: It has a steeper learning curve than Flyway. The XML syntax is undeniably verbose and can feel heavy for very simple, single-table applications.

3. The Verdict: Liquibase shines in complex applications, multi-tenant systems, projects that support multiple database vendors, and enterprise environments that require fine-grained control over CI/CD deployment pipelines.

Liquibase Best Practices

Now that you understand the mechanics of Liquibase, you need to know how to use it in a professional environment. Writing a migration that works on your local machine is only half the battle. Writing a migration that your entire team can safely deploy to production requires discipline.

Here are the engineering best practices you should adopt when managing database changes.

1. One Logical Change Per ChangeSet (The Atomic Rule)

We discussed this in the common mistakes section, but it's important enough to repeat. Never bundle a table creation, an index creation, and a data insertion into a single changeSet.

If you're adding a salary column and an idx_employee_salary index, put them in two separate changeSets within the same file. This ensures that if the index creation fails, the column creation is still safely recorded, and you don't end up in a fractured database state.

2. Meaningful File Organization and Naming

Don't name your files update1.xml or new_changes.xml. Your file names should tell a story about how your database evolved.

Adopt a strict prefix system. In our project, we used 01-create-employees.xml and 02-add-employee-email.xml. In a real team, you might use Jira ticket numbers or release versions (for example, v1.2.0_ticket-482_add_email.xml). Whatever convention you choose, enforce it rigorously during code reviews.

3. Treat Database Changes Like Application Code

Database migrations belong in source control right next to your Java code. They should be reviewed with the exact same level of scrutiny.

When reviewing a pull request that includes a Liquibase file, engineers should ask:

• Does this column need an index?

• Is this a destructive change (like renaming a column) that will break the currently running application?

• Did the author include explicit rollback instructions for custom SQL?

4. Integrate Migrations into CI/CD

Human hands should never run database migrations against a production server. Your deployment pipeline should handle this automatically.

When you merge code into your main branch, your CI/CD pipeline (like GitHub Actions or GitLab CI) should build your Spring Boot application and deploy it. Because we bundled Liquibase into our Spring Boot startup sequence, the application will automatically migrate the production database before it starts accepting web traffic.

Here's what a safe, automated deployment pipeline looks like:

In a mature deployment pipeline, human hands never touch the production database. When you merge a pull request, the CI/CD pipeline builds the code and runs unit tests. It deploys the Spring Boot application to a staging environment, where Liquibase automatically acquires a lock and runs the migrations during startup. Once validated, that exact same artifact is promoted to production, triggering the identical automated migration process.

5. Never Fix Forward by Deleting History

If a migration fails in an upper environment (like staging or production), never log into the database to delete the DATABASECHANGELOG row so you can try again.

You must respect the immutability of the changelog. If you made a mistake, write a new changeSet that drops the broken table or fixes the data type, and push it through your Git workflow just like you would a Java bug fix.

Final Thoughts

Managing database schema changes doesn't have to be a source of anxiety.

By treating your database schema as code, you eliminate the chaos of manual SQL scripts. You prevent the dreaded "schema drift" where every developer's local machine behaves differently. Most importantly, you make your deployments predictable and boring (which is exactly what you want deployments to be).

In this tutorial, you built a practical Spring Boot application from scratch. You learned how Liquibase intercepts the application startup, locks the database, calculates cryptographic checksums, and safely applies incremental changes. You evolved a single table into a relational schema, added seed data, and learned how to avoid the most common traps beginners fall into.

The next time you start a Spring Boot project, don't reach for a manual SQL client. Add the Liquibase dependency, create your master changelog, and start version controlling your database from day one. Your future self (and your team) will thank you.

But a lot of teams still treat these numbers like something you glance at once. A screenshot in a deck. A one-off notebook cell. A quick check in a UI before a meeting.

That works until you need to answer basic questions that show up in real workflows:

What did TSLA's surface look like at 10:32? When did skew start steepening? Did the change come from the wings moving or the ATM shifting?

If you don't store the data as it arrives, you can't replay it, compare it, or audit it. You're stuck with whatever you happened to look at in the moment.

In this walkthrough, we'll build something small but practical: an internal database that continuously captures SpiderRock MLink's LiveImpliedQuote analytics for TSLA, stores each snapshot as queryable history, and also maintains a "latest view" table so you can pull the current surface state without scanning the full history.

The goal is not to build a trading system. It's to build a reliable internal dataset that you can monitor and query.

Note: SpiderRock MLink's LiveImpliedQuote analytics is a product offered for a fee, which includes exchange charges for the underlying market data used in its creation.

Table of Contents

• Prerequisites

• What Data We're Using

• Setup: Importing Packages

• Database Design

• Pulling LiveImpliedQuote

• Normalizing the Response Into Rows

• Writing To The Database

• Running a Short Polling Capture

• Analysis: Smile Reconstruction From the Database

  • Pick an Expiry with Good Coverage

  • Rebuild the Smile Across Snapshots

  • Zoom-In Around Spot

• Analysis: ATM IV and Skew Over Time

• Alert-Style Thresholds

• Wrapping Up

Prerequisites

Before running any of the code in this walkthrough, there are a few things you need to have in place.

On the API side, you need a SpiderRock MLink account with access to the LiveImpliedQuote feed. The examples use the REST interface, so no websocket setup is required, but you do need a valid API key. If you don't have one yet, you can reach out to SpiderRock directly to get access.

On the Python side, the environment is minimal. You need Python 3.10 or later for the tuple type hint syntax used in one of the function signatures. The external packages are requests, pandas, numpy, and matplotlib. Everything else – sqlite3, time, datetime – is part of the standard library. You can install the external dependencies with:

pip install requests pandas numpy matplotlib

No database setup is required beyond a writable local path. SQLite creates the file automatically on first run, so there's nothing to install or configure separately.

Finally, the walkthrough uses TSLA as the target symbol because it has a liquid and active options chain. If you want to swap in a different underlying, the only thing you need to change is the symbol variable in the config block.

What Data We're Using

This build is driven by one OptAnalytics message type from SpiderRock MLink: LiveImpliedQuote.

Each message represents an option contract and comes with the analytics you actually need for monitoring:

• the option identifier (symbol, expiry, strike, call or put)

• surface IV (sVol) and related surface fields

• Greeks (delta, gamma, theta, vega)

• context fields like underlying price (uPrc), time to expiry (years), and rate (rate)

• timestamps and calc source markers, which matter when you're turning a live feed into a database

We'll treat sVol as the main volatility field for the article and refer to it as surface IV. That keeps the workflow consistent when we rebuild smiles or compute skew proxies from stored history.

The demo uses TSLA because it has a rich and active options chain, which makes the database and queries more interesting even in a short capture window. The same pipeline works for any other underlying – the only thing you change is the symbol filter.

Setup: Importing Packages

Before touching the database or the API, we set up a small, repeatable environment. This section is intentionally minimal. We only import what we need for three things: making REST calls, storing data in SQLite, and doing basic analysis and plots.

import requests
import sqlite3
import pandas as pd
import numpy as np
import time
from datetime import datetime, timezone
import matplotlib.pyplot as plt
plt.style.use('ggplot')

• requests is used for calling MLink REST endpoints.

• sqlite3 gives us a lightweight database we can write to locally without extra setup.

• pandas and numpy are only for shaping and filtering the data once it comes back.

• time and datetime help us run a polling loop and timestamp each snapshot so the database becomes a real-time series.

Database Design

If the goal is to make live analytics queryable, the database design has to support two different needs.

First, you want an audit trail. Every snapshot should be preserved so you can reconstruct what the surface looked like at a specific time.

Second, you also want a fast way to answer "what does it look like right now" without scanning everything you've ever stored.

So we use two tables:

• implied_quote_history: Append-only. Every poll inserts a full snapshot.

• implied_quote_latest: One row per option contract. Each poll upserts into this table so it always reflects the most recent snapshot.

The core of both tables is a stable option identifier. In the feed, the option key is nested, so we normalize it into a single option_key string that includes symbol, expiry, strike, call or put, and venue fields. This becomes the primary key for the latest table and the main join key for queries.

#config
api_key = "YOUR SPIDERROCK API KEY"
mlink_url = "https://mlink-live.nms.saturn.spiderrockconnect.com/rest/json"

msg_type = "LiveImpliedQuote"

symbol = "TSLA"
poll_interval_s = 10
poll_duration_s = 120
limit = 2000

#create db connection
db_path = "/mnt/data/optanalytics_iv_greeks.db"

def get_conn(path: str = db_path):
    conn = sqlite3.connect(path)
    conn.execute("PRAGMA journal_mode=WAL;")
    conn.execute("PRAGMA synchronous=NORMAL;")
    return conn

#create db schema
def setup_db(path: str = db_path):
    conn = get_conn(path)
    cur = conn.cursor()

    cur.execute("""
    create table if not exists implied_quote_history (
        id integer primary key autoincrement,
        asof_ts text not null,

        option_key text not null,
        symbol text not null,
        expiry text not null,
        strike real not null,
        cp text not null,

        calc_source text,
        u_prc real,
        years real,
        rate real,

        s_vol real,
        atm_vol real,
        s_mark real,

        o_bid real,
        o_ask real,
        o_bid_iv real,
        o_ask_iv real,

        delta real,
        gamma real,
        theta real,
        vega real,

        src_ts text
    );
    """)

    cur.execute("""
    create index if not exists idx_hist_symbol_expiry_asof
    on implied_quote_history(symbol, expiry, asof_ts);
    """)

    cur.execute("""
    create index if not exists idx_hist_option_asof
    on implied_quote_history(option_key, asof_ts);
    """)

    cur.execute("""
    create table if not exists implied_quote_latest (
        option_key text primary key,

        last_asof_ts text not null,
        symbol text not null,
        expiry text not null,
        strike real not null,
        cp text not null,

        calc_source text,
        u_prc real,
        years real,
        rate real,

        s_vol real,
        atm_vol real,
        s_mark real,

        o_bid real,
        o_ask real,
        o_bid_iv real,
        o_ask_iv real,

        delta real,
        gamma real,
        theta real,
        vega real,

        src_ts text
    );
    """)

    cur.execute("""
    create index if not exists idx_latest_symbol_expiry
    on implied_quote_latest(symbol, expiry);
    """)

    conn.commit()
    conn.close()

setup_db()

This creates the SQLite database file and both tables. The history table is append-only and indexed for the two queries we'll run later: pulling snapshots by expiry and time, and pulling a specific option's timeline by option_key. The latest table is keyed by option_key, which lets us upsert and maintain a consistent "current view."

The columns we store are intentionally opinionated. We keep surface IV (s_vol), surface mark (s_mark), Greeks, and a few context fields. We also store timestamps so later we can reason about when a value was produced.

Pulling LiveImpliedQuote

Now we do the first live pull. The goal here is not to build a perfect filter. It's to confirm that we can retrieve a meaningful slice of TSLA option analytics and that the response structure is what we expect.

We request LiveImpliedQuote and filter by symbol using the where clause. The response is a list where most rows are actual LiveImpliedQuote messages, and one row at the end is a QueryResult summary.

def fetch_live_implied_quote(symbol: str, limit: int = 2000):
    where = f"okey.tk:eq:{symbol}"

    params = {
        "apiKey": api_key,
        "cmd": "getmsgs",
        "msgType": msg_type,
        "where": where,
        "limit": limit
    }

    r = requests.get(mlink_url, params=params)
    r.raise_for_status()
    return r.json()

raw = fetch_live_implied_quote(symbol, limit=limit)
print("raw messages:", len(raw))
print("first type:", raw[0].get("header", {}).get("mTyp") if raw else None)

This is a straight REST getmsgs call. We pass the API key, message type, and a simple symbol filter. The limit is important. It caps how many messages we get back in one poll, so for active underlyings, the returned set of strikes and expiries can vary between polls. That's fine for this tutorial, because the goal is to show the database pattern and the types of monitoring queries it enables.

This is the output you should see:

Normalizing the Response Into Rows

Right now, raw is a list of nested message objects. That format is fine for transport, but it's not something you can store or query directly. So now, we turn each LiveImpliedQuote message into one flat row with a consistent schema.

def make_option_key(okey: dict) -> str:
    return "|".join([
        str(okey.get("tk")),
        str(okey.get("dt")),
        str(okey.get("xx")),
        str(okey.get("cp")),
        str(okey.get("at")),
        str(okey.get("ts")),
    ])

def normalize_liq(raw: list, asof_ts: str, keep_calc_source: str = "Loop") -> pd.DataFrame:
    rows = []

    for row in raw:
        if row.get("header", {}).get("mTyp") != "LiveImpliedQuote":
            continue

        m = row.get("message", {})
        if keep_calc_source and m.get("calcSource") != keep_calc_source:
            continue

        pkey = m.get("pkey", {})
        okey = pkey.get("okey", {})
        if not okey:
            continue

        s_vol = m.get("sVol")
        if s_vol is None or s_vol == 0:
            continue

        o_bid = m.get("oBid", 0) or 0
        o_ask = m.get("oAsk", 0) or 0

        quote_ok = int(not (o_bid == 0 and o_ask == 0))

        rows.append({
            "asof_ts": asof_ts,
            "option_key": make_option_key(okey),

            "symbol": okey.get("tk"),
            "expiry": okey.get("dt"),
            "strike": okey.get("xx"),
            "cp": okey.get("cp"),

            "calc_source": m.get("calcSource"),
            "u_prc": m.get("uPrc"),
            "years": m.get("years"),
            "rate": m.get("rate"),

            "s_vol": s_vol,
            "atm_vol": m.get("atmVol"),
            "s_mark": m.get("sMark"),

            "o_bid": o_bid,
            "o_ask": o_ask,
            "o_bid_iv": m.get("oBidIv"),
            "o_ask_iv": m.get("oAskIv"),
            "quote_ok": quote_ok,

            "delta": m.get("de"),
            "gamma": m.get("ga"),
            "theta": m.get("th"),
            "vega": m.get("ve"),

            "src_ts": m.get("timestamp"),
        })

    df = pd.DataFrame(rows)
    if df.empty:
        return df

    df = (
        df.sort_values("src_ts")
          .drop_duplicates(subset=["option_key"], keep="last")
          .reset_index(drop=True)
    )
    return df

asof_ts = datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")
snapshot_df = normalize_liq(raw, asof_ts)

print("snapshot rows:", len(snapshot_df))
print("quote_ok distribution:", snapshot_df["quote_ok"].value_counts().to_dict() if not snapshot_df.empty else {})
snapshot_df.head()

There are three practical decisions baked into this normalization step:

• First, we build a stable option_key from the option identifier so we have a consistent primary key for the latest table.

• Second, we keep only calcSource="Loop". LiveImpliedQuote can include both Tick and Loop records. Loop records tend to be more consistent for snapshot-style analysis because the underlying reference price is stable across the surface.

• Third, we avoid aggressive filtering. In this dataset, the top-of-book bid and ask fields can be zero even when the analytics fields are populated. So instead of dropping those rows, we store a quote_ok flag and keep the record. That keeps the pipeline usable while still making it obvious later which rows had live quotes.

This is the output:

At this point, one row represents one option contract snapshot. The fact that quote_ok is 0 across the board simply means bid and ask are not populated in this slice, even though surface IV, Greeks, and other analytics fields are present. That's still useful for building a monitoring database, because the core idea here is tracking the evolution of analytics over time, not reconstructing executable markets.

Writing to the Database

Now that we have a clean snapshot DataFrame, the job is to persist it in two places.

History table: Append everything. This is the audit log. Latest table: Upsert by option_key. This is the fast "current view."

This separation is what makes the database useful. History lets you reconstruct any past snapshot. Latest lets you answer "what does the surface look like right now" without scanning time series.

def safe_add_column(table: str, col: str, col_type: str, path: str = db_path):
    conn = get_conn(path)
    cur = conn.cursor()
    existing = [r[1] for r in cur.execute(f"PRAGMA table_info({table});").fetchall()]
    if col not in existing:
        cur.execute(f"ALTER TABLE {table} ADD COLUMN {col} {col_type};")
    conn.commit()
    conn.close()

safe_add_column("implied_quote_history", "quote_ok", "INTEGER")
safe_add_column("implied_quote_latest", "quote_ok", "INTEGER")

def write_snapshot_to_db(df: pd.DataFrame, path: str = db_path) -> tuple[int, int]:
    if df.empty:
        return 0, 0

    conn = get_conn(path)
    cur = conn.cursor()

    cols = [
        "asof_ts",
        "option_key","symbol","expiry","strike","cp",
        "calc_source","u_prc","years","rate",
        "s_vol","atm_vol","s_mark",
        "o_bid","o_ask","o_bid_iv","o_ask_iv",
        "delta","gamma","theta","vega",
        "quote_ok","src_ts"
    ]

    for c in cols:
        if c not in df.columns:
            df[c] = None

    insert_df = df[cols].copy()

    cur.executemany(
        """
        insert into implied_quote_history (
            asof_ts,
            option_key, symbol, expiry, strike, cp,
            calc_source, u_prc, years, rate,
            s_vol, atm_vol, s_mark,
            o_bid, o_ask, o_bid_iv, o_ask_iv,
            delta, gamma, theta, vega,
            quote_ok, src_ts
        ) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """,
        insert_df.itertuples(index=False, name=None)
    )
    history_inserted = cur.rowcount

    cur.executemany(
        """
        insert into implied_quote_latest (
            option_key,
            last_asof_ts, symbol, expiry, strike, cp,
            calc_source, u_prc, years, rate,
            s_vol, atm_vol, s_mark,
            o_bid, o_ask, o_bid_iv, o_ask_iv,
            delta, gamma, theta, vega,
            quote_ok, src_ts
        ) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        on conflict(option_key) do update set
            last_asof_ts=excluded.last_asof_ts,
            symbol=excluded.symbol,
            expiry=excluded.expiry,
            strike=excluded.strike,
            cp=excluded.cp,
            calc_source=excluded.calc_source,
            u_prc=excluded.u_prc,
            years=excluded.years,
            rate=excluded.rate,
            s_vol=excluded.s_vol,
            atm_vol=excluded.atm_vol,
            s_mark=excluded.s_mark,
            o_bid=excluded.o_bid,
            o_ask=excluded.o_ask,
            o_bid_iv=excluded.o_bid_iv,
            o_ask_iv=excluded.o_ask_iv,
            delta=excluded.delta,
            gamma=excluded.gamma,
            theta=excluded.theta,
            vega=excluded.vega,
            quote_ok=excluded.quote_ok,
            src_ts=excluded.src_ts
        """,
        insert_df[[
            "option_key","asof_ts","symbol","expiry","strike","cp",
            "calc_source","u_prc","years","rate",
            "s_vol","atm_vol","s_mark",
            "o_bid","o_ask","o_bid_iv","o_ask_iv",
            "delta","gamma","theta","vega",
            "quote_ok","src_ts"
        ]].itertuples(index=False, name=None)
    )
    latest_upserted = cur.rowcount

    conn.commit()
    conn.close()
    return history_inserted, latest_upserted

hist_n, latest_n = write_snapshot_to_db(snapshot_df)
print("history inserted:", hist_n)
print("latest upserted:", latest_n)

We batch write using executemany so inserts are fast even with thousands of option rows. The history insert is straightforward. The latest write uses a SQLite upsert keyed on option_key, which means if the contract already exists in the latest table, its fields are overwritten with the newest snapshot.

You should see:

After the first write, both tables have the same number of rows. That's expected, because there is only one snapshot in history so far. Once we start polling multiple snapshots, the history table will grow every cycle, while the latest table will stay roughly flat and continue updating in place.

Running a Short Polling Capture

At this point, the pipeline works end-to-end for a single snapshot. The whole point of the database, though, is to turn live analytics into a time series. So we run a short capture window and store multiple snapshots back-to-back.

This isn't meant to be a production scheduler. It's just a simple loop that runs for a couple of minutes, polls every few seconds, timestamps the snapshot, and writes it to both tables.

def poll_and_write(symbol: str, duration_s: int = poll_duration_s, interval_s: int = poll_interval_s):
    start = time.time()
    polls = 0
    total_hist = 0

    while time.time() - start < duration_s:
        asof_ts = datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")

        raw = fetch_live_implied_quote(symbol, limit=limit)
        df = normalize_liq(raw, asof_ts)

        hist_n, latest_n = write_snapshot_to_db(df)
        polls += 1
        total_hist += hist_n

        print(f"[{polls}] {asof_ts} snapshot_rows={len(df)} history+={hist_n} latest_upsert={latest_n}")
        time.sleep(interval_s)

    print(f"done. polls={polls}, total_history_added={total_hist}")

poll_and_write(symbol, duration_s=120, interval_s=10)

Each loop iteration represents one snapshot. We generate a UTC timestamp (asof_ts), pull the latest batch from LiveImpliedQuote, normalize it into rows, then write it into the database. The history table accumulates every snapshot. The latest table overwrites by option_key, so it always represents the most recent view.

One practical detail is worth calling out. The API call is capped by limit, so you're not guaranteed to receive an identical set of strikes and expiries every poll. That's why snapshot_rows can vary between iterations.

In production, you usually stabilize the slice by pinning specific expiries and a strike band or by interpolating IV to fixed moneyness points. For this tutorial, we're keeping ingestion simple and focusing on the database pattern and the monitoring queries it enables.

You should see per-poll telemetry like this:

[1] 2026-04-14T18:09:29Z snapshot_rows=1454 history+=1454 latest_upsert=1454
...
done. polls=9, total_history_added=12806

This confirms the database is building a time series. Over nine polls, you stored 12,806 option rows in history. The latest table is updated each time, but it doesn't grow in the same way as history because it overwrites per contract key.

From the next section, we'll stop writing and start querying.

Analysis: Smile Reconstruction From the Database

Once the data is in implied_quote_history, the workflow flips. We stop thinking in terms of "API responses" and start thinking in terms of "queries." This section does two things. First, it picks an expiry that has enough rows to be representative. Then it reconstructs the call-side volatility smile for that expiry across a few timestamps.

Pick an Expiry with Good Coverage

If you pick an expiry that only appears sporadically in the captured snapshots, the smile plot will be misleading. So we start by looking at which expiries have the most rows in the history table.

conn = get_conn()

expiry_counts = pd.read_sql_query(
    """
    select expiry, count(*) as n
    from implied_quote_history
    where symbol = ?
    group by expiry
    order by n desc
    limit 10
    """,
    conn,
    params=(symbol,)
)

conn.close()
expiry_counts

This query scans only the history table, filters to TSLA, and counts how many option rows exist per expiry across the capture window. We keep the top 10 and pick the first one as the expiry we'll reconstruct.

The expiry date 2026-11-20 has the highest count.

Here, the count doesn't mean this expiry is "best" in any trading sense. It just means it showed up most consistently in the captured data. That makes it a practical choice for a clean smile comparison.

Rebuild the Smile Across Snapshots

Now we query the stored history for one expiry, keep only calls, and plot surface IV (s_vol) against strike for multiple snapshot timestamps.

chosen_expiry = "2026-11-20" 

conn = get_conn()
smile = pd.read_sql_query(
    """
    select asof_ts, strike, cp, s_vol, u_prc
    from implied_quote_history
    where symbol = ? and expiry = ?
    """,
    conn,
    params=(symbol, chosen_expiry)
)
conn.close()

smile_calls = smile[smile["cp"] == "Call"].copy()

ts_list = sorted(smile_calls["asof_ts"].unique())
pick = [ts_list[0], ts_list[len(ts_list)//2], ts_list[-1]]

plt.figure(figsize=(9,5))
for ts in pick:
    g = smile_calls[smile_calls["asof_ts"] == ts].sort_values("strike")
    plt.plot(g["strike"], g["s_vol"], label=ts)

plt.title(f"{symbol} Vol Smile (Calls) | Expiry {chosen_expiry} | 3 snapshots")
plt.xlabel("Strike")
plt.ylabel("Implied Vol (s_vol)")
plt.grid(True)
plt.legend()
plt.show()

We pull all rows for the chosen expiry from history, then filter to calls so we don't mix put and call shapes. To keep the plot readable, we only plot three snapshots. First, middle, and last.

Over a short capture window, the smiles often overlap heavily. That doesn't mean the system isn't working. It usually means the surface didn't move much in those two minutes. The important part is that we can reconstruct and compare it purely from stored history.

Zoom-In Around Spot

The full-range plot is useful for shape, but it can hide small shifts near the region people actually care about. So we zoom to a band around the underlying price.

s0 = float(smile_calls["u_prc"].dropna().median())
low, high = s0 * 0.6, s0 * 1.4

for ts in pick:
    g = smile_calls[smile_calls["asof_ts"] == ts].sort_values("strike")
    g = g[(g["strike"] >= low) & (g["strike"] <= high)]
    plt.plot(g["strike"], g["s_vol"], label=ts)

plt.title(f"{symbol} Vol Smile (Calls) | Expiry {chosen_expiry} | zoomed")
plt.xlabel("Strike")
plt.ylabel("Implied Vol (s_vol)")
plt.grid(True)
plt.legend(fontsize=8)
plt.show()

We take a robust spot proxy from the stored u_prc values and then keep strikes within a range around it. The goal is not precision. It's to make the chart readable and show whether the near-ATM region is drifting.

Here, even small changes become visible. This is also why storing history matters. If you only looked at one snapshot in isolation, these shifts would be easy to miss or dismiss.

Analysis: ATM IV and Skew Over Time

A full smile plot is useful, but it's not always the fastest way to monitor a surface. In practice, teams usually track a few summary numbers per expiry so they can spot changes quickly, then drill down only when something looks off.

Here we reduce each stored snapshot into two metrics for a single expiry.

• ATM IV: Surface IV at the strike closest to spot.

• Skew proxy: Surface IV at 0.9 times spot minus surface IV at 1.1 times spot, using the closest available strikes.

chosen_expiry = "2026-11-20"

conn = get_conn()
df = pd.read_sql_query(
    """
    select asof_ts, strike, s_vol, u_prc
    from implied_quote_history
    where symbol = ? and expiry = ? and cp = 'Call'
    """,
    conn,
    params=(symbol, chosen_expiry)
)
conn.close()

df["strike"] = df["strike"].astype(float)
df["s_vol"] = df["s_vol"].astype(float)

def closest_iv(grp: pd.DataFrame, target_strike: float):
    g = grp.iloc[(grp["strike"] - target_strike).abs().argsort()[:1]]
    return float(g["s_vol"].iloc[0]), float(g["strike"].iloc[0])

rows = []
for ts, grp in df.groupby("asof_ts"):
    spot = float(grp["u_prc"].dropna().median())
    atm_target = spot
    down_target = spot * 0.9
    up_target = spot * 1.1

    atm_iv, atm_k = closest_iv(grp, atm_target)
    down_iv, down_k = closest_iv(grp, down_target)
    up_iv, up_k = closest_iv(grp, up_target)

    rows.append({
        "asof_ts": ts,
        "spot": spot,
        "atm_strike": atm_k,
        "atm_iv": atm_iv,
        "k90": down_k,
        "iv_90": down_iv,
        "k110": up_k,
        "iv_110": up_iv,
        "skew_90_110": down_iv - up_iv
    })

metrics = pd.DataFrame(rows).sort_values("asof_ts").reset_index(drop=True)
metrics

We query the history table for one expiry and keep only calls, then group by snapshot timestamp. For each snapshot, we use the median u_prc as a spot proxy and pick the closest available strike to spot. That gives ATM IV. We repeat the same approach for 0.9 times spot and 1.1 times spot and compute a skew proxy as the difference.

The table also stores the actual strikes used (atm_strike, k90, k110). Options strikes are discrete, so the nearest strike can change between snapshots. Keeping the chosen strikes visible makes the metric explainable when it moves.

The output is a table with one row per snapshot timestamp and the computed metrics.

Now that we have a clean time series table, we can visualize the two metrics. First, ATM IV. Then, the skew proxy.

plt.plot(metrics["asof_ts"], metrics["atm_iv"])
plt.title(f"{symbol} ATM IV over time | Expiry {chosen_expiry}")
plt.xticks(rotation=30, ha="right")
plt.ylabel("ATM IV (s_vol)")
plt.grid(True)
plt.show()

plt.plot(metrics["asof_ts"], metrics["skew_90_110"])
plt.title(f"{symbol} Skew proxy (IV@0.9S - IV@1.1S) | Expiry {chosen_expiry}")
plt.xticks(rotation=30, ha="right")
plt.ylabel("Skew proxy")
plt.grid(True)
plt.show()

Here is the first chart, ATM IV over time.

ATM IV tends to move slowly over short windows unless there is a sharp repricing event. In this run, it stays fairly stable, which is a realistic outcome for a short capture. The value here is that the database turns "fairly stable" into something you can quantify and compare later, rather than a vague impression.

Here is the second chart, Skew proxy over time.

The skew proxy is more sensitive because it's based on wing points. If it changes, it usually means the downside is being repriced differently from the upside for that expiry. One nuance is that the nearest available strike can change between snapshots, which can create step-like moves even when the surface isn't moving dramatically. That's why we keep k90 and k110 in the metrics table. It keeps the skew plot explainable.

Alert-Style Thresholds

Once you have a metrics table per snapshot, adding a monitoring layer is straightforward. The idea isn't to generate trades. It's to flag when the surface moves enough that someone should look closer.

Here we do two checks:

• ATM IV change alert: Flag if ATM IV changes more than a small threshold between snapshots.

• Skew change alert: Flag if the skew proxy changes more than a threshold between snapshots.

alerts = metrics.copy()

alerts["atm_iv_change"] = alerts["atm_iv"].diff()
alerts["skew_change"] = alerts["skew_90_110"].diff()

atm_thresh = 0.002    
skew_thresh = 0.003   

alerts["atm_alert"] = alerts["atm_iv_change"].abs() >= atm_thresh
alerts["skew_alert"] = alerts["skew_change"].abs() >= skew_thresh

alerts[[
    "asof_ts",
    "atm_iv", "atm_iv_change", "atm_alert",
    "skew_90_110", "skew_change", "skew_alert",
    "atm_strike", "k90", "k110"
]]

We take the per-snapshot metrics table and compute first differences. Then we compare those changes to thresholds and store boolean flags. The output table keeps both the metrics and the strikes used for the calculations, so any alert is explainable rather than a black box.

In this run, the ATM IV alerts are all false, while the skew alert triggers once.

The skew alert fires because the skew proxy jumps by more than the threshold between two snapshots. This is explainable. If you see the table, you can see the strikes used for the proxy changed around the same time (k90 shifts from 340 to 315). Because strikes are discrete, nearest-strike metrics can step even when the surface is not moving dramatically.

To make this easier to read, we also plot the two series and mark alert points.

plt.plot(alerts["asof_ts"], alerts["atm_iv"])
for i, r in alerts[alerts["atm_alert"]].iterrows():
    plt.scatter(r["asof_ts"], r["atm_iv"],  s=30, edgecolors="r", alpha=0.6, linewidth=2)
plt.title(f"{symbol} ATM IV with alerts | Expiry {chosen_expiry}")
plt.xticks(rotation=30, ha="right")
plt.grid(True)
plt.show()

plt.plot(alerts["asof_ts"], alerts["skew_90_110"])
for i, r in alerts[alerts["skew_alert"]].iterrows():
    plt.scatter(r["asof_ts"], r["skew_90_110"], s=30, edgecolors="r", alpha=0.6, linewidth=2)
plt.title(f"{symbol} Skew proxy with alerts | Expiry {chosen_expiry}")
plt.xticks(rotation=30, ha="right")
plt.grid(True)
plt.show()

Both plots use the same pattern. Plot the metric as a line, then overlay a marker on any timestamp where the corresponding alert flag is true. This makes it obvious when something crossed the threshold.

This chart represents skew proxy with alerts.

This chart shows one alert marker, which matches what we saw in the table.

The ATM IV plot isn't featured since there are no alert points.

Wrapping Up

In this walkthrough, we used SpiderRock MLink's LiveImpliedQuote feed for TSLA and turned it into a small internal database you can query. We stored every snapshot in an append-only history table, maintained a latest view keyed by a stable option identifier, then used that stored data to rebuild a smile, track ATM surface IV and a simple skew proxy, and add a basic alert rule on top.

This fits well in B2B workflows because it turns live analytics into something operational: a dataset you can audit, replay, and monitor. The same pattern works whether you're building an internal dashboard, running routine surface checks for a desk, or doing a quick post-event review without relying on screenshots and one-off notebook runs.

If you want to extend it, the most practical next steps are longer capture windows, tracking multiple symbols, and moving from SQLite to Postgres once the data volume grows. If metric stability becomes important, you can also standardize the slice you track per poll or interpolate IV to fixed moneyness points so skew measures don't step when nearest strikes change.

With that being said, you've reached the end of the article. Hope you learned something new and useful.

So I built a benchmark suite that pits vanilla PostgreSQL against a feature-optimized PostgreSQL instance — measuring caching, message queues, full-text search, and pub/sub under controlled conditions.

In this article, you'll learn how to use PostgreSQL's built-in features for caching, job queues, full-text search, and pub/sub. You'll see actual benchmark results (latency percentiles, throughput, and error rates) comparing naive PostgreSQL patterns against optimized ones, and understand where PostgreSQL's limits are so you can decide whether you really need that extra service in your stack.

Table of Contents

• Prerequisites

• The Setup

• Benchmark 1: Caching with UNLOGGED Tables

• Benchmark 2: Job Queues with SKIP LOCKED

• Benchmark 3: Full-Text Search with tsvector

• Benchmark 4: Pub/Sub with LISTEN/NOTIFY

• The Combined Workload: The Honest Test

• What I Learned

Prerequisites

To follow along or reproduce the benchmarks, you'll need:

• Docker and Docker Compose

• Node.js 20+ (for the Express TypeScript API layer)

• k6 for load testing

• Basic familiarity with SQL and PostgreSQL

The full benchmark project is open source on GitHub — you can clone it and run every test yourself.

The Setup

The benchmark uses two identical PostgreSQL 17 instances running in Docker containers, each with fixed resource constraints (2 CPUs, 2 GB RAM). Both share the same Express TypeScript API layer — the only difference is which PostgreSQL features are enabled.

┌─────────┐     ┌──────────────────┐     ┌─────────────────┐
│   k6    │────>│  Express API     │────>│  PG Baseline    │
│  (load  │     │  (TypeScript)    │     │  (vanilla PG17) │
│  test)  │────>│  Port 3001/3002  │────>│  PG Modded      │
└─────────┘     └──────────────────┘     │  (features on)  │
                                         └─────────────────┘

The baseline instance uses naïve approaches (regular tables, ILIKE search, polling). The modded instance uses PostgreSQL's built-in features (UNLOGGED tables, tsvector with GIN indexes, LISTEN/NOTIFY, partial indexes). Same hardware, same API code, same data. Only the database features differ.

Both instances share this tuned postgresql.conf:

# Memory allocation
shared_buffers = 512MB           # 25% of available RAM
effective_cache_size = 1536MB    # 75% of RAM — helps the query planner
work_mem = 16MB                  # per-sort/hash operation memory

# SSD-optimized planner settings
random_page_cost = 1.1           # default 4.0 assumes spinning disks
effective_io_concurrency = 200   # allow parallel I/O on SSDs

These settings matter. The defaults assume spinning disks from the early 2000s. Setting random_page_cost = 1.1 tells the query planner that random reads are nearly as fast as sequential reads on SSDs, which encourages index usage over sequential scans.

Benchmark 1: Caching with UNLOGGED Tables

The idea: Use an UNLOGGED table as an in-database cache. UNLOGGED tables skip PostgreSQL's Write-Ahead Log (WAL) — the mechanism that guarantees durability. Since cache data is ephemeral by nature, losing it on a crash is acceptable, and skipping WAL removes the biggest write bottleneck.

-- Modded: UNLOGGED table for cache entries
CREATE UNLOGGED TABLE cache_entries (
    key TEXT PRIMARY KEY,
    value JSONB NOT NULL,
    expires_at TIMESTAMPTZ
);

-- Baseline: same schema, but a regular (logged) table
CREATE TABLE cache_entries (
    key TEXT PRIMARY KEY,
    value JSONB NOT NULL,
    expires_at TIMESTAMPTZ
);

Results (200 Virtual Users)

A consistent 13% improvement across all percentiles. Not dramatic, but free — you change one keyword in your CREATE TABLE statement.

Under Stress (1,000 Virtual Users, No Sleep)

The relative improvement stays locked at 12-13% regardless of load level. The UNLOGGED advantage is a per-write optimization — it saves the same amount of I/O whether you are doing 100 or 10,000 writes per second. The modded instance served 37,000 more requests in the same time window.

The Verdict

UNLOGGED tables won't match Redis for sub-millisecond hot-path caching (real-time bidding, gaming leaderboards). But for web applications where the difference between 2ms and 5ms is invisible to users, they eliminate an entire infrastructure dependency for zero additional complexity.

You do give up Redis data structures (sorted sets, HyperLogLog, streams). If you need those, a dedicated cache is still the right call.

Benchmark 2: Job Queues with SKIP LOCKED

The idea: Use PostgreSQL as a job queue with SELECT ... FOR UPDATE SKIP LOCKED. Multiple workers poll the same table, and SKIP LOCKED ensures each worker gets a different row — no duplicates, no contention.

-- Queue table with a partial index on pending jobs only
CREATE TABLE job_queue (
    id SERIAL PRIMARY KEY,
    payload JSONB NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Partial index: only indexes pending jobs
-- As jobs complete, they leave the index — it stays small forever
CREATE INDEX idx_pending_jobs ON job_queue (created_at)
    WHERE status = 'pending';

The dequeue pattern:

-- Atomic dequeue: select + update in one statement
UPDATE job_queue SET status = 'processing'
WHERE id = (
    SELECT id FROM job_queue
    WHERE status = 'pending'
    ORDER BY created_at
    LIMIT 1
    FOR UPDATE SKIP LOCKED  -- skip rows locked by other workers
) RETURNING *;

How SKIP LOCKED works: Worker A locks row 1. Worker B tries row 1, sees the lock, skips it, and takes row 2 instead. No blocking, no duplicates. If a worker crashes, the transaction rolls back and the row becomes available again.

Results (100 Producers + 50 Consumers)

They're virtually identical. The partial index doesn't show its value in a 60-second benchmark because the table doesn't accumulate enough completed rows for the index size difference to matter. In a production system with millions of completed jobs, the partial index keeps the index at kilobytes while a full index grows to gigabytes.

The Verdict

SKIP LOCKED is production-ready for job queues. Libraries like pg-boss (Node.js) and river (Go) build on this exact pattern.

You do give up exchange/routing patterns (fan-out, topic-based routing) and consumer groups with message replay. If you need those, a dedicated message broker is still the right tool. For simple "process this job once" workloads, PostgreSQL handles it.

Benchmark 3: Full-Text Search with tsvector

The idea: Use PostgreSQL's built-in full-text search instead of a separate search service. A tsvector column stores pre-processed search tokens, and a GIN (Generalized Inverted Index) enables fast lookups using the same inverted index concept that powers Elasticsearch.

-- Search-optimized article table
CREATE TABLE articles (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    body TEXT NOT NULL,
    search_vector tsvector  -- pre-computed search tokens
);

-- GIN index for full-text search
CREATE INDEX idx_search ON articles USING GIN (search_vector);

-- Auto-update search_vector on insert/update
CREATE OR REPLACE FUNCTION update_search_vector() RETURNS trigger AS $$
BEGIN
    NEW.search_vector := to_tsvector('english',
        COALESCE(NEW.title, '') || ' ' || COALESCE(NEW.body, ''));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_search
    BEFORE INSERT OR UPDATE ON articles
    FOR EACH ROW EXECUTE FUNCTION update_search_vector();

The baseline uses ILIKE with a leading wildcard — the approach most developers reach for first:

-- Baseline: sequential scan on every query
SELECT * FROM articles
WHERE title ILIKE '%postgresql%' OR body ILIKE '%postgresql%';

-- Modded: GIN index lookup with relevance ranking
SELECT id, title,
    ts_rank(search_vector, plainto_tsquery('english', 'postgresql')) AS rank
FROM articles
WHERE search_vector @@ plainto_tsquery('english', 'postgresql')
ORDER BY rank DESC LIMIT 20;

Results (500 Virtual Users)

This is the standout result. The baseline's p95 of 101ms versus the modded's 10ms is a 10x improvement.

Why the baseline's p50 (1.96ms) is slightly better than the modded's (2.76ms): simple ILIKE queries on small result sets can be fast when the data fits in shared_buffers. But as load increases and the buffer cache is contested, sequential scans degrade dramatically. The GIN index stays stable.

Under Stress (500 Virtual Users, No Sleep)

ILIKE collapses to 1-second p95 latencies. Each query forces a sequential scan of all 10,000 articles, blocking shared buffers and starving concurrent queries. The tsvector approach serves 2.6x more requests in the same time window because the GIN index lookup is O(log n) regardless of concurrency.

The Verdict

This is the strongest argument in the entire benchmark. The fix requires zero extensions — to_tsvector(), plainto_tsquery(), and CREATE INDEX USING GIN are all built into core PostgreSQL. If you're doing WHERE column ILIKE '%term%' on any table with more than a few thousand rows, you're leaving massive performance on the table.

You do give up distributed search across shards, complex analyzers for CJK languages, and aggregation/faceted search pipelines. For a product search bar, blog search, or internal tool — PostgreSQL is enough.

Benchmark 4: Pub/Sub with LISTEN/NOTIFY

The idea: Use PostgreSQL's native LISTEN/NOTIFY for pub/sub messaging, triggered automatically on INSERT via a database trigger.

-- Trigger that fires pg_notify on every new message
CREATE OR REPLACE FUNCTION notify_message() RETURNS trigger AS $$
BEGIN
    PERFORM pg_notify(NEW.channel, NEW.payload::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_notify
    AFTER INSERT ON messages
    FOR EACH ROW EXECUTE FUNCTION notify_message();

Results (200 Virtual Users)

Here we have a 20% improvement at p95. The trigger-based approach does more work per INSERT (INSERT + NOTIFY), but the reduced round trips and better connection reuse patterns offset the overhead.

The Verdict

LISTEN/NOTIFY works for real-time features where you would otherwise reach for Redis pub/sub. The main limitation is payload size (8,000 bytes maximum) and the requirement for dedicated connections (incompatible with PgBouncer in transaction mode).

The Combined Workload: The Honest Test

Individual benchmarks are flattering. The real question: can one PostgreSQL instance handle caching, queues, search, and pub/sub simultaneously without degrading?

Results (All Four Workloads Running Together)

Under combined load, the baseline marginally outperforms the modded setup. The modded PostgreSQL does more work per operation — maintaining GIN indexes, firing triggers, running pg_cron in the background. When all these features are active simultaneously, the overhead is measurable: about 15% higher p95 latency.

But both setups stay comfortably under 10ms at p95. For most web applications, that's more than good enough.

What I Learned

After running all these benchmarks, here's what I would tell a team evaluating whether to "just use Postgres":

1. Do it for full-text search: Switching from ILIKE to tsvector with a GIN index is a 10x improvement that requires zero extensions. This is the single highest-ROI change in the entire PostgreSQL ecosystem, and most developers don't know it exists.

2. Do it for job queues: SKIP LOCKED is production-ready and eliminates RabbitMQ for simple "process this job" workloads. Use a library like pg-boss or river rather than rolling your own.

3. Consider it for caching: UNLOGGED tables give a steady 13% improvement over regular tables. If sub-millisecond latency is not a hard requirement (and for most web apps, it is not), you can drop Redis entirely.

4. Be honest about the overhead: Running all four roles simultaneously adds about 15% latency compared to running any single role. Whether that matters depends on your latency budget.

5. Know where to stop: PostgreSQL won't match Redis for sub-millisecond caching, Kafka for millions of messages per second, or Elasticsearch for distributed multi-node search with complex analyzers. The line is at extreme throughput or extreme specialization.

The honest conclusion is not "PostgreSQL does everything." It is: for most applications, a single well-configured PostgreSQL instance handles 80% of what you would otherwise need three to five additional services for. That is less infrastructure to deploy, monitor, and maintain — and fewer things to break at 3 AM.

Enterprise-scale applications processing millions of messages per second, serving sub-millisecond cache hits to millions of concurrent users, or running distributed search across terabytes of documents will still need specialized tools. Those tools exist for a reason, and at that scale the operational cost of running them is justified by the performance you get back.

But most of us aren't building at that scale — and may never need to. Starting with PostgreSQL for these roles means you ship faster with fewer moving parts. If and when you outgrow what PostgreSQL can handle, your benchmarks will tell you exactly which role needs to be extracted into a dedicated service. That is a much better position than starting with five services on day one because you assumed you would need them.

The benchmark project is open source if you want to reproduce these results or adapt the tests for your own workload.

You can find more of my writing at site.aaronhsyong.com.

The fix, more often than not, is an index.

A database index is a data structure that helps the database find rows faster without scanning the entire table. It works a lot like the index at the back of a textbook: instead of reading every page to find a topic, you look it up in the index, get the page number, and go straight there.

In this tutorial, you'll learn how indexes work under the hood, how to create and use them effectively in PostgreSQL, and how to avoid the common mistakes that make indexes useless or even harmful.

Table of Contents

• Prerequisites

• Why Do You Need Indexes?

• How Indexes Work Under the Hood

• How to Create Your First Index

• How to Use EXPLAIN ANALYZE to Measure Performance

• Types of Indexes in PostgreSQL

• How to Create a Composite Index

• How to Create a Partial Index

• How to Create an Expression Index

• How to Create a Unique Index

• How to Manage Indexes

• When Indexes Hurt Instead of Help

• Common Mistakes That Prevent Index Usage

• Best Practices for Indexing

• Conclusion

Prerequisites

To follow along with the examples, you'll need:

• Basic knowledge of SQL (SELECT, INSERT, UPDATE, DELETE, WHERE, JOIN)

• A running PostgreSQL instance (version 12 or later)

• A SQL client like psql, pgAdmin, or DBeaver

If you don't have PostgreSQL installed locally, you can use a free cloud-hosted instance from services like Neon or Supabase.

Why Do You Need Indexes?

When you run a query like SELECT * FROM users WHERE email = 'jane@example.com', the database needs to find the matching row. Without an index, PostgreSQL performs a sequential scan — it reads every single row in the table and checks whether the email column matches.

For a table with 100 rows, this is fine. For a table with 10 million rows, it's painfully slow.

An index solves this by creating a separate, sorted data structure that maps column values to their row locations. Instead of scanning 10 million rows, PostgreSQL can look up the value in the index and jump directly to the matching row. This can reduce query time from seconds to milliseconds.

But indexes aren't free. They come with trade-offs you need to understand before adding them everywhere. You'll learn about those trade-offs throughout this tutorial.

How Indexes Work Under the Hood

PostgreSQL's default index type is the B-tree (balanced tree). Understanding how a B-tree works will help you make smarter decisions about when and how to index.

A B-tree organizes data into a sorted, hierarchical structure with three levels:

1. Root node — the top of the tree. It holds a few values that divide the data into broad ranges.

2. Internal nodes — each one further narrows down the range.

3. Leaf nodes — the bottom level. These hold the actual indexed values along with pointers to the corresponding rows in the table.

When PostgreSQL uses a B-tree index to find a value, it starts at the root and follows the path that matches the target value, moving through internal nodes until it reaches the correct leaf node. This path is called a tree traversal, and it typically requires only 3–4 steps even for tables with millions of rows.

Think of it like a phone book. You don't start at page one and read every name. You open to roughly the right section (root), narrow it down to the right page (internal nodes), and scan the entries on that page (leaf node).

This sorted structure is also why B-tree indexes work well for range queries like WHERE price > 50 AND price < 100. The database finds the starting point in the tree and then scans forward through the leaf nodes, which are already in order.

How to Create Your First Index

Let's build a practical example. You'll create a table, load it with data, and see the difference an index makes.

Step 1 – Create the Table and Insert Sample Data

CREATE TABLE customers (
    id SERIAL PRIMARY KEY,
    first_name VARCHAR(50) NOT NULL,
    last_name VARCHAR(50) NOT NULL,
    email VARCHAR(100) NOT NULL,
    city VARCHAR(50),
    created_at TIMESTAMP DEFAULT NOW()
);

Now insert a large number of rows so the performance difference is visible. This generates 500,000 rows of sample data:

INSERT INTO customers (first_name, last_name, email, city)
SELECT
    'User' || gs,
    'Last' || gs,
    'user' || gs || '@example.com',
    (ARRAY['Lagos', 'London', 'New York', 'Berlin', 'Tokyo'])[1 + (gs % 5)]
FROM generate_series(1, 500000) AS gs;

Step 2 – Query Without an Index

EXPLAIN ANALYZE
SELECT * FROM customers WHERE email = 'user250000@example.com';

You'll see output similar to this:

Seq Scan on customers  (cost=0.00..11374.00 rows=1 width=52) (actual time=45.123..91.456 rows=1 loops=1)
  Filter: ((email)::text = 'user250000@example.com'::text)
  Rows Removed by Filter: 499999
Planning Time: 0.085 ms
Execution Time: 91.502 ms

The key detail here is Seq Scan — PostgreSQL scanned all 500,000 rows to find a single match. It filtered out 499,999 rows. That's a lot of wasted work.

Step 3 – Create an Index

CREATE INDEX idx_customers_email ON customers (email);

This creates a B-tree index on the email column. The name idx_customers_email follows a common naming convention: idx_ prefix, then the table name, then the column name.

Step 4 – Query With the Index

Run the same query again:

EXPLAIN ANALYZE
SELECT * FROM customers WHERE email = 'user250000@example.com';

Now you'll see something like this:

Index Scan using idx_customers_email on customers  (cost=0.42..8.44 rows=1 width=52) (actual time=0.034..0.036 rows=1 loops=1)
  Index Cond: ((email)::text = 'user250000@example.com'::text)
Planning Time: 0.112 ms
Execution Time: 0.058 ms

The scan type changed from Seq Scan to Index Scan. The execution time dropped from ~91ms to ~0.06ms. That's roughly a 1,500x improvement — from one line of SQL.

How to Use EXPLAIN ANALYZE to Measure Performance

EXPLAIN ANALYZE is your most important tool for understanding how PostgreSQL executes a query. You already saw it in the previous section, but let's break down what the output means.

EXPLAIN ANALYZE SELECT * FROM customers WHERE city = 'Lagos';

The output will tell you several things:

• Scan type — whether PostgreSQL used a sequential scan, index scan, bitmap index scan, or another access method

• Cost — the estimated cost in arbitrary units. The first number is the startup cost, the second is the total cost

• Rows — how many rows PostgreSQL estimated it would find versus how many it actually found

• Actual time — the real time in milliseconds to execute the query

• Rows Removed by Filter — how many rows were scanned but didn't match the condition

If you see Seq Scan on a large table with a selective WHERE clause, that's usually a sign you need an index. If you see Index Scan or Index Only Scan, your index is working.

One thing to keep in mind: EXPLAIN without ANALYZE shows the plan without actually running the query. EXPLAIN ANALYZE runs the query and shows real timing data. Always use EXPLAIN ANALYZE when you're investigating performance, but be careful with it on destructive queries — EXPLAIN ANALYZE DELETE FROM ... will actually delete the rows. Wrap those in a transaction and roll back:

BEGIN;
EXPLAIN ANALYZE DELETE FROM customers WHERE city = 'Berlin';
ROLLBACK;

Types of Indexes in PostgreSQL

PostgreSQL supports several index types, each optimized for different query patterns.

B-tree (Default)

B-tree is the default index type and covers the vast majority of use cases. It supports equality checks (=), range queries (<, >, <=, >=, BETWEEN), sorting (ORDER BY), and IS NULL / IS NOT NULL checks.

-- These are equivalent – B-tree is the default
CREATE INDEX idx_name ON customers (last_name);
CREATE INDEX idx_name ON customers USING btree (last_name);

Use B-tree when you don't have a specific reason to use something else.

Hash

Hash indexes are optimized purely for equality comparisons (=). They don't support range queries or sorting. In practice, B-tree handles equality checks almost as fast, so hash indexes are rarely necessary.

CREATE INDEX idx_email_hash ON customers USING hash (email);

Consider a hash index only if you have a very large table with frequent equality-only lookups and want to save a small amount of index space.

GIN (Generalized Inverted Index)

GIN indexes are designed for values that contain multiple elements — like arrays, JSONB documents, or full-text search vectors. Instead of indexing a single value per row, GIN indexes every element within the value.

-- Add a JSONB column
ALTER TABLE customers ADD COLUMN preferences JSONB DEFAULT '{}';

-- Index the JSONB column
CREATE INDEX idx_preferences ON customers USING gin (preferences);

-- Now this query uses the GIN index
SELECT * FROM customers WHERE preferences @> '{"newsletter": true}';

Use GIN when you're querying inside JSONB data, searching arrays with @> or &&, or doing full-text search with tsvector.

GiST (Generalized Search Tree)

GiST indexes support geometric data, ranges, and full-text search. They're commonly used with PostGIS for geospatial queries.

-- Range type example
CREATE TABLE events (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    duration TSRANGE
);

CREATE INDEX idx_event_duration ON events USING gist (duration);

-- Find overlapping events
SELECT * FROM events WHERE duration && '[2025-01-01, 2025-01-31]'::tsrange;

Use GiST when you're working with spatial data, range types, or need overlap/containment operators.

BRIN (Block Range Index)

BRIN indexes are extremely small and work well on large tables where the physical row order correlates with the indexed column's value. A common example is a timestamp column on an append-only table where new rows always have later timestamps.

CREATE INDEX idx_created_at_brin ON customers USING brin (created_at);

BRIN stores summary information (min/max values) for each block of rows rather than indexing every row individually. This makes the index much smaller than a B-tree, but it only works well when the data is naturally ordered.

Use BRIN for very large, append-only tables with naturally ordered data — like logs, events, or time-series data.

How to Create a Composite Index

A composite index (also called a multi-column index) covers more than one column. It's useful when your queries frequently filter or sort by multiple columns together.

CREATE INDEX idx_city_lastname ON customers (city, last_name);

The order of columns in a composite index matters. PostgreSQL can use this index for queries that filter on city alone, or on both city and last_name. But it can't efficiently use this index for queries that filter only on last_name.

Think of it like a phone book sorted by city first, then by last name within each city. You can easily look up everyone in Lagos. You can also look up everyone named "Adeyemi" in Lagos. But finding all people named "Adeyemi" across all cities requires scanning the whole book.

This principle is called the leftmost prefix rule: PostgreSQL can use a composite index for queries that include the leftmost column(s) of the index, but not for queries that skip them.

-- ✅ Uses the index (matches leftmost column)
SELECT * FROM customers WHERE city = 'Lagos';

-- ✅ Uses the index (matches both columns, left to right)
SELECT * FROM customers WHERE city = 'Lagos' AND last_name = 'Adeyemi';

-- ❌ Cannot use this index efficiently (skips the leftmost column)
SELECT * FROM customers WHERE last_name = 'Adeyemi';

When deciding column order, place the most selective column first — the one that narrows down the results the most.

How to Create a Partial Index

A partial index covers only a subset of rows in a table. You define the subset with a WHERE clause in the index definition.

This is useful when you only query a specific portion of the data. For example, if you have an orders table and you frequently query for pending orders but rarely look at completed ones:

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    customer_id INT NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'pending',
    total NUMERIC(10, 2),
    created_at TIMESTAMP DEFAULT NOW()
);

-- Only index rows where status is 'pending'
CREATE INDEX idx_orders_pending ON orders (customer_id)
WHERE status = 'pending';

This index is smaller than a full index because it skips all rows that don't match the WHERE condition. Smaller indexes use less disk space, consume less memory, and are faster to maintain during writes.

For the index to be used, your query's WHERE clause must match the index's condition:

-- ✅ Uses the partial index
SELECT * FROM orders WHERE status = 'pending' AND customer_id = 42;

-- ❌ Cannot use the partial index (different status)
SELECT * FROM orders WHERE status = 'shipped' AND customer_id = 42;

How to Create an Expression Index

Sometimes you need to index the result of a function or expression rather than a raw column value. Expression indexes (also called functional indexes) handle this.

A common scenario is case-insensitive email lookups. If your queries use LOWER(email), a regular index on email won't help — PostgreSQL sees the function call as a different expression.

-- Regular index on email – won't help with LOWER() queries
CREATE INDEX idx_email ON customers (email);

-- This query does NOT use the index above
SELECT * FROM customers WHERE LOWER(email) = 'user100@example.com';

To fix this, create an index on the expression itself:

CREATE INDEX idx_email_lower ON customers (LOWER(email));

Now queries that use LOWER(email) in their WHERE clause will use this index:

-- ✅ Uses the expression index
SELECT * FROM customers WHERE LOWER(email) = 'user100@example.com';

The rule is straightforward: the expression in your query must match the expression in the index exactly. If the index is on LOWER(email), your query must also use LOWER(email).

How to Create a Unique Index

A unique index guarantees that no two rows have the same value (or combination of values) in the indexed columns. It serves a dual purpose: it enforces data integrity and provides fast lookups.

CREATE UNIQUE INDEX idx_customers_email_unique ON customers (email);

If you try to insert a duplicate value, PostgreSQL will reject the operation:

INSERT INTO customers (first_name, last_name, email, city)
VALUES ('Test', 'User', 'user1@example.com', 'Lagos');
-- ERROR: duplicate key value violates unique constraint "idx_customers_email_unique"

You might wonder how this differs from a UNIQUE constraint. Under the hood, PostgreSQL implements UNIQUE constraints by creating a unique index. The two are functionally identical.

The difference is intent — a UNIQUE constraint expresses a data integrity rule, while a unique index explicitly focuses on query performance with uniqueness as a bonus.

How to Manage Indexes

As your database grows, you'll need to inspect, monitor, and maintain your indexes.

How to List All Indexes on a Table

SELECT
    indexname,
    indexdef
FROM pg_indexes
WHERE tablename = 'customers';

This shows the name and full definition of every index on the table.

How to Check Index Size

SELECT
    pg_size_pretty(pg_relation_size('idx_customers_email')) AS index_size;

For a broader view of all indexes and their sizes:

SELECT
    indexrelname AS index_name,
    pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE relname = 'customers'
ORDER BY pg_relation_size(indexrelid) DESC;

How to Find Unused Indexes

Indexes that are never used waste disk space and slow down writes. You can find them by checking pg_stat_user_indexes:

SELECT
    indexrelname AS index_name,
    idx_scan AS times_used,
    pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE relname = 'customers'
AND idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;

If an index has idx_scan = 0 after a reasonable period of normal usage, it's a candidate for removal. Just make sure to check across a full business cycle — some indexes are only used during monthly reports or seasonal operations.

How to Drop an Index

DROP INDEX IF EXISTS idx_customers_email;

If you're dropping an index on a production table and want to avoid locking writes, use CONCURRENTLY:

DROP INDEX CONCURRENTLY IF EXISTS idx_customers_email;

How to Rebuild an Index

Over time, indexes can become bloated as rows are inserted, updated, and deleted. You can rebuild an index to reclaim space:

REINDEX INDEX idx_customers_email;

Or rebuild all indexes on a table:

REINDEX TABLE customers;

On production systems, use REINDEX CONCURRENTLY (PostgreSQL 12+) to avoid locking the table:

REINDEX INDEX CONCURRENTLY idx_customers_email;

When Indexes Hurt Instead of Help

Indexes aren't free. Every index you add comes with costs:

1. Write overhead — every INSERT, UPDATE, or DELETE must also update every index on the table. If a table has 10 indexes and you insert a row, PostgreSQL performs 11 write operations (one for the table and one for each index). On write-heavy tables, excessive indexes can significantly slow down data modification.

2. Storage cost — indexes consume disk space. On large tables, indexes can take up as much space as the table itself, sometimes more. You can check this with pg_relation_size.

3. Memory consumption — PostgreSQL caches frequently used indexes in memory. More indexes means more memory pressure, which can push useful data out of the cache and slow down other queries.

4. Maintenance burden — indexes need periodic maintenance (vacuuming, reindexing) and add complexity to schema migrations.

The question to ask is not "should I add an index?" but rather "does the read performance gain justify the write performance cost for this table's workload?"

Common Mistakes That Prevent Index Usage

You can have the perfect index and PostgreSQL might still ignore it. Here are the most common reasons.

Wrapping the Indexed Column in a Function

-- Index on email
CREATE INDEX idx_email ON customers (email);

-- ❌ PostgreSQL cannot use the index because of LOWER()
SELECT * FROM customers WHERE LOWER(email) = 'user1@example.com';

-- ✅ Fix: create an expression index on LOWER(email)
CREATE INDEX idx_email_lower ON customers (LOWER(email));

Any function applied to the indexed column in a WHERE clause prevents the standard index from being used. You need an expression index that matches the function.

Implicit Type Casting

-- id is an INTEGER column with an index
-- ❌ Passing a string forces a type cast, which may prevent index usage
SELECT * FROM customers WHERE id = '42';

-- ✅ Use the correct type
SELECT * FROM customers WHERE id = 42;

When the query's value type doesn't match the column type, PostgreSQL may cast the column to match, which prevents index usage.

Using OR Conditions Across Different Columns

-- ❌ OR across different columns can prevent index usage
SELECT * FROM customers WHERE email = 'user1@example.com' OR city = 'Lagos';

-- ✅ Rewrite as UNION for better index utilization
SELECT * FROM customers WHERE email = 'user1@example.com'
UNION
SELECT * FROM customers WHERE city = 'Lagos';

Leading Wildcards in LIKE Queries

-- ❌ Leading wildcard cannot use a B-tree index
SELECT * FROM customers WHERE email LIKE '%@example.com';

-- ✅ Trailing wildcard CAN use a B-tree index
SELECT * FROM customers WHERE email LIKE 'user1%';

A B-tree index is sorted from left to right. A leading wildcard (%something) means the database can't use the sorted structure and falls back to a sequential scan. If you need to search by suffix or substring, consider a GIN index with the pg_trgm extension.

Low Selectivity

If a column has very few distinct values relative to the number of rows (low selectivity), PostgreSQL may decide a sequential scan is faster than using the index.

For example, if a status column has only three possible values ('pending', 'shipped', 'delivered') and each value covers roughly a third of the table, an index on status alone provides little benefit. PostgreSQL would still need to read a large portion of the table, and the extra index lookup adds overhead.

A partial index is often the better solution in these cases.

Best Practices for Indexing

Here's a summary of the key principles to follow:

1. Index columns that appear in WHERE, JOIN, and ORDER BY clauses. These are the columns the database needs to search, match, or sort by. Start with the queries that run most frequently or take the longest.

2. Measure before and after with EXPLAIN ANALYZE. Never add an index based on guesswork. Run your query with EXPLAIN ANALYZE, add the index, and run it again. If the execution time doesn't improve meaningfully, the index isn't helping.

3. Don't index every column. Each index slows down writes and consumes storage. Be deliberate about which columns you index based on actual query patterns.

4. Use composite indexes for multi-column filters. If your queries commonly filter on city and last_name together, a composite index on (city, last_name) is more efficient than two separate single-column indexes.

5. Put the most selective column first in composite indexes. The column that narrows the results the most should come first.

6. Use partial indexes when you only query a subset of data. If 90% of your queries target rows where status = 'active', a partial index on that subset is smaller and faster than a full index.

7. Monitor index usage regularly. Query pg_stat_user_indexes to find unused indexes and remove them.

8. Rebuild bloated indexes periodically. On tables with heavy update/delete activity, indexes can become bloated. Use REINDEX CONCURRENTLY on production systems.

Conclusion

In this tutorial, you learned what database indexes are and why they matter for query performance. You explored how B-tree indexes work under the hood, created several types of indexes (single-column, composite, partial, expression, and unique), and used EXPLAIN ANALYZE to measure the impact.

You also learned about the trade-offs indexes introduce — write overhead, storage cost, and memory pressure — and the common mistakes that silently prevent PostgreSQL from using your indexes.

The core principle is simple: index deliberately based on your actual query patterns, measure the results, and remove anything that isn't pulling its weight.

If you found this tutorial helpful, you can find more of my writing on freeCodeCamp and connect with me on LinkedIn and X.

A database trigger is a function that the database executes automatically when a specific event occurs on a table. You don't call it manually. Instead, you define the conditions, and the database handles the rest.

In this tutorial, you'll learn what triggers are, how they work, when to use them, and when to avoid them. You'll work through practical examples using PostgreSQL, but the core concepts apply to most relational databases.

Table of Contents

• Prerequisites

• How Triggers Work

• How to Create Your First Trigger

• BEFORE vs AFTER Triggers

• How to Build an Audit Log with an AFTER Trigger

• How to Use a BEFORE Trigger for Validation

• Row-Level vs Statement-Level Triggers

• The NEW and OLD Variables Reference

• How to Manage Triggers

• When to Use Triggers

• When to Avoid Triggers

• Conclusion

Prerequisites

To follow along with the examples, you'll need:

• Basic knowledge of SQL (SELECT, INSERT, UPDATE, DELETE)

• A running PostgreSQL instance (version 12 or later)

• A SQL client like psql, pgAdmin, or DBeaver

If you don't have PostgreSQL installed, you can use a free cloud-hosted instance from services like Neon or Supabase to follow along.

How Triggers Work

At a high level, a trigger has three parts:

1. The event: what action activates the trigger (INSERT, UPDATE, DELETE, or TRUNCATE)

2. The timing: when the trigger fires relative to the event (BEFORE or AFTER)

3. The function: what logic runs when the trigger fires

Here's the general flow: a user or application performs an operation on a table, the database checks if any triggers are associated with that operation, and if a match is found, the database executes the trigger function automatically.

You can think of triggers as event listeners for your database. Just like a JavaScript addEventListener watches for a click or keypress, a database trigger watches for row-level changes on a table.

How to Create Your First Trigger

In PostgreSQL, creating a trigger is a two-step process. You first create a trigger function, then you attach that function to a table with a CREATE TRIGGER statement.

Let's build a concrete example. Say you have a products table and you want to automatically set the updated_at timestamp every time a row is modified.

Step 1 – Create the Table

CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    price NUMERIC(10, 2) NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

Step 2 – Create the Trigger Function

A trigger function in PostgreSQL is a special function that returns the TRIGGER type. Inside the function body, you have access to two important variables: NEW (the row after the operation) and OLD (the row before the operation).

CREATE OR REPLACE FUNCTION set_updated_at()
RETURNS TRIGGER AS $$
BEGIN
    NEW.updated_at = NOW();
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

This function sets the updated_at column to the current timestamp every time it runs. It then returns NEW, which tells PostgreSQL to proceed with the modified row.

Step 3 – Attach the Trigger to the Table

CREATE TRIGGER trigger_set_updated_at
BEFORE UPDATE ON products
FOR EACH ROW
EXECUTE FUNCTION set_updated_at();

Let's break down each part of this statement:

• BEFORE UPDATE – the trigger fires before the update is applied to the table

• ON products – the trigger is associated with the products table

• FOR EACH ROW – the function runs once for every row affected by the update

• EXECUTE FUNCTION set_updated_at() – the function to call

Step 4 – Test It

INSERT INTO products (name, price) VALUES ('Wireless Keyboard', 49.99);

-- Wait a moment, then update the row
UPDATE products SET price = 44.99 WHERE name = 'Wireless Keyboard';

SELECT name, price, created_at, updated_at FROM products;

You'll see that updated_at has been automatically updated to the time of the UPDATE operation, even though you didn't explicitly set it in your query. That's the trigger doing its job.

BEFORE vs AFTER Triggers

The timing of a trigger determines when the function executes relative to the actual data change.

BEFORE triggers run before the row is inserted, updated, or deleted. They are useful when you want to modify or validate the incoming data. Since the change hasn't been applied yet, you can alter the NEW row or even cancel the operation entirely by returning NULL.

AFTER triggers run after the row change has been committed to the table. They are useful for side effects like logging, sending notifications, or updating related tables. At this point, the change is already done, so you can't modify the row – but you can read both OLD and NEW to see what changed.

Here's a rule of thumb: use BEFORE triggers when you need to change or reject data, and use AFTER triggers when you need to react to a completed change.

How to Build an Audit Log with an AFTER Trigger

One of the most common uses for triggers is audit logging – keeping a record of every change made to an important table. Let's build one.

Step 1 – Create an Audit Table

CREATE TABLE product_audit (
    audit_id SERIAL PRIMARY KEY,
    product_id INT NOT NULL,
    action VARCHAR(10) NOT NULL,
    old_price NUMERIC(10, 2),
    new_price NUMERIC(10, 2),
    changed_by TEXT DEFAULT current_user,
    changed_at TIMESTAMP DEFAULT NOW()
);

Step 2 – Create the Audit Trigger Function

CREATE OR REPLACE FUNCTION log_product_changes()
RETURNS TRIGGER AS $$
BEGIN
    IF TG_OP = 'UPDATE' THEN
        INSERT INTO product_audit (product_id, action, old_price, new_price)
        VALUES (OLD.id, 'UPDATE', OLD.price, NEW.price);
    ELSIF TG_OP = 'DELETE' THEN
        INSERT INTO product_audit (product_id, action, old_price)
        VALUES (OLD.id, 'DELETE', OLD.price);
    ELSIF TG_OP = 'INSERT' THEN
        INSERT INTO product_audit (product_id, action, new_price)
        VALUES (NEW.id, 'INSERT', NEW.price);
    END IF;

    RETURN COALESCE(NEW, OLD);
END;
$$ LANGUAGE plpgsql;

There are a few important things happening here. The TG_OP variable is a special string that PostgreSQL provides inside trigger functions. It tells you which operation activated the trigger: 'INSERT', 'UPDATE', or 'DELETE'. This lets you handle different operations with a single function.

The RETURN COALESCE(NEW, OLD) at the end ensures the function returns the correct row. For INSERT and UPDATE operations, NEW exists and is returned. For DELETE operations, NEW is null, so OLD is returned instead.

Step 3 – Attach the Trigger

CREATE TRIGGER trigger_product_audit
AFTER INSERT OR UPDATE OR DELETE ON products
FOR EACH ROW
EXECUTE FUNCTION log_product_changes();

Notice the AFTER INSERT OR UPDATE OR DELETE syntax. You can bind a single trigger to multiple events, which keeps your setup clean.

Step 4 – Test It

-- Insert a new product
INSERT INTO products (name, price) VALUES ('USB-C Hub', 29.99);

-- Update the price
UPDATE products SET price = 24.99 WHERE name = 'USB-C Hub';

-- Delete the product
DELETE FROM products WHERE name = 'USB-C Hub';

-- Check the audit log
SELECT * FROM product_audit ORDER BY changed_at;

You'll see three rows in product_audit (one for each operation) with the old and new prices recorded automatically. No application code needed.

How to Use a BEFORE Trigger for Validation

Triggers can also enforce business rules at the database level. Let's say you want to prevent any product from having a negative price.

CREATE OR REPLACE FUNCTION prevent_negative_price()
RETURNS TRIGGER AS $$
BEGIN
    IF NEW.price < 0 THEN
        RAISE EXCEPTION 'Product price cannot be negative. Got: %', NEW.price;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_check_price
BEFORE INSERT OR UPDATE ON products
FOR EACH ROW
EXECUTE FUNCTION prevent_negative_price();

Now test it:

INSERT INTO products (name, price) VALUES ('Faulty Item', -10.00);
-- ERROR: Product price cannot be negative. Got: -10.00

The insert is rejected entirely. The row never makes it into the table. This is powerful because the rule is enforced at the database level regardless of which application or script sends the query.

Row-Level vs Statement-Level Triggers

All the triggers you've seen so far use FOR EACH ROW, which means the function runs once per affected row. If you update 100 rows in a single query, the trigger function runs 100 times.

PostgreSQL also supports FOR EACH STATEMENT triggers, which run once per SQL statement regardless of how many rows are affected.

CREATE OR REPLACE FUNCTION log_bulk_update()
RETURNS TRIGGER AS $$
BEGIN
    RAISE NOTICE 'A bulk operation was performed on the products table';
    RETURN NULL;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_bulk_update_notice
AFTER UPDATE ON products
FOR EACH STATEMENT
EXECUTE FUNCTION log_bulk_update();

Statement-level triggers are less common, but they're useful for operations like refreshing a materialized view or sending a single notification after a batch update instead of one notification per row.

Important: in statement-level triggers, the NEW and OLD variables are not available because the trigger isn't tied to any specific row.

The NEW and OLD Variables Reference

Here's a quick reference for when NEW and OLD are available in row-level triggers:

Understanding when each variable is available will save you from runtime errors in your trigger functions.

How to Manage Triggers

As you add more triggers to your database, you'll need to know how to inspect, disable, and remove them.

How to List All Triggers on a Table

SELECT trigger_name, event_manipulation, action_timing
FROM information_schema.triggers
WHERE event_object_table = 'products';

How to Disable a Trigger Temporarily

-- Disable a specific trigger
ALTER TABLE products DISABLE TRIGGER trigger_product_audit;

-- Disable all triggers on a table
ALTER TABLE products DISABLE TRIGGER ALL;

This is useful during bulk data migrations where you want to skip trigger execution for performance reasons.

How to Re-Enable a Trigger

ALTER TABLE products ENABLE TRIGGER trigger_product_audit;

How to Drop a Trigger

DROP TRIGGER IF EXISTS trigger_product_audit ON products;

Note that dropping a trigger does not drop the associated function. You'll need to drop the function separately if you no longer need it:

DROP FUNCTION IF EXISTS log_product_changes();

When to Use Triggers

Triggers work well for specific use cases. Here are the scenarios where they're a strong choice:

• Audit logging: automatically recording who changed what and when, as you saw earlier in this tutorial.

• Derived data maintenance: keeping computed columns, counters, or summary tables in sync with the source data.

• Data validation: enforcing business rules that go beyond what CHECK constraints can express, like cross-table validations.

• Automatic timestamping: setting created_at and updated_at fields without relying on the application layer.

When to Avoid Triggers

Triggers are powerful, but they come with trade-offs. Here are cases where you should think twice before using them:

• Complex business logic: if the logic involves calling external APIs, sending emails, or orchestrating multi-step workflows, it belongs in your application layer. Triggers should stay lightweight.

• Performance-sensitive bulk operations: row-level triggers on tables that frequently receive bulk inserts or updates can create significant overhead. If you're inserting millions of rows, those triggers fire millions of times.

• Cascading triggers: when one trigger's action fires another trigger, which fires another, debugging becomes extremely difficult. If you find yourself building a chain of triggers, reconsider the design.

• Logic that developers need to discover easily: triggers are sometimes called "hidden logic" because they execute automatically without appearing in application code. If your team frequently asks "why did this column change?" and the answer is always "there's a trigger," that's a sign the logic might be more discoverable if placed in your application layer or a stored procedure that's called explicitly.

A good rule of thumb: if the logic is tightly coupled to the data and should always execute regardless of which client or service touches the table, a trigger is appropriate. If the logic depends on application context (like the current user's session, feature flags, or external state), it belongs in the application.

Conclusion

In this tutorial, you learned what database triggers are and how they work in PostgreSQL. You built three practical triggers: an automatic timestamp updater, a full audit logging system, and a data validation guard. You also learned the difference between BEFORE and AFTER triggers, row-level and statement-level triggers, and when NEW and OLD variables are available.

Triggers are a powerful tool for keeping your data consistent and your business rules enforced at the database level. Use them for focused, data-centric operations, and keep the logic simple.

If you found this tutorial helpful, you can connect with me on LinkedIn and X.

In this article, we'll talk about what it takes to design a highly-functional database using some key best practices.

This article is aimed at developers and those looking to start a career in managing Databases. We'll discuss what a database actually is, the components of a Database System, what we mean by database design, the stages of database design, and what's involved in Database System Design.

Table of Contents

1. Prerequisites and Setup

   • 1. Foundational Knowledge

   • 2. Software and Installation

   • 3. Verifying Your Setup

2. What is a Database?

3. Components of a Database System

4. Types of Database Systems

5. Database System vs. DBMS

6. Characteristics of a Good Database

7. Stages of Database Design

8. The Role of Normalisation

9. Practical: Designing a Library System

   • Step 1: Requirements & ER Diagram

   • Step 2: Normalization in Action

   • Step 3: Implementation (SQL)

10. Conclusion

Prerequisites and Setup

To get the most out of this guide, you should have the following foundational skills and tools ready. This will help ensure that you aren't just reading theory, but that you're actually building a functional system.

1. Foundational Knowledge

• Data Types: You should be able to distinguish between basic data formats. In database design, choosing the wrong type can lead to storage waste or application errors.

  • Strings/Varchars: Textual data (for example, "John Doe", "123 Main St").

  • Integers: Whole numbers used for math or unique IDs (for example, 10, 500).

  • Floats/Decimals: Numbers with decimal points, usually for currency (for example, 19.99).

  • Booleans: Simple True/False toggles (for example, is_available).

• Logical Thinking: You should be comfortable identifying "entities." If you're building an app for a school, you'll need to recognize that "Students," "Teachers," and "Classrooms" are separate objects that must be linked via relationships.

• Terminal/CLI Basics: While we'll use visual tools, you should know how to open your Command Prompt (Windows) or Terminal (Mac/Linux) and understand that commands are often case-sensitive.

2. Software and Installation

We'll use PostgreSQL (the database engine) and pgAdmin 4 (the visual management tool).

1. Download: Visit the official PostgreSQL Downloads page and select the installer for your operating system.

2. Installation Wizard: Run the installer. When asked which components to include, make sure that PostgreSQL Server, pgAdmin 4, and Command Line Tools are all checked.

3. The "Postgres" User: During setup, you will be prompted to create a password for the default "postgres" superuser. Note: Write this password down. You can't easily reset it, and you'll need it to access your data.

4. Port Selection: The default port is 5432. Keep this as the default unless you're an advanced user with a specific reason to change it.

3. Verifying Your Setup

Before moving to the practical section, let's verify that everything is installed correctly:

1. Open pgAdmin 4 from your applications menu.

2. In the left-hand sidebar, click on Servers.

3. Enter the master password you created during installation.

4. If you see "PostgreSQL [Version Number]" appear with a green icon, your database environment is successfully configured.

What is a Database?

A Database is a collection of structured data usually stored electronically in a computer. Databases are controlled and managed using a Database Management System (DBMS). Database Management Software is an application that constructs and maintain (and sometimes expands) Databases. Examples of DBMS are IBM's DB2, Oracle Corporation's Oracle, Microsoft Access, and Microsoft's SQL Server.

We use databases everyday, whether knowingly or unknowingly. And as a developer, you'll likely need to at least understand Database basics so you can effectively work with them.

It's also important for you to how to know how to design a scalable database, as well as be familiar with the environment in which the database will be housed (called, not surprisingly, a Database Environment). The hardware and operating system that house the database makes up this Database Environment.

Components of a Database System

A Database System is a computerized record-keeping system designed to store, manage, and retrieve data efficiently. It acts as a centralized repository that allows multiple users to access and manipulate data simultaneously while ensuring the integrity, security, and persistence of that information over time.

A Database System consists of four basic components:

1. Hardware

This includes the secondary storage where the database resides alongside other necessary components. Examples are Hard disks, Processors, RAM, and so on. Since a database can span from a single workstation to a global mainframe, hardware selection is a priority. Proper investment in processing power and storage is essential to handling the projected user load and data volume.

2. Software

In this case, Database Management Software (DBMS) is in charge of the maintenance and management of databases. It's robust software that acts as an intermediary, restricting users from the complex hardware-level details of data storage. The Software layer (DBMS) handles data storage, retrieval, and processing. Examples of DBMS are Microsoft's SQL Server, IBM's DB2, and Oracle.

3. Data

Data serves as the bridge connecting the machine components (hardware and software) to the human users. In a database system, data is organized into two main types:

• User Data: The actual structured information stored in tables, made up of columns (attributes) and rows (records).

• Metadata: Often defined as "data about data," metadata is stored in system tables and describes the actual structure of the database, such as the number of tables, field names, and defined primary keys

4. Users

These are the people who interact with the database to carry out their business responsibilities. Users generally fall into three distinct categories:

• Database Administrators (DBAs): Technical experts who hold central responsibility for the database. They monitor performance, define security and integrity checks, and establish backup and recovery strategies.

• Database Designers/Programmers: The engineers who actually write the code and use the DBMS to create the database's logical structure.

• End Users: The everyday people who access the database using query languages or simple menu-driven application interfaces

Types of Database Systems

It's important to know that not all databases store data in the same way. The choice of database depends on the specific needs of the application. The primary types include:

Hierarchical and Network Databases

These are older, legacy models. Hierarchical databases structure data in a tree-like, parent-child format where a child can only have one parent. Network databases improved on this by allowing a graph-like structure where records can have multiple parent and child relationships, making it easier to model complex associations.

Relational Databases (RDBMS)

The most widely used type today. They organize data into structured tables consisting of rows and columns. These tables are linked using primary and foreign keys, and they use Structured Query Language (SQL) for operations. They are ideal for applications requiring strong consistency, like banking systems.

Object-Oriented Databases (OODBMS)

These combine database capabilities with object-oriented programming principles (like Java or C++). Data is stored as "objects" that contain both the data and the methods (functions) that operate on it, making them great for complex data like multimedia or engineering designs.

NoSQL Databases

Designed to handle large volumes of unstructured or semi-structured data. Unlike relational databases, they don't rely on rigid table structures and are highly scalable. Types of NoSQL include document stores (for example, MongoDB), key-value stores (for example, Redis), column-family stores, and graph databases.

Cloud and Distributed Databases

Cloud databases are hosted on cloud platforms (like AWS or Microsoft Azure) and offer elasticity, scalability, and cost-efficiency (pay-as-you-go).

Distributed databases store data across multiple physical locations but function as a single unified system to the user, providing high availability and fault tolerance.

Database System vs. Database Management System (DBMS)

People often use "Database" and "DBMS" interchangeably, but there is a distinct difference:

• Database Management System (DBMS): This is purely the software that helps users interact with the database. It handles data storage, retrieval, security, and concurrency control. Examples include MySQL, PostgreSQL, and Oracle DB.

• Database System: This is the broader concept that encompasses the entire setup. It includes the actual database (where data is stored), the DBMS software, the physical hardware, the network, and the users interacting with it.

Characteristics of a Good Database

To make sure that your database design is successful, it should exhibit several core characteristics:

• Data integrity and consistency: Ensuring data is accurate, reliable, and uniform across the entire system.

• Data security: Protecting sensitive information from unauthorized access and potential breaches.

• Scalability and performance: The ability to handle increasing amounts of data and users efficiently while providing fast query processing.

• Redundancy management (normalization): Avoiding unnecessary duplication of data to save storage space and prevent errors during updates.

• Concurrency control: Allowing multiple users to access and modify data simultaneously without causing conflicts or data corruption.

• Backup and recovery: Supporting robust mechanisms to recover data in the event of hardware or system failures.

Stages of Database Design

Database design is a structured process consisting of several stages to ensure that data is efficiently stored, accessed, and managed. There are four key stages in this process:

Requirements Analysis

This is the foundational stage where designers gather and analyse the specific needs of users and the business. It involves identifying the overall purpose of the database, understanding data requirements, defining key entities and attributes, and establishing both functional and non-functional requirements.

Conceptual Design

In this phase, a high-level visual blueprint of the database is created, which is independent of any specific software implementation. This makes it easy for non-technical stakeholders to understand.

Designers typically use Entity-Relationship (ER) models or UML diagrams to identify entities, map out relationships, and define constraints like primary keys.

Logical Design

This stage involves translating the conceptual blueprint into a logical model that aligns with a specific type of Database Management System (DBMS), such as a relational or NoSQL system.

Key steps include converting the ER diagram into relational schemas (tables and columns), defining foreign and primary keys, and normalising the database to remove anomalies and reduce data redundancy.

Physical Design

The final stage translates the logical model into an actual physical structure optimized for high performance and efficient storage. Activities here include selecting the specific DBMS, establishing indexing strategies to speed up data retrieval, defining access paths, and configuring essential security policies and backup mechanisms.

The Role of Normalisation in Database Design

When building a relational database, one of the most critical steps in the logical design phase is a process called Normalisation.

Normalisation is a systematic approach to organising data to minimise redundancy (duplicate data) and improve overall data integrity. Essentially, it involves taking large, clunky tables and decomposing them into smaller, more focused tables, then linking them together using defined relationships.

Why is Normalisation Important?

A poorly designed database often suffers from errors that happen when you try to insert, update, or delete data. For example, if a teacher's phone number is stored in multiple places, updating it in one row but forgetting another creates an update anomaly. Normalisation solves this by ensuring each piece of information is stored in only one place.

The main objectives of normalisation are:

• Eliminating redundancy: By reducing duplicate data, you save valuable storage space and keep your data consistent.

• Avoiding anomalies: It prevents data corruption that arises during insertion, updating, or deletion.

• Ensuring data integrity: It maintains the accuracy and reliability of the data across the entire database.

• Enhancing query performance: Organising the data efficiently helps optimise how data is retrieved and updated.

Stages of Normalisation

Normalisation happens in sequential stages known as Normal Forms (NFs), where each stage builds upon the rules of the previous one to further refine the database structure.

For beginners, the first three forms are the most important to understand:

• First Normal Form (1NF): This stage ensures "atomicity". This means that every column in a table should hold a single, indivisible value, and duplicate columns are eliminated. For instance, you would not store two different phone numbers in a single "Phone" cell; you would separate them.

• Second Normal Form (2NF): To achieve 2NF, the table must first be in 1NF. Then, it ensures that all non-key attributes (the regular data columns) are fully dependent on the entire primary key. This often involves creating separate tables for distinct entities, like putting "Courses" into their own table rather than mixing them with "Student" details.

• Third Normal Form (3NF): A table in 3NF is already in 2NF and has removed all "transitive dependencies". This means that a non-key column shouldn't depend on another non-key column. For example, if a table has an "Instructor Name" and "Instructor Phone," those details should live in a dedicated "Instructor" table, not inside a "Course" table.

• Boyce-Codd Normal Form (BCNF): This is a stricter version of 3NF used to resolve any remaining, complex anomalies.

Finding the Right Balance

While normalisation is crucial for maintaining data consistency, it's important to strike a balance. Minimising redundancy is great, but excessive normalisation creates dozens of tiny tables. When you need to retrieve a complete record, the database system has to piece all those tables back together (using complex queries called "joins"), which can slow down performance.

So the ultimate goal of a good database designer is to find the sweet spot between a highly normalised structure and efficient query performance.

Practical: Designing a Library System

To move from theory to practice, let’s build a database for a small local library. We'll go through the design stages to ensure the data is structured efficiently.

Step 1: Requirements & ER Diagram

First, we'll identify what our library needs to track. We have three main entities:

• Authors: The writers of the books.

• Books: The actual items available for loan.

• Members: The people who borrow the books.

The Relationships:

• One Author can write many Books (One-to-Many).

• One Member can borrow many Books (One-to-Many).

Here's the ER diagram I've created for this example:

Step 2: Normalization in Action

To ensure our database is "well-designed" and free of redundancy, we'll apply the normalization rules discussed earlier. Instead of one giant spreadsheet, we split the data into three distinct tables:

1. Authors Table: * author_id (Primary Key)

   • author_name

2. Books Table: * book_id (Primary Key)

   • title

   • isbn

   • author_id (Foreign Key linking to the Authors table)

3. Members Table: * member_id (Primary Key)

   • first_name

   • last_name

   • email (Unique constraint)

Step 3: Implementation (SQL)

Now, let’s use the PostgreSQL Query Tool in pgAdmin 4 to actually create these tables and insert some dummy data.

-- 1. Create the Authors table
CREATE TABLE Authors (
    author_id SERIAL PRIMARY KEY,
    author_name VARCHAR(100) NOT NULL
);

-- 2. Create the Books table with a relationship to Authors
CREATE TABLE Books (
    book_id SERIAL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    isbn VARCHAR(20) UNIQUE,
    author_id INT REFERENCES Authors(author_id)
);

-- 3. Create the Members table
CREATE TABLE Members (
    member_id SERIAL PRIMARY KEY,
    first_name VARCHAR(50),
    last_name VARCHAR(50),
    email VARCHAR(100) UNIQUE NOT NULL
);

-- 4. Insert dummy data to test the design
INSERT INTO Authors (author_name) 
VALUES ('J.R.R. Tolkien'), ('George R.R. Martin');

INSERT INTO Books (title, isbn, author_id) 
VALUES ('The Hobbit', '978-0261102217', 1), 
       ('A Game of Thrones', '978-0553103540', 2);

Understanding the Schema Design:

By running the SQL script above, you’ve successfully transitioned from a logical design to a physical database. Here is a breakdown of the key concepts we applied:

• Primary Keys (PK): Using SERIAL PRIMARY KEY automatically creates a unique, incrementing ID for every new entry. This ensures no two authors or books are ever confused by the system.

• Foreign Keys (FK): The REFERENCES Authors(author_id) command is where the "Relational" part of a Relational Database happens. It tells the Books table that it must point to a valid ID in the Authors table, preventing "orphan" books without creators.

• Constraints: By using UNIQUE on the isbn and email columns, we've programmed the database to reject any duplicate data, ensuring high data integrity.

How to Fetch Your Data

Now that the data is stored, you need to know how to get it back out. In SQL, we can do this using the SELECT statement.

1. See Everything in a Table

To see all books currently in the library:

SELECT * FROM Books;

2. Filtering Results

Often, you don't want every single row. You can use the WHERE clause to filter for specific data. For example, to find an author by their exact name:

SELECT * FROM Authors 
WHERE author_name = 'J.R.R. Tolkien';

3. Joining Tables

In a normalized database, information is spread across tables. To see a list of book titles alongside their actual author names (instead of just an ID number), you use a JOIN.

SELECT Books.title, Authors.author_name
FROM Books
JOIN Authors ON Books.author_id = Authors.author_id;

The database is instructed by this query to "give the titles from the Books table and the names from the Authors table where the author_id matches in both." This enables you to effectively store data in distinct tables while viewing it as a single, comprehensive report.

This ability to link information across tables is what makes Relational Databases the industry standard for most business applications. But while the relational model is powerful, it isn't the only way to store data. Depending on whether you're handling social media connections, real-time sensor data, or simple document storage, you might need a different architectural approach.

Conclusion

Designing a database is much more than simply throwing information into a computer. It's the process of building a robust, efficient, and secure foundation for decision-making and business operations.

As we have explored here, a successful database relies on a carefully orchestrated ecosystem of hardware, software (the DBMS), data, and users.

By following the four stages of design – Requirements Analysis, Conceptual Design, Logical Design, and Physical Design – you can avoid costly structural mistakes and ensure that your system perfectly aligns with user needs.

Applying critical techniques like normalisation during this process guarantees that your data remains consistent, accurate, and free from frustrating anomalies.

Also, as the digital landscape continues to evolve, mastering these foundational concepts is your stepping stone into the future. Traditional relational databases remain incredibly powerful, but modern data demands are rapidly driving the adoption of cloud-based, AI-powered, and serverless database systems.

A well-designed system today must not only focus on data integrity and query performance but also prioritise scalability and stringent data security to protect against modern cyber threats.

Whether you are building a simple address book or architecting a backend for the next big application, keeping these core principles of database system design in mind will empower you to create resilient, high-performing, and future-proof data solutions.

These two operations look simple, but they hide a dangerous reliability problem. What if the database write succeeds but the message broker is temporarily unreachable? Or your service crashes between the two steps? You end up in an inconsistent state: your database has the new data, but the rest of the system never heard about it.

The Outbox Pattern is a well-established solution to this problem. In this tutorial, you'll learn what the pattern is, why it works, and how to implement it in Go with PostgreSQL and Google Cloud Pub/Sub.

Prerequisites

Before reading this tutorial, you should be familiar with:

• The basics of the Go programming language

• SQL and PostgreSQL

• The concept of database transactions

• Basic familiarity with event-driven or distributed systems (helpful but not required)

Table of Contents

1. The Problem: Two Operations, No Atomicity

2. How the Outbox Pattern Works

3. The Outbox Table Schema

4. The Message Relay

5. Go and PostgreSQL Implementation

   • The Orders Service

   • The Relay Service

6. Why Messages Can Be Delivered More Than Once

7. Alternative: PostgreSQL Logical Replication

8. Conclusion

The Problem: Two Operations, No Atomicity

To understand why the Outbox Pattern exists, you need to understand a core challenge in distributed systems: atomicity across different systems.

In a relational database, a transaction lets you group multiple operations so they either all succeed or all fail together. If you insert a row and update another row in the same transaction, you're guaranteed that both happen – or neither does.

The problem arises when you try to extend this guarantee across two different systems: for example, your database and your message broker (like Kafka, RabbitMQ, or Pub/Sub). These systems don't share a transaction boundary.

Here's a typical event-driven flow that breaks without the Outbox Pattern:

1. A user places an order.

2. Your service saves the order to the database ✅

3. Your service publishes an order.created event to the message broker ❌ (broker is down)

4. The order exists in the database, but downstream services never learned about it.

Or the reverse failure:

1. Your service publishes the event first ✅

2. Your service tries to save the order to the database ❌ (database times out)

3. Downstream services received a notification for an order that doesn't exist.

Either scenario leaves your system in an inconsistent state. This is the core problem the Outbox Pattern solves.

Here's what the process looks like when not using the Outbox Pattern:

How the Outbox Pattern Works

The Outbox Pattern solves the atomicity problem by keeping both operations inside the database:

1. Saves your business data (for example, a new order) to your database.

2. Writes the event message to a special table called the outbox table in the same database transaction.

3. A separate background process called the Message Relay polls the outbox table and publishes pending messages to the broker.

4. Once the broker confirms receipt, the relay marks the message as processed.

Because steps 1 and 2 happen in the same database transaction, they are atomic. Either both succeed or neither does. You can never end up with saved data but no corresponding event queued – or an event queued for data that was never saved.

The message is never published directly to the broker in your main application code. Instead, the database acts as a reliable staging area.

The Outbox Table Schema

The outbox table stores pending messages until the relay picks them up. Here's a typical PostgreSQL schema:

CREATE TABLE outbox (
    id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    topic       varchar(255)  NOT NULL,
    message     jsonb         NOT NULL,
    state       varchar(50)   NOT NULL DEFAULT 'pending',
    created_at  timestamptz   NOT NULL DEFAULT now(),
    processed_at timestamptz
);

Let's walk through each column:

• id: A unique identifier for each message. Using UUIDs makes it easy to reference specific messages.

• topic: The destination topic or queue name in your message broker (for example, orders.created).

• message: The event payload, stored as JSON. This is the data your consumers will receive.

• state: Tracks whether the message has been sent. The two main values are pending (waiting to be published) and processed (successfully published).

• created_at: When the message was inserted. The relay uses this to process messages in order.

• processed_at: When the relay successfully published the message.

You may want to add additional columns depending on your needs: for example, a retry_count column to track how many times the relay has attempted to send a message, or an error column to log failure reasons.

The Message Relay

The Message Relay is a background process (often a goroutine, a sidecar, or a separate service) that bridges the outbox table and the message broker.

Its responsibilities are:

1. Periodically query the outbox table for messages with state = 'pending'.

2. Publish each message to the appropriate topic in the broker.

3. Once the broker confirms delivery, update the row's state to 'processed'.

4. Handle failures gracefully: if publishing fails, leave the message as 'pending' so it will be retried.

This design gives you at-least-once delivery: a message will always be sent, even if the relay crashes and restarts. The trade-off is that a message might occasionally be sent more than once (more on this below), so your consumers should handle duplicates.

Go and PostgreSQL Implementation

Let's build a concrete example. Imagine you have an orders service. When a new order is created, you want to:

1. Save the order to a PostgreSQL orders table.

2. Publish an order.created event to Google Cloud Pub/Sub.

You'll use pgx for the PostgreSQL driver.

The Orders Service

The key insight is that the order insert and the outbox insert happen inside the same transaction. If anything goes wrong, both are rolled back.

// orders/main.go

package main

import (
	"context"
	"encoding/json"
	"log"
	"os"

	"github.com/google/uuid"
	"github.com/jackc/pgx/v5"
	"github.com/jackc/pgx/v5/pgxpool"
)

// Order represents a customer order in our system.
type Order struct {
	ID       uuid.UUID `json:"id"`
	Product  string    `json:"product"`
	Quantity int       `json:"quantity"`
}

// OrderCreatedEvent is the payload published to the message broker.
// It contains only the fields that downstream services need to know about.
type OrderCreatedEvent struct {
	OrderID uuid.UUID `json:"order_id"`
	Product string    `json:"product"`
}

// createOrderInTx saves a new order and its outbox event atomically.
// Both operations share the same transaction (tx), so either both succeed
// or both are rolled back — ensuring consistency.
func createOrderInTx(ctx context.Context, tx pgx.Tx, order Order) error {
	// Step 1: Insert the business data (the actual order).
	_, err := tx.Exec(ctx,
		"INSERT INTO orders (id, product, quantity) VALUES (\(1, \)2, $3)",
		order.ID, order.Product, order.Quantity,
	)
	if err != nil {
		return err
	}
	log.Printf("Inserted order %s into database", order.ID)

	// Step 2: Serialize the event payload that consumers will receive.
	event := OrderCreatedEvent{
		OrderID: order.ID,
		Product: order.Product,
	}
	msg, err := json.Marshal(event)
	if err != nil {
		return err
	}

	// Step 3: Write the event to the outbox table.
	// This does NOT publish to Pub/Sub — it just queues it for the relay.
	_, err = tx.Exec(ctx,
		"INSERT INTO outbox (topic, message) VALUES (\(1, \)2)",
		"orders.created", msg,
	)
	if err != nil {
		return err
	}
	log.Printf("Inserted outbox event for order %s", order.ID)

	return nil
}

func main() {
	ctx := context.Background()

	pool, err := pgxpool.New(ctx, os.Getenv("DATABASE_URL"))
	if err != nil {
		log.Fatalf("Unable to connect to database: %v", err)
	}
	defer pool.Close()

	// Begin a transaction that will cover both the order insert
	// and the outbox insert.
	tx, err := pool.Begin(ctx)
	if err != nil {
		log.Fatalf("Unable to begin transaction: %v", err)
	}
	// If anything fails, the deferred Rollback is a no-op after a successful Commit.
	defer tx.Rollback(ctx)

	newOrder := Order{
		ID:       uuid.New(),
		Product:  "Super Widget",
		Quantity: 10,
	}

	if err := createOrderInTx(ctx, tx, newOrder); err != nil {
		log.Fatalf("Failed to create order: %v", err)
	}

	// Committing the transaction makes both writes permanent simultaneously.
	if err := tx.Commit(ctx); err != nil {
		log.Fatalf("Failed to commit transaction: %v", err)
	}

	log.Println("Successfully created order and queued outbox event.")
}

Notice that createOrderInTx receives a pgx.Tx (a transaction) rather than a pool connection. This is intentional: it enforces that the caller is responsible for managing the transaction boundary, making the atomicity guarantee explicit.

The Relay Service

The relay runs as a separate background process. It polls the outbox table, publishes messages, and marks them as processed.

A critical detail here is the use of FOR UPDATE SKIP LOCKED in the SQL query. This PostgreSQL feature lets you run multiple relay instances concurrently without them stepping on each other. When one instance locks a row to process it, other instances skip that row and move on to the next one.

// relay/main.go

package main

import (
	"context"
	"log"
	"time"

	"cloud.google.com/go/pubsub"
	"github.com/google/uuid"
	"github.com/jackc/pgx/v5/pgxpool"
)

// OutboxMessage mirrors the columns we need from the outbox table.
type OutboxMessage struct {
	ID      uuid.UUID
	Topic   string
	Message []byte
}

// processOutboxMessages picks up one pending message, publishes it to Pub/Sub,
// and marks it as processed — all within a single database transaction.
func processOutboxMessages(ctx context.Context, pool *pgxpool.Pool, pubsubClient *pubsub.Client) error {
	tx, err := pool.Begin(ctx)
	if err != nil {
		return err
	}
	defer tx.Rollback(ctx)

	// Query for the next pending message.
	// FOR UPDATE SKIP LOCKED ensures that if multiple relay instances are
	// running, they won't try to process the same message simultaneously.
	rows, err := tx.Query(ctx, `
		SELECT id, topic, message
		FROM outbox
		WHERE state = 'pending'
		ORDER BY created_at
		LIMIT 1
		FOR UPDATE SKIP LOCKED
	`)
	if err != nil {
		return err
	}
	defer rows.Close()

	var msg OutboxMessage
	if rows.Next() {
		if err := rows.Scan(&msg.ID, &msg.Topic, &msg.Message); err != nil {
			return err
		}
	} else {
		// No pending messages — nothing to do.
		return nil
	}

	log.Printf("Publishing message %s to topic %s", msg.ID, msg.Topic)

	// Publish the message to the Pub/Sub topic and wait for confirmation.
	result := pubsubClient.Topic(msg.Topic).Publish(ctx, &pubsub.Message{
		Data: msg.Message,
	})
	if _, err = result.Get(ctx); err != nil {
		// Publishing failed. We return the error here without committing,
		// so the transaction rolls back and the message stays 'pending'.
		// The relay will retry it on the next polling interval.
		return err
	}

	// Mark the message as processed now that the broker has confirmed receipt.
	_, err = tx.Exec(ctx,
		"UPDATE outbox SET state = 'processed', processed_at = now() WHERE id = $1",
		msg.ID,
	)
	if err != nil {
		return err
	}
	log.Printf("Marked message %s as processed", msg.ID)

	// Commit the transaction: the state update becomes permanent.
	return tx.Commit(ctx)
}

func main() {
	// In production, initialize real connections using environment variables
	// or a config file. These are left as placeholders for clarity.
	var (
		pool         *pgxpool.Pool
		pubsubClient *pubsub.Client
	)

	// Poll the outbox table every second.
	// Adjust the interval based on your latency requirements.
	ticker := time.NewTicker(1 * time.Second)
	defer ticker.Stop()

	for range ticker.C {
		if err := processOutboxMessages(context.Background(), pool, pubsubClient); err != nil {
			log.Printf("Error processing outbox: %v", err)
		}
	}
}

The polling interval (1 second in this example) controls the maximum latency between an event being written to the outbox and it being published to the broker. For most use cases, 1–5 seconds is perfectly acceptable. If you need lower latency, you can reduce the interval, or consider using PostgreSQL's LISTEN/NOTIFY feature to wake up the relay immediately when a new row is inserted.

Why Messages Can Be Delivered More Than Once

You might wonder: isn't the Outbox Pattern supposed to guarantee exactly once delivery?

It does not. It guarantees at-least-once delivery. Here's the edge case:

1. The relay publishes the message to Pub/Sub successfully.

2. Before it can update the outbox row to 'processed', the relay process crashes.

3. On restart, the relay sees the message is still 'pending' and publishes it again.

This is a rare but possible scenario. The standard way to handle it is to design your message consumers to be idempotent. This means that they can safely receive and process the same message multiple times without causing incorrect behavior.

Common strategies for idempotency include:

• Using the message's id as a deduplication key, and checking if you've already processed it before acting.

• Making your operations naturally idempotent. For example, using INSERT ... ON CONFLICT DO NOTHING instead of a plain INSERT.

Alternative: PostgreSQL Logical Replication

The polling approach described above is simple and works well, but it has two drawbacks: it introduces some latency (up to one polling interval), and it issues database queries even when there's nothing to process.

For high-throughput systems where these trade-offs matter, PostgreSQL offers a more advanced alternative: logical replication via the Write-Ahead Log (WAL).

Every change made to a PostgreSQL database is first written to the WAL – an append-only log used for crash recovery and replication. With logical replication, you can subscribe to changes in specific tables and receive them as a stream in near real-time.

Instead of your relay asking "Are there any new messages?" on a schedule, PostgreSQL will proactively notify your relay the moment a new row is inserted into the outbox table.

This approach is lower latency and more resource-efficient for high-volume workloads. The trade-off is added implementation complexity: you need to manage a replication slot in PostgreSQL and handle the WAL stream correctly.

In Go, you can use the pglogrepl library to interact with PostgreSQL's logical replication protocol.

For more details on how WAL and change data capture work in PostgreSQL, see the official Write-Ahead Logging documentation.

Conclusion

The Outbox Pattern solves a fundamental problem in distributed systems: how do you reliably perform a database write and publish a message to a broker in a consistent way?

The key idea is to use your database as the source of truth for both the business data and the pending messages. By writing to the outbox table in the same transaction as your business data, you get atomic guarantees from the database itself: no distributed transaction protocol required.

Here's a quick summary of the key concepts:

• The outbox table stores pending events as part of your regular database schema.

• The transaction wraps both the business write and the outbox write, making them atomic.

• The Message Relay is a background process that reads from the outbox and publishes to the broker.

• At-least-once delivery means your consumers must be idempotent.

• FOR UPDATE SKIP LOCKED allows multiple relay instances to run safely in parallel.

• Logical replication is an advanced alternative that avoids polling for high-throughput systems.

The pattern is simple in concept, but there are several ways to implement it depending on your scale and infrastructure. The polling approach shown in this tutorial is a solid starting point for most applications.

Resources

• Source code on GitHub

• PostgreSQL Write-Ahead Logging (WAL)

• pglogrepl – Go library for PostgreSQL logical replication

• pgx – PostgreSQL driver and toolkit for Go

• Explore more Go tutorials on packagemain.tech

Disaster recovery testing is what separates assumed resilience from proven recovery, but it’s still skipped, rushed or treated as a checkbox exercise. For developers and technical teams, that gap can turn a manageable failure into a prolonged outage.

Table of Contents

• What is Disaster Recovery Testing?

• How Disaster Recovery Testing Works in Practice

• Disaster Recovery Testing Methods Developers Should Know

• What Technology Disaster Recovery Testing Evaluates

• How to Test a Disaster Recovery Plan

• Disaster Recovery Test Scenarios: Practical Examples

• Disaster Recovery Test Report: Turning Tests Into Improvements

• Disaster Recovery Audits and Continuous Validation

• Conclusion

What is Disaster Recovery Testing?

Disaster recovery (DR) testing is the process of validating that systems, data and applications can be restored after a disruptive event within defined recovery objectives. It generally evaluates:

• Recovery Time Objective (RTO): How quickly systems must be restored.

• Recovery Point Objective (RPO): How much data loss is acceptable.

• Operational readiness: Whether teams know what to do during an incident.

A disaster recovery test plan documents how these elements are tested, who is responsible and what success looks like. Without testing, DR plans are assumptions, not guarantees.

How Disaster Recovery Testing Works in Practice

In real environments, disaster recovery testing is used to check all elements of the disaster recovery plan and is rarely a single event. It’s a structured exercise that simulates failure, observes system behavior and measures outcomes against expectations.

A typical DR test involves:

1. Defining scope – Which applications, services, or data sets are included.

2. Selecting a scenario – Outage, corruption, ransomware, region failure, and so on.

3. Executing recovery actions – Restore data, fail over systems, reconfigure dependencies.

4. Measuring results – Time to recovery, data consistency, service availability.

5. Documenting findings – What worked, what failed, what needs improvement.

For developers, the key shift is recognizing that DR testing isn’t just an ops exercise. Application architecture, data handling and deployment patterns all influence recovery outcomes.

Importantly, regulatory pressure is also reshaping how organizations approach recovery validation. Frameworks such as the NIS2 Directive require essential and important entities in the EU to implement robust cybersecurity risk management measures, including incident response and business continuity capabilities.

Disaster Recovery Testing Methods Developers Should Know

Different testing methods provide different levels of confidence. Mature teams use more than one. Each method has a place, but relying only on low-impact testing creates blind spots that surface during real incidents.

Checklist Testing

The simplest method: Teams review documented recovery steps without executing them. This helps validate documentation completeness but does not confirm real-world recoverability.

Tabletop Exercises

Stakeholders walk through a simulated disaster scenario and discuss responses. Tabletop tests are useful for identifying communication gaps and unclear responsibilities, especially for cross-team coordination.

Partial or Component Testing

Specific systems, such as databases or backup restores, are tested in isolation. Developers often encounter this when validating recovery procedures for individual services or environments.

Full-scale Testing

This is the most comprehensive method. It involves actual failover or full recovery in production-like environments. While disruptive, full-scale tests provide the highest confidence.

What Technology Disaster Recovery Testing Evaluates

Modern environments are complex, and disaster recovery testing must validate more than just data restores.

DR testing evaluates:

• Backup integrity – Are backups usable, consistent and complete?

• Application dependencies – Do services come back in the correct order?

• Infrastructure recovery – Can compute, storage and networking be re-provisioned?

• Identity and access – Do credentials, secrets and permissions still function?

• Automation and scripts – Do recovery workflows still match current architectures?

For developers, this often reveals hidden coupling between services, outdated scripts or environment-specific assumptions that were never documented.

How to Test a Disaster Recovery Plan

Testing a disaster recovery plan doesn’t require shutting down production on day one. A practical, incremental approach works best.

1. Start with a single application: Pick a service with well-defined data and dependencies. Avoid starting with your most complex system.

2. Validate backup restores: Restore data into a non-production environment and confirm application functionality, not just file presence.

3. Measure RTO and RPO: Time the recovery process and compare results to stated objectives. At this stage, many teams can discover that their objectives were unrealistic.

4. Test failure assumptions: Simulate real-world issues like missing credentials, expired certificates or partial data loss.

5. Document gaps immediately: Update the disaster recovery test plan while findings are fresh. Untested fixes are just new assumptions.

This approach makes disaster recovery testing part of standard processes rather than a once-a-year compliance task.

Automating Restore Validation

One of the most common gaps in disaster recovery testing is stopping at “restore completed” instead of validating that the application actually works. A restored database that can’t serve queries or contains incomplete data doesn’t meet recovery objectives.

Teams can reduce this risk by automating post-restore validation. For example, after restoring a PostgreSQL database into a staging or isolated DR environment, a simple validation script can confirm connectivity and basic data integrity:

import psycopg2

import sys


def validate_restore():

    try:

        conn = psycopg2.connect(

            host="restored-db.internal",

            database="appdb",

            user="dr_test_user",

            password="securepassword"

        )

        cur = conn.cursor()

        cur.execute("SELECT COUNT(*) FROM users;")

        result = cur.fetchone()



        if result and result[0] > 0:

            print("Restore validation successful.")

        else:

            print("Restore validation failed: No data found.")

            sys.exit(1)


        conn.close()

    except Exception as e:

        print(f"Restore validation error: {e}")

        sys.exit(1)


validate_restore()

This script does three important things:

• Confirms the database is reachable

• Executes a real query, not just a connection check

• Fails explicitly if the expected data is missing

In practice, teams can integrate scripts like this into CI/CD pipelines or scheduled recovery drills. The goal isn’t to test every edge case, but to move from “backup exists” to “restore is functionally verified.” Over time, these automated checks become part of the disaster recovery test plan, helping teams measure RTO accurately and detect configuration drift before a real incident exposes it.

Disaster Recovery Test Scenarios: Practical Examples

Effective disaster recovery testing focuses on realistic failures, not idealized outages.

Accidental Deletion or Misconfiguration

A dropped database table, deleted storage bucket or bad configuration change tests how quickly teams can restore specific data without rolling back entire systems. These everyday incidents often reveal slow or overly manual recovery processes.

Data Corruption and Application Failure

Buggy releases can silently corrupt data while systems remain online. This scenario validates point-in-time recovery and whether teams can identify when corruption started, not just restore the latest backup.

Ransomware Simulation

Ransomware testing checks whether clean, uncompromised backups can be restored in isolation. It often exposes gaps in backup immutability, credential handling and realistic recovery times.

Infrastructure or Platform Outage

Simulating the loss of a cluster, availability zone or region tests automation and infrastructure-as-code maturity. In virtualized environments, most commonly VMware disaster recovery, testing involves restoring virtual machines at a secondary site and validating networking and application dependencies.

Credential and Access Failure

Recovery can stall if credentials, certificates or secret keys are unavailable. Testing this scenario validates identity systems and whether recovery procedures rely on fragile access assumptions.

Disaster Recovery Test Report: Turning Tests Into Improvements

Testing without documentation is wasted effort. A disaster recovery test report turns results into actionable improvements.

A valuable DR test report includes:

• Test scope and scenario

• Expected vs. actual RTO/RPO

• Recovery steps executed

• Failures, delays and root causes

• Recommended changes

For developers, this often results in concrete action items: refactoring startup dependencies, adding health checks, improving automation or adjusting data protection policies. The report should feed directly into backlog planning.

Disaster Recovery Audits and Continuous Validation

Audits often expose what teams already suspect: Disaster recovery plans exist, but haven’t been tested recently (or at all).

Rather than treating audits as one-time events, teams should adopt continuous validation:

• Regular restore tests integrated into CI/CD pipelines.

• Scheduled DR tests tied to major architecture changes.

• Automated alerts when recovery objectives drift.

This shifts disaster recovery testing from an annual obligation to an ongoing practice that evolves alongside the environment.

Conclusion

Disaster recovery testing is not about pessimism, it’s about realism. Systems and people change, and failure modes evolve faster than documentation. Without testing, even the best-designed recovery plan can become outdated.

For developers and technical teams, practicing disaster recovery testing builds confidence rooted in evidence, not assumptions. It exposes hidden dependencies, validates data protection strategies and ensures that when something goes wrong, recovery is predictable instead of chaotic.

A query that looks like a normal join can execute like a distributed system: rows move across the network, remote statements get executed repeatedly, and the local planner quietly becomes a coordinator. In that world, “fast SQL” is not mainly about CPU or indexes. It’s about data movement and round-trips.

This handbook covers the mechanism that determines whether a federated query behaves like a clean remote query or a chatty distributed workflow: pushdown.

Pushdown is not “moving compute”. Pushdown determines whether filtering, joining, ordering, and aggregation occur at the data source or after the data has already crossed the wire. When pushdown works, the local server receives a reduced result set. When it doesn’t, Postgres often has to fetch broad intermediate sets and finish the work locally.

The chapters ahead will help you build a practical mental model of what is “shippable” in postgres_fdw, why some expressions are blocked, and how to read EXPLAIN (ANALYZE, BUFFERS, VERBOSE) without getting tricked by familiar plan shapes.

After the core method, the handbook covers tuning knobs that matter in production, schema and indexing considerations, benchmarking methodology, monitoring and logging, and a case study that shows what a real pushdown win looks like end-to-end.

The later sections go deeper into advanced shippability edge cases, cost model calibration, and regression-proofing FDW workloads.

Table of Contents

• Prerequisites

• Executive Summary

• Motivation

• FDW Basics Without the Setup Tax

• Pushdown Mechanics

• Shippable Operations: a Deep Dive

• Pushdown Blockers and Why They Exist

• Reading EXPLAIN Like a Pro

• How to Tune postgres_fdw

• Schema and Index Recommendations

• Benchmarking Methodology

• Monitoring and Logging

• Case Study: Refactoring a Keycloak Coverage Query

• Checklist and Troubleshooting Guide

• Case Study Takeaways

• Advanced Operations: A Deeper Dive into Shippability

• Common Anti‑Patterns and How to Avoid Them

• Extending Tuning: Calibrating Cost Models

• Further Case Studies and Practical Examples

• Monitoring, Diagnostics, and Regression Testing

• Extended Guidelines for Advanced DBAs

• Bringing it All Together

• References

Prerequisites

This handbook assumes basic comfort with Postgres query plans. It builds on EXPLAIN (ANALYZE, BUFFERS) rather than reintroducing SQL fundamentals, indexing, or join algorithms.

The focus here is federated execution: how foreign queries behave, and how to reason about them with the same clarity as local plans.

Here’s what you should already be comfortable with:

• Reading EXPLAIN (ANALYZE, BUFFERS) output and spotting obvious plan smells (row explosions, bad join order, missed indexes).

• Basic join mechanics (nested loop, hash join, merge join) and why cardinality estimates matter.

• Postgres statistics at a practical level (ANALYZE, correlation, and what “estimated rows vs actual rows” implies).

And here’s what you need to follow along with the examples:

• A Postgres “local” instance that will run postgres_fdw and act as the coordinator.

• A Postgres “remote” instance that holds the foreign tables.

• Permission on the local side to:

  • CREATE EXTENSION postgres_fdw;

  • create a SERVER and USER MAPPING

  • create FOREIGN TABLE objects (or permission to use existing ones)

• A way to run queries and capture plans:

  • psql is enough, and so is any GUI, as long as you can run EXPLAIN (ANALYZE, BUFFERS, VERBOSE).

We won’t go through a long environment setup walkthrough. The examples assume the FDW objects exist and focus on plans and behavior.

We also won’t go into general distributed systems theory. Only the pieces that show up in an FDW plan are used.

Executive Summary

The single most important lesson of this handbook is that FDW pushdown reduces data movement. It’s tempting to think of pushdown as merely changing where a calculation happens (“move the work to the remote”). But what really matters is whether the remote server is asked for only the rows you need.

When pushdown is working, the remote server performs the selective join and filtering, and the local Postgres receives a small, already reduced result set. When pushdown fails, the local server becomes a distributed query coordinator: it pulls large intermediate sets over the network and then finishes the heavy lifting locally.

Why does this matter? Because a refactor that makes more of your query shippable to the remote server can slash end‑to‑end latency without changing a single row of output. In the case study we'll explore later, rewriting a query so that the FDW can ship a joined remote query instead of performing multiple foreign scans and local joins reduces runtime from approximately 166 ms to 25 ms. The business logic did not change – the shape of the work changed.

Below is a simple bar chart illustrating that dramatic drop. The chart uses actual timings from the case study. If you run the experiment yourself, the numbers may differ depending on your hardware and network, but the relative difference should be clear.

Motivation

Foreign data wrappers let you query remote data using the same SQL syntax you use locally. That convenience is exactly why they can be so deceptive.

A federated query may look like a normal join, but under the hood, it behaves like a distributed system: some part of the plan runs on the remote server, some on the local server, and every boundary between them is a network hop. The slow path is rarely “bad SQL” – it’s usually a combination of two things:

1. Too many rows are pulled over the network. Without pushdown, the FDW retrieves a large slice of the remote table and applies your filters and joins locally. This may lead to tens of thousands or millions of rows being shipped across the network when you only needed hundreds or fewer.

2. Too many round-trips. If the plan performs a nested loop that drives a foreign scan, it can end up executing the same remote query hundreds or thousands of times. Each call might be fast on its own, but latency adds up.

This isn't speculation. PostgreSQL's documentation makes clear that a foreign table has no local storage and that Postgres “asks the FDW to fetch data from the external source” [1]. There is no local buffer cache or heap storage to hide mistakes. Every row you retrieve must traverse the network at least once. If your plan fetches more rows than it needs, or repeatedly does so, performance can degrade quickly.

That’s why you should treat the Remote SQL shown in EXPLAIN (VERBOSE) as part of your query plan. It tells you exactly what the remote server is being asked to do. If it’s missing your filters or joins, you know the local server will have to finish the job. The rest of this handbook will teach you how to read that plan, how to force pushdown when possible, and how to recognize the signs that something has gone wrong.

FDW Basics Without the Setup Tax

You might be tempted to skip this section if you've already created foreign tables in your own databases. Don't. Understanding the architecture of foreign data wrappers is essential to understanding why pushdown matters.

SQL/MED in a nutshell

PostgreSQL implements the SQL/MED (Management of External Data) standard through its FDW framework. To access a remote Postgres server via postgres_fdw, you perform four steps:

1. Install the extension: CREATE EXTENSION postgres_fdw tells Postgres to load the FDW code.

2. Create a foreign server: CREATE SERVER foreign_server FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '...', port '...', dbname '...')defines where the remote server resides and how to connect.

3. Create a user mapping: CREATE USER MAPPING FOR your_user SERVER foreign_server OPTIONS (user 'remote_user', password '...') tells Postgres how to authenticate on the remote side.

4. Create a foreign table: CREATE FOREIGN TABLE remote_table (...) SERVER foreign_server OPTIONS (schema_name '...', table_name '...'); defines the columns and references the remote table.

Once you've done that, you can run SELECT statements against the foreign table as if it were local. But the definition hides an important detail: there is no storage associated with that foreign table [1]. Every time you SELECT, INSERT, UPDATE, or DELETE, the FDW must connect to the remote server, build a remote query, send it, and read the results. This overhead is small for simple queries but becomes critical as queries get more complex.

What postgres_fdw does and does not do

postgres_fdw does two things for you:

1. It builds remote SQL from your query, including pushing down safe filters, joins, sorts, and aggregates when it can.

2. It fetches rows from the remote server and hands them to the local executor. If some part of your query cannot be executed remotely, the local executor performs that part.

The FDW tries hard to minimize data transfer by sending as much of your WHERE clause as possible to the remote server and by not retrieving unused columns [2]. It also has a number of tuning knobs that we'll explore later (such as fetch_size, use_remote_estimate, fdw_startup_cost, and fdw_tuple_cost[3]). But the real win often comes from structuring your query so that the FDW can push work down.

There's one last architectural point to keep in mind: the remote server runs with a restricted session environment. In remote sessions opened by postgres_fdw, the search_path is set to pg_catalog only, and TimeZone, DateStyle, and IntervalStyle are set to specific values [4]. This means that any functions you expect to run remotely must be schema‑qualified or packaged in a way that the FDW can find them. It also underscores why you should not override session settings for FDW connections unless you know exactly what you are doing [4].

Pushdown Mechanics

At a high level, “pushdown” means pushing as much of your SQL query as possible to the remote server. But the FDW cannot simply send arbitrary SQL. It must be safe and portable for remote evaluation. Postgres uses the term shippable to describe expressions and operations that can be evaluated on the foreign server.

What “shippable” means in practice

An expression is considered shippable if it meets several conditions:

1. It uses built‑in functions, operators, or data types, or functions/operators from extensions that have been explicitly allow‑listed via the extensions option on the foreign server [2]. If you use a custom function or an extension that has not been declared, the FDW assumes it cannot run remotely.

2. It’s marked IMMUTABLE. Postgres distinguishes between IMMUTABLE, STABLE, and VOLATILE functions. Only immutable functions – those that always return the same output for the same inputs and don’t depend on session state – are candidates for pushdown [5]. This rule prevents time‑dependent functions, such as now() or random() from being evaluated remotely, because the result might differ between the local and remote servers.

3. It doesn’t depend on local collations or type conversions. PostgreSQL’s docs warn that type or collation mismatches can lead to semantic anomalies [1]. If the FDW cannot guarantee that a comparison behaves identically on both servers, it will refuse to push it down. For example, comparing a citext column to a text constant could be unsafe if the remote server doesn’t have the citext extension installed.

From these rules, you can derive a mental checklist: avoid non‑immutable functions in your WHERE clause, keep your join conditions simple and typed correctly, and list any third‑party extensions you want to use in the foreign server’s extensions option so that they are considered shippable [2].

WHERE pushdown

If a WHERE clause consists entirely of shippable expressions, it will be included in the remote query. Otherwise, it will be evaluated locally. This matters because pushing a filter down reduces the number of rows returned to the local server.

Consider a predicate like this:

WHERE created_at >= now() - interval '30 days'

Because now() is volatile (it returns a different value each time it’s called), Postgres cannot assume the remote server will interpret now() the same way. The FDW therefore pulls the entire table and applies the filter locally.

A better approach is to pass a parameter into the query or compute the cutoff timestamp once in the application and embed it into the SQL.

Join pushdown conditions

Joins are the next big lever. When postgres_fdw encounters a join between foreign tables on the same foreign server, it will send the entire join to the remote server unless it believes it will be more efficient to fetch the tables individually or unless the tables use different user mappings [6].

It applies the same precautions described for WHERE clauses: the join condition must be shippable, and both tables must be on the same server. Cross‑server joins are never pushed down – the FDW will perform them locally.

Shippability decision tree

It can be helpful to visualize the shippability rules as a flowchart. Below is a simple decision tree that you can use when inspecting an expression or join clause.

It starts with the question of whether an expression is in a WHERE or JOIN clause. Further decisions are made based on factors like using volatile functions, built-in functions, type mismatches, or cross-server joins. The flowchart concludes with outcomes like "Not shippable, evaluated locally" or "Shippable, included in Remote SQL."

If you reach the left side of the tree, the expression will be evaluated locally. If you reach the right side, the FDW can ship it.

Shippable Operations: a Deep Dive

Postgres has been expanding what postgres_fdw can be pushed down over several versions. This section walks through each operation class and the conditions required for pushdown.

Filters (WHERE clauses)

As explained above, simple filters that use built‑in operators and immutable functions are generally pushed down. If you see a Filter: node above a Foreign Scan in your plan, it means some part of your predicate didn’t qualify. Common reasons include using now(), timezone() or other volatile functions, referencing a non‑allow‑listed extension, or comparing different collation settings.

When this happens, the entire table (or at least all rows matching other shippable conditions) is fetched, and the filter is applied locally.

Plan smell: Look for a Foreign Scan node with a Filter: line directly above it. That means filtering happened locally. Also look for broad Remote SQL such as:

SELECT * FROM remote_table WHERE (name = 'Hamdaan')

with no group constraints. That's a sign that the filter was not pushed down.

Joins

Simple inner joins between foreign tables on the same foreign server are usually pushable. The join condition must satisfy the same shippability rules as filters. If the join involves more than one foreign server, if the join condition uses an unshippable function, or if the foreign tables use different user mappings, the FDW will fetch each table separately and join them locally [6]. This can lead to large intermediate sets being transferred.

Plan smell: A Hash Join or Merge Join where both inputs are Foreign Scan nodes indicates that the join was performed locally. Conversely, a single Foreign Scan representing a join and containing the JOIN ... ON clause in Remote SQL indicates that the join was pushed down.

Aggregates (GROUP BY, COUNT, SUM, and so on)

Starting in PostgreSQL 10, aggregates can be pushed to the remote server when possible. The release notes state explicitly: “push aggregate functions to the remote server,” and explain that this reduces the amount of data that must be transferred from the remote server and offloads aggregate computation [7].

To qualify, both the grouping expressions and the aggregate functions themselves must be shippable. If the FDW cannot push an aggregate, it will fetch the raw rows and perform the aggregation locally.

Plan smell: Look for a GroupAggregate node above a Foreign Scan that returns many rows. When the aggregate is pushed down, there will be no local aggregate node. Instead, the Remote SQL will include a GROUP BY clause.

ORDER BY and LIMIT

Prior to PostgreSQL 12, sorting and limiting were rarely pushed down. In version 12, Etsuro Fujita’s patch allows ORDER BY sorts and LIMIT clauses to be pushed to postgres_fdw foreign servers in more cases [8]. For the sort or limit to be pushed, the underlying scan must be pushable, and the ordering expression must be shippable. Partitioned queries or complicated join trees may still cause the sort or limit to be applied locally.

Plan smell: A local Sort or Limit node above a Foreign Scan indicates the operation was not pushed down. Conversely, a Remote SQL statement containing ORDER BY and LIMIT indicates that pushdown succeeded.

DISTINCT

Distinct operations can be pushed down when the distinct expression list is shippable. But if the distinct is combined with unshippable expressions, or if the distinct is applied after a join that cannot be pushed down, the FDW will retrieve all rows and perform the distinct locally.

Window functions

In practice, window functions are rarely pushed down through postgres_fdw. They often require ordering or partitioning semantics that are difficult to represent portably. If you see a WindowAgg node in your plan, it’s almost always local. That doesn’t mean you can't use window functions with foreign tables, but you should expect them to incur network and CPU costs.

Version differences

Postgres developers continue to improve the FDW layer. Here are some notable changes by version:

1. PostgreSQL 9.6 introduced remote join pushdown and allowed UPDATE/DELETE pushdown. Before 9.6, all joins were local.

2. PostgreSQL 10 introduced aggregate pushdown, enabling remote GROUP BY and aggregate functions [7].

3. PostgreSQL 12 expanded ORDER BY and LIMIT pushdown [8].

4. PostgreSQL 15 added pushdown for certain CASE expressions and other improvements.

If you learned FDW behavior on an older version, revisit your assumptions.

Pushdown Blockers and Why They Exist

When pushdown fails, it’s not due to bad luck. There’s always a reason grounded in safety or correctness. Here are the most common blockers and how to diagnose them.

Non‑immutable functions

Functions marked VOLATILE or STABLE cannot be pushed down because their results may differ between the local and remote server. Examples include now(), random(), current_user, and user‑defined functions that look at session variables or query the database. Even functions you might think are harmless, like age() or clock_timestamp(), can cause pushdown to fail.

Fix: Compute volatile values in your application or in a CTE before referencing the foreign table. For example, compute timestamp 'now' - interval '30 days' as a constant and compare your created_at column against that constant. Alternatively, move the logic into a stored generated column on the remote table.

Type and collation mismatches

The documentation warns that when types or collations don’t match between the local and remote tables, the remote server may interpret conditions differently [1]. This is particularly insidious when text comparisons, case‑insensitive collations, or non‑default locale settings are used. If Postgres can't guarantee the same semantics, it will pull rows locally and evaluate the expression.

Fix: Make sure that your foreign table definition uses the same data types and collations as the remote table. When in doubt, explicitly cast values to a common type.

Cross‑server joins

Joins across different foreign servers cannot be pushed down. The FDW can only ship a join when both tables reside on the same remote server and use the same user mapping [6]. Otherwise, it will perform two separate scans and join the results locally.

Fix: If you frequently join tables across servers, consider consolidating the tables on a single server, materializing a view on one side, or pulling the smaller table into a temporary local table before joining.

Mixed local and foreign joins

A join between a local table and a foreign table will not be pushed down. Even though the foreign side might be pushdown‑eligible, the FDW cannot join it with local data on the remote server. A nested loop with a parameterized foreign scan is the typical pattern here, resulting in many remote calls.

Fix: Filter or aggregate as much as possible on the foreign side first (via a CTE or by materializing a subset) before joining to local tables.

Remote session settings and search paths

Because postgres_fdw sets a restricted search_path, TimeZone, DateStyle, and IntervalStyle in remote sessions [4], any functions you call must be schema‑qualified or otherwise compatible. If a function relies on the current search path or session settings, it may break or produce different results on the remote side.

Fix: Schema‑qualify remote functions and ensure that any environment‑dependent logic is safe to execute under the default FDW session settings. If necessary, attach SET search_path or other settings to your remote functions.

Troubleshooting matrix

The table below maps symptoms in your EXPLAIN plan to likely causes and fixes. Use it as a quick diagnostic tool when something looks off.

Reading EXPLAIN Like a Pro

Many developers skim EXPLAIN plans for local queries, looking at the top nodes and overall cost. For FDW queries, you must invert that habit: read the foreign parts first. The Remote SQL string tells you what the remote server is being asked to do, and the loops field tells you how many times that remote call is executed.

Inspect the Foreign Scan nodes

Start by finding the Foreign Scan node(s). In EXPLAIN (VERBOSE), each foreign scan includes a line like:

Remote SQL: SELECT ...

This line is not a trivial – it’s the actual SQL that will run on the remote server. Read it carefully. Does it include your WHERE predicates? Does it include your join conditions? If not, you know the local server will pick up the slack.

Look at the loops column. If the loops exceed 1, the same remote query is executed multiple times. For example:

Foreign Scan on public.user_entity  (rows=1 loops=416)
  Remote SQL: SELECT id, tenant_id FROM public.user_entity WHERE enabled AND service_account_client_link IS NULL AND id = $1

This is the “N+1” problem in disguise. The plan executes the foreign scan once per outer row. Multiply the per‑loop cost by the number of loops to understand why the query is slow. The fix is to rewrite the query so that the join and filters are applied in a single remote call.

Recognize InitPlan vs SubPlan

An InitPlan runs once and caches its result. A SubPlan can run per outer row. In FDW queries, subplans often drive parameterized remote scans. If you see a SubPlan attached to a nested loop that feeds a foreign scan, suspect a parameterized remote lookup and look for ways to turn it into an InitPlan or merge it into a single remote query.

Understand CTE materialization

Common table expressions (CTEs) behave differently depending on whether they are marked MATERIALIZED or NOT MATERIALIZED. A materialized CTE is computed once and stored in a temporary structure, then read by the rest of the query. A non‑materialized CTE is inlined into the parent query, allowing optimizations to span across the boundary.

In PostgreSQL 12 and later, CTEs are inlined by default unless they’re referenced multiple times or explicitly marked MATERIALIZED. Materializing a CTE that contains a foreign scan can freeze a broad remote fetch and prevent later clauses from being pushed down. On the other hand, materialization can prevent repeated remote scans if the CTE is referenced multiple times. Use this lever deliberately to control where remote work happens.

Annotated example

Let's annotate a simplified excerpt from a real plan. The goal is to show how to quickly read the relevant parts.

Nested Loop  (rows=414 loops=1)
  -> Hash Join  (rows=416 loops=1)
       -> Foreign Scan on public.user_entity (rows=1 loops=416)
            Remote SQL: SELECT id, tenant_id FROM public.user_entity WHERE enabled AND service_account_client_link IS NULL AND id = $1
  -> Foreign Scan on public.user_attribute (rows=671 loops=1)
       Remote SQL: SELECT ua.user_id, ua.value FROM user_attribute ua JOIN user_entity u ON ua.user_id = u.id JOIN tenant r ON u.tenant_id = r.id WHERE ua.name = 'attribute A' AND r.name = 'demo' AND u.enabled AND u.service_account_client_link IS NULL AND (g.name = 'keycloak-group-a' OR g.parent_group = $1)

In the old plan, the first Foreign Scan executed 416 times, each time retrieving a single row. The Remote SQL only applies the filter on enabled and service_account_client_link – it doesn’t include the tenant or group scoping. That scoping is applied by the nested loop outside the foreign scan.

In the refactored plan, the second Foreign Scan results from combining user_attribute, user_entity, user_group_membership, keycloak_group, and tenant into a single remote query. It retrieves 671 rows in a single query and includes all relevant filters. There is no repeated remote call. The timing difference is driven by the different loop values and the selectivity of the Remote SQL.

How to Tune postgres_fdw

Once you've structured your query for maximum pushdown, tuning knobs let you squeeze out further performance improvements and adjust planner decisions.

fetch_size

fetch_size controls how many rows postgres_fdw retrieves per network fetch. The default is 100 rows [9]. A small fetch size means more round-trips and lower memory usage. A larger fetch size reduces network overhead at the cost of buffering more rows in memory.

In practice, increasing fetch_size to a few thousand can reduce latency for large result sets. It’s specified either at the foreign server or foreign table level:

ALTER SERVER foreign_server OPTIONS (ADD fetch_size '1000');
ALTER FOREIGN TABLE remote_table OPTIONS (ADD fetch_size '1000');

use_remote_estimate

By default, the planner estimates the cost of foreign scans using local statistics. This can be wildly inaccurate if the foreign table has a different data distribution. Setting use_remote_estimate to true tells postgres_fdw to run EXPLAIN on the remote server to get row count and cost estimates. This can dramatically improve join order selection at the cost of an additional remote query during planning [3]. You can set this per table or per server:

ALTER SERVER foreign_server OPTIONS (SET use_remote_estimate 'true');

fdw_startup_cost and fdw_tuple_cost

These cost parameters model the overhead of starting a foreign scan and the cost per row fetched. Adjusting them can influence the planner’s choice of join strategy. A higher fdw_startup_cost discourages the planner from choosing plans with many small foreign scans (which might generate many remote calls). A higher fdw_tuple_cost discourages plans that fetch large numbers of rows [3]. Use these only after you have solid evidence from EXPLAIN and experiments.

ANALYZE and analyze_sampling

Running ANALYZE on a foreign table collects local statistics by sampling the remote table [3]. Accurate stats are essential for good estimates when use_remote_estimate is false.

But if the remote table changes frequently, these stats become stale quickly. The analyze_sampling option controls whether sampling happens on the remote side or locally. When analyze_sampling is set to random, system, bernoulli, or auto, ANALYZE will sample rows remotely instead of pulling all rows into the local server[3].

extensions

The extensions option lists extensions whose functions and operators can be shipped to the remote server [2]. If you rely on functions from citext, pg_trgm, or other extensions, add them to the server definition:

ALTER SERVER foreign_server OPTIONS (SET extensions 'citext,pg_trgm');

A quick knob impact table

Schema and Index Recommendations

Pushdown doesn’t eliminate the need for good indexes. In fact, effective pushdown depends on the remote server having indexes that support the filter and join predicates you’re shipping.

Below are some patterns to watch for in FDW queries and the indexes that support them. You can adapt these to your own schema.

In general, indexes should reflect the join order you expect the remote planner to use. If your Remote SQL starts with:

FROM user_attribute ua JOIN user_entity u ON ua.user_id = u.id JOIN user_group_membership ugm ON ...

ensure that indexes exist on user_attribute(user_id) and user_group_membership(user_id).

Benchmarking Methodology

It’s easy to claim a performance improvement without proper measurement. Here's a repeatable method you can use to benchmark FDW query changes.

1. Warm the caches. Run each query once to load data into the remote buffer cache and the local FDW connection. Discard the timings.

2. Measure latencies. Use EXPLAIN (ANALYZE, BUFFERS, VERBOSE) to capture execution times, buffer usage, and remote row counts. Be aware that EXPLAIN ANALYZE adds overhead, so record the raw execution time if possible by running the query directly.

3. Record remote metrics. On the remote server, enable pg_stat_statements and track the calls, total_time, and rows for each remote query. This gives you a per‑query breakdown and confirms what Remote SQL is executed.

4. Control for concurrency and network latency. Run benchmarks during a quiet period or isolate the test cluster. If your environment has high network latency, record the round‑trip time separately to attribute delays.

5. Compare apples to apples. Benchmark the old and new queries under identical conditions. Use the same sample data, same remote server, and same connection settings.

6. Look at row counts. The primary goal of pushdown is to reduce the number of rows shipped. Compare the rows column of each Foreign Scan node.

Here's a simple matrix you can use to record your experiments:

Monitoring and Logging

In production, you need to know when a query starts misbehaving. There are two places to look: the local server and the remote server.

Local metrics

1. pg_stat_statements. This extension tracks planning and execution times, row counts, and buffer hits for each query. Look for high total times relative to rows or calls.

2. Auto Explain or auto_explain. Turn on auto_explain.log_min_duration_statement to capture slow queries with plans. This will show you the Remote SQL executed and whether the plan changed.

3. Connection pool metrics. Monitor connection counts and wait events related to FDW operations (for example, PostgresFdwConnect, PostgresFdwGetResult) as described in the documentation [10].

Remote metrics

1. pg_stat_statements on the remote server. This lets you see which Remote SQL queries are being executed, how often, and how long they take. Compare these with the Remote SQL strings in your local EXPLAIN plans.

2. Server logs. Increase log_statement or log_min_duration_statement on the remote server to capture long-running remote queries.

Correlating local and remote metrics can reveal patterns such as a new code path causing a surge in remote queries or pushdown failures, leading to heavy remote scans.

Case Study: Refactoring a Keycloak Coverage Query

The theory above may seem abstract until you see it play out in practice. Let's walk through a real example inspired by a Keycloak integration.

The original query calculated coverage: given a list of category IDs, it returned the percentage of users who had attributes mapped to those categories and a JSON array of entity counts. The query used a CTE to build a list of scoped users, then joined it with user attributes, category mappings, and a few other tables.

Symptom

In a test environment with 100K user records, the query averaged 166 ms. This was slower than expected. Running EXPLAIN (ANALYZE, BUFFERS, VERBOSE) showed two foreign scans on the Keycloak database. The first scanned user_entity 416 times (loops = 416). The second pulled all rows from user_attribute where name = 'attributeA' before filtering by tenant and group locally.

Here's a simplified excerpt (numbers are approximate):

Foreign Scan on public.user_entity  (actual time=0.117..0.117 rows=1 loops=416)
  Remote SQL: SELECT id, tenant_id FROM public.user_entity WHERE (enabled AND service_account_client_link IS NULL AND id = $1)
Foreign Scan on public.user_attribute  (actual time=41.267..80.352 rows=80739 loops=1)
  Remote SQL: SELECT value, user_id FROM public.user_attribute WHERE (('attributeA' = name))

The first scan performed a single-row lookup 416 times. The second scan retrieved 80,739 rows because the only condition pushed down was name = 'attributeA'. Tenant and group scoping occurred locally. That meant 80k rows were transferred over the network and then filtered down to about 671 on the local side.

Diagnosis

There were two main issues.

First was the N+1 remote calls on user_entity. The join to user_entity was not pushed down, so the plan executed a remote lookup for each row from user_group_membership. This created 416 remote queries.

Second was the unscoped attribute fetch. Because the WHERE clause included user_entity.tenant_id = tenant.id and keycloak_group.name = 'groupA' in a higher CTE, the FDW could not see those predicates when scanning user_attribute. It therefore fetched all rows with name = 'attributeA' and left the tenant and group filters to the local side.

Refactor

The fix was to inline the tenant and group joins into the user_attribute scan to avoid the nested-loop pattern. The refactored selected_user_attributes CTE looked like this (simplified for readability):

WITH selected_user_attributes AS (
  SELECT DISTINCT ua.user_id, ua.value
  FROM public.user_attribute ua
  JOIN public.user_entity u ON u.id = ua.user_id
  JOIN public.user_group_membership ugm ON ugm.user_id = u.id
  JOIN public.keycloak_group g ON g.id = ugm.group_id
  JOIN public.tenant r ON r.id = u.tenant_id
  WHERE ua.name = 'attributeA'
    AND u.enabled
    AND u.service_account_client_link IS NULL
    AND r.name = 'tenantA'
    AND (g.name = 'groupA' OR g.parent_group = (
         SELECT id FROM public.keycloak_group WHERE name = 'groupA' AND tenant_id= r.id
    ))
)

This single query expresses the same scoping logic that previously lived in separate CTEs. Because all the join conditions are on the same foreign server and use built‑in operators, the FDW can push down the entire join. The new plan looked like this:

Foreign Scan  (actual time=7.840..7.856 rows=671 loops=1)
  Remote SQL: SELECT ua.user_id, ua.value FROM user_attribute ua JOIN user_entity u ON ua.user_id = u.id JOIN user_group_membership ugm ON ugm.user_id = u.id JOIN keycloak_group g ON g.id = ugm.group_id JOIN tenant r ON u.tenant_id= r.id WHERE ua.name = 'attributeA' AND u.enabled AND u.service_account_client_link IS NULL AND r.name = 'tenantA' AND (g.name = 'groupA' OR g.parent_group = $1)

Only one remote query is executed, and it returns 671 rows. Tenant and group scoping occur on the remote server. There is no nested loop or repeated remote scan. The final runtime dropped to about 25 ms.

Why it improved

1. Fewer rows crossing the network. The old plan fetched 80k attribute rows and filtered them locally. The new plan fetched only the 671 scoped rows.

2. No repeated remote calls. The old plan executed 416 remote scans of user_entity. The new plan performs one joined remote query.

3. Less local work. Because the join and filtering happen remotely, the local side no longer hashes or filters large sets.

Key takeaway

If you see a Foreign Scan with a high loops count or a Remote SQL that doesn’t contain your filters and joins, you’re leaving performance on the table. Merging filters and joins into a single remote query (subject to shippability rules) often yields orders-of-magnitude improvements.

Checklist and Troubleshooting Guide

The following steps summarize how to approach FDW performance tuning:

1. Inspect the Remote SQL. Always run EXPLAIN (VERBOSE) and look at what is being sent to the remote. If your predicates are missing, the FDW isn't pushing them down.

2. Check loops. If the loops are greater than 1 on a Foreign Scan, you are paying for repeated remote calls. Rewrite the query or reorder the joins to make the foreign scan run once.

3. Make predicates shippable. Replace volatile functions with constants or parameters. Ensure operators and functions are built‑in or explicitly allow‑listed via the extensions option [2].

4. Align types and collations. Use the same data types and collations on both sides to avoid semantic mismatches [1].

5. Push joins to the same server. Consolidate tables on one foreign server if possible. Joins across servers cannot be pushed down [6].

6. Use use_remote_estimate when planning seems off. Enabling remote estimates can improve join order selection [3].

7. Tune fetch_size and costs if your queries transfer many rows. A bigger fetch_size reduces round-trip; adjusting fdw_startup_cost and fdw_tuple_cost influences the planner [3].

8. Analyze foreign tables if you rely on local cost estimates. Keep in mind that stats can get stale quickly [3].

9. Monitor both servers. Use pg_stat_statements on local and remote servers to see how often remote queries run and how long they take.

10. Test version upgrades. Each major release improves FDW pushdown semantics (for example, aggregates in 10 [7], ORDER/LIMIT in 12 [8]). Retest after upgrading.

Case Study Takeaways

Querying remote data with PostgreSQL’s postgres_fdw can be fast and convenient if you respect the underlying mechanics. Pushdown is the difference between streaming a trickle of relevant rows and hauling an ocean of data across the network. It isn't simply a matter of moving CPU cycles – it changes how much data moves, how many network round-trip occur, and how much your local server has to do.

The rules may seem restrictive – use only immutable functions, avoid cross‑server joins, align types and collations – but they exist to preserve correctness while enabling optimization.

By reading EXPLAIN from the bottom up, inspecting the Remote SQL, and understanding the shippability rules, you can spot slow patterns quickly. Armed with tuning knobs like fetch_size and use_remote_estimate, and a willingness to rewrite queries to make joins and filters pushable, you can often achieve dramatic performance gains without touching your hardware.

This case study shows that rewriting a query to enable a single-joined remote query reduced runtime from around 166 ms to 25 ms. That sort of improvement is not rare. It’s what happens when you treat FDW queries as distributed queries rather than local queries in disguise.

The next time you debug a slow FDW query, remember this handbook. Check the Remote SQL. Count the loops. Ask yourself: “Am I doing the work close to the data, or am I bringing the data to the work?” Adjust accordingly, and you'll write queries that make the most of Postgres's federated capabilities while keeping your latency in check.

This section closes the case study loop and summarizes exactly what changed in the plan and why it produced a large end-to-end win. The following sections of the handbook turn that single win into a repeatable method: how Postgres determines what is shippable, how to quickly read FDW plans, which operations and versions matter, and how to debug common failure modes that prevent pushdown.

Advanced Operations: A Deeper Dive into Shippability

The previous sections introduced the basic rules around what can be pushed to the remote and why. To really make sense of those rules, you need to see how they play out on the operations you use every day.

This section walks through filters, joins, aggregates, ordering, and limits, DISTINCT queries, and window functions in more detail. By the end, you should have a mental map of which operations to trust and which to double‑check when reading your plans.

Filters and simple predicates

WHERE clauses matter more than you think

When you specify WHERE attribute = 'value' on a foreign table, the FDW will happily transmit that predicate to the remote server as long as the comparison uses built‑in types and immutable operators. For example:

• WHERE id = 42 is fine

• WHERE lower(username) = 'hamdaan' is fine if lower() is allow‑listed and immutable

• WHERE created_at >= now() - interval '7 days' is not shippable because now() is volatile

When such a predicate cannot be pushed, the FDW will fetch every row that matches all the shippable predicates and apply the rest locally. That means that a seemingly innocuous call to now() can blow up your network traffic.

The lesson is simple: compute volatile values up front (in your application or in a CTE) and reference them as constants in the query against the foreign table.

Complex expressions are not automatically unsafe

Suppose you have WHERE (status = 'active' AND (age BETWEEN 18 AND 29 OR age > 65)). This entire expression is shippable because it uses built‑in boolean logic, simple comparisons, and immutable operators. The FDW will deparse it into remote SQL and forward it. You only need to worry when one of the subexpressions introduces a function or operator that the FDW doesn’t recognize or cannot safely assume exists on the remote.

A good heuristic is: if you can express your filter using only simple comparisons, boolean logic, and built‑in functions, pushdown should work. When in doubt, check the Remote SQL.

Array and JSON operators

Modern Postgres makes heavy use of array and JSON functions. Many of these functions, like the array overlap operator && used in the case study, are built‑in and can be shipped. But some JSON functions are provided by extensions (like jsonb_path_query or functions from the pgjson family).

If your filter uses one of these, ensure that the extension is available and allow‑listed on the foreign server. Otherwise, the FDW will fetch rows and perform the JSON logic locally. This is rarely what you want when dealing with large JSON columns.

Joins: the good, the bad, and the ugly

Same‑server joins are your friend

If you join multiple foreign tables that are all defined on the same foreign server and user mapping, and if the join condition uses only shippable expressions, then the FDW can generate a single remote join. This is the ideal case.

For example, joining orders and customers on orders.customer_id = customers.id is pushable, as long as both tables reside on the same foreign server. The remote planner will use its own statistics and indexes to plan the join, and the local server will simply iterate through the result. Postgres 9.6 and later support this pattern [6].

Cross‑server joins break pushdown

If you attempt to join two foreign tables that live on different servers (or even on the same remote server but with different user mappings), postgres_fdw will fetch the tables separately and join them locally. This is almost always slower than pushing the join down, because you end up transferring both tables in their entirety.

The FDW design team chose not to support cross‑server joins because there is no portable way to tell two remote servers to cooperate on a join. Your options are: replicate one table on the other server, materialize the smaller table locally before joining, or restructure the query to filter aggressively on each side before joining locally.

Mixed local/foreign joins are tricky

Joining a local table to a foreign table cannot be pushed down, for straightforward reasons: the remote server has no access to your local data. A common pattern that triggers repeated remote calls looks like this:

SELECT u.id, a.value
FROM users u
LEFT JOIN user_attribute a
  ON a.user_id = u.id AND a.name = 'favorite_color';

If users is a local table and user_attribute is foreign, the plan may use a nested loop: for each local u, it executes a remote lookup in user_attribute to retrieve attributes.

The fix is to flip the query: retrieve all relevant rows from user_attribute in one remote scan, then join them locally. Or, if possible, create a small temporary table on the remote side with your u.id values, perform the join entirely remotely, and then fetch the results.

Join conditions matter

Even when joining two foreign tables on the same server, an unshippable join condition will force the join to be local. For example, JOIN ON textcol ILIKE '%foo%' is not pushable because ILIKE might not exist or behave identically on the remote.

If you need case‑insensitive matching, consider lowercasing both sides: LOWER(textcol) = 'foo' (assuming the remote server has the lower() function available and allowed). Similarly, joining on a cast expression (for example, JOIN ON CAST(a.id AS text) = b.text_id) can block pushdown. Define your columns with matching types instead.

Aggregates and grouping

Aggregates are where the data movement story shines. When you can push down a GROUP BY and aggregate functions like COUNT, SUM, AVG, or MAX, you reduce the result set to just the aggregated rows. This can be a difference of several orders of magnitude.

Postgres 10 introduced aggregate pushdown [7]. But not all aggregates are equal:

Simple aggregates such as COUNT(*), SUM(col), AVG(col), MIN(col), and MAX(col) are shippable when applied to shippable expressions. Even COUNT(DISTINCT col) is often shippable, because the remote can deduplicate before counting. The FDW will wrap the aggregate in a remote query and return just the aggregated row.

If you see a GroupAggregate node on the local side, check whether all involved columns and functions are shippable. If they are, ensure that the join conditions above are also pushable.

Filtered aggregates such as COUNT(*) FILTER (WHERE x > 5) or SUM(col) FILTER (WHERE status = 'active') are often pushable, because they translate into SUM(CASE WHEN condition THEN col ELSE 0 END) or COUNT(...). As long as the filter is shippable, the FDW will push it into the remote aggregate.

User‑defined aggregates are rarely pushable. If you have a custom aggregate function, the FDW will not assume that it exists or behaves the same on the remote server. Even if you install the function on both servers, postgres_fdw won't push it unless the function is in an allow‑listed extension.

Grouping sets and rollups are not currently pushable. When you write GROUP BY GROUPING SETS (...) or ROLLUP(...), Postgres will compute the grouping locally even if the underlying scan is remote.

If you need complex rollups, consider performing them in two steps: push down the initial grouping to the remote server to reduce rows, then perform the rollup locally.

ORDER BY, LIMIT, and DISTINCT

Ordering and limiting rows may seem like purely cosmetic features, but they affect how much data is transferred. If the remote can sort and limit, the local server only receives the top N rows. If it cannot, the local server must sort everything.

Postgres 12 expanded the cases where ORDER BY and LIMIT are pushed down [8]. Here are guidelines:

• Single foreign scan with simple sort: If your query selects from one foreign table and sorts by a shippable expression (for example, ORDER BY created_at DESC), the FDW will include ORDER BY in Remote SQL. It will also push down LIMIT and OFFSET. This is ideal because the remote server does the sort and sends only the top rows.

• Sort after join: If you sort after joining two foreign tables on the same server, and the join and sort expressions are shippable, the FDW may push both down. But if the sort requires columns from the local side or from a different remote server, the FDW cannot push it down.

• Sort after aggregation: Sorting aggregated results is often pushable as long as the aggregate itself is pushable. But when grouping occurs locally, the sort remains local.

• DISTINCT behaves like GROUP BY. If the distinct expression list is shippable, the FDW can push it down. If you write SELECT DISTINCT ON (col1) col2, col3 FROM ... and col3 is not part of the DISTINCT list, Postgres will treat this as GROUP BY and may push it. Be aware that DISTINCT ON semantics differ from plain DISTINCT and may not be pushable in older Postgres versions.

Window functions

Window functions (for example, ROW_NUMBER() OVER (PARTITION BY ...), RANK(), LAG(), LEAD()) rely on ordering and partitioning across rows.

Postgres has not yet taught postgres_fdw how to push window functions. When you see a WindowAgg node in your plan, it’s almost always local. The FDW will fetch the rows, and the local server will sort, partition, and compute the window. If you need to run window functions on remote data, plan to transfer the data locally.

Version‑specific quirks

The exact pushdown capabilities vary by release. When planning migrations or deciding whether to rely on a pushdown behavior, check the release notes:

• 9.6: first version to support pushdown of joins and sorts, and remote updates and deletes.

• 10: introduced aggregate pushdown [7], significantly reducing network use for GROUP BY queries.

• 11: improved partition pruning and join ordering for foreign tables.

• 12: expanded ORDER BY and LIMIT pushdown [8].

• 15: added pushdown for simple CASE expressions and additional built‑in functions.

• 17 (development at the time of writing) continues to expand shippable constructs. Always test on your target version because subtle improvements can change what the FDW can ship.

Common Anti‑Patterns and How to Avoid Them

Everyone has run into FDW queries that seemed reasonable but turned out to be bottlenecks. Here are a few of the most common mistakes and how to correct them. These examples are deliberately simplified – so you can adapt them to your schema.

Using volatile functions in predicates

Anti‑pattern:

SELECT *
FROM audit_logs
WHERE event_ts >= now() - interval '1 day';

now() is a volatile function, so the FDW refuses to push this predicate. It pulls all rows from audit_logs and filters them locally.

Better:

SELECT *
FROM audit_logs
WHERE event_ts >= $1;

Compute $1 (a timestamp) in your application or upstream query. Or compute it once in a CTE:

WITH cutoff AS (SELECT now() - interval '1 day' AS ts) SELECT * FROM audit_logs, cutoff WHERE event_ts >= cutoff.ts;

The FDW sees a constant and pushes the predicate.

Joining local and foreign data first

Anti‑pattern:

SELECT u.email, ua.value
FROM users u
LEFT JOIN user_attribute ua ON u.id = ua.user_id AND ua.name = 'favorite_movie';

This uses a local table (users) to drive a join to a foreign table (user_attribute). The FDW receives 10,000 individual remote queries if users have 10,000 rows. Each call fetches one or zero rows from user_attribute.

Better:

-- Fetch all favorite movies remotely and join locally
WITH remote_movies AS (
  SELECT ua.user_id, ua.value
  FROM user_attribute ua
  WHERE ua.name = 'favorite_movie'
)
SELECT u.email, rm.value
FROM users u
LEFT JOIN remote_movies rm ON u.id = rm.user_id;

Now the FDW issues one query to fetch all relevant attributes, and the join is done locally in one pass.

Cross‑server joins without materialization

Anti‑pattern:

SELECT *
FROM remote_db1.orders o
JOIN remote_db2.customers c ON o.customer_id = c.id;

This is not pushable because the two tables are on different foreign servers. Postgres will fetch orders and customers separately and join them locally. If orders have 1 million rows and customers have 50,000 rows, you will transfer 1.05 million rows.

Better: Replicate or materialize one side on the other server (or locally) before joining. For example, create a materialized view m_customers on remote_db1 containing just the id and name of the customers you need, then join orders and m_customers on the same server. Alternatively, copy customers into a temporary table on the local server and join there.

Complex expressions on join keys

Anti‑pattern:

SELECT *
FROM remote_table a
JOIN remote_table b ON CAST(a.key AS text) = b.key_text;

Casting a numeric key to text prevents pushdown. The remote server cannot use indexes and must return both tables. The local server performs the join and cast.

Better: Align your schemas so that the join columns use the same type. If you cannot change the schema, create a computed column on the remote server with the appropriate type and use it in the join.

Ignoring collation and type mismatches

Anti‑pattern:

SELECT *
FROM remote_table
WHERE citext_col = 'abc';

If the remote server doesn’t have the citext extension installed, the comparison semantics will differ, and the FDW will refuse to ship the filter. This appears harmless until you see the plan and realize all rows were fetched.

Better: Install the same extensions and collations on the remote server, or convert the column to a base type like text on both sides.

Extending Tuning: Calibrating Cost Models

Earlier, we discussed fetch_size, use_remote_estimate, and the cost knobs. This section expands on how to use them strategically.

Balancing fetch size and memory

fetch_size controls how many rows the FDW asks for in each round trip [9]. Think of it as the batch size. The default (100) works well for small result sets. If you expect to retrieve tens of thousands of rows, a higher fetch size reduces the overhead of many network requests. But there are trade‑offs:

• Memory consumption: Each foreign scan buffers rows until they are consumed. A huge fetch size (for example, 10,000) may allocate more memory than you expect, especially when multiple scans run concurrently. Monitor memory usage as you increase this setting.

• Latency hiding: If network latency is high, overlapping network requests with local processing can hide some latency. But postgres_fdw does not pipeline multiple fetches – it waits for one batch before requesting the next. This means that a larger batch size reduces the number of waits, but cannot overlap them. If you operate across data centers, consider using a connection pooler or caching layer instead of just increasing fetch_size.

Remote estimates vs. local estimates

The planner uses statistics to estimate how many rows each node will produce, which in turn influences join order. When use_remote_estimate is false (the default), the planner guesses based on local stats collected by ANALYZE on the foreign table. This can be wrong if the remote table has a different distribution than the local sample, or if the table has changed since the last ANALYZE.

Setting use_remote_estimate to true instructs the FDW to run EXPLAIN on the remote server during planning to obtain row counts and cost estimates [3]. This can improve join ordering, especially when joining multiple foreign tables or mixing local and foreign tables. The downside is increased planning time because each remote estimate runs an extra query.

In practice:

• Enable use_remote_estimate on queries with complex joins where the planner picks obviously wrong join orders. If enabling it improves the plan, consider leaving it on for that server or table.

• Use ANALYZE on foreign tables periodically if your remote data is relatively static. This populates local stats and can avoid the overhead of remote estimates.

• Don’t enable use_remote_estimate indiscriminately on simple lookups. The cost of additional round-trip remote flights may outweigh the benefit.

Tuning cost parameters

fdw_startup_cost and fdw_tuple_cost control how much the planner thinks it costs to start a foreign scan and fetch each row [3]. If these are too low, the planner may choose a nested loop that generates many small remote calls. If they are too high, the planner might avoid remote scans even when they are efficient.

You can adjust these parameters based on empirical measurement:

• Increase fdw_startup_cost to discourage the planner from using nested loops that call the remote table repeatedly. You might set it to the average cost of a round-trip remote.

• Increase fdw_tuple_cost if network bandwidth is limited or expensive. This indicates to the planner that each remote row incurs higher fetch costs than a local row. The planner will prefer plans that filter early on the remote side.

Always adjust these settings gradually and observe the effect on the plan. Keep separate settings per foreign server if network conditions differ.

When to analyze foreign tables

Running ANALYZE on a foreign table collects sample statistics by pulling a subset of rows from the remote server. This helps the planner estimate row counts when use_remote_estimate is off. It also helps decide whether to use an index on the remote side. You should analyze foreign tables when:

• The remote table is large and static, and you want accurate local estimates without the overhead of remote estimates.

• You have just defined a foreign table, and the default stats are empty.

• You changed the extensions allow‑list to enable more pushdown and want the planner to see the effect.

Conversely, if the remote data changes constantly, ANALYZE results will quickly become stale. In that case, rely on use_remote_estimate instead.

Further Case Studies and Practical Examples

The Keycloak coverage example is not the only place where pushdown matters. The following scenarios illustrate other patterns you may encounter.

Reporting on a sharded logging system

Imagine you store application logs across multiple shards, each a separate Postgres database. You want to produce a report of the number of error logs per service per day.

A naïve approach might join all shards in one query:

SELECT shard, service, date_trunc('day', log_time) AS day, COUNT(*)
FROM shard1.logs
UNION ALL
SELECT shard, service, date_trunc('day', log_time) AS day, COUNT(*)
FROM shard2.logs
...;

This approach will fetch all log rows to the local server and aggregate them locally. A better solution is to push the grouping to each shard:

SELECT shard, service, day, sum(count)
FROM (
  SELECT 1 AS shard, service, date_trunc('day', log_time) AS day, COUNT(*) AS count
  FROM shard1.logs
  WHERE log_time >= $1 AND log_time < $2
  GROUP BY service, day
  UNION ALL
  SELECT 2 AS shard, service, date_trunc('day', log_time) AS day, COUNT(*)
  FROM shard2.logs
  WHERE log_time >= $1 AND log_time < $2
  GROUP BY service, day
  ...
) x
GROUP BY shard, service, day;

Here, each foreign server returns a small set of aggregated rows instead of raw logs. The outer aggregation sums across shards. This pattern generalizes: push grouping and filtering to the remote side, then combine locally.

Combining remote and local data for analytics

Suppose you have a local table users and a remote table orders. You want to compute the average order amount per user segment. A naïve query might look like:

SELECT u.segment, AVG(o.amount)
FROM users u
JOIN orders o ON o.user_id = u.id
GROUP BY u.segment;

This is a local join driving a remote nested loop. The better approach is to aggregate orders remotely by user_id and join on the small result:

WITH remote_totals AS (
  SELECT user_id, SUM(amount) AS total, COUNT(*) AS n
  FROM orders
  GROUP BY user_id
)
SELECT u.segment, AVG(rt.total / rt.n)
FROM users u
JOIN remote_totals rt ON u.id = rt.user_id
GROUP BY u.segment;

This pushes the heavy aggregation to the remote and transfers only one row per user. The local join then groups by segment. As with other examples, the key is to reduce remote rows before they cross the network.

Avoiding pushdown for correctness

There are legitimate cases where you should prevent pushdown because of semantic differences. Postgres allows you to do this by adding OFFSET 0 or wrapping the foreign table in a CTE.

For example, if a built‑in function behaves differently on the remote due to a version mismatch, you can force local evaluation:

WITH local_eval AS (SELECT  FROM remote_table)  -- CTE prevents pushdown
SELECT 
FROM local_eval
WHERE some_complex_expression(local_eval.col) > 0;

Alternatively, a WHERE clause like random() < 0.1 will not push down because random() is volatile – you don't need to force it. But adding OFFSET 0 is a simple hack that prevents any pushdown:

SELECT * FROM remote_table OFFSET 0;

Knowing how to disable pushdown intentionally helps you debug. If a query returns different results when pushdown occurs, suspect type/collation mismatches or remote session settings [4].

Monitoring, Diagnostics, and Regression Testing

Monitoring doesn't end at counting remote rows. To make pushdown reliable in production, you need to set up mechanisms to detect regressions and gather evidence when performance changes.

Automate EXPLAIN regression tests

In addition to unit tests and integration tests, you can add tests that assert the shape of your plans. For instance, if a mission‑critical report must always push down a WHERE clause, you can write a test that runs EXPLAIN (VERBOSE) and checks that the Remote SQL contains the filter. You might even parse loops and assert that it is 1. When a developer inadvertently adds a non‑immutable function or changes a join, the test will fail. This is akin to snapshot testing for SQL.

Monitor pg_stat_statements across servers

Enable pg_stat_statements on both the local and remote servers. On the local side, track the total time, planning time, and rows for each FDW query. On the remote side, track which queries are being executed.

Look for outliers: a query whose remote calls spike or whose average remote rows jump from hundreds to thousands. Those are early signs of pushdown failure.

Log remote SQL with auto_explain

Setting auto_explain.log_min_duration_statement (for example, to 500ms) causes Postgres to automatically log slow queries with their plans. Combine this with auto_explain.log_verbose = true and auto_explain.log_nested_statements = true to capture remote SQL as well. When a federated query slows down, the log will show you exactly what remote SQL was executed and how often. This is invaluable in production, where you cannot always run EXPLAIN interactively.

Use connection pooling and prepare statements

postgres_fdw maintains a connection pool keyed on the user mapping. It reuses connections between queries, but you can also use connection pooling at the network level (for example, pgbouncer or pgcat).

Keeping connections warm reduces the startup cost, as captured by fdw_startup_cost. Meanwhile, preparing statements on the remote server (via PREPARE and EXECUTE) can save parse time when the same remote SQL is executed frequently. postgres_fdw can use server‑side prepared statements for parameterized scans.

Regression testing after version upgrades

Every major Postgres release brings improvements to postgres_fdw pushdown semantics. But new releases also change planner heuristics and remote SQL generation. After an upgrade, rerun your key queries with EXPLAIN (VERBOSE), compare the Remote SQL, and benchmark them.

In some cases, a release may push down something previously local, revealing a latent type mismatch or a function difference. In other cases, pushdown may be withheld due to a new rule. Don’t assume that an upgrade automatically improves performance – test it.

Extended Guidelines for Advanced DBAs

To close this handbook, here are consolidated guidelines distilled from the previous sections. They go beyond simple bullet points to capture nuances. Keep them handy for reference or print them out for your team.

1. Respect the FDW safety model. Immutable functions and built‑in operators are your friends. Anything outside that scope must be explicitly allowed or evaluated locally. Understand which items belong to each category and plan accordingly.

2. Always read the Remote SQL. Don’t trust your intuition about what is being pushed down. The Remote SQL string is the only source of truth. It indicates whether a predicate, join, sort, or limit operation is occurring remotely. It also shows parameter placeholders (for example, $1) that correspond to values passed from the local plan.

3. Reduce before you fetch. The network is the highest cost. If the remote can reduce rows through filtering, grouping, or limiting, let it. If it cannot, structure your query to enable it. Avoid queries that require pulling large raw tables and processing them locally.

4. Beware of join order. The planner sometimes chooses a nested loop with a foreign table as the inner side, resulting in repeated remote calls. Examine loops: if you see a high number, consider rewriting the query or adjusting cost parameters.

5. Use CTEs strategically. A CTE can isolate remote scans and let you control whether they are materialized once or inlined. Use MATERIALIZED to avoid repeated remote scans when a CTE is referenced multiple times. Use NOT MATERIALIZED to allow optimizations across CTE boundaries.

6. Instrument, monitor, iterate. Good FDW performance is not a one‑off fix. Monitor queries and plans. Use tests to catch regressions. Adjust tuning knobs and indexes as your data or workload changes. Document your reasoning so others can understand why a particular plan is expected.

7. Educate your team. Federated queries invite subtle bugs and performance traps. Share the high‑level rules – immutable functions only, cross‑server joins are local, always check remote SQL – so engineers write safer queries by default. A 30‑minute training can save hours of debugging later.

Bringing it All Together

This handbook has covered a lot of ground: from the high‑level principle that pushdown is about data movement, to the nitty‑gritty of join conditions and tuning knobs, to troubleshooting steps and case studies. It is intentionally opinionated and personal: these are the patterns and pitfalls encountered in real systems, not abstract guidelines. By sharing specific examples, I hoped to make the rules memorable and show how they interplay with actual workloads.

The goal is not just to tell you what to do, but to show you how to think and problem solve: review the plan, trace data movement, and determine whether the query is doing the heavy work in the right place.

That thinking process, practiced enough times, becomes second nature. When you write a new query, you'll automatically consider whether your predicates are immutable, whether the join can be shipped, and whether you are about to trigger an N+1 pattern. When you review plans, you'll start from the Foreign Scan nodes and remote SQL, not the top‑level node. When you tune, you'll know which knobs to twist and in which order.

Keep experimenting. Use the examples here as starting points. Try different structures in a test environment and measure the difference. The more you play with pushdown, the more comfortable you'll become with its constraints and superpowers.

If this handbook helps you avoid one performance incident or saves you from shipping a broken query, it has done its job. Enjoy exploring the federated world of Postgres.

References

[1] [2] [3] [4] [5] [6] [9] [10] PostgreSQL: Documentation: 18: F.38. postgres_fdw – access data stored in external PostgreSQL servers (https://www.postgresql.org/docs/current/postgres-fdw.html)

[7] PostgreSQL: Release Notes (https://www.postgresql.org/docs/release/10.0/)

[8] PostgreSQL: Release Notes (https://www.postgresql.org/docs/release/12.0/)

We just posted a course on the freeCodeCamp.org YouTube channel that will help you learn relational database design from the ground up. This course covers SQL fundamentals, entity-relationship modeling, normalization (1NF through BCNF), data types and constraints, indexing strategies, and query optimization.

This course is based on the book Grokking Relational Database Design by Dr. Qiang Hao and Dr. Michael Tsikerdekis (Manning Publications, 2025).

Here are the sections in the course:

• Relational Databases for Beginners — Tables, Entities, Keys & SQL

• SQL Filtering & Aggregation

• SQL Table Commands

• Foreign Keys in SQL

• How SQL JOINs Work

• How to learn SQL on your own

• Database Design Goals

• Database Design Lifecycle

• From Real-World Ideas to Tables

• Primary Key, Candidate Key, and Super Key

• Don't Use the Wrong SQL String Type

• The FLOAT Mistake That Crashed a Stock Exchange

• SQL Date and Time Types Explained

• Connecting Entities in an ER Diagram

• One-to-One Relationships

• One-to-Many Relationships

• Many-to-Many Relationships

• Strong vs Weak Entities

• First Normal Form - Primary Keys and Atomic Values

• Second Normal Form - Partial Keys and Functional Dependencies

• Third Normal Form - Transitive Dependencies

• The Untold Story of BCNF

• Primary Key vs Unique Constraints

• Foreign Key Constraints - ON DELETE & ON UPDATE

• Other Constraints: NOT NULL, DEFAULT, and CHECK

• Access Control, Hashing & Encryption

• B-Tree vs Full-Text Indexes

• Denormalization

Watch the full course on the freeCodeCamp.org YouTube channel (6-hour watch).

Not quite. While blue-green deployments work beautifully for stateless applications, they become significantly more complex when you introduce databases and stateful services into the equation. The moment your blue and green environments need to share a database, you're facing a fundamental challenge: how do you evolve your schema and data without breaking either version?

In this article, we'll tackle the real-world complexities of implementing blue-green deployments on Amazon ECS when your application depends on shared state. You'll learn practical strategies for handling database migrations, managing sessions, and maintaining data consistency across application versions.

💡 Complete Working Example: All code examples in this article are available in the bluegreen-deployment-ecs repository on GitHub. You can clone it and deploy the entire infrastructure to your AWS account.

Table of Contents

• The Problem with State in Blue-Green Deployments

• Database Migration Strategies for Blue-Green

• Handling Stateful Services in ECS

• Complete Implementation: End-to-End Example

• Rollback Strategies

• Monitoring During Deployments

• Best Practices

• When NOT to Use Blue-Green

• Alternative Deployment Strategies

• Cleanup

• Conclusion

• Further Resources

The Problem with State in Blue-Green Deployments

The elegance of blue-green deployments starts to crumble when you consider databases. Here's why: your blue environment runs application version 1, your green environment runs version 2, but they both connect to the same RDS instance.

Consider this scenario: you're adding a new feature that requires a new database column. Version 2 of your application expects this column to exist. You deploy green, run your migration to add the column, and switch traffic.

Everything works great until you need to rollback. Now version 1 is receiving traffic, but it doesn't know what to do with that new column. Worse, if your migration removed or renamed a column that version 1 depends on, your rollback will fail catastrophically.

Here are the specific challenges you'll face:

• Schema versioning conflicts: Your blue environment expects schema version N, while green expects version N+1. Any breaking schema change will cause one environment to fail.

• Data inconsistencies: If version 2 writes data in a new format that version 1 can't read, switching back to blue will result in errors or data corruption.

• Irreversible migrations: Some database changes are inherently destructive. Dropping a column, changing data types, or restructuring tables can't be easily undone.

• Failed rollbacks: The promise of instant rollback becomes hollow when your database has evolved beyond what the blue environment can handle.

Let's explore the strategies that solve these problems.

Database Migration Strategies for Blue-Green

Strategy 1: The Expand-Contract Pattern (Recommended)

The expand-contract pattern is the most practical approach for blue-green deployments with shared databases. It works by breaking schema changes into three phases, ensuring backwards compatibility throughout.

Phase 1: Expand

In this phase, you add new schema elements while keeping old ones intact. If you're renaming a column, add the new column without removing the old one. If you're changing table structure, create new tables alongside existing ones.

-- Example: Renaming 'user_name' to 'username'
-- Phase 1: Expand - Add new column
ALTER TABLE users ADD COLUMN username VARCHAR(255);

-- Populate new column from old column
UPDATE users SET username = user_name WHERE username IS NULL;

At this point, your database supports both the old schema (used by blue) and the new schema (used by green). Your application code needs to handle both as well.

Phase 2: Deploy

Now, deploy your green environment with code that uses the new schema. But this code should still write to both old and new columns to maintain compatibility.

# Version 2 code - writes to both columns
def update_user(user_id, username):
    db.execute(
        "UPDATE users SET username = %s, user_name = %s WHERE id = %s",
        (username, username, user_id)
    )

Traffic shifts from blue to green. Both environments work because the database supports both schemas. If you need to rollback, blue still functions perfectly because the old columns are intact.

Phase 3: Contract

After you're confident green is stable and you've decommissioned blue, remove the old schema elements in a separate deployment.

-- Phase 3: Contract - Remove old column
ALTER TABLE users DROP COLUMN user_name;

Update your application code to stop writing to the old columns. This is now version 3, deployed as a standard release.

When to use: This should be your default approach for most schema changes including adding/removing columns, renaming fields, changing constraints, and restructuring tables.

Strategy 2: Parallel Schemas or Databases

For major breaking changes where backwards compatibility is impractical, you might maintain entirely separate database versions. Version 1 connects to database A, version 2 connects to database B. This approach requires data synchronization between databases. AWS Database Migration Service (DMS) can replicate data in near real-time, or you can build custom replication logic using change data capture.

# Configuration for version-specific database connections
DATABASE_CONFIG = {
    'v1': {
        'host': 'blue-db.cluster-xxxxx.us-east-1.rds.amazonaws.com',
        'database': 'app_v1'
    },
    'v2': {
        'host': 'green-db.cluster-yyyyy.us-east-1.rds.amazonaws.com',
        'database': 'app_v2'
    }
}

During the transition period, you run DMS to keep both databases synchronized, with the understanding that writes go to the active version's database.

The challenge is that you're now managing data synchronization, dealing with replication lag, and paying for two databases. Eventually, you need to consolidate back to one database, which requires another migration. This is expensive and complex, which is why it's the "nuclear option."

When to use: Only for major architectural changes, complete data model redesigns, or when migrating between database types (for example, MySQL to PostgreSQL). If expand-contract can possibly work, use that instead.

Strategy 3: Feature Flags for Gradual Rollout

Feature flags allow you to decouple deployment from release. Both blue and green run the same codebase, but features are toggled on or off via configuration. This shifts the problem from schema compatibility to code-level compatibility.

def create_user(user_data):
    config = get_feature_config()
    if config['use_new_user_schema']:
        return create_user_v2(user_data)
    else:
        return create_user_v1(user_data)

Instead of having two separate deployments (blue and green), you have ONE deployment with conditional logic. The "switch" from old to new behavior happens via configuration change, not infrastructure change. This is technically not pure blue-green, but it's a powerful hybrid approach.

How it works

Your application checks AWS AppConfig (or similar service) for feature flags before executing code paths. When a flag is off, it uses the old schema/logic. When on, it uses the new schema/logic. You can even enable features for a percentage of users (5% get new behavior, 95% get old behavior) for gradual rollout.

The tradeoff is that your codebase temporarily contains both old and new logic with conditional branches everywhere. This increases complexity and requires disciplined cleanup after the feature is fully released. However, you gain fine-grained control and can toggle features on/off instantly without deploying new infrastructure.

When to use: For large features with uncertain stability, gradual rollouts to monitor impact, or when you want instant rollback capability without touching infrastructure. Also useful when combined with expand-contract for extra safety.

Handling Stateful Services in ECS

Beyond databases, several other stateful components require careful consideration during blue-green deployments.

Session Management

It’s a good idea to store sessions in ElastiCache or DynamoDB rather than application memory:

app.config['SESSION_TYPE'] = 'dynamodb'
app.config['SESSION_DYNAMODB'] = boto3.client('dynamodb')

Shared Resources

Beyond database sessions, your application likely depends on other stateful components that need coordination during blue-green deployments:

1. S3 buckets

If your application stores files or data in S3, schema changes to object metadata or file formats can cause compatibility issues between versions. To address this, you can enable S3 versioning to maintain multiple format versions simultaneously.

For example, if version 2 writes JSON files with a new structure, version 1 should still be able to read the old format. You can include a version prefix in object keys (like v1/user-data.json and v2/user-data.json) or embed version metadata in the objects themselves.

Message queues (SQS/SNS)

Messages sent by one version must be readable by the other during the transition. You can use versioned message schemas with a schema_version field in your message payload. Both blue and green should be able to parse messages from either version, even if they only produce messages in their preferred format. Consider using a schema registry or validation library to ensure compatibility.

Cache layers (ElastiCache/Redis)

Cached data structure changes can cause deserialization errors when switching between versions. Try versioning your cache keys by including the schema version: CACHE_VERSION = 'v2' and then cache_key = f"user:{CACHE_VERSION}:{user_id}". This ensures blue and green maintain separate cache namespaces, preventing cross-contamination. When you fully migrate to green, you can flush the old cache keys or let them expire naturally.

CACHE_VERSION = 'v2'
cache_key = f"user:{CACHE_VERSION}:{user_id}"

Implementation: End-to-End Example

Let's walk through a complete blue-green deployment with ECS, handling a database schema change using the expand-contract pattern. We'll migrate from a single address text field to structured street_address, city, state, and zip_code fields.

Here’s the scenario: You're running an e-commerce application on ECS. The current version (blue) stores customer addresses in a single address text field. Version 2 (green) splits this into structured fields: street_address, city, state, and zip_code.

Architecture Setup

Your infrastructure includes:

• ECS cluster running Fargate tasks

• Application Load Balancer with two target groups (blue and green)

• RDS PostgreSQL database (shared between environments)

• CodeDeploy for managing traffic shifts

• Parameter Store for database connection strings

💡 Implementation Note: The complete Terraform code for this architecture is available in the companion GitHub repository.

Prerequisites

Before starting, make sure that you have the following tools installed and your AWS credentials properly configured:

# Required tools
aws --version      # AWS CLI
terraform --version # Terraform >= 1.0
docker --version   # Docker
psql --version     # PostgreSQL client

# Configure AWS credentials
aws configure
aws sts get-caller-identity  # Verify your identity

Step 1: Deploy Infrastructure and Blue Environment

We’ll start by setting up the entire AWS infrastructure from scratch using Terraform, then deploying the initial version of our application (blue environment).

First, clone the repository and set up your environment:

# Clone the repository
git clone https://github.com/Caesarsage/bluegreen-deployment-ecs.git
cd bluegreen-deployment-ecs

# Create terraform variables
cd terraform
cat > terraform.tfvars <<EOF
aws_region         = "us-east-1"
project_name       = "ecommerce-bluegreen"
environment        = "production"
vpc_cidr           = "10.0.0.0/16"

# Database credentials (CHANGE THESE!)
db_username = "dbadmin"
db_password = "ChangeThisPassword123!"

# Container configuration
container_image = "PLACEHOLDER"  # Will update after building image
container_port  = 8080

# Scaling configuration
desired_count = 2
cpu           = "256"
memory        = "512"

# Notifications
notification_email = "your-email@example.com"
EOF

Security Note: Never commit terraform.tfvars to Git. It's already in .gitignore.

Next, initialize Terraform and create the ECR repository:

# Initialize Terraform
terraform init
terraform validate

# Create ECR repository
terraform apply -target=aws_ecr_repository.app

# Get ECR repository URL
export ECR_REPO=$(terraform output -raw ecr_repository_url)
echo "ECR Repository: $ECR_REPO"