The company lost about $440 million in just 45 minutes. Knight Capital nearly collapsed and had to be rescued by investors. It was later acquired by another firm.

When Target expanded into Canada, the company relied on a new supply chain system that contained incorrect product and inventory data. Product information in the database was incomplete and inaccurate. Prices, sizes, and product descriptions were entered incorrectly.

Inventory systems reported items in stock that were actually unavailable. Customers found empty shelves in stores despite the system showing stock. The company lost over $2 billion in the Canadian market. Target eventually shut down all Canadian stores in 2015.

One employee made the statement “Even though we had a great supply chain system on paper, we didn’t have accurate data. Bad data leads to bad decisions’’

Another famous example of data-related engineering failures involves the Mars Climate Orbiter spacecraft. One engineering team used metric units (newtons). Another team used imperial units (pounds-force). The system failed to convert the data correctly. The spacecraft entered Mars' atmosphere at the wrong altitude. The mission failed and the spacecraft was destroyed. The loss was about $125 million.

In this article, we'll delve deep into what data quality truly means, the types of data errors that silently break systems, the developer’s responsibility in preventing them, and the validation layers that work together to keep bad data out of production.

What We'll Cover:

• Prerequisites

• The Importance of Data Quality

  • How Does Bad Data Happen in the First Place?

  • The Cost of Bad Data

• Types of Data Errors

  • Required Field Errors

  • Format Validation Errors

  • Range and Limit Errors

  • Logical Consistency Errors

  • Duplicate and Data Integrity Errors

  • Relational Errors (Reference Integrity)

  • Structural Errors (Dropdowns, Radio Buttons, Enums)

• What Makes Good Data?

  • Completeness:

  • Uniqueness:

  • Validity:

  • Timeliness:

  • Accuracy:

  • Consistency:

  • Fitness for Purpose:

• Data Validation Layers

  • Frontend Layer — “Protect the User, Not the System”

  • Backend Validation — “The Real Gatekeeper”

  • Database Layer — “Protect the Data at Rest”

  • Service Layer / Business Logic — “Validate Real-World Rules”

  • Jobs / Queues / Data Ingestion — “Validate External Data”

• Testing Strategies to Protect Data Quality

  • Unit Testing: The Schema & Constraint Check

  • Integration Testing: The Flow & Lineage Check

  • Functional Testing: The Business Rule Check

• Conclusion

Prerequisites

• A basic understanding of what data is

• A basic understanding of data structures

• An understanding of what an API is

• An understanding of what a database is and what it does

The Importance of Data Quality

As you can see from just these few examples, the quality of the data you're working with really matters.

Gartner reports that organisations attribute around $15 million in annual losses to poor‑quality data. The same research also shows that nearly 60% of companies have no clear idea what bad data actually costs them, largely because they don’t track or measure data‑quality issues at all.

A 2016 study by IBM is even more eye-popping. IBM found that poor data quality strips $3.1 trillion from the U.S. economy annually due to lower productivity, system outages, and higher maintenance costs.

Bad data is, and will continue to be, the kryptonite of any organisation. This is even more concerning as more organisations now depend on data for strategy execution than ever before.

When data is wrong, incomplete, duplicated, or inconsistent, the consequences ripple outward: Incorrect dashboards mislead teams, which leads to making incorrect decisions. Implementing these decisions can lead to faulty strategy and policy implementation.

Eventually, the organisation pays the price, financially, operationally, and reputationally. And while money can be recovered, reputation rarely bounces back so easily.

How Does Bad Data Happen in the First Place?

Form fields are usually the first place where data enters an application, so they’re often where bad data begins. This is why the developer’s role is so critical.

Many of the most damaging data errors don’t originate from malicious users or complex edge cases – they come from simple oversights that the system should never have allowed in the first place.

But it's equally important to recognise that data quality issues often originate before the data ever reaches an application. Upstream processes — how data is collected, measured, recorded, or pre‑validated — can introduce inaccuracies long before the system receives it.

For example, a nurse might weigh a patient using an uncalibrated mechanical scale, record the incorrect value on a paper form, and later have that value transcribed into the hospital system. By the time the data enters the application, the error is already embedded.

This means that maintaining data quality requires attention both to upstream data collection practices and to the system-level validation that developers control.

When the UI, backend, or API layer permits invalid, incomplete, inconsistent, or logically impossible data to enter the pipeline, the organisation inherits a long‑term liability. Even small choices — such as allowing empty fields, ignoring duplicates, or failing to enforce validation rules — can introduce errors that may only surface months later in reports or dashboards, leading to confusion and inaccurate insights.

The Cost of Bad Data

Data quality can also be impacted at any stage of the data pipeline: before ingestion, in production, or even during analysis.

If bad data is caught in the UI, it's almost free, if we're thinking in terms of cost. If it's caught at the API layer, that's still pretty cheap. If it's caught in the database, the cost is moderate. And if it's caught in a report or ML model months later, that's expensive, and sometimes irreversible.

A key principle in modern data management is: the cheapest and safest place to catch bad data is at the source, and that is before ingestion. The well-known 1-10-100 Rule, introduced by George Labovitz and Yu Sang Chang in 1992, clearly illustrates this idea.

According to the rule, it costs about \(1 to validate data at the point of entry, \)10 to correct it after it has entered the system, and $100 per record if the error goes unnoticed and causes problems further down the line.

As the saying goes, an ounce of prevention is worth a pound of cure – and this is especially true when it comes to maintaining high-quality data.

To help buttress my point, I’ve categorised the different types of errors and oversights that developers should never allow that can and should be prevented before they ever reach the database, analytics layer, or reporting systems.

Types of Data Errors

Required Field Errors

If you build a form that allows a user to submit a registration form with important fields left empty (like first name, last name, email address, phone number, date of birth, or address), you're directly letting incomplete data enter the system.

I remember a scenario from my time as a data analyst where I was analysing a dataset containing different types of alarms triggered across several buildings. These alarms fell into categories such as aquarium alarms, intruder alarms, fire alarms, and maintenance alarms.

The purpose of the analysis was simple: identify which buildings had the highest frequency of alarms so that maintenance, resources, or investigations could be allocated appropriately.

Whenever an alarm went off, the security team recorded it using a software system. By the end of each month, we could view the cumulative alarms and generate insights.

But I encountered a major data quality issue. The security officers often selected the alarm category but failed to submit the building where the alarm occurred — and the system allowed this incomplete record to be saved into the database.

Every alarm had to occur in a specific building. Yet during analysis, I would see entries like “20 fire alarms” with no building information attached. Since I couldn’t determine where these alarms happened, the data became unusable. I had no choice but to delete those records because they provided no actionable value.

This is a classic example of poor data validation. If the developer had implemented proper constraints, the system would never allow an alarm to be submitted without a building name.

Required fields should be enforced at the UI and backend levels to prevent missing data from entering the system in the first place. These gaps lead to missing or unusable data in the database, often forcing teams to delete or manually repair records later.

To prevent these errors, you can use required‑field validation, disable the submit button until all mandatory fields are completed, and visually highlight missing fields with inline error messages.

Here's a practical code example of some bad code (no required checks):

<form id="signup">
  <input type="text" id="name" placeholder="Full name">
  <input type="email" id="email" placeholder="Email">
  <button type="submit">Sign up</button>
</form>

<script>
document.getElementById("signup").addEventListener("submit", e => {
  const name = document.getElementById("name").value;
  const email = document.getElementById("email").value;
  console.log("Submitted:", { name, email });
});
</script>

From the above code snippet, the core problem is that the form doesn't enforce required input. Neither HTML‑level validation (using the required attribute) nor JavaScript‑based checks are implemented. This omission allows users to submit the form without providing necessary information, making the form unreliable for collecting valid and complete user data.

From a usability and data quality perspective, this is problematic. Forms are typically designed to collect meaningful and complete information, and fields such as “Full name” and “Email” are usually essential. Without marking these inputs as required or validating them programmatically, we risk receiving blank or invalid submissions, which can compromise the quality of stored data and any processes that depend on it.

Here's an example of a better version (UI prevents empty submission):

<form id="signup">
  <input type="text" id="name" placeholder="Full name" required>
  <input type="email" id="email" placeholder="Email" required>
  <button type="submit">Sign up</button>
</form>

<script>
document.getElementById("signup").addEventListener("submit", e => {
  if (!e.target.checkValidity()) {
    e.preventDefault();
    alert("Please fill in all required fields.");
  }
});
</script>

In this revised version of the code, the addition of the required attribute to both the name and email input elements ensures that the browser won't allow the form to be submitted unless these fields are filled. This is an important step toward maintaining data completeness and improving the overall reliability of the form.

Also, by checking e.target.checkValidity(), we now ensure that the form is evaluated before submission proceeds.

Another positive aspect is the conditional use of e.preventDefault(). When the form is invalid, the default submission behavior is stopped, preventing incomplete or incorrect data from being sent.

Format Validation Errors

If you have a form that allows a user to enter an email without an @ symbol, an email without a domain, a phone number containing letters, or a postcode/ZIP code in the wrong format, that allows invalid data to enter the system.

The same applies when you allow a user to submit an impossible date (32/15/2025) or a credit card number with the wrong length.

These issues will cause the data analyst to spend more time cleaning the data, if it's even cleanable. And such incorrect inputs create unreliable data that breaks downstream processes and increases cleanup costs.

To prevent these types of errors, you can use regex validation, input masks, and field‑type restrictions (for example, numeric‑only fields for phone numbers) to enforce correct formats before submission.

Here's a bad example of allowing format validation errors:

<input id="phone" placeholder="Phone number">
<button onclick="save()">Save</button>

<script>
function save() {
  const phone = document.getElementById("phone").value;
  console.log("Saving phone:", phone);
}
</script>

This code doesn't perform any checks on the format or structure of the phone number. The function simply retrieves whatever value exists – whether valid, invalid, or blank – and logs it to the console without any condition.

Here's the fixed version:

<input id="phone" placeholder="Phone number" required>
<button onclick="save()">Save</button>

<script>
function save() {
  const phone = document.getElementById("phone").value;

  if (!/^\d+$/.test(phone)) {
    alert("Phone number must contain digits only.");
    return;
  }

  console.log("Saving phone:", phone);
}
</script>

This version fixes the earlier mistake by introducing a clear validation rule. Before the system accepts the phone number, it checks whether the input contains only digits. The regular expression ^\d+$ ensures that the value is made up entirely of numbers, with no letters or symbols allowed. If the user enters anything invalid, the function stops and displays an error message instead of saving bad data.

This approach prevents the format error that occurred in the previous example. Instead of blindly trusting whatever the user types, the code now enforces a rule that matches the expected format of a phone number. This is what a responsible developer should do: verify the input before using it.

Range and Limit Errors

Allowing users to enter values outside acceptable limits – such as negative ages, quantities below zero, discounts above 100%, or measurements far beyond realistic ranges – that enables the ingestion of data that violates business rules. These errors distort analytics, break calculations, and create operational inconsistencies.

To mitigate these errors, you can apply min/max constraints, sliders, steppers, and numeric boundaries to ensure values fall within valid ranges.

Here's a bad example of allowing range and limit errors:

<input id="age" type="number">
<button onclick="submitAge()">Submit</button>

<script>
function submitAge() {
  console.log("Age:", document.getElementById("age").value);
}
</script>

As seen above, we've created an input field for age but doesn't specify any limits or constraints. The browser allows the user to type any number — including values that make no sense, such as negative ages, extremely large ages, or decimals. The JavaScript function simply reads the value and logs it without checking whether the age is realistic.

Here's a better version:

<input id="age" type="number" min="0" max="120" required>
<button onclick="submitAge()">Submit</button>

<script>
function submitAge() {
  const ageInput = document.getElementById("age");
  if (!ageInput.checkValidity()) {
    alert("Age must be between 0 and 120.");
    return;
  }
  console.log("Age:", ageInput.value);
}
</script>

Now in this version, the inclusion of the min="0" and max="120" attributes sets clear boundaries for acceptable input values. This ensures that only realistic age values within a defined range are allowed, preventing invalid entries such as negative numbers or excessively large ages.

The JavaScript function further enhances this validation by using the checkValidity() method. This method checks whether the input satisfies all defined constraints, including the required condition and the specified numeric range. If the input doesn't meet these conditions, the function prevents further execution and displays an alert message, informing the user that the entered age must fall within the allowed range.

Logical Consistency Errors

If you allow a user to select an end date before the start date, choose a checkout date earlier than check‑in at a hotel, or enter a delivery date before the order date, this will result in logically impossible data. The same applies when you allow a user to enter a graduation year earlier than their admission to a program, or submit working hours that exceed 24 hours in a day.

You can mitigate this by implementing cross‑field validation, business‑rule checks, and conditional logic that ensures related fields remain consistent.

Here's a bad example of a logical consistency error:

<input type="date" id="start">
<input type="date" id="end">
<button onclick="save()">Save</button>

<script>
function save() {
  console.log({
    start: document.getElementById("start").value,
    end: document.getElementById("end").value
  });
}
</script>

In the code above, the core issue is the complete absence of validation. Although the inputs use type="date", which provides a structured way for users to select dates, the code doesn't enforce that either field is required. This means the user can leave one or both date fields empty, and the save() function will still run and log the values. As a result, the system may end up processing incomplete or meaningless data.

Beyond missing required checks, the code also fails to validate the logical relationship between the two dates. In any scenario involving a start date and an end date, it's expected that the start date shouldn't occur after the end date. But this code performs no such comparison.

This means that the user can select a start date that's later than the end date, and the system will accept it without warning. This leads to inconsistent or impossible data being recorded.

Also, the function simply logs the values without providing any feedback to the user. There's no mechanism to alert the user when a field is empty or when the dates are logically incorrect. This reduces usability and makes it difficult for users to understand or correct their mistakes.

Here's the fixed version:

<input type="date" id="start" required>
<input type="date" id="end" required>
<button onclick="save()">Save</button>

<script>
function save() {
  const startValue = document.getElementById("start").value;
  const endValue = document.getElementById("end").value;

  // Extra safety: check empties (in case required is bypassed)
  if (!startValue || !endValue) {
    alert("Both start and end dates are required.");
    return;
  }

  const start = new Date(startValue);
  const end = new Date(endValue);

  if (end < start) {
    alert("End date cannot be before start date.");
    return;
  }

  console.log({ start, end });
}
</script>

In this improved version, first, both date fields now include the required attribute, ensuring that the user can't leave either field empty without triggering validation.

Second, we've added a logical validation check to ensure that the relationship between the two dates is correct. After retrieving the values, the function converts them into Date objects and compares them to verify that the end date doesn't occur before the start date. If this condition is violated, the function stops execution and displays an alert informing the user of the error.

This prevents inconsistent or impossible date ranges from being accepted.

Duplicate and Data Integrity Errors

When you let a user submit an email that's already registered, choose a username that's already taken, or enter a duplicate employee ID or student number, this results in identity conflicts and duplicate records. Problems also arise when you allow users to upload unsupported file types, oversized files, or corrupted images.

Security risks can emerge when users are able to enter HTML/script tags (XSS), SQL‑injection patterns, or disallowed special characters. These issues compromise data quality, system integrity, and security.

You can prevent these types of issues by using uniqueness checks, file‑type and size validation, and input sanitization to block duplicates, invalid uploads, and malicious inputs.

Here's an example of a duplicate error:

<input id="email" placeholder="Enter email" required>
<button onclick="save()">Save</button>

<script>
const savedEmails = [];

function save() {
  const email = document.getElementById("email").value;
  savedEmails.push(email);
  console.log("Saved emails:", savedEmails);
}
</script>

This code blindly pushes every email into the savedEmails array without checking whether the email already exists. Because there is no duplicate detection, the user can enter the same email multiple times.

Here is the fixed version:

<input id="email" placeholder="Enter email" required>
<button onclick="save()">Save</button>

<script>
const savedEmails = [];

function save() {
  const email = document.getElementById("email").value.trim();

  // Check if the field is empty
  if (!email) {
    alert("Please enter an email before saving.");
    return;
  }

  // Check for duplicate
  if (savedEmails.includes(email)) {
    alert("This email has already been saved.");
    return;
  }

  savedEmails.push(email);
  console.log("Saved emails:", savedEmails);
}
</script>

In this improved version of the code, we've implemented proper validation steps to prevent duplicate email entries. Before saving the email, the function checks whether the value already exists in the savedEmails array using the includes() method. If the email is found, the function stops execution and displays an alert informing the user that the email has already been saved. This ensures that each email is stored only once, maintaining the uniqueness and integrity of the data.

Relational Errors (Reference Integrity)

If you let a user select a city that doesn’t belong to the chosen country, a product ID that no longer exists, a retired SKU, or a shipping method unavailable in the selected region, this can result in broken references.

The same applies when users can select a manager from a different department or choose a fully booked time slot, not setting the right roles and permissions. These errors break relationships between tables and corrupt downstream joins and reports.

Here, you can use dependent dropdowns, real‑time lookups, and foreign‑key validation to help ensure that users can only select valid, existing, and compatible options.

Here's a bad example of a relational error:

<select id="country">
  <option value="uk">United Kingdom</option>
  <option value="usa">United States</option>
</select>

<select id="city">
  <option value="london">London</option>
  <option value="manchester">Manchester</option>
  <option value="newyork">New York</option>
  <option value="losangeles">Los Angeles</option>
</select>

<button onclick="save()">Save</button>

<script>
function save() {
  const country = document.getElementById("country").value;
  const city = document.getElementById("city").value;

  console.log("Saving:", { country, city });
}
</script>

From the above, the mistake in this code is that we've treated country and city as completely independent fields, even though one is supposed to depend on the other. By presenting all cities regardless of the selected country, the interface allows users to create combinations that make no sense — such as choosing “United Kingdom” with “New York” or “United States” with “Manchester.”

Also, because the save() function performs no validation and simply logs whatever the user selects, the system ends up accepting and storing relationships that should never exist. This breaks the logical link between the two fields and leads to invalid, inconsistent data that can corrupt downstream.

Here's the fixed, production-ready version:

<select id="country" onchange="loadCities()" required>
  <option value="">Select country</option>
  <option value="uk">United Kingdom</option>
  <option value="usa">United States</option>
</select>

<select id="city" required disabled>
  <option value="">Select city</option>
</select>

<button onclick="save()">Save</button>

<script>
const citiesByCountry = {
  uk: ["London", "Manchester"],
  usa: ["New York", "Los Angeles"]
};

function loadCities() {
  const country = document.getElementById("country").value;
  const citySelect = document.getElementById("city");

  // Reset city dropdown
  citySelect.innerHTML = '<option value="">Select city</option>';

  // Disable if no country selected
  if (!country) {
    citySelect.disabled = true;
    return;
  }

  // Enable dropdown
  citySelect.disabled = false;

  // Load cities safely
  (citiesByCountry[country] || []).forEach(city => {
    const option = document.createElement("option");
    option.value = city.toLowerCase().replace(/\s+/g, ""); // remove ALL spaces
    option.textContent = city;
    citySelect.appendChild(option);
  });
}

function save() {
  const country = document.getElementById("country").value;
  const city = document.getElementById("city").value;

  // Required validation
  if (!country || !city) {
    alert("Please select both a country and a city.");
    return;
  }

  // Build list of valid cities for this country
  const validCities = (citiesByCountry[country] || [])
    .map(c => c.toLowerCase().replace(/\s+/g, ""));

  // Relational validation
  if (!validCities.includes(city)) {
    alert("Selected city does not belong to the chosen country.");
    return;
  }

  console.log("Saving:", { country, city });
}
</script>

This improved code turns the country–city form into a controlled, relationship‑aware flow instead of two loose dropdowns.

When the user selects a country, the loadCities() function runs. It first clears the city dropdown and, if no country is selected, keeps the city field disabled so the user can't choose a city on its own.

Once a valid country is chosen, the city dropdown is enabled and populated only with the cities that belong to that specific country, using the citiesByCountry mapping. Also, the city values are normalised (lowercased and stripped of spaces) so they’re consistent and safe to compare.

When the user clicks “Save,” the save() function checks that both a country and a city have been selected. If either is missing, it shows an alert and stops. It then rebuilds the list of valid city values for the chosen country and verifies that the selected city is actually in that list.

Structural Errors (Dropdowns, Radio Buttons, Enums)

If users can type a country as “U.S.A”, “USA”, “United States”, or “us”, enter gender as “male”, “Male”, “M”, or “man”, or type a department as “Engineering”, “Eng”, or “engineer”, this can result in inconsistent categorical data.

The same applies to currencies typed as “usd”, “USD”, “US Dollars”, product categories spelled differently, status values like “active”, “Active”, “ACT”, “enabled”, or boolean values like “yes”, “Yes”, “Y”, “1”.

These inconsistencies make analytics, grouping, and reporting unreliable, and the analyst will spend time cleaning and standardizing these files.

You should replace free‑text fields with dropdowns, radio buttons, and enums to enforce standardized categorical values.

Bad example of a structural error:

<form id="profile">
  <label>Country</label>
  <input type="text" id="country" placeholder="Enter country">
  <button type="submit">Save</button>
</form>

<script>
document.getElementById("profile").addEventListener("submit", e => {
  e.preventDefault();
  const country = document.getElementById("country").value;
  console.log("Saving:", country);
});
</script>

The problem with this code is that it pretends to save a country value without doing any real validation or enforcing any rules, which makes the form unreliable and prone to bad data.

The form uses a plain text input for “country,” meaning the user can type anything they want — misspellings, random characters, invalid countries, or even leave it blank. Because the input isn’t marked as required and the JavaScript doesn’t check whether the field contains a meaningful value, the form will happily “save” an empty string or nonsense text.

The submit handler prevents the default form submission but does nothing beyond logging whatever the user typed, so the system accepts invalid, incomplete, or malformed data without question. In short, the code collects input but doesn't validate it, doesn't enforce correctness, and doesn't protect the system from bad or unusable values.

Here's the fixed version:

<form id="profile">
  <label>Country</label>
  <select id="country" required>
    <option value="">Select country</option>
    <option value="uk">United Kingdom</option>
    <option value="usa">United States</option>
    <option value="canada">Canada</option>
  </select>

  <button type="submit">Save</button>
</form>

<script>
document.getElementById("profile").addEventListener("submit", e => {
  e.preventDefault();

  const country = document.getElementById("country").value;

  // Required validation
  if (!country) {
    alert("Please select a country before saving.");
    return;
  }

  console.log("Saving:", country);
});
</script>

The biggest improvement is that we're no longer relying on a free‑text field for the country. By switching to a dropdown, the form now limits the user to a controlled set of valid options. This prevents misspellings, random text, or invalid country names from ever entering the system.

These are the main types of data errors you might come across in your work. Now that we've discussed what causes them and some key fixes/preventative measures you can take, let's move on to data quality itself.

What Makes Good Data?

So what, in fact, is data quality? IBM defines it as the degree of accuracy, consistency, completeness, reliability, and relevance of the data collected, stored, and used within an organization or a specific context.

Let's look at each of these features of quality data a bit more closely to understand what they entail.

Completeness:

Completeness measures how much of the required data is actually present. When large portions of fields are missing, the dataset stops representing reality and any analysis built on it becomes unreliable.

An example would be a sign‑up form that stores users, but half of them are missing an email address. If you run an analysis on “email engagement,” your results will be skewed because a big chunk of users can’t even receive emails. This means that this data is incomplete.

Uniqueness:

Uniqueness checks whether each real‑world entity appears only once in the dataset. Duplicate records inflate counts, break joins, and distort metrics.

An example would be a customer table containing two rows for the same person with the same customer ID. When calculating “active customers,” the system counts them twice, inflating revenue projections.

Validity:

Validity evaluates whether data follows the expected format, type, or business rules. This includes correct data types, allowed ranges, and patterns defined by the system.

An example would be a field meant to store dates contains values like “32/99/2025” or “tomorrow.” These invalid entries break downstream ETL jobs that expect a proper date format.

Timeliness:

Timeliness reflects whether data is available when it’s needed. Even accurate data becomes useless if it arrives too late for the process that depends on it. For example, after a customer places an order, the system should generate an order ID instantly.

Accuracy:

Accuracy measures how closely data matches the real‑world truth. When multiple systems report the same metric, one must be designated as the authoritative source to avoid conflicting values.

Consistency:

Consistency checks whether data aligns across different datasets or within related fields. If two systems describe the same concept, their values shouldn't contradict each other.

For example, a company’s HR system reports 50 employees in Engineering, but the payroll system lists only 42. Since both describe the same group, the mismatch signals a data quality issue.

Fitness for Purpose:

Fitness for purpose assesses whether the data is suitable for the specific business task at hand. Even complete, accurate, and timely data may be unhelpful if it doesn’t answer the intended question.

A dataset of website clicks might be perfect for analysing user engagement, for example, but it’s useless for forecasting revenue because it contains no purchase or pricing information.

Data Validation Layers

Now that we've highlighted the characteristics that ensure quality data, it's important to discuss the layers of data validation.

There are five layers you'll need to check to enforce data quality.

Frontend Layer — “Protect the User, Not the System”

Frontend validation plays an important role in enhancing the user experience – but it doesn't provide real protection for a system.

Since frontend logic operates within the user’s environment, we can't trust it as a mechanism for enforcing data quality. Any code executed in the browser is ultimately under the user’s control, meaning it can be disabled, modified, intercepted, or bypassed entirely.

For instance, a user can simply open browser developer tools, remove validation rules, and submit invalid or malicious data without restriction.

Frontend validation is incapable of enforcing complex business rules. Constraints such as ensuring that a discounted price is lower than the original price, validating that a start date precedes an end date, preventing stock levels from becoming negative, or confirming that a product belongs to a valid category within the database require deeper system-level checks.

At the frontend level, what is being validated is: required fields, email format, password strength, address fields, and payment input format.

So frontend validation doesn't guarantee data quality or security, as it can be bypassed through API tools (like Postman), disabled JavaScript, malicious bots, and third-party integrations.

Because of this, it's best to treat the front-end as a usability layer, not a trust layer.

Backend Validation — “The Real Gatekeeper”

You can only guarantee true data quality and system integrity at the backend and database layers.

The backend is responsible for enforcing request validation, implementing business logic, and managing authentication and authorization.

If validation fails here, invalid data is rejected before it can propagate. Without this layer, data corruption begins at ingestion.

For example:

$request->validate([
   'name' => 'required|string|max:255',
   'price' => 'required|numeric|min:0',
   'stock' => 'required|integer|min:0',
   'category_id' => 'required|exists:categories,id',
]);

The code snippet above demonstrates how you can use request validation in Laravel to ensure that incoming data meets specific requirements before it's processed or stored in the database. This is an essential practice in web development, as it helps maintain data integrity, prevents errors, and enhances application security.

In this example, we're using the $request->validate() method to define a set of validation rules for four input fields: name, price, stock, and category_id. Each field is assigned a series of constraints that the incoming data must satisfy.

The name field is marked as required, meaning it must be included in the request and can't be empty. It must also be a string, ensuring that only textual data is accepted, and it's limited to a maximum length of 255 characters using max:255. This prevents excessively long inputs that could potentially cause issues in the database or user interface.

Similarly, the price field is required and must be numeric, allowing only numbers such as integers or decimal values. The rule min:0 ensures that the price can't be negative, which is logically consistent for most product pricing scenarios.

The stock field is also required and must be an integer, meaning it can only accept whole numbers. This is appropriate for counting physical items. Like the price field, it includes a min:0 rule to prevent negative stock values, which would not make sense in an inventory system.

Finally, the category_id field is validated to ensure it is both present and valid. The required rule ensures that a category is selected, while the exists:categories,id rule checks that the provided value corresponds to an existing id in the categories database table. This prevents invalid or non-existent category references, thereby preserving relational integrity within the database.

This layer validates null values, data types and formats, allowed ranges, and referential integrity (exists).

Database Layer — “Protect the Data at Rest”

Validation at the application level is insufficient on its own. You'll also need to enforce database-level constraints like NOT NULL constraints, UNIQUE constraints (email, SKU, order number), foreign keys (orders.user_id → users.id), and check constraints (for example, price >= 0).

This layer is critical because application bugs may bypass validation, background jobs and imports may skip controllers, and malicious actors may attempt direct access.

The database layer acts as the final line of defense, ensuring structural integrity regardless of application failures. Database constraints are the last hard stop: they enforce correctness even when code is bypassed.

Service Layer / Business Logic — “Validate Real-World Rules”

This layer enforces domain-specific logic that can't be captured by simple validation rules. The service layer is where the application stops asking “Is this data shaped correctly?” and starts asking “Is this allowed to happen in the real world?”.

This layer enforces domain‑specific rules that can't be captured by simple request validation or database constraints. These rules reflect business truth, not structural correctness.

Example:

if (\(product->stock < \)quantity) {
   throw new OutOfStockException();
}

This prevents overselling and ensures the system reflects physical reality.

if (\(cartTotal !== \)calculatedTotal) {
   throw new PriceMismatchException();
}

This protects revenue and prevents tampering.

In this layer, you enforce real‑world business rules by ensuring inventory correctness, recalculating totals, applying discount logic, and checking user‑specific limits.

Jobs / Queues / Data Ingestion — “Validate External Data”

When importing or processing external data (for example, supplier feeds), validation must occur before processing. You'll need to ensure schema conformity, that the required columns are present, that you have the correct data types, that the JSON structure is valid, and that you're detecting duplicate batches.

This is because external data sources are a major source of data quality issues. Without validation here, corrupted data can silently enter the system at scale.

Now that we've discussed the layers of a modern application stack, it should be clear that data quality isn't something you “check once” at the UI.

It must be enforced repeatedly, at multiple depths of the system. Each layer catches a different class of defects, and together they form a defensive wall that prevents bad data from ever reaching storage, analytics, or downstream consumers.

Testing Strategies to Protect Data Quality

To wrap up, here are the three foundational testing strategy every developer should apply to protect data quality.

Unit Testing

Unit tests are the first line of defense in data quality. In this context, a “unit” refers to a single column, a single transformation, or a single validation rule.

The purpose is straightforward: verify that the smallest building blocks of your data logic behave exactly as intended. This matters because if these low‑level rules are not tested and validated, incorrect or inconsistent data will flow into the database and contaminate everything built on top of it.

By isolating each rule or transformation, you can guarantee that schema constraints, field‑level assumptions, and low‑level logic remain correct before data ever flows into larger pipelines or business processes.

Typical questions answered at this layer include:

1. Does this column allow nulls?

2. Does this regex correctly strip whitespace from email strings?

3. Does this transformation produce the expected output for a single row?

This is where you can verify that the data contract is sound. If a column must be non‑null, unique, or follow a specific pattern, the unit test enforces it. When these rules fail here, they fail cheaply – before they can corrupt a table or mislead a dashboard.

To make this concrete, here’s what a unit test looks like in a real codebase. Even though this example comes from Laravel, the testing principle is identical to data‑quality unit tests: one rule, one expectation, isolated from everything else.

Example: Testing a Discount Calculation Rule

Imagine your e‑commerce shop has this rule:

• If a product costs more than £100, apply a 10% discount.

• Otherwise, apply no discount.

Let's say this is your discount logic:

<?php

namespace App\Services;

class DiscountService
{
    public function calculate(float $price): float
    {
        if ($price > 100) {
            return $price * 0.10; // 10% discount
        }

        return 0;
    }
}

The unit test for this logic will be:

<?php

namespace Tests\Unit;

use Tests\TestCase;
use App\Services\DiscountService;

class DiscountServiceTest extends TestCase
{
    /** @test */
    public function it_applies_10_percent_discount_when_price_is_above_100()
    {
        $service = new DiscountService();

        \(discount = \)service->calculate(200);

        \(this->assertEquals(20, \)discount);
    }

    /** @test */
    public function it_applies_no_discount_when_price_is_100_or_below()
    {
        $service = new DiscountService();

        \(discount = \)service->calculate(100);

        \(this->assertEquals(0, \)discount);
    }
}

The DiscountService contains a simple rule: if a price is greater than 100, a 10% discount is applied. Otherwise, no discount is applied. The unit test verifies this rule in isolation, without involving controllers, databases, or HTTP requests. By testing the service directly, the developer ensures that the core calculation behaves exactly as intended.

The first test checks the positive case — a price of 200 should produce a discount of 20. The second test checks the boundary condition — a price of 100 should produce no discount. Together, these tests confirm both sides of the rule and protect against regressions if the logic changes in the future.

Now, since this is Laravel example, Laravel tests help you verify both your logic (unit tests) and your full application behaviour (feature tests). You can run them using php artisan test, which executes tests in a separate testing environment, ensuring your real database and main codebase remain safe and unaffected.

Integration Testing: The Flow & Lineage Check

While unit tests validate the correctness of individual rules, integration tests validate the movement of data across components. Integration testing verifies that multiple layers work together as a single data flow.

In this example, the controller receives an order, calls the discount service, applies the transformation, and persists the result to the database. That interaction across layers is what elevates this from a unit test to an integration test. This is where you test the real‑world flow:

1. Controller → Service → Repository → MySQL

2. Check if MySQL migrations run correctly

3. Check foreign keys enforce relationships

4. Check to ensure services interact with the database as expected

5. Check to ensure models and repositories behave consistently

Integration tests reveal issues that only appear when components interact: incorrect joins, broken migrations, mismatched field names, or subtle type mismatches that unit tests cannot detect.

This is the layer where you catch the bugs that would otherwise silently corrupt data lineage.

Here's an example:

<?php

namespace Tests\Feature;

use Tests\TestCase;
use App\Models\Order;
use Illuminate\Foundation\Testing\RefreshDatabase;

class ApplyDiscountTest extends TestCase
{
    use RefreshDatabase;

    /** @test */
    public function check_it_persists_the_correct_discounted_total_to_the_database()
    {
        $order = Order::factory()->create(['subtotal' => 150]);

        \(response = \)this->postJson("/orders/{$order->id}/apply-discount");

        $response->assertStatus(200);

        $this->assertDatabaseHas('orders', [
            'id' => $order->id,
            'grand_total' => 135, // 150 - 10% discount
            'discount_total' => 15
        ]);
    }
}

This represents a full flow rather than a single rule:

• Controller → Service

• Service → Calculation

• Controller → Database write

• Database → Final state

This test begins by creating an order using an Eloquent factory. It immediately steps beyond the boundaries of a unit test, since it interacts with the database and relies on Laravel’s model layer to persist real data.

From there, the test sends an actual HTTP POST request to the /orders/{id}/apply-discount endpoint, which means it's not calling a method directly, but instead it's traveling through Laravel’s routing layer, invoking the controller responsible for handling the request, and triggering whatever business logic is responsible for calculating and applying the discount.

This movement through multiple layers (routing, controller, service logic, and model persistence) is precisely what defines integration testing: the goal is to verify that these components work together correctly as a system.

Once the request is processed, the test asserts that the response returns a successful status code, which confirms that the HTTP layer behaved as expected.

But the most important part comes afterward, when the test checks the database to ensure that the correct grand_total and discount_total were saved. This final assertion proves that the discount logic was executed, the model was updated, and the changes were successfully written to the database.

In other words, the test isn't merely checking whether a calculation is correct. It's also checking whether the entire pipeline – from receiving the request to updating the database – functions as a coherent whole.

Functional Testing: The Business Rule Check

Functional tests validate the entire user experience, from the moment a request enters the system to the moment a response is returned. This includes:

• HTTP requests

• Controller logic

• Validation rules

• Service operations

• Database writes

• Redirects or rendered views

This is where you test the business rules that govern real‑world behaviour:

“A student can't register for two exams at the same time.”

“A cart can't have negative quantities.”

“A user can't update their profile without a valid email.”

Functional tests ensure that the system behaves correctly from the perspective of the user and the business, not just the code.

Here's an example: Functional Test

<?php

namespace Tests\Feature;

use Tests\TestCase;
use App\Models\Product;
use Illuminate\Foundation\Testing\RefreshDatabase;

class CartQuantityFunctionalTest extends TestCase
{
    use RefreshDatabase;

    /** @test */
    public function a_user_cannot_set_a_negative_cart_quantity()
    {
        // Arrange: create a product
        $product = Product::factory()->create(['price' => 40]);

        // Simulate existing cart
        $this->withSession([
            'cart' => [
                $product->id => ['quantity' => 2]
            ]
        ]);

        // Act: user tries to update quantity to a negative number
        \(response = \)this->post('/cart/update', [
            'product_id' => $product->id,
            'quantity' => -5
        ]);

        // Assert: system rejects invalid business behaviour
        $response->assertStatus(302); // redirect back with errors
        $response->assertSessionHasErrors(['quantity']);

        // Assert: cart remains unchanged (business rule preserved)
        \(this->assertEquals(2, session('cart')[\)product->id]['quantity']);
    }
}

The test begins by creating a realistic environment in which a user interacts with a shopping cart. This is essential for understanding the behaviour the system is meant to enforce.

First, it generates a real product in the database using a factory, giving the product a price so that it resembles an item a customer might genuinely add to their cart.

Once the product exists, the test manually seeds the session with a cart containing that product and a quantity of two. This simulates a user who has already added the item to their cart in a previous interaction, and it establishes the baseline state the system must preserve if the user attempts an invalid update.

With the environment prepared, the test then imitates a user action by sending a POST request to the /cart/update endpoint. Instead of calling a method directly, it uses Laravel’s HTTP layer to reproduce the exact behaviour of a browser submitting a form. The request includes the product ID and a deliberately invalid quantity of negative five.

This is the heart of the scenario: the user is attempting something that violates the business rules of the application, and the test is designed to confirm that the system responds appropriately.

Now, when the request is processed, the test expects the application to reject the input, redirect the user back, and attach validation errors to the session. The assertion that the response has a 302 status code and contains validation errors confirms that the validation layer is functioning correctly and that the controller is enforcing the rule that quantities can't be negative.

The final part of the test is where the business rule is truly verified. After the failed update attempt, the test inspects the session to ensure that the cart remains unchanged. This is crucial because rejecting invalid input is only half of the requirement: the system must also protect the integrity of the existing cart data.

Functional tests answer questions like:

• Does the system prevent invalid real‑world behaviour?

• Does the user get the correct feedback?

• Does the data remain consistent after the request?

• Does the final output match the business expectation?

Conclusion

Data quality is never the result of a single check or a single team. It emerges from a disciplined, layered approach where each testing level catches a different category of defects.

Unit tests safeguard the smallest rules, integration tests validate the flow of data across components, and functional tests enforce the business logic that governs real‑world behaviour.

When these layers operate together, bad data has nowhere to hide. When they don’t, even a minor oversight can slip through the cracks and escalate into a costly downstream failure.

So as you can see, your role in data quality is fundamentally proactive, not reactive. By designing systems with validation, integrity, and monitoring in mind, you ensure that data flowing through the pipeline is accurate, timely, complete, unique, and fit for purpose – supporting reliable analytics, reporting, and intelligent systems.

The truth is, most developers don’t realize their code is vulnerable until something breaks. Or, worse, until someone attacks it. So if you want secure code, you first have to know what bad code looks like.

In this tutorial, we’ll see 10 clear signs that your code might be vulnerable to attacks. And more importantly, how to fix it.

Here’s what we’ll cover:

• 1. Hardcoded Credentials

• 2. No Input Validation

• 3. Poor Error Handling

• 4. Outdated Dependencies

• 5. No Authentication or Weak Authentication

• 6. Missing Authorization Checks

• 7. Exposed Sensitive Data in URLs

• 8. No Rate Limiting

• 9. Unsafe File Uploads

• 10. Missing HTTPS

• Final Thoughts

1. Hardcoded Credentials

This one is everywhere. Maybe you’ve seen it yourself – an API key sitting right there in the code. A database password written in plain text.

It looks like this:

DB_PASSWORD = "supersecret123"
API_KEY = "sk_test_abc123"

If this code leaks (and it will), attackers can do whatever they want. They can log into your systems, steal your data, or run up huge bills on cloud services – all without breaking a sweat.

And here’s the scary part: this kind of leak doesn’t just happen when your whole project gets hacked. It can happen when someone pushes code to GitHub and forgets to add .env to .gitignore. Boom – your secret keys are now public.

How to Protect Against It

Never hardcode sensitive data like API keys, database passwords, or tokens. Instead, use environment variables.

These are hidden from the source code and can be safely managed per environment (dev, test, production). For example, a .env file imported into your codebase:

import os
db_password = os.getenv("DB_PASSWORD")

2. No Input Validation

If you trust user input, you’re already in trouble. Attackers love sending weird stuff, like super long strings, funky characters, or unexpected formats.

Here’s what it looks like:

username = request.GET['username']
print("Hello " + username)

Now someone enters:

username=Robert'); DROP TABLE users; --

Boom. You’ve just been SQL injected. Your database table? Gone.

Without validation, your app can break or even be hijacked. Bad input can lead to issues like SQL injection, cross-site scripting (XSS), and general bugs.

Basically, you’re giving attackers a blank check.

How to Protect Against It

Make sure you validate all inputs. For example:

import re
email = request.GET.get('email')
if not re.match(r"[^@]+@[^@]+\.[^@]+", email):
    return "Invalid email format"

Use parameterized queries. Never build SQL strings from raw user input:

cursor.execute("SELECT * FROM users WHERE email = %s", (email,))

And use strict data types. Don’t just assume input is clean. Make it pass a test.Limit input length. No one needs a 5,000-character username.Escape special characters especially if you’re using input in HTML or SQL.

3. Poor Error Handling

This is what lazy error handling looks like:

except Exception as e:
    print(e)  # Exposes internal errors to the user

Or worse:

except:
    pass  # Silently swallows all errors

In the first example, the error is fully displayed to the user. The second example ignores all errors.

Silent errors are dangerous. And showing full error messages to users? That’s handing over a map to your system.

Imagine a database error pops up in production, and your app spits out something like:

psycopg2.OperationalError: could not connect to server: Connection refused

Great – now attackers know what database you’re using, and they might start poking around.

How to Protect Against It

• Log detailed errors – but do it securely. Use logging tools or services, and don’t store logs where users can see them.

• Show users simple messages like:
  "Oops! Something went wrong. Please try again later."
  That’s all they need to know.

• Never expose stack traces in production. Turn off debug mode and use proper error pages.

• Handle specific exceptions where possible, so you know exactly what failed and why.

Example:

try:
    process_data()
except ValueError as e:
    logger.error(f"Data error: {e}")
    return "Invalid input. Please check your data."
except Exception as e:
    logger.exception("Unexpected error")
    return "Something went wrong. Try again later."

Use error monitoring tools like Sentry, Rollbar, or LogRocket. They catch errors, track them, and help you fix them – before users even notice.

4. Outdated Dependencies

Using old packages is like leaving your front door wide open. Attackers know exactly where the weak spots are – and they actively scan for them.

If your package.json or requirements.txt file hasn’t changed in years, that’s a red flag.

How to Protect Against It

• Update regularly. New versions often patch security flaws.

• Audit your dependencies. Use tools like npm audit and pip-audit based on your codebase.

• Automate updates with tools like Dependabot, Renovate, or PyUp.

pip-audit
# or
npm audit

Even small packages can have big impacts. Stay updated, stay safe.

5. No Authentication or Weak Authentication

If your app lets anyone in without verifying who they are, that’s game over. Weak logins are just as dangerous.

Common mistakes include:

• No password complexity rules – Weak passwords like “123456” or “password” can be cracked in seconds using brute-force or dictionary attacks.

• Storing passwords in plain text – If your database is ever breached, all user credentials are exposed instantly, leading to massive data leaks and account takeovers.

• No account lockout after repeated failed logins – Without a limit on login attempts, attackers can keep guessing passwords endlessly using automated tools.

How to Protect Against It

First, you can hash passwords using strong algorithms like bcrypt.

Here’s an example in Python:

import bcrypt
hashed = bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt())

You can also enforce strong password policies (min length, symbols, and so on) and use multi-factor authentication (MFA) if available for extra protection.

A few extra lines of code can stop a full-blown breach.

6. Missing Authorization Checks

Authentication checks who you are. Authorization checks what you can do. Skipping the second one is like giving everyone admin access.

Example:

@app.route('/user/<id>')
def get_user(id):
    return User.query.get(id)

Here, there’s no check to see if the current user is allowed to view that data.

How to Protect Against It

@app.route('/user/<id>')
@login_required
def get_user(id):
    if current_user.id != int(id):
        return "Unauthorized", 403
    return User.query.get(id)

In the above code, a login is required and the user is verified before giving them access to the data.

• Always verify ownership and roles before showing or modifying data.

• Implement access control rules across your API and frontend.

• Don’t trust IDs from the frontend – verify on the backend too.

7. Exposed Sensitive Data in URLs

Ever seen a password reset link like this?

https://example.com/reset-password?token=abcd1234

Looks harmless – but it’s not. Tokens, session IDs, and API keys should never be in URLs. They get saved in:

• Browser history

• Server logs

• Analytics tools

How to Protect Against It

Make sure you only send sensitive data in POST requests or headers, like this:

POST /reset-password
Authorization: Bearer abcd1234

8. No Rate Limiting

Rate limiting is a security technique that controls how many times a user (or system) can make a request to your server within a given time frame – for example, no more than 10 login attempts per minute.

Without rate limits,

• An attacker can make 1,000 login attempts in a minute

• Your server may crash under fake requests

How to Protect Against It

Set a max request limit per IP or user. You can use tools like Cloudflare or inbuilt tools in programming languages to do this. For example, in Python, we can use flask_limiter.

from flask_limiter import Limiter
from flask_limiter.util import get_remote_address

limiter = Limiter(app, key_func=get_remote_address)

@app.route("/login")
@limiter.limit("5 per minute")
def login():
    # login logic

In the above code, the login attempts are limited to 5 per minute. Stop abuse before it starts.

9. Unsafe File Uploads

Letting users upload files? Cool. But if you’re not careful, they can:

• Upload malware

• Overwrite key files

• Execute scripts on your server

Here’s an example of a common mistake:

file. Save(f"/uploads/{file.filename}")

Any type of file could be uploaded this way.

How to Protect Against It

To start, you can rename files before saving:

pythonCopyEditimport uuid
filename = str(uuid.uuid4()) + ".jpg"

You can check content type (not just file extension):

pythonCopyEditif file.content_type not in ["image/jpeg", "image/png"]:
    return "Invalid file type"

You also can store files outside public directory, and finally limit file size in your server config and backend code

10. Missing HTTPS

If your app still uses plain old HTTP, all data travels in the open – including:

• Passwords

• Tokens

• Personal info

Attackers can sniff it all with tools like Wireshark.

How to Protect Against It

To start, you can use HTTPS everywhere and get a free SSL cert from Let’s Encrypt.

You can also redirect insecure traffic – here’s how you’d do it in Flask, for example:

@app.before_request
def before_request():
    if not request.is_secure:
        return redirect(request.url.replace("http://", "https://"))

Encrypting traffic is not optional – it’s table stakes for modern apps.

Final Thoughts

Writing secure code isn’t about being perfect. It’s about being careful. Slow down. Look at your code with fresh eyes. Think like an attacker. Plan for failure before it happens.

The best security isn’t patched in later – it’s baked in from the start.

For more cybersecurity tutorials, join my newsletter. New to cybersecurity? Check out my Security Starter Course.

The act of application security is a shared responsibility amongst the client and server developers and negligence of one’s role can be disastrous. Statistics show that data breaches in 2023 resulted in exposure of over 8 million data records worldwide.

In this article, I'll be highlighting key areas of API security, which involves data validation. This concept is quite crucial in helping you protect your API from web attacks via malicious user data. This tutorial is well-suited for all backend developers regardless of years of experience.

To be able to follow this tutorial, here are some prerequisites:

• Knowledge of Node.js
• Knowledge of npm and package installation

With that in place, let’s get started.

How Does Data Validation Work?

First of all, what is data validation? Data validation simply entails ensuring the accuracy and reliability of the data received from external sources before onward data processing.

It is a key component of web API security as it is essential for preventing the occurrence of web injection attacks, SQL attacks and NoSQL attacks. To know more about these, you can check this link.

Note that data validation is needed but not limited to the following backend operations.:

• User login and sign up
• Response query
• Updating server databases

All these can be used as avenues by mischievous black hat hackers to gain access to the server database and obtain sensitive user details or even wreak havoc by formatting the entire database.

Popular Data Validation Tools

So far, there are lot of tools that can help the programmer achieve efficient data validation in API development.

They help you avoid reinventing the wheel of using long regex code to validate data. They provide a whole lot of features, including error handling and validation customization functionalities.

Some of these tools include:
• Joi
• Zod
• Yup
• AJv
• Valibot
• Validator.js
• Superstruct

To further shed light on these tools, we'll compare some of the most popular data validation tools mentioned above.

Pros and Cons of Data Validation Tools

To further enlighten you about these JavaScript validation tools, I will be highlighting some pros and cons of three of these popular JavaScript validation tools.

Joi

Pros

• It has a strong, large user community and development support
• It has built-in capabilities to handle complex validations

Cons

• It’s syntax is quite verbose

Zod

Pros

• It is easily compatible with Typescript projects
• It has efficient error-handling capabilities

Cons

• Async validation isn’t supported.

Yup

Pros

• It mainly uses declarative syntax to set its validation tool which confers its simplicity
• It has a comparable fast performance.

Cons

• It doesn’t provide customization features
• It has limited ability to handle complex validations

For the purpose of this tutorial, we'll use Joi as our data validation tool.

Introduction to Joi

Joi is a simple and efficient JavaScript-based data validation tool that is based on the schema-type configuration.

It has built-in capabilities for validating the occurrence of data in various forms, but not limited to Booleans, strings, functions and intervals. It can also handle complex validation operations.

Additionally, it provides minimal caching functionalities. More information about the tool can be found here.

How to Set Up Joi

In this section, we'll set up Joi in our local environment. To install Joi, navigate to the code folder via the command line and run this:

npm i joi

A message confirming successful installation should be displayed. With that completed, we can demonstrate the power of Joi in validating user registration in our demo API.

Demo Project

In this project, you'll use Joi to validate the input received from the client with the intent to sign up on the server. The default code for the user sign-up function for the Node.js application can be found here.

Go on and import the installed Joi package into your code:

const Joi = require("joi");

Before writing our signup controller, we'll initialize the Joi library within the code file:

const SignUpSchema = Joi.object({});

In this project, we'll validate the email, password and username parameters received from the client.

const SignUpSchema = Joi.object({
    email: Joi.string().email({
        minDomainSegments: 2,
        tlds: {
            allow: ['com', 'net']
        }
    }),
    username: Joi.string().alphanum().min(3).max(15).required(),
    password: Joi.string().min(8).required()
});

The email parameter object ensures that the email address is a string, and the domain site is limited to .com and .net, disallowing other forms of domains.

For the username parameter, it ensures that it is a string containing both letters and numbers with a minimum character count of 3 and a maximum character count of 15. The required function ensures that these conditions must be met or the entire request won't be validated.

The password parameter ensures that the password supplied is in a string format with a minimum character count of 8, and it is also required.

To apply it to our endpoints, we include this within the controller function:

const { error, value } = SignUpSchema.validate(req.body, { abortEarly: false });
if (error) {
    res.status(400).json(error.details);
    return;
}

This function gets executed before inserting the user details into the database. The schema tries to validate the received input and then proceeds to the database if successfully validated.

The abortEarly feature is included to allow for all parameters to be assessed. All the errors will be displayed if there is any.

The above can also be replicated in the Login controller function. You can also check out the documentation for other examples of complex validation options using Joi.

The final code for the project is displayed below:

const jwt = require("jsonwebtoken");
const userSchema = require("../Schema/User");
const Joi = require("joi");
const bcrypt = require("bcrypt");
const { createNewColumn, checkRecordsExists, insertRecord } = require('../utils/sqlSchemaFunction');

const generateAccessToken = (use) => {
    return jwt.sign({ userID: use }, process.env.JWT, { expiresIn: "1d" });
}

const SignUpSchema = Joi.object({
    email: Joi.string().email({ minDomainSegments: 2, tlds: { allow: ['com', 'net'] } }),
    username: Joi.string().alphanum().min(3).max(15).required(),
    password: Joi.string().min(8).required()
});

const loginSchema = Joi.object({
    email: Joi.string().email({ minDomainSegments: 2, tlds: { allow: ['com', 'net'] } }),
    password: Joi.string().min(8).required()
});

const register = async (req, res) => {
    const email = req.body.email;
    const password = req.body.password;

    if (!email || !password) {
        res.status(400).json("Please supply the email or password");
        return; 
    }

    const { error, value } = SignUpSchema.validate(req.body, { abortEarly: false });
    if (error) {
        res.status(400).json(error.details);
        return;
    }

    const salt = await bcrypt.genSalt(10);
    const hashedPassword = await bcrypt.hash(password, salt);
    const user = {
        username: req.body.username,
        email: email,
        password: hashedPassword
    };

    try {
        const userAlreadyExists = await checkRecordsExists("users", "email", email);
        if (userAlreadyExists) {
            res.status(400).json("Email must be unique");
        } else {
            await insertRecord("users", user);
            res.status(200).json("User created successfully");
        }
    } catch (err) {
        res.status(500).json({ err: err.message });
    }
};

module.exports = { register };

API testing in Postman

Ensuring that the code followed our defined schema resulted in it being successfully executed.

Conclusion

With this, we have come to the end of the tutorial. I hope you’ve learned about data validation, various data validation tools and data validation best practices.

You can also reach out to me and check out my other articles here. Till next time, keep on coding!

But the excitement soon disappeared when I received feedback from users. Simply put, the users felt the model was bad.

I was saddened by this, but looking back, they were correct. The model may have performed well in terms of top-level metrics. But from the perspective of the consumer, if a machine learning model provides a poor forecast, that person's experience with the model will be bad.

The issue was that specific model features, or slices of data, were causing the model to perform poorly.

In short, before deploying any machine learning model, the onus is on machine learning engineers to assess it, make sure it satisfies strict quality standards, and acts as predicted for all pertinent slices of data.

What is TensorFlow Model Analysis?

To enable Machine Learning engineers to look at the performance of their models at a deeper level, Google created TensorFlow Model Analysis (TFMA). According to the docs, "TFMA performs its computations in a distributed manner over large amounts of data using Apache Beam."

TFMA, as a tool, enables you to really dig into the model's performance and understand how it varies on different slices of data. It provides support for calculating metrics that were used at training time (that is built-in metrics) as well as metrics defined after the model was saved as part of the TFMA configuration settings.

In this tutorial, you will analyze and evaluate results on a previously trained machine learning model. The model you will use is trained for a Chicago Taxi Example, which uses the Taxi Trips dataset released by the city of Chicago. You can check out the full dataset here.

When you are done with this tutorial, you will be able to use Apache Beam to do a full pass over the specified evaluation dataset. Also, you will not only have a more accurate calculation of metrics, but you'll be able to scale up to massive evaluation datasets, since Beam pipelines can be run using distributed processing back-ends.

Prerequisites

• Fundamental knowledge of Apache Beam. The Beam Programming Guide is a great place to start.
• Fundamental understanding of the workings of machine learning models.
• A new Google Colab notebook to run the Python code in your Google Drive. You can set this up by following this tutorial.

Step 1 – How to Install TensorFlow Model Analysis (TFMA)

With your Google Colab notebook ready, the first thing to do is to pull in all the dependencies. This will take a while.

A blank (new) notebook in dark mode

Rename the file from Untitled.ipynb to TFMA.ipynb.

!pip install -U pip
!pip install tensorflow-model-analysis`

The first line upgrades pip to the latest version. pip is the package management system used to install and manage software packages written in Python. It stands for “preferred installer program”. The second line will install TensorFlow Model Analysis, TFMA.

Now, after that is done, restart the runtime before running the cells below. It is important to restart the runtime before running the cells.

import sys
assert sys.version_info.major==3 
import tensorflow as tf
import apache_beam as beam
import tensorflow_model_analysis as tfma

This block of code imports the needed libraries – sys, tensorflow, apache_beam and tensorflow_model_analysis. You use the assert sys.version_info.major==3 command to verify that the notebook is being run using Python 3.

Step 2 – How to Load the dataset

You will download the tar file and extract it.

import io, os, tempfile
TAR_NAME = 'saved_models-2.2'
BASE_DIR = tempfile.mkdtemp()
DATA_DIR = os.path.join(BASE_DIR, TAR_NAME, 'data')
MODELS_DIR = os.path.join(BASE_DIR, TAR_NAME, 'models')
SCHEMA = os.path.join(BASE_DIR, TAR_NAME, 'schema.pbtxt')
OUTPUT_DIR = os.path.join(BASE_DIR, 'output')

!curl -O https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/{TAR_NAME}.tar
!tar xf {TAR_NAME}.tar
!mv {TAR_NAME} {BASE_DIR}
!rm {TAR_NAME}.tar

The dataset downloaded is in the tar file format. It includes the training datasets, evaluation datasets, the data schema and the training and serving saved models along with eval saved models. You will need all of them in this tutorial.

Step 3 – How to Parse the Schema

You need to parse the downloaded schema so that you can use it with TFMA.

import tensorflow as tf
from google.protobuf import text_format
from tensorflow.python.lib.io import file_io
from tensorflow_metadata.proto.v0 import schema_pb2
from tensorflow.core.example import example_pb2

schema = schema_pb2.Schema()
contents = file_io.read_file_to_string(SCHEMA)
schema = text_format.Parse(contents, schema)

You will parse the schema using the text_format method of the google.protobuf library to convert the protobuf message to text format and TensorFlow's schema_pb2.

Step 4 – How to Use the Schema to Create TFRecords

The next course of action would be to give TFMA access to our dataset. For this, we need to create a TFRecords file. We used our schema to create it, since it gives us the correct type for each feature.

import csv
datafile = os.path.join(DATA_DIR, 'eval', 'data.csv')
reader = csv.DictReader(open(datafile, 'r'))
examples = []
for line in reader:
  example = example_pb2.Example()
  for feature in schema.feature:
    key = feature.name
    if feature.type == schema_pb2.FLOAT:
      example.features.feature[key].float_list.value[:] = (
          [float(line[key])] if len(line[key]) > 0 else [])
    elif feature.type == schema_pb2.INT:
      example.features.feature[key].int64_list.value[:] = (
          [int(line[key])] if len(line[key]) > 0 else [])
    elif feature.type == schema_pb2.BYTES:
      example.features.feature[key].bytes_list.value[:] = (
          [line[key].encode('utf8')] if len(line[key]) > 0 else [])
  # Add a new column 'big_tipper' that indicates if the tip was > 20% of the fare. 
  # TODO(b/157064428): Remove after label transformation is supported for Keras.
  big_tipper = float(line['tips']) > float(line['fare']) * 0.2
  example.features.feature['big_tipper'].float_list.value[:] = [big_tipper]
  examples.append(example)
tfrecord_file = os.path.join(BASE_DIR, 'train_data.rio')
with tf.io.TFRecordWriter(tfrecord_file) as writer:
  for example in examples:
    writer.write(example.SerializeToString())
!ls {tfrecord_file}

It is worthy of note that TFMA supports a number of different model types including TF Keras models, models based on generic TF2 signature APIs, as well TF estimator-based models. However, for this tutorial, you will configure a Keras-based model.

In your Keras setup, you will add your metrics and plots manually as part of the configuration (see the metrics guide for information on the metrics and plots that are supported).

Step 5 – How to Set Up and Run TFMA using Keras

import tensorflow_model_analysis as tfma

You'll finally call and use the instance of tfma that you previously imported at this point.

# You will setup tfma.EvalConfig settings
keras_eval_config = text_format.Parse("""
  ## Model information
  model_specs {
    # For keras (and serving models) we need to add a `label_key`.
    label_key: "big_tipper"
  }

  ## You will post training metric information. These will be merged with any built-in
  ## metrics from training.
  metrics_specs {
    metrics { class_name: "ExampleCount" }
    metrics { class_name: "BinaryAccuracy" }
    metrics { class_name: "BinaryCrossentropy" }
    metrics { class_name: "AUC" }
    metrics { class_name: "AUCPrecisionRecall" }
    metrics { class_name: "Precision" }
    metrics { class_name: "Recall" }
    metrics { class_name: "MeanLabel" }
    metrics { class_name: "MeanPrediction" }
    metrics { class_name: "Calibration" }
    metrics { class_name: "CalibrationPlot" }
    metrics { class_name: "ConfusionMatrixPlot" }
    # ... add additional metrics and plots ...
  }

  ## You will slice the information
  slicing_specs {}  # overall slice
  slicing_specs {
    feature_keys: ["trip_start_hour"]
  }
  slicing_specs {
    feature_keys: ["trip_start_day"]
  }
  slicing_specs {
    feature_values: {
      key: "trip_start_month"
      value: "1"
    }
  }
  slicing_specs {
    feature_keys: ["trip_start_hour", "trip_start_day"]
  }
""", tfma.EvalConfig())

It's also important that you create a tfma.EvalSharedModel that points at the Keras model.

keras_model_path = os.path.join(MODELS_DIR, 'keras', '2')
keras_eval_shared_model = tfma.default_eval_shared_model(
    eval_saved_model_path=keras_model_path,
    eval_config=keras_eval_config)

keras_output_path = os.path.join(OUTPUT_DIR, 'keras')

And then you finally run TFMA, ending this step.

keras_eval_result = tfma.run_model_analysis(
    eval_shared_model=keras_eval_shared_model,
    eval_config=keras_eval_config,
    data_location=tfrecord_file,
    output_path=keras_output_path)

Now that you have run the evaluation, look at the visualizations using TFMA. For the following examples, you can visualize the results from running the evaluation on the Keras model.

To view metrics, you will use [tfma.view.render_slicing_metrics](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/view/render_slicing_metrics). By default, the views will display the Overall slice. To view a particular slice, you can either use the name of the column (by setting slicing_column) or provide a tfma.SlicingSpec.

Step 6 – How to Visualize the Metrics and Plots

At this point, it is important that you note that the columns used in the dataset are as follows:

• pickup_community_area
• fare
• trip_start_month
• trip_start_hour
• trip_start_day
• trip_start_timestamp
• pickup_latitude
• pickup_longitude
• dropoff_latitude
• dropoff_longitude
• trip_miles
• pickup_census_tract
• dropoff_census_tract
• payment_type
• company
• trip_seconds
• dropoff_community_area, and
• tips

For a first trial and as an example, you can set slicing_column to look at the trip_start_hour feature from our previous slicing_specs. You are then able to visualize the column.

tfma.view.render_slicing_metrics(keras_eval_result, slicing_column='trip_start_hour')

On running this, you will see that the metrics visualization supports the following interactions:

• Click and drag to pan
• Scroll to zoom
• Right click to reset the view
• Hover over the desired data point to see more details.
• Select from four different types of views using the selections at the bottom.

Note that your initial tfma.EvalConfig has created a whole list of slicing_specs, which you can visualize by updating slice information passed to tfma.view.render_slicing_metrics. Here you can select the trip_start_day slice (days of the week).

tfma.view.render_slicing_metrics(keras_eval_result, slicing_column='trip_start_day')

TFMA also supports creating feature crosses to analyze combinations of features. To test this, you will create a cross between trip_start_hour and trip_start_day.

tfma.view.render_slicing_metrics(
    keras_eval_result,
    slicing_spec=tfma.SlicingSpec(
        feature_keys=['trip_start_hour', 'trip_start_day']))

Now, crossing the two columns creates a lot of combinations! But you will narrow down your cross to only look at trips that start at 1pm. Then, you will select binary_accuracy from the visualization as shown below.

tfma.view.render_slicing_metrics(
    keras_eval_result,
    slicing_spec=tfma.SlicingSpec(
        feature_keys=['trip_start_day'], feature_values={'trip_start_hour': '13'}))

Step 7 – How to Track Your Model's Performance Over Time

You'll use your training dataset for training your model. It will hopefully be representative of your test dataset and the data that will be sent to your model in production.

But while the data in inference requests may remain the same as your training data, in many cases it will start to change enough so that the performance of your model will change.

That means that you need to monitor and measure your model's performance on an ongoing basis, so that you can be aware of and react to changes.

Let's look at how TFMA can help.

output_paths = []
for i in range(3):
  # Create a tfma.EvalSharedModel that points to our saved model.
  eval_shared_model = tfma.default_eval_shared_model(
      eval_saved_model_path=os.path.join(MODELS_DIR, 'keras', str(i)),
      eval_config=keras_eval_config)

  output_path = os.path.join(OUTPUT_DIR, 'time_series', str(i))
  output_paths.append(output_path)

  # Run TFMA
  tfma.run_model_analysis(eval_shared_model=eval_shared_model,
                          eval_config=keras_eval_config,
                          data_location=tfrecord_file,
                          output_path=output_path)

  eval_results_from_disk = tfma.load_eval_results(output_paths[:2])

tfma.view.render_time_series(eval_results_from_disk)

Using the tfma, you can validate and evaluate your machine learning models across different slices of data.

You can see from the image above that you can evaluate the auc (area under the curve), auc_precision_recall, binary_accuracy, binary_crossentropy, calibration, example_count, mean_label, mean_prediction, precision, and recall metrics of the machine learning model.

Conclusion

Finally, it is important that TFMA can be configured to evaluate multiple models at the same time. Typically, you do this to compare a new model against a baseline (such as the currently serving model) to determine what the performance differences in metrics (for example AUC) are relative to the baseline.

When thresholds are configured, TFMA will produce a tfma.ValidationResult record indicating whether the performance matches expectations.

If at this point, you have questions about the difference between evaluating machine learning models using TensorBoard and TensorFlow Metrics Analysis (TFMA), this is a valid concern. Both are tools for providing the measurements and visualizations needed during the Machine Learning workflow.

But it is important to note that you use them in different stages of the development process. At a high level, you use TensorBoard to analyze the training process itself while TFMA is concerned with the deep analysis of the 'finished' trained model.

Thank you for reading!

If you have an existing Xamarin.Forms codebase and want to add validation without affecting your code behind or view model, then this is for you. ?

What made me write this article?

Our team designed and developed a Xamarin mobile app with multiple forms with zero validations, since we had less time and we were okay with back end validations alone.

As time passed our need to add front end validation grew with the app. So we decided to add validations without affecting our code behind or view model. There were no articles to help us with this strategy, so I decided to write one.

Prerequisites

I assume that you are comfortable with Xamarin Behaviors. If not, please read the below documentation — it is very simple and straightforward.

Xamarin.Forms Behaviors - Xamarin
_Behaviors lets you add functionality to user interface controls without having to subclass them. Behaviors are written…_docs.microsoft.com

What we will end up with

The result will be highly readable and declarative validations on the xaml, keeping the code behind and view model away from validation!

Things to note

Whenever the code is too long to digest in a single explanation, I will break it into multiple sections and will use three ellipses(…) in place of code from previous or upcoming sections.

So what is the plan?

We can plan our way pretty well if we can think about the basic things that happens while validating:

The above flowchart is a very abstract representation of our entire design. To shed more light onto the actual code, take a look at the following image:

Enough is enough — let’s code!

The following interface helps in maintaining the consistency across different validators we define.

Now let us write some implementations for a few common validators.

Some validators may need few extra parameters. For example, FormatValidator requires a format parameter, and those parameters can be added as fields in the class.

There are number of ways to display an error. You might prefer to display it right below the control, or you might prefer to show it as a summary at the top of the form. To accommodate these different styles of showing errors, we define an interface for it to maintain consistency.

For the sake of simplicity, I have implemented a very basic error style which just adds and removes a label below the control being validated.

Now it is time for the Xamarin Behavior part, which glues together all the above code.

_style: This field is initialised with our custom BasicErrorStyle implementation. This will help us show and remove the error whenever its needed.

_view: This is the control on which this validation is placed.

PropertyName: This is the property of the control which is to be validated against validation rules, let's say Text property of a Entry control or SelectedItem property of a picker control.

Validators: This is the list of validation rules against which the control property will be validated.

The Validate() method runs through all the validation rules and executes the Check method on the property value of the control. If all the validation rules pass, then the RemoveError method is called on _style. Otherwise the ShowError method is called on _style.

The remaining part of code just attaches and detaches the OnPropertyChanged, OnUnFocussed method to PropertyChanged and UnFocussed event of the control whenever this behavior is added to the xaml.

When these events are triggered, they call the Validate() method we defined earlier which in-turn adds or removes the error as needed.

How to use it

Yes the wait is finally over! Now you can add these validations to your xaml by importing the necessary xaml namespaces.

If you run the app now, you should experience something like this:

Validating the entire form

Now comes the next challenge: how we know whether the entire form is valid? That is, how do we know that all the controls in the form have valid data?

To accomplish this, we create another behavior called ValidationGroupBehavior, in which we group the controls that are to be validated together to validate an entire form.

_validationBehaviors: this is a list of validation behaviors from various controls to be validated.

IsValidProperty: this is a bindable property which you can access in the xaml to check whether the form is valid or not.

We expose two methods, namely Add() and Remove(), to help the controls add or remove themselves from a particular validation group.

We also expose a method, Update(), to help the controls for refreshing the validity of the form whenever their data changes. This method in turn runs all the validations in the _validationBehaviors and sets the value of the IsValid property accordingly.

Refactoring our ValidationBehavior

Let’s do a little refactoring to our ValidationBehavior class to accommodate the validation grouping of controls:

We have added a property called Group to store the validation group to which the control belongs. The Group.Add() and Group.Remove() methods are called when this behavior is attached or detached, respectively, from a control. In turn, this adds or remove the control from a particular validation group. Whenever the control loses focus, it updates the validity of the group by calling the Group.Update() method.

ValidationGroupBehavior in action

Update the xaml file as shown below to see the awesomeness of the validation groups:

The above code adds the ValidationGroupBehavior to the parent element of all controls, and then it adds its reference to the Group property of the Validator in the control. You can then use the IsValid property of the validation group to enable or disable a submit button. Fancy stuff, right?

If you run the app now, you should see something like this:

What about async validators?

I can hear the question buzzing through your mind. One good thing about our design is that it can be extended very easily with little effort. This can be achieved by just declaring another interface as shown below:

You can now create an implementation for an async validator, let's say AsyncUserExists, which returns a task that resolves to a boolean. Add a new property to ValidationBehavior called AsyncValidators and add async validators to that property. A little async and await addition to the validation process will solve our problem.

Are we done at last?

We’ve covered quite a bit of info in a very simple way. But there are few nice-to-have features that you can try yourself:

• Debounce for async validators
• Check if the control is dirty (user typed something) or not before showing an error

Confused?

If you are confused or have some gray areas, feel free to reach out to me in the comments. You can also refer to my demo app in GitHub which I have linked below.

ameerthehacker/XamarinFormValidationDemo
_XamarinFormValidationDemo - This is a demo app for my medium blog on xamarin form validations_github.com

TL;DR

If you are lazy like me, or don't want to write a whole lot to get the above functionality, then you can wait until I develope a NuGet package for xamarin form validations using the above strategy. You are welcome to contribute to it, and I have linked to it below.

ameerthehacker/XamarinFormValidation
_XamarinFormValidation - Delcarative, flexible and highly configurable validations in Xamarin without a sweat :hearts:_github.com

If you found this article helpful, give me some claps ?