<![CDATA[ image processing - freeCodeCamp.org ]]> https://www.freecodecamp.org/news/ https://cdn.freecodecamp.org/universal/favicons/favicon.png <![CDATA[ image processing - freeCodeCamp.org ]]> https://www.freecodecamp.org/news/ Eleventy Fri, 22 May 2026 17:39:53 +0000 60 <![CDATA[ How to Build and Deploy an Image Hosting Service on Sevalla ]]> https://www.freecodecamp.org/news/build-and-deploy-an-image-hosting-service-on-sevalla/ 68d691dd9aa70c44b703deb0 Fri, 26 Sep 2025 13:15:09 +0000 When most people think of image hosting, they imagine uploading photos to a cloud service and getting back a simple link.

It feels seamless, but behind that experience sits a powerful set of technologies. At the core is something called object storage, which is a different way of handling files compared to traditional databases or file systems.

In this article, we’ll build a complete image hosting service using Node.js and Express, connect it to object storage, and finally, deploy the whole project to Sevalla.

By the end, you will have a working application that lets users upload images and retrieve them through hosted URLs, all running live on the cloud.

Table of Contents

What is Object Storage?

To understand why our project is designed the way it is, we need to first understand object storage.

Traditional file storage systems save files in a hierarchy of folders, like your computer’s file explorer. Block storage systems, often used in databases, split data into chunks and manage them for speed and reliability.

Object storage is different. It treats each file, whether an image, video, or document, as a single object. Each object is stored with its metadata and a unique identifier inside a flat structure, usually called a bucket.

This flat architecture makes object storage scalable almost without limit. Instead of worrying about file paths or directories, you simply place an object in a bucket and get back an identifier.

Amazon S3 is the industry standard for object storage, offering massive scale, global replication, and advanced features, but it comes with added complexity and often unpredictable costs. Sevalla’s object storage, on the other hand, is designed for developers who want the same durability and scalability without the steep learning curve.

It provides a simpler setup, and is compatible with S3, so interacting with it is same as using a S3 bucket without the additional setup and complexity. While S3 is ideal for enterprises with petabytes of data, Sevalla’s solution is perfect for projects like image hosting, blogs, or mobile apps where ease of use and speed matter most.

What We Will Be Building

We will create a simple yet practical image hosting service. At its core, the service allows a user to send an image through an HTTP request. The server will accept this image, process it, and store it in object storage.

The usefulness of such a project goes far beyond a coding exercise. If you are building a blog, you could use this service to store images for your posts without worrying about file management on your web server.

If you are developing a mobile app that requires profile pictures or image sharing, this backend can serve as your foundation. Even if you simply want to understand how cloud-native applications handle file uploads, this project gives you a clear, hands-on experience.

By the end, you will not just have code running locally. We will deploy the application on Sevalla, meaning your image hosting service will be live, scalable, and accessible to anyone with a link.

How to Set Up the Project

Let us start by setting up a Node.js project. You can clone this repository if you don’t want to setup the project from scratch.

Create a new project directory, initialize it with npm, and install the required dependencies.

npm init -y
npm i express multer dotenv @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

We will use Express for our web server, Multer for handling file uploads, and the AWS SDK to connect to object storage. Multer acts as middleware, giving us easy access to uploaded files. The AWS SDK gives us programmatic access to object storage, allowing us to upload files and generate links.

Let’s write a quick index.html and put it inside the public/ directory to serve as the UI for file upload.

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8" /> <!-- Set character encoding -->
  <meta name="viewport" content="width=device-width,initial-scale=1" /> <!-- Mobile-friendly -->
  <title>Pic Host</title>

  <!-- Simple CSS styling for layout and form -->
  <style>
    :root { color-scheme: light dark; } /* Support dark/light themes */
    body { 
      font-family: system-ui, sans-serif; 
      max-width: 560px; 
      margin: 4rem auto; 
      padding: 0 1rem; 
    }
    h1 { font-size: 1.25rem; margin-bottom: 1rem; }
    form, .card { 
      border: 1px solid #9993; 
      padding: 1rem; 
      border-radius: 12px; 
    }
    input[type="file"] { margin: .5rem 0 1rem; }
    button { 
      padding: .6rem 1rem; 
      border-radius: 10px; 
      border: 1px solid #9995; 
      background: #0000FF; 
      cursor: pointer; 
    }
    #result { margin-top: 1rem; display: none; }
    #result a { word-break: break-all; } /* Break long URLs nicely */
  </style>
</head>
<body>
  <!-- Page heading -->
  <h1>Simple Image Host</h1>

  <!-- Upload form -->
  <form id="uploadForm" class="card">
    <label for="file">Choose image</label><br/>
    <input id="file" name="file" type="file" accept="image/*" required />
    <br/>
    <button type="submit">Upload</button>
    <!-- Status text (uploading, success, error) -->
    <div id="status" aria-live="polite" style="margin-top:.75rem;"></div>
  </form>

  <!-- Result card: hidden until an image is uploaded -->
  <div id="result" class="card">
    <div>
      <strong>Share this page:</strong> 
      <a id="pageUrl" href="#" target="_blank" rel="noopener"></a>
    </div>
  </div>

  <!-- Client-side JavaScript -->
  <script>
    const form = document.getElementById('uploadForm');   // Form element
    const statusEl = document.getElementById('status');   // Upload status
    const result = document.getElementById('result');     // Result box
    const pageUrlEl = document.getElementById('pageUrl'); // Share link
    const directUrlEl = document.getElementById('directUrl'); // (unused here)

    // Event listener for form submission
    form.addEventListener('submit', async (e) => {
      e.preventDefault(); // Prevent full-page reload
      statusEl.textContent = 'Uploading...'; 
      result.style.display = 'none';

      const fd = new FormData(); // FormData object for sending file
      const file = document.getElementById('file').files[0];
      if (!file) {
        statusEl.textContent = 'Pick a file first.';
        return;
      }
      fd.append('file', file); // Attach file to request

      try {
        // Send file to backend /upload route
        const res = await fetch('/upload', { method: 'POST', body: fd });
        if (!res.ok) throw new Error('Upload failed');
        const data = await res.json();

        // Show returned page URL
        pageUrlEl.textContent = data.pageUrl;
        pageUrlEl.href = data.pageUrl;

        // Display result card and reset form
        result.style.display = 'block';
        statusEl.textContent = 'Done!';
        form.reset();
      } catch (err) {
        // Handle error
        statusEl.textContent = 'Error: ' + err.message;
      }
    });
  </script>
</body>
</html>

When a user visits the page, they’ll see a simple upload form with a file picker. They can select an image from their computer and click Upload. Then JavaScript intercepts the form submission using addEventListener('submit'), prevents the browser from doing a full page refresh, and instead, packages the selected file into a FormData object.

That file is then sent to the server with a fetch call to the /upload route. If the server responds successfully, the JSON returned contains a pageUrl. This URL is displayed inside the result card, which was initially hidden. The user can now copy this link and share it with others.

If something goes wrong, like no file being selected, the server erroring out, or the upload failing, the script updates the status message to inform the user.

Here’s how it looks to the user.

Index.html

Now let’s create the backend using server.js file.

import path from "path"; // For working with file paths
import express from "express"; // Web framework to handle HTTP routes
import multer from "multer"; // Middleware for handling file uploads
import crypto from "crypto"; // Used to generate random unique IDs
import dotenv from "dotenv"; // Loads environment variables from .env file
import { fileURLToPath } from "url"; // For handling ES module file paths
import {
  S3Client,
  PutObjectCommand,
  HeadObjectCommand,
  GetObjectCommand,
} from "@aws-sdk/client-s3"; // AWS SDK commands for S3 operations
import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; // To generate temporary signed URLs

dotenv.config(); // Load environment variables

// Setup paths for __dirname and __filename in ES modules
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

// Bucket name from environment
const S3_BUCKET = process.env.S3_BUCKET;

// Create an S3 client (works with Sevalla-compatible storage as well)
const s3 = new S3Client({
  region: "auto", // Auto-region for Sevalla
  endpoint: process.env.ENDPOINT, // Custom endpoint for object storage
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID, // From .env
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, // From .env
  },
});

// Initialize Express app
const app = express();

// Serve static files (like index.html, CSS, JS) from "public" folder
app.use(express.static(path.join(__dirname, "public")));

// Multer setup: store uploaded files in memory (not on disk)
// Limit file size to 10MB
const upload = multer({
  storage: multer.memoryStorage(),
  limits: { fileSize: 10 * 1024 * 1024 },
});

// ---------- ROUTE 1: GET / ----------
// Serves the main HTML file (upload form)
app.get("/", (req, res) => {
  res.sendFile(path.join(__dirname, "public", "index.html"));
});

// ---------- ROUTE 2: POST /upload ----------
// Handles image uploads and stores them in object storage
app.post("/upload", upload.single("file"), async (req, res) => {
  try {
    // Check if file exists
    if (!req.file) return res.status(400).json({ error: "file is required" });

    // Generate a random ID for the file
    const id = crypto.randomUUID().replace(/-/g, "");
    const key = id;

    // Create a PutObjectCommand to upload file to S3/Sevalla
    const put = new PutObjectCommand({
      Bucket: S3_BUCKET,
      Key: key,
      Body: req.file.buffer,
      ContentType: req.file.mimetype,
      Metadata: {
        originalname: req.file.originalname || "",
      },
    });

    // Upload the file
    await s3.send(put);

    // Build a page URL for retrieving the image later
    const baseUrl = `${req.protocol}://${req.get("host")}`;
    const pageUrl = `${baseUrl}/i/${id}`;

    // Respond with the page URL
    res.json({ id, pageUrl });
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: "upload_failed" });
  }
});

// ---------- ROUTE 3: GET /i/:id ----------
// Redirects to a signed URL for secure access to the uploaded file
app.get("/i/:id", async (req, res) => {
  const { id } = req.params;
  const key = id;

  try {
    // Ensure the object exists in storage
    await s3.send(new HeadObjectCommand({ Bucket: S3_BUCKET, Key: key }));

    // Create a signed URL valid for 1 hour
    const command = new GetObjectCommand({ Bucket: S3_BUCKET, Key: key });
    const signedUrl = await getSignedUrl(s3, command, { expiresIn: 3600 });

    // Redirect user to the signed URL
    return res.redirect(302, signedUrl);
  } catch (err) {
    console.error(err);
    return res.status(404).send("Not found");
  }
});

// ---------- Boot the Server ----------
app.listen(process.env.PORT || 3000, () => {
  console.log(`Image host server listening for requests...`);
});

Route 1: GET /

This is the entry point of the app. When you open the browser and go to the root URL, it serves the index.html file from the public folder. That file contains the upload form where the user can select an image and submit it.

Route 2: POST /upload

This is where the magic happens. When a user selects an image and clicks “Upload,” the file is sent to this endpoint. Multer handles the file upload in memory, and then the file is pushed to object storage using the PutObjectCommand. A random unique ID is generated as the key for the file. Once uploaded, the server responds with a pageUrl that can be used to view the uploaded image later.

Route 3: GET /i/:id

This route retrieves an uploaded image. Instead of serving the file directly, it generates a signed URL valid for one hour using getSignedUrl. This signed URL gives temporary access to the file stored in object storage. The server then redirects the user to that signed URL. If the file doesn’t exist, it returns a 404 error.

Before you run this code, we need access to the object storage and add the value in an environment file. The code you see process.env fetches these values and helps us authenticate with the object storage to read and write files.

How to Create Your Object Storage

Login to Sevalla and click “Object Storage”. Click “Create Object Storage” and give it a name.

Object Storage Creation

Once created, click “Settings” and you will see the access key and secret key. We need these four values

  • Bucket name

  • Endpoint URL

  • Access Key

  • Secret Key

Object Storage Access Keys

Copy them into a file named .env within your project.

AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID_HERE
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY_HERE
S3_BUCKET=YOUR_BUCKET_NAME_HERE
ENDPOINT=YOUR_ENDPOINT_URL_HERE

Additionally, enable public access in the settings so that you can push files from your local environment.

public access enabled

Testing the Application Locally

Let’s make sure our code works locally.

node server.js

Go to http://localhost:3000/ and try uploading a file. It should give you the URL to view the file after a successful upload.

File upload success

You can visit the URL to see your uploaded file. You can also double check if it has been uploaded using the Object Storage UI.

Object Storage UI

Great. We have built a simple image hosting and sharing service. Now let’s get this into the cloud.

How to Deploy Your Project on Sevalla

First, push your project to GitHub or fork my repository. Then log in to your Sevalla dashboard and create a new application.

Create application

Connect your GitHub account, choose the repository that contains your image hosting service, and select the branch you want to deploy. Sevalla will automatically detect that it is a Node.js project and install dependencies. It will also run the application on the specified port.

To configure AWS credentials and bucket information, go to the environment variables section in your app and add your AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, and S3_BUCKET_NAME. These values will be injected into your application at runtime, ensuring that sensitive data is not hardcoded into your source code.

Adding environment variables

Once environment variables are added, go to “Overview” and click “Deploy”.

fbcfcd74-d74e-43ad-9b99-7f02421cf5df

Wait for a few minutes. Once the deployment is complete, Sevalla will give you a live URL. Click “Visit APP” to go to your application’s page.

Live url

Congratulations! Your app is now live. You can share the URL with others or even add a custom domain to your app to have your own image hosting solution.

Why This Project Matters

This project is more than just a coding exercise. It teaches you how modern applications manage files at scale, introduces you to object storage, and shows how to integrate cloud services into your own projects.

With Sevalla, you also learned how to deploy production-ready applications, giving you the full cycle from local prototype to live cloud service.

For developers building blogs, mobile apps, or even internal tools, the ability to host images reliably and at scale is invaluable. With object storage and a simple Node.js service, you can avoid reinventing the wheel and rely on proven cloud infrastructure.

Conclusion

We began by exploring object storage and why it is ideal for handling files like images. We then built a Node.js application that accepts uploads, stores them in Sevalla’s Object Storage, and returns accessible URLs. Finally, we deployed the application on Sevalla, turning a local project into a live image hosting service. Along the way, you gained not only working code but also a deeper understanding of how to build cloud-native services.

By completing this project, you now have a working image hosting service you can extend and adapt. You could add features like authentication, image resizing, or even a better front-end interface with drag-and-drop UI. Most importantly, you have experienced how development and deployment fit together in modern software.

]]> <![CDATA[ How to Use Nano Banana for Image Generation - Explained with Code Examples ]]> https://www.freecodecamp.org/news/nano-banana-for-image-generation/ 68cd5897ffbf18457f7bb85a Fri, 19 Sep 2025 13:20:23 +0000 AI is changing the image generation and editing process into a smooth workflow. Now, with just a single prompt, you can tell your computer to generate or edit an existing image. Google just launched its new model for image generation or editing, "Nano Banana" – Gemini 2.5 Flash. It's a powerful, nimble tool that's changing how we think about image generation and manipulation, and it's something you'll definitely want in your developer toolkit.

In this article, you will learn how to use “Nano Banana” for Image Generation using Gemini’s 2.5 Flash Image. So, let’s get started!

Table of Contents

What is "Nano Banana"?

Nano Banana is the latest image-editing cum generation tool from Google DeepMind. Forget the formal jargon for a second. Imagine you have an incredibly talented, lightning-fast artist at your beck and call. You can describe anything to them – "an astronaut riding a horse on the Moon" – and poof, it appears. Or, you hand them a picture of your dog and say, "Make the dog wear a cap on his head," and they do it instantly, keeping your cat looking like your dog.

That's essentially Nano Banana. It's an advanced AI model from the Gemini family, specifically engineered for rapid, intelligent image generation and nuanced editing. It understands your natural language commands, enabling you to bring complex visual ideas to life or make surgical changes to existing images with surprising ease.

Why "Nano Banana"?

Because it's small (flash!), packed with goodness, and leaves you feeling like you just peeled back a new layer of creative possibility. It's fast, efficient, and incredibly versatile.

The Superpowers You Get:

  • Prompt-Perfect Editing: Want to change a background, alter a pose, or add a specific object? Just ask. Nano Banana understands and executes.

  • Character Consistency: This is a big one. If you're creating a story or a series of images, maintaining the look of a specific character or object is crucial. Nano Banana excels at this, ensuring your protagonist looks the same whether they're in a forest or on the moon.

  • Visual Mashups (Multi-Image Fusion): Got a few different visual elements you want to combine seamlessly? It can blend them into a cohesive new image.

and much more!

Interested? Let's get our hands dirty. But wait! To use “Nano Banana, “ you have two ways to do this:

  1. Using Google AI Studio: The simplest and easiest way to generate or edit images in Google Studio. This is a web-based tool that gives you direct access to the Gemini models without writing a single line of code. It's the absolute best place to test and start, and is useful for developers and non-developers, also. Also, there's no need to install libraries, manage API keys, or write any code

  2. Building with the Gemini API: This is beneficial if you want more custom solutions for your application. For any serious application—whether it's a web app, a mobile app, or a backend service—you'll need to integrate directly with the Gemini API. This is where the real power lies, as it allows you to automate tasks and create interactive experiences.

In this tutorial, you will see how we can use this tool in our own applications, using nothing but Python. So, let’s get started.

How to Set Up Your Project

Step 1: Get an API key from Google Gemini

The very first step for using “Nano Banana” is to get an API key. Head over to Google AI Studio, click on “Create API key“, and generate a new one by specifying a project from your existing Google Cloud projects.

API key generated from Google Gemini

Once you have generated an API key, save it securely somewhere.

Step 2: Install the SDK and Other Dependencies

Open your terminal and run:

pip install google-generativeai pillow python-dotenv

We’ll use Pillow for easy image handling and python-dotenv to safely manage our API key.

Step 3: Set Up Your Environment

It’s crucial to keep your API key out of your code for security. For this, we usually use environment variables. So, create a file named .env in your project root and add your API key:

GEMINI_API_KEY="YOUR_API_KEY_HERE"

Step 4: Image Generation & Editing

Example 1: Text-to-Image Generation

Text-to-Image is like an artist who can draw anything you describe. In this, you simply write the prompt (a sentence or a description), even a very detailed one, and the AI will generate a unique, high-quality image that matches your description. It’s perfect for bringing your most imaginative ideas to life with just a few words.

import os
import google.generativeai as genai
from PIL import Image
from io import BytesIO
from dotenv import load_dotenv

# Configuration
load_dotenv()
genai.configure(api_key=os.getenv("GEMINI_API_KEY"))
model = genai.GenerativeModel('gemini-2.5-flash-image-preview')

# Prompt, Image, and Response Setup
prompt = "A golden retriever puppy sitting in a field of daisies, bright and cheerful"
output_filename = "text_to_image_result.png"

# saving image helper function from text prompt response
def save_image_from_response(response, filename):
    """Helper function to save the image from the API response."""
    if response.candidates and response.candidates[0].content.parts:
        for part in response.candidates[0].content.parts:
            if part.inline_data:
                image_data = BytesIO(part.inline_data.data)
                img = Image.open(image_data)
                img.save(filename)
                print(f"Image successfully saved as {filename}")
                return filename
    print("No image data found in the response.")
    return None

def main():
    print(f"Generating image for prompt: '{prompt}'...")
    response = model.generate_content(prompt)
    save_image_from_response(response, output_filename)

if __name__ == "__main__":
    main()

Output:

A golden retriever puppy sitting happily in a sunny meadow filled with white daisies, surrounded by bright green grass and a cheerful, vibrant atmosphere.

The code used in the example handles everything needed to communicate with the Gemini API and save the image.

  • First, we import the required libraries and load the API key from .env using load_dotenv(). This makes the key available so we can connect to Google’s service with genai.configure().

  • The model we’re using is gemini-2.5-flash-image-preview, which is designed for fast image generation.

  • We define a prompt (“A golden retriever puppy...”) and a filename for saving the image.

  • The helper function save_image_from_response(...) looks at the API’s response, extracts the raw image data, and saves it as a PNG file.

  • In main(), we call the model with the prompt, then pass the response to the helper function to save the result.

  • The if __name__ == "__main__": block ensures the script runs only when executed directly, not when imported.

Example 2: Image-to-Image Editing

Image-to-Image is like a photo editor. Instead of starting from scratch, you can upload an existing picture and describe how to change it. For instance, you can request background removal, addition of new objects, or even a complete artistic style change.

import os
import google.generativeai as genai
from PIL import Image
from io import BytesIO
from dotenv import load_dotenv

# Configuration
load_dotenv()
genai.configure(api_key=os.getenv("GEMINI_API_KEY"))
model = genai.GenerativeModel('gemini-2.5-flash-image-preview')

# Prompt, Image, and Response Setup
input_image_path = "input_dog.png"
prommpt = "Make the dog wear a small wizard hat and spectacles."
output_filename = "edited_image_result.png"

# saving image helper function from text prompt response
def save_image_from_response(response, filename):
    """Helper function to save the image from the API response."""
    if response.candidates and response.candidates[0].content.parts:
        for part in response.candidates[0].content.parts:
            if part.inline_data:
                image_data = BytesIO(part.inline_data.data)
                img = Image.open(image_data)
                img.save(filename)
                print(f"Image successfully saved as {filename}")
                return filename
    print("No image data found in the response.")
    return None

def main():
    print(f"Editing image '{input_image_path}' with prompt: '{prommpt}'...")
    try:
        img_to_edit = Image.open(input_image_path)
        response = model.generate_content([prommpt, img_to_edit])
        save_image_from_response(response, output_filename)
    except FileNotFoundError:
        print(f"Error: The file '{input_image_path}' was not found.")

if __name__ == "__main__":
    main()

Output:

A before and after image of a playful dog wearing a small pointed wizard hat and round spectacles, sitting upright with a charming and magical look, giving a whimsical, storybook-like feel.

This code is very similar to the first example, but the key difference is in the core logic.

  • input_image_path: This variable now holds the file path to the image you want to edit.

  • Image.open(input_image_path): This line uses the Pillow library to open your local image file to be used.

  • model.generate_content([prommpt, img_to_edit]): This is the most important part. Unlike before, we now pass a list to the generate_content function that contains both the text prompt and the image object. This tells the API to use the provided image as a starting point for its generation.

  • try...except block: Here, we are handling the errors. It tries to open the image file, and if it fails (because the file isn't there), it will except the FileNotFoundError and print a friendly message to the user instead of crashing.

Example 3: Multi-Image Fusion

Multi-image fusion is like merging two or more images or objects. Upload several images and instruct the AI to blend them into one composite picture seamlessly. This is a tool for creating new scenes, combining people and backgrounds, or creating detailed product mockups.

import os
import google.generativeai as genai
from PIL import Image
from io import BytesIO
from dotenv import load_dotenv

# Configuration
load_dotenv()
genai.configure(api_key=os.getenv("GEMINI_API_KEY"))
model = genai.GenerativeModel('gemini-2.5-flash-image-preview')

# Prompt, Images, and Response Setup
image1_path = "dog_image.png"
image2_path = "cap_image.png"
prompt = "Make the dog from the first image wear the cap from the second image. The cap should fit realistically on the dog's head."
output_filename = "dog_with_cap_result.png"

def save_image_from_response(response, filename):
    """Helper function to save the image from the API response."""
    if response.candidates and response.candidates[0].content.parts:
        for part in response.candidates[0].content.parts:
            if part.inline_data:
                image_data = BytesIO(part.inline_data.data)
                img = Image.open(image_data)
                img.save(filename)
                print(f"Image successfully saved as {filename}")
                return filename
    print("No image data found in the response.")
    return None

def main():
    print(f"Fusing images '{image1_path}' and '{image2_path}'...")
    try:
        img1 = Image.open(image1_path)
        img2 = Image.open(image2_path)
        response = model.generate_content([prompt, img1, img2])
        save_image_from_response(response, output_filename)
    except FileNotFoundError:
        print("Error: One or both image files were not found.")

if __name__ == "__main__":
    main()

Output:

Three-part image showing a golden retriever puppy in a daisy field, a red baseball cap with the letter “A,” and the final edited version where the puppy is wearing the red cap while sitting happily among the daisies

The logic of the code above is an extension of the Image-to-Image example.

  • image1_path and image2_path: These variables hold the paths to the two images you want to fuse or merge.

  • model.generate_content([prompt, img1, img2]): Here, the list passed to the generate_content function contains three items: the text prompt and both image objects. This tells the AI to use the prompt to combine the elements from both images into a single output.

Example 4: Image Restoration

This feature can restore old, faded, or damaged photos. Upload a picture and request Gemini to restore it. This includes sharpening low-quality images, colorizing old black-and-white photos, and enhancing textures, which can make your memories look new again.

import os
import google.generativeai as genai
from PIL import Image
from io import BytesIO
from dotenv import load_dotenv

# Configuration
load_dotenv()
genai.configure(api_key=os.getenv("GEMINI_API_KEY"))
model = genai.GenerativeModel('gemini-2.5-flash-image-preview')

# Prompt, Image, and Response Setup
input_image_path = "old_photo.png"
prompt = "Restore this old, faded photograph. Sharpen the details, remove any scratches or damage, and enhance the colors to make it look like a new, high-quality photo."
output_filename = "restored_image_result.png"

def save_image_from_response(response, filename):
    """Helper function to save the image from the API response."""
    if response.candidates and response.candidates[0].content.parts:
        for part in response.candidates[0].content.parts:
            if part.inline_data:
                image_data = BytesIO(part.inline_data.data)
                img = Image.open(image_data)
                img.save(filename)
                print(f"Image successfully saved as {filename}")
                return filename
    print("No image data found in the response.")
    return None

def main():
    print(f"Attempting to restore image: '{input_image_path}'...")
    try:
        old_photo = Image.open(input_image_path)
        response = model.generate_content([prompt, old_photo])
        save_image_from_response(response, output_filename)
    except FileNotFoundError:
        print(f"Error: The file '{input_image_path}' was not found.")

if __name__ == "__main__":
    main()

Output:

Side-by-side comparison of an old photograph and a restored version. The left shows a scratched, sepia-toned photo of a vintage car on a rural road, while the right shows the same scene digitally restored in color, with a blue classic car under a bright sky in a golden countryside

The structure here is identical to the Image-to-Image Editing example because, from a technical perspective, image restoration is a form of image-to-image editing.

  • Now the prompt is where the magic happens. The text prompt explicitly tells the model what to do with the image, outlining the restoration steps like "sharpen the details," "remove scratches," and "enhance the colors." The model's intelligence allows it to understand these abstract instructions and apply them to the visual data to give you a better and a realistic update to your old image.

Beyond the Basics: What Else Can You Do?

This is just the tip of the iceberg! Nano Banana is incredibly versatile. Here are some ideas for where you can take your projects:

  • Batch Processing: Automate the generation of multiple images from a list of prompts.

  • Creative Assets: Design icons, backgrounds, or character sprites for games or apps directly from your Python script.

  • Data Processing: Integrate Nano Banana into a data pipeline to programmatically edit or generate images based on data inputs.

  • AI Art Galleries: Build a backend service that allows users to submit prompts and receive images.

Wrapping Up

"Nano Banana" (Gemini 2.5 Flash Image) isn't just a cool tech tool; it's a practical, powerful tool for developers and creatives alike. With just a few lines of code, you can tap into its capabilities and bring your visual ideas to real life. This streamlined approach makes it easy to get started, experiment, and integrate this visual magic into your projects.

If you found this article helpful and want to discuss AI development, LLMs, or software development, feel free to connect with me on X/Twitter, LinkedIn, or check out my portfolio on my Blog. I regularly share insights about AI, development, technical writing, and much more.

Happy coding, and may your creations be as vibrant as a field of fresh bananas!

]]> <![CDATA[ How to Enhance Images with Neural Networks ]]> https://www.freecodecamp.org/news/how-to-enhance-images-with-neural-networks/ 68b8e1073c8fb81fc2265eef Thu, 04 Sep 2025 00:44:55 +0000 Artificial intelligence is changing how we work with images. What once took hours in Photoshop can now happen in seconds with AI-powered tools. You can take a blurry picture, enlarge it without losing sharpness, fix the lighting, remove unwanted noise, or even bring color to a black-and-white photo, all with a single click.

The magic you see in these tools is powered by algorithms which are trained AI models that understand how images should look and then reconstruct them accordingly. These models have studied millions of examples to learn patterns, textures, and details, so they can “predict” what’s missing and fill it in naturally.

For developers, photographers, and content creators, knowing the basics of these algorithms can help you pick the right tools for your workflow. Even if you never plan to code an AI model yourself, this knowledge will help you make better choices for image processing, web apps, or creative projects.

Let’s look at five of the most important algorithms used in AI image enhancement today. Along the way, you’ll see real-world tools that use these algorithms and how you can try them yourself.

Table of Contents

Image Colorization

Automatic image colorization might be the most visually dramatic AI enhancement of all. It takes a black-and-white image and predicts the colors that should be there, often producing results that look like the photo was taken in full color.

The AI behind this uses convolutional neural networks (CNNs) trained on huge datasets of color images. The model sees both the grayscale and the color versions during training, so it learns how certain objects typically appear. For example, it might learn that grass is usually green, the sky is often blue, and human skin falls within a certain range of tones.

Image Colorization

One of the most famous models is DeOldify, which combines CNNs with GANs. The GAN setup helps refine the results, making colors more natural and avoiding strange or overly bright tones.

Colorization has practical uses beyond restoring old family photos. It’s used in film restoration, historical projects, digital storytelling, and even concept art.

See Image Colorization in action.

GAN-Based Image Enhancement

GANs, or Generative Adversarial Networks, are one of the most powerful AI techniques in image enhancement. They consist of two neural networks: the generator, which tries to create realistic-looking images, and the discriminator, which evaluates them. Over many iterations, the generator becomes extremely good at producing images that pass as real.

Image Enhancement

In image retouching, GANs can handle many tasks at once, like fixing lighting, improving sharpness, enhancing textures, and even subtly changing elements to make the picture more appealing. Because GANs learn from real-world images, the results often feel more natural than traditional editing filters.

GAN-based retouching is used in professional portrait editing, e-commerce product photos, real estate listings, and even game asset creation. It’s also behind many “one-click enhance” buttons you see in modern apps.

See a GAN powered photo enhancer here.

Noise Reduction (Denoising Autoencoders)

Noise in images looks like random specks of color or brightness that shouldn’t be there. It often happens in low-light photos or in images taken with high ISO settings. Noise makes photos look grainy and less professional.

Traditional noise removal methods simply blurs the image to hide the noise, but this also destroyed fine details. AI noise reduction works differently.

Denoising Autoencoders, one of the most common approaches, learn from pairs of images—one clean and one noisy. The AI studies how noise distorts details, then learns to reverse the process.

Image denoising

When you pass a noisy photo through a denoising autoencoder, it removes the noise while preserving edges, textures, and important small details.

Noise reduction isn’t just for photography. It’s also used in document scanning to make text easier to read, medical imaging to clarify scans, cleaning up screenshots or UI mockups for presentations

See Noise Reduction in action here.

Image Upscaling using Super-Resolution

Super-resolution is the process of increasing the resolution of an image to make it sharper and larger without simply stretching the pixels.

In the past, enlarging a small image just made it blurry. AI super-resolution works differently. It studies the image, detects patterns, and then generates new pixels that match what would have been there in a higher-quality original.

One of the first big breakthroughs was SRCNN (Super-Resolution Convolutional Neural Network). SRCNN works by breaking the image into patches, analyzing them, and then predicting what higher-resolution patches should look like. This early approach was effective but sometimes produced overly smooth images.

Then came ESRGAN (Enhanced Super-Resolution Generative Adversarial Network), which took things further. ESRGAN uses a GAN architecture, a generator creates enhanced images, while a discriminator judges how real they look. Through this back-and-forth training, the generator learns to produce fine textures like hair strands, fabric weaves, or building details that look realistic to the human eye.

Image Upscaling

Super-resolution is widely used in e-commerce (for clearer product photos), printing (turning web images into high-resolution posters), and web apps (making user-uploaded images look professional).

See Super resolution powered image upscaler in action.

Artifact Removal

When a JPEG image is heavily compressed, it develops blocky patches, fuzzy edges, and strange halos around lines. These are called compression artifacts, and they appear because JPEG reduces file size by removing fine detail. Traditional fixes blur the image to hide these defects, but that also softens important edges and textures.

JPEG Aartifact Removal

FBCNN, or Flexible Blind Convolutional Neural Network, takes a smarter approach. Instead of needing to know the exact compression level beforehand, FBCNN is trained to handle a wide range of artifact severities without extra input. This is what makes it “blind”, it doesn’t require metadata about how the JPEG was compressed. It can adapt its restoration process on the fly.

FBCNN works in two main steps. First, it extracts features from the image, analyzing patterns in edges, textures, and flat areas to identify where artifacts are most likely. Then, it applies a learned mapping to reconstruct what those regions should look like without the damage.

Because it can estimate the compression quality itself, FBCNN avoids the common problem of over-smoothing lightly compressed images or under-restoring heavily compressed ones.

This flexibility makes FBCNN useful in many scenarios: cleaning up low-quality images from social media, restoring graphics and text in screenshots, or preparing old compressed web images for printing. Modern AI tools often integrate FBCNN-style processing as a first step before applying super-resolution or general enhancement.

FBCNN’s ability to adapt without manual tuning makes it one of the most practical and developer-friendly models for real-world JPEG restoration today.

See artifact removal in action.

Why These Algorithms Matter to Developers

Even if you have never trained your own AI model, understanding these algorithms gives you a better sense of what’s possible and how to apply it. Many of the tools mentioned here offer APIs, which means developers can build them into their own apps and websites.

If you run a social platform, you can automatically enhance user-uploaded images before they appear in feeds. If you build e-commerce platforms, you can clean and upscale product images for better sales conversions. If you work in media archiving, you can restore and preserve images without spending hours on manual edits.

The real value comes from knowing which algorithm is right for the problem you’re solving. Super-resolution for enlarging, denoising for cleaning, colorization for restoration, artifact removal for fixing compression, and GAN retouching for overall beautification.

Conclusion

AI image enhancement has moved from research labs to everyday tools, making it possible for anyone to transform low-quality images into something sharp, vibrant, and professional. The algorithms behind these tools like super-resolution, denoising, colorization, artifact removal, and GAN retouching are the building blocks of modern visual AI.

Whether you’re a developer looking to integrate image processing into your app or a creator who wants to improve your visuals, knowing how these algorithms work will help you get the most out of AI. This is only the beginning and future models will be even more precise, faster, and capable of things we haven’t yet imagined. Developers who understand these foundations will be ready to make the most of the next wave of AI-powered creativity.

Hope you enjoyed this article. Signup for my free AI newsletter TuringTalks.ai for more hands-on tutorials on AI. You can also visit my website.

]]> <![CDATA[ How to Blend Images in Rust Using Pixel Math ]]> https://www.freecodecamp.org/news/how-to-blend-images-in-rust-using-pixel-math/ 66cda9b4e220ecb31e3a2239 Tue, 27 Aug 2024 10:25:56 +0000 For anyone looking to learn about image processing as a programming niche, blending images is a very good place to start. It's one of the simplest yet most rewarding techniques when it comes to image processing.

To help your intuition, it's best to imagine an image as a mathematical graph of pixel values plotted along the x and y coordinates. The top right pixel in an image is your origin, which corresponds to an x value of 0 and a y value of 0.

Once you imagine this, any pixel in an image can be read or modified using it's coordinate in this x-y graph. For example, for a square image of size 5px x 5px, the coordinate of the center pixel is 2, 2. You may have expected it to be 3, 3, but image coordinates in this context work similar to array indexes and start from 0 for both axis.

mathematical graph with x and y axis

Approaching image processing this way also helps you address each pixel individually, making the process much simpler.

Prerequisites

The focus of this article is for you to understand and learn how to blend images using the Rust programming language, without going into the details of the language or it's syntax. So being comfortable writing Rust programs is required.

If you're not familiar with Rust, I highly encourage you to learn the basics. Here's an interactive Rust course that can get you started.

Table Of Contents

  1. Introduction

  2. How Image Blending Works

  3. Project Setup

  4. How to Read Pixel Values

  5. How to Blend Functions

    1. Average Blend

    2. Multiply Blend

    3. Lighten Blend

    4. Darken Blend

    5. Screen Blend

    6. Addition Blend

    7. Subtraction Blend

  6. How to Apply Blend Functions To Images

  7. Putting It All Together

  8. Glossary

Introduction

Image blending refers to the technique of merging pixels from multiple images to create a single output image that is derived from all of its inputs. Depending on which blending operation is used, the image output can vary widely given the same inputs.

This technique serves as the basis for many complex image processing tools, some of which you may already be familiar with. Things such as removing moving people from images if you have multiple images, merging images of the night sky to create star trails, and merging multiple noise-heavy images to create a noise reduced image are all examples of this technique at play.

To achieve the blending of images in this tutorial, we will make use of "pixel math", which while not being a truly standard term, refers to the technique of performing mathematical operations on a pixel or set of pixels to generate an output pixel.

For example, to blend two images using the "average" blend mode, you will perform the mathematical average operation on all input pixels at a given location, to generate the output at the same location.

Pixel math is not limited to point operations, which are basically operations performed during image processing that generate a given output pixel based on input pixel from single or multiple images from the same location in the x-y coordinate system.

In my experience so far, the entirety of image processing field is 99% mathematics and 1% black magic. Mathematical operations on pixels and it's surrounding pixels is the basis of image manipulation techniques such as compression, resizing, blurring and sharpening, noise reduction, and so on.

How Image Blending Works

The technique is technically simple to implement. Let's take the example of a simple average blend. Here's how it works:

  1. Read the pixel data of both images into memory, usually into an array for each image.

    • The array is usually 2 dimensional. Each entry in array is another array for color images, the secondary array holds the 3 pixel values corresponding to Red, Green, and Blue color channels.

  2. For each pixel location:

    1. For each channel:
      a. Take the value of the channel from the 2nd image, let's consider it y.
      b. Perform the averaging operation x/2 + y/2.
      c. Save the output value of this operation as the value of the output channel

    2. Save the result of previous operation as the value of the output pixel.

  3. Construct the output image with the same dimensions from the computed data.

You'll notice that pixel math is performed on a per-channel basis. This is always true for the blend modes we cover in this tutorial, but many techniques involve applying blends between the channels themselves and many times within the same image.

Project Setup

Let's get started by setting up a project that gives us a good baseline to work with.

cargo new --bin image-blender
cd image-blender

You will also need a single dependency to help you perform these operations:

cargo add image

image is a Rust library we'll use to work with images of all of the standard formats and encodings. It also helps us convert between various formats, and provides easy access to pixel data as buffers.

For more information on the image crate, you can refer to the official documentation.

To follow along, you can use any two images, the only requirement being that they should be of the same size and in the same format. You can also find the images used in this tutorial, along with complete code, in the GitHub repository here.

How to Read Pixel Values

The first step is to load the images and read their pixel values into a data structure that facilitates our operation. For this tutorial, we're going to use a Vec of arrays (Vec<[u8; 3]>). Each entry in the outer Vec represents a pixel, and the channel-wise values of each pixel are stored in [u8; 3] array.

Let's start by creating a new file to hold this code called io.rs.

// src/io.rs

use image::GenericImageView;

pub struct SourceData {
    pub width: usize,
    pub height: usize,
    pub image1: Vec<[u8; 3]>,
    pub image2: Vec<[u8; 3]>,
}

pub fn read_pixel_data(image1_path: String, image2_path: String) -> SourceData {
    // Open the images
    let image1 = image::open(image1_path).unwrap();
    let image2 = image::open(image2_path).unwrap();

    // Compute image dimensions
    let (width, height) = image1.dimensions();
    let (width, height) = (width as usize, height as usize);

    // Create arrays to hold input pixel data
    let mut image1_data: Vec<[u8; 3]> = vec![[0, 0, 0]; width * height];
    let mut image2_data: Vec<[u8; 3]> = vec![[0, 0, 0]; width * height];

    // Iterate over all pixels in the input image, along with their positions in x & y
    // coordinates.
    for (x, y, pixel) in image1.to_rgb8().enumerate_pixels() {
        // Compute the raw values for each channel in the RGB pixel.
        let [r, g, b] = pixel.0;

        // Compute linear index based on 2D index. This is basically computing index in
        // 1D array based on the row and column index of the pixel in the 2D image.
        let index = (y * (width as u32) + x) as usize;

        // Save the channel-wise values in the correct index in data arrays.
        image1_data[index] = [r, g, b];
    }

    // Iterate over all pixels in the input image, along with their positions in x & y
    // coordinates.
    for (x, y, pixel) in image2.to_rgb8().enumerate_pixels() {
        // Compute the raw values for each channel in the RGB pixel.
        let [r, g, b] = pixel.0;

        // Compute linear index based on 2D index. This is basically computing index in
        // 1D array based on the row and column index of the pixel in the 2D image.
        let index = (y * (width as u32) + x) as usize;

        // Save the channel-wise values in the correct index in data arrays.
        image2_data[index] = [r, g, b];
    }

    SourceData {
        width,
        height,
        image1: image1_data,
        image2: image2_data,
    }
}

How to Blend Functions

The next step is to implement the blending functions, which are pure functions that take two pixel values as input and return the output value. This is implemented through the BlendOperation trait defined below. Let's create a new file to host all the operations called operations.rs.

// src/operations.rs

pub trait BlendOperation {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3];
}

Next, we need to implement this trait for all of the blending methods we want to support.

For showcasing the result of each of the blending modes, the following two input images are blended together

Source image 1: Fireflies in a dark forest area

Source image 2: Fireflies in a bright forest area

Average Blend

An average blend involves channel-wise averaging the input pixel values to get the output pixel.

// src/operations.rs

pub struct AverageBlend;

impl BlendOperation for AverageBlend {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3] {
        [
            pixel1[0] / 2 + pixel2[0] / 2,
            pixel1[1] / 2 + pixel2[1] / 2,
            pixel1[2] / 2 + pixel2[2] / 2,
        ]
    }
}

Result of average blending source images

Multiply Blend

A multiply blend involves channel-wise multiplication of input pixel values after they've been normalized[¹] to get the output pixel. The output pixel is then rescaled back to the original range by multiplying with 255.

// src/operations.rs

pub struct MultiplyBlend;

impl BlendOperation for MultiplyBlend {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3] {
        [
            ((pixel1[0] as f32 / 255. * pixel2[0] as f32 / 255.) * 255.) as u8,
            ((pixel1[1] as f32 / 255. * pixel2[1] as f32 / 255.) * 255.) as u8,
            ((pixel1[2] as f32 / 255. * pixel2[2] as f32 / 255.) * 255.) as u8,
        ]
    }
}

Result of multiply blending source images

Lighten Blend

Lighten blend involves channel-wise comparison of input pixel values, selecting the pixel with higher value (intensity) as the output pixel.

// src/operations.rs

pub struct LightenBlend;

impl BlendOperation for LightenBlend {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3] {
        [
            pixel1[0].max(pixel2[0]),
            pixel1[1].max(pixel2[1]),
            pixel1[2].max(pixel2[2]),
        ]
    }
}

Result of lighten blending source images

Darken Blend

Darken blend is the opposite operation of lighten blend. It involves channel-wise comparison of input pixel values, selecting the pixel with least value (intensity) as the output pixel.

// src/operations.rs

pub struct DarkenBlend;

impl BlendOperation for DarkenBlend {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3] {
        [
            pixel1[0].min(pixel2[0]),
            pixel1[1].min(pixel2[1]),
            pixel1[2].min(pixel2[2]),
        ]
    }
}

Result of darken blending source images

Screen Blend

Screen blend refers to multiplying the inverse of two images, and then inverting the result. In our implementation, the pixels first need to be normalized[¹]. The normalized[¹] values are then inverted by subtracting them from 1, then they're multiplied and inverted again.

Finally, the output is multiplied by 255 to de-normalize the output pixel value.

// src/operations.rs

pub struct ScreenBlend;

impl BlendOperation for ScreenBlend {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3] {
        [
            ((1. - ((1. - (pixel1[0] as f32 / 255.)) * (1. - (pixel2[0] as f32 / 255.)))) * u8::MAX as f32) as u8,
            ((1. - ((1. - (pixel1[1] as f32 / 255.)) * (1. - (pixel2[1] as f32 / 255.)))) * u8::MAX as f32) as u8,
            ((1. - ((1. - (pixel1[2] as f32 / 255.)) * (1. - (pixel2[2] as f32 / 255.)))) * u8::MAX as f32) as u8,
        ]
    }
}

Result of screen blending source images

Addition Blend

Addition blend involves adding the input values and then clamping the result to the maximum range of the color depth we're targeting. In this case, that would be 0-255 as we're targeting 8-bit color depth.

We also have to convert the values to u16 in order to avoid loss of value due to overflow. We can also use normalized[¹] values here to achieve the same result.

// src/operations.rs

pub struct AdditionBlend;

impl BlendOperation for AdditionBlend {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3] {
        [
            (pixel1[0] as u16 + pixel2[0] as u16).clamp(0, u8::MAX as u16) as u8,
            (pixel1[1] as u16 + pixel2[1] as u16).clamp(0, u8::MAX as u16) as u8,
            (pixel1[2] as u16 + pixel2[2] as u16).clamp(0, u8::MAX as u16) as u8,
        ]
    }
}

Result of addition blending source images

Subtraction Blend

Addition blend involves subtracting the input values and then clamping the result to the maximum range of the color depth we're targeting. In this case, that would be 0-255 as we're targeting 8-bit color depth.

We also convert the values to i16 in order to avoid loss of value due to overflow and lack of sign. We can also use normalized[¹] values here to achieve the same result.

// src/operations.rs

pub struct SubtractionBlend;

impl BlendOperation for SubtractionBlend {
    fn perform_operation(&self, pixel1: [u8; 3], pixel2: [u8; 3]) -> [u8; 3] {
        [
            (pixel1[0] as i16 - pixel2[0] as i16).clamp(0, u8::MAX as i16) as u8,
            (pixel1[1] as i16 - pixel2[1] as i16).clamp(0, u8::MAX as i16) as u8,
            (pixel1[2] as i16 - pixel2[2] as i16).clamp(0, u8::MAX as i16) as u8,
        ]
    }
}

Result of subtraction blending source images

How to Apply Blend Functions To Images

The final step is to actually use the blending operations we created previously and apply them to pairs of images.

To achieve this, we need a function that can take the SourceData type we defined previously as input, along with a blending operation as the arguments, and gives us the final output buffer. Let's start by creating a new file for it called blend.rs.

// src/blend.rs

use image::{ImageBuffer, Rgb};
use crate::{operations::BlendOperation, SourceData};

impl SourceData {
    pub fn blend_images(&self, operation: impl BlendOperation)  -> ImageBuffer<Rgb<u8>, Vec<u8>> {
        let SourceData {
            width,
            height,
            image1,
            image2,
        } = self;

        // Create a new buffer that has the same size as input images, which will serve as our output data
        let mut buffer = ImageBuffer::new(*width as u32, *height as u32);

        // Iterate over all pixels in the output buffer, along with their coordinates
        for (x, y, output_pixel) in buffer.enumerate_pixels_mut() {
            // Compute linear index form x & y coordinates. In other words, you have the
            // row and column indexes here, and you want to compute the array index based
            // on these two positions.
            let index = (y * *width as u32 + x) as usize;

            // Store pixel values in the given position into variables
            let pixel1 = image1[index];
            let pixel2 = image2[index];

            // Compute the blended pixel and convert it into the `Rgb` type, which is then
            // assigned to the output pixel in the buffer.
            *output_pixel = Rgb::from(operation.perform_operation(pixel1, pixel2));
        }

        buffer
    }
}

Putting It All Together

It's now time to make use of all the new things you've learnt so far, and put them together in main.rs file.

// src/main.rs

mod blend;
mod io;
mod operations;

use io::*;
use operations::{
    AdditionBlend, AverageBlend, DarkenBlend, LightenBlend, MultiplyBlend, ScreenBlend,
    SubtractionBlend,
};

fn main() {
    let source_data = read_pixel_data("image1.jpg".to_string(), "image2.jpg".to_string());

    let output_buffer = source_data.blend_images(AdditionBlend);
    output_buffer.save("addition.jpg").unwrap();

    let output_buffer = source_data.blend_images(AverageBlend);
    output_buffer.save("average.jpg").unwrap();

    let output_buffer = source_data.blend_images(DarkenBlend);
    output_buffer.save("darken.jpg").unwrap();

    let output_buffer = source_data.blend_images(LightenBlend);
    output_buffer.save("lighten.jpg").unwrap();

    let output_buffer = source_data.blend_images(MultiplyBlend);
    output_buffer.save("multiply.jpg").unwrap();

    let output_buffer = source_data.blend_images(ScreenBlend);
    output_buffer.save("screen.jpg").unwrap();

    let output_buffer = source_data.blend_images(SubtractionBlend);
    output_buffer.save("subtraction.jpg").unwrap();
}

You can now run the program using the following command, and you should have all the images generated and saved in the project folder:

cargo run --release

As you might have guessed already, this implementation only works for 8-bit RGB images. This code, however, can be extended very easily to support the other color formats such as 8-bit Luma (Monochrome), 16-bit RGB (Many RAW camera images), and so on.

I highly encourage you to try that out. You can also reach out to me for help with anything in this tutorial or with extending the code in this tutorial. I'd be happy to answer all your queries. Email is the best way to reach me, you can email me at anshul@anshulsanghi.tech.

Glossary

Normalization refers to the process of rescaling the pixel values so that the values are in floating point format and are in the range of 0-1. For example, for an 8 bit image, the color black is represented by 0 (0 in de-normalized value) and the color white is represented by 1 (255 in de-normalized value). Intermediary decimal values between 0 & 1 represent different intensities of the pixel between black and white. Normalization is done for many different reasons such as:

  • Preventing overflows during calculations.

  • Re-scaling images to the same range irrespective of their individual color depth.

  • Expanding possible dynamic range of the image.

Enjoying my work?

Consider buying me a coffee to support my work!

?text=Buy%20me%20a%20coffee&emoji=%E2%98%95&slug=anshulsanghi&button_colour=FFDD00&font_colour=000000&font_family=Cookie&outline_colour=000000&coffee_colour=ffffff

Till next time, happy coding and wishing you clear skies!

]]> <![CDATA[ Rust Tutorial – How to Build a Naïve Star Detector for Images ]]> https://www.freecodecamp.org/news/rust-tutorial-naive-star-detector-for-images/ 66bb57a029aa951a4c0628bc Tue, 16 Apr 2024 19:34:07 +0000 Star detection is a crucial step in many of the processing and analysis routines that we perform on astronomical images. It is extremely important for a process called plate-solving, which is the process of figuring out which part of the sky an image shows, or which part of the sky your telescope is pointed at.

All modern telescope mounts can make use of plate solving software to automatically figure out where they're pointed at, and in which direction they need to move to point at the correct location.

Star detection, sometimes, is also used in correcting the effect of atmosphere on the sharpness of targets such as galaxies. It is also crucial for combining astronomical images from multiple nights, telescopes, locations and so on into a single output image that has a very high signal-to-noise ratio.

With this tutorial, I'd like to introduce a very naïve technique for detecting stars in an image.

A quick note:

Star detection is a very complex topic, and I've only scratched the surface both in my own understanding and in this article.

The steps I use and describe in this article are derived from public documentation on existing real world applications (both for star detection and for edge detection), as well as some blog posts from incredibly knowledgeable people (which I link to at the end of the article, be sure to check them out).

As such, this implementation is intended for learning purposes only.

Before You Read

Prerequisites for the first part of the tutorial

The process described builds upon the concept of multi-scale processing of images using a trous wavelet transform. If you're not aware of what that is, I encourage you to learn more about it using my previous article that I just linked to, and then come back to this one.

This article also assumes that you have a basic understanding of Centroids. Just knowing what they mean is enough, as you don't have to calculate them yourself. Since the article focuses on image processing and analysis, a basic understanding of how pixels work in digital format is helpful, but not mandatory.

Prerequisites for the second part of this tutorial

Here, we focus on implementing the algorithm using the Rust programming language, without going much into the details of the language itself. So being comfortable writing Rust programs, and comfortable reading crate documentations is required.

If this is not you, you can still read Part 1 and learn the technique, and then maybe you'll want to then try it out in a language of your choice.

If you're not familiar with Rust, I highly encourage you to learn the basics. Here's an interactive Rust course that can get you started.

Table of contents

  1. How Star Detection Works
    1. What is Star Detection?
    2. How Star Detection Works
    3. An Intermediary Look at the Process
    4. Picking It Apart
  2. How to Implement it in Rust
    1. Prerequisites
    2. How to read and decompose the input image
    3. Noise reduction
    4. How to optimize the threshold and binarization
    5. How to construct polygons around stars
    6. How to detect star size and location using contours
    7. How to encapsulate the process
    8. How to test the implementation on astronomical images
    9. How to optimize minimum star count
    10. But there is one more thing...
  3. Further Reading
  4. Wrapping Up

How Star Detection Works

Since this process involves a lot of steps, let's see how it works, with an increasing level of detail about what actually happens as we go along. With each increasing level, we'll be unwrapping the black box bit by bit.

What is Star Detection?

Star detection, at it's simplest form, involves isolating the stars from the rest of the image, and then performing edge detection on it.

Image 1. Input image

Image 2. Detected stars visualised using green circles

How Star Detection Works

First, you try to extract away the pixels that you think might be stars from the rest of the pixels in the image. This new image, that only contains the extracted pixels, is then analysed using edge detection techniques to find the star positions in 2D space.

Image 1. Input image

Image 2. Extracted pixels that are potentially stars

Image 3. Detected stars visualised using green circles

An Intermediary Look At The Process

Then, you decompose your input image into multiple layers, each layer containing a part of the original data such that adding all layers gives us back the original data.

You then isolate the layers that would only contain small sized structures, such as noise and stars, and throw away the rest of the data.

Image Different layers of structure in the image that the input is decomposed into. We throw away the final layer and retain the rest in this example

With this filtered data, you find the edges in the image using the contouring technique (which is explained in the next section). Each contour gives us multiple "points" in the 2D space. You then try to draw a closed shape using the points you have.

Once you've done this, all you need is to find the center of this shape and you have the location of the stars.

Image 1. Input image

Image 2. Image after decomposing into layers and throwing away large scale data

Image 3. Image after binarisation

Image 4. Detected contours visualised using green outlines

Image 5. Detected stars visualised using green circles

Picking It Apart

Using a multi-scale analysis technique facilitated by the à trous transform algorithm, you break down the image into multiple layers, each containing different scaled structures from the original image. You take the layers containing smaller scale structures and throw away the rest.

To these layers, you apply a bilateral denoising filter to reduce noise so that you can ensure that you're only left with stars and not noise that the algorithm might pick up as stars later on.

Image Different layers of structure in the image that the input is decomposed into. We throw away the final layer and retain the rest in this example.

Image 1. Input image

Image 2. Image after decomposing into layers and throwing away large scale data

Image 3. Noise reduced image

Once you've filtered out the noise, you binarize your image using thresholding. Thresholding and binarization is the process of converting all of the pixels to either pure black or pure white, so that they're easier to work with. You can do this by selecting a certain intensity value, and all pixels with intensity less than this become black and all pixels with intensity more than this become white.

To find the optimum intensity value to binarize the image with, you define a minimum number of stars that you expect to find in the image, which is usually determined based on what you actually need to do with your star locations.

In our example, we'll start with a minimum of 500 and slowly push it to the limit of the sample image to see what happens.

Image Binarizing noise-reduced and wavelet filtered image

This makes the process of edge detection (which is the next step in our process) using contouring much more reliable.

Contouring is a term that describes the process of figuring out where the structures are in your image, and drawing a border along those structures – these are known as contours.

It is similar to edge-detection, but edge-detection helps you differentiate between individual neighbouring pixels, whereas contours are designed to work with a complete boundary of any structures in an image.

The library we'll be using finds the contours in an image using the algorithm proposed by Suzuki and Abe: Topological Structural Analysis of Digitized Binary Images by Border Following. Contouring in this manner will give you a collection of points that lie on the border of each contour.

For each contour it finds, you create a polygon by joining all of the border points within that contour. If this shape is an open shape, then you just extrapolate the final border to create a polygon, which needs to be a closed shape. You then use the centroid formulae on this polygon to find the center of mass of your shape, which gives you the center of your star (in most cases).

You also need to find the euclidean distances between the center of mass and each border point, the longest of which becomes the size of the star.

Image Contouring the binarized image to find closed polygons around stars visualised here using green outlines

Once you have your star size, you reject any stars that are either smaller than 1 pixel or larger than 24 pixels. These are educated guesses that I use, and they seem to give me the best results for sample images (but this is definitely a potential point of improvement).

After all of this, you should have the x and y coordinates of the star, as well as its size in pixels.

Image Detected stars visualised using green circles around them

We're going to stop there, but there's a lot more that you can do after this step to remove false-positives and fix the centroid/size of stars.

How to Implement it in Rust

Let's create a new library project:

cargo new --lib stardetect-rs && cd stardetect-rs

Prerequisites

You need a couple of dependencies to get started. Let's add them and I'll explain why you need them:

cargo add image imageproc image-dwt geo
  • image is a Rust library we'll use to work with images of all of the standard formats and encodings. It also helps us convert between various formats, and provides easy access to pixel data as buffers.
  • imageproc is another library by the people who created the image library. It's an extension for the same as it implements image processing functions and algorithms for the image lib.
  • image-dwt is my own library (shameless plug) that implements the à trous wavelet decomposition algorithm for image crate. This is needed to break down our image into multiple scales that I mentioned previously.
  • geo is a Rust library that allows us to easily work with geometric types (like points in 2d space), shapes (such as polygons), and algorithms implemented for them. We use this library to build our polygon based on contour data, and to also find the centroid of the polygon that I described above. It also helps us compute euclidean distances between points, which we use for determining star size.

How to read and decompose the input image

You start by reading the input image and decomposing it so that you're only left with stars (and noise).

You need to define a new struct that will act as a wrapper for your input image, and add a constructor for it to create an instance of this struct based on input:

// lib.rs
use image::{DynamicImage, GrayImage};

pub struct StarDetect {
    source: GrayImage,
}

impl From<DynamicImage> for StarDetect {
    fn from(source: DynamicImage) -> Self {
        Self {
            source: source.to_luma8(),
        }
    }
}

You then need to add the ability to extract the first n layers from wavelet decomposition of your image:

// lib.rs

use image_dwt::kernels::LinearInterpolationKernel;
use image_dwt::recompose::{OutputLayer, RecomposableWaveletLayers};
use image_dwt::transform::ATrousTransform;

impl StarDetect {
    fn extract_small_scale_structures(&mut self) {
        let (width, height) = self.source.dimensions();

        // Decompose the image into 8 layers
        let filtered_image = ATrousTransform::new(
            &DynamicImage::ImageLuma8(self.source.clone()),
            8,
            LinearInterpolationKernel,
        )
        // Filter out the residue image and keep the rest
        .filter(|item| item.pixel_scale.is_some())
        // Recompose the first 3 layers into a grayscale image.
        .recompose_into_image(width as usize, height as usize, OutputLayer::Grayscale);

        // Update the source image that we will work with
        // going forward.
        self.source = filtered_image.to_luma8();
    }
}

Noise reduction

Now that you have the input image (which should only contain noise and stars), let's get rid of the noise:

// lib.rs

impl StarDetect {
    fn apply_noise_reduction(&mut self) {
        self.source = imageproc::filter::bilateral_filter(&self.source, 10, 10., 3.);
    }
}

Next, you need to determine the optimum threshold value for a given minimum star count. You find it by picking a value and iteratively optimising it until you hit a star count that's more than the minimum.

How to optimize the threshold and binarization

Start by creating a new file threshold.rs and defining a trait with necessary methods. You need a method to optimise your threshold value and another for performing the binarization operation:

// threshold.rs

pub(crate) trait ThresholdingExtensions {
    fn optimize_threshold_for_star_count(&self, min_star_count: usize) -> u8;
    fn binarize(&mut self, threshold: u8);
}

Let's implement both of these:

// threshold.rs

use crate::centroid::find_star_centres_and_size;
use crate::StarDetect;

impl ThresholdingExtensions for StarDetect {
    fn optimize_threshold_for_star_count(&self, min_star_count: usize) -> u8 {
        // Current star count
        let mut star_count = 0;

        // Starting threshold value
        let mut threshold = u8::MAX;

        // Iterate until you've found the best threshold
        while star_count < min_star_count {
            // Panic if we reach the 0 intensity value while iterating.
            // This means that there are fewer stars than we hoped for.
            if threshold == 0 {
                panic!("Maximum iteration count reached");
            }

            // Reduce threshold to 95% of its previous value.
            // Using this, we check finer and finer differences
            // in threshold for each iteration.
            threshold = (0.95 * threshold as f32) as u8;

            // Clone the source data since we need to modify it
            // without affecting original data.
            let mut source = self.clone();

            // Binarize the source data image using current threshold
            ThresholdingExtensions::binarize(&mut source, threshold);

            // Find the number of stars detected with the current threshold
            star_count = find_star_centres_and_size(&source.source).len();
        }

        threshold
    }

    fn binarize(&mut self, threshold: u8) {
        // Iterate over every pixel in source image
        for pixel in self.source.iter_mut() {
            if *pixel > threshold {
                // If pixel intensity is greater than threshold
                // set it to maximum intensity instead.
                *pixel = u8::MAX;
            } else {
                // Otherwise, set it to 0 intensity.
                *pixel = 0;
            }
        }
    }
}

You might notice that we use the find_star_centres_and_size function when trying to find the optimised threshold value. We'll get to that shortly, as we need to declare some types that will hold the state of our computation before we implement the function.

Create a new file centroid.rs.

Define a new struct that will hold the coordinates and size of the star:

// centroid.rs

use imageproc::point::Point;

#[derive(Eq, PartialEq, Copy, Clone, Debug)]
pub struct StarCenter {
    coord: Point<u32>,
    radius: u32,
}

impl StarCenter {
    pub fn coord(&self) -> &Point<u32> {
        &self.coord
    }
    pub fn radius(&self) -> u32 {
        self.radius
    }
}

We've also defined methods to retrieve these fields. Point is a type provided to you by imageproc crate to store coordinates in an image.

How to construct polygons around stars

We're going to implement this function inside out. We first need a way to construct our polygon from contours. Let's implement that:

// centroid.rs

use geo::LineString;
use imageproc::contours::Contour;

pub(crate) fn construct_closed_polygon(contour: &Contour<u32>) -> LineString<f32> {
    // Create a new line string that connects all points
    // in the contour. This can create either an open
    // or a closed shape.
    let mut line_string = LineString::from_iter(contour.points.iter().map(|point| Coord {
        x: point.x as f32,
        y: point.y as f32,
    }));

    // If it is an open shape, close the shape to create a
    // polygon. This does nothing otherwise.
    line_string.close();

    line_string
}

Contour is a type provided by the imageproc crate, which is what it returns as the result of contouring operation on an image. It contains a list of points that lie on the border of the contour.

LineString is a type provided by geo and is defined by them as "An ordered collection of two or more Coords, representing a path between locations.". In this case, we use this type to construct the polygon shape.

How to detect star size and location using contours

Next, you need a way to compute the StarCenter type we declared previously from contour data:

// centroid.rs

use geo::{Centroid, Coord, EuclideanDistance};

pub(crate) fn filter_map_contour_to_star_centers(contour: &Contour<u32>) -> Option<StarCenter> {
    // If there are no points in the contour
    // it is not a star.
    if contour.points.is_empty() {
        return None;
    }

    if contour.points.len() == 1 {
        // If there's only 1 point in the contour
        // consider it to be the center of the star
        // of size 1px.
        let center = contour.points.first().unwrap();
        let radius = 1_u32;

        return Some(StarCenter {
            coord: *center,
            radius,
        });
    }

    // Otherwise, construct a polygon around the star based on
    // contour information.
    let polygon = construct_closed_polygon(contour);

    // Find the centre of gravity of this polygon (centroid)
    let center = polygon.centroid().unwrap();

    // Find the radius of the star based on maximum distance between
    // the centroid and any of the points in contour.
    let radius = polygon.points().fold(0., |distance, point| {
        point.euclidean_distance(&center).max(distance)
    });

    // If the radius is less than 1px or more than 24px
    // we reject it as a non-star.
    if !(1. ..=24.).contains(&radius) {
        return None;
    }

    // Construct star center based on previously computed information
    Some(StarCenter {
        coord: Point {
            x: center.x() as u32,
            y: center.y() as u32,
        },
        radius: radius as u32,
    })
}

This function utilises the construct_closed_polygon function you defined previously to compute the final star centers and sizes. Now for the easy part: let's implement the missing find_star_centres_and_size:

// centroid.rs

use image::GrayImage;

pub(crate) fn find_star_centres_and_size(image: &GrayImage) -> Vec<StarCenter> {
    // Compute the contours in source image
    let contours = imageproc::contours::find_contours::<u32>(image);

    contours
        .iter()
        // Iterate over all contours and create a list
        // of star center and size data.
        .filter_map(filter_map_contour_to_star_centers)
        .collect()
}

How to encapsulate the process

All you need now is to implement one last method on the StarDetect struct that encapsulates the entire process:

// lib.rs

use crate::centroid::{find_star_centres_and_size, StarCenter};
use crate::threshold::ThresholdingExtensions;

impl StarDetect {
    pub fn find_stars(&mut self, min_stars: usize) -> Vec<StarCenter> {
        self.extract_small_scale_structures();
        self.apply_noise_reduction();

        let threshold = self.optimize_threshold_for_star_count(min_stars);
        self.binarize(threshold);

        find_star_centres_and_size(&self.source)
    }
}

This method only calls the functions we've written so far. The user of your library will only need to call this function and nothing else.

You can now use what you've created to find stars in an image. For this article going forward, the image I'll be using to demonstrate is shown below. If you'd like to follow along, you can download the image I'll be using from here.

Image M42 Orion Nebula, The Dark Horse Nebula, The Flaming Star Nebula And The Surrounding H-Alpha Gas

As you might notice, we have a wide range of star shapes, sizes and colors in this image, but the same goes for noise and other large-scale nebulae structures too.

How to test the implementation on astronomical images

Create a new file main.rs and declare it as a binary target in the Cargo.toml file. It should look like this:

[package]
name = "stardetector"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "stardetector"
path = "src/main.rs"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
geo = "0.28.0"
image = "0.25.1"
image-dwt = "0.3.2"
imageproc = "0.24.0"

You can finally use the lib we created to process the sample image. The final code in main.rs should look like this:

use image::Rgba;
use stardetector::StarDetect;

fn main() {
    // Load the image as mutable. You need mutability so that
    // you can draw on this image.
    let mut image = image::open("m42-star-detection.jpg").unwrap();

    // Create a new star detector instance. You clone the image
    // here because you need to also draw on the image for
    // visualisation purposes in this example.
    let mut star_detector = StarDetect::from(image.clone());

    // Run the star finder function with a minimum star count of
    // 500
    let stars = star_detector.find_stars(500);

    // Iterate over all stars you've found
    for star in stars {
        // Draw a hollow circle on the image so that you
        // can see what the algorithm found
        imageproc::drawing::draw_hollow_circle_mut(
            &mut image,
            (star.coord().x as i32, star.coord().y as i32),
            // Extend the radius by 4px so that it's easier to see
            // in the visualisation.
            star.radius() as i32 + 4,
            // Draw the circle with a pure green color
            Rgba([0, u8::MAX, 0, 1]),
        );
    }

    // Save the image with star positions annotated with
    // green circles.
    image.save("annotated.jpg").unwrap();
}

Ensure that the downloaded image is present at the root of this project folder.

We can finally run the program and see what it gives us:

cargo run --release

Image A part of the orion region with detected stars annotated with green circles

That looks pretty good! If we zoom in to a small part of the image:

Image A part of the orion region with detected stars annotated with green circles

We can see that there are some minor issues with the algorithm, such as stars that are very close to each other and have an overlap of their halos are considered as a single star. The problem is quite an interesting one.

There are various techniques to solve this issue, but they're out of the scope of this article.

How to optimize minimum star count

Let's crank up the minimum star count to 1000 and see what happens:

Image A part of the orion region with detected stars annotated with green circles

Image A part of the orion region with detected stars annotated with green circles

This time, it picked up many of the fainter stars since the threshold had to be lower to accommodate for the higher minimum star count.

It's time to crank it up further! Let's try 2000.

Image A part of the orion region with detected stars annotated with green circles

Image A part of the orion region with detected stars annotated with green circles

It picked up even more stars this time, but it has also started hallucinating some stars where there are none. This is being caused by lower threshold retaining more noise in the image, which is then picked up as a star. But noise isn't as visible in the final image unless you really pixel-peep, which is why it appears that the algorithm is hallucinating stars.

Noise, in this particular situation, not only refers to the noise in the traditional sense – but also to any pixels that do no belong to a star for this particular purpose.

But there is one more thing...

Let's crank the minimum star count up to the absolute maximum for this particular image, which I found to be 3500.

Image A part of the orion region with detected stars annotated with green circles

Image A part of the orion region with detected stars annotated with green circles

The algorithm now seems to have failed us miserably, which is expected when the noise is too high. There are too many false-positives for this data to be of any use at all.

I wanted to show you this anyway because it shows you the flaws in the algorithm. It also shows you what star detection on noise signal looks like and why we need to pre-process an image to remove everything that isn't a star before we run the star detection.

We're going to stop here for the implementation, but there's many resources you can find below if you're interested in learning more about the topic.

The complete code for everything I talked about today can be found here: https://github.com/anshulsanghi-blog/stardetector

Further Reading

These are some of the resources that were very helpful to me when I was trying to figure out how star detection works. The resources are more about plate-solving (the process of figuring out the exact coordinates in the night sky of things in an image), but star detection is a crucial part of that process.

Stars have pretty interesting characteristics, some of which are unique such as their point-spread function estimates. These characteristics can be implemented to further improve the star detection and filtering of false-positives.

In addition, I've created a Rust library that implements this algorithm, but has some additional features already, and more robust processes are in the works.

Things such as handling RGB images properly instead of converting them to grayscale are already implemented. It also has the ability to work with RAW images.

I'm also soon going to be working on performance improvements for the same.

If you want to learn more, or contribute to the library, feel free to do so. The repository can be found here: https://github.com/anshap1719/stardetect

Wrapping Up

I hope you enjoyed the journey so far. If image processing and analysis techniques or their implementation in Rust is something that interests you, then stay tuned for more as these are the topics I love writing about.

Also, feel free to contact me if you have any questions or opinions on this topic.

Enjoying my work?

Consider buying me a coffee to support my work!

Till next time, happy coding and wishing you clear skies!

]]> <![CDATA[ Rust Tutorial – Learn Multi-Scale Processing of Astronomical Images ]]> https://www.freecodecamp.org/news/multi-scale-analysis-of-images-in-rust/ 66bb5795f55324ca867c88e2 Wed, 10 Apr 2024 15:48:11 +0000 Recently, there's been a massive amount of effort put into developing novel image processing techniques. And many of them are derived from digital signal processing methods such as Fourier and Wavelet transforms.

These techniques have not only enabled a wide range of image processing techniques such as noise reduction, sharpening, and dynamic-range extension, but have also enabled many techniques used in compute vision such as edge detection, object detection, and so on.

Multi-scale analysis is one of the newer techniques (relatively speaking) that has been adopted in a wide range of applications, especially in the astronomical image and data processing applications. This technique, which is based on Wavelet transform, allows us to divide our data into multiple signals, that all add up to make the final signal.

We can then perform our processing or analysis work on this individual sub-signals, allowing us to do targeted operations that do not affect other sub-signals.

In this tutorial, we'll first be exploring what the technique is all about, through the lens of a particular algorithm for performing multi-scale analysis on images. We'll then move on to looking at how we can implement what we discussed in the first part in Rust programming language and recreate the examples you see in the first half of the article.

Before You Read:

Prerequisites for Part 1:

The technique described is derived from the concept of "Wavelet Transforms". You don't need to know everything about it, but a very basic understanding will help you grasp the material better.

Since the article focuses on image processing and analysis, a basic understanding of how pixels work in digital format is helpful, but not mandatory.

Prerequisites for Part 2:

Here, we focus on implementing the algorithm using the Rust programming language, without going much into the details of the language itself. So being comfortable writing Rust programs, and comfortable reading crate documentations is required.

If this is not you, you can still read Part 1 and learn the technique, and then maybe you'll want to then try it out in a language of your choice. If you're not familiar with Rust, I highly encourage you to learn the basics. Here's an interactive Rust course that can get you started.

Table Of Contents

  1. Part 1: Understanding Multi-Scale Processing Technique And Algorithm
    1. What is multi-scale image processing
    2. The À Trous Wavelet Transform
    3. Scaling Functions
    4. Convolution Pixels At Each Scale
    5. Handling Boundary Conditions
    6. Computing Maximum Possible Scales For Any Given Image
    7. Closing Notes
  2. Part 2: How to Implement À Trous Tranform in Rust
    1. Prerequisites
    2. The À Trous Transform
    3. Iterators And The À Trous Transform
    4. Convolution
    5. Implementing the Iterator
    6. Recomposition
    7. Using The À Trous Transform
    8. Further Reading
  3. Wrapping Up

Part 1: Understanding the Multi-Scale Processing Technique and Algorithm

So what do we mean when we talk about multi-scale processing or analysis of some data? Well, we usually mean breaking down the input data into multiple signals, each representing a particular scale of information.

Scale, when talking about image analysis, simply refers to the size of structures that we are looking at at any given time. It ignores everything else that's either smaller or larger than the current scale.

What is multi-scale image processing?

For images, "scales" generally refer to the size in pixels of various structures or details in the image. You'll be able to get an intuitive understanding by looking at the following example:

Image Messier 33, AKA Triangulum Galaxy

Assuming our naïve understanding is correct, we can derive images of at-least the following 3 scales:

  • Very small structures, usually the size of a single pixel. This layer, when separated from the rest of the image, will only contain the noise and some sharp stars for the most part.
  • Small structures, usually a few pixels in size. This layer, when separated, will contain all of the stars and the very fine details in the galaxy arms.
  • Large and very large scale structures, usually 100s of pixels in size. This layer, when separated, will contain the general size and shape of the galaxy at the center.

Now the question becomes, why do we need to do all of this in the first place?

The answer is simple: it allows us to make targeted enhancements and changes to an image.

For example, noise reduction on the overall image will usually result in a loss of sharpness in the galaxy. But since we have broken our image down into multiple scales, we can easily apply noise reduction to only the first few layers, as most of the random noise that is easy to remove resides only in lower scale layers.

We then re-combine the noise-reduced low-scale layers with unmodified large-scale ones, and we have an output that gives us noise reduction without a loss in quality.

Another peculiar thing about noise is that it's almost always present in just one of these layers, making noise reduction process both easy and non-destructive.

If you're more of a visual learner, let's see this in practice using the image we used above. We're gonna be working with the following grayscale version of that image, where I've also added random gaussian noise:

Image Messier 33 AKA Triangulum Galaxy, Converted to grayscale and with added Gaussian noise

Performing scale-based layer separation on this image, we get the following results. Note that the results are rescaled to a range where they can be viewed as an image for representational purpose. The actual transform produces pixel values that don't make sense when looked at independently, but all of the techniques and calculations described in this tutorial can still be safely applied without rescale. The recomposition process automatically gives us back the correct range:

Image 9-level À Trous Decomposition. From top-left to bottom-right, we have images at the following pixel scales: 1, 2 4, 8, 16, 32, 64, 128, 256 (powers of 2)

  1. The first and second layers contain the noise and stars. In this particular example, noise is mixed in with the stars. But using the first and second layers, we can easily target areas that are not present in the second layer, as we can be sure that those are where the noise is present in the first layer.
  2. With the third layer, we still see the residue luminance from stars. But if you look closely, we also see very faintly the arms of the galaxy starting to appear.
  3. From the fourth layer onwards, we see the galaxy at varying scales and detail levels, completely without the stars. We start with the finer details (relatively small scale details) and increasingly move on to larger and larger scale samples. By the end, we only see a vague shape where the galaxy used to be.

From here on, we can selectively apply noise reduction to the first two layers. Then we can recombine all of the layers to create the following image that has very little noise while preserving the same amount of details in the stars and the galaxy arms:

Image Messier 33 AKA Triangulum Galaxy, result of recombining all layers but with noise reduction applied to the pixel scale 1 & 2 layers

In its most basic form, multi-scale analysis involves breaking up your source image, commonly referred to as the "signal", into multiple "signals" – each containing the data for a particular scale in the source signal.

Scale, when talking about image signal here, refers to the distance between adjacent pixels that we take when creating the layer from the source image.

In practice, this technique is used as the one of the first steps in all kinds of astronomical data analysis and image processing.

As an example, you can use the technique to detect locations of stars while ignoring larger structures much more easily than would be possible otherwise.

The À Trous Wavelet Transform

All of what I've showed you previously, and all of what you're going to see in this tutorial, was achieved with wavelet decomposition and recomposition using the à trous algorithm for discreet wavelet transforms.

This algorithm has been used throughout the years for various applications. But it's become particularly important recently in astronomical image processing applications, where different objects and signals in an image can be completely separated based on structural scales.

Here's how the algorithm works:

  1. We start with the source image input and number of levels to decompose into n.
  2. For each level n:
    • We convolve the image with our scaling function (we'll see what this is in a bit), where adjacent pixels are considered to be 2n units apart from each other, giving us the result resultn. This is where the "À Trous" name comes from, which literally translates to "with holes".
    • The layer output outputn is then computed using input - resultn.
    • We then update input to equal resultn. This is also known as residue data which serves as the source data for next layer.
  3. Repeat the above steps for all levels.
  4. In the end, we have 9 wavelet layers, and 1 residue layer. All 10 layers are required for the recomposition.

For a more mathematical approach to understanding this algorithm, I encourage you to read about the à trous algorithm here.

The recomposition process is very straightforward: we just need to add all 10 layers together. We can chose to apply positive or negative bias to any of the layers, which is a factor by which to multiply the layer pixel values during recomposition. You can use it either to enhance or diminish the characteristics of that particular layer.

Scaling Functions

Scaling functions are specific convolution kernels that help us better represent data at a particular scale based on our use case. There are 3 most commonly used scaling functions, which are shown below:

Image

Image

Image

The images above show the 3 most commonly used scaling functions in the À Trous algorithm, visualised using 3rd level decomposition of the triangulum galaxy image used previously:

  • B3 Spline is a very smooth kernel. It is mostly used in isolation of large scale structures. If we wanted to sharpen our galaxy, we would have used this kernel.
  • Low-scale is a very sharply peaked kernel, and is best at working with small scale structures.
  • Linear interpolation kernel gives us the best of both worlds, and hence is used when we need to work with both small scale and large scale structures. This is what we have used in all of our previous examples.

Convolution Pixels At Each Scale

I mentioned in the algorithm that at each scale, the pixels in the image are considered to be 2n units apart. Let's try to grasp a better understanding of this using the following visualisation:

Consider the following 8px by 8px image. Each pixel is labeled 1 through 64, which is their index.

Image A representational pixel grid of a 8x8px image

We're going to focus on a convolution operation of one of the center pixels only for this example, let's say pixel number 28.

Scale 0: At scale 0, the value of 2n becomes 1. This means that for convolution, we'll consider pixels that are 1 unit apart from our target center pixel. These pixels are highlighted below:

Image 8x8px grid with pixels that are involved in convolution for pixel number 28 highlighted at scale 0

Scale 1: This is where things get interesting. At scale 1, the value of 2n becomes 2. This means that for convolution, we'll jump directly to pixels that are 2 locations apart from the target pixel:

Image 8x8px grid with pixels that are involved in convolution for pixel number 28 highlighted at scale 1

As you can see, we've created "holes" in our computation of the value of the target pixel by skipping 2n - 1 adjacent pixels and selecting the 2nth pixel. This is the basis of the algorithm.

This process is repeated for every pixel in the image, just like a regular convolution process. And each time, we consider increasing distances between pixels for computation of final values at increasing scales.

Let's look at just one more scale.

Scale 2: This is where things get even more interesting. At scale 2 the value of 2n becomes 4. This means that for convolution, we'll jump directly to pixels that are 4 locations apart from the target pixel:

Image 8x8px grid with pixels that are involved in convolution for pixel number 28 highlighted at scale 2

Wait what? Why are we choosing pixels 1, 4, 8, 25, & 57? 1 & 4 are only 3 locations apart, 25 is only 2 locations apart, and 8 & 57 are not even diagonally aligned with the target pixel. What's going on?

Handling Boundary Conditions

As we've mentioned that this process is executed for all of the pixels in an image, we also need to consider cases where the pixel locations for convolution lie outside of the image.

This is not a concept unique to this algorithm. During convolution, this is referred to as a boundary condition or handling boundary pixels. There are various techniques for dealing with this, and all of them involve virtually extending the image in order to make it seem like we're not encountering the boundary at all.

Some of the techniques are:

  • Extending as much as needed by copying the value of the last row/column
  • Mirroring the image on all edges and corners
  • Wrapping the image around the edges.

In our example, we're employing the "mirroring" technique. When implementing such an algorithm, we don't need to actually create an extended image. Any boundary handling is implementable using just basic mathematical formulae.

Our extended image, with the correct pixels selected for scale 2, is as follows:

Image Source image extended on all edges and corners using the mirroring technique. All of the faded regions represent extended areas.

Again, the extension is only logical and is completely computed using formulae, as opposed to actually extending the source image and then checking. We can easily see that with the mirrored images in place, our basic rule of picking pixels that are 2n locations apart is still followed.

Computing Maximum Possible Scales for Any Given Image

If you think about it carefully, you'll see that the maximum layers an image can be decomposed into can be calculated by computing the log2 of the image width or height (whichever is lower) and throwing away the fractional part.

In our 5x5 image, log2(5) ~= 2.32. If we throw away the fractional part, that leaves us with 2 layers. Similarly, for a 1000x1000px image, log21000 ~= 9.96, which means we can decompose a 1000x1000 px image into a maximum of 9 layers. It simply implies that our "holes" cannot be larger than the width or height.

Even with the mirroring extension we used above, if the holes are larger than the width of the image, they'll still end up outside of the extended regions, specially for corner or boundary pixels, making it impossible to perform convolution at that scale.

Closing Notes

Thinking about the examples and visualisations a bit more, you can clearly see how and why this algorithm works, and how it's able to separate out structures in an image based on their sizes. The increasing hole sizes make it so that only structures larger than the hole itself are retained for any given layer.

A big advantage of using this algorithm is the computational cost. Since this doesn't involve Fourier or Wavelet transforms, the computational cost is quite low, relatively speaking. The memory cost, however, is indeed higher. But more often than not that is a good tradeoff.

Another advantage of this algorithm when comparing it to other discreet wavelet transform algorithms is that the size of source image is preserved throughout the entire process. There's no decimation or upscaling happening here, making this algorithm one of the easiest ones to understand and implement.

The algorithm is used in almost all of the astronomical image processing softwares such as PixInsight, Siril, and many others.

This algorithm is also known by other names such as Stationary Wavelet Transform and Starlet Transform.

Part 2: How to Implement À Trous Tranform in Rust

Now I'm going to show you how you can implement this algorithm in Rust.

For the purposes of this tutorial, I'm going to assume that you're pretty familiar with Rust and its basic concepts, such as data-types, iterators, and traits and are comfortable writing programs that use these concepts.

I'm also going to assume that you have an understanding of what convolution and convolution kernels mean in this context.

Prerequisites

We're going to need a couple of dependencies. Before we get to that, let's quickly create a new project:

cargo new --lib atrous-rs
cd atrous-rs

Now let's all of the dependencies we need. We actually only need 2:

cargo add image ndarray

image is a Rust library we'll use to work with images of all of the standard formats and encodings. It also helps us convert between various formats, and provides easy access to pixel data as buffers.

ndarray is a Rust library that helps you you create, manipulate, and work with 2D, 3D, or N-Dimensional arrays. We can use nested Vectors, but using a project like ndarray is better in this case because we need to perform a lot of operations on both individual values as well as their neighbours. Not only is it much easier to do with ndarray, but they also have performance optimisations built in for many operations and CPU types.

Although I'll be covering the basic functions/traits/methods/data-types we use from these crates, I'm not going to go into too much detail for them. I encourage you to read the docs instead.

We're actually going to jump straight to algorithm implementation, and come back later to see how we can use it.

The À Trous Transform

Create a new file that will hold our implementation. Let's name it transform.rs.

Start with adding the following struct, that will hold the information we need to perform the transform:

// transform.rs

use ndarray::Array2;

pub struct ATrousTransform {
    input: Array2<f32>, // `Array2<f32>` is a 2D array where each value is of type `f32`. This will hold our pixel data for input image.
    levels: usize, // The number of levels or scales to decompose the image into
    current_level: usize, // Current level that we need to generate. This holds the state of our iterator.
    width: usize, // Width of input image
    height: usize, // Height of input image
}

We also need a way to create this struct easily. In our case, we want to be able to create it from the input image directly. Also, input image can be of any of the supported format and encoding, but we want a consistent color-type to implement the calculations, so we'll also need to convert the image to our expected format.

It's helpful to extract all of this logic away using the "constructor" pattern in Rust. Let's implement that:

// transform.rs

use image::GenericImageView;

impl ATrousTransform {
    pub fn new(input: &image::DynamicImage, levels: usize) -> Self {
        let (width, height) = input.dimensions();
        let (width, height) = (width as usize, height as usize);

        // Create a new 2D array with proper size for each dimension to hold all of our input's pixel data. Method `zeros` takes a "shape" parameter, which is a tuple of (rows_count, columns_count).
        let mut data = Array2::<f32>::zeros((height, width));

        // Convert the image to be a grayscale image where each pixel value is of type `f32`. Loop over all pixels in the input image along with its 2D location.
        for (x, y, pixel) in input.to_luma32f().enumerate_pixels() {
            // Put the pixel value at appropriate location in our data array. The `[[]]` syntax is used to provide a 2-dimensional index such as `[[row_index, col_index]]`
            data[[y as usize, x as usize]] = pixel.0[0];
        }

        Self {
            input: data,
            levels,
            current_level: 0,
            width,
            height
        }
    }
}

This takes care of converting the image to grayscale and converting the pixel values to f32. If you're not already aware, for images with floating-point pixel values, the values are always normalized. This means that they are always between 0 and 1 – 0 representing black and 1 representing white.

Iterators and the À Trous Transform

Before we continue, let's think about the algorithm for a second. We need to be able to generate images at increasing scales, until we hit the maximum number of levels we need.

We want the consumer of our library to have access to all of these scales, and be able to manipulate them and also easily recombine once they're done. They need to be able to filter layers to ignore structures at certain scales, manipulate or "map" them to change their characteristics, perform operations on them, or even store each image if they so need.

This sounds an awful lot like Iterators! Iterators give us methods like filter, skip, take, map, for_each, and so on, all of which are exactly all we need to work with our layers before recomposition.

One added advantage of Iterators is that it allows you to finish processing each layer all the way through before you move on to the next one. If you're unsure why this is, I suggest reading more about processing a series of items with Iterators in Rust.

We're going implement the Iterator trait for our ATrousTransform type which should produce a wavelet layer as output for each iteration.

We're going to be implementing the inner-most parts of the algorithm first, and build out from there. So we first need a way to convolve an input data buffer with the scaling function while making sure that adjacent pixels are 2n locations apart, which is the first step in our loop.

Convolution

We need to define our convolution kernel before we can do anything else. Create a new file kernel.rs and add it to lib.rs with the following contents:

// kernel.rs

#[derive(Copy, Clone)]
pub struct LinearInterpolationKernel {
    values: [[f32; 3]; 3]
}

impl Default for LinearInterpolationKernel {
    fn default() -> Self {
        Self {
            values: [
                [1. / 16., 1. / 8., 1. / 16.],
                [1. / 8., 1. / 4., 1. / 8.],
                [1. / 16., 1. / 8., 1. / 16.],
            ]
        }
    }
}

We define it using a struct instead of a constant array of arrays because we need to define some tiny helpful methods on it related to index handling. We'll come back to that later.

Create another file convolve.rs. This is where all of the code for handling convolution for individual pixels will go. We'll define a Convolution trait that will define methods needed to perform the convolution on every pixel in current layer.

// convolve.rs

pub trait Convolution {
    fn compute_pixel_index(
        &self,
        distance: usize,
        kernel_index: [isize; 2],
        target_pixel_index: [usize; 2]
    ) -> [usize; 2];

    fn compute_convoluted_pixel(
        &self, 
        distance: usize, 
        index: [usize; 2]
    ) -> f32;
}

You may ask why we need a trait here instead of a simple impl block. We are only working with Grayscale images in this article, but you may want to extend it to implement it for RGB or other color modes as well.

Now, you need to implement this trait for your ATrousTransform struct:

// convolve.rs

impl Convolution for ATrousTransform {
    fn compute_pixel_index(
        &self, 
        distance: usize, 
        kernel_index: [isize; 2], 
        target_pixel_index: [usize; 2]
    ) -> [usize; 2] {
        let [kernel_index_x, kernel_index_y] = kernel_index;

        // Compute the actual distance of adjacent pixel
        // by multiplying their relative position with the
        // size of the hole.
        let x_distance = kernel_index_x * distance as isize;
        let y_distance = kernel_index_y * distance as isize;

        let [x, y] = target_pixel_index;

        // Compute the index of adjacent pixel in the 2D
        // image based on the index of current pixel.
        let mut x = x as isize + x_distance;
        let mut y = y as isize + y_distance;

        // If x index is out of bounds, consider x to be
        // the nearest boundary location
        if x < 0 {
            x = 0;
        } else if x > self.width as isize - 1 {
            x = self.width as isize - 1;
        }

        // If y index is out of bounds, consider y to be
        // the nearest boundary location
        if y < 0 {
            y = 0;
        } else if y > self.height as isize - 1 {
            y = self.height as isize - 1;
        }

        // The final 2D index of pixel.
        [y as usize, x as usize]
    }

    fn compute_convoluted_pixel(
        &self, 
        distance: usize, 
        [x, y]: [usize; 2]
    ) -> f32 {
        // Create new variable to hold the result of convolution
        // for current pixel.
        let mut pixels_sum = 0.0;

        let kernel = LinearInterpolationKernel::default();

        // Iterate over relative position of pixels from the center
        // pixel to perform convolution with. In other words, 
        // these are the indexes of neighbouring pixels from the
        // center pixel.
        for kernel_index_x in -1..=1 {
            for kernel_index_y in -1..=1 {
                // Get the computed pixel location that maps to
                // the current position in kernel
                let pixel_index = self.compute_pixel_index(
                    distance,
                    [kernel_index_x, kernel_index_y],
                    [x, y]
                );

                // Get the multiplicative factor (kernel value) for 
                // this relative location from the kernel.
                let kernel_value = kernel.value_from_relative_index(
                    kernel_index_x,
                    kernel_index_y
                );

                // Multiply the pixel value with kernel scaling
                // factor and add it to the pixel sum.
                pixels_sum += kernel_value * self.input[pixel_index];
            }
        }

        // Return the value of computed pixel from convolution process.
        pixels_sum
    }
}

We need to do computations to figure out each pixel's location based on the relative position in the kernel from the center pixel as well as ensure that the "hole size" is also being taken into consideration for the final pixel index. As you might notice, you also want to handle the boundary conditions when computing indexes.

I encourage to take your time here and go through the code and the comments.

Implementing the Iterator

It's finally time to implement the Iterator trait for your ATrousTransform:

// transform.rs

impl Iterator for ATrousTransform {
    // Our output is an image as well as the current level for each
    // iteration. The current level is an `Option` to represent the
    // final residue layer after the intermediary layers have been
    // generated.
    type Item = (Array2::<f32>, Option<usize>);

    fn next(&mut self) -> Option<Self::Item> {
        let pixel_scale = self.current_level;
        self.current_level += 1;

        // We've already generated all the layers. Return None to 
        // exit the iterator.
        if pixel_scale > self.levels {
            return None;
        }

        // We've generated all intermediary layers, return the 
        // residue layer.
        if pixel_scale == self.levels {
            return Some((self.input.clone(), None))
        }

        let (width, height) = (self.width, self.height);

        // Distance between adjacent pixels for convolution (also 
        // referred to as size of "hole").
        let distance = 2_usize.pow(pixel_scale as u32);

        // Create new buffer to hold the computed data for this layer.
        let mut current_data = Array2::<f32>::zeros((height, width));

        // Iterate over each pixel location in the 2D image
        for x in 0..width {
            for y in 0..height {
                // Set the current pixel in current layer to
                // the result of convolution on the current
                // pixel in input data.
                current_data[[y, x]] = self.compute_convoluted_pixel(
                    distance, 
                    [x, y]
                );
            }
        }

        // Create current layer by subtracting currently computed pixels 
        // from previous layer
        let final_data = self.input.clone() - &current_data;

        // Set the input layer to equal the current computed layer so 
        // that it can be used as the "previous layer" in next iteration.
        // This is also our residue data for each layer.
        self.input = current_data;

        // Return the current layer data as well as current level information.
        Some((final_data, Some(self.current_level)))
    }
}

I'm going to point out that there's a lot of potential for optimizing for performance here, but that's out of the scope of this article.

We'll finally look at how we can take all of these layers and reconstruct our input image.

Recomposition

As I've said previously, reconstructing an image that was decomposed with the A Trous transform is as simple as summing all of the layers together.

We're going to define a trait for this. Why we need a trait here should be clear once you look at the implementation.

Create a new file recompose.rs with the following contents:

// recompose.rs

use image::{DynamicImage, ImageBuffer, Luma};
use ndarray::Array2;

pub trait RecomposableLayers: Iterator<Item = (Array2<f32>, Option<usize>)> {
    fn recompose_into_image(
        self,
        width: usize,
        height: usize,
    ) -> DynamicImage
        where
            Self: Sized,
    {
        // Create a result buffer to hold the pixel data for our output image.
        let mut result = Array2::<f32>::zeros((height, width));

        // For each layer, add the layer data to current value of result buffer.
        for layer in self {
            result += &layer.0;
        }

        // Compute min and max pixel intensity values in the final data so that
        // we can perform a "rescale", which normalizes all pixel values to be
        // between the range of 0 & 1, as is expected by float 32 images.
        let min_pixel = result.iter().copied().reduce(f32::min).unwrap();
        let max_pixel = result.iter().copied().reduce(f32::max).unwrap();

        // Create a new `ImageBuffer`, which is a type provided by `image` crate to
        // serve as buffer for pixel data of an image. Here, we're creating a new
        // `Luma` ImageBuffer with pixel value of type `u16`. Luma just refers to
        // grayscale.
        let mut result_img: ImageBuffer<Luma<u16>, Vec<u16>> =
            ImageBuffer::new(width as u32, height as u32);

        // Pre-compute the denominator for scaling computation so that we don't
        // repeat this unnecessarily for every iteration.
        let rescale_ratio = max_pixel - min_pixel;

        // Iterate over all pixels in the `ImageBuffer` and fill it based on data
        // from the `result` buffer after rescaling the value.
        for (x, y, pixel) in result_img.enumerate_pixels_mut() {
            let intensity = result[(y as usize, x as usize)];

            *pixel =
                Luma([((intensity - min_pixel) / rescale_ratio * u16::MAX as f32) as u16]);
        }

        // Convert the `ImageBuffer` into `DynamicImage` and return it
        DynamicImage::ImageLuma16(result_img)
    }
}

// Implement this trait for anything that implements the Iterator trait
// with the given item type
impl<T> RecomposableLayers for T where T: Iterator<Item = (Array2<f32>, Option<usize>)> {}

If you haven't noticed, since we implement this trait for a generic, this will work with any iterator, such as Filter, Map, and so on. If you didn't use a trait here, you'll have had to implement the same thing again and again for every built-in iterator type, and your code wouldn't have worked with 3rd party types.

Using the À Trous Transform

After all of that, it's finally time to reproduce the processing that I showed you for the galaxy image with lots of noise. Create a new file main.rs with the following contents:

use image::{DynamicImage, ImageBuffer, Luma};
use atrous::recompose::RecomposableLayers;
use atrous::transform::ATrousTransform;

fn main() {
    // Open our noisy image
    let image = image::open("m33-noise-lum.jpg").unwrap();

    // Create a new instance of the transform with 9 layers
    let transform = ATrousTransform::new(&image, 9);

    // Map over each layer
    transform.map(|(mut buffer, pixel_scale)| {
        // Create a new image buffer to hold the pixel data. This
        // will be populated from the raw buffer for this layer.
        let mut new_buffer =
            ImageBuffer::<Luma<u16>, Vec<u16>>::new(buffer.ncols() as u32, buffer.nrows() as u32);

        // Iterate over all pixels of the `ImageBuffer` to populate it. We also
        // convert from `f32` pixels to `u16` pixels.
        for (x, y, pixel) in new_buffer.enumerate_pixels_mut() {
            *pixel = Luma([(buffer[[y as usize, x as usize]] * u16::MAX as f32) as u16])
        }

        // If the present layer is a small scale layer (< 3), 
        // perform noise reduction
        if pixel_scale.is_some_and(|scale| scale < 3) {
            let mut image = DynamicImage::ImageLuma16(new_buffer).to_luma8();

            // Bilateral filter is a de-noising filter. Apply it to the image.
            image = imageproc::filter::bilateral_filter(&image, 10, 10., 3.);

            // Modify the raw buffer to contain the updated pixel values after
            // filtering.
            for (x, y, pixel) in image.enumerate_pixels() {
                buffer[[y as usize, x as usize]] = pixel.0[0] as f32 / u8::MAX as f32;
            }

            // Return the updated buffer.
            (buffer, pixel_scale)
        } else {
            // Return the unmodified buffer for larger scale layers.
            (buffer, pixel_scale)
        }
    })
        // Call the recomposition method on iterator
        .recompose_into_image(image.width() as usize, image.height() as usize)
        // Convert output to 8-bit grayscale image
        .to_luma8()
        // Save it to jpg file
        .save("noise-reduced.jpg")
        .unwrap()
}

You also need to add a new dependency, imageproc, which provides useful image processing implementations on top of the image crate.

cargo add imageproc

To make this work, we also need to modify our Cargo.toml to explicitly define both binary and library targets:

// Cargo.toml

[package]
name = "atrous-rs"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "atrous"
path = "src/main.rs"

[lib]
name = "atrous"
path = "src/lib.rs"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
image = "0.25.1"
imageproc = "0.24.0"
ndarray = "0.15.6"

You may download the test image from here. Move it to the root directory of your project, and run cargo run --release. Once it finishes, you should have a new file noise-reduced.jpg as the output of our process.

And there we have it.

Further Reading

These are some of the resources that were very helpful to me when I was learning about this algorithm and how to use it. I highly encourage anyone who wants a more technical understanding of the algorithm to check these out.

In addition, I've created a Rust library for working with À Trous transform. It closely matches with what I've showed you here, but has some additional features already, and will have even more.

Things such as handling RGB images and working with all the 3 different kernels is already implemented. It also has better logic for handling boundary conditions, where it uses the image folding technique.

I'm also soon going to be working on performance improvements for the same.

If you want to learn more, or contribute to the library, feel free to do so. The repository can be found here: https://github.com/anshap1719/image-dwt

Wrapping Up

I hope you enjoyed the journey so far. If image processing techniques or their implementation in Rust is something that interests you, then stay tuned for more as these are the topics I love writing about.

Also, feel free to contact me if you have any questions or opinions on this topic.

Enjoying my work?

Consider buying me a coffee to support my work!

Till next time, happy coding and wishing you clear skies!

]]> <![CDATA[ Next.js Image Tutorial – How to Upload, Crop, and Resize Images in the Browser in Next ]]> https://www.freecodecamp.org/news/how-to-upload-crop-resize-images-in-the-browser-in-nextjs/ 66b905da2898aa52dab670af Mon, 18 Apr 2022 17:19:23 +0000 Two of the most fundamental image editing functions are resizing and cropping. But you should do these carefully because they have the potential to degrade image quality.

Cropping always includes removing a portion of the original image, resulting in the loss of some pixels.

This post will teach you how to upload, crop, and resize images in the browser.

I built this project in a Codesandbox. To get started quickly, fork the Codesandbox or run the project.

Prerequisites

To follow along with this tutorial, you should have some JavaScript and React.js experience. Experience with Next.js isn't a requirement, but it's nice to have.

You also need a Cloudinary account to store the media files.

Cloudinary offers a safe and complete API for quickly and efficiently uploading media files from the server, browser, or a mobile application.

Finally you'll need Next.js. It's an open-source React-based front-end development web framework that allows server-side rendering and the generation of static websites and applications.

Project Setup and Installation

Use the npx create-next-app command to scaffold a new project in a directory of your choice to create a new project.

You can do this with the command:

npx create-next-app <project name>

To install the dependencies, use these commands:

cd <project name> 
npm install cloudinary-react

Once the app is created, and the dependencies are installed, you'll see a message with instructions for navigating to your site and running it locally.

You can do this with the command:

npm run dev

Next.js will start a hot-reloading development environment accessible by default at http://localhost:3000.

How to Build the User Interface

For our project, we'll want the user interface to upload, crop, and resize images on the home page. We will do this by updating the pages/index.js file to a component:

import React, { useState } from "react";
import Head from "next/head";

const IndexPage = () => {

  return (
    <>
      <div className="main">
        <div className="splitdiv" id="leftdiv">
          <h1 className="main-h1">
            How to Crop, Resize & Upload Image in the Browser using Cloudinary
            Transformation
          </h1>
          <div id="leftdivcard">
            <h2 className="main-h2">Resize Options</h2>
          </div>

          <button type="button" id="leftbutton">
            Upload Image
          </button>
        </div>

        <div className="splitdiv" id="rightdiv">
        <h1> Image will appear here</h1>
        </div>
      </div>
    </>
  );
};
export default IndexPage;

The current user interface doesn't look that great, though. We'll add some styling with CSS in the style.css file like this:

@import url("https://fonts.googleapis.com/css?family=Acme|Lobster");

/* This allow me to have the full width of the page without the initial padding/margin*/
body,
html {
  margin: 0;
  padding: 0;
  height: 100%;
  width: 100%;
  font-family: Acme;
  min-width: 700px;
}

.splitdiv {
  height: 100%;
  width: 50%;
}

/* This part contains all of the left side of the screen */
/* ----------------------------------------- */
#leftdiv {
  float: left;
  background-color: #fafafa;
  height: 932px;
}

#leftdivcard {
  margin: 0 auto;
  width: 50%;
  background-color: white;
  margin-top: 25vh;
  transform: translateY(-50%);
  box-shadow: 10px 10px 1px 0px rgba(78, 205, 196, 0.2);
  border-radius: 10px;
}

#leftbutton {
  background-color: #512cf3;
  border-radius: 5px;
  color: #fafafa;
  margin-left: 350px;
}

/* ----------------------------------------- */

/* This part contains all of the right side of the screen */
/* ----------------------------------------- */
#rightdiv {
  float: right;
  background-color: #cbcfcf;
  height: 932px;
}

#rightdivcard {
  margin: 0 auto;
  width: 50%;
  margin-top: 50vh;
  transform: translateY(-50%);
  background-position: bottom;
  background-size: 20px 2px;
  background-repeat: repeat-x;
}

/* ----------------------------------------- */

/* Basic styling */
/* ----------------------------------------- */

button {
  outline: none !important;
  font-family: Lobster;
  margin-bottom: 15px;
  border: none;
  font-size: 20px;
  padding: 8px;
  padding-left: 20px;
  padding-right: 20px;
  margin-top: -15px;
  cursor: pointer;
}

h1 {
  font-family: Lobster;
  color: #512cf3;
  text-align: center;
  font-size: 40px;
}

input {
  font-family: Acme;
  font-size: 16px;
  font-family: 15px;
}

input {
  width: 30%;
  height: 20px;
  padding: 16px;
  margin-left: 1%;
  margin-right: 2%;
  margin-top: 15px;
  margin-bottom: 10px;
  display: inline-block;
  border: none;
}

input:focus {
  outline: none !important;
  border: 1px solid #512cf3;
  box-shadow: 0 0 1px round #719ece;
}

/* ----------------------------------------- */

.main {
  height: 100%;
  width: 100%;
  display: inline-block;
}

.main-h2 {
  padding-top: 20px;
  text-align: center;
}

.body-h1 {
  padding-top: 20px;
  text-align: center;
  color: white;
}

.inner-p {
  color: white;
  text-align: center;
}

.main-align {
  text-align: center;
}

.form-control {
  margin-left: 15px;
}

Our application should now look like this on http://localhost:3000/:

How to Upload, Crop, & Resize Image in the Browser in Next.js

How to Create the Image Upload Widget

Cloudinary's upload widget lets us upload media assets from multiple sources, including Dropbox, Facebook, Instagram, and images that were taken right from our device's camera. We'll use the upload widget in this project.

Create a free cloudinary account to obtain your cloud name and upload_preset.

upload_presets allows us to define a set of asset upload choices centrally rather than providing them in each upload call. A Cloudinary cloud name is a unique identifier associated with your Cloudinary account.

First, from a content delivery network (CDN), we will add the Cloudinary widget's JavaScript file in our index.js located in pages/index.js. We will include this file using next/head to include all meta tags, which lets us add data to the Head portion of our HTML document in React.

Next, in the pages/index.js file, we'll import Head from next/head and add the script file.

import React, { useState } from "react";
import Head from "next/head";

const IndexPage = () => {

  return (
    <>
      <Head>
        <title>How to Crop and Resize Image in the Browser</title>
        <link rel="icon" href="/favicon.ico" />
        <meta charSet="utf-8" />
        <script
          src="https://widget.Cloudinary.com/v2.0/global/all.js"
          type="text/javascript"
        ></script>
      </Head>
      <div className="main">
          [...]
      </div>
    </>
  );
};
export default IndexPage;

In the pages/index.js file, we will create an instance of the widget in a method triggered when clicking a button and a state variable imagePublicId..

import React, { useState } from "react";
import Head from "next/head";

const IndexPage = () => {
  const [imagePublicId, setImagePublicId] = useState("");

  const openWidget = () => {
    // create the widget
    const widget = window.cloudinary.createUploadWidget(
      {
        cloudName: "olanetsoft",
        uploadPreset: "w42epls7"
      },
      (error, result) => {
        if (
          result.event === "success" &&
          result.info.resource_type === "image"
        ) {
          console.log(result.info);
          setImagePublicId(result.info.public_id);
        }
      }
    );
    widget.open(); // open up the widget after creation
  };

  return (
    <>
      //...
    </>
  );
};
export default IndexPage;

The widget requires our Cloudinary cloud_name and uploadPreset. The createWidget() function creates a new upload widget. On successfully uploading an image, we assign the public_id of the asset to the relevant state variable.

To get our cloudname and uploadPreset, we follow the steps below:

You can get the cloud name from your Cloudinary dashboard, as shown below.

How to Upload, Crop, & Resize Image in the Browser in Next.js

You can find an upload preset in the Upload tab of your Cloudinary settings page. You access this by clicking on the gear icon in the top right corner of the dashboard page.

How to Upload, Crop, & Resize Image in the Browser in Next.js

How to Upload, Crop, & Resize Image in the Browser in Next.js

Scroll down to the bottom of the page to the upload presets section, where you'll see your upload preset or the option to create one if you don't have any.

We'll proceed to call the openWidget function in the onClick handler of our image upload button, as shown below:

//...

const IndexPage = () => {
//...
  return (
    <>
     //....
      <div className="main">
        <div className="splitdiv" id="leftdiv">
          //...
          <div id="leftdivcard">
            <h2 className="main-h2">Resize Options</h2>
             //...
            </div>

          <button type="button" id="leftbutton" onClick={openWidget}>
            Upload Image
          </button>
        </div>

        <div className="splitdiv" id="rightdiv">
        <h1> Image will appear here</h1>
        </div>
      </div>
    </>
  );
};
export default IndexPage;

When we open our app in the browser and click the Upload Image button, we should see something like this:

How to Upload, Crop, & Resize Image in the Browser in Next.js

How to Implement Custom Transformation Functions

We need to create a component that handles the transformation depending on the props passed to it. We will create a components/ directory in the root folder. Inside it, we will create a file called image.js with the following content:

import { CloudinaryContext, Transformation, Image } from "cloudinary-react";

const TransformImage = ({ crop, image, width, height }) => {
  return (
    <CloudinaryContext cloudName="olanetsoft">
      <Image publicId={image}>
        <Transformation width={width} height={height} crop={crop} />
      </Image>
    </CloudinaryContext>
  );
};

export default TransformImage;

In the code snippet above, we imported CloudinaryContext, a wrapper Cloudinary component used to manage shared information across all its children Cloudinary components. The rendered TransformImage component takes data of the image transformation as props.

The above code block will render the uploaded image when we import it into pages/index.js:

//...
import TransformImage from "../components/image";

const IndexPage = () => {
  const [imagePublicId, setImagePublicId] = useState("");
  const [alt, setAlt] = useState("");
  const [crop, setCrop] = useState("scale");
  const [height, setHeight] = useState(200);
  const [width, setWidth] = useState(200);

  return (
    <>
     //...
      <div className="main">
        <div className="splitdiv" id="leftdiv">
          //...
       </div>
        <div className="splitdiv" id="rightdiv">
          <h1> Image will appear here</h1>
          <div id="rightdivcard">
            {imagePublicId ? (
              <TransformImage
                crop={crop}
                image={imagePublicId}
                width={width}
                height={height}
              />
            ) : (
              <h1> Image will appear here</h1>
            )}
          </div>
        </div>
      </div>
    </>
  );
};
export default IndexPage;

Next, we will add the Resize Options radio button so that we can select different resize and crop options with the following code snippet:

//...

const IndexPage = () => {
//...

  return (
    <>
    //...
      <div className="main">
        <div className="splitdiv" id="leftdiv">
          //...
          <div id="leftdivcard">
            <h2 className="main-h2">Resize Options</h2>

          <label className="form-control">Select Crop Type</label>
            <div>
              <label className="form-control">Scale</label>
              <input
                type="radio"
                value="scale"
                name="crop"
                onChange={(event) => setCrop(event.target.value)}
              />
            </div>
            <div>
              <label className="form-control">Crop</label>
              <input
                type="radio"
                value="crop"
                name="crop"
                onChange={(event) => setCrop(event.target.value)}
              />
            </div>
            <input
              type="number"
              placeholder="Height"
              onChange={(event) => setHeight(event.target.value)}
            />
            <input
              type="number"
              placeholder="Width"
              onChange={(event) => setWidth(event.target.value)}
            />
          </div>

          <button type="button" id="leftbutton" onClick={openWidget}>
            Upload Image
          </button>
        </div>

        <div className="splitdiv" id="rightdiv">
          //...
        </div>
      </div>
    </>
  );
};
export default IndexPage;

In the code snippet above, we:

  • Added crop type and also width and height options
  • Added an onChange property to keep track of the changes in the height and width input field, respectively

Our application's final output should look similar to what we have below:

How to Upload, Crop, & Resize Image in the Browser in Next.js

How to Upload, Crop, & Resize Image in the Browser in Next.js

Here's the GitHub Repository for the project if you want to have a look at the full code: https://github.com/Olanetsoft/how-to-upload-crop-and-resize-images-in-the-browser-in-next.js

Conclusion

This post shows how to upload, crop, and resize images in the browser in Next.js.

Resources

You may find these resources helpful.

]]> <![CDATA[ How to generate product images for Amazon, Instagram, Zalando, and Tmall ]]> https://www.freecodecamp.org/news/generation-of-product-images-for-amazon-zalando-tmall-instagram-asos/ 66d45d9c052ad259f07e4a61 Tue, 26 Nov 2019 15:54:54 +0000 By Anton Garcia Diaz

Millions of people have already shifted from traditional tv to online content, and from traditional malls to online stores. Because of this, e-commerce and marketing teams need to deploy and maintain strong online presences for their businesses.

This usually means running the brand's own online store and having a presence in different marketplaces that cover different regions and population segments. The never-ending list of possible marketplaces in which to showcase, promote, and sell products just gets longer and longer.

To make matters worse, different marketplaces have different requirements and restrictions on images, which can add a burden for the devops and marketing teams. It's also a source of inconsistency in the public image of a brand.

Here, we'll review the main aspects to consider when setting up a clean pipeline for the seamless production of omnichannel images.

A single master image through a single pipeline

To simplify workflows and keep them sustainable, a good practice is to apply the principles of omnichannel to images. This basically means to set a single, easy to configure pipeline for the creation of variants, from the same master or pristine images. Under this approach, we can use the same product image for every channel.

Our pipeline should receive master images and produce the derivatives needed to feed the marketplaces. At a minimum, it should cope with a workflow like this.

https://store.abraia.me/05bf471cbb3f9fa9ed785718e6f60e28/product-images-for-amazon-zalando-tmall-lamoda-ssg/generation-of-variants/index.html

Of course, a front-end and cloud storage are not necessary. The pipeline may just work by watching a hot folder and creating the variants as master images land there. We'll also take a look at this.

Image transformation and optimization

Each web channel has its own web design and layout. As for images, this means different and specific aspect ratios. Besides, each marketplace usually has an image policy in place, which limits the resolution and the weight of the image and sets the admissible image format. Usually, it also specifies other style guidelines.

Let's look at the main operations we'll want to accomplish with our pipeline.

Resizing, cropping, padding

To change the aspect ratio of an image we may crop it or pad it. To get a squared image from a vertical one we may cut out the upper and bottom parts or we may fill in the left and right sides with white stripes.

There are open source tools – like ImageMagick – that allow you to perform these operations effectively. Resizing an image with ImageMagick to limit its maximum dimensions to 800 px is as simple as this:

convert input.jpg -resize 800x800 resized.jpg

This instruction respects the aspect ratio. If the original image is not squared, then the resized image has one dimension lower than 800 px. Let's say the image is vertical and we want it for Tmall, which requires a squared image of 800x800 px. Then we may pad it like this:

convert resized.jpg  -gravity center -extent 800x800 padded.jpg

Also, we may simply crop it to fit the dimensions:

convert input.jpg -gravity Center  -crop 800x800+0+0 +repage crop.jpg

While some marketplaces like Tmall encourage padding images with white stripes and branding them with logos to use them in category pages, others like Amazon or Lamoda forbid this practice.

Image Cropping (left), Resizing (center), Resizing and padding (right)

When we pad an image to match the aspect ratio, we don’t risk cropping out important parts. In fact, padding is a trick to keep the aspect ratio unchanged. However, the risk is real when we crop the image.

So, it is a good practice to ensure in the studio that we comply with some composition requirements set by each channel. We should produce master images with a view of the product compatible with the different aspect ratios that we'll deliver.

Smart cropping

There are algorithms inspired by human attention and aesthetic perception that provide an enhanced protection against bad automatic crops. In the next example, with smart image cropping (white line) we avoid cutting the face unlike a simple center cropping (red line) would do.

Image Example of smart cropping with a cloud service vs center cropping

This option is available in some cloud services. If we're going to use it, we should verify that it works properly for us because many solutions only use an attention map and do not consider aesthetic aspects. Usually, choosing a number of representative images, making some tests with them, and finally verifying the results is enough to get a good grasp.

Overlaying logos and text

We may also need to add our brand logo or to add a message to the image by overlaying a vector graphic or a text. Moreover, in many cases we need some content localization strategy in place – like tailoring discounts and language to a market region. Sticking to our example, with ImageMagick we can overlay text on a padded image.

convert -fill black -pointsize 70 -gravity center -draw "rotate -90 text 0,-330 'MyBrandHere'" padded.jpg padded-with-brand.jpg

Once we configure it for one image, we may apply it to any other with the same dimensions.

Image Examples of batch image branding using Abraia's cloud service

Otherwise, handling typographies and different settings may end being tricky in workflows with certain complexity. In this regard, a cloud service usually provides a front-end to make the configuration intuitive and fast, and more convenient to handle. It also deals with other stuff like typographies or quality preservation in image recompression.

The workflows

There are many ways to deploy an image processing pipeline. Depending on the flow rate of images, we may need to support different types of workflows.

Batch processing

In the most simple case – when the flow rate is low – a batch image processing solution may be enough. With ImageMagick, we can use mogrify (instead of convert) to process all the images inside a folder.

In certain cases, like image versions with a text in different languages, we may need to code a script, but that's not a big deal either. To make it even easier, we may use a cloud batch processing tool in which we drop images and it gives us back all the variants we need, like in the video at the beginning of the post.

Hot folders

For in-house deployments where we need something more than simple batch image processing, the use of hot folders may be a good option. In this case, we should set a worker that watches a folder. Any time an image lands the folder the watcher triggers the process that creates all the variants we need.

In this regard, Gulp comes very handy to implement a folder watching pipeline. This GitHub repository brings a ready-to-use implementation of hot folder based on Gulp. It allows us to transform images using Abraia's cloud service or optimize them using Imagemin (an open source solution). Once installed, the watcher is easily started with just one command in the terminal.

$ gulp

This video shows the process at work.

https://store.abraia.me/05bf471cbb3f9fa9ed785718e6f60e28/product-images-for-amazon-zalando-tmall-lamoda-ssg/hot-folder-gulp/index.html

Full cloud

Cloud services usually bring the most flexible and fast-to-deploy solution. Still, there are different ways to go full cloud. In the most simple approach from a user perspective, an image management and optimization service takes charge of the transformation. It also manages the delivery to end users (through a CDN) or other web channels like marketplaces and social networks. The user only needs to upload the master images and to configure the transformations, usually with an intuitive graphic interface.

In medium to large companies that manage their own cloud, services from different providers are usually combined. In this case, we are likely to have to manage private and public buckets. We can have a service accessing a bucket, creating the variants, and delivering the resources or just returning them to a different bucket.

Also, a cloud pipeline may be partially implemented in-house. In this case we have endless possibilities. However, such development effort only makes sense when no service complies with the requirements and there is a justified need for a tailored solution.

Summary

Studio shooting and photography retouching are time-consuming and costly operations. Being able to use the same master material everywhere is very important to keep times and cost under control.

We have reviewed the main aspects of a complete pipeline in charge of creating image variants. On one hand, we looked at the transformations you need to perform from resizing, cropping, or padding, to the overlay of texts and graphics. On the other, we looked at the workflows to implement from batch processing to hot folders or full cloud solutions. We have reviewed some important open source resources (like ImageMagick or Gulp) that make it possible to implement a pipeline you develop yourself.

In the end, there are two main factors to consider when deciding whether to use an in-house or a cloud service. First, you must evaluate your willingness to take on the development effort. Second, you need to decide what features you require, from an easy to use interface for the configuration of variants to advanced features like smart cropping.

]]> <![CDATA[ How to build an image type convertor in six lines of Python ]]> https://www.freecodecamp.org/news/how-to-build-an-image-type-convertor-in-six-lines-of-python-d63c3c33d1db/ 66c350311283974fd2bb0775 Wed, 13 Mar 2019 16:26:01 +0000 By AMR

One of the advantage of being a programmer is your ability to build utility tools to improve your life. Unlike a non-programmer, you are probably not spending hours digging through multiple Google search result pages to find a tool that, in the first place, was supposed to improve your productivity (irony wins). This likely makes you feel more powerful knowing a programming language — especially if that Programming language is as versatile and awesome as Python is.

One of the points in the The Zen of Python says:

Simple is better than complex.

With this philosophy in place, a lot of niche tool development using Python can be done so succinctly that it makes me wonder if it’s worth calling it a tool at all. Sometimes the word script would be more accurate. Either way, we’re setting out here to build one such script that converts images from one file format (image type) to another — in just 6 lines of Python code.

Disclaimer: The number of lines (6) excludes empty lines and comments

In this tutorial, we’re going to build an image type convertor that converts a PNG image to a JPG image. Before your grey matter cells are rushing to judge whether I’m crazy to build this tool, let me say that this is not just for one image — but for all the images inside a folder. That’d definitely require more manual effort to do without coding (I know you can smell bash ing).

Python Package

We’re going to use the Python package PIL (which stands for Python Image Library) for this purpose. The original PIL didn’t get any updates for the latest Python version, so some good souls have created a friendly fork called Pillow that supports even > Python 3.0.

Install it using pip3 install Pillow.

Beginning Script

There are two primary sections in this code. The first section is where we import the required packages, and the second section is where the actual operation happens. The actual operation can be further broken down as follows:

  • Iterate through all the files with the given extension — in our case .png — and repeat all the following:
  • Open the image file (as an image file)
  • Convert the image file to a different format ( RGB )
  • Finally save the file — with the new extension .jpg

Lines 1 and 2:

from PIL import Image  # Python Image Library - Image Processing

import glob

This section just imports the required packages. PIL for Image Processing and glob for iterating through files of the given folder in the OS.

Lines 3–6:

# based on SO Answer: https://stackoverflow.com/a/43258974/5086335

for file in glob.glob("*.png"):

 im = Image.open(file)

 rgb_im = im.convert('RGB')

 rgb_im.save(file.replace("png", "jpg"), quality=95)

FIN

So that’s the end of our tool! You can save these 6 lines as a .py file and then invoke them in your computer where you’ve got images to convert.

Further Development

If you are planning on to improve this script further, you can convert this entire script into a Command Line Interface Tool — then all these details like File Format and Folder Path can be given as arguments thus extending its power further.

References

]]> <![CDATA[ How to use DeepLab in TensorFlow for object segmentation using Deep Learning ]]> https://www.freecodecamp.org/news/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b/ 66c355a1df235c0b59e2533a Mon, 24 Sep 2018 20:20:58 +0000 By Beeren Sahu

Modifying the DeepLab code to train on your own dataset for object segmentation in images

Image _Photo by [Unsplash](https://unsplash.com/photos/FmD8tIkf8bo?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" rel="noopener" target="_blank" title="">Nick Karvounis on <a href="https://unsplash.com/search/photos/images?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" rel="noopener" target="blank" title=")

I work as a Research Scientist at FlixStock, focusing on Deep Learning solutions to generate and/or edit images. We identify coherent regions belonging to various objects in an image using Semantic Segmentation.

DeepLab is an ideal solution for Semantic Segmentation. The code is available in TensorFlow.

In this article, I will be sharing how we can train a DeepLab semantic segmentation model for our own data-set in TensorFlow. But before we begin…

What is DeepLab?

DeepLab is one of the most promising techniques for semantic image segmentation with Deep Learning. Semantic segmentation is understanding an image at the pixel level, then assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.

Installation

DeepLab implementation in TensorFlow is available on GitHub here.

Preparing Dataset

Before you create your own dataset and train DeepLab, you should be very clear about what you want to want to do with it. Here are the two scenarios:

  • Training the model from scratch: you are free to have any number of classes of objects (number of labels) for segmentation. This needs a very long time for training.
  • Use the pre-trained model: you are free to have any number of classes of objects for segmentation. Use the pre-trained model and only update your classifier weights with transfer learning. This will take far less time for training compared to the prior scenario.

Let us name your new dataset as “PQR”. Create a new folder “PQR” as: tensorflow/models/research/deeplab/datasets/PQR.

To start, all you need is input images and their pre-segmented images as ground-truth for training. Input images need to be color images and the segmented images need to be color indexed images. Refer to the PASCAL dataset.

Create a folder named “dataset” inside “PQR”. It should have the following directory structure:

+ dataset    -JPEGImages    -SegmentationClass    -ImageSets+ tfrecord

JPEGImages

It contains all the input color images in *.jpg format.

Image A sample input image from PASCAL VOC dataset

SegmentationClass

This folder contains all the semantic segmentation annotations images for each of the color input images, which is the ground truth for the semantic segmentation.

These images should be color indexed. Each color index represents a unique class (with unique color) known as a color map.

Image _Sample Color Map [source: [https://github.com/DrSleep/tensorflow-deeplab-resnet](https://github.com/DrSleep/tensorflow-deeplab-resnet" rel="noopener" target="blank" title=")]

Note: Files in the “SegmentationClass” folder should have the same name as in the “JPEGImage” folder for corresponding image-segmentation file pair.

Image A sample semantic segmentation ground truth image from PASCAL VOC dataset

ImageSets

This folder contains:

  • train.txt: list of image names for the training set
  • val.txt: list of image names for the validation set
  • trainval.txt: list of image names for training + validation set

Sample *.txt file looks something like this:

pqr_000032pqr_000039pqr_000063pqr_000068pqr_000121

Remove the color-map in the ground truth annotations

If your segmentation annotation images are RGB images instead of color indexed images. Here is a Python script that will be of help.

Here, the palette defines the “RGB:LABEL” pair. In this sample code (0,0,0):0 is background and (255,0,0):1 is the foreground class. Note, the new_label_dir is the location where the raw segmentation data is stored.

Next, the task is to convert the image dataset to a TensorFlow record. Make a new copy of the script file./dataset/download_and_convert_voc2012.sh as ./dataset/convert_pqr.sh. Below is the modified script.

The converted dataset will be saved at ./deeplab/datasets/PQR/tfrecord

Defining the dataset description

Open the file segmentation_dataset.py present in the research/deeplab/datasets/ folder. Add the following code segment defining the description for your PQR dataset.

_PQR_SEG_INFORMATION = DatasetDescriptor(    splits_to_sizes={        'train': 11111, # number of file in the train folder        'trainval': 22222,        'val': 11111,    },    num_classes=2, # number of classes in your dataset    ignore_label=255, # white edges that will be ignored to be class)

Make the following changes as shown bellow:

_DATASETS_INFORMATION = {    'cityscapes': _CITYSCAPES_INFORMATION,    'pascal_voc_seg': _PASCAL_VOC_SEG_INFORMATION,    'ade20k': _ADE20K_INFORMATION,    'pqr': _PQR_SEG_INFORMATION}

Training

In order to train the model on your dataset, you need to run the train.py file in the research/deeplab/ folder. So, we have written a script file train-pqr.sh to do the task for you.

Here, we have used xception_65 for your local training. You can specify the number of training iterations to the variable NUM_ITERATIONS. and set “ — tf_initial_checkpoint” to the location where you have downloaded or pre-trained the model *.ckpt. After training, the final trained model can be found in the TRAIN_LOGDIR directory.

Finally, run the above script from the …/research/deeplab directory.

# sh ./train-pqr.sh

Voilà! You have successfully trained DeepLab on your dataset.

In the coming months, I will be sharing more of my experiences with Images & Deep Learning. Stay tuned and don’t forget to spare some claps if you like this article. It will encourage me immensely.

]]> <![CDATA[ Image Augmentation: Make it rain, make it snow. How to modify photos to train self-driving cars ]]> https://www.freecodecamp.org/news/image-augmentation-make-it-rain-make-it-snow-how-to-modify-a-photo-with-machine-learning-163c0cb3843f/ 66c357bccf1314a450f0d6b8 Mon, 09 Apr 2018 04:02:55 +0000 By Ujjwal Saxena

Image Augmentation is a technique for taking an image and using it to generating new ones. It’s useful for doing things like training a self-driving car.

Think of a person driving a car on a sunny day. If it starts raining, they may initially find it difficult to drive in rain. But slowly they get accustomed to it.

An artificial neural network too finds it confusing to drive in a new environment unless it has seen it earlier. Their are various augmentation techniques like flipping, translating, adding noise, or changing color channel.

In this article, I’ll explore the weather part of this. I used the OpenCV library for processing images. I found it pretty easy after a while, and was able to introduce various weather scenarios into an image.

I’ve pushed a fully implemented Jupyter Notebook you can play with on GitHub.

Lets’ have a look.

I’ll first show you an original test image and will then augment it.

Image

Sunny and Shady

After adding random sunny and shady effect, the image’s brightness changes. This is an easy and quick transformation to perform.

def add_brightness(image):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    image_HLS = np.array(image_HLS, dtype = np.float64)     random_brightness_coefficient = np.random.uniform()+0.5 ## generates value between 0.5 and 1.5    image_HLS[:,:,1] = image_HLS[:,:,1]*random_brightness_coefficient ## scale pixel values up or down for channel 1(Lightness)    image_HLS[:,:,1][image_HLS[:,:,1]>255]  = 255 ##Sets all values above 255 to 255    image_HLS = np.array(image_HLS, dtype = np.uint8)    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

The brightness of an image can be changed by changing the pixel values of “Lightness”- channel 1 of image in HLS color space. Converting the image back to RGB gives the same image with enhanced or suppressed lighting.

Image Sunny

Image Shady

Shadows

To a car, a shadow is nothing but the dark portions of an image, which can also be bright at times. So a self-driving car should always learn to drive with or without shadows. Randomly changing brightness on the hills or in the woods often boggle a car’s perception if not trained properly. This is even more prevalent on sunny days and differently tall buildings in a city, allowing beams of light to peep through.

Brightness is good for perception but uneven, sudden or too much brightness create perception issues. Let’s generate some fake shadows.

def generate_shadow_coordinates(imshape, no_of_shadows=1):    vertices_list=[]    for index in range(no_of_shadows):        vertex=[]        for dimensions in range(np.random.randint(3,15)): ## Dimensionality of the shadow polygon            vertex.append(( imshape[1]*np.random.uniform(),imshape[0]//3+imshape[0]*np.random.uniform()))        vertices = np.array([vertex], dtype=np.int32) ## single shadow vertices         vertices_list.append(vertices)    return vertices_list ## List of shadow vertices

def add_shadow(image,no_of_shadows=1):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    mask = np.zeros_like(image)     imshape = image.shape    vertices_list= generate_shadow_coordinates(imshape, no_of_shadows) #3 getting list of shadow vertices    for vertices in vertices_list:         cv2.fillPoly(mask, vertices, 255) ## adding all shadow polygons on empty mask, single 255 denotes only red channel        image_HLS[:,:,1][mask[:,:,0]==255] = image_HLS[:,:,1][mask[:,:,0]==255]*0.5   ## if red channel is hot, image's "Lightness" channel's brightness is lowered     image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

OpenCV’s fillPoly() function is really handy in this case. Let’s create some random vertices and impose the polygon on an empty mask using fillPoly(). Having done this, the only thing left to do is to check the mask for hot pixels and reduce the “Lightness” in the HLS image wherever these hot pixels are found.

Image Random shadow polygon on the road

Snow

Well this is something new. We often wonder how would our vehicle behave on snowy roads. One way to test that is to get pics of snow clad roads or do something on the images to get a similar effect. This effect is not a complete alternative to snowy roads, but it’s an approach worth trying.

def add_snow(image):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    image_HLS = np.array(image_HLS, dtype = np.float64)     brightness_coefficient = 2.5     snow_point=140 ## increase this for more snow    image_HLS[:,:,1][image_HLS[:,:,1]<snow_point] = image_HLS[:,:,1][image_HLS[:,:,1]<snow_point]*brightness_coefficient ## scale pixel values up for channel 1(Lightness)    image_HLS[:,:,1][image_HLS[:,:,1]>255]  = 255 ##Sets all values above 255 to 255    image_HLS = np.array(image_HLS, dtype = np.uint8)    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

Yup! That’s it. This code generally whitens the darkest parts of the image, which are mostly roads, trees, mountains and other landscape features, using the same HLS “Lightness” increase method used in the other approaches above. This technique doesn’t work well for dark images, but you can modify it to do so. Here’s what you get:

Image winter is here

You can tweak some parameters in the code for more or less snow than this. I have tested this on other images too, and this technique gives me chills.

Rain

Yes, you heard that right. Why not rain? When humans experience difficulty driving in rain, why should vehicles be spared from that? In fact, this is one of the situations for which I want my self-driving car to be trained the most. Slippery roads and blurred visions are risky, and cars should know how to handle them.

def generate_random_lines(imshape,slant,drop_length):    drops=[]    for i in range(1500): ## If You want heavy rain, try increasing this        if slant<0:            x= np.random.randint(slant,imshape[1])        else:            x= np.random.randint(0,imshape[1]-slant)        y= np.random.randint(0,imshape[0]-drop_length)        drops.append((x,y))    return drops            def add_rain(image):        imshape = image.shape    slant_extreme=10    slant= np.random.randint(-slant_extreme,slant_extreme)     drop_length=20    drop_width=2    drop_color=(200,200,200) ## a shade of gray    rain_drops= generate_random_lines(imshape,slant,drop_length)        for rain_drop in rain_drops:        cv2.line(image,(rain_drop[0],rain_drop[1]),(rain_drop[0]+slant,rain_drop[1]+drop_length),drop_color,drop_width)    image= cv2.blur(image,(7,7)) ## rainy view are blurry        brightness_coefficient = 0.7 ## rainy days are usually shady     image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    image_HLS[:,:,1] = image_HLS[:,:,1]*brightness_coefficient ## scale pixel values down for channel 1(Lightness)    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

What I did here is that again I generated random points all over the image and then used the OpenCV’s line() function to generate small lines all over the image. I have also used a random slant in the rain drops to have a feel of actual rain. I have also reduced image’s brightness because rainy days are usually shady, and also blurry because of the rain. You can change the dimension of your blur filter and the number of rain drops for desired effect.

Here is the result:

Image Fake rain but not much blur

Fog

This is yet another scenario that hampers the vision of a self-driving car a lot. Blurry white fluff in the image makes it very difficult to see beyond a certain stretch and reduces the sharpness in the image.

Fog intensity is an important parameter to train a car for how much throttle it should give. For coding such a function, you can take random patches from all over the image, and increase the image’s lightness within those patches. With a simple blur, this gives a nice hazy effect.

def add_blur(image, x,y,hw):    image[y:y+hw, x:x+hw,1] = image[y:y+hw, x:x+hw,1]+1    image[:,:,1][image[:,:,1]>255]  = 255 ##Sets all values above 255 to 255    image[y:y+hw, x:x+hw,1] = cv2.blur(image[y:y+hw, x:x+hw,1] ,(10,10))    return image

def generate_random_blur_coordinates(imshape,hw):    blur_points=[]    midx= imshape[1]//2-hw-100    midy= imshape[0]//2-hw-100    index=1    while(midx>-100 or midy>-100): ## radially generating coordinates        for i in range(250*index):            x= np.random.randint(midx,imshape[1]-midx-hw)            y= np.random.randint(midy,imshape[0]-midy-hw)            blur_points.append((x,y))        midx-=250*imshape[1]//sum(imshape)        midy-=250*imshape[0]//sum(imshape)        index+=1    return blur_points    def add_fog(image):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    mask = np.zeros_like(image)     imshape = image.shape    hw=100    image_HLS[:,:,1]=image_HLS[:,:,1]*0.8    haze_list= generate_random_blur_coordinates(imshape,hw)    for haze_points in haze_list:         image_HLS[:,:,1][image_HLS[:,:,1]>255]  = 255 ##Sets all values above 255 to 255        image_HLS= add_blur(image_HLS, haze_points[0],haze_points[1], hw) ## adding all shadow polygons on empty mask, single 255 denotes only red channel    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

Coding this was the hardest of all the functions above. I have tried a radial approach to generate patches here. Since on a foggy day usually most of the fog is at the far end of the road and as we approach near, vision keeps clearing itself.

Image Foggy Highway

It’s a real difficult task for a machine to detect nearby cars and lanes in such a foggy condition, and is a good way to train and test the robustness of the driving model.

Torrential rain

I thought of making the rain part a little better by combining fog and rain. As there is always some haze during rains and it’s good to train the car for that also. There’s no new function is required for this. We can achieve the effect by sequentially calling both.

Image

The car on the right is barely visible in this image, and this is a real world scenario. We can hardly make out anything on the road in heavy rain.

I hope this article will help you train the model in various weather conditions. For my complete code, you can visit my GitHub profile. And I’ve written a lot of other articles, which you can read on Medium and on my WordPress site.

Enjoy!

]]> <![CDATA[ How to Build a Simple Image Recognition System with TensorFlow (Part 2) ]]> https://www.freecodecamp.org/news/how-to-build-a-simple-image-recognition-system-with-tensorflow-part-2-c83348b33bce/ 66c34fec9972b7c5c7624eba Mon, 02 Jan 2017 20:40:16 +0000 By Wolfgang Beyer

This is the second part of my introduction to building an image recognition system with TensorFlow. In the first part we built a softmax classifier to label images from the CIFAR-10 dataset. We achieved an accuracy of around 25–30%. Since there are 10 different and equally likely categories, labeling the images randomly we’d expect an accuracy of 10%. So we’re already a lot better than random, but there’s still plenty of room for improvement.

In this post, I’ll describe how to build a neural network that performs the same task. Let’s see by how much we can increase our prediction accuracy!

Neural Networks

Neural networks are very loosely based on how biological brains work. They consist of a number of artificial neurons which each process multiple incoming signals and return a single output signal. The output signal can then be used as an input signal for other neurons.

Let’s take a look at an individual neuron:

Image An artificial neuron. Its output is the result of the ReLU function of a weighted sum of its inputs.

What happens in a single neuron is very similar to what happens in the the softmax classifier. Again we have a vector of input values and a vector of weights. The weights are the neuron’s internal parameters. Both input vector and weights vector contain the same number of values, so we can use them to calculate a weighted sum.

Image

So far, we’re doing exactly the same calculation as in the softmax classifier, but now comes a little twist: as long as the result of the weighted sum is a positive value, the neuron’s output is this value. But if the weighted sum is a negative value, we ignore that negative value and the neuron generates an output of 0 instead. This operation is called a Rectified Linear Unit (ReLU).

Image Rectified Linear Unit, which is defined by f(x) = max(0, x)

The reason for using a ReLU is that this creates a nonlinearity. The neuron’s output is now not strictly a linear combination (= weighted sum) of its inputs anymore. We’ll see why this is useful when we stop looking at individual neurons and instead look at the whole network.

The neurons in artificial neural networks are usually not connected randomly to each other. Most of the time they are arranged in layers:

Image _An artificial neural network with 2 layers, a hidden and an output layer. The input is not considered a layer, since it just feeds the data (without transforming it) to the first proper layer.
(Image is part of the [here](https://commons.wikimedia.org/wiki/Main_Page" rel="noopener" target="_blank" title="">Wikimedia Commons and was taken from <a href="https://commons.wikimedia.org/wiki/File:Colored_neural_network.svg" rel="noopener" target="blank" title="))

The input image’s pixel values are the inputs for the network’s first layer of neurons. The output of the neurons in layer 1 is the input for neurons of layer 2 and so forth. This is the reason why having a nonlinearity is so important. Without the ReLU at each layer, we would only have a sequence of weighted sums. And stacked weighted sums can be merged into a single weighted sum, so the multiple layers would give us no improvement over a single layer network. Introducing the ReLU nonlinearity solves this problem as each additional layer really adds something to the network.

The network’s final layer’s output are the values we are interested in, the scores for the image categories. In this network architecture each neuron is connected to all neurons of the previous layer, therefore this kind of network is called a fully connected network. As we shall see in Part 3 of this Tutorial, that is not necessarily always the case.

And that’s already the end of my very brief part on the theory of neural networks. Let’s get started building one!

The Code

The full code for this example is available on Github. It requires TensorFlow and the CIFAR-10 dataset (see Part 1) on how to install the prerequisites).

If you’ve made your way through my previous blog post, you’ll see that the code for the neural network classifier is pretty similar to the code for the softmax classifier. But in addition to switching out the part of the code that defines the model, I’ve added a couple of small features to show some of the things TensorFlow can do:

  • Regularization: this is a very common technique to prevent overfitting of a model. It works by applying a counter-force during the optimization process which aims to keep the model simple.
  • Visualization of the model with TensorBoard: TensorBoard is included with TensorFlow and allows you to generate charts and graphs from your models and from data generated by your models. This helps with analyzing your models and is especially useful for debugging.
  • Checkpoints: this feature allows you to save the current state of your model for later use. Training a model can take quite a while, so it’s essential to not have to start from scratch each time you want to use it.

The code is split into two files this time: there’s two_layer_fc.py, which defines the model, and run_fc_model.py, which runs the model (in case you’re wondering: ‘fc’ stands for fully connected).

2-Layer Fully Connected Neural Network

Let’s look at the model itself first and deal with running and training it later. two_layer_fc.py contains the following functions:

  • inference() gets us from input data to class scores.
  • loss() calculates the loss value from class scores.
  • training() performs a single training step.
  • evaluation() calculates the accuracy of the network.

Generating Class Scores: inference()

inference() describes the forward pass through the network. How are the class scores calculated, starting from input images?

The images parameter is the TensorFlow placeholder containing the actual image data. The next three parameters describe the shape/size of the network. image_pixels is the number of pixels per input image, classes is the number of different output labels and hidden_units is the number of neurons in the first/hidden layer of our network.

Each neuron takes all values from the previous layer as input and generates a single output value. Each neuron in the hidden layer therefore has image_pixels inputs and the layer as a whole generates hidden_units outputs. These are then fed into the classes neurons of the output layer which generate classes output values, one score per class.

reg_constant is the regularization constant. TensorFlow allows us to add regularization to our network very easily by handling most of the calculations automatically. I’ll go into a bit more detail when we get to the loss function.

Since our neural network has 2 similar layers, we’ll define a separate scope for each. This allows us to reuse variable names in each scope. The biases variable is defined in the way we already know, by using tf.Variable().

The definition of the weights variable is a bit more involved. We use tf.get_variable(), which allows us to add regularization. weights is a matrix with dimensions of image_pixels by hidden_units (input vector size x output vector size). The initializer parameter describes the weight variable’s initial values.

Up to now, we’ve initialized our variables to 0, but this wouldn’t work here. Think about the neurons in a single layer. They all receive exactly the same input values. If they all had the same internal parameters as well, they would all make the same calculation and all output the same value. To avoid this, we need to randomize their initial weights. We use an initialization scheme which usually works well, the weights are initialized to normally distributed values. We drop values which are more than 2 standard deviations from the mean, and the standard deviation is set to the inverse of the square root of the number of input pixels. Luckily TensorFlow handles all these details for us, we just need to specify that we want to use a truncated_normal_initializer which does exactly what we want.

The final parameter for the weights variable is the regularizer. All we have to do at this point is to tell TensorFlow we want to use L2-regularization for the weights variable. I’ll cover regularization here.

To create the first layer’s output we multiply the images matrix and the weights matrix witch each other and add the bias variable. This is exactly the same as in the softmax classifier from the previous blog post. Then we apply tf.nn.relu(), the ReLU function to arrive at the hidden layer’s output.

Layer 2 is very similar to layer 1. The number of inputs is hidden_units, the number of outputs is classes. Therefore the dimensions of the weights matrix are [hidden_units, classes]. Since this is the final layer of our network, there’s no need for a ReLU anymore. We arrive at the class scores (logits) by multiplying input (hidden) and weights with each other and adding bias.

The summary operation tf.histogram_summary() allows us to record the value of the logits variable for later analysis with TensorBoard. I’ll cover this later.

To sum it up, the inference() function as whole takes in input images and returns class scores. That’s all a trained classifier needs to do, but in order to arrive at a trained classifier, we first need to measure how good those class scores are. That’s the job of the loss function.

Calculating the Loss: loss()

First we calculate the cross-entropy between logits(the model’s output) and labels(the correct labels from the training dataset). That has been our whole loss function for the softmax classifier, but this time we want to use regularization, so we have to add another term to our loss.

Let’s take a step back first and look at what we want to achieve by using regularization.

Overfitting and Regularization

When a statistical model captures the random noise in the data it was trained on instead of the true underlying relationship, this is called overfitting.

Image _The red and blue circles represent two different classes. The green line represents an overfitted model whereas the black line represents a model with a good fit.
(Image is part of the [here](https://commons.wikimedia.org/wiki/Main_Page" rel="noopener" target="_blank" title="">Wikimedia Commons and was taken from <a href="https://en.wikipedia.org/wiki/File:Overfitting.svg" rel="noopener" target="blank" title="))

In the above image there are two different classes, represented by the blue and red circles. The green line is an overfitted classifier. It follows the training data perfectly, but it is also heavily dependent on it and is likely to handle unseen data worse than the black line, which represents a regularized model.

So our goal for regularization is to arrive at a simple model without any unnecessary complications. There are different ways to achieve this, and the option we are choosing is called L2-regularization. L2-regularization adds the sum of the squares of all the weights in the network to the loss function. This corresponds to a heavy penalty if the model is using big weights and a small penalty if the model is using small weights.

That’s why we used the regularizer parameter when defining the weights and assigned a l2_regularizer to it. This tells TensorFlow to keep track of the L2-regularization terms (and weigh them by the parameter reg_constant) for this variable. All regularization terms are added to a collection called tf.GraphKeys.REGULARIZATION_LOSSES, which the loss function accesses. We then add the sum of all regularization losses to the previously calculated cross-entropy to arrive at the total loss of our model.

Optimizing the Variables: training()

global_step is a scalar variable which keeps track of how many training iterations have already been performed. When repeatedly running the model in our training loop, we already know this value. It’s the iteration variable of the loop. The reason we’re adding this value directly to the TensorFlow graph is that we want to be able to take snapshots of the model. And these snapshots should include information about how many training steps have already been performed.

The definition of the gradient descent optimizer is simple. We provide the learning rate and tell the optimizer which variable it is supposed to minimize. In addition, the optimizer automatically increments the global_step parameter with every iteration.

Measuring Performance: evaluation()

The calculation of the model’s accuracy is the same as in the softmax case: we compare the model’s predictions with true labels and calculate the frequency of how often the prediction is correct. We’re also interested in how the accuracy evolves over time, so we’re adding a summary operation which keeps track of the value of accuracy. We’ll cover this in the section about TensorBoard.

To summarize what we have done so far, we have defined the behavior of a 2-layer artificial neural network using 4 functions: inference() constitutes the forward pass through the network and returns class scores. loss() compares predicted and true class scores and generates a loss value. training() performs a training step and optimizes the model’s internal parameters and evaluation() measures the performance of our model.

Running the Neural Network

Now that the neural network is defined, let’s look at how run_fc_model.py runs, trains and evaluates the model.

After the obligatory imports we’re defining the model parameters as external flags. TensorFlow has its own module for command line parameters, which is a thin wrapper around Python’s argparse. We’re using it here for convenience, but you can just as well use argparse directly instead.

In the first couple of lines, the various command line parameters are being defined. The parameters for each flag are the flag’s name, its default value and a short description. Executing the file with the -h flag displays these descriptions.

The second block of lines calls the function which actually parses the command line parameters. Then the values of all parameters are printed to the screen.

Here we define constants for the number of pixels per image (32 x 32 x 3) and the number of different image categories. Then we start measuring the runtime by creating a timer.

We want to log some info about the training process and use TensorBoard to display that info. TensorBoard requires the logs for each run to be in a separate directory, so we’re adding date and time info to the name of the log directory.

load_data() loads the CIFAR-10 data and returns a dictionary containing separate training and test datasets.

Generate the TensorFlow Graph

We’re defining TensorFlow placeholders. When performing the actual calculations, these will be filled with training/testing data.

The images_placeholder has dimensions of batch size x pixels per image. A batch size of ‘None’ allows us to run the graph with different batch sizes (the batch size for training the net can be set via a command line parameter, but for testing we’re passing the whole test set as a single batch).

The labels_placeholder is a vector of integer values containing the correct class label, one per image in the batch.

Here we’re referencing the functions we covered earlier in two_layer_fc.py.

  • inference() gets us from input data to class scores.
  • loss() calculates a loss value from class scores.
  • training() performs a single training step.
  • evaluation() calculates the accuracy of the network.

Defines a summary operation for TensorBoard (covered here).

Generates a saver object to save the model’s state at checkpoints (covered here).

We start the TensorFlow session and immediately initialize all variables. Then we create a summary writer which we will use to periodically save log information to disk.

These lines are responsible for generating batches of input data. Let’s pretend we have 100 training images and a batch size of 10. In the softmax example we just picked 10 random images for each iteration. This means that after 10 iterations each image will have been picked once on average(!). But in fact some images will have been picked multiple times while some images haven’t been part of any batch so far. As long as you repeat this often enough, it’s not that terrible that randomness causes some images to be part of the training batches somewhat more often than others.

But this time we want to improve the sampling process. What we do is we first shuffle the 100 images of the training dataset. The first 10 images of the shuffled data are our first batch, the next 10 images are our second batch and so forth. After 10 batches we’re at the end of our dataset and the process starts again. We shuffle the data another time and run through it from front to back. This guarantees that no image is being picked more often than any other while still ensuring that the order in which the images are returned is random.

In order to achieve this, the gen_batch() function in data_helpers() returns a Python generator, which returns the next batch each time it is evaluated. The details of how generators work are beyond the scope of this post (a good explanation can be found here). We’re using the Python’s built-in zip() function to generate a list of tuples of the from [(image1, label1), (image2, label2), ...], which is then passed to our generator function.

next(batches) returns the next batch of data. Since it’s still in the form of [(imageA, labelA), (imageB, labelB), ...], we need to unzip it first to separate images from labels, before filling feed_dict, the dictionary containing the TensorFlow placeholders, with a single batch of training data.

Every 100 iterations the model’s current accuracy is evaluated and printed to the screen. In addition, the summary operation is being run and its results are added to the summary_writer which is responsible for writing the summaries to disk. From there they can be read and displayed by TensorBoard (see this section).

This line runs the train_step operation (defined previously to call two_layer_fc.training(), which contains the actual instructions for the optimization of the variables).

When training a model takes a longer period of time, there is an easy way to save a snapshot of your progress. This allows you to come back later and restore the model in exactly the same state. All you need to do is to create a tf.train.Saver object (we did that earlier) and then call its save() method every time you want to take a snapshot.

Restoring a model is just as easy, just call the saver’s restore() method. There is a working code example showing how to do this in the file [restore_model.py](https://github.com/wolfib/image-classification-CIFAR10-tf/blob/master/restore_model.py)in the github repository.

After the training is finished, the final model is evaluated on the test set (remember, the test set contains data that the model has not seen so far, allowing us to judge how well the model is able to generalize to new data).

Results

Let’s run the model with the default parameters via “python run_fc_model.py”. My output looks like this:

Parameters: batch_size = 400 hidden1 = 120 learning_rate = 0.001 max_steps = 2000 reg_constant = 0.1 train_dir = tf_logs

Step 0, training accuracy 0.09 Step 100, training accuracy 0.2675 Step 200, training accuracy 0.3925 Step 300, training accuracy 0.41 Step 400, training accuracy 0.4075 Step 500, training accuracy 0.44 Step 600, training accuracy 0.455 Step 700, training accuracy 0.44 Step 800, training accuracy 0.48 Step 900, training accuracy 0.51 Saved checkpoint Step 1000, training accuracy 0.4425 Step 1100, training accuracy 0.5075 Step 1200, training accuracy 0.4925 Step 1300, training accuracy 0.5025 Step 1400, training accuracy 0.5775 Step 1500, training accuracy 0.515 Step 1600, training accuracy 0.4925 Step 1700, training accuracy 0.56 Step 1800, training accuracy 0.5375 Step 1900, training accuracy 0.51 Saved checkpoint Test accuracy 0.4633 Total time: 97.54s

We can see that the training accuracy starts at a level we would expect from guessing randomly (10 classes -> 10% chance of picking the correct one). Over the first about 1000 iterations the accuracy increases to around 50% and fluctuates around that value for the next 1000 iterations. The test accuracy of 46% is not much lower than the training accuracy. This indicates that our model is not significantly overfitted. The performance of the softmax classifier was around 30%, so 46% is an improvement of about 50%. Not bad!

Visualization with TensorBoard

TensorBoard allows you to visualize different aspects of your TensorFlow graphs and is very useful for debugging and improving your networks. Let’s look at the TensorBoard-related lines of code spread throughout the codebase.

In two_layer_fc.py we find the following:

Each of these three lines creates a summary operation. By defining a summary operation you tell TensorFlow that you are interested in collecting summary information from certain tensors (logits, loss and accuracy in our case). The other parameter for the summary operation is just a label you want to attach to the summary.

There are different kinds of summary operations. We’re using scalar_summary to record information about scalar (non-vector) values and histogram_summary to collect info about a distribution of multiple values (more info about the various summary operations can be found in the TensorFlow docs).

In run_fc_model.py the following lines are relevant for the TensorBoard visualization:

An operation in TensorFlow doesn’t run by itself, you need to either call it directly or call another operation which depends on it. Since we don’t want to call each summary operation individually each time we want to collect summary information, we’re using tf.merge_all_summaries to create a single operation which runs all our summaries.

During the initialization of the TensorFlow session we’re creating a summary writer. The summary writer is responsible for actually writing summary data to disk. In its constructor we supply logdir, the directory where we want the logs to be written. The optional graph argument tells TensorBoard to render a display of the whole TensorFlow graph.

Every 100 iterations we execute the merged summary operation and feed the results to the summary writer which writes them to disk.

To view the results we run TensorBoard via “tensorboard --logdir=tf_logs” and open localhost:6006 in a web browser. In the “Events”-tab we can see how the network’s loss decreases and how its accuracy increases over time.

Image TensorBoard charts displaying the model’s loss and accuracy over a training run.

The “Graphs”-tab shows a visualization of the TensorFlow graph we have defined. You can interactively rearrange it until you’re satisfied with how it looks. I think the following image shows the structure of our network pretty well.

Image TensorBoard displays the TensorBoard graph in an interactive visualization.

In the “Distribution”- and “Histograms”-tabs you can explore the results of the tf.histogram_summary operation we attached to logits, but I won’t go into further details here. More info can be found in the relevant section of the offical TensorFlow documentation.

Further Improvements

Maybe you’re thinking that training the softmax classifier took a lot less computation time than training the neural network. While that’s true, even if we kept training the softmax classifier as long as it took the neural network to train, it wouldn’t reach the same performance. The longer you train a model, the smaller the additional gains get and after a certain point the performance improvement is miniscule. We’ve reached this point with the neural network too. Additional training time would not improve the accuracy significantly anymore. There’s something else we could do though:

The default parameter values are chosen to be pretty ok, but there is some room for improvement left. By varying parameters such as the number of neurons in the hidden layer or the learning rate, we should be able to improve the model’s accuracy some more. A testing accuracy greater than 50% should definitely be possible with this model with some further optimization. Although I would be very surprised if this model could be tuned to reach 65% or more. But there’s another type of network architecture for which such an accuracy is easily doable: convolutional neural networks. These are a class of neural networks which are not fully connected. Instead they try to make sense of local features in their input, which is very useful for analyzing images. It intuitively makes a lot of sense to take spatial information into account when looking at images. In part 3 of this series we will see the principles of how convolutional neural networks work and build one ourselves.

Stay tuned for part 3 on convolutional neural networks and thanks a lot for reading! I’m happy about any feedback you might have!

aYou can also check out other articles I’ve written on my blog.

]]>