These terms are frequently used interchangeably, even among experienced developers. But while they may sound similar and occasionally overlap in practice, they address distinctly different problems and serve separate architectural goals. Understanding this distinction is not just an academic exercise. It directly impacts how you build scalable, efficient systems.

Whether you’re developing a high-traffic web server, training complex machine learning models, or optimising application performance, a solid grasp of these concepts can mean the difference between a solution that merely functions and one that scales elegantly under real-world conditions.

This article provides a comprehensive breakdown of both concepts through visual analogies, practical examples, and technical implementations. By the end, you will be equipped to confidently apply these principles in your software projects.

Here’s what we’ll cover:

1. Understanding the Fundamental Concepts

   • The Kitchen Analogy

2. What Concurrency Looks Like in Practice

   • Python Example: Implementing Concurrency with asyncio

3. What Parallelism Looks Like in Practice

   • Python Example: Implementing Parallelism with multiprocessing

4. Concurrency vs. Parallelism: A Detailed Comparison

   • When to Use Each

5. Real-World Applications and Use Cases

   • Concurrency in Production Systems

   • Parallelism in Production Systems

   • Hybrid Approaches

6. Choosing the Right Approach for Your Problem

   • Common Pitfall to Avoid

7. Why This Distinction Matters in Practice

8. Common Misconceptions and Clarifications

9. Practical Implementation Strategies

   • When Implementing Concurrency

   • When Implementing Parallelism

10. Tools and Technologies by Language

11. Further Learning Resources

12. Conclusion

Understanding the Fundamental Concepts

Before diving into implementations, let’s establish some clear definitions:

Concurrency refers to the ability of a system to manage multiple tasks within overlapping time periods. It does not necessarily mean these tasks execute at the exact same instant. Rather, concurrency is about structuring a program to handle multiple operations by interleaving their execution, often on a single processor core.

Parallelism, by contrast, involves the simultaneous execution of multiple tasks. This typically requires multiple CPU cores or processors working in tandem, with each handling a separate portion of the workload at the same time.

The Kitchen Analogy

Consider the process of cooking as a helpful mental model:

A concurrent kitchen employs a single chef who rapidly switches between preparing multiple dishes. The chef might chop vegetables for one dish, then stir a sauce for another, then return to the first dish to continue preparation. From an observer's perspective, it appears that multiple dishes are being prepared "at once," but in reality, the chef is performing one action at a time in rapid succession.

A parallel kitchen has multiple chefs, each working on different dishes simultaneously. One chef prepares the appetiser while another works on the main course, and a third handles dessert. True simultaneous work is happening across multiple workers.

Same kitchen, different strategies, different outcomes.

What Concurrency Looks Like in Practice

Concurrency is fundamentally about task scheduling, coordination, and resource management. It enables a program to handle multiple operations by strategically interleaving their execution, whether on a single core or across multiple threads.

A practical example: when you stream a video on YouTube while your device downloads a file in the background and your messaging app checks for new messages, your CPU is rapidly context-switching between these tasks. Each task gets a slice of processing time, creating the illusion of simultaneous execution even on a single-core processor.

Python Example: Implementing Concurrency with asyncio

To examine concurrency in more detail, we’ll create a simple application which gets data on various APIs asynchronously. This is an example of how Python’s library, asyncio, lets us spawn multiple network operations without blocking so we can effectively use the waiting time.

In this implementation, we’ll be simulating API calls to a weather service, a news service, and a user profile database. Pay attention to the fact that all three requests begin nearly at the same time, yet the program doesn’t wait until one of them is completed before it begins the next one.

import asyncio

async def fetch_data_from_api(api_name, delay):
    print(f"Starting request to {api_name}...")
    await asyncio.sleep(delay)  # Simulates network I/O wait
    print(f"Received response from {api_name}")
    return f"Data from {api_name}"

async def fetch_user_profile(user_id):
    print(f"Fetching profile for user {user_id}...")
    await asyncio.sleep(1.5)
    print(f"Profile loaded for user {user_id}")
    return {"user_id": user_id, "name": "John Doe"}

async def main():
    # All tasks start and are managed concurrently
    results = await asyncio.gather(
        fetch_data_from_api("Weather API", 2),
        fetch_data_from_api("News API", 1),
        fetch_user_profile(12345)
    )
    print("\nAll operations completed!")
    print("Results:", results)

asyncio.run(main())

What happens during execution:

1. All three async functions are initiated at approximately the same time.

2. The event loop manages their execution, switching between tasks when one is waiting (during await statements).

3. While one task waits for simulated I/O, the event loop allows other tasks to make progress.

4. The task with the shortest delay completes first, even though all were started together.

5. No task blocks the others, resulting in efficient use of the single thread.

Key insight: Concurrency optimises responsiveness and resource utilisation. It doesn’t inherently make individual tasks complete faster. Instead, it allows multiple tasks to make progress during the same time period, particularly when those tasks involve waiting for external resources.

What Parallelism Looks Like in Practice

Parallelism concerns itself with genuine simultaneous execution. This approach leverages multiple CPU cores or processors to divide work and execute portions concurrently in real time.

Parallelism shines when dealing with CPU-intensive operations such as mathematical computations, image processing, video rendering, or training deep learning models.

Python Example: Implementing Parallelism with multiprocessing

To better understand parallel execution, we’re going to make a program that carries out intensive calculations in a set of cores of CPUs. The given example relies on Python and the multiprocessing module to create different processes that are executed on different processor cores.

To work with a sufficiently complex example, we’ll compute the sum of the squares of millions of numbers. In contrast to the concurrent code sample, where we were waiting to receive I/O, we are actually doing some CPU-intensive work. You’ll notice the reduction in the time taken to execute the work when it’s shared by a number of cores.

from multiprocessing import Process, current_process
import time

def compute_heavy_task(task_name, iterations):
    """Simulates a CPU-intensive operation"""
    process_name = current_process().name
    print(f"{task_name} started on {process_name}")

    # Simulate CPU-bound work
    result = 0
    for i in range(iterations):
        result += i ** 2

    time.sleep(1)  # Additional simulated work
    print(f"{task_name} completed on {process_name}. Result: {result}")
    return result

if __name__ == "__main__":
    start_time = time.time()

    # Create separate processes for each task
    p1 = Process(target=compute_heavy_task, args=("Task 1", 10000000))
    p2 = Process(target=compute_heavy_task, args=("Task 2", 10000000))
    p3 = Process(target=compute_heavy_task, args=("Task 3", 10000000))

    # Start all processes (they run on separate CPU cores)
    p1.start()
    p2.start()
    p3.start()

    # Wait for all processes to complete
    p1.join()
    p2.join()
    p3.join()

    end_time = time.time()
    print(f"\nAll tasks completed in {end_time - start_time:.2f} seconds")

What happens during execution:

1. Three separate processes are spawned, each allocated to available CPU cores.

2. Each process runs independently with its own memory space and Python interpreter.

3. All three CPU-intensive calculations execute truly simultaneously across multiple cores.

4. The total runtime is determined by the longest-running task, not the cumulative sum of all tasks.

5. On a multi-core system, this completes approximately three times faster than sequential execution.

Key insight: Parallelism achieves actual speedup by distributing computational workload across multiple processors. This directly reduces total execution time for CPU-bound operations.

Concurrency vs. Parallelism: A Detailed Comparison

When to Use Each:

Use concurrency when you want to handle more tasks effectively within the same time period, particularly when those tasks spend time waiting for external resources.

Use parallelism when you want to complete tasks faster by leveraging multiple processors to divide the computational workload.

Real-World Applications and Use Cases

Concurrency in Production Systems

1. Web Servers and APIs

Modern web frameworks like Node.js, Django with async views, and FastAPI handle thousands of simultaneous client connections. Each request may involve database queries, external API calls, or file operations. Concurrency allows the server to handle new requests while waiting for I/O operations from previous requests to complete.

2. Real-Time Communication

Chat applications, collaborative editing tools, and live streaming platforms manage multiple simultaneous connections. Messages must be received, processed, and broadcast to multiple clients concurrently without blocking any single connection.

3. Mobile Applications

Mobile apps perform background synchronization, push notification handling, and data caching while maintaining a responsive user interface. The UI thread remains free while background operations proceed concurrently.

4. Microservices Orchestration

Service meshes coordinate multiple API calls to different microservices, aggregating results efficiently without waiting for each call to complete sequentially.

Parallelism in Production Systems

1. Machine Learning and AI

Training neural networks involves massive matrix computations that can be distributed across multiple GPU cores or even multiple machines. Frameworks like TensorFlow and PyTorch automatically parallelise operations across available hardware.

2. Big Data Processing

Distributed computing frameworks such as Apache Spark, Hadoop, and Dask divide large datasets across cluster nodes. Each node processes its portion of the data in parallel, enabling analysis of petabyte-scale datasets.

3. Media Processing

Video transcoding, image batch processing, and audio rendering leverage multiple CPU cores or GPUs. Each frame or segment can be processed independently in parallel.

4. Scientific Computing

Computational physics simulations, genome sequencing, and climate modelling require enormous computational resources. Parallelism across supercomputer clusters enables these calculations to complete in reasonable time frames.

5. Financial Modelling

Risk analysis and portfolio optimisation involve running thousands of scenarios. Parallel processing allows these computations to execute simultaneously, providing results quickly enough for real-time decision making.

Hybrid Approaches

In practice, sophisticated systems frequently combine both paradigms. Consider a modern web application:

1. The web server handles client requests concurrently (handling multiple users simultaneously).

2. Each request may trigger parallel data processing tasks (such as image resizing across multiple cores).

3. The database connection pool manages concurrent query execution.

4. Background job workers process tasks in parallel (such as sending emails or generating reports).

This layered approach leverages the strengths of both concurrency and parallelism to create systems that are both responsive and computationally efficient.

Choosing the Right Approach for Your Problem

Understanding which paradigm to apply requires analysing the nature of your workload:

Common Pitfall to Avoid

A frequent mistake is attempting to use threading for CPU-bound tasks in languages with a Global Interpreter Lock (like Python's CPython) and expecting parallel speedups. In such cases, threads provide concurrency but not true parallelism.

The GIL ensures only one thread executes Python bytecode at a time, leading to context-switching overhead without genuine parallel execution. For CPU-bound work in Python, multiprocessing or C extensions are necessary for true parallelism.

Why This Distinction Matters in Practice

Grasping the difference between concurrency and parallelism extends beyond writing faster code. It fundamentally influences how you architect systems and make technological decisions:

First of all, choosing the appropriate execution model for each component of your system leads to cleaner, more maintainable code. You avoid over-engineering solutions or applying the wrong tool to a problem.

Understanding these concepts also prevents wasteful patterns such as spawning unnecessary processes for I/O-bound work or using single-threaded approaches for parallelizable computations. This directly translates to reduced infrastructure costs.

Systems designed with proper concurrency models also scale horizontally more effectively. Those leveraging parallelism appropriately utilise hardware resources fully as you scale vertically.

In addition, you’ll get some key performance optimisations by choosing the right approach. When profiling reveals bottlenecks, knowing whether to optimise for concurrency or parallelism guides your refactoring efforts in the right direction.

Beyond this, in cloud environments where you pay for compute resources, efficient use of concurrency and parallelism directly affects operational costs. An efficiently concurrent system might handle 10x the load on the same hardware compared to a poorly designed synchronous alternative.

And these concepts are fundamental to backend engineering, distributed systems, DevOps, machine learning engineering, and systems programming. They appear frequently in technical interviews and are essential for senior engineering roles.

Common Misconceptions and Clarifications

"Using threads automatically gives me parallelism."

In reality, threads enable concurrency but do not guarantee parallel execution. In systems with a Global Interpreter Lock (like CPython) or on single-core machines, threads run concurrently but not in parallel. True parallelism requires multiple CPU cores and mechanisms that avoid locking constraints.

"Parallelism is always faster than sequential execution."

In fact, parallelism introduces overhead, including process creation, inter-process communication, and data synchronisation costs. For small tasks or I/O-bound operations, this overhead can outweigh benefits. Parallelism shows gains when the computational work justifies the overhead.

"Concurrency and parallelism are mutually exclusive."

As you’ve learned, modern high-performance systems routinely combine both. A web server can handle requests concurrently, with each request triggering parallel processing. Understanding how to layer these approaches is key to building sophisticated systems.

"More threads or processes always mean better performance."

Beyond a certain point, adding more threads or processes leads to diminishing returns and even performance degradation due to increased context switching and resource contention. The optimal number depends on workload characteristics and available hardware.

“Async/await makes my code run faster."

Async/await improves efficiency for I/O-bound operations by reducing idle time, but it does not speed up CPU-bound computations. It changes how waiting is handled, not how quickly individual operations execute.

Practical Implementation Strategies

How to Implement Concurrency

To introduce concurrency into your programs, first you’ll need to find where time is wasted. Blocking operations that are held waiting on external resources are the best candidates to be put under concurrent execution.

Say you’re building a web scraper to fetch a bunch of data on a variety of sites. Every single HTTP request is most likely waiting until the server gets a response back. Other requests might be underway in your program instead of waiting around during this waiting period. These wait points are identifiable by profiling your application and searching the operations with network calls, file I/O, or database queries.

After you’ve discovered these wait points, the next big step will be to select the concurrency primitive. In Python, I/O-bound operations perform very well using the patterns of async/await with the support of the asyncio framework. It also comes with a minimal cost.

Take a situation when you have to retrieve user data in a REST API and query a database at the same time. With asyncio, you can write code that initiates both tasks almost simultaneously, and then have the event loop alternate between them during periods of waiting.

Here's a practical example:

import asyncio
import aiohttp

async def fetch_user_api(user_id):
    async with aiohttp.ClientSession() as session:
        async with session.get(f'https://api.example.com/users/{user_id}') as response:
            return await response.json()

async def query_database(user_id):
    # Simulating database query
    await asyncio.sleep(0.5)
    return {'preferences': 'theme:dark', 'notifications': True}

async def get_complete_user_data(user_id):
    api_data, db_data = await asyncio.gather(
        fetch_user_api(user_id),
        query_database(user_id)
    )
    return {**api_data, **db_data}

This gives a thorough look at how concurrency works in practice.

When Implementing Parallelism

Before committing parallelism into your system, you’ll need to profile the system and make sure that what causes your bottleneck is CPU-bound computation. Many developers think that their code requires parallelism when it should actually employ concurrency.

You can use profiling tools such as Python cProfile or line profilers to determine where time is being used or wasted in your program. When the time spent in computational loops is as large as compared to waiting in I/O, then parallelism can be beneficial.

To take an example, when processing images, the execution time in pixel manipulation algorithms consumes 90% of the execution time. This is a good sign that parallelism would be useful.

Deciding how to partition the work between multiple processors is sometimes a complex issue that you should consider carefully (in terms of dividing tasks into independent points). These chunks should be able to be processed individually without needing to communicate with each other on a regular basis.

Imagine that you have to examine the log files of several servers. Processing each file may happen on a different core, and the results will get added at the final stage.

Here's how you might structure this:

from multiprocessing import Pool
import re

def analyze_log_file(filepath):
    error_count = 0
    with open(filepath, 'r') as f:
        for line in f:
            if re.search(r'ERROR|CRITICAL', line):
                error_count += 1
    return filepath, error_count

if __name__ == '__main__':
    log_files = ['server1.log', 'server2.log', 'server3.log', 'server4.log']

    with Pool(processes=4) as pool:
        results = pool.map(analyze_log_file, log_files)

    for filepath, count in results:
        print(f'{filepath}: {count} errors found')

In this example, each log file is processed entirely on one core without needing to communicate with other processes until the final result aggregation.

Tools and Technologies by Language

Various programming languages offer different methods of achieving concurrency and parallelism with their own advantages and disadvantages. And when you understand the available tools in your language of choice, you’ll be able to make a wise architectural choice.

Python

Python has a concurrent and parallel environment. For concurrent programming, the asyncio library offers a more modern syntax of async/await that’s ideal in I/O-bound tasks such as web scraping or API communication.

The threading module allows shared memory execution, but is restricted on CPU-bound tasks by the Global Interpreter Lock. The concurrent futures module is a high-level interface to concurrent task execution, which can be useful when you want to parallelize I/O operations without having to write the low-level code of asynchronous operations.

Sometimes you’ll need actual parallelism because your job requires a lot of CPU time. Multiprocessing starts individual Python processes, which don’t use the GIL at all.

In the case of data science and machine learning processes, distributed parallelism is offered in libraries such as joblib, ray, and dask and can run on your laptop up to a cluster of computers.

JavaScript and Node.js

The event loop architecture had concurrency as its foundation in JavaScript and Node.js. Asynchronous programming is now intuitive with native syntax and Promises being used as the standard model of dealing with I/O operations (like HTTP requests or file system access).

JavaScript is single-threaded, and Node.js is designed to execute single-thread programs that make good use of I/O bound concurrent tasks, such as web servers, which support thousands of parallel connections.

In cases of actual parallelism (for example, image processing or cryptographic tasks), worker threads enable you to execute JavaScript on multiple cores. The child processes module can launch individual instances of Node.js, and the cluster module allows you to launch a pool of workers to accept incoming connections and make the most of all CPU cores in a web server.

Java

Java has mature and battle-tested concurrency and parallelism support. CompletableFuture offers a fluent interface to asynchronous operations, so it’s easier to sequence dependent asynchronous tasks together without any callback hell.

The ExecutorService model also provides detailed management of thread pools and task scheduling, which is necessary in developing high performance server programs. Parallelism Java thread pools are effective at handling worker threads to execute CPU-bound tasks, whereas ForkJoinPool uses work-stealing algorithms that are useful in divide-and-conquer problems.

Java 8 offers parallel streams, which allow you to process collections in parallel with a minimal amount of code rewrites – but you have to pay close attention to when they actually will or will not improve performance.

Go

Go introduced concurrency as a first-class language: goroutines and channels. Goroutines are lightweight threads controlled by the Go runtime, which means that you can run thousands or even millions of operations concurrently with minimal overhead.

The philosophy of communication in Channels offers a secure means of communication between goroutines, and it includes the expression "do not communicate by sharing memory; share memory by communicating." Such a design makes concurrent programming more user-friendly and error-free.

In parallelism, Go automatically allocates goroutines to multiple CPU cores according to the GOMAXPROCS environment variable, and parallel execution is achieved automatically. This renders Go especially effective in the construction of parallel systems such as web servers, network tools, and distributed systems.

Rust

Rust provides concurrent and parallel programming with memory safety without performance degradation. The ownership system of the language eliminates all forms of data races at compile-time, which means that the entire category of concurrency bugs found in other languages doesn’t exist.

In the case of async operations, you can apply the syntax of Rust to operations of an asynchronous type with runtime libraries such as tokio or async-std and achieves similar performance to C++ without sacrificing safety.

The Rayon library makes parallelism of data exceedingly easy. At times, you can parallelise a calculation by substituting .iter() with .par_iter(). Rust thread pools and channels give you low-level control where necessary, and the type system keeps the threads safe, making sure that problems don’t arise in your code.

Conclusion

Concurrency and parallelism represent fundamental pillars of modern computing architecture. They are not interchangeable buzzwords, but are rather distinct paradigms that address different challenges:

Concurrency focuses on program structure and efficient task coordination. It allows systems to handle multiple operations within overlapping time periods, maximizing resource utilization and responsiveness.

Parallelism focuses on computational throughput and execution speed. It divides work across multiple processors to complete tasks faster through simultaneous execution.

The most powerful systems strategically combine both approaches, applying each where it provides the greatest benefit.

The next time you face a performance challenge, ask yourself these critical questions:

1. Is my bottleneck caused by waiting (I/O-bound) or by computation (CPU-bound)?

2. Am I trying to handle more tasks simultaneously or complete tasks faster?

3. Do I need better resource utilisation or raw computational throughput?

Your answers will guide you toward the right solution. Understanding when to apply concurrency, when to leverage parallelism, and when to combine them is what separates adequate solutions from exceptional ones. This knowledge empowers you to build systems that are not only fast but also efficient, scalable, and economically viable.

Master these concepts, and you will find yourself equipped to tackle increasingly complex engineering challenges with confidence and precision.

Further Learning Resources

• "Go Concurrency Patterns" by Rob Pike (Google Tech Talk)

• "Concurrency is not Parallelism" by Rob Pike

• Python AsyncIO Official Documentation

• Real Python: Concurrency and Parallelism in Python

• Java Concurrency in Practice by Brian Goetz

• Apache Spark Documentation for Big Data Parallelism

In this article, you’ll learn how overworked managers, busy waiters, and train stations can help bring home the fundamental concept of the event loop. If you’re working with Node, you’ll need to understand how the event loop works, as it lies at the root of some of the most powerful applications today.

Table of Contents

• Prerequisites

• What are Synchronous and Asynchronous Code?

• What Concurrency and Parallelism Mean

  • Concurrency in Node.js

  • Parallelism in Node.js

• What is the Event Loop?

• The Phases of the Event Loop

• Conclusion

Prerequisites

In order to seamlessly follow along with this article, it would help if you are familiar with the following concepts:

1. A basic understanding of JavaScript: Node.js runs on JavaScript, so you’ll need to understand variables, functions, and control flow.

2. Familiarity with Node.js basics: Running simple scripts with Node and requiring modules.

3. Some exposure to asynchronous patterns: Knowing what patterns such as setTimeout() do.

4. Some familiarity with basic CPU concepts (cores and threads): This will help you better understand concurrency and parallelism.

5. Awareness of promises and async/await: This is optional and not a strict requirement, but will be helpful.

What are Synchronous and Asynchronous Code?

When writing code for Node.js applications, there are two different ways it can run: synchronous (sync) and asynchronous (async). Synchronous code is referred to as blocking because when it runs, no other code runs until execution is complete.

An analogy for this is a busy restaurant. Picture a waiter who refuses to wait on other tables until the table they’re presently serving has received their orders and has started eating. While the food is being prepared, the waiter waits around doing nothing and only approaches your table to take your order when they are completely finished with the previous table. Needless to say, the waiter may not receive a great tip for that service.

This is what synchronous code is. It halts the execution of other processes until it’s complete. You can see how it works in the example below:

const syncWaiter = (name) => {
    console.log(`${name} attends to tables pretty slowly.`);
};

syncWaiter("Devin");
console.log("At least all the orders are correct!");

The code above will be run in sequence, in the order it appears.

Asynchronous code, unlike synchronous code, doesn’t halt all other processes until one task is executed – rather, it proceeds to carry out other tasks while a longer process runs in the background.

Using our waiter analogy, in this case the waiter in the restaurant would go take an order from one table, pass the order to the kitchen, and while it’s being prepared, proceeds to your table to take your order as well. This way, the waiter is able to ensure different processes are started even if one process takes a bit longer than the rest. Check out the example below:

const asyncWaiter = (name) => {
    setTimeout(()=> {console.log(`${name} attends to tables pretty quickly.`)}, 3000)
};

asyncWaiter("James");
console.log("Wow! All the tables are attended to in a short time.");

Unlike synchronous code, this code does run the function asyncWaiter() – but the callback inside the function executes later. When the duration elapses, the result is then shown on the screen. This is why asynchronous programs are referred to as non-blocking. They don’t halt the program, but move from one available task to another.

The code above returns the following:

Wow! All the tables are attended to in a short time.
James attends to tables pretty quickly.

This print order happens because of how the event loop manages tasks: the synchronous console.log() that comes after asyncWaiter() runs immediately, while the asynchronous callback inside asyncWaiter() (from setTimeout) is scheduled to run later. If you don’t understand this just yet, don’t worry as I’ll break it down in detail shortly.

What Concurrency and Parallelism Mean

Node.js is single-threaded but often gives the appearance of a multi-threaded environment due to how it handles concurrency and parallelism. A thread is a single sequence of instructions executed by the CPU independently. Think of it like a single waiter named James in a restaurant.

If James handles multiple tasks around the same time and quickly, an onlooker outside the restaurant who sees the number of customers moving in and out of the restaurant may assume that there are a ton of waiters serving tables. In reality, James just handles his tasks asynchronously.

Before grasping the concept of the event loop, it’s good to understand what concurrency and parallelism are, as they help explain this.

Concurrency in Node.js

Concurrency means having multiple processes run around the same time. In the waiter analogy, it is like James carrying out different tasks, though not simultaneously. He could, for instance take an order from a table and, while waiting for the food to arrive, request that extra salt be provided to another table. While the salt is on its way, he uses the waiting time to read the bill to a third table.

The key idea is that James never sits idle — he works on other tasks while waiting for one to finish. If this sounds an awful lot like asynchronous programming, it is because asynchronous code is just one way to achieve concurrency.

Other ways to execute concurrency are multithreading on a single CPU core and coroutines which are just functions that pause their execution to resume at a later time.

Parallelism in Node.js

Parallelism, on the other hand, also means having several tasks run at the same time – but instead of the tasks just being processed around the same time, they are executed at exactly the same time, simultaneously. In this case, the restaurant manager decides to hire multiple waiters and each table has a waiter who is taking orders at exactly the same time.

Parallelism can be achieved using multithreading on multiple CPU cores. In this setup, the threads share the same memory and run simultaneously while using clusters which run independently – each with its own memory space. Here’s a clear example of parallelism using the worker_threads module:

const { Worker}  = require('worker_threads');

new Worker('./worker.js');
new Worker('./worker.js');
new Worker('./worker.js');

console.log("Main thread keeps running in the process...");

The code above creates three worker threads in parallel on a multi-core machine. This doesn’t stop the main thread which continues to run, allowing each worker thread independently do its task. worker.js could be a simple file carrying out any task. In this case, it simply logs a message to the screen:

console.log("This worker thread is running here!");

Note that the argument for the Worker constructor can be any file path, and the order in which they are executed isn’t dependent on the order they appear in code. Each worker runs independently of the others and they run in parallel.

Concurrency and parallelism allow Node.js (which is single-threaded) to appear to manage multiple tasks simultaneously. Understanding these concepts sets the stage for the event loop, showing how Node.js manages to give the appearance of concurrency while still executing code in a single-threaded environment.

What is the Event Loop?

The event loop listens for events in the Node.js environment. It essentially listens for actions and then processes tasks or outputs values.

To better understand how this works, you can picture the Node.js environment as a fast-paced organization and the event loop as an overworked manager who refuses to hire a personal assistant. The manager oversees the operations of the entire office, and has a dedicated desk that contains whatever they are working on at that particular time. Let’s call this desk the call stack.

The call stack consists of whatever processes or tasks that Node.js is currently working on. When input is entered or code is written to do something, it gets moved to the call stack and from there gets executed.

The order in which this execution takes place is important, as synchronous code makes it to the call stack before asynchronous code. What happens to the asynchronous code you may ask? It goes into something known as the callback queue first before ending up on the call stack.

The callback queue is a lineup of asynchronous tasks that make it to the call stack only if the stack is empty. You can think of it like a file cabinet in the office, where asynchronous code that is processed by a specialized team of workers under the manager go to stay until the manager’s desk is cleared. The manager only heads to the cabinet when they’re done handling all the synchronous task on the call stack. This specialized team that handles asynchronous code like callbacks and async/await are the Node APIs or the Web APIs.

Node or Web APIs process asynchronous code. When the code comes in, it’s processed here and then placed in the callback queue for the event loop to pick up and take to the call stack. But there are some asynchronous tasks that are prioritized. These are known as microtasks, such as promises.

Microtasks are given particular priority and are queued in a special microtask queue. This is usually checked after an operation before checking the callback queue. If nothing is present, the event loop checks the callback queue but if some task exists such as process.nextTick(), it gets handled immediately. Macrotasks consist of tasks that are regularly scheduled and handled by the event loop only after the microtasks are treated, such as setTimeout() and setInterval().

So as you can see, the event loop is basically what it sounds like – a loop. It looks through events and handles tasks based on a prioritized schedule.

One thing to note, though, is that even within callback queues and microtask queues, there are phases. The event loop, for instance, must handle certain tasks before others even within the same category. This is where the phases of the event loop come in.

The Phases of the Event Loop

By analogy, the event loop is akin to a manager who checks the status of projects and tasks at regular intervals. In this case, they have a specific schedule for checking the status of projects. Some projects or tasks take priority over others, and the manager has to look through them in a set order.

You can also visualize event loop phases as a train moving from station to station. It starts from one location and moves to others in a particular order until it’s complete, then starts the journey again. This arrangement determines what tasks get executed before others.

Here are the phases of the event loop in order:

1. The timers phase: This phase executes the setTimeout() and setInterval() callbacks after the duration is run. The event loop starts here, like the first station on a train’s journey.

2. The pending callbacks phase: These are system-level callbacks, checked after the timers phase operations.

3. The poll phase: This phase handles input/output (I/O) events and executes the callbacks. In the absence of callbacks, the event loop waits for new ones here.

4. The check phase: This phase executes setImmediate() callbacks.

5. Close callbacks: This phase is concerned with executing close events like socket closes.

These callback events are checked in order and run accordingly, so that if setTimeout() and setImmediate() are in the same code, setTimeout() runs first unless the “train” is say, in the Poll Phase of the loop so that setImmediate() runs before setTimeout().

You can see this illustrated with the example below:

const fs = require('fs');

fs.readFile('trainMap.txt', () => {
    setTimeout(() => {
        console.log("Train takes off");
    }, 0);
    setImmediate(() => {
        console.log("Oops! Immediate halt! There's a cat on the tracks!");
    })
});

You see in the code above that the callbacks are handled asynchronously. Recall that the event loop waits for new callbacks in the poll phase. What this means is that since fs.readfile() is a callback, it gets processed in the poll phase.

setTimeout() is set to run in the timers phase but the event loop proceeds to the check phase (which comes next) where setImmediate() is executed. This is why setImmediate() runs before setTimeout() in this case. The event loop then continues from the check phase to the close phase, back to the timers phase, repeating this cycle continuously.

This explains why you see the output below printed to the screen:

Oops! Immediate halt! There's a cat on the tracks!
Train takes off

This illustrates how the event loop enforces order of execution across the different phases, ensuring that asynchronous operations run in the correct sequence.

Conclusion

The Node.js event loop can sometimes appear mysterious, but it really isn’t as complex as it first seems. At its core, it really is just the engine that ensures JavaScript can handle multiple tasks without freezing.

In this article, you’ve learnt about synchronous and asynchronous code, concurrency, parallelism, and how these concepts help explain the event loop and the phases of the event loop. Understanding how they work gives you the confidence to write asynchronous code without fear, debug more efficiently, and appreciate the power behind Node.js’s ability to handle concurrent tasks.

Go's concurrency model is based on the concept of goroutines, lightweight threads that can run multiple functions concurrently, and channels, a built-in communication mechanism for safe and efficient data exchange between goroutines.

Go's concurrency features enable developers to write programs that can:

• Handle multiple requests simultaneously, improving responsiveness and throughput.
• Utilize multi-core processors efficiently, maximizing system resources.
• Write concurrent code that is safe, efficient, and easy to maintain.

Go's concurrency model is designed to minimize overhead, reduce latency, and prevent common concurrency errors like race conditions and deadlocks.

With Go, developers can build high-performance, scalable, and concurrent systems with ease, making it an ideal choice for building modern distributed systems, networks, and cloud infrastructure.

Table of Contents

• Case study: A Bank Teller
• Sequential Processing
• Concurrency
• What are Goroutines and Channels?
• What is a Goroutine?
• How to Implement a Goroutine
• How Does a Goroutine Work?
• What are waitGroups?
• What are Channels?
• How to Write Data to a Channel
• How to Read Data from a Channel
• How to Implement Channels with Goroutine
• What are Channel Buffers?
• What is an Unbuffered Channel?
• How to Create a Buffered Channel
• What are Channel Directions?
• How to Handle Multiple Communication Operations with Channel Select
• How to Timeout Long Running Processes in a Channel
• How to Close a Channel
• How to Iterate Over Channel Messages
• Conclusion

Let's consider a scenario to illustrate concurrency:

Case Study: A Bank Teller

Imagine a busy bank with two tellers, Maria and David. Customers arrive at the bank to conduct various transactions like deposits, withdrawals, and transfers. The goal is to serve customers quickly and efficiently.

Sequential Processing (No Concurrency)

Maria and David work sequentially, one at a time. When a customer arrives, Maria helps the customer, and David waits until Maria is finished before helping the next customer. This leads to a long wait time for customers.

Concurrency

Maria and David work concurrently, serving customers simultaneously. When a customer arrives, Maria helps the customer with a transaction, and David simultaneously helps another customer with a different transaction. They work together, sharing resources like the bank's database and cash supplies, to serve multiple customers at the same time.

In this scenario, concurrency enables Maria and David to work together efficiently, serving multiple customers simultaneously, and improving the overall customer experience. This same concept applies to computer programming, where concurrency enables multiple tasks to run simultaneously, improving responsiveness, efficiency, and performance.

What are Goroutines and Channels?

A goroutine is a lightweight thread managed by the Go runtime. It is a function that runs on the Go runtime. It helps address concurrency and async flow requirements.

Goroutines allow you to start up and run other threads of execution concurrently within your program.

Channels are used to communicate between goroutines. It is a typed conduit through which you can send and receive values with the channel operator: <-.

How to Implement a Goroutine

To use and implement a goroutine, the go keyword is used to precede a function.

package main

import (
  "fmt"
  "math/rand"
  "time"
)

func pause() {
  time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
}

func sendMsg(msg string) {
  pause()
  fmt.Println(msg)
}

func main() {
  sendMsg("hello") // sync

  go sendMsg("test1") // async
  go sendMsg("test2") // async
  go sendMsg("test3") // async

  sendMsg("main") // sync

  time.Sleep(2 * time.Second)
}

From the example above,

• The sendMsg function is called synchronously and asynchronously.
• The sendMsg function is called synchronously when the sendMsg function is called without the go keyword.
• The sendMsg function is called asynchronously when the sendMsg function is called with the go keyword.

How Does a Goroutine Work?

When the sendMsg function is called with the go keyword, the main function will not wait for the sendMsg function to finish executing before it continues to the next line of code and will return immediately after the sendMsg function is called.

Otherwise, the function is called synchronously, and the main function will wait for the sendMsg function to finish executing before it continues to the next line of code.

The order of the output when you run the above example will differ from the order of the code because the three goroutine all run concurrently and since the functions pause for a period of time, the order which they wake will differ and be outputted.

The time.Sleep(2 * time.Second) is a quick and simple method used to keep the main function running for 2 seconds to allow the goroutine to finish executing before the main function exits. Otherwise, the main function will exit immediately after the goroutine is called and the goroutine will not have enough time to finish executing resulting to errors.

What are WaitGroups?

Unlike the time.Sleep(2 * time.Second) used in the example above, the WaitGroups are more standard to wait for a collection of goroutines to finish executing. It is a simple way to synchronize multiple goroutines.

A goroutine can also be declared with anonymous functions

package main

import (
  "fmt"
  "sync"
  "time"
  "math/rand"
)

func pause() {
  time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
}

func sendMsg(msg string, wg *sync.WaitGroup) {
  defer wg.Done()
  pause()
  fmt.Println(msg)
}

func main() {
  var wg sync.WaitGroup

  wg.Add(3)

  go func(msg string) {
    defer wg.Done()
    pause()
    fmt.Println(msg)
  }("test1")


  go sendMsg("test2", &wg)
  go sendMsg("test3", &wg)

  wg.Wait()
}

From the example above, the sync.WaitGroup is used to wait for the three goroutine to finish executing before the main function exits. It synchronizes the three goroutine and the main function.

• The sync.WaitGroup (wg) manages the goroutines and keeps track of the number of goroutines that are running.
• The sync.WaitGroup.Add (wg.Add) method is used to add the number of goroutines as arguments that are running.
• The sync.WaitGroup.Done (wg.Done) method is used to decrement the number of goroutines that are running.
• The **sync.WaitGroup.Wait (wg.Wait)** method is used to wait for all the goroutines to finish executing before the main function exits.

What are Channels?

Channels are used to communicate between goroutines. It is a typed conduit through which you can send and receive messages with the channel operator, **<-**.

In their simplest form, one goroutine writes messages into the channel and another goroutine reads the same messages out of the channel.

Channels are created using the make method and the chan keyword together with its type. Channels are used to transfer messages of which type it was declared with.

Example:

package main

func main(){
    msgChan := make(chan string)
}

The example above creates a channel msgChan of type string.

How to Write Data to a Channel

To write data to a channel, first specify the name (msgChan) of the channel, followed by the <- operator and the message. This is considered the Sender.

msgChan <- "hello world"

How to Read Data from a Channel

To read data from a channel, simple move the operator (<-) to front of the channel name (msgChan) and you can assign it to a variable. This is considered the Receiver.

msg := <- msgChan

How to Implement Channels with Goroutine

package main

import (
  "fmt"
  "math/rand"
  "time"
)

func main() {

  msgChan := make(chan string)

  go func() {
    time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
    msgChan <- "hello" // Write data to the channel
    msgChan <- "world" // Write data to the channel
  }()

  msg1 := <- msgChan
  msg2 := <- msgChan

  fmt.Println(msg1, msg2)
}

The example above shows how to write and read data from a channel. The msgChan channel is created and the go keyword is used to create a goroutine that writes data to the channel. The msg1 and msg2 variables are used to read data from the channel.

Channels behave as a first-in-first-out queue. So, when one goroutine writes data to the channel, the other goroutine reads the data from the channel in the same order it was written.

What are Channel Buffers?

Channels can be buffered or unbuffered. The previous examples include the use of an unbuffered channels.

What is an Unbuffered Channel?

An unbuffered channel causes the sender to block immediately after sending a message into the channel until the receiver receives the message.

What is a Buffered Channel?

A buffered channel allows the sender to send messages into the channel without blocking until the buffer is full. So, the sender blocks only once the buffer has filled up and waits until another goroutine reads off the channel, making sure the space size becomes available before unblocking.

How to Create a Buffered Channel

When creating a buffered channel, use the make function and specify a second parameter to indicate the buffer size.

msgBufChan := make(chan string, 2)

The example above creates a buffered channel msgBufChan of type string with a buffer size of 2. This means that the channel can hold up to two messages before it blocks.

package main

import (
  "time"
)

func main() {
  size := 3
  msgBufChan := make(chan int, size)

  // reader (receiver)
  go func() {
    for {
      _ = <- msgBufChan
      time.Sleep(time.Second)
    }
  }()

  //writer (sender)
  writer := func() {
    for i := 0; i <=> 10; i++ {
      msgBufChan <- i
      println(i)
    }
  }

  writer()
}

The example above creates a buffered channel msgBufChan of type int with a buffer size of 3.

• The writer function writes data to the channel and the reader function reads data from the channel.
• When the program runs, you will see that the number 0 through to 3 printed out immediately and the remaining numbers 5 through to 10 are printed out slowly about one per second (time.Sleep(time.Second).
• This is showing the effect of buffered channel that specify the size it can hold before it blocks.

What are Channel Directions?

When using channels as function parameters, by default, you can send and receive messages within the function. To provide additional safety at compile time, channel function parameters can be defined with a direction. That is, they can be defined to be read-only or write-only.

Example:

package main

import (
  "fmt"
  "time"
)

func writer(channel chan<- string, msg string) {
  channel <- msg
}

func reader(channel <-chan string) {
  msg := <- channel
  fmt.Println(msg)
}

func main() {
  msgChan := make(chan string, 1)

  go reader(msgChan)


  for i :- 0; i < 10; i++ {
    writer(msgChan, fmt.Sprintf("msg %d", i))
  }

  time.Sleep(time.Second * 5)
}

The example above shows how to define a channel with a direction.

• The writer function is defined with a write-only channel and
• The reader function is defined with a read-only channel.

The msgChan channel is created with a buffer size of 1. The writer function writes data to the channel and the reader function reads data from the channel.

How to Handle Multiple Communication Operations with Channel Select

The select statement lets a goroutine wait on multiple communication operations. A select blocks until one of its cases can run, then it executes that case. It chooses one at random if multiple are ready.

The select and case statements are used to simplify the management and readability of wait across multiple channels.

Example:‌

package main

import (
  "fmt"
  "time"
  "math/rand"
)

func pause() {
  time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
}

func test1(c chan<- string) {
  for {
    pause()
    c <- "hello"
  }
}

func test2(c chan<- string) {
  for {
    pause()
    c <- "world"
  }
}

func main() {
  rand.Seed(time.Now().Unix())

  c1 := make(chan string)
  c2 := make(chan string)

  go test1(c1)
  go test2(c2)

  for {
    select {
    case msg1 := <- c1:
      fmt.Println(msg1)
    case msg2 := <- c2:
      fmt.Println(msg2)
    }
  }
}

The example above shows how to use the select statement to wait on multiple channels. The test1 and test2 functions write data to the c1 and c2 channels respectively. The main function reads data from the c1 and c2 channels using the select statement.

The select statement will block until one of the channels is ready to send or receive data. If both channels are ready, the select statement will choose one at random.

How to Timeout Long Running Processes in a Channel

The time.After function is used to create a channel that sends a message after a specified duration. This can be used to implement a timeout for a channel.

It can be specified in a select statement to help manage situations where it's taking too long to receive a message from any of the channels being monitored.

Also consider using timeout when working with external resources as you can never guarantee the response time and, therefore may need to proactively take action after a predetermined time has passed.

Implementing a timeout with a select statement is very straightforward.

Example:

package main

import (
  "fmt"
  "time"
)

func main() {
     c1 := make(chan string)

    go func(channel chan string) {
        time.Sleep(1 * time.Second)
        channel <- "hello world"
    }(c1)

    select {
    case msg2 := <-c1:
        fmt.Println(msg2)
    case <-time.After(2 * time.Second): //Timeout after 2 second
        fmt.Println("timeout")
  }
}

• The example above shows how to use the time.After function to create a channel that sends a message after a specified duration.
• The main function reads data from the c1 channel using the select statement.
• The select statement will block until one of the channels is ready to send or receive data.
• If the c1 channel is ready, the main function will print the message.
• If the c1 channel is not ready after 2 seconds, the main function will print a timeout message.

How to Close a Channel

Closing a channel is used to indicate that no more values will be sent on the channel. It is used to signal to the receiver that the channel has been closed and no more values will be sent.

Go channels can be explicitly closed to help with synchronization issues. The default implementation will close the channel when all the values have been sent.

Closing a channel is done by invoking the built-in close function.‌

close(channel)

Example:

package main

import (
  "fmt"
  "bytes"
)

func process(work <-chan string, fin chan<- string) {
  var b bytes.Buffer
  for {
    if msg, notClosed := <-work; notClosed {
      fmt.Printf("%s received...\n", msg)
    } else {
      fmt.Println("Channel closed")
      fin <- b.String()
      return
    }
  }
}

func main() {
  work := make(chan string, 3)
  fin := make(chan string)

  go process(work, fin)

  word := "hello world"

  for i := 0; i < len(word); i++ {
    letter := string(word[i])
    work <- letter
    fmt.Printf("%s sent ...\n", letters)
  }

  close(work)

  fmt.Printf("result: %s\n", <-fin)
}

The example above shows how to close a channel. The work channel is created with a buffer size of 3. The process function reads data from the work channel and writes data to the fin channel. The main function writes data to the work channel and closes the work channel. The process function will print the message if the work channel is not closed. If the work channel is closed, the process function will print a message and write the data to the fin channel.

How to Iterate Over Channel Messages

Channels can be iterated over by using the range keyword, similar to arrays, slice, and/or maps. This allows you to quickly and easily iterate over the messages within a channel.

Example:

package main

import (
  "fmt"
)

func main() {
  c := make(chan string, 3)

  go func() {
    c <- "hello"
    c <- "world"
    c <- "goroutine"
    close(c) // Closing the channel is very important before proceeding to the iteration hence deadlock error
  }()

  for msg := range c {
    fmt.Println(msg)
  }
}

The example above shows how to iterate over a channel using the range keyword. The c channel is created with a buffer size of 3. The go keyword is used to create a goroutine that writes data to the c channel. The main function iterates over the c channel using the range keyword and prints the message.

Conclusion

In this article, we learned how to handle concurrency with goroutines and channels in Go. We learned how to create goroutines, and how to use WaitGroups and channels to communicate between goroutines.

We also learned how to use channel buffers, channel directions, channel select, channel timeout, channel closing, and channel range.

Goroutines and channels are powerful features in Go that help address concurrency and async flow requirements.

As always, I hope you enjoyed the article and learned something new. If you want, you can also follow me on LinkedIn or Twitter.

Many tutorials discuss these concepts, but they can be challenging to grasp due to the lack of clear examples. I will guide you through the concepts and delve into the world of Concurrency in Python, providing straightforward explanations for better understanding.

Prerequisites

You should be familiar with Python coding and have a basic understanding of multi-threading.

Although I have provided non-multi-threading code examples to help those who are not acquainted with these concepts, if you already have knowledge in these areas, you'll be able to gain a deeper understanding.

Table of Contents

1. Sequential Programming
2. What Are Threads?
3. What Are Processes?
4. Threads vs Processes
5. What is Multi-Threading?
6. What is Multi-Processing?
7. What is Asynchronous Programming?
8. What is Concurrency?
9. Global Interpreter Lock (GIL)
10. How to Remove the Limitations of GIL
    – Using another flavor of Python
    – Multiprocessing module
    – Asyncio
11. Summary

Sequential Programming

Don't be afraid when you see the word "Concurrency". Let's set it aside for now.

When you begin learning programming languages like Python or Java, your program usually runs in a sequence, starting from the top and going down.

For example:

a = 25
b = 30

def add(a, b):
    print("Addition value: ", a+b)

def sub(a, b):
    print("Subtraction Value: ", a-b)

add(a,b)
sub(a,b)

When you run the program above, your Python interpreter starts executing it line by line from top to bottom in a sequence. It starts from line a=25 and ends at the sub() function.

No matter how long your code is, Python executes it from the top. This way of executing code is also known as the top-to-bottom or sequential approach – or simply Sequential programming!

What Are Threads?

When you run your program, the Python interpreter creates a small program internally and instructs it to start running. This small program, created by the interpreter, is called a Thread. A thread is a small program that performs a certain task.

For any program or application, Python creates and instructs a thread to start execution. This thread is called the Main thread. It's important to note that the Main thread can also create multiple other threads if you tell it to do so.

That's correct! You can write a program to tell the Main thread to create multiple other threads that perform your tasks. So, a thread performs some work or a task (here main thread runs your program from top to bottom).

So you can create multiple tasks and assign one thread to perform one task. Or you can also create one thread and run multiple tasks – just like Python interpreter created Main thread and executed both the tasks add and sub in sequential order.

What Are Processes?

Imagine you own a company and have many tasks to do. To handle them, you will begin hiring people and assigning a task to each person.

Each person works independently on their task. They have their own set of tools and resources, and if one person finishes their task, it doesn't directly affect what the others are doing.

In the world of computers, each person doing a task is like a separate process. Processes are independent, have their own memory space, and don't directly share resources.

For instance, when you turn on your computer and begin opening a browser, notepad, and MS Office, you are essentially starting several processes. Each program you open, like the browser or notepad, is a separate process with its own memory. These processes are not connected to each other.

When you run a program, your operating system creates a process, assigns some memory to it, dedicates some CPU time, and executes the program within it. How the program functions within the process is determined by how it's designed. In essence, we can say that an application is a process.

Threads vs Processes

Now, that may seem confusing, right? But there is a small difference between threads and processes. Let me clarify it for you.

A process is like a single task performed by the operating system. It works on its own and has its own memory space.

A thread is like a smaller part of a task within a program. It handles small tasks. Threads are created within a process and are controlled by the program.

A program can create multiple threads and they can share resources, communicate with each other, but only within the same process.

To simplify, think of a browser as a process. Within this process, it can run single or multiple threads.

When you learn about concurrency in programming, you will see terms like "Multi-Threading", "Multi-Processing", "Asynchronous", and finally the big one "Concurrency". Don't let these terms intimidate you – the concepts are simpler than you think. And they're what you'll learn about next.

What is Multi-Threading?

Life is easier when you have a lot of workers to complete all the necessary tasks, isn't it? Similarly, if you have multiple threads, they help you divide all your program tasks into smaller parts.

Of course, if you have a lot of workers, then it can be tough to manage them, and it costs more. These are the advantages and disadvantages to multi-threading as well.

Using multiple threads to perform your tasks is called multi-threading.

Let's revisit the example:

a = 25
b = 30

def add(a, b):
    print("Addition value: ", a+b)

def sub(a, b):
    print("Subtraction Value: ", a-b)

add(a,b)
sub(a,b)

You can tell the Python interpreter to create two threads and execute the add() with thread-1 and sub() with thread-2. This is fine, but why would you want to do that?

Consider the sequential approach: the interpreter runs add() first, then sub(). Why wait for sub() while add() is running? In this case, addition and subtraction are entirely different and don't depend on each other, so why wait? Can't both functions run simultaneously?

This is where multiple threads come into play. If your tasks are independent and don't need to wait for one another to complete, then you have the privilege to run your tasks using multiple threads.

Note that whether you create one thread or more, your Python interpreter will create the main thread and handle the execution of your program. If you instruct your program to create 2 threads, the main thread will create them for you and assign which task needs to be done by each thread.

So, irrespective of whether you create a thread in your program or not, the main thread will be created, and it takes care of your program execution.

What is Multi-Processing?

From the name itself, you may be able to understand that multiprocessing means running multiple processes separately without sharing resources directly.

For instance, imagine two restaurants, each with its own chefs. The method one chef in Restaurant 1 uses to cook a dish doesn't affect or rely on Restaurant 2. They operate independently, even if you place the same order at both restaurants.

But you might be wondering: how do we talk about multitasking on a computer with one processor but many CPU cores?

In modern computers, some processors have multiple cores, and each core works like its own processor. If a processor has 4 cores, it can handle 4 tasks simultaneously by assigning each task to a different core. This means the CPU, memory, and storage resources are shared among these tasks without them relying on each other.

This approach to multitasking on multiple cores within one processor is known as Symmetric Multiprocessing.

If you have multiple processors, you can assign each task to a different processor to execute independently. This type of multi-processing is called Distributed Multi-processing.

I want to remind you that multi-threading occurs within a process. Simply put, multi-threading only happens within a process.

Let's gain a practical understanding of multiprocessing. I will be using the same example that we discussed in the Sequential Programming and Multi-threading sections, but this time using a multiprocessing version.

Just as you ran the program using threads, we will now run the program using processes.

from multiprocessing import Process

a = 25
b = 30

def add(a, b):
    try:
        print("Addition value: ", a + b)
        time.sleep(10)
    except Exception as e:
        print(e)

def sub(a, b):
    try:
        print("Subtraction Value: ", a - b)
        time.sleep(20)
    except Exception as e:
        print(e)


if __name__ == "__main__":
    # Create two processes, one for add and one for sub

    add_process = Process(target=add, args=(a, b))
    sub_process = Process(target=sub, args=(a, b))

    # Start the processes
    add_process.start()
    sub_process.start()

    # Wait for both processes to finish
    add_process.join()
    sub_process.join()

We are using the multiprocessing module in this program to create 2 processes: add_process and sub_process. One process runs the add() function and another runs the sub() function, respectively.

We then start both the processes using the .start() method of both the process objects. After this, we are using the .join() method of the process objects to make the main thread wait until both processes complete.

Let's understand this in a bit more detail. I am running the program in a Windows operating system-based computer. Open the task manager application to view all the running processes in your operating system. For running your program, you have to use your Window's command prompt or cmd or your terminal in your Linux or Mac.

TaskManager and CMD

The image above shows task manager on the left and CMD on the right. You can easily see the Windows Command Processor, or CMD, running as a process in the task manager. You can also see that I am running Google Chrome and the Snipping Tool in the background, both of which are also listed in my Task Manager.

I am going to run the program now from CMD by executing the command python sample.py, where sample is the name of the program.

Running the sample.py in CMD

In the image above, I am running the program in CMD on the right. The program immediately printed the output of both the functions add() and sub(). The program is still running, because we used time.sleep() in both functions, which pauses the execution of the add_process for 10 seconds and sub_process for 20 seconds.

Setting the output aside, let's shift our focus to the Task Manager. If you expand the Windows Command Processor, you will find 3 Python processes running. Let's explore why there are three Python processes instead of two (add_process and sub_process).

Since we initiated the program via CMD, your Python application is triggered through the Windows Command Processor or CMD and is therefore listed under CMD. We already know that when you run a Python program, your Python interpreter starts the Main Thread, which manages the execution of the program.

This Main Thread is listed as one process. Then, as we are creating two processes within the program, they are listed as an additional two processes under the Windows Command Processor (as seen in the image above).

Note that both the processes are completely independent and don't share any resources. If you want to share anything between the processes then you have to implement special mechanisms like a Queue.

What is Asynchronous Programming?

Asynchronous programming is a specific approach to managing tasks that involves waiting for external operations to complete. It allows a program to continue executing other tasks while waiting for these operations to finish, rather than blocking until they complete.

Note: If you are not familiar with Multi-threaded coding in Python, you can skip to example-2.

Example 1:

Same example again, but this time the functions are executed using threads. add() is executed by thread-1 and sub() is executed by thread-2. Following is the threading version of the example:

import threading
import time

a = 25
b = 30

def add(a, b):
    print("Inside add function\n")
    print("Waiting for 20 seconds in add \n")
    time.sleep(20)
    print("Addition value:", a + b)

def sub(a, b):
    print("Inside sub function\n")
    print("Subtraction Value {}:".format(a - b))
    print("Waiting for 10 seconds in sub\n")
    time.sleep(10)

# Create threads for add and sub functions
add_thread = threading.Thread(target=add, args=(a, b))
sub_thread = threading.Thread(target=sub, args=(a, b))

# Start both threads
add_thread.start()
sub_thread.start()

# Wait for both threads to finish
add_thread.join()
sub_thread.join()

print("Complete")

We created add_thread and sub_thread and assigned the functions add and sub as target. If you run this program, you can see the following output:

Program output

Let's breakdown the output:

• Both add and sub functions are started using separate threads (add_thread and sub_thread).
• The add function prints "Inside add function" and then indicates that it is waiting for 20 seconds using time.sleep(20).
• The sub function prints "Inside sub function" and "Subtraction Value -5:" (result of the subtraction operation), followed by a message that it is waiting for 10 seconds using time.sleep(10).
• As the threads are running at the same time, the messages from both functions may appear interleaved, depending on the execution order of the threads.
• After the waiting periods, both threads print the results of their respective operations. The add function prints "Addition value: 55," and the sub function prints nothing else.
• Finally, the main program prints "Complete" once both threads have finished their execution.

While the add_thread() is waiting for 20 seconds, your sub_thread() has provided the result and started to sleep for 10 seconds. Meanwhile, add_thread completed its wait time, computed the addition value, and printed the result. So, while one thread is waiting, your second thread started executing, and while your second thread was waiting, your first thread started running.

This way of managing the tasks to execute simultaneously when one task is waiting for external resources is called Asynchronous programming.

The key point to note is that threads run simultaneously, allowing the program to perform tasks in parallel. The interleaved output demonstrates the asynchronous nature of threading, where different parts of the program can execute simultaneously.

Example 2:

Let's consider two tasks: Task 1 retrieves a list of employees from a company's database, and Task 2 retrieves a list of active projects by the same company. It's clear that these tasks are not connected or dependent on each other – they are independent tasks.

Let's write some sample code for these tasks:

def get_employees():
    # connect to the database
    # code to get employees list from database
    return employees_list

def get_active_projects():
    # connect to the database
    # code to get the projects under development
    return active_projects

# run task1
get_employees()

# run task2
get_active_projects()

Don't worry about the code inside the functions – we want to understand the concept here.

When you run the program above, the Python interpreter tells the main thread to execute the program step by step, starting from the top and going downwards.

First, it performs task-1, which let's say takes 2 seconds, and then it moves on to task-2, which let's say also takes 2 seconds. We know that these tasks are independent, so why should task-2 wait for task-1 to finish?

If there's a mechanism where I can tell the interpreter to start task-1 (getting a list of employees from the database), and while it waits for task-1 to complete, I can also instruct it to start task-2 (getting a list of active projects from the database), then I'm essentially executing both tasks almost simultaneously. This helps me reduce the total execution time of both tasks. This is another example of asynchronous programming.

Now it's clear. How do we do this? How do we ensure that task-2 runs while we're waiting for task-1 to get the list of employees from the database? We have many tools for that. Multithreading is one of them.

In the example above, you use two threads: thread-1 for task-1 and then immediately start thread-2 for task-2. Thread-2 doesn't have to wait for Thread-1 to finish.

You can also use concepts like async/await, callbacks, and promises in various programming languages to implement asynchronous programming. You can read more about those concepts in JavaScript here.

What is Concurrency?

Finally, we've arrived at concurrency!

You should now understand that multi-threading or multi-processing involves using several threads and processes to perform multiple tasks simultaneously to decrease the time it takes for a program or application to run.

But how does this occur? What goes on in the background? How does the processor ensure that the threads or processes run simultaneously?

Imagine you have two tasks: driving a car and making a phone call. You decide to do both at once. While driving, you start calling your friend using your cell phone and speaking.

You're doing these tasks simultaneously, but there's a small detail that's important: your brain quickly scans the road, checks for other cars, and ensures you're focused and steady. It spends about one millisecond here, then switches to speaking with your friend, which might take another millisecond, and then shifts back to driving every millisecond. It continuously switches between both tasks.

As the time spent on each task is very short (just one millisecond), you might think you're doing both tasks at the same time. But there's a very tiny difference in time for each task, which makes it seem like both tasks are being done simultaneously.

In a similar way, when there are two tasks being executed by either two threads or two processes, your processor switches between these tasks very quickly. It runs thread-1 for 1 millisecond, then saves its state and switches to thread-2, running it for another 1 millisecond. After saving the state of thread-2, it shifts back to thread-1 and runs it for another millisecond.

I'm using 1 millisecond as an example, but in reality, it happens even faster. The speed of switching depends on your processor.

As the switching between tasks is so fast, it gives the impression that both tasks are running simultaneously. But it's important to note that even if you ask for both thread-1 and thread-2 to run at the same time, your processor and operating system decide which one to prioritize first, how much time to allocate to each, and in what order to execute them.

In summary, it's like juggling multiple tasks simultaneously. You start one task, switch to another when needed, and keep cycling through them until everything is done. This concept is called concurrency.

"Concurrency is a concept of managing the progress of multiple tasks at the same time, even if they are not executing simultaneously."

How do you achieve this? Using multi-threading and multi-processing again!
By now, you may have understood that Concurrency and Asynchronous Programming are the core concepts, while multi-threading and multi-processing are implementations of these concepts.

Global Interpreter Lock (GIL)

The concepts of concurrency and asynchronous programming are the same, regardless of what programming language you use. But the implementation of these concepts does depend on the programming language you choose.

When it comes to multi-threading, Python behaves bit weird. Let's understand this with a small example.

In my childhood, my younger brother and I used to play computer games. My mother made a rule that my brother should play first for 30 min and then I could play second for 30 min. This rule was to make sure that no one messed up the game by trying to do things at the same time. So I would wait for 30 min until my turn came to play the game for 30 min.

In the world of computers, programs are like me and my brother trying to play the game. The Global Interpreter Lock (GIL) is like the rule that only allows one, either me or my brother (or thread), to play the game (or execute Python bytecode) at a time.

GIL is a rule that allows only one thread at a time to run the Python bytecode. Global Interpreter Lock is a lock that protects access to Python objects, preventing multiple threads from executing Python bytecode simultaneously in a single process. This means that even in a multi-threaded Python program, only one thread can execute Python bytecode at any given time.

As a result, this causes performance limitations of multi-threading in CPU-bound tasks. Note that CPU-bound tasks are the tasks that rely a lot on CPU rather than IO operations. Mathematical computations, compression and decompression of files, and compilation of programs by a program compiler are a few examples of CPU-bound tasks that use more CPU.

Let's look at an example:

import time

def count_up():
    count = 0
    for i in range(100000000):
        count = count + i

def count_down():
    count = 0
    for i in range(100000000):
        count = count + i

if __name__ == "__main__":
    start_time = time.time()

    count_up()
    count_down()

    end_time = time.time()

    print(f"Time taken: {end_time - start_time} seconds")

In the above program, we are running the count_up() and count_down() functions sequentially one after the other. The count_up and count_down functions each iterate through a large range, calculating the cumulative sum of numbers. The output of the program is:

Time taken: 25.86127805709839 seconds

Let's write the same program using multi-threading as follows:

import threading
import time

def count_up():
    count = 0
    for i in range(100000000):
        count = count + i

def count_down():
    count = 0
    for i in range(100000000):
        count = count + i

if __name__ == "__main__":
    start_time = time.time()

    # Create two threads, each running a CPU-bound task
    thread1 = threading.Thread(target=count_up)
    thread2 = threading.Thread(target=count_down)

    # Start both threads
    thread1.start()
    thread2.start()

    # Wait for both threads to finish
    thread1.join()
    thread2.join()

    end_time = time.time()

    print(f"Time taken: {end_time - start_time} seconds")

Two threads thread1 and thread2 are created to run the count_up and count_down functions simultaneously. The program measures the time it takes for both threads to complete using the time module. The output of the program is:

Time taken: 24.498752117156982 seconds

Note that in your PC, the time taken for this program to complete can be different. If you observe, the output for the sequential and multi-threading versions of the program (25.8 seconds vs 24.98 seconds) there is not much difference between the time taken. This is because of GIL.

Global Interpreter Lock (GIL) in CPython, which allows only one thread to execute Python bytecode at a time, means that the execution time won't show a significant improvement compared to running the tasks sequentially.

This highlights the limitation of multithreading in Python for CPU-bound tasks due to GIL.

But it's worth noting that while the GIL prevents multiple threads from executing Python bytecode at the same time, it does not prevent threading altogether. Python threads can still be useful for I/O-bound tasks where threads spend most of their time waiting for external operations (like network or disk I/O) rather than performing CPU-intensive computations.

So how do you remove the limitations of GIL?

Use another flavor of Python

Python standard implementation is Cpython. This is the Python designed using the C language and is used most all over the world. To avoid GIL, we can use Jython (Python developed using Java), IronPython (Python developed using .NET), or PyPy (Python developed using Python).

You can check out the following resources for more info:

• Home | Jython
• IronPython.net
• PyPy

Use the multiprocessing module

Multiprocessing is a module that helps you take advantage of your multiple CPU cores for running your program in separate processes.

Each process has its own CPU core, memory, resources, and interpreter. Yes, every process has its own interpreter to run your Python code. If you have 4 CPU cores then you can run 4 processes, each having its own interpreter that executes your application in its own memory.

Let's write the program we discussed in the GIL concept using multi-processing as follows:

import multiprocessing
import time

def count_up():
    count = 0
    for i in range(100000000):
        count = count + i

def count_down():
    count = 0
    for i in range(100000000):
        count = count + i

if __name__ == "__main__":
    start_time = time.time()

    # Create two threads, each running a CPU-bound task
    process1 = multiprocessing.Process(target=count_up)
    process2 = multiprocessing.Process(target=count_down)

    # Start both threads
    process1.start()
    process2.start()

    # Wait for both threads to finish
    process1.join()
    process2.join()

    end_time = time.time()

    print(f"Time taken: {end_time - start_time} seconds")

The output for the program when you run it via your terminal or CMD is:

Time taken: 8.376060724258423 seconds

This represents a significant improvement in the time taken compared to techniques such as Sequential and Multi-threading (25.8 vs 24.4 vs 8.3 seconds). This illustrates how using multiprocessing can help reduce your CPU time.

Note that there will be overhead if you want to communicate between the processes. As a result, while multiprocessing can be effective for CPU-bound tasks, it may not be suitable for all scenarios, especially those involving frequent communication or large data transfers between processes.

Implement asynchronous programming!

If your program or application is not CPU bound heavy or depends more on CPU processing then using the asyncio module of Python, you can implement the concept of Asynchronous Programming (not waiting for task 1 to complete before starting task 2).

This module is mainly used in I/O operations. Asyncio allows you to write asynchronous code that cooperatively multitasks without relying on threads, which can help in situations where threads would otherwise be blocked waiting for I/O operations to complete.

By using asyncio, you can write non-blocking code that allows other tasks to run while waiting for I/O operations to finish, thus potentially improving overall concurrency and responsiveness of your application.

It's important to note that asyncio doesn't eliminate the GIL itself – rather, it provides an alternative concurrency model that can be more efficient for certain types of applications.

For CPU-bound tasks, asyncio may not provide the same performance benefits as multiprocessing or other parallelism techniques, as it still operates within the constraints of the GIL. In such cases, multiprocessing or other concurrency approaches may be more suitable.

Summary

In this article, you learned about the differences between Multi-threading, Multi-processing, Asynchronous Programming, and Concurrency. Let's review them briefly now:

Multi-threading:

• Definition: Multi-threading involves using multiple threads to execute tasks concurrently within a single process. Concurrently is nothing but the concept Concurrency here.
• Clarification: Threads share the same memory space, allowing them to communicate easily but requiring careful synchronization to avoid conflicts.

Multi-processing:

• Definition: Multi-processing involves using multiple processes to run your application or tasks. Each process has its own memory space and resources.
• Clarification: Processes are independent, and communication between them often requires inter-process communication (IPC) mechanisms. Each process operates in its own memory space.

Asynchronous Programming:

• Definition: Asynchronous programming is a programming concept that allows tasks to be executed separately from the main program flow. It doesn't necessarily imply concurrent execution.
• Clarification: Asynchronous tasks don't block the main program, enabling the program to continue its execution while waiting for the asynchronous tasks to complete. This is commonly used in I/O operations to improve efficiency.

Concurrency:

• Definition: Concurrency is a broader concept that refers to the ability of a system to execute multiple tasks in overlapping time intervals, that is switching between the tasks in specific intervals.
• Clarification: Concurrency doesn't guarantee simultaneous execution but rather focuses on efficiently managing multiple tasks by switching between them. It encompasses multi-threading, multi-processing, and asynchronous programming as ways to achieve concurrent execution.

I hope you now have a clear understanding of the concepts that may have seemed scary before. I understand it's mostly theoretical, but I want to ensure you grasp the concept well before diving into the practical aspect.

Until then, your friend here, Hemachandra, is signing off…

For more courses, you can visit my personal website.

Have a great day!

Isolation is a guarantee that concurrently running transactions should not interfere with each other. This is arguably the most important ACID property, because different DBMS can often have different default isolation levels. And you may need to change this based on what is needed for your application.

In a previous article, I explained the two main isolation levels used by most DBMS. These are the read committed and repeatable read isolation levels.

Pessimistic and optimistic concurrency controls essentially explain some of the ways a database is able to achieve these two isolation guarantees.

Table of Contents

1. Pessimistic Concurrency Control

2. Pessimistic Concurrency Control Analogy

3. Real-World Example of Pessimistic Concurrency Control

4. Pros and Cons of Pessimistic Concurrency Control

5. How it Guarantees the Read Committed Isolation Level

6. Optimistic Concurrency Control

7. Real-World Example of Optimistic Concurrency Control

8. Pros and Cons of Optimistic Concurrency Control

9. How it Guarantees the Repeatable Read Isolation Level

10. Bringing it Together

Pessimistic Concurrency Control

With pessimistic concurrency control, the DBMS assumes that conflicts between transactions are likely to occur. It is pessimistic – that is, it assumes that if something can go wrong, it will go wrong. This pessimism prevents conflicts from occurring by blocking them before they get a chance to start.

To prevent these conflicts, it locks the data that a transaction is using until the transaction is completed. This approach is 'pessimistic' because it assumes the worst-case scenario – that every transaction might lead to a conflict. The data is therefore locked in order to prevent conflicts from happening.

I've mentioned two technical terms here that need clarification: locks and conflict.

What are locks?

A lock is a mechanism used to control access to a database item, like a row or table. Locks ensure data integrity, if multiple transactions are occurring at the same time.

In very simple terms, a lock is analogous to a reservation on the database item. A reservation, be it a restaurant, hotel, or a train, prevents other people from using the resource you reserved for a fixed duration of time. Locks work in a similar way.

There are two types of locks: a read lock and a write lock.

A read lock can be shared by multiple transactions trying to read the same database item. But it blocks other transactions from updating that database item.

A write lock is exclusive – that is, it can only be held by a single transaction. A transaction with a write lock on a database item blocks every other transaction from reading or updating that database item.

What are conflicts?

A conflict refers to a situation where multiple transactions are attempting to access and modify the same data concurrently, in a way that could lead to inconsistencies or errors in the database.

A Library Analogy for Pessimistic Concurrency Control

First, let us describe an analogy for a write lock.

Imagine you're at a library, and you want to borrow a hard copy of a popular book, say, The Great Gatsby by F. Scott Fitzgerald.

Write locks are analogous to borrowing a physical book from the library

With a write lock, the librarian assumes that there can be conflicts over who gets to borrow the book. So, they implement a strict rule to avoid conflicts: only one person can hold the reservation for a physical book at a time.

When you reserve the book, no one else can borrow it. The book is available to be reserved again only once it is returned. This is similar to how a write lock works.

Write locks are exclusive. This means that they can only he held by a single transaction at any time. Similarly, reserving a physical book from the library means no one else has access to it. Only the person with the reservation can read the book, or write in it (although writing in a library book is bad form).

Read locks work a bit differently.

A read lock is analogous to someone making a reservation to borrow an e-book. Borrowing an e-book is not a very popular thing to do, but some libraries do have such a service.

Many people can make the same reservation for the same e-book without any conflict. One person borrowing an e-book version of The Great Gatsby does not stop others from doing the same. But no one who borrows an e-book can update it, by scribbling notes in it that can be seen by others, for example.

Read locks are analogous to borrowing an e-book from the library

Pessimistic concurrency control is very safe because it prevents conflicts from occurring by blocking them before they get a chance to start. A write lock on a database item prevents other transactions from reading or updating that item while that lock is held, similar to to how a library stops more than one person from trying to borrow the same physical book at the same time.

A read lock on a database item allows other transactions to also obtain a read lock for that item, but prevents transactions from updating that item. This is analogous to borrowing an e-book, where multiple people can borrow the same e-book at the same time, but can’t make any updates to it.

A Simple Real-World Example of Pessimistic Concurrency Control in Action

Let's illustrate how pessimistic concurrency control works using a simple example involving a bank balance database table. Assume we have a table named Accounts with the following columns: AccountID and Balance.

Database columns for AccountID and Balance

Two transactions, T1 and T2, intend to update the balance of account 12345. T1 wants to withdraw $300, and T2 wants to deposit $400. At the end of these two transactions, the account balance should read $1600

Here are the steps of how this will work using write locks:

1. Start of T1 (Withdrawal): T1 requests to update the balance of AccountID 12345. The database system places an exclusive write lock on the row for AccountID 12345, preventing other transactions from reading or writing to this row until T1 is completed. T1 reads the balance ($1500).

2. T1 Processing: T1 calculates the new balance as $1200 ($1500 - $300).

3. Commit T1: T1 writes the new balance ($1200) back to the database. Upon successful commit, T1 releases the exclusive lock on AccountID 12345.

4. Start of T2 (Deposit) After T1 Completes: Now that T1 has completed and the lock is released, T2 can start. T2 attempts to read and update the balance for AccountID 12345. The database system places an exclusive lock on the row for AccountID 12345 for T2, ensuring no other transactions can interfere. T2 reads the updated balance ($1200).

5. T2 Processing: T2 calculates the new balance as $1600 ($1200 + $400).

6. Commit T2: T2 writes the new balance ($1600) back to the database. Upon successful commit, T2 releases the exclusive lock on AccountID 12345.

7. Result: The Accounts table is updated using locks After T1: $1200 After T2: $1600

Without a write lock in this example, T1 and T2 could read the original balance of $1500 at the same time. So, instead of a balance of $1200 after T1 has committed, T2 still reads the original balance of $1500 and adds $400. This would cause the final balance to be $1500 + $400 = $1900 (instead of $1600).

Absence of locking has created free money, which is never a bad thing for a customer. But, if money can be conjured out of thin air because of these conflicts, it can also vanish, and accidentally shrinking bank balances are a quick way to make customers unhappy.

Benefits and Challenges of Pessimistic Concurrency Control

Just like reserving a book ensures that it's set aside for one person, pessimistic concurrency control locks data for a single transaction. Other transactions cannot access or modify this data until the lock is released.

This method prevents two people from trying to take out the same popular book at the same time, thereby avoiding disputes. Similarly, in databases, it stops conflicts due to concurrent transactions before they get a chance to start.

But this approach can be inefficient. The reserved book might sit on the reserved shelf for a while, stopping other people from reading it.

In databases, this locking mechanism can lead to underutilisation of resources and a slowdown in the speed transactions take to complete, since a subset of the data is locked and inaccessible to other transactions.

How Pessimistic Concurrency Controls Guarantee the Read Committed Isolation Level

So, how exactly does pessimistic concurrency control work in ensuring the isolation guarantee, that is the “I” in ACID? The implementation details can vary across different DBMS. But the explanation here shows the general approach.

Recall that the read committed isolation level prevents dirty writes and dirty reads.

Preventing Dirty Writes

Overwriting data that has already been written by another transaction but not yet committed is called a dirty write. A common approach to preventing dirty writes is to use pessimistic concurrency control. For example, by using a write lock at the row level.

When a transaction wants to modify a row, it acquires a lock on that row and holds it until the transaction is complete. Recall that write locks can only be held by a single transaction. This prevents another transaction from acquiring a lock to modify that row.

Preventing Dirty Reads

Reading data from another transaction that has not yet been committed is called a dirty read. Dirty reads are prevented using either a read or write lock. Once a transaction acquires a read lock on a database item, it will prevent updates to that item.

But what happens if you are trying to read something that is already being updated but the transaction has not yet committed? In this instance, the write lock saves the day again.

Since write locks are exclusive (can’t be shared with other transactions), any transaction wanting to read the same database item will have to wait until the transaction with the write lock is committed (or aborted, if it fails). This prevents other transactions from reading uncommitted changes.

Optimistic Concurrency Control

With optimistic concurrency control, transactions do not obtain locks on data when they read or write. The "Optimistic" in the name comes from assuming that conflicts are unlikely to occur, so locks are not needed. If something does go wrong though, conflicts will still be prevented and everything will be OK.

Unlike pessimistic concurrency control – which prevents conflicts from occurring by blocking them before they get a chance to start – optimistic concurrency control checks for conflicts at the end of a transaction.

With optimistic concurrency control, multiple transactions can read or update the same database item without acquiring locks. How exactly does this work?

Every time a transaction wants to update a database item, say a row, it will also read two additional columns added to every table by the DBMS – the timestamp and the version number. Before that transaction is committed, it checks if another transaction has made any change(s) to that row by confirming if the version number and timestamp are the same.

If they have changed, that means another transaction has updated that row, so the initial transaction will have to be retried.

A Simple Real-World Example of Optimistic Concurrency Control in Action

Let's illustrate how optimistic concurrency control works using a simple example involving a bank balance database table. Assume we have a table named Accounts with the following columns: AccountID, Balance, VersionNumber, and Timestamp.

Table showing AccountID, Balance, VersionNumber, and Timestamp columns

Two transactions, T1 and T2, intend to update the balance of account 12345 at the same time. T1 wants to withdraw $200, and T2 wants to deposit $300. At the end of these two transactions, the account balance should read $1100

Here are the steps of how this will work:

1. Start of Transactions: T1 reads the balance, version number, and timestamp for AccountID 12345. Simultaneously, T2 reads the same row with the same balance, version number, and timestamp.

2. Processing: T1 calculates the new balance as $800 ($1000 - $200) but does not write it back immediately. T2 calculates the new balance as $1300 ($1000 + $300) but also waits to commit.

3. Attempt to Commit T1: Before committing, T1 checks the current VersionNumber and Timestamp of AccountID 12345 in the database. Since no other transaction has modified the row, T1 updates the balance to $800, increments the VersionNumber to 2, updates the Timestamp, and commits successfully.

4. Attempt to Commit T2: T2 attempts to commit by first verifying the VersionNumber and Timestamp. T2 finds that the VersionNumber and Timestamp have changed (now VersionNumber is 2, and Timestamp is updated), indicating another transaction (T1) has updated the row. Since the version number and timestamp have changed, T2 realises there was a conflict.

5. Resolution for T2: T2 must restart its transaction. It re-reads the updated balance of $800, the new VersionNumber 2, and the updated Timestamp. T2 recalculates the new balance as $1100 ($800 + $300), updates the VersionNumber to 3, updates the Timestamp, and commits successfully.

Result: The Accounts table is updated sequentially and safely without any locks: After T1: $800, VersionNumber: 2. After T2: $1100, VersionNumber: 3.

Benefits and Challenges of Optimistic Concurrency Control

On the positive side, avoiding locks allows for high levels of concurrency. This is particularly beneficial in read-heavy workloads where transactions are less likely to conflict, allowing the system to handle more transactions in a given period. For example, database backups and analytical queries typically used in a data warehouse.

But in scenarios where conflicts are frequent, the cost of repeatedly rolling back and retrying transactions can outweigh the benefits of avoiding locks, making optimistic concurrency control less efficient

How Optimistic Concurrency Controls Guarantee the Repeatable Read Isolation level

The repeatable read is more strict isolation level in that it has the same guarantees as read committed isolation, plus it guarantees that reads are repeatable.

A repeatable read guarantees that if a transaction reads a row of data, any subsequent reads of that same row of data within the same transaction will yield the same result, regardless of changes made by other transactions. This consistency is maintained throughout the duration of the transaction.

How can a repeatable read be achieved? Pessimistic control using a read lock can help with this, since a transaction with a read lock on a database item will prevent that item from being updated. But this can be inefficient, since a long running read transaction can block updates from happening to that database item.

Multi-Version Concurrency Control (MVCC) is a concurrency control method used by some DBMS to allow multiple transactions to access the same data simultaneously without locking the data. This makes it a popular choice for reducing lock contention and improving the scalability of databases.

MVCC achieves this by keeping multiple versions of data objects, which helps to manage different visibility levels for transactions depending on their timestamps or version numbers.

Bringing it Together

A lock is a mechanism used to control access to a database item, like a row or table. In very simple terms, it is analogous to a reservation on a database item.

Pessimistic concurrency control assumes the worst. It assumes that conflicts are likely to happen, so locks are used to block transactions that can cause conflicts before they even get a chance to start.

In situations where conflicts are common, such as a write heavy application, this approach can prevent the overhead associated with frequent rollbacks and retries (which happens in optimistic concurrency control) by ensuring exclusive access to database items during transactions.

Optimistic concurrency control assumes the best. It assumes that conflicts are unlikely to occur, so locks are not needed to stop transactions before they start. Instead, potential conflicts are checked at the end of a transaction and if any are found, the transaction is aborted or retried.

Optimistic concurrency control is useful for read heavy transactions with infrequent writes, as it allows multiple transactions to proceed without the need to use a lock, which can be inefficient.

Concurrency is a programming term you'll need to be familiar with if you code in C++. It's especially relevant if you want to get more out of your available compute resources.

But concurrency comes with certain problems. Some of them can be solved using mutexes, although mutexes have their own set of challenges.

This is where atomics come in. However, atomics require an understanding of the memory model of a computer, which is a challenging topic to fully grasp.

This is what we'll cover in this article. Hopefully you'll gain a good understanding of how memory ordering works and how to use atomics in conjunction with memory ordering to build a lock-free queue in C++.

Pre-requisites

To get the most out of this guide, you need to have written some basic concurrent programs. Programming experience with C++ helps.

Note: If you want to actually compile the code and run it, make sure to do so with the TSan flag enabled for the CLang compiler. TSan is a reliable way of detecting data races in your code, instead of trying to repeatedly run the code in a loop hoping for a data race to occur.

Getting Started

Imagine you have some concurrent code operating on some shared data in memory. You have two threads or processes, one writing and one reading on some shared piece of state.

Processes sharing memory

The "safe" way of dealing with this is mutexes. But mutexes tend to add overhead. Atomics are a more performant but much more complicated way of dealing with concurrent operations.

What Are Atomics in C++?

This section is very simple. Atomics are simply operations or instructions that cannot be split by the compiler or the CPU or re-ordered in any way.

The simplest possible example of an atomic in C++ is an atomic flag. It looks like this:

#include <atomic>

std::atomic<bool> flag(false);

int main() {
  flag.store(true);
  assert(flag.load() == true);
}

We define an atomic boolean, initialise it, and then call store on it. This method sets the value on the flag. You can then load the flag from memory and assert its value.

Operations with Atomics

The operations you can perform with atomics are straightforward:

1. You can store some value into them with the store() method. This is a write operation.
2. You can load some value from them with the load() method. This is a read operation.
3. You can do a Compare-and-Set (CAS) with them using the compare_exchange_weak() or compare_exchange_strong() method. This is a read-modify-write (RMW) operation.

The important thing to remember is that each of these cannot be split into separate instructions.

Note: There are more methods available, but this is all we need for now.

There are various atomics available in C++ and you can use them in combination with memory orderings.

Memory Ordering

This section is a lot more complicated and is the meat of the matter. There are some great references to help you understand memory ordering in more depth that I've linked at the bottom.

Why Memory Ordering Matters

The compiler and CPU are capable of re-ordering your program instructions, often independently of one another. That is, your compiler can re-order instructions and your CPU can re-order instructions again. See this post for more detail.

But this is only allowed if the compiler can definitely not establish a relationship between the two sets of instructions.

For instance, the below code can be re-ordered because there is no relationship between the assignment to x and the assignment to y. That is, the compiler or CPU might assign y first and then x. But, this doesn't change the semantic meaning of your program.

int x = 10;
int y = 5;

On the other hand, the following code example cannot be re-ordered because the compiler cannot establish the absence of a relationship between x and y. It's pretty easy to see here because y depends on the value of x.

int x = 10;
int y = x + 1;

This doesn't seem like a big problem – until there's multi-threaded code. Let's see what I mean with an example.

Intuition for Ordering

#include <cassert>
#include <thread>

int data = 0;

void producer() {
  data = 100;  // Write data
}

void consumer() {
  assert(data == 100);
}

int main() {
  std::thread t1(producer);
  std::thread t2(consumer);
  t1.join();
  t2.join();
  return 0;
}

The multi-threaded example above will fail to compile with TSan because there is a clear data race when thread 1 is trying to set the value of data and thread 2 is trying to read the value of data. The easy answer here is a mutex to protect the write and read of data, but there's a way to do this with an atomic boolean.

We loop on the atomic boolean until we find that it is set to the value we are looking for and then check the the value of data.

#include <atomic>
#include <cassert>
#include <thread>

int data = 0;
std::atomic<bool> ready(false);

void producer() {
  data = 100;
  ready.store(true);  // Set flag
}

void consumer() {
  while (!ready.load())
    ;
  assert(data == 100);
}

int main() {
  std::thread t1(producer);
  std::thread t2(consumer);
  t1.join();
  t2.join();
  return 0;
}

When you compile this with TSan, it doesn't complain about any race conditions. Note: I'm going to refer back to why TSan doesn't complain here a little further on.

Now, I'm going to break it by adding a memory ordering guarantee to it. Just replace ready.store(true); with ready.store(true, std::memory_order_relaxed); and replace while (!ready.load()) with while (!ready.load(std::memory_order_relaxed)).

TSan will complain that there is a data race. But, why is it complaining?

The issue here is that we have no order among the operations of the two threads anymore. The compiler or CPU is free to re-order instructions in the two threads. If we go back to our abstract visualisation from earlier, this is what it looks like now:

Relaxed memory model ordering

The visualisation above shows us that our two processes (threads) have no way of agreeing upon the current state or the order in which that state has changed.

Once process 2 determines that the flag has been set to true, so it tries to read the value of data. But, thread 2 believes that the value of data has not yet changed even though it believes the value of flag has been set to true.

This is confusing because the classic model of interleaving concurrent operations doesn't apply here. In the classic model of concurrent operations, there is always some order that can be established. For instance, we can say that this is one possible scenario of operations:

Thread 1                  Memory                  Thread 2
---------                 -------                 ---------
  |                          |                          |
  |   write(data, 100)       |                          |
  | -----------------------> |                          |
  |                          |     load(ready) == true  |
  |                          | <----------------------  |
  |                          |                          |
  |   store(ready, true)     |                          |
  | -----------------------> |                          |
  |                          |                          |
  |                          |       read(data)         |
  |                          | <----------------------  |
  |                          |                          |

But, the above graphic assumes that both threads have agreed upon some global order of events, which isn't true at all anymore! This is still confusing for me to wrap my head around.

In a memory_order_relaxed mode, two threads have no way of agreeing upon the order of operations on shared variables. From thread 1's point of view, the operations it executed were the following:

write(data, 100)
store(ready, true)

However, from thread 2's point of view, the order of operations it saw thread 1 execute were:

store(ready, true)
write(data, 100)

Without agreeing upon the order in which operations occurred on shared variables, it isn't safe to make changes to those variables across threads.

Okay, let's fix the code by replacing std::memory_order_relax with std::memory_order_seq_cst.

So, ready.store(true, std::memory_order_relaxed); becomes ready.store(true, std::memory_order_seq_cst); and while (!ready.load(std::memory_order_relaxed)) becomes while (!ready.load(std::memory_order_seq_cst)).

If you run this again with TSan, there are no more data races. But why did that fix it?

Memory Barrier

So, we saw our problem earlier was about two threads being unable to agree upon a single view of events and we wanted to prevent that. So, we introduced a barrier using sequential consistency.

Thread 1                  Memory                  Thread 2
---------                 -------                 ---------
  |                          |                          |
  |   write(data, 100)       |                          |
  | -----------------------> |                          |
  |                          |                          |
  |  ================Memory Barrier===================  |
  |   store(ready, true)     |                          |
  | -----------------------> |                          |
  |                          |   load(ready) == true    |                   
  |                          | <----------------------  |
  |  ================Memory Barrier===================  |
  |                          |                          |
  |                          |       read(data)         |
  |                          | <----------------------  |
  |                          |                          |

The memory barrier here says that nothing before the store operation and nothing after the load operation can be re-ordered. That is, thread 2 now has a guarantee that the compiler or the CPU will not place the write to data after the write to the flag in thread 1. Similarly, the read operation in thread 2 cannot be re-ordered above the memory barrier.

The region inside the memory barrier is akin to a critical section that a thread needs to a mutex to enter. We now have a way to synchronise the two threads on the order of events across them.

This brings us back to our classic model of interleaving in concurrency because we now have an order of events both threads agree upon.

Types Of Memory Order

There are 3 main types of memory order:

1. Relaxed memory model (std::memory_order_relaxed)
2. Release-acquire memory model (std::memory_order_release and std::memory_order_acquire)
3. Sequentially consistent memory order (std::memory_order_seq_cst)

We already covered 1 and 3 in the examples above. The second memory model literally lies between the other two in terms of consistency.

#include <atomic>
#include <cassert>
#include <iostream>
#include <thread>

int data = 0;
std::atomic<bool> ready(false);

void producer() {
  data = 100;
  ready.store(true, std::memory_order_release);  // Set flag
}

void consumer() {
  while (!ready.load(std::memory_order_acquire))
    ;
  assert(data == 100);
}

int main() {
  std::thread t1(producer);
  std::thread t2(consumer);
  t1.join();
  t2.join();
  return 0;
}

The above is the same example from earlier, expect with std::memory_order_release used for ready.store() and memory_order_acquire used for read.load(). The intuition here for ordering is similar to the pervious memory barrier example.

Except, this time the memory barrier is formed on the pair of ready.store() and ready.load() operations and will only work when used on the same atomic variable across threads.

Assuming you have a variable x being modified across 2 threads, you could do x.store(std::memory_order_release) in thread 1 and x.load(std::memory_order_acquire) in thread 2 and you would have a synchronization point across the two threads on this variable.

The difference between the sequentially consistent model and the release-acquire model is that the former enforces a global order of operations across all threads, while the latter enforces an order only among pairs of release and acquire operations.

Now, we can revisit why TSan didn't complain about a data race initially when there was no memory order specified. It's because C++ by default assumes a std::memory_order_seq_cst when no memory order is specified. Since this is the strongest memory mode, there is no data race possible.

Hardware Considerations

Different memory models have different performance penalties on different hardware.

For instance, the x86 architectures instruction set implements something called total store ordering (TSO). The gist of it is that the model resembles all threads reading and writing to a shared memory. You can read more about that here.

This means that the x86 processors can provide sequential consistency for a relatively low computational penalty.

On the other side are the ARM family of processors has a weakly ordered instruction set architecture. This is because each thread or process reads and writes to its own memory. Again, the link above provides context.

This means that the ARM processors provide sequential consistency for a much higher computational penalty.

How to Build a Concurrent Queue

I'm going to use the operations we have discussed so far to build the basic operations of a lock-free concurrent queue. This is by no means even a complete implementation, just my attempt to re-create something basic using atomics.

I'm going to represent the queue using a linked list and wrap each node inside an atomic.

class lock_free_queue {
 private:
  struct node {
    std::shared_ptr<T> data;
    std::atomic<node*> next;

    node() : next(nullptr) {}  //  initialise the node
  };

  std::atomic<node*> head;
  std::atomic<node*> tail;
}

Now, for the enqueue operation, this is what it's going to look like:

void enqueue(T value) {
    std::shared_ptr<T> new_data = std::make_shared<T>(value);
    node* new_node = new node();
    new_node->data = new_data;

    //  do an infinite loop to change the tail
    while (true) {
      node* current_tail = this->tail.load(std::memory_order_acquire);
      node* tail_next = current_tail->next;

      //  everything is correct so far, attempt the swap
      if (current_tail->next.compare_exchange_strong(
              tail_next, new_node, std::memory_order_release)) {
        this->tail = new_node;
        break;
      }
    }
  }

The main focus is on the load and compare_exchange_strong operations. The load works with an acquire and the CAS works with a release so that reads and writes to the tail are synchronized.

Similarly for the dequeue operation:

std::shared_ptr<T> dequeue() {
    std::shared_ptr<T> return_value = nullptr;

    //  do an infinite loop to change the head
    while (true) {
      node* current_head = this->head.load(std::memory_order_acquire);
      node* next_node = current_head->next;

      if (this->head.compare_exchange_strong(current_head, next_node,
                                             std::memory_order_release)) {
        return_value.swap(next_node->data);
        delete current_head;
        break;
      }
    }
    return return_value;
  }

Note: This queue doesn’t handle the ABA problem. It's out of the scope of this tutorial to bring in hazard pointers, so I’m leaving that out – but you can look into them if you're curious.

Conclusion

So there you have it – atomics in C++. They're difficult to reason about but give you massive performance benefits. If you want to see production grade implementations of lock-free data structures, you can read more here.

Typically, you'd see lock-based concurrent data structures put into production, but there is increasing research around the area of lock-free and wait-free implementations of concurrent algorithms to improve compute efficiency.

If you want to read this and other tutorials, you can visit my website here.

Imagine you're running a bustling restaurant. Orders fly in, dishes need cooking, tables require clearing, and happy customers demand attention. How can you handle all this chaos without everything turning into a total disaster? This is where concurrency comes in.

In the world of server-side development, things can get just as hectic. Users fire off requests, databases need querying, calculations demand crunching, and all this needs to happen smoothly and efficiently. Without concurrency, it's like having one overworked chef trying to do everything.

But with concurrency, it's like adding a whole team of skilled multitaskers to your kitchen. You have specialized "coroutines" handling different tasks – one taking orders, another cooking, one washing dishes, and so on. Each works simultaneously, but in a coordinated way, ensuring your customers get their delicious meals fast and with a smile.

Examples of Concurrency

Here are some real-world examples of how concurrency works in server-side development:

Serving multiple users at once: Imagine your website gets slammed by eager shoppers during a flash sale. Without concurrency, each user request would have to wait its turn, leading to frustratingly slow loading times.

But with coroutines, multiple requests can be processed simultaneously, keeping everyone happy and shopping with ease.

Processing data pipelines: Do you work with really large datasets? Concurrency can break them down and analyze them bit by bit.

Handling asynchronous requests: Think of API calls, database queries, or file uploads. These often take time to complete. With concurrency, your main program doesn't have to sit and wait. It can launch separate coroutines to handle these requests and get back to other tasks, keeping everything running smoothly.

Concurrency isn't just some fancy buzzword – it's a powerful tool that lets you handle the ever-growing demands of server-side development with ease and efficiency.

Now, we'll dive deeper into the differences between threads and coroutines, the fundamental building blocks of concurrent programming in Kotlin.

Threads vs. Coroutines

Now, let's talk about the two main players on this team: threads and coroutines.

What are Threads?

Threads are powerful, and capable of handling complex tasks – but they're also a bit resource-hungry. Each thread requires its memory space and attention from the operating system.

This can be great for demanding tasks like breaking down big numbers or sending critical emails, but if you have too many threads running around, your server starts getting overwhelmed, leading to slowdowns and even crashes.

What are Coroutines?

Coroutines are lightweight, flexible, and able to switch between tasks in a flash. They don't need their dedicated tables, they share resources efficiently, and they can even pause their work gracefully if something else needs immediate attention.

Think of them as taking orders, checking on tables, and cleaning dishes – always contributing, but in a light and adaptable way.

Table showing differences bewteen threads and coroutines.

So, which of them should you choose? Well, it depends! For heavy lifting, threads are strong and reliable. But for the everyday hustle and bustle, coroutines are your agile friends.

Kotlin favours coroutines for concurrent programming because they're more efficient and easier to manage, especially when dealing with lots of asynchronous tasks.

How to Launch Coroutines

In your server-side code, there are two main ways to launch coroutines.

Launch and Forget: Using the launch Keyword

This fires up a new coroutine and sends it off to its assigned task, but doesn't wait for it to finish.

Here's how it looks in code:

fun updateCustomerOrder(
  orderId: Int,
  newDish: String
) {
  launch {
    updateOrderInDatabase(orderId, newDish)
  }
  sendConfirmationEmail(orderId)
}

This is ideal for tasks that don't need immediate feedback. You can launch multiple coroutines like this without overloading your server, as they work independently and share resources efficiently.

Wait and Receive: The async Keyword

Now, let's say you need to know the total bill before you can settle up with the customer. This is where async comes in. It's like sending an assistant to calculate the bill and then patiently waiting for them to return with the answer.

Here's how it works:

fun calculateBill(orderId: Int): Double {
  val billResult = async {
    calculateBillFromDatabase(orderId)
  }
  val totalBill = billResult.await()
  return totalBill
}

This is useful for tasks where you need the result before proceeding. The async keyword creates a coroutine that calculates the bill in the background (performs the task), but the main program pauses and waits until the result is ready.

Channels: Communication Lifelines

Channels allow coroutines to send and receive messages in a safe and organized way, even when things get crazy. Here's how they work:

One-Way Streets: Send Channels

A send channel allows one coroutine to send a message that another coroutine can receive on the other end. Only one direction, just like a one-way street.

Bi-Directional Bridges: Receive Channels

Receive channels allow a coroutine to wait for and receive a message that was sent on the corresponding send channel. Two-way communication, like a bridge in both directions.

Putting Channels to Work:

Here's an example of using channels to update customer orders:

val orderUpdatesChannel = Channel<Pair<Int, String>>()

fun updateCustomerOrder(orderId: Int, newDish: String) {
  launch {
    updateOrderInDatabase(orderId, newDish)
    orderUpdatesChannel.send(orderId to newDish)
  }
}

fun sendOrderConfirmationEmails() {
  while (true) {
    val orderUpdate = orderUpdatesChannel.receive()
    val (updatedOrderId, newDish) = orderUpdate
    sendConfirmationEmail(updatedOrderId)
  }
}

In this case, the updateCustomerOrder coroutine sends the updated order details through the orderUpdatesChannel. The sendOrderConfirmationEmails coroutine constantly waits on the channel, receiving updates one by one and sending out confirmation emails.

Structured Concurrency

Think of structured concurrency like assigning roles and responsibilities in your kitchen. It helps you:

1. Define the scope: You can create coroutine contexts that provide specific resources and boundaries for your coroutines to work within. This keeps things organized and prevents rogue coroutines from mucking up the system.
2. Cancellation: Imagine a customer changing their order halfway through cooking. Structured concurrency allows you to easily cancel the wrong dish and start preparing the new one, without everything crumbling into a greasy mess.
3. Supervision: Structured concurrency lets you set up supervisors who monitor their "child" coroutines, ensuring they finish their tasks properly or handle errors gracefully.

Key tools for structured concurrency:

Dispatchers: They control where and how your coroutines run, like assigning them to specific threads or pools.

Jobs: Think of them as the tasks themselves, assigned to specific coroutines within a context. You can track their progress, cancel them if needed, and keep things organized.

Supervisor Jobs: They supervise child jobs, handling failures and propagating them properly, preventing errors from cascading and ruining the whole meal.

Different types of tasks in your server-side code demand different environments. This is where the various dispatcher types come in:

1. The Main Dispatcher: Think of it as the front of the house, handling UI updates and interacting directly with the customer (user). It's a single thread, ensuring smooth and responsive interactions without any chaotic multitasking.
2. The Default Dispatcher: This is the workhorse for general computation tasks. It uses a thread pool, assigning coroutines to available threads. Great for CPU-intensive calculations, database processing, and other heavy lifting.
3. The IO Dispatcher: This dispatcher handles blocking I/O operations like network calls, file access, or database queries.
4. Custom Dispatchers: You can create custom dispatchers with specific thread pools or even single threads, tailoring them to the needs of your unique tasks.

Choosing the right dispatcher is very important for optimizing performance and avoiding congestion. Here are some tips:

• Use the main dispatcher only for UI updates and user interaction.
• Default dispatcher for most CPU-intensive tasks.
• IO dispatcher for any blocking I/O operations.
• Consider custom dispatchers for specific high-volume or sensitive tasks.

Flows

Imagine your restaurant is booming, orders flying in faster than the chefs can cook. How do you handle this data deluge without everything turning into a soggy mess?

Think of Flows as streams of data flowing through your server-side code, like the continuous stream of orders coming from the tables. Instead of dealing with each dish individually, Flows let you handle the data continuously, adapting to the pace and transforming it on the fly.

Here's why Flows are a game-changer:

• No more waiting around for data to arrive. Flows let you launch coroutines to handle the stream asynchronously, keeping your main program free to keep taking orders and managing the overall experience.
• Flows allow you to extract insights, generate reports, or trigger actions in real-time.
• Flows automatically adjust their pace based on downstream processing capabilities, preventing backlogs and ensuring smooth data flow through your system.
• Errors are handled delicately with flows, which transmit them without bringing down the entire service.

Here's an example of using Flows to process a stream of sensor readings:

val sensorReadingsFlow = sensorManager.readings()
    .filter { reading -> reading.temperature > 80 } 
    .map { reading -> "Alert! Sensor ${reading.id} at ${reading.temperature} degrees!" }    
    .launchIn(Dispatchers.IO) 
    .onEach { message -> logger.warn(message) } 
    .collectLatest { message -> sendPushNotification(message) }

This Flow reads sensor readings continuously, filters for high temperatures, transforms them into alert messages, and sends them to users, all in a non-blocking, resilient way.

Flows are a powerful tool for modern server-side development. They let you embrace the world of asynchronous data processing with grace and efficiency, keeping your code responsive, scalable, and ready to handle whatever data banquet your users throw at you.

Testing Your Concurrency Setup

So, you've built your server-side kitchen with agile coroutines, smooth communication channels, and efficient dispatchers. Your data flows like a rushing stream, and your customers (users) are singing your praises. But before you hang the "Open" sign permanently, there's one more crucial step: testing your concurrency magic.

Testing helps you uncover edge cases, prevent crashes, and ensure everything runs smoothly even under heavy load.

Here are some key points for testing concurrent code:

• Focus on isolation: Test individual coroutines and channels in isolation to identify problems in specific components. You can use libraries like kotlinx-coroutines-test to control time and simulate asynchronous behavior.
• Mock dependencies: External dependencies like databases or APIs can add complexity. Mocking these dependencies allows you to focus on coroutine behavior without introducing real-world uncertainties.
• Verify state changes: Ensure coroutines update data structures and flags correctly as they execute. Assert expected values at different points in the execution flow.
• Stress test performance: Push your code to its limits with high volume or complex scenarios. This helps you identify bottlenecks and ensure your system scales gracefully.

Remember, testing concurrent code is not always straightforward. Be patient, experiment with different methods, and don't be afraid to seek help from experienced developers and testing frameworks.

Conclusion

We've explored agile coroutines dancing across tasks, efficient channels keeping them in sync, and smart dispatchers directing the flow. We've seen how Flows seamlessly handle data streams, and we've learned the importance of robust testing to ensure everything runs smoothly.

Now, with this knowledge, you should understand the concept of concurrency in your Kotlin applications. Embrace it, use it wisely, and watch your applications soar to new heights of responsiveness, scalability, and efficiency.

Concurrency refers to a programming language's ability to deal with lots of things at once.

A good way to understand concurrency is by imagining multiple cars traveling on two lanes. Sometimes the cars overtake each other, and sometimes they stop and let others pass by.

Another good example is when your computer runs multiple background tasks like messaging, downloading movies, running the operating system, and so on – all at once.

Parallelism means doing lots of things simultaneously and independently. It might sound similar to concurrency, but it’s actually quite different.

Let's understand it better with the same traffic example. In this case, cars travel on their own road without intersecting each other. Each task is isolated from all other tasks. Concurrent tasks can be executed in any given order.

This is a non-deterministic way to achieve multiple things at once. True parallel events require multiple CPUs.

Illustration showing difference between parallelism and concurrency

What is a Goroutine?

A goroutine is an independent function that executes simultaneously in some separate lightweight threads managed by Go. GoLang provides it to support concurrency in Go.

Here's an example of what a goroutine looks like:

package main

import (
    "fmt"
    "time"
)

func main() {
    go helloworld()
    time.Sleep(1 * time.Second)
    goodbye()
}

func helloworld() {
    fmt.Println("Hello World!")
}

func goodbye() {
    fmt.Println("Good Bye!")
}

In this example, first, the main goroutine starts. Then it invokes the helloworld() function, and the helloworld goroutine starts.

After the helloworld goroutine finishes its operation, the main goroutine waits for 1 second and invokes the goodbye() function.

If you omit the time function in main, then it will exit before the helloworld() finishes its execution.

Let's understand the steps involved here:

1. main goroutine starts
2. Invokes helloworld and helloworld goroutine starts
3. If there is no pause using the sleep method, the main will then invoke goodbye() and exit before the helloworld goroutine finishes its execution.

Without time.Sleep():

$ go run HelloWorld.go 
Good Bye!

After adding time.Sleep(), the helloworld goroutine is able to finish its execution before main exits:

$ go run HelloWorld.go 
Hello World!
Good Bye!

What are WaitGroups?

You can use WaitGroups to wait for multiple goroutines to finish. A WaitGroup blocks the execution of a function until its internal counter becomes 0.

Let's see a simple code snippet:

package main

import (
    "fmt"
)

func main() {
    go helloworld()
    go goodbye()
}

func helloworld() {
    fmt.Println("Hello World!")
}

func goodbye() {
    fmt.Println("Good Bye!")
}

Output

$ go run HelloWorld.go 

$

If we run the above program, it doesn't print anything. This is because the main function got terminated as soon as those two goroutines started executing. So, we can use Sleep which pauses the execution of the main function. It looks like this:

package main

import (
    "fmt"
    "time"
)

func main() {
    go helloworld()
    go goodbye()
    time.Sleep(2 * time.Second)
}

func helloworld() {
    fmt.Println("Hello World!")
}

func goodbye() {
    fmt.Println("Good Bye!")
}

Here's the output:

$ go run HelloWorld.go 
Good Bye!
Hello World!

Here, the main function was blocked for 2 seconds and all the goroutines were executed successfully.

Blocking the method for 2 seconds might not create any problems. But at the production level, where each millisecond is vital, millions of concurrent requests can create a huge problem.

You can solve this problem using sync.WaitGroup like this:

package main

import (
    "fmt"
    "sync"
)

func main() {
    var wg sync.WaitGroup
    wg.Add(2)
    go helloworld(&wg)
    go goodbye(&wg)
    wg.Wait()
}

func helloworld(wg *sync.WaitGroup) {
    defer wg.Done()
    fmt.Println("Hello World!")
}

func goodbye(wg *sync.WaitGroup) {
    defer wg.Done()
    fmt.Println("Good Bye!")
}

Output

$ go run HelloWorld.go 
Good Bye!
Hello World!

The output is the same as the previous one, but it doesn't block the main for 2 seconds.

1. wg.Add(int): This method indicates the number of goroutines to wait. In the above code, I have provided 2 for 2 different goroutines. Hence the internal counter wait becomes 2.
2. wg.Wait(): This method blocks the execution of code until the internal counter becomes 0.
3. wg.Done(): This will reduce the internal counter value by 1.

NOTE: If a WaitGroup is explicitly passed into functions, it should be added by a pointer.

What are Channels?

In concurrent programming, Go provides channels that you can use for bidirectional communication between goroutines.

Bidirectional communication means that one goroutine will send a message and the other will read it. Sends and receives are blocking. Code execution will be stopped until the write and read are done successfully.

Channels are one of the more convenient ways to send and receive notifications.

There are a couple different types of channels:

Unbuffered channel: Unbuffered channels require both the sender and receiver to be present to be successful operations. It requires a goroutine to read the data, otherwise, it will lead to deadlock. By default, channels are unbuffered.

Buffered channel: Buffered channels have the capacity to store values for future processing. The sender is not blocked until it becomes full and it doesn't necessarily need a reader to complete the synchronization with every operation.

If a space in the array is available, the sender can send its value to the channel and complete its send operation immediately.

After its execution, if a receiver comes, the channel will start sending values to the receiver and it will start its operation once it receives the values. As the sender and receiver are operating at different times, this is called asynchronous communication.

Here's an example:

Syntax to declare a channel
ch := make(chan Type)

Declaration of channels based on directions
1. Bidirectional channel : chan T
2. Send only channel: chan <- T
3. Receive only channel: <- chan T

How to write and read from a channel

package main

import (
    "fmt"
    "time"
)

func main() {
    msg := make(chan string)
    go greet(msg)
    time.Sleep(2 * time.Second)

    greeting := <-msg

    time.Sleep(2 * time.Second)
    fmt.Println("Greeting received")
    fmt.Println(greeting)
}

func greet(ch chan string) {
    fmt.Println("Greeter waiting to send greeting!")

    ch <- "Hello Rwitesh"

    fmt.Println("Greeter completed")
}

$ go run main.go 
Greeter waiting to send greeting!
Greeter completed
Greeting received
Hello Rwitesh

In the above code snippet, msg := make(chan string) is declaring a channel of type string. Then I passed the channel in goroutine greet. ch <-"Hello Rwitesh" allows us to write the message to ch.

The ch <-"Hello Rwitesh" blocks the execution of the goroutine, as no one reads its value written in a channel. So in the main goroutine time.Sleep(2 * time.Second) terminates the execution without waiting for greet.

The second time.Sleep(2* time.Second) statement gives us the time to read from the channel. We read from the channel using <-msg.

Closing the channel: Closing the channel indicates that no more values should be sent on it. We want to show that the work has been completed and there is no need to keep a channel open.

package main

import (
    "fmt"
    "time"
)

func main() {
    msg := make(chan string)
    go greet(msg)

    time.Sleep(2 * time.Second)

    greeting := <-msg

    time.Sleep(2 * time.Second)
    fmt.Println("Greeting received")
    fmt.Println(greeting)

    _, ok := <-msg
    if ok {
        fmt.Println("Channel is open!")
    } else {
        fmt.Println("Channel is closed!")
    }
}

func greet(ch chan string) {
    fmt.Println("Greeter waiting to send greeting!")

    ch <- "Hello Rwitesh"
    close(ch)

    fmt.Println("Greeter completed")
}

We close a channel by using close() like close(ch) on the above code snippet.

$ go run main.go 
Greeter waiting to send greeting!
Greeter completed
Greeting received
Hello Rwitesh
Channel is closed!

Conclusion

Let's recap what we've learned: concurrency in Go refers to the ability to perform multiple tasks simultaneously, using goroutines and tools like WaitGroups and channels to synchronize and communicate between them.

Goroutines are lightweight threads of execution used in Go to support concurrency. WaitGroups are used to wait for multiple goroutines to finish. They block the execution of a function until their internal counter becomes 0.

Channels are a way for goroutines to communicate and can be used to send and receive data between goroutines.

I hope you found this tutorial helpful and informative. If you enjoyed reading it, I encourage you to share it with your friends and followers on social media.

Don't forget to also follow me on Twitter for more updates on coding and tech. Thanks for reading!

Concurrency in iOS is a massive topic. So in this article I want to zoom in on a sub-topic concerning queues and the Grand Central Dispatch (GCD) framework.

In particular, I wish to explore the differences between serial and concurrent queues, as well as the differences between synchronous and asynchronous execution.

If you've never used GCD before, this article is a great place to start. If you have some experience with GCD, but are still curious about the topics mentioned above, I think you will still find it useful. And I hope you will pick up one or two new things along the way.

I created a SwiftUI companion app to visually demonstrate the concepts in this article. The app also has a fun short quiz that I encourage you to try before and after reading this article. Download the source code here, or get the public beta here.

I will begin with an introduction to GCD, followed by a detailed explanation on sync, async, serial and concurrent. Afterwards, I will cover some pitfalls when working with concurrency. Finally, I will end with a summary and some general advice.

Introduction

Let’s start with a brief intro to GCD and dispatch queues. Feel free to skip to the Sync vs Async section if you are already familiar with the topic.

Concurrency and Grand Central Dispatch

Concurrency lets you take advantage of the fact that your device has multiple CPU cores. To make use of these cores, you will need to use multiple threads. However, threads are a low-level tool, and managing threads manually in an efficient manner is extremely difficult.

Grand Central Dispatch was created by Apple over 10 years ago as an abstraction to help developers write multi-threaded code without manually creating and managing the threads themselves.

With GCD, Apple took an asynchronous design approach to the problem. Instead of creating threads directly, you use GCD to schedule work tasks, and the system will perform these tasks for you by making the best use of its resources. GCD will handle creating the requisite threads and will schedule your tasks on those threads, shifting the burden of thread management from the developer to the system.

A big advantage of GCD is that you don’t have to worry about hardware resources as you write your concurrent code. GCD manages a thread pool for you, and it will scale from a single-core Apple Watch all the way up to a many-core MacBook Pro.

Dispatch Queues

These are the main building blocks of GCD which let you execute arbitrary blocks of code using a set of parameters that you define. The tasks in dispatch queues are always started in a first-in, first-out (FIFO) fashion. Note that I said started, because the completion time of your tasks depends on several factors, and is not guaranteed to be FIFO (more on that later.)

Broadly speaking, there are three kinds of queues available to you:

• The Main dispatch queue (serial, pre-defined)
• Global queues (concurrent, pre-defined)
• Private queues (can be serial or concurrent, you create them)

Every app comes with a Main queue, which is a serial queue that executes tasks on the main thread. This queue is responsible for drawing your application’s UI and responding to user interactions (touch, scroll, pan, etc.) If you block this queue for too long, your iOS app will appear to freeze, and your macOS app will display the infamous beach ball/spinning wheel.

When performing a long-running task (network call, computationally intensive work, etc), we avoid freezing the UI by performing this work on a background queue. Then we update the UI with the results on the main queue:

URLSession.shared.dataTask(with: url) { data, response, error in
    if let data = data {
        DispatchQueue.main.async { // UI work
            self.label.text = String(data: data, encoding: .utf8)
        }
    }
}

As a rule of thumb, all UI work must be executed on the Main queue. You can turn on the Main Thread Checker option in Xcode to receive warnings whenever UI work gets executed on a background thread.

In addition to the main queue, every app comes with several pre-defined concurrent queues that have varying levels of Quality of Service (an abstract notion of priority in GCD.)

For example, here’s the code to submit work asynchronously to the user interactive (highest priority) QoS queue:

DispatchQueue.global(qos: .userInteractive).async {
    print("We're on a global concurrent queue!") 
}

Alternatively, you can call the default priority global queue by not specifying a QoS like this:

DispatchQueue.global().async {
    print("Generic global queue")
}

Additionally, you can create your own private queues using the following syntax:

let serial = DispatchQueue(label: "com.besher.serial-queue")
serial.async {
    print("Private serial queue")
}

When creating private queues, it helps to use a descriptive label (such as reverse DNS notation), as this will aid you while debugging in Xcode’s navigator, lldb, and Instruments:

By default, private queues are serial (I’ll explain what this means shortly, promise!) If you want to create a private concurrent queue, you can do so via the optional attributes parameter:

let concurrent = DispatchQueue(label: "com.besher.serial-queue", attributes: .concurrent)
concurrent.sync {
    print("Private concurrent queue")
}

There is an optional QoS parameter as well. The private queues that you create will ultimately land in one of the global concurrent queues based on their given parameters.

What’s in a task?

I mentioned dispatching tasks to queues. Tasks can refer to any block of code that you submit to a queue using the sync or async functions. They can be submitted in the form of an anonymous closure:

DispatchQueue.global().async {
    print("Anonymous closure")
}

Or inside a dispatch work item that gets performed later:

let item = DispatchWorkItem(qos: .utility) {
    print("Work item to be executed later")
}

Regardless of whether you dispatch synchronously or asynchronously, and whether you choose a serial or concurrent queue, all of the code inside a single task will execute line by line. Concurrency is only relevant when evaluating multiple tasks.

For example, if you have 3 loops inside the same task, these loops will always execute in order:

DispatchQueue.global().async {
    for i in 0..<10 {
        print(i)
    }

    for _ in 0..<10 {
        print("?")
    }

    for _ in 0..<10 {
        print("?")
    }
}

This code always prints out ten digits from 0 to 9, followed by ten blue circles, followed by ten broken hearts, regardless of how you dispatch that closure.

Individual tasks can also have their own QoS level as well (by default they use their queue’s priority.) This distinction between queue QoS and task QoS leads to some interesting behaviour that we will discuss in the priority inversion section.

By now you might be wondering what serial and concurrent are all about. You might also be wondering about the differences between sync and async when submitting your tasks. This brings us to the crux of this article, so let’s dive in!

Sync vs Async

When you dispatch a task to a queue, you can choose to do so synchronously or asynchronously using the sync and async dispatch functions. Sync and async primarily affect the source of the submitted task, that is the queue where it is being submitted from.

When your code reaches a sync statement, it will block the current queue until that task completes. Once the task returns/completes, control is returned to the caller, and the code that follows the sync task will continue.

Think of sync as synonymous with ‘blocking’.

An async statement, on the other hand, will execute asynchronously with respect to the current queue, and immediately returns control back to the caller without waiting for the contents of the async closure to execute. There is no guarantee as to when exactly the code inside that async closure will execute.

Current queue?

It may not be obvious what the source, or current, queue is, because it’s not always explicitly defined in the code.

For example, if you call your sync statement inside viewDidLoad, your current queue will be the Main dispatch queue. If you call the same function inside a URLSession completion handler, your current queue will be a background queue.

Going back to sync vs async, let’s take this example:

DispatchQueue.global().sync {
    print("Inside")
}
print("Outside")
// Console output:
// Inside
// Outside

The above code will block the current queue, enter the closure and execute its code on the global queue by printing “Inside”, before proceeding to print “Outside”. This order is guaranteed.

Let’s see what happens if we try async instead:

DispatchQueue.global().async {
    print("Inside")
}
print("Outside")
// Potential console output (based on QoS): 
// Outside
// Inside

Our code now submits the closure to the global queue, then immediately proceeds to run the next line. It will likely print “Outside” before “Inside”, but this order isn’t guaranteed. It depends on the QoS of the source and destination queues, as well as other factors that the system controls.

Threads are an implementation detail in GCD — we do not have direct control over them and can only deal with them using queue abstractions. Nevertheless, I think it can be useful to ‘peek under the covers’ at thread behaviour to understand some challenges we might encounter with GCD.

For instance, when you submit a task using sync, GCD optimizes performance by executing that task on the current thread (the caller.)

There is one exception however, which is when you submit a sync task to the main queue —  doing so will always run the task on the main thread and not the caller. This behaviour can have some ramifications that we will explore in the priority inversion section.

[From Dispatcher on Github](https://github.com/almaleh/Dispatcher" rel="noopener nofollow)

Which one to use?

When submitting work to a queue, Apple recommends using asynchronous execution over synchronous execution. However, there are situations where sync might be the better choice, such as when dealing with race conditions, or when performing a very small task. I will cover these situations shortly.

One large consequence of performing work asynchronously inside a function is that the function can no longer directly return its values (if they depend on the async work that’s being done). It must instead use a closure/completion handler parameter to deliver the results.

To demonstrate this concept, let’s take a small function that accepts image data, performs some expensive computation to process the image, then returns the result:

func processImage(data: Data) -> UIImage? {
    guard let image = UIImage(data: data) else { return nil }
    // calling an expensive function
    let processedImage = upscaleAndFilter(image: image)
    return processedImage 
}

In this example, the function upscaleAndFilter(image:) might take several seconds, so we want to offload it into a separate queue to avoid freezing the UI. Let’s create a dedicated queue for image processing, and then dispatch the expensive function asynchronously:

let imageProcessingQueue = DispatchQueue(label: "com.besher.image-processing")

func processImageAsync(data: Data) -> UIImage? {
    guard let image = UIImage(data: data) else { return nil }

    imageProcessingQueue.async {
        let processedImage = upscaleAndFilter(image: image)
        return processedImage
    }
}

There are two issues with this code. First, the return statement is inside the async closure, so it is no longer returning a value to the processImageAsync(data:) function, and currently serves no purpose.

But the bigger issue is that our processImageAsync(data:) function is no longer returning any value, because the function reaches the end of its body before it enters the async closure.

To fix this error, we will adjust the function so that it no longer directly returns a value. Instead, it will have a new completion handler parameter that we can call once our asynchronous function has completed its work:

let imageProcessingQueue = DispatchQueue(label: "com.besher.image-processing")

func processImageAsync(data: Data, completion: @escaping (UIImage?) -> Void) {
    guard let image = UIImage(data: data) else {
        completion(nil)
        return
    }

    imageProcessingQueue.async {
        let processedImage =  self.upscaleAndFilter(image: image)
        completion(processedImage)
    }
}

As evident in this example, the change to make the function asynchronous has propagated to its caller, who now has to pass in a closure and handle the results asynchronously as well. By introducing an asynchronous task, you can potentially end up modifying a chain of several functions.

Concurrency and asynchronous execution add complexity to your project as we just observed. This indirection also makes debugging more difficult. That’s why it really pays off to think about concurrency early in your design — it’s not something you want to tack on at the end of your design cycle.

Synchronous execution, by contrast, does not increase complexity. Rather, it allows you to continue using return statements as you did before. A function containing a sync task will not return until the code inside that task has completed. Therefore it does not require a completion handler.

If you are submitting a tiny task (for example, updating a value), consider doing it synchronously. Not only does that help you keep your code simple, it will also perform better — Async is believed to incur an overhead that outweighs the benefit of doing the work asynchronously for tiny tasks that take under 1ms to complete.

If you are submitting a large task, however, like the image processing we performed above, then consider doing it asynchronously to avoid blocking the caller for too long.

Dispatching on the same queue

While it is safe to dispatch a task asynchronously from a queue into itself (for example, you can use .asyncAfter on the current queue), you can not dispatch a task synchronously from a queue into the same queue. Doing so will result in a deadlock that immediately crashes the app!

This issue can manifest itself when performing a chain of synchronous calls that lead back to the original queue. That is, you sync a task onto another queue, and when the task completes, it syncs the results back into the original queue, leading to a deadlock. Use async to avoid such crashes.

Blocking the main queue

Dispatching tasks synchronously from the main queue will block that queue, thereby freezing the UI, until the task is completed. Thus it’s better to avoid dispatching work synchronously from the main queue unless you’re performing really light work.

[prefer to use async from the main queue](https://github.com/almaleh/Dispatcher" rel="noopener nofollow)

Serial vs Concurrent

Serial and concurrent affect the destination —  the queue in which your work has been submitted to run. This is in contrast to sync and async, which affected the source.

A serial queue will not execute its work on more than one thread at a time, regardless of how many tasks you dispatch on that queue. Consequently, the tasks are guaranteed to not only start, but also terminate, in first-in, first-out order.

Moreover, when you block a serial queue (using a sync call, semaphore, or some other tool), all work on that queue will halt until the block is over.

[From Dispatcher on Github](https://github.com/almaleh/Dispatcher" rel="noopener nofollow)

A concurrent queue can spawn multiple threads, and the system decides how many threads are created. Tasks always start in FIFO order, but the queue does not wait for tasks to finish before starting the next task, therefore tasks on concurrent queues can finish in any order.

When you perform a blocking command on a concurrent queue, it will not block the other threads on this queue. Additionally, when a concurrent queue gets blocked, it runs the risk of thread explosion. I will cover this in more detail later on.

[From Dispatcher on Github](https://github.com/almaleh/Dispatcher" rel="noopener nofollow)

The main queue in your app is serial. All the global pre-defined queues are concurrent. Any private dispatch queue you create is serial by default, but can be set to be concurrent using an optional attribute as discussed earlier.

It’s important to note here that the concept of serial vs concurrent is only relevant when discussing a specific queue. All queues are concurrent relative to each other.

That is, if you dispatch work asynchronously from the main queue to a private serial queue, that work will be completed concurrently with respect to the main queue. And if you create two different serial queues, and then perform blocking work on one of them, the other queue is unaffected.

To demonstrate the concurrency of multiple serial queues, let’s take this example:

let serial1 = DispatchQueue(label: "com.besher.serial1")
let serial2 = DispatchQueue(label: "com.besher.serial2")

serial1.async {
    for _ in 0..<5 { print("?") }
}

serial2.async {
    for _ in 0..<5 { print("?") }
}

Both queues here are serial, but the results are jumbled up because they execute concurrently in relation to each other. The fact that they’re each serial (or concurrent) has no effect on this result. Their QoS level determines who will generally finish first (order not guaranteed).

If we want to ensure that the first loop finishes first before starting the second loop, we can submit the first task synchronously from the caller:

let serial1 = DispatchQueue(label: "com.besher.serial1")
let serial2 = DispatchQueue(label: "com.besher.serial2")

serial1.sync { // <---- we changed this to 'sync'
    for _ in 0..<5 { print("?") }
}
// we don't get here until first loop terminates
serial2.async {
    for _ in 0..<5 { print("?") }
}

This is not necessarily desirable, because we are now blocking the caller while the first loop is executing.

To avoid blocking the caller, we can submit both tasks asynchronously, but to the same serial queue:

let serial = DispatchQueue(label: "com.besher.serial")

serial.async {
    for _ in 0..<5 { print("?") }
}

serial.async {
    for _ in 0..<5 { print("?") }
}

Now our tasks execute concurrently with respect to the caller, while also keeping their order intact.

Note that if we make our single queue concurrent via the optional parameter, we go back to the jumbled results, as expected:

let concurrent = DispatchQueue(label: "com.besher.concurrent", attributes: .concurrent)

concurrent.async {
    for _ in 0..<5 { print("?") }
}

concurrent.async {
    for _ in 0..<5 { print("?") }
}

Sometimes you might confuse synchronous execution with serial execution (at least I did), but they are very different things. For example, try changing the first dispatch on line 3 from our previous example to a sync call:

let concurrent = DispatchQueue(label: "com.besher.concurrent", attributes: .concurrent)

concurrent.sync {
    for _ in 0..<5 { print("?") }
}

concurrent.async {
    for _ in 0..<5 { print("?") }
}

Suddenly, our results are back in perfect order. But this is a concurrent queue, so how could that happen? Did the sync statement somehow turn it into a serial queue?

The answer is no!

This is a bit sneaky. What happened is that we did not reach the async call until the first task had completed its execution. The queue is still very much concurrent, but inside this zoomed-in section of the code. it appears as if it were serial. This is because we are blocking the caller, and not proceeding to the next task, until the first one is finished.

If another queue somewhere else in your app tried submitting work to this same queue while it was still executing the sync statement, that work will run concurrently with whatever we got running here, because it’s still a concurrent queue.

Which one to use?

Serial queues take advantage of CPU optimizations and caching, and help reduce context switching.

Apple recommends starting with one serial queue per subsystem in your app —  for example one for networking, one for file compression, etc. If the need arises, you can later expand to a hierarchy of queues per subsystem using the setTarget method or the optional target parameter when building queues.

If you run into a performance bottleneck, measure your app’s performance then see if a concurrent queue helps. If you do not see a measurable benefit, it’s better to stick to serial queues.

Pitfalls

Priority Inversion and Quality of Service

Priority inversion is when a high priority task is prevented from running by a lower priority task, effectively inverting their relative priorities.

This situation often occurs when a high QoS queue shares a resources with a low QoS queue, and the low QoS queue gets a lock on that resource.

But I wish to cover a different scenario that is more relevant to our discussion —  it’s when you submit tasks to a low QoS serial queue, then submit a high QoS task to that same queue. This scenario also results in priority inversion, because the high QoS task has to wait on the lower QoS tasks to finish.

GCD resolves priority inversion by temporarily raising the QoS of the queue that contains the low priority tasks that are ‘ahead’ of, or blocking, your high priority task.

It’s kind of like having cars stuck in front of an ambulance. Suddenly they’re allowed to cross the red light just so that the ambulance can move (in reality the cars move to the side, but imagine a narrow (serial) street or something, you get the point :-P)

To illustrate the inversion problem, let’s start with this code:

enum Color: String {
    case blue = "?"
    case white = "⚪️"
}

func output(color: Color, times: Int) {
    for _ in 1...times {
        print(color.rawValue)
    }
}

let starterQueue = DispatchQueue(label: "com.besher.starter", qos: .userInteractive)
let utilityQueue = DispatchQueue(label: "com.besher.utility", qos: .utility)
let backgroundQueue = DispatchQueue(label: "com.besher.background", qos: .background)
let count = 10

starterQueue.async {

    backgroundQueue.async {
        output(color: .white, times: count)
    }

    backgroundQueue.async {
        output(color: .white, times: count)
    }

    utilityQueue.async {
        output(color: .blue, times: count)
    }

    utilityQueue.async {
        output(color: .blue, times: count)
    }

    // next statement goes here
}

We create a starter queue (where we submit the tasks from), as well as two queues with different QoS. Then we dispatch tasks to each of these two queues, each task printing out an equal number of circles of a specific colour (utility queue is blue, background is white.)

Because these tasks are submitted asynchronously, every time you run the app, you’re going to see slightly different results. However, as you would expect, the queue with the lower QoS (background) almost always finishes last. In fact, the last 10–15 circles are usually all white.

no surprises here

But watch what happens when we submit a sync task to the background queue after the last async statement. You don’t even need to print anything inside the sync statement, just adding this line is enough:

// add this after the last async statement, 
// still inside starterQueue.async
backgroundQueue.sync {}

priority inversion

The results in the console have flipped! Now, the higher priority queue (utility) always finishes last, and the last 10–15 circles are blue.

To understand why that happens, we need to revisit the fact that synchronous work is executed on the caller thread (unless you’re submitting to the main queue.)

In our example above, the caller (starterQueue) has the top QoS (userInteractive.) Therefore, that seemingly innocuous sync task is not only blocking the starter queue, but it’s also running on the starter’s high QoS thread. The task therefore runs with high QoS, but there are two other tasks ahead of it on the same background queue that have background QoS. Priority inversion detected!

As expected, GCD resolves this inversion by raising the QoS of the entire queue to temporarily match the high QoS task. Consequently, all the tasks on the background queue end up running at user interactive QoS, which is higher than the utility QoS. And that’s why the utility tasks finish last!

Side-note: If you remove the starter queue from that example and submit from the main queue instead, you will get similar results, as the main queue also has user interactive QoS.

To avoid priority inversion in this example, we need to avoid blocking the starter queue with the sync statement. Using async would solve that problem.

Although it’s not always ideal, you can minimize priority inversions by sticking to the default QoS when creating private queues or dispatching to the global concurrent queue.

Thread explosion

When you use a concurrent queue, you run the risk of thread explosion if you’re not careful. This can happen when you try to submit tasks to a concurrent queue that is currently blocked (for example with a semaphore, sync, or some other way.) Your tasks will run, but the system will likely end up spinning up new threads to accommodate these new tasks, and threads aren’t cheap.

This is likely why Apple suggests starting with a serial queue per subsystem in your app, as each serial queue can only use one thread. Remember that serial queues are concurrent in relation to other queues, so you still get a performance benefit when you offload your work to a queue, even if it isn’t concurrent.

Race conditions

Swift Arrays, Dictionaries, Structs, and other value types are not thread-safe by default. For example, when you have multiple threads trying to access and modify the same array, you will start running into trouble.

There are different solutions to the readers-writers problem, such as using locks or semaphores. But the relevant solution I wish to discuss here is the use of an isolation queue.

Let’s say we have an array of integers, and we want to submit asynchronous work that references this array. As long as our work only reads the array and does not modify it, we are safe. But as soon as we try to modify the array in one of our asynchronous tasks, we will introduce instability in our app.

It’s a tricky problem because your app can run 10 times without issues, and then it crashes on the 11th time. One very handy tool for this situation is the Thread Sanitizer in Xcode. Enabling this option will help you identify potential race conditions in your app.

this option is only available on the simulator**

To demonstrate the problem, let’s take this (admittedly contrived) example:

class ViewController: UIViewController {

    let concurrent = DispatchQueue(label: "com.besher.concurrent", attributes: .concurrent)
    var array = [1,2,3,4,5]

    override func viewDidLoad() {
        for _ in 0...1 {
            race()
        }
    }

    func race() {

        concurrent.async {
            for i in self.array { // read access
                print(i)
            }
        }

        concurrent.async {
            for i in 0..<10 {
                self.array.append(i) // write access
            }
        }
    }
}

One of the async tasks is modifying the array by appending values. If you try running this on your simulator, you might not crash. But run it enough times (or increase the loop frequency on line 7), and you will eventually crash. If you enable the thread sanitizer, you will get a warning every time you run the app.

To deal with this race condition, we are going to add an isolation queue that uses the barrier flag. This flag allows any outstanding tasks on the queue to finish, but blocks any further tasks from executing until the barrier task is completed.

Think of the barrier like a janitor cleaning a public restroom (shared resource.) There are multiple (concurrent) stalls inside the restroom that people can use.

Upon arrival, the janitor places a cleaning sign (barrier) blocking any newcomers from entering until the cleaning is done, but the janitor does not start cleaning until all the people inside have finished their business. Once they all leave, the janitor proceeds to clean the public restroom in isolation.

When finally done, the janitor removes the sign (barrier) so that the people who are queued up outside can finally enter.

Here’s what that looks like in code:

class ViewController: UIViewController {
    let concurrent = DispatchQueue(label: "com.besher.concurrent", attributes: .concurrent)
    let isolation = DispatchQueue(label: "com.besher.isolation", attributes: .concurrent)
    private var _array = [1,2,3,4,5]

    var threadSafeArray: [Int] {
        get {
            return isolation.sync {
                _array
            }
        }
        set {
            isolation.async(flags: .barrier) {
                self._array = newValue
            }
        }
    }

    override func viewDidLoad() {
        for _ in 0...15 {
            race()
        }
    }

    func race() {
        concurrent.async {
            for i in self.threadSafeArray {
                print(i)
            }
        }

        concurrent.async {
            for i in 0..<10 {
                self.threadSafeArray.append(i)
            }
        }
    }
}

We have added a new isolation queue, and restricted access to the private array using a getter and setter that will place a barrier when modifying the array.

The getter needs to be sync in order to directly return a value. The setter can be async, as we don’t need to block the caller while the write is taking place.

We could have used a serial queue without a barrier to solve the race condition, but then we would lose the advantage of having concurrent read access to the array. Perhaps that makes sense in your case, you get to decide.

Conclusion

Thank you so much for reading this far! I hope you learned something new from this article. I will leave you with a summary and some general advice:

Summary

• Queues always start their tasks in FIFO order
• Queues are always concurrent relative to other queues
• Sync vs Async concerns the source
• Serial vs Concurrent concerns the destination
• Sync is synonymous with ‘blocking’
• Async immediately returns control to caller
• Serial uses a single thread, and guarantees order of execution
• Concurrent uses multiple-threads, and risks thread explosion
• Think about concurrency early in your design cycle
• Synchronous code is easier to reason about and debug
• Avoid relying on global concurrent queues if possible
• Consider starting with a serial queue per subsystem
• Switch to concurrent queue only if you see a measurable performance benefit

I like the metaphor from the Swift Concurrency Manifesto of having an ‘island of serialization in a sea of concurrency’. This sentiment was also shared in this tweet by Matt Diephouse:

When you apply concurrency with that philosophy in mind, I think it will help you achieve concurrent code that can be reasoned about without getting lost in a mess of callbacks.

If you have any questions or comments, feel free to reach out to me on Twitter

Besher Al Maleh

_Cover photo by Onur K on Unsplash_

Download the companion app here:

Check out some of my other articles:

Further reading:

http://khanlou.com/2016/04/the-GCD-handbook/

WWDC Videos:

The call stack

First let’s learn about what a call stack is.

A call stack is a simple data structure that records where in the code we are currently. So if we step into a function that is a function invocation it is pushed to the call stack. When we return from a function it is popped out of the stack.

Let’s look at a code example to understand the call stack:

function multiply(x,y) {
   return x * y;
}

function squared(n) {
     return multiply(n,n)
  }

function printSquare(n) {
   return squared(n)
}

let numberSquared = printSquare(5);
console.log(numberSquared);

First when the code executes the runtime will read through each of the function definitions. But when it reaches the line where the first function printSquare(5) is invoked it will push this function into the call stack.

Next, this function will execute. Before returning it will encounter another function, squared(n), so it will suspend its current operation and push this function on top of the existing function.

It executes the function (in this case the squared function) and finally it encounters another function multiply(n,n). Then it suspends its current executions and pushes this function into the call stack. The function multiply executes and it returns with the multiplied value.

Finally the squared function returns and is popped off the stack and then the same goes with printSquare. The final squared value is allocated to the numberSquared variable.

We encounter again a function invocation (in this case it’s a console.log() statement) so the runtime pushes this to the stack. This executes it thus printing the squared number on the console.

Note that the first function that gets pushed into the stack before any of the above code runs is the main function. In the runtime this is denoted as an ‘anonymous function’.

So to summarize: whenever a function is invoked it is pushed into the call stack where it executes. Finally when the function is done with its execution and is returning either implicitly or explicitly it will be popped off the stack.

The call stack just records at what point in time which function was executing. And it keeps track of which function is currently executing.

The browser

Now we know from this that Javascript can execute one thing at a time but that’s not the case with the Browser. The Browser has its own set of APIs like setTimeout and XMLHttpRequests which are not specified in the Javascript runtime.

In fact if you look through the source code of V8, the popular Javascript runtime that powers browsers like Google Chrome, you won’t find any definitions for it. That's because these special web API’s exist in the browser environment not inside the javascript environment. So you can say that these APIs introduce concurrency into the mix.

Let’s look at a diagram to understand the whole picture.

Some more terms are introduced here, so let's go through them:

Heap: It’s mostly the place where objects are allocated.

Callback Queue: It’s a data structure that stores all the callbacks. Since it’s a queue, the elements are processed based on FIFO which is First in First Out.

Event Loop: This is where all these things come together. The event loop simply checks the call stack, and if it is empty (which means there are no functions in the stack) it takes the oldest callback from the callback queue and pushes it into the call stack which eventually executes the callback.

Let’s understand this with a code example:

console.log('hi');

setTimeout(function() {
     console.log('freecodeCamp')
},5000);

console.log('JS')

When the first line executes it’s a console.log(). This is a function invocation which means that this function is pushed into the call stack where it executes printing ‘hi’ to the console. Finally it’s returned and is popped off the stack.

Then when the runtime goes to execute setTimeout() it knows that this is a web API. Therefore it gives it off to the browser to handle its execution. The browser starts the timer and then JS runtime pops the setTimeout() out of the stack. It encounters another console.log() invocation and so it pushes this into the call stack, the message ‘JS’ is logged into the console, and then it’s finally returned. Then the last console.log() is popped off the stack. Now the call stack is empty.

Meanwhile while all of this was going on the timer finishes. When 5 seconds have elapsed the browser goes ahead and pushes the callback function into the callback queue.

Next the event loop checks if the call stack is free or not. Since it is free it takes the callback function and pushes it again back to the call stack which executes the code inside it.

Again inside the code there is a console.log() invocation so this function goes to the top of the stack executes which logs ‘freecodecamp’ into the console and finally it returns. This means it gets popped off the stack and finally the callback gets popped off the stack and we are done.

To visualize this better try this tool by Phillip Roberts: Loupe Event Loop Visualizer

There are 7,713,468,100 people in the world in 2019, around 26.3% of which are under 15 years old. This works out to 2,028,642,110 children (persons under 15 years of age) in the world this year.

Santa doesn’t seem to visit children of every religion, so we’ll generalize and only include Christians and non-religious folks. Collectively that makes up approximately 44.72% of the population. If we assume that all kids take after their parents, then 907,208,751.6 children would appear to be Santa-eligible.

What percentage of those children are good? It’s impossible to know; however, we can work on a few assumptions. One is that Santa Claus functions more on optimism than economics and would likely have prepared for the possibility that every child is a good child in any given year. Thus, he would be prepared to give a toy to every child. Let’s assume it’s been a great year and that all 907,208,751.6 children are getting toys.

That’s a lot of presents, and, as we know, they’re all made by Santa’s elves at his North China Pole workshop. Given that there are 365 days in a year and one of them is Christmas, let’s assume that Santa’s elves collectively have 364 days to create and gift wrap 907,208,752 (rounded up) presents. That works out to 2,492,331.74 presents per day.

Almost two-and-a-half million presents per day is a heavy workload for any workshop. Let’s look at two paradigms that Santa might employ to hit this goal: concurrency, and parallelism.

A sequential process

Suppose that Santa’s workshop is staffed by exactly one, very hard working, very tired elf. The production of one present involves four steps:

1. Cutting wood
2. Assembly and gluing
3. Painting
4. Gift-wrapping

With a single elf, only one step for one present can be happening at any instance in time. If the elf were to produce one present at a time from beginning to end, that process would be executed sequentially. It’s not the most efficient method for producing two-and-a-half million presents per day; for instance, the elf would have to wait around doing nothing while the glue on the present was drying before moving on to the next step.

Concurrency

In order to be more efficient, the elf works on all presents concurrently.

Instead of completing one present at a time, the elf first cuts all the wood for all the toys, one by one. When everything is cut, the elf assembles and glues the toys together, one after the other. This concurrent processing means that the glue from the first toy has time to dry (without needing more attention from the elf) while the remaining toys are glued together. The same goes for painting, one toy at a time, and finally wrapping.

Since one elf can only do one task at a time, a single elf is using the day as efficiently as possible by concurrently producing presents.

Parallelism

Hopefully, Santa’s workshop has more than just one elf. With more elves, more toys can be built simultaneously over the course of a day. This simultaneous work means that the presents are being produced in parallel. Parallel processing carried out by multiple elves means more work happens at the same time.

Elves working in parallel can also employ concurrency. One elf can still tackle only one task at a time, so it’s most efficient to have multiple elves concurrently producing presents.

Of course, if Santa’s workshop has, say, two-and-a-half million elves, each elf would only need to finish a maximum of one present per day. In this case, working sequentially doesn’t detract from the workshop’s efficiency. There would still be 7,668.26 elves left over to fetch coffee and lunch.

Santa Claus, and threading

After all the elves’ hard work is done, it’s up to Santa Claus to deliver the presents – all 907,208,752 of them.

Santa doesn’t need to make a visit to every kid; just to the one household tree. So how many trees does Santa need to visit? Again with broad generalization, we’ll say that the average number of children per household worldwide is 2.45, based on the year’s predicted fertility rates. That makes 370,289,286.4 houses to visit. Let’s round that up to 370,289,287.

How long does Santa have? The lore says one night, which means one earthly rotation, and thus 24 hours. NORAD confirms.

This means Santa must visit 370,289,287 households in 24 hours (86,400 seconds), at a rate of 4,285.75 households per second, never mind the time it takes to put presents under the tree and grab a cookie.

Clearly, Santa doesn’t exist in our dimension. This is especially likely given that, despite being chubby and plump, he fits down a chimney (with a lit fire, while remaining unhurt) carrying a sack of toys containing presents for all the household’s children. We haven’t even considered the fact that his sleigh carries enough toys for every believing boy and girl around the world, and flies.

Does Santa exist outside our rules of physics? How could one entity manage to travel around the world, delivering packages, in under 24 hours at a rate of 4,285.75 households per second, and still have time for milk and cookies and kissing mommy?

One thing is certain: Santa uses the Internet. No other technology has yet enabled packages to travel quite so far and quite so quickly. Even so, attempting to reach upwards of four thousand households per second is no small task, even with the best gigabit internet hookup the North Pole has to offer. How might Santa increase his efficiency?

There’s clearly only one logical conclusion to this mystery: Santa Claus is a multithreaded process.

A single thread

Let’s work outward. Think of a thread as one particular task, or the most granular sequence of instructions that Santa might execute. One thread might execute the task, put present under tree. A thread is a component of a process, in this case, Santa’s process of delivering presents.

If Santa Claus is single-threaded, he, as a process, would only be able to accomplish one task at a time. Since he’s old and a bit forgetful, he probably has a set of instructions for delivering presents, as well as a schedule to abide by. These two things guide Santa’s thread until his process is complete.

Single-threaded Santa Claus might work something like this:

1. Land sleigh at Timmy’s house
2. Get Timmy’s present from sleigh
3. Enter house via chimney
4. Locate Christmas tree
5. Place Timmy’s present under Christmas tree
6. Exit house via chimney
7. Take off in sleigh

Rinse and repeat… another 370,289,286 times.

Multithreading

Multithreaded Santa Claus, by contrast, is the Doctor Manhattan of the North Pole. There’s still only one Santa Claus in the world; however, he has the amazing ability to multiply his consciousness and accomplish multiple instruction sets of tasks simultaneously. These additional task workers, or worker threads, are created and controlled by the main process of Santa delivering presents.

Each worker thread acts independently to complete its instructions. Since they all belong to Santa’s consciousness, they share Santa’s memory and know everything that Santa knows, including what planet they’re running around on, and where to get the presents from.

With this shared knowledge, each thread is able to execute its set of instructions in parallel with the other threads. This multithreaded parallelism makes the one and only Santa Claus as efficient as possible.

If an average present delivery run takes an hour, Santa need only spawn 4,286 worker threads. With each making one delivery trip per hour, Santa will have completed all 370,289,287 trips by the end of the night.

Of course, in theory, Santa could even spawn 370,289,287 worker threads, each flying to one household to deliver presents for all the children in it! That would make Santa’s process extremely efficient, and also explain how he manages to consume all those milk-dunked cookies without getting full. ????

An efficient and merry multithreaded Christmas

Thanks to modern computing, we now finally understand how Santa Claus manages the seemingly-impossible task of delivering toys to good girls and boys the world-over. From my family to yours, I hope you have a wonderful Christmas. Don’t forget to hang up your stockings on the router shelf.

Of course, none of this explains how reindeer manage to fly.

What is a Thread?

A thread is a lightweight process. Any process can have multiple threads running in it.

For example in a web browser, we can have one thread which will load the user interface and another thread which will actually retrieve all the data that needs to be displayed in that interface.

What is MultiThreading?

Multithreading enables us to run multiple threads concurrently.

For example in a web browser, we can have one thread which handles the user interface, and in parallel we can have another thread which fetches the data to be displayed.

So multithreading improves the responsiveness of a system.

What is Concurrency?

Concurrency in the context of threads enables us to run multiple threads at the same time.

But do the threads really run at the same time?

Single Core Systems

The Thread Scheduler provided by the JVM decides which thread runs at any given time. The scheduler gives a small time slice to each thread.

So at any given time we have only one thread which is actually running in the processor. But because of the time slicing we get the feeling that multiple threads are running at the same time.

Multi Core Systems

Even in multiple core systems the thread scheduler is involved. But since we have multiple cores, we can actually have multiple threads running at the exact same time.

For example if we have a dual core system, then we can have 2 threads running at the exact same time. The first thread will run in the first core, and the second thread will run in the second core.

Why is Multithreading needed?

Multithreading enables us to improve the responsiveness of a system.

For example in a web browser, if everything ran in a single thread, then system would be completely unresponsive whenever data was being fetched to display. For example, if it takes 10 seconds to fetch the data, then in that 10 seconds we wont be able to do anything else in the web browser like opening new tabs, or even closing the web browser.

So running different parts of a program in different threads concurrently helps improve the responsiveness of a system.

How to write Multithreaded Programs in Java

We can create threads in Java using the following

• Extending the thread class
• Implementing the runnable interface
• Implementing the callable interface
• By using the executor framework along with runnable and callable tasks

We will look at callables and the executor framework in a separate blog. In this article I will be mainly focussing on extending the thread class and implementing the runnable interface.

Extending the Thread Class

In order to create a piece of code which can be run in a thread, we create a class and then extend the thread class. The task being done by this piece of code needs to be put in the run() function.

In the below code you can see that worker is a class which extends the thread class, and the task of printing numbers 0 to 5 is being done inside the run() function.

class Worker extends Thread {

    @Override
    public void run() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

In the above code Thread.currentThread().getName() is used to get the name of the current thread which is running the code.

In order to create a thread, we just need to create an instance of the worker class. And then we can start the thread using the start() function.

public class ThreadClassDemo {
    public static void main(String[] args) {
        Thread t1 = new Worker();
        Thread t2 = new Worker();
        Thread t3 = new Worker();
        t1.start();
        t2.start();
        t3.start();

    }
}

In the above code, we are creating 3 threads (t1,t2 and t3) from the worker class. Then we are starting the threads using the start() function.

Here is the final code for creating a thread by extending a thread class:

class Worker extends Thread {

    @Override
    public void run() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

public class ThreadClassDemo {
    public static void main(String[] args) {
        Thread t1 = new Worker();
        Thread t2 = new Worker();
        Thread t3 = new Worker();
        t1.start();
        t2.start();
        t3.start();

    }
}

Here is the output we get by running the above code:

You can see that all the 3 threads have printed the numbers from 0 to 5.

You can also clearly see from the output that the 3 threads do not run in any particular sequence

Implementing the Runnable Interface

In order to create a piece of code which can be run in a thread, we create a class and then implement the runnable interface. The task being done by this piece of code needs to be put in the run() function.

In the below code you can see that RunnableWorker is a class which implements runnable interface, and the task of printing numbers 0 to 4 is being done inside the run() function.

class RunnableWorker implements Runnable{

    @Override
    public void run() {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

In order to create a thread, first we need to create an Instance of RunnableWorker which implements the runnable interface.

Then we can create a new thread by creating an instance of the thread class and passing the instance of RunnableWorker as the argument. This is shown in the code below:

public class RunnableInterfaceDemo {

    public static void main(String[] args) {
        Runnable r = new RunnableWorker();
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();

    }

}

The above code creates a runnable instance r. Then it create 3 threads (t1, t2 and t3) and passes r as the argument to the 3 threads. Then the start() function is used to start all 3 threads.

Here is the complete code for creating a thread by implementing the runnable interface:

class RunnableWorker implements Runnable{

    @Override
    public void run() {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

public class RunnableInterfaceDemo {

    public static void main(String[] args) {
        Runnable r = new RunnableWorker();
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();

    }

}

On running the above code, we will get the following output. The sequence of the output will change every time the code is run.

Implementing the runnable interface is a better option than extending the thread class since we can extend only one class, but we can implement multiple interfaces in Java.

Runnable Interface in Java 8

In Java 8, the runnable interface becomes a FunctionalInterface since it has only one function, run().

The below code shows how we can create a runnable instance in Java 8.

public class RunnableFunctionalInterfaceDemo {

    public static void main(String[] args) {

        Runnable r = () -> {
            for (int i = 0; i <= 4; i++) {
                System.out.println(Thread.currentThread().getName() + ": " + i);
            }
        };

        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();
    }

}

Here, instead of creating a class and then implementing the runnable interface, we can directly use a lambda expression to create a runnable instance as shown below:

Runnable r = () -> {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    };

Code

The code in this article is available in the following GitHub repo: https://github.com/aditya-sridhar/basic-threads-demo

Congrats ?

You now know how to create threads by extending the thread class and by implementing the runnable interface.

I will discuss the thread life cycle and challenges while using threads in my next blog post.

My Website: https://adityasridhar.com/

Feel free to connect with me on LinkedIn or follow me on Twitter

At the Apache Druid community, we are currently preparing a detailed checklist to be used during code reviews. I decided to publish parts of the checklist as posts on Medium to gather more ideas for checklist items. Hopefully, somebody will find it useful in practice.

By the way, it seems me that creating project-specific checklists for code reviews should be a powerful idea, yet I don’t see any existing examples among large open source projects.

This post contains checklist items about problems that arise with the multithreaded Java code.

Thanks to Marko Topolnik, Matko Medenjak, Chris Vest, Simon Willnauer, Ben Manes, Gleb Smirnov, Andrey Satarin, Benedict Jin, and Petr Janeček for reviews and contributions to this post. The checklist is not considered complete, comments and suggestions are welcome!

Update: this checklist is now available on Github.

1. Design

1.1. If the patch introduces a new subsystem with concurrent code, is the necessity for concurrency rationalized in the patch description? Is there a discussion of alternative design approaches that could simplify the concurrency model of the code (see the next item)?

1.2. Is it possible to apply one or several design patterns (some of them are listed below) to significantly simplify the concurrency model of the code, while not considerably compromising other quality aspects, such as overall simplicity, efficiency, testability, extensibility, etc?

Immutability/Snapshotting. When some state should be updated, a new immutable object (or a snapshot within a mutable object) is created, published and used, while some concurrent threads may still use older copies or snapshots. See [EJ Item 17], [JCIP 3.4], items 4.5 and 9.2 in this checklist, CopyOnWriteArrayList, CopyOnWriteArraySet, persistent data structures.

Divide and conquer. Work is split into several parts that are processed independently, each part in a single thread. Then the results of processing are combined. Parallel Streams (see section 14) or ForkJoinPool (see items 10.4 and 10.5) can be used to apply this pattern.

Producer-consumer. Pieces of work are transmitted between worker threads via queues. See [JCIP 5.3], item 6.1 in this checklist, CSP, SEDA.

Instance confinement. Objects of some root type encapsulate some complex hierarchical child state. Root objects are solitarily responsible for the safety of accesses and modifications to the child state from multiple threads. In other words, composed objects are synchronized rather than synchronized objects are composed. See [JCIP 4.2, 10.1.3, 10.1.4].

Thread/Task/Serial thread confinement. Some state is made local to a thread using top-down pass-through parameters or ThreadLocal. See [JCIP 3.3]. Task confinement is a variation of the idea of thread confinement that is used in conjunction with the divide-and-conquer pattern. It usually comes in the form of lambda-captured “context” parameters or fields in the per-thread task objects. Serial thread confinement is an extension of the idea of thread confinement for the producer-consumer pattern, see [JCIP 5.3.2].

2. Documentation

2.1. For every class, method, and field that has signs of being thread-safe, such as the synchronized keyword, volatile modifiers on fields, use of any classes from java.util.concurrent.*, or third-party concurrency primitives, or concurrent collections: do their Javadoc comments include

• The justification for thread safety: is it explained why a particular class, method or field has to be thread-safe?
• Concurrent control flow documentation: is it enumerated from what methods and in contexts of what threads (executors, thread pools) each specific method of a thread-safe class is called?

Wherever some logic is parallelized or the execution is delegated to another thread, are there comments explaining why it’s worse or inappropriate to execute the logic sequentially or in the same thread? See also item 14.1 in this checklist about parallel Stream use.

2.2. If the patch introduces a new subsystem that uses threads or thread pools, are there high-level descriptions of the threading model, the concurrent control flow (or the data flow) of the subsystem somewhere, e. g. in the Javadoc comment for the package in package-info.java or for the main class of the subsystem? Are these descriptions kept up-to-date when new threads or thread pools are added or some old ones deleted from the system?

Description of the threading model includes the enumeration of threads and thread pools created and managed in the subsystem, and external pools used in the subsystem (such as ForkJoinPool.commonPool()), their sizes and other important characteristics such as thread priorities, and the lifecycle of the managed threads and thread pools.

A high-level description of concurrent control flow should be an overview and tie together concurrent control flow documentation for individual classes, see the previous item. If the producer-consumer pattern is used, the concurrent control flow is trivial and the data flow should be documented instead.

Describing threading models and control/data flow greatly improves the maintainability of the system, because in the absence of descriptions or diagrams developers spend a lot of time and effort to create and refresh these models in their minds. Putting the models down also helps to discover bottlenecks and the ways to simplify the design (see item 1.2).

2.3. For classes and methods that are parts of the public API or the extensions API of the project: is it specified in their Javadoc comments whether they are (or in case of interfaces and abstract classes designed for subclassing in extensions, should they be implemented as) immutable, thread-safe or not thread-safe? For classes and methods that are (or should be implemented as) thread-safe, is it documented precisely with what other methods (or themselves) they may be called concurrently from multiple threads? See also [EJ Item 82] and [JCIP 4.5].

If the @com.google.errorprone.annotations.Immutable annotation is used to mark immutable classes, Error Prone static analysis tool is capable to detect when a class is not actually immutable (see the relevant bug pattern).

2.4. For subsystems, classes, methods, and fields that use some concurrency design patterns, either high-level (such as those mentioned in item 1.2 in this checklist) or low-level (such as double-checked locking, see section 8 in this checklist): are the used concurrency patterns pronounced in the design or implementation comments for the respective subsystems, classes, methods, and fields? This helps readers to make sense out of the code quicker.

2.5. Are ConcurrentHashMap and ConcurrentSkipListMap objects stored in fields and variables of ConcurrentHashMap or ConcurrentSkipListMap or **ConcurrentMap** type, but not just Map?

This is important, because in code like the following:

ConcurrentMap<String, Entity> entities = getEntities();if (!entities.containsKey(key)) {  entities.put(key, entity);} else {  ...}

It should be pretty obvious that there might be a race condition because an entity may be put into the map by a concurrent thread between the calls to containsKey() and put() (see item 4.1 about this type of race conditions). While if the type of the entities variable was just Map<String, Entity> it would be less obvious and readers might think this is only slightly suboptimal code and pass by.

It’s possible to turn this advice into an inspection in IntelliJ IDEA.

2.6. An extension of the previous item: are ConcurrentHashMaps on which compute(), computeIfAbsent(), computeIfPresent(), or merge() methods are called stored in fields and variables of ConcurrentHashMap type rather than ConcurrentMap? This is because ConcurrentHashMap (unlike the generic ConcurrentMap interface) guarantees that the lambdas passed into compute()-like methods are performed atomically per key, and the thread safety of the class may depend on that guarantee.

This advice may seem to be overly pedantic, but if used in conjunction with a static analysis rule that prohibits calling compute()-like methods on ConcurrentMap-typed objects that are not ConcurrentHashMaps (it’s possible to create such inspection in IntelliJ IDEA too) it could prevent some bugs: e. g. calling compute() on a ConcurrentSkipListMap might be a race condition and it’s easy to overlook that for somebody who is used to rely on the strong semantics of compute() in ConcurrentHashMap.

2.7. Is **@GuardedBy** annotation used? If accesses to some fields should be protected by some lock, are those fields annotated with @GuardedBy? Are private methods that are called from within critical sections in other methods annotated with @GuardedBy? If the project doesn’t depend on any library containing this annotation (it’s provided by jcip-annotations, error_prone_annotations, jsr305 and other libraries) and for some reason it’s undesirable to add such dependency, it should be mentioned in Javadoc comments for the respective fields and methods that accesses and calls to them should be protected by some specified locks.

See [JCIP 2.4] for more information about @GuardedBy.

Using GuardedBy is especially beneficial in together with Error Prone, which is able to statically check for unguarded accesses to fields and methods with @GuardedBy annotations.

2.8. If in a thread-safe class some fields are accessed both from within critical sections and outside of critical sections, is it explained in comments why this is safe? For example, unprotected read-only access to a reference to an immutable object might be benignly racy (see item 4.5).

2.9. Regarding every field with a volatile modifier: does it really need to be volatile? Does the Javadoc comment for the field explain why the semantics of volatile field reads and writes (as defined in the Java Memory Model) are required for the field?

2.10. Is it explained in the Javadoc comment for each mutable field in a thread-safe class that is neither volatile nor annotated with @GuardedBy, why that is safe? Perhaps, the field is only accessed and mutated from a single method or a set of methods that are specified to be called only from a single thread sequentially (as described in item 2.1). This recommendation also applies to final fields that store objects of non-thread-safe classes when those objects could be mutated from the methods of the enclosing thread-safe class. See items 4.2–4.4 in this checklist about what could go wrong with such code.

3. Excessive thread safety

3.1. An example of excessive thread safety is a class where every modifiable field is volatile or an AtomicReference or other atomic, and every collection field stores a concurrent collection (e. g. ConcurrentHashMap), although all accesses to those fields are synchronized.

There shouldn’t be any “extra” thread safety in code, there should be just enough of it. Duplication of thread safety confuses readers because they might think the extra thread safety precautions are (or used to be) needed for something but will fail to find the purpose.

The exception from this principle is the volatile modifier on the lazily initialized field in the safe local double-checked locking pattern which is the recommended way to implement double-checked locking, despite that volatile is excessive for correctness when the lazily initialized object has all final fields*. Without that volatile modifier the thread safety of the double-checked locking could easily be broken by a change (addition of a non-final field) in the class of lazily initialized objects, though that class should not be aware of subtle concurrency implications. If the class of lazily initialized objects is specified to be immutable (see item 2.3) the volatile is still unnecessary and the UnsafeLocalDCL pattern could be used safely, but the fact that some class has all final fields doesn’t necessarily mean that it’s immutable.

See also section 8 in this post about double-checked locking.

3.2. Aren’t there **AtomicReference**, AtomicBoolean, AtomicInteger or AtomicLong fields on which only get() and set() methods are called? Simple fields with volatile modifiers can be used instead, but volatile might not be needed too; see item 2.9.

4. Race conditions

4.1. Aren’t ConcurrentHashMaps updated with multiple separate containsKey(), get(), put() and remove() calls instead of a single call to compute()/computeIfAbsent()/computeIfPresent()/replace()?

4.2. Aren’t there point read accesses such as Map.get(), containsKey() or List.get() outside of critical sections to a non-thread-safe collection such as HashMap or ArrayList, while new entries can be added to the collection concurrently, even though there is a happens-before edge between the moment when some entry is put into the collection and the moment when the same entry is point-queried outside of a critical section?

The problem is that when new entries can be added to a collection, it grows and changes its internal structure from time to time (HashMap rehashes the hash table, ArrayList reallocates the internal array). At such moments races might happen and unprotected point read accesses might fail with NullPointerException, ArrayIndexOutOfBoundsException, or return null or some random entry.

Note that this concern applies to ArrayList even when elements are only added to the end of the list. However, a small change in ArrayList’s implementation in OpenJDK could have disallowed data races in such cases at very little cost. If you are subscribed to the concurrency-interest mailing list, you could help to bring attention to this problem by reviving this thread.

4.3. A variation of the previous item: isn’t a non-thread-safe collection such as HashMap or ArrayList iterated outside of a critical section, while it can be modified concurrently? This could happen by accident when an Iterable, Iterator or Stream over a collection is returned from a method of a thread-safe class, even though the iterator or stream is created within a critical section.

Like the previous item, this one applies to growing ArrayLists too.

4.4. More generally, aren’t non-trivial objects that can be mutated concurrently returned from getters on a thread-safe class?

4.5. If there are multiple variables in a thread-safe class that are updated at once but have individual getters, isn’t there a race condition in the code that calls those getters? If there is, the variables should be made final fields in a dedicated POJO, that serves as a snapshot of the updated state. The POJO is stored in a field of the thread-safe class, directly or as an AtomicReference. Multiple getters to individual fields should be replaced with a single getter that returns the POJO. This allows avoiding a race condition in the client code by reading a consistent snapshot of the state at once.

This pattern is also very useful for crafting safe and reasonably simple non-blocking code: see item 9.2 in this checklist and [JCIP 15.3.1].

4.6. If some logic within some critical section depends on some data that principally is part of the internal mutable state of the class, but was read outside of the critical section or in a different critical section, isn’t there a race condition because the local copy of the data may become out of sync with the internal state by the time when the critical section is entered? This is a typical variant of check-then-act race condition, see [JCIP 2.2.1].

4.7. Aren’t there race conditions between the code (i. e. program runtime actions) and some actions in the outside world or actions performed by some other programs running on the machine? For example, if some configurations or credentials are hot reloaded from some file or external registry, reading separate configuration parameters or separate credentials (such as username and password) in separate transactions with the file or the registry may be racing with a system operator updating those configurations or credentials.

Another example is checking that a file exists (or not exists) and then reading, deleting, or creating it, respectively, while another program or a user may delete or create the file between the check and the act. It’s not always possible to cope with such race conditions, but it’s useful to keep such possibilities in mind. Prefer static methods from [java.nio.file.Files](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/file/Files.html) class instead of methods from the old java.io.File for file system operations. Methods from Files are more sensitive to file system race conditions and tend to throw exceptions in adverse cases, while methods on File swallow errors and make it hard even to detect race conditions.

5. Replacing locks with concurrency utilities

5.1. Is it possible to use concurrent collections and/or utilities from java.util.concurrent.* and avoid using locks with Object.wait()/notify()/notifyAll()? Code redesigned around concurrent collections and utilities is often both clearer and less error-prone than code implementing the equivalent logic with intrinsic locks, Object.wait() and notify() (Lock objects with await() and signal() are not different in this regard). See [EJ Item 81] for more information.

5.2. Is it possible to simplify code that uses intrinsic locks or Lock objects with conditional waits by using Guava’s [Monitor](https://google.github.io/guava/releases/27.0.1-jre/api/docs/com/google/common/util/concurrent/Monitor.html) instead?

6. Avoiding deadlocks

6.1. If a thread-safe class is implemented so that there are nested critical sections protected by different locks, is it possible to redesign the code to get rid of nested critical sections? Sometimes a class could be split into several distinct classes, or some work that is done within a single thread could be split between several threads or tasks which communicate via concurrent queues. See [JCIP 5.3] for more information about the producer-consumer pattern.

6.2. If restructuring a thread-safe class to avoid nested critical sections is not reasonable, was it deliberately checked that the locks are acquired in the same order throughout the code of the class? Is the locking order documented in the Javadoc comments for the fields where the lock objects are stored?

6.3. If there are nested critical sections protected by several (potentially different) dynamically determined locks (for example, associated with some business logic entities), are the locks ordered before the acquisition? See [JCIP 10.1.2] for more information.

6.4. Aren’t there calls to some callbacks (listeners, etc.) that can be configured through public API or extension interface calls within critical sections of a class? With such calls, the system might be inherently prone to deadlocks because the external logic executed within a critical section might be unaware of the locking considerations and call back into the logic of the project, where some more locks may be acquired, potentially forming a locking cycle that might lead to deadlock. Let alone the external logic could just perform some time-consuming operation and by that harm the efficiency of the system (see the next item). See [JCIP 10.1.3] and [EJ Item 79] for more information.

7. Improving scalability

7.1. Are critical sections as small as possible? For every critical section: can’t some statements in the beginning and the end of the section be moved out of it? Not only minimizing critical sections improves scalability, but also makes it easier to review them and spot race conditions and deadlocks.

This advice equally applies to lambdas passed into ConcurrentHashMap’s compute()-like methods.

See also [JCIP 11.4.1] and [EJ Item 79].

7.2. Is it possible to increase locking granularity? If a thread-safe class encapsulates accesses to map, is it possible to turn critical sections into lambdas passed into ConcurrentHashMap.compute() or computeIfAbsent() or computeIfPresent() methods to enjoy effective per-key locking granularity? Otherwise, is it possible to use Guava’s Striped or an equivalent? See [JCIP 11.4.3] for more information about lock striping.

7.3. Is it possible to use non-blocking collections instead of blocking ones? Here are some possible replacements within JDK:

• Collections.synchronizedMap(HashMap), Hashtable → ConcurrentHashMap
• Collections.synchronizedSet(HashSet) → ConcurrentHashMap.newKeySet()
• Collections.synchronizedMap(TreeMap) → ConcurrentSkipListMap. By the way, ConcurrentSkipListMap is not the state of the art concurrent sorted dictionary implementation. [SnapTree](https://github.com/nbronson/snaptree) is more efficient than ConcurrentSkipListMap and there have been some research papers presenting algorithms that are claimed to be more efficient than SnapTree.
• Collections.synchronizedSet(TreeSet) → ConcurrentSkipListSet
• Collections.synchronizedList(ArrayList), Vector → CopyOnWriteArrayList
• LinkedBlockingQueue → ConcurrentLinkedQueue
• LinkedBlockingDeque → ConcurrentLinkedDeque

Was it considered to use one of the array-based queues from the JCTools library instead of ArrayBlockingQueue? Those queues from JCTools are classified as blocking, but they avoid lock acquisition in many cases and are generally much faster than ArrayBlockingQueue.

7.4. Is it possible to use [ClassValue](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ClassValue.html) instead of ConcurrentHashMap<Class, ...>? Note, however, that unlike ConcurrentHashMap with its computeIfAbsent() method ClassValue doesn’t guarantee that per-class value is computed only once, i. e. ClassValue.computeValue() might be executed by multiple concurrent threads. So if the computation inside computeValue() is not thread-safe, it should be synchronized separately. On the other hand, ClassValue does guarantee that the same value is always returned from ClassValue.get() (unless remove() is called).

7.5. Was it considered to replace a simple lock with a ReadWriteLock? Beware, however, that it’s more expensive to acquire and release a ReentrantReadWriteLock than a simple intrinsic lock, so the increase in scalability comes at the cost of reduced throughput. If the operations to be performed under a lock are short, or if a lock is already striped (see item 7.2) and therefore very lightly contended, replacing a simple lock with a ReadWriteLock might have a net negative effect on the application performance. See this comment for more details.

7.6. Is it possible to use a [**StampedLock**](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/locks/StampedLock.html) instead of a ReentrantReadWriteLock when reentrancy is not needed?

7.7. Is it possible to use [**LongAdder**](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/atomic/LongAdder.html) for “hot fields” (see [JCIP 11.4.4]) instead of AtomicLong or AtomicInteger on which only methods like incrementAndGet(), decrementAndGet(), addAndGet() and (rarely) get() is called, but not set() and compareAndSet()?

8. Lazy initialization and double-checked locking

8.1. For every lazily initialized field: is the initialization code thread-safe and might it be called from multiple threads concurrently? If the answers are “no” and “yes”, either double-checked locking should be used or the initialization should be eager.

8.2. If a field is initialized lazily under a simple lock, is it possible to use double-checked locking instead to improve performance?

8.3. Does double-checked locking follow the SafeLocalDCL pattern, as noted in item 3.1 in this checklist?

If the initialized objects are immutable a more efficient UnsafeLocalDCL pattern might also be used. However, if the lazily-initialized field is not volatile and there are accesses to the field that bypass the initialization path, the value of the field must be carefully cached in a local variable. For example, the following code is buggy:

private MyClass lazilyInitializedField;

void foo() {  if (lazilyInitializedField != null) { // (1)    // Can throw NPE!    lazilyInitializedField.bar();     // (2)  }}

It might result in a NullPointerException, because although a non-null value is observed when the field is read the first time at line 1, the second read at line 2 could observe null.

The above code could be fixed as follows:

void foo() {  MyClass lazilyInitialized = this.lazilyInitializedField;  if (lazilyInitialized != null) {    lazilyInitialized.bar();  }}

See “Wishful Thinking: Happens-Before Is The Actual Ordering” for more information.

8.4. In each particular case, doesn’t the net impact of double-checked locking and lazy field initialization on performance and complexity overweight the benefits of lazy initialization? Isn’t it ultimately better to initialize the field eagerly?

8.5. If a field is initialized lazily under a simple lock or using double-checked locking, does it really need locking? If nothing bad may happen if two threads do the initialization at the same time and use different copies of the initialized state, a benign race could be allowed. The initialized field should still be volatile (unless the initialized objects are immutable) to ensure there is a happens-before edge between threads doing the initialization and reading the field.

See also [EJ Item 83] and “Safe Publication this and Safe Initialization in Java”.

9. Non-blocking and partially blocking code

9.1. If there is some non-blocking or semi-symmetrically blocking code that mutates the state of a thread-safe class, was it deliberately checked that if a thread on a non-blocking mutation path is preempted after each statement, the object is still in a valid state? Are there enough comments, perhaps before almost every statement where the state is changed, to make it relatively easy for readers of the code to repeat and verify the check?

9.2. Is it possible to simplify some non-blocking code by confining all mutable state in an immutable POJO and update it via compare-and-swap operations? This pattern is also mentioned in item 4.5. Instead of a POJO, a single long value could be used if all parts of the state are integers that can together fit 64 bits. See also [JCIP 15.3.1].

10. Threads and Executors

10.1. Are Threads given names when created? Are ExecutorServices created with thread factories that name threads?

It appears that different projects have different policies regarding other aspects of Thread creation: whether to make them daemon with setDaemon(), whether to set thread priorities and whether a ThreadGroup should be specified. Many of such rules can be effectively enforced with forbidden-apis.

10.2. Aren’t there threads created and started, but not stored in fields, a-la **new Thread(...).start()**, in some methods that may be called repeatedly? Is it possible to delegate the work to a cached or a shared ExecutorService instead?

10.3. Aren’t some network I/O operations performed in an Executors.newCachedThreadPool()-created ExecutorService? If a machine that runs the application has network problems or the network bandwidth is exhausted due to increased load, CachedThreadPools that perform network I/O might begin to create new threads uncontrollably.

10.4. Aren’t there blocking or I/O operations performed in tasks scheduled to a ForkJoinPool (except those performed via a [managedBlock()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ForkJoinPool.html#managedBlock(java.util.concurrent.ForkJoinPool.ManagedBlocker)) call)? Parallel Stream operations are executed in the common ForkJoinPool implicitly, as well as the lambdas passed into CompletableFuture’s methods whose names end with “Async”.

This advice should not be taken too far: occasional transient IO (such as that may happen during logging) and operations that may rarely block (such as ConcurrentHashMap.put() calls) usually shouldn’t disqualify all their callers from execution in a ForkJoinPool or in a parallel Stream. See Parallel Stream Guidance for the more detailed discussion of those tradeoffs.

See also section 14 in this checklist about parallel Streams.

10.5. Opposite of the previous item: can non-blocking computations be parallelized or executed asynchronously by submitting tasks to ForkJoinPool.commonPool() or via parallel Streams instead of using a custom thread pool (e. g. created by one of the static factory methods from ExecutorServices)? Unless the custom thread pool is configured with a ThreadFactory that specifies a non-default priority for threads or a custom exception handler (see item 10.1) there is little reason to create more threads in the system instead of reusing threads of the common ForkJoinPool.

11. Thread interruption and Future cancellation

11.1. If some code propagates InterruptedException wrapped into another exception (e. g. RuntimeException), is the interruption status of the current thread restored before the wrapping exception is thrown?

Propagating InterruptedException wrapped into another exception is a controversial practice (especially in libraries) and it may be prohibited in some projects completely, or in specific subsystems.

11.2. If some method returns normally after catching an InterruptedException, is this coherent with the (documented) semantics of the method? Returning normally after catching an InterruptedException usually makes sense only in two types of methods:

• Runnable.run() or Callable.call() themselves, or methods that are intended to be submitted as tasks to some Executors as method references. Thread.currentThread().interrupt() should still be called before returning from the method, assuming that the interruption policy of the threads in the Executor is unknown.
• Methods with “try” or “best effort” semantics. Documentation for such methods should be clear that they stop attempting to do something when the thread is interrupted, restore the interruption status of the thread and return.

If a method doesn’t fall into either of these categories, it should propagate InterruptedException directly or wrapped into another exception (see the previous item), or it should not return normally after catching an InterruptedException, but rather continue execution in some sort of retry loop, saving the interruption status and restoring it before returning (see an example from JCIP). Fortunately, in most situations, it’s not needed to write such boilerplate code: one of the methods from [Uninterruptibles](https://google.github.io/guava/releases/27.0.1-jre/api/docs/com/google/common/util/concurrent/Uninterruptibles.html) utility class from Guava can be used.

11.3. If an **InterruptedException** or a TimeoutException is caught on a Future.get() call and the task behind the future doesn’t have side effects, i. e. get() is called only to obtain and use the result in the context of the current thread rather than achieve some side effect, is the future canceled?

See [JCIP 7.1] for more information about thread interruption and task cancellation.

12. Time

12.1. Are values returned from **System.nanoTime()** compared in an overflow-aware manner, as described in the documentation for this method?

12.2. Does the code that compares values returned from **System.currentTimeMillis()** have precautions against “time going backward”? This might happen due to time correction on a server. Values that are returned from currentTimeMillis() that are less than some other values that have already been seen should be ignored. Otherwise, there should be comments explaining why this issue is not relevant for the code.

Alternatively, System.nanoTime() could be used instead of currentTimeMillis(). Values returned from nanoTime() never decrease (but may overflow — see the previous item). Warning: nanoTime() didn’t always uphold to this guarantee in OpenJDK until 8u192 (see JDK-8184271). Make sure to use the freshest distribution.

In distributed systems, the leap second adjustment causes similar issues.

12.3. Do variables that store time limits and periods have suffixes identifying their units, for example, “timeoutMillis” (also -Seconds, -Micros, -Nanos) rather than just “timeout”? In method and constructor parameters, an alternative is providing a [TimeUnit](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/TimeUnit.html) parameter next to a “timeout” parameter. This is the preferred option for public APIs.

12.4. Do methods that have “timeout” and “delay” parameters treat negative arguments as zeros? This is to obey the principle of least astonishment because all timed blocking methods in classes from java.util.concurrent.* follow this convention.

13. Thread safety of Cleaners and native code

13.1. If a class manages native resources and employs java.lang.ref.Cleaner (or sun.misc.Cleaner; or overrides Object.finalize()) to ensure that resources are freed when objects of the class are garbage collected, and the class implements Closeable with the same cleanup logic executed from close() directly rather than through Cleanable.clean() (or sun.misc.Cleaner.clean()) to be able to distinguish between explicit close() and cleanup through a cleaner (for example, clean() can log a warning about the object not being closed explicitly before freeing the resources), is it ensured that even if the cleanup logic is called concurrently from multiple threads, the actual cleanup is performed only once? The cleanup logic in such classes should obviously be idempotent because it’s usually expected to be called twice: the first time from the close() method and the second time from the cleaner or finalize(). The catch is that the cleanup must be concurrently idempotent, even if close() is never called concurrently on objects of the class. That’s because the garbage collector may consider the object to become unreachable before the end of a close() call and initiate cleanup through the cleaner or finalize() while close() is still being executed.

Alternatively, close() could simply delegate to Cleanable.clean() (sun.misc.Cleaner.clean()) which is thread-safe. But then it’s impossible to distinguish between explicit and automatic cleanup.

See also JLS 12.6.2.

13.2. In a class with some native state that has a cleaner or overrides finalize(), are bodies of all methods that interact with the native state wrapped with
try { ... } finally { Reference.reachabilityFence(this); },
including constructors and the close() method, but excluding finalize()? This is needed because an object could become unreachable and the native memory might be freed from the cleaner while the method that interacts with the native state is being executed, that might lead to use-after-free or JVM memory corruption.

reachabilityFence() in close() also eliminates the race between close() and the cleanup executed through the cleaner or finalize() (see the previous item), but it may be a good idea to retain the thread safety precautions in the cleanup procedure, especially if the class in question belongs to the public API of the project because otherwise if close() is accidentally or maliciously called concurrently from multiple threads, the JVM might crash due to double memory free or, worse, memory might be silently corrupted, while the promise of the Java platform is that whatever buggy some code is, as long as it passes bytecode verification, thrown exceptions should be the worst possible outcome, but the virtual machine shouldn’t crash. Item 13.4 also stresses on this principle.

Reference.reachabilityFence() has been added in JDK 9. If the project targets JDK 8 and Hotspot JVM, any method with an empty body is an effective emulation of reachabilityFence().

See the documentation for [Reference.reachabilityFence()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ref/Reference.html#reachabilityFence(java.lang.Object)) and this discussion in the concurrency-interest mailing list for more information.

13.3. Aren’t there classes that have cleaners or override finalize() not to free native resources, but merely to return heap objects to some pools, or merely to report that some heap objects are not returned to some pools? This is an antipattern because of the tremendous complexity of using cleaners and finalize() correctly (see the previous two items) and the negative impact on performance (especially of finalize()), that might be even larger than the impact of not returning objects back to some pool and thus slightly increasing the garbage allocation rate in the application. If the latter issue arises to be any important, it should better be diagnosed with async-profiler in the allocation profiling mode (-e alloc) than by registering cleaners or overriding finalize().

This advice also applies when pooled objects are direct ByteBuffers or other Java wrappers of native memory chunks. [async-profiler -e malloc](https://stackoverflow.com/questions/53576163/interpreting-jemaloc-data-possible-off-heap-leak/53598622#53598622) could be used in such cases to detect direct memory leaks.

13.4. If some classes have some state in native memory and are used actively in concurrent code, or belong to the public API of the project, was it considered making them thread-safe? As described in item 13.2, if objects of such classes are inadvertently accessed from multiple threads without proper synchronization, memory corruption and JVM crashes might result. This is why classes in the JDK such as [java.util.zip.Deflater](http://hg.openjdk.java.net/jdk/jdk/file/a772e65727c5/src/java.base/share/classes/java/util/zip/Deflater.java) use synchronization internally despite Deflater objects are not intended to be used concurrently from multiple threads.

Note that making classes with some state in native memory thread-safe also implies that the native state should be safely published in constructors. This means that either the native state should be stored exclusively in final fields, or [VarHandle.storeStoreFence()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/invoke/VarHandle.html#storeStoreFence()) should be called in constructors after full initialization of the native state. If the project targets JDK 9 and VarHandle is not available, the same effect could be achieved by wrapping constructors’ bodies in synchronized (this) { ... }.

14. Parallel Streams

14.1. For every use of parallel Streams via Collection.parallelStream() or Stream.parallel(): is it explained why parallel Stream is used in a comment preceding the stream operation? Are there back-of-the-envelope calculations or references to benchmarks showing that the total CPU time cost of the parallelized computation exceeds 100 microseconds?

Is there a note in the comment that parallelized operations are generally I/O-free and non-blocking, as per item 10.4? The latter might be obvious momentary, but as codebase evolves the logic that is called from the parallel stream operation might become blocking accidentally. Without comment, it’s harder to notice the discrepancy and the fact that the computation is no longer a good fit for parallel Streams. It can be fixed by calling the non-blocking version of the logic again or by using a simple sequential Stream instead of a parallel Stream.

Bonus: is forbidden-apis configured for the project and are java.util.StringBuffer, java.util.Random and Math.random() prohibited? StringBuffer and Random are thread-safe and all their methods are synchronized, which is never useful in practice and only inhibits the performance. In OpenJDK, Math.random() delegates to a global static Random instance. StringBuilder should be used instead of StringBuffer, ThreadLocalRandom or SplittableRandom should be used instead of Random.

Reading List

• [JLS] Java Language Specification, Memory Model and [final](https://docs.oracle.com/javase/specs/jls/se11/html/jls-17.html#jls-17.5) field semantics.
• [EJ] “Effective Java” by Joshua Bloch, Chapter 11. Concurrency.
• [JCIP] “Java Concurrency in Practice” by Brian Goetz, Tim Peierls, Joshua Bloch, Joseph Bowbeer, David Holmes, and Doug Lea.
• Posts by Aleksey Shipilёv:
  Safe Publication and Safe Initialization in Java
  Java Memory Model Pragmatics
  Close Encounters of The Java Memory Model Kind
• When to use parallel streams written by Doug Lea, with the help of Brian Goetz, Paul Sandoz, Aleksey Shipilev, Heinz Kabutz, Joe Bowbeer, …

We use object pool in jasync-sql to manage connections to the database. In this post, I will share how it is done in a performant, lock-free manner using Kotlin coroutines with an Actor.

An object pool has a very simple API to work with. It is a pool of objects with two methods: take() and return().

On first sight it looks like a very simple problem. The main catch here is that it has to be both performant and thread-safe, and that’s what makes it interesting and tricky to implement.

But hey! Why do we need an object pool anyway?

jasync-sql is a library to access relational databases like MySQL and PostgreSQL. Database connections are a great example of the need for object pools. The access to the database is done by obtaining a connection from a Connection-Pool, using it and returning it back to the pool.

With a connection pool we get a couple of advantages over creating connections per each SQL query:

• Reusing connections — since the overhead of initiating a connection to the database is high (handshake, etc), connection pools allow keeping connections alive, thus reducing that overhead.
• Limiting resources — creating a DB connection per user request can be overwhelming to the DB. Using a pool effectively adds a barrier, limiting the number of maximum number of concurrent connections.

Well, I am sold, but…

Isn’t a Connection Pool a solved problem in the Java world?

Yes it is a solved problem if you’re using JDBC. In that case HikariCP is an excellent choice from my experience, but there are a lot of others. In the case of jasync-sql it is not possible to use HikariCP, because HikariCP works with the JDBC API, and the jasync-sql driver is not implementing that full-fledged API, only a subset of it.

What about other Object pools in Java world?

There are numerous implementations, but it turns out that you usually find some specific requirement that was not implemented by that pool you’re using.

In our case, that requirement was non-blocking. In our pool, all operations have to be non-blocking since the library is async. For example, the take() operation in most implementations returns an object immediately or blocks until an object is ready. Our take() returns a Future<Connection>, which will be completed and continued when the connection is ready to use.

I haven’t seen such an implementation in the wild.

I really like this answer from Stack Exchange:

Is object pooling a deprecated technique?
_Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working…_softwareengineering.stackexchange.com

Another requirement that makes it hard to find an alternative is the need to try and stay compatible as much as possible with the current implementation we have.

In case you want to see other implementations you can check here:

object pool in java - Google Search
_object pool is a collection of a particular object that an application will create and keep on hand for those…_www.google.co.il

So how did we implement Object Pool?

Before we dive into the details, let’s observe other requirements from the object pool that were omitted above for clarity but are necessary details.

Interfaces

The Object pool interface looks like this:

interface AsyncObjectPool<T&gt; {  fun take(): CompletableFuture&lt;T>  fun giveBack(item: T): CompletableFuture<AsyncObjectPool<T>>  fun close(): CompletableFuture<AsyncObjectPool<T>>

}

In addition, when a pool wants to create new objects (connections) it will call the ObjectFactory. The factory has a couple more methods to handle the object lifecycle:

• validate — a method to check that the object is still valid. The method should be fast and check only in-memory constructs. For connections we usually check that the last query did not throw an exception and did not get a termination message from netty.
• test — similar to validate, but a more exhaustive check. We allow test method to be slow and access the network etc. This method is used to check that idle objects are still valid. For connections, that will be something similar to select 0.
• destroy — called to clean up the object when the pool is not using it anymore.

The complete interface is:

interface ObjectFactory<T> {  fun create(): CompletableFuture<;out T>  fun destroy(item: T)  fun validate(item: T): Try<T>  fun test(item: T): CompletableFuture<T>

}

For pool configuration we have the following properties:

• maxObjects — maximum number of connections we allow.
• maxIdle — time that we leave the connection open without use. After that time it will be reclaimed.
• maxQueueSize — when a request for a connection arrives and no connection is available, we put the request on hold in a queue. In case the queue is full (its size passed maxQueueSize) it will not wait but instead return an error.
• createTimeout — maximum time to wait for a new connection to be created.
• testTimeout — maximum time to wait for a test query on an idle connection. If it passes we will consider the connection as erroneous.
• validationInterval — on this interval, we will test if the idle connections are active and free up connections that passed maxIdle. We will also remove connections that passed testTimeout.

Original implementation

The first implementation of object pool was single threaded. All operations were sent to a worker thread that was responsible to execute them. This method is known as thread-confinement. Object creation and test operations were blocking and query execution itself was non-blocking.

This method is problematic because operations are done one after another. On top of that, there are a couple of operations that are blocking as mentioned above. There were various cases of high latency when working in some scenarios and use cases (like here for example).

As a workaround PartitionedPool was introduced. This is a workaround to the block issue with the above single-threaded approach. The partitioned pool creates multiple SingleThreadedObjectPools, each with its own worker. When a connection is requested, a pool is selected by a modulus on the thread id. The partitioned pool is actually a pool of pools ;-)

I mentioned this is a workaround since it has its own problems: you might still be blocking, but at a lower rate — plus it consume more threads and resources.

Actor based implementation

An Actor is an entity that has a mailbox. It receives messages to its mailbox and processes them one after the other. The mailbox is a sort of a channel to pass events from the outside world to the actor.

A coroutines actor employs lock-free algorithms to allow fast and performant execution of events without the need for locks and synchronized blocks.

_“wall rack filled with paper document lot” by [Unsplash](https://unsplash.com/@californong?utm_source=medium&utm_medium=referral" rel="noopener" target="_blank" title="">Nong Vang on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" rel="noopener" target="blank" title=")

You can see an elaborated explanation here.

In our case those events will be take and giveBack. In addition to those, we will have internal messages that the actor sends to itself like objectCreated etc. That allows the actor to have states that does not suffer from concurrency problems, as it is always confined to the same sequential execution. In addition the channel that passes those events is a queue that is using lock-free algorithms so it is very efficient, avoids contention, and generally has very high performance.

There is an excellent video explaining how this was implemented (note that this is “heavy” algorithmic staff):

Let’s recap what we have until now:

• An actor receives messages and processes them one by one.
• Usually messages will contain a CompletableFuture that should be completed when the actor processes it.

Messages will be completed immediately or delayed (like in case we are waiting for a connection to be created). If it is delayed the actor will put the Future in a queue, and will use a callback mechanism to notify itself when the original future can be completed.

• Message processing in the actor should not be blocked or delay the actor. If this happens, it will delay all messages waiting to be processed in the queue and will slow down the entire actor operation.

That’s why, in case we have long running operations inside the actor, we use the callback mechanism.

Let’s see more details on the use cases

Take — someone wants an object from the pool. It will send a message with a callback to the actor. The actor will do one of the following things:

• If the object is available — the actor will simply return it.
• If the pool hasn’t passed the limit of created objects — the actor will create a new object and return it when the object is ready.

In such a case, object creation can take time, so the actor will connect the callback from the object creation to the original take request callback.

• Will put the request in a queue for an available object (unless the queue is full and in that case will just return an error).

GiveBack — someone wants to give an object back to the pool (release it). This is also done by a message to the actor. The actor will do one of the following:

• If someone is waiting on the wait queue — it will borrow the object to it.
• In other cases it will just keep the object on the pool for requests to come, so the object remains idle.

Test — periodically, someone from outside will notify the actor to test connections:

• The actor will release the idle connection that hasn’t been used for a long time (it’s configurable).
• The actor will test other idle objects using the ObjectFactory. It will send a callback to the factory and mark those objects as In Use, to prevent from borrowing them until the test is completed.
• The actor will check for timeouts in tests and destroy time-outed objects.

Those are the main use cases.

Leaks

_“selective focus photography of brown faucet” by [Unsplash](https://unsplash.com/@leipuri?utm_source=medium&utm_medium=referral" rel="noopener" target="_blank" title="">Jouni Rajala on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" rel="noopener" target="blank" title=")

There can be all sort of leaks in an object pool. Some are internal bugs which I hope are easier to spot and fix, and others are objects that were taken but not returned due to some user error. In such cases, objects might remain in the “In Use” queue forever.

To avoid such cases, the “In Use” Map is using Java’s WeakHashMap. So if a user lost a connection it will be automatically removed from the map when it is cleaned by Java’s Garbage-Collector.

In addition we added a log message in such cases that says: “LEAK-DETECTED”.

That’s it!

The full Kotlin source code of the object pool is available here:

jasync-sql/jasync-sql
_Java async database driver for MySQL and PostgreSQL written in Kotlin - jasync-sql/jasync-sql_github.com

In an upcoming post I will compare performance metrics of the different implementations.

If you want to read more about Kotlin there is a nice introduction here:

And for coroutines in general check out this video:

Finally if you want to learn more about Actors implementation using coroutines in Kotlin, then head over here:

Kotlin/kotlinx.coroutines
_Library support for Kotlin coroutines . Contribute to Kotlin/kotlinx.coroutines development by creating an account on…_github.com

Thanks for reading! ❤️

_“aerial photography of woman on pink swimming floats” by [Unsplash](https://unsplash.com/@tom_grimbert?utm_source=medium&utm_medium=referral" rel="noopener" target="_blank" title="">Tom Grimbert on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" rel="noopener" target="blank" title=")

Usually, the machine that executes our code has limited resources. Doing everything at once might not only hurt, but can also hang our process and make it stop responding altogether.

When we want to crawl 100 websites, we should crawl, for example, 5 at once, so that we don’t take up all the available bandwidth. As soon as one website is crawled, the next one is ready to go.

Generally speaking, all “heavy” operations should be laid out in time. They should not be executed all-at-once, for better performance and to save resources.

Implementation

If you are familiar with my previous post about implementing promises, then you are going to notice many similarities.

class Concurrently<T = any> {
  private tasksQueue: (() => Promise<T>)[] = [];
  private tasksActiveCount: number = 0;
  private tasksLimit: number;

  public constructor(tasksLimit: number) {
    if (tasksLimit < 0) {
      throw new Error('Limit cant be lower than 0.');
    }

    this.tasksLimit = tasksLimit;
  }

  private registerTask(handler) {
    this.tasksQueue = [...this.tasksQueue, handler];
    this.executeTasks();
  }

  private executeTasks() {
    while (this.tasksQueue.length && this.tasksActiveCount < this.tasksLimit) {
      const task = this.tasksQueue[0];
      this.tasksQueue = this.tasksQueue.slice(1);
      this.tasksActiveCount += 1;

      task()
        .then((result) => {
          this.tasksActiveCount -= 1;
          this.executeTasks();

          return result;
        })
        .catch((err) => {
          this.tasksActiveCount -= 1;
          this.executeTasks();

          throw err;
        });
    }
  }

  public task(handler: () => Promise<T>): Promise<T> {
    return new Promise((resolve, reject) =>
      this.registerTask(() =>
        handler()
          .then(resolve)
          .catch(reject),
      ),
    );
  }
}

export default Concurrently;

We register a given task by adding it to our tasksQueue and then we call executeTasks.

Now we execute as many tasks as our limit allows us — one by one. Each time adding 1 to our counter called tasksActiveCount.

When the executed task finishes, we remove 1 from tasksActiveCount and again call executeTasks.

Below we can see an example of how it works.

The limit is set to 3. The first two tasks are taking very long to process. We can see the third “slot” getting opened from time to time, allowing the next task in the queue to be executed.

Always there are three, no more, no less.

Executing heavy and light tasks with the limit of 3.

You can see the code in the repository.

Thank you very much for reading! Can you think of any other way of achieving the same effect? Share them down below.

If you have any questions or comments feel free to put them in the comment section below or send me a message.

Check out my social media!

Join my newsletter!

Originally published at www.mcieslar.com on August 28, 2018.