But do you know what actually happens when your application sends a request? What travels through the wire? Why does HTTP/2 make things faster? Why do WebSockets exist when HTTP already works? What makes Protocol Buffers different from JSON at a fundamental level?

And when you're designing a system, how do you decide which communication approach to use?

These are the questions this handbook answers.

This isn't a beginner's guide to APIs. This is a deep dive into how clients and servers actually communicate: the protocols, the trade-offs, the history of why each approach was built, and the engineering thinking behind choosing one over another.

By the end, you won't just know what these technologies are. You'll understand why they exist, how they work at a level that makes you a better engineer, and how to make deliberate architectural decisions about communication in your systems.

Table of Contents

• The Foundation: How Two Machines Talk to Each Other

• HTTP/1.1: The Protocol That Built the Web

• The Problems HTTP/1.1 Could Not Solve

• HTTP/2: Rebuilding the Foundation

• HTTP/3 and QUIC: The Next Evolution

• Data Formats: How Information Is Encoded

• REST: The Architecture That Took Over the World

• The Limits of REST

• GraphQL: Letting the Client Decide

• WebSockets: When HTTP Is Not Enough

• Server-Sent Events: The Simpler Real-Time Option

• Protocol Buffers: A New Language for Data

• gRPC: Remote Procedure Calls at Scale

• The Complete Comparison

• How to Choose: The Engineering Decision Framework

• Conclusion

The Foundation: How Two Machines Talk to Each Other

Before any protocol, data format, or architectural style enters the picture, two machines need to establish a connection. Understanding this foundation makes everything else click.

IP Addresses and Ports

Every device on a network has an IP address: a unique identifier that works like a postal address. When your application sends a request to api.example.com, the first thing that happens is a DNS lookup, which translates that human-readable name into an IP address like 93.184.216.34. That IP address is where the packet is going.

But an IP address alone isn't enough. A single server might be running dozens of different services simultaneously: a web server, a database, an email server, an SSH daemon.

Ports tell the operating system which service should handle the incoming connection. Port 80 is the conventional port for HTTP. Port 443 is for HTTPS. Port 5432 is for PostgreSQL. Port 22 is for SSH. When you call api.example.com/users, you are actually calling api.example.com:443/users. The browser fills in the port automatically.

TCP: The Reliable Foundation

Most web communication runs over TCP (Transmission Control Protocol). TCP is a connection-oriented protocol, which means before any data is exchanged, both parties go through a handshake to establish a connection.

The TCP handshake works in three steps, which is why it's called the three-way handshake:

Client                    Server
  |                          |
  |-------- SYN ----------->|   "I want to connect"
  |                          |
  |<------- SYN-ACK --------|   "Okay, I acknowledge. Ready?"
  |                          |
  |-------- ACK ----------->|   "Great, let's go"
  |                          |
  [Connection established]

SYN stands for synchronize. ACK stands for acknowledge. After these three packets, the connection exists and data can flow.

TCP guarantees three things that make it the foundation of reliable communication:

1. Delivery: if a packet is lost in transit, TCP detects this and retransmits it automatically. The application layer never has to worry about lost packets.

2. Order: packets arrive in the same order they were sent. If packets arrive out of order (which happens frequently on real networks), TCP reorders them before delivering them to the application.

3. Error detection: every TCP packet includes a checksum. If the data is corrupted in transit, TCP detects and discards the corrupted packet, then requests a retransmission.

This reliability comes at a cost: the overhead of the handshake, the acknowledgment packets, and the retransmission logic.

For many use cases, this cost is worth it. For some (live video streaming, online gaming, DNS lookups), UDP (User Datagram Protocol) is preferred because it sends packets without any of this overhead, accepting some loss in exchange for speed. HTTP/3, which we'll cover later, is built on a protocol that brings reliability to UDP.

TLS: Encrypting the Connection

On the modern web, most connections use HTTPS rather than plain HTTP. The S stands for Secure, and the security is provided by TLS (Transport Layer Security), the successor to SSL.

TLS adds an additional handshake on top of the TCP connection. During the TLS handshake:

1. The client and the server agree on which version of TLS to use and which encryption algorithms to support

2. The server presents its digital certificate (issued by a trusted Certificate Authority)

3. The client verifies the certificate is valid and belongs to the server it intended to reach

4. They exchange encryption keys using asymmetric cryptography

5. From that point forward, all communication is encrypted with symmetric encryption

The TLS handshake adds latency. In TLS 1.2, it takes two round trips before any application data can flow. TLS 1.3, released in 2018, reduced this to one round trip, and even supports zero round-trip resumption for returning connections.

Understanding TCP and TLS matters because every protocol we discuss runs on top of them (until HTTP/3, which changes the underlying transport). When people talk about the "overhead" of HTTPS or the "cost" of establishing a connection, they're talking about the time and packets spent on these handshakes before a single byte of your actual request travels.

HTTP/1.1: The Protocol That Built the Web

HTTP (HyperText Transfer Protocol) was invented by Tim Berners-Lee in 1991 to transfer HTML documents between computers. HTTP/1.0 was simple: one request per connection, then the connection closes.

HTTP/1.1, standardized in 1997, brought significant improvements and became the dominant version of HTTP for nearly two decades. It introduced persistent connections (keep connections open across multiple requests), chunked transfer encoding, and more sophisticated caching mechanisms.

How an HTTP/1.1 Request Works

An HTTP request is a text message with a specific structure:

POST /api/users HTTP/1.1
Host: api.example.com
Content-Type: application/json
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9...
Accept: application/json
Content-Length: 45
User-Agent: MyApp/2.0

{"name": "John Smith", "email": "john@example.com"}

The first line is the request line: the HTTP method (POST), the path (/api/users), and the protocol version.

Below that are the headers: key-value pairs that provide metadata about the request. The host, the content type, the authorization token, what format the client accepts, and how large the body is.

After a blank line comes the body: the actual data being sent.

The server processes this and responds:

HTTP/1.1 201 Created
Content-Type: application/json
Location: /api/users/usr_789
Date: Mon, 21 Jul 2026 09:15:00 GMT
Content-Length: 89

{"id": "usr_789", "name": "John Smith", "email": "john@example.com", "created_at": "..."}

The response has a status line (the protocol version, the status code, and a reason phrase), headers, and a body.

HTTP Methods and Their Semantics

HTTP/1.1 defines several methods, each with specific semantics:

• GET retrieves a resource. A GET request should have no side effects. It shouldn't create or modify anything. It's safe and idempotent, meaning calling it multiple times has the same effect as calling it once.

• POST submits data to create a new resource or trigger an action. It's neither safe nor idempotent: calling POST twice typically creates two resources.

• PUT replaces a resource entirely with the provided data. It's idempotent: calling PUT twice with the same data has the same effect as calling it once.

• PATCH partially updates a resource. Only the fields provided are changed.

• DELETE removes a resource. It's idempotent: deleting something that doesn't exist is still considered successful.

• HEAD is identical to GET but the server only returns headers, not the body. It's used to check if a resource exists or has been modified without downloading the full content.

• OPTIONS asks the server what methods are allowed for a resource. It's used in CORS preflight requests.

Status Codes

HTTP status codes are three-digit numbers grouped into five categories:

1xx Informational — the server has received the request and is continuing to process it. These are rarely seen in practice outside of specific use cases like HTTP upgrade (used to establish WebSocket connections).

2xx Success — the request was received, understood, and accepted.

• 200 OK: standard success response

• 201 Created: a new resource was created

• 204 No Content: success but nothing to return (common for DELETE)

3xx Redirection — further action is required to complete the request.

• 301 Moved Permanently: the resource has a new URL forever

• 302 Found: temporary redirect

• 304 Not Modified: the cached version is still valid (used with ETags)

4xx Client Error — the request contains bad syntax or can't be fulfilled.

• 400 Bad Request: the request is malformed

• 401 Unauthorized: authentication is required (despite the name, it means unauthenticated)

• 403 Forbidden: authenticated but not authorized to access this resource

• 404 Not Found: the resource doesn't exist

• 422 Unprocessable Entity: the request is syntactically valid but semantically wrong (common for validation errors)

• 429 Too Many Requests: rate limit exceeded

5xx Server Error — the server failed to fulfill a valid request.

• 500 Internal Server Error: something went wrong on the server

• 502 Bad Gateway: the server received an invalid response from an upstream server

• 503 Service Unavailable: the server is temporarily unavailable

• 504 Gateway Timeout: the upstream server did not respond in time

Caching in HTTP/1.1

One of HTTP/1.1's most powerful features is its built-in caching model. Responses can include headers that tell clients and intermediate caches how long to store a response and when to revalidate it.

• Cache-Control: max-age=3600 tells the client to cache this response for one hour.

• Cache-Control: no-cache tells the client to always revalidate with the server before using a cached response.

• Cache-Control: no-store tells the client never to cache this response.

ETag is a fingerprint of the response content. When the client makes a subsequent request, it sends the ETag back in an If-None-Match header. If the content hasn't changed, the server responds with 304 Not Modified and no body, saving bandwidth.

Last-Modified works similarly: the client sends If-Modified-Since and the server confirms whether the content has changed.

Caching is one of the key reasons REST over HTTP became dominant. GET requests to well-designed REST APIs can be cached at the CDN level, meaning the same response is served to thousands of users without the request ever reaching your origin server.

The Problems HTTP/1.1 Could Not Solve

HTTP/1.1 served the web well for two decades. But as the web grew more complex, applications more dynamic, and user expectations higher, its architectural limitations became significant performance bottlenecks.

Head-of-Line Blocking

HTTP/1.1 processes requests sequentially on a single connection. The server must finish responding to one request before the next one on the same connection begins.

Connection 1:
Request 1 (slow database query) -----> [3 seconds] -----> Response 1
Request 2 (fast in-memory read) -----> [waits 3 seconds] -----> Response 2
Request 3 (static file) -----------> [waits 3+ seconds] -----> Response 3

Request 2 and Request 3 are fast operations. But they're stuck waiting for Request 1 to complete. This is head-of-line blocking: the head of the queue blocks everything behind it.

Browsers worked around this by opening multiple parallel TCP connections to the same server, typically six. But each connection requires its own TCP handshake and TLS negotiation, consuming resources on both the client and server.

Verbose Headers on Every Request

Every HTTP/1.1 request sends its complete headers as plain text. Consider a mobile application making fifty requests during a session. On every single request, the following headers are sent in full:

Host: api.example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c3JfMTIzIn0...
Content-Type: application/json
Accept: application/json
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X)...

The Authorization header alone, carrying a JWT, can be 400 to 600 bytes. Multiplied by fifty requests, that is 20 to 30 kilobytes of data carrying nothing but headers that haven't changed between requests.

On a 4G mobile connection with limited bandwidth, this is waste. On a 2G connection in a network-constrained environment, it's a significant performance penalty.

No Server Push

HTTP/1.1 is strictly request-response. The server can't send data until the client asks for it. This fundamental limitation means the server can never proactively inform the client of changes.

For applications requiring real-time updates, short polling became a common workaround: the client sends a request every few seconds asking "has anything changed?" This is inefficient because most polling requests receive a "no, nothing has changed" response, consuming bandwidth and server resources for no purpose.

Long polling was a refinement: the client sends a request and the server holds it open until something changes or a timeout occurs. This reduces unnecessary responses but keeps connections open indefinitely, consuming server resources.

Both are workarounds for a fundamental limitation of HTTP/1.1's request-response model.

Inefficient Use of Connections

Opening a new TCP connection requires the three-way handshake plus the TLS handshake: a process that can take 200 to 500 milliseconds on a mobile connection.

HTTP/1.1 introduced keep-alive connections to reuse connections across multiple requests, but head-of-line blocking made this only partially effective. Browsers opened multiple connections to compensate, but six parallel connections per domain is both a client limitation and a server resource concern at scale.

HTTP/2: Rebuilding the Foundation

Google published a protocol called SPDY (pronounced "speedy") in 2009, designed to address HTTP/1.1's performance limitations. SPDY demonstrated that significant improvements were possible without changing the fundamental HTTP semantics. HTTP/2, standardized by the IETF in 2015, was heavily based on SPDY and became the successor to HTTP/1.1.

HTTP/2 doesn't change what you send. From the application developer's perspective, requests still have methods, paths, headers, and bodies. Responses still have status codes, headers, and bodies. What HTTP/2 changes is how all of this is transmitted.

Binary Framing: The Core Change

HTTP/1.1 is a text protocol. Headers, status lines, and method names are all ASCII text. Machines must parse this text character by character to interpret it.

HTTP/2 is a binary protocol. Every piece of information is encoded as binary frames rather than text. Binary is more compact and significantly faster for machines to parse. Instead of tokenizing a string looking for colons and newlines to separate header names from values, a binary parser reads fixed-length fields directly from memory.

The binary framing layer is the foundation everything else in HTTP/2 is built upon.

Multiplexing: Many Streams, One Connection

HTTP/2 introduces the concept of streams. A stream is an independent, bidirectional sequence of frames within a single TCP connection. Multiple streams can exist simultaneously on the same connection.

Single TCP connection to api.example.com

Stream 1: GET /user/profile ---------> Response arrives
Stream 2: GET /user/balance ---------> Response arrives
Stream 3: POST /transactions --------> Response arrives
Stream 4: GET /notifications --------> Response arrives

All four streams active simultaneously
No stream waits for any other stream

This is multiplexing: many independent requests and responses interleaved on the same connection. Head-of-line blocking at the HTTP level is eliminated. A slow request on Stream 1 doesn't prevent Stream 2, 3, or 4 from receiving their responses.

One connection replaces six parallel connections. The TCP handshake and TLS negotiation happen once. Connection overhead drops dramatically.

Header Compression with HPACK

HTTP/2 compresses headers using an algorithm called HPACK specifically designed for HTTP headers.

HPACK works in two ways. First, it maintains a table of previously seen headers. Instead of retransmitting a header that was sent on the previous request, it sends a reference to the table entry: a single integer instead of hundreds of bytes of text.

Second, HPACK uses Huffman encoding for new header values, reducing the size of strings that can't be referenced from the table.

The result: a mobile application sending the same Authorization header on every request transmits it in full on the first request, then sends a one-byte or two-byte reference on every subsequent request. What was 500 bytes of overhead becomes 2 bytes.

Across fifty requests in a session, this eliminates thousands of bytes of redundant header transmission.

Stream Prioritization

HTTP/2 allows clients to assign priority to streams. A browser loading a web page can signal that the CSS file (needed to render anything) is higher priority than the analytics script (not needed for initial render). The server can use these priorities to decide the order in which it sends frames when multiple streams are active.

In practice, stream prioritization has been inconsistently implemented and is being redesigned in HTTP/3.

Server Push

HTTP/2 allows the server to proactively send resources to the client without waiting for a request. When a browser requests an HTML file, the server can immediately push the CSS and JavaScript files it knows the browser will need next, before the browser has even parsed the HTML to discover it needs them.

Client: GET /index.html
Server: Here is index.html
Server: (push) Here is styles.css — you will need this
Server: (push) Here is app.js — you will need this too

In practice, server push has had mixed adoption due to implementation complexity and the risk of pushing resources the client already has cached. HTTP/3 is reconsidering how push should work.

HTTP/2 and gRPC

HTTP/2's multiplexing and persistent connections make it the ideal transport for gRPC. A single HTTP/2 connection can carry many concurrent gRPC calls, including long-running streaming calls that push data continuously. This is why gRPC requires HTTP/2: the features that make gRPC efficient are provided by the transport layer.

HTTP/3 and QUIC: The Next Evolution

Even with HTTP/2's improvements, one fundamental problem remained: TCP head-of-line blocking.

HTTP/2 eliminated head-of-line blocking at the HTTP level. Multiple HTTP/2 streams can proceed independently. But all of those streams share a single TCP connection. TCP guarantees ordered delivery of all bytes in a connection. If a single TCP packet is lost, the entire connection stalls while TCP retransmits that packet, even for streams that have nothing to do with the lost packet.

HTTP/2 over TCP — packet loss scenario:

Stream 1: data in flight...
Stream 2: data in flight...
Stream 3: packet LOST — TCP retransmission required

Stream 1: STALLED (waiting for TCP retransmission)
Stream 2: STALLED (waiting for TCP retransmission)
Stream 3: retransmission in progress...

Both streams 1 and 2 are blocked by a packet loss that affected only stream 3. This is TCP head-of-line blocking, and HTTP/2 can't eliminate it because it operates above the TCP layer.

QUIC: A New Transport Protocol

Google developed QUIC (Quick UDP Internet Connections) to solve this problem. QUIC is a new transport protocol built on UDP instead of TCP, designed to provide some very helpful new features:

1. Multiplexing without head-of-line blocking: QUIC understands streams natively. A packet loss in one QUIC stream only stalls that stream. Other streams on the same connection continue flowing freely.

2. Built-in encryption: Unlike TLS which runs on top of TCP, QUIC has TLS 1.3 built into the protocol itself. The transport and security layers are integrated, reducing the number of round trips required before data can flow.

3. Faster connection establishment: A new QUIC connection requires one round trip before data can flow. For returning connections where a session ticket exists, QUIC can send data in zero round trips (0-RTT).

4. Connection migration: A TCP connection is identified by the four-tuple of source IP, source port, destination IP, and destination port. If any of these change (say, a mobile device switches from WiFi to cellular), the TCP connection breaks and must be re-established. QUIC connections are identified by a connection ID that survives network changes, enabling seamless handoff.

HTTP/3

HTTP/3 is HTTP semantics over QUIC. The request and response model remains the same. Headers, status codes, and methods are all identical. The transport underneath is QUIC instead of TCP.

HTTP/3 is particularly impactful for:

1. Mobile networks where packet loss is more common and devices frequently switch between networks.

2. High-latency connections where the reduced handshake round trips save meaningful time.

3. Applications with many concurrent streams where TCP head-of-line blocking was a real bottleneck.

As of 2026, HTTP/3 is supported by major browsers, CDNs, and an increasing number of backend servers. Adoption continues to grow.

Data Formats: How Information Is Encoded

Independent of which protocol carries data, systems need to agree on how data is encoded. The most important formats for API communication are JSON and Protocol Buffers.

JSON: The Universal Language

JSON (JavaScript Object Notation) was derived from JavaScript syntax and formalized as a standalone data format. Its design philosophy is human readability and simplicity.

A JSON object is a collection of key-value pairs enclosed in curly braces. Keys are always strings. Values can be strings, numbers, booleans, null, arrays, or other objects.

{
  "id": "usr_001",
  "name": "John Smith",
  "age": 28,
  "is_verified": true,
  "scores": [98, 87, 92],
  "address": {
    "city": "Lagos",
    "country": "Nigeria"
  }
}

JSON became the dominant API data format for several reasons. It's human-readable: a developer can look at a JSON response in a browser's developer tools and immediately understand it. It maps naturally to data structures in virtually every programming language. It requires no special tooling or schema definition. And it's flexible: fields can be added or removed without necessarily breaking existing clients.

The Structural Cost of JSON

JSON's human-readable design comes with a structural cost that becomes significant at scale.

Every field name is a string that travels over the network on every single response. In the example above, the strings "is_verified", "address", "country" aren't data. They're labels for data. They consume bytes, they must be tokenized and parsed, and they're repeated on every response for every user.

JSON is a text format, which means it must be parsed from text into the application's native data structures. This parsing isn't free: it requires allocating memory for strings, walking the text byte by byte to find delimiters, and constructing objects from the parsed values.

For a fintech platform with an internal API that returns a 1000-field response and is called by dozens of internal services millions of times per day, the cumulative cost of JSON's verbosity and parsing overhead becomes measurable in bandwidth bills and server CPU time.

JSON also has no formal schema at the network level. There's nothing in the JSON format itself that prevents a backend from changing "account_balance" to "balance". The change compiles fine. The server deploys. Clients that depend on "account_balance" break silently at runtime.

XML: The Predecessor

Before JSON, XML (eXtensible Markup Language) was the dominant data format for web services (used in SOAP, the predecessor to REST). XML is more verbose than JSON, wrapping every value in opening and closing tags:

<user>
  <id>usr_001</id>
  <name>John Smith</name>
  <age>28</age>
  <is_verified>true</is_verified>
</user>

XML has advantages: it supports schemas (XSD), namespaces, and complex document structures. It's still used in enterprise systems, document formats (DOCX, SVG, RSS), and configuration files. But for API communication, JSON's simplicity won.

REST: The Architecture That Took Over the World

REST (Representational State Transfer) was defined by Roy Fielding in his doctoral dissertation in 2000. Fielding was one of the principal authors of the HTTP specification, and REST emerged from his analysis of what made HTTP architecturally successful.

REST isn't a protocol. It's an architectural style: a set of constraints that, when applied to a distributed system, produce desired properties including scalability, simplicity, and modifiability.

The Six REST Constraints

Fielding defined six constraints that define a RESTful architecture. Most APIs described as "REST" implement a subset of these, which is why the term "RESTful" covers a wide spectrum.

1. Client-Server: The client and server are separate concerns. The client manages the user interface. The server manages data storage and business logic. They evolve independently. This separation allows each to scale and change without affecting the other.

2. Stateless: Each request from the client to the server must contain all the information needed to understand and process the request. The server doesn't store any session state between requests. If a client needs to be authenticated, the authentication information (typically a token) travels with every request.

Statelessness is what makes REST APIs horizontally scalable. Any server instance can handle any request because no session state needs to be co-located with the request. Load balancers can route requests freely.

3. Cacheable: Responses must define themselves as cacheable or non-cacheable. If a response is cacheable, clients and intermediate layers (CDN, reverse proxies) can store and reuse the response without hitting the server.

Caching is one of the most powerful properties of REST. A well-designed REST API can serve millions of identical GET requests from CDN cache, with only a fraction ever reaching the origin server.

4. Uniform Interface: The interface between client and server is standardized. Resources are identified by URIs. Resources are manipulated through representations. Messages are self-descriptive. This uniformity is what makes REST APIs universally accessible: a developer in any language can call a REST API using standard HTTP tooling.

5. Layered System: The client doesn't need to know whether it's connected directly to the server or to an intermediary (load balancer, CDN, API gateway, caching proxy). Each layer only sees the layer it is interacting with. This enables transparent scaling and security.

6. Code on Demand (Optional): Servers can extend client functionality by sending executable code (JavaScript). This is the only optional constraint and is the basis for how browsers work, but rarely relevant to API design.

Resources and URIs

The central concept in REST is the resource. A resource is any piece of information that can be named, like a user, an order, a product, or a collection of transactions.

Resources are identified by URIs (Uniform Resource Identifiers). The URI identifies what the resource is, not what to do with it. The HTTP method expresses the operation.

GET    /users           — retrieve all users
GET    /users/123       — retrieve user 123
POST   /users           — create a new user
PUT    /users/123       — replace user 123 entirely
PATCH  /users/123       — partially update user 123
DELETE /users/123       — delete user 123

GET    /users/123/orders        — orders belonging to user 123
POST   /users/123/orders        — create an order for user 123
GET    /users/123/orders/456    — order 456 belonging to user 123

The URI structure forms a hierarchy that reflects the relationships between resources. This makes APIs predictable: a developer who understands the resource model can guess the correct URIs.

Why REST Won

REST became the dominant architectural style for web APIs for reasons that go beyond technical merit:

Universal accessibility: Any device, any language, any framework that can make an HTTP request can call a REST API. There's no special client library needed.

HTTP alignment: REST leverages HTTP's existing infrastructure. CDN caching works for free. Load balancers understand HTTP. Monitoring tools speak HTTP. The entire ecosystem is built around HTTP semantics.

Simplicity: A REST API can be designed, documented, and consumed with minimal tooling. A developer can test endpoints in a browser or with curl immediately.

Developer experience: JSON over HTTP is something every web developer already understands. The learning curve is essentially zero.

Ecosystem maturity: OpenAPI/Swagger provides standardized documentation. Postman provides testing. Every programming language has robust HTTP client libraries.

The Limits of REST

REST's success is real. But so are its limitations, and understanding them is essential to knowing when to reach for something else.

Overfetching: Getting More Than You Need

A REST endpoint returns a fixed shape of data. The /users/123 endpoint returns the full user object: name, email, phone, address, preferences, account status, and thirty other fields.

A mobile screen that displays only the user's name and avatar must receive all of those fields to use two of them. The rest is waste: wasted bandwidth, serialization on the server, and deserialization on the client.

On a constrained mobile connection, this overfetching isn't just inefficient. It's a measurable degradation of user experience.

Underfetching: Not Getting Enough at Once

The opposite problem is equally common. A screen needs data from multiple resources: the user's profile, their recent orders, their notification count, and their account balance.

A REST API typically models these as separate endpoints. Loading this screen requires four separate HTTP requests, each with its own round-trip latency.

GET /users/123         → profile data
GET /users/123/orders  → orders data
GET /notifications?user=123 → notification count
GET /accounts/123/balance   → balance data

Four sequential round trips. On a 200ms latency connection, that's 800ms of network time before the screen can render completely.

The N+1 Problem

A common variant of underfetching: you fetch a list of resources, then must fetch additional data for each item in the list.

GET /orders            → returns 20 orders (each with a user_id)
GET /users/1           → user for order 1
GET /users/2           → user for order 2
...
GET /users/20          → user for order 20

21 requests to load one screen. This pattern appears constantly in REST APIs and is addressed in various ways: including nested data in responses, adding query parameters to expand related resources, or creating purpose-built endpoints for specific screens.

All of these workarounds create tension: the API becomes less general as it's optimized for specific client needs.

No Native Real-Time Support

REST is request-response. The client initiates every interaction. The server can never proactively push data.

Real-time features like live notifications, collaborative editing, and streaming data require either polling (inefficient), long-polling (complex), or a separate real-time technology bolted alongside the REST API.

The Documentation Drift Problem

A REST API contract lives in documentation. Nothing in the HTTP protocol enforces that the documentation accurately reflects the API's actual behavior. As APIs evolve, documentation falls behind. Fields are renamed, types change, endpoints are deprecated. Clients built against outdated documentation break.

This isn't a theoretical problem. It's a daily reality in engineering teams where the backend and frontend evolve at different speeds.

GraphQL: Letting the Client Decide

GraphQL was developed at Facebook starting in 2012 and open-sourced in 2015. Facebook built it to solve a specific problem: their mobile app needed to fetch complex, interconnected social data from a REST API, and the resulting overfetching and multiple round trips were degrading performance on mobile devices.

GraphQL's core insight is simple and radical: instead of the server deciding what data to return, let the client specify exactly what it needs.

The Query Language

GraphQL is both a query language for APIs and a runtime for executing those queries. Rather than calling different endpoints for different data, all GraphQL requests go to a single endpoint (typically /graphql) and include a query that describes precisely what data is needed.

A GraphQL query for a user profile screen:

query UserProfile {
  user(id: "usr_123") {
    name
    avatarUrl
    recentOrders(limit: 3) {
      id
      total
      status
      createdAt
    }
    notificationCount
  }
}

The response contains exactly and only the fields requested. Nothing more. If the client needs only name and avatarUrl, it requests only those two fields. The response contains only two fields.

Mutations and Subscriptions

GraphQL has three operation types:

1. Queries fetch data. They're the GraphQL equivalent of GET requests.

2. Mutations modify data: creating, updating, or deleting resources. They're the GraphQL equivalent of POST, PUT, PATCH, and DELETE.

3. Subscriptions establish a persistent connection and push data in real-time when specified events occur. A subscription to orderStatusChanged receives a push every time any order's status changes. This is GraphQL's real-time capability, typically implemented over WebSockets.

The Schema

Every GraphQL API is defined by a schema written in the Schema Definition Language (SDL). The schema declares every type, query, mutation, and subscription the API supports.

type User {
  id: ID!
  name: String!
  email: String!
  orders: [Order!]!
  notificationCount: Int!
}

type Order {
  id: ID!
  total: Float!
  status: OrderStatus!
  createdAt: String!
}

enum OrderStatus {
  PENDING
  PROCESSING
  SHIPPED
  DELIVERED
}

type Query {
  user(id: ID!): User
  orders(userId: ID!, limit: Int): [Order!]!
}

type Mutation {
  createOrder(userId: ID!, items: [OrderItemInput!]!): Order!
}

The schema is introspectable: clients can query the schema itself to discover what types and operations are available. This enables powerful tooling: GraphQL IDEs can autocomplete queries, validate them against the schema before sending, and display documentation inline.

Where GraphQL Wins

Precise data fetching: Clients request exactly what they need. Overfetching is eliminated by design.

Single round trip for complex data: Data from multiple resources is fetched in a single request. The N+1 problem is solved at the query level rather than requiring the client to make multiple requests.

Strongly typed schema: The schema is the contract. Clients can validate their queries against it at build time. Type mismatches are caught before deployment.

Frontend agility: Frontend teams can evolve their data requirements without asking backend teams to create new endpoints. New screens, data combinations, and features are all handled by writing a new query.

Excellent tooling: GraphiQL and Apollo Studio provide interactive schema exploration, query building, and performance analysis.

Where GraphQL Struggles

Query complexity: A malicious or poorly written query can request enormous amounts of nested data. A query that fetches every user, each user's orders, each order's items, and each item's product details can bring a server to its knees.

REST endpoints can be individually optimized. GraphQL requires query complexity analysis, depth limiting, and rate limiting to protect the server.

Caching is harder: REST GET requests are cacheable at the HTTP level by default. GraphQL queries all go through POST requests to a single endpoint, breaking standard HTTP caching. Clients must implement their own caching (Apollo Client does this), but CDN-level caching is essentially unavailable for dynamic queries.

Over-engineering simple APIs: If your API is straightforward CRUD operations with no complex data relationships and no mobile clients with aggressive data constraints, GraphQL's added setup cost exceeds its benefit.

Real-time at scale is complex: GraphQL subscriptions work, but scaling WebSocket connections for thousands of concurrent subscribers is infrastructure-intensive and requires careful architecture.

Error handling is non-standard: A GraphQL request can partially succeed: some fields resolve successfully while others fail. The response includes both data and errors simultaneously. Handling this gracefully requires more nuanced error handling logic than a simple HTTP status code.

WebSockets: When HTTP Is Not Enough

HTTP, in all its versions, is fundamentally request-response. The client speaks first. The server responds. The conversation ends. Even with HTTP/2's server push, the client initiates every new exchange.

But some applications genuinely need both sides to be able to speak at any moment, without waiting for the other to ask first. For example, a chat application where both parties send messages freely. A live collaborative document where every keystroke is broadcast to co-editors. An online game where the server pushes state updates as they happen and the client sends actions continuously.

For these cases, WebSockets provide a fundamentally different communication model.

The WebSocket Handshake

A WebSocket connection starts as an HTTP request and then upgrades to a WebSocket connection. This upgrade mechanism means WebSockets work through existing HTTP infrastructure (firewalls, proxies, load balancers) without requiring special configuration.

The upgrade request:

GET /chat HTTP/1.1
Host: api.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

The server confirms the upgrade:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

Status code 101 means "Switching Protocols." From this point forward, the HTTP connection is replaced by a WebSocket connection. The protocol has changed. HTTP headers, status codes, and methods no longer apply.

Full-Duplex, Persistent Communication

The WebSocket connection is:

• Full-duplex: both the client and server can send messages at any time, simultaneously, without waiting for the other to finish.

• Persistent: the connection stays open until explicitly closed by either party or until a network interruption occurs.

• Low overhead: once established, WebSocket messages have minimal framing overhead compared to HTTP. A small WebSocket message may have only 2 to 10 bytes of overhead, versus potentially hundreds of bytes of HTTP headers.

WebSocket connection open

Client: "Hello, I'm user 123"
Server: "Welcome, user 123"
Server: "User 456 just sent you a message: Hey!"
Client: "Thanks, here's my reply: Hi there!"
Server: "New notification: your payment was confirmed"
Client: "Great, show me my balance"
Server: "Your balance is NGN 500,000"
Server: "Another notification: transfer from user 789 received"

[Both sides communicate freely, at any time, simultaneously]

Where WebSockets Win

True real-time bidirectional communication: Applications where both client and server need to send messages at unpredictable times and at high frequency. For example, chat, live collaboration, multiplayer games, financial trading terminals.

Low-latency messaging: Once the connection is established, message round-trip times can be in the single-digit milliseconds, limited only by network latency rather than connection setup overhead.

Native browser support: The WebSocket API is built into every modern browser. No libraries are needed for the fundamental connection.

Event-driven architecture on the client: WebSocket events (message, close, error) map naturally to event-driven client code.

Where WebSockets Struggle

Stateful connections: Each WebSocket connection must be maintained by a specific server instance. When scaling horizontally, a client connected to Server A can't receive messages from Server B without a shared pub/sub layer (like Redis) that all server instances publish to and subscribe from. This adds infrastructure complexity.

No built-in request-response correlation: WebSockets are a message stream. If you send a message and expect a response, there's no built-in mechanism to correlate which response corresponds to which request. You have to build this yourself.

No schema or contract: WebSockets send raw text or binary. The format of messages is defined entirely by the application. Two systems communicating over WebSockets must agree on message format out of band, in documentation, and there's nothing to enforce it at the connection level.

Firewall and proxy complications: Some corporate networks and older proxies don't support the HTTP upgrade mechanism correctly, breaking WebSocket connections. This is less common than it was but still occurs in enterprise environments.

Reconnection must be handled manually: WebSocket connections can drop due to network instability. Applications must implement reconnection logic, including managing state across reconnections.

Server-Sent Events: The Simpler Real-Time Option

Between REST's pure request-response and WebSocket's full bidirectional communication lies a middle option that most developers overlook: Server-Sent Events (SSE).

SSE establishes a one-directional persistent connection: the server pushes data to the client over a regular HTTP connection, and the client listens. The client can't send data back through the same connection.

How SSE Works

The client makes a standard HTTP GET request with an Accept: text/event-stream header:

GET /notifications HTTP/1.1
Host: api.example.com
Accept: text/event-stream
Authorization: Bearer token123

The server responds with a 200 OK and keeps the connection open, periodically sending events:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

data: {"type": "balance_update", "balance": 500000}

data: {"type": "transaction", "id": "txn_001", "amount": -5000}

event: notification
data: {"message": "Your transfer has been confirmed"}

id: 42
data: {"type": "order_status", "status": "shipped"}

Each event is separated by a blank line. Events can include a data field, an optional event type, and an optional id for resumability.

Automatic Reconnection

One of SSE's most practical features is automatic reconnection. If the connection drops, the browser automatically reconnects, sending the last received event ID in a Last-Event-ID header. The server can resume from that point, ensuring no events are missed.

Where SSE Wins

Simplicity: SSE works over plain HTTP. There's no protocol upgrade needed, and no special infrastructure. It works through every HTTP/2 connection, load balancer, and CDN that supports streaming.

Native browser support: The EventSource API is built into every modern browser. Automatic reconnection is built in.

Perfect for one-directional feeds: Live dashboards, notification streams, news feeds, real-time analytics, server logs: any scenario where the server pushes a continuous stream of updates and the client only reads.

HTTP/2 multiplexing: Over HTTP/2, multiple SSE connections can share a single TCP connection. The browser connection limit that affected SSE over HTTP/1.1 doesn't apply.

Natural fit for existing infrastructure: SSE responses are just HTTP responses. Existing load balancers, authentication middleware, and monitoring tools work without modification.

Where SSE Struggles

One direction only: The client can't send data back through the SSE connection. For bidirectional scenarios, SSE isn't sufficient on its own.

Text only (natively): SSE events are text. Binary data must be base64-encoded, adding overhead.

No native support in all environments. SSE is a browser API. In other environments (mobile apps, server-to-server), it requires an HTTP client configured to handle streaming responses.

SSE vs WebSockets: The Decision

Choose SSE when the server pushes data and the client only reads: notifications, live feeds, dashboards, or streaming responses from an AI model. SSE is simpler, works over plain HTTP, and has automatic reconnection built in.

Choose WebSockets when both the client and server need to send messages freely and simultaneously: chat, collaborative editing, and games. The added complexity of WebSockets is justified when you genuinely need bidirectional communication.

Protocol Buffers: A New Language for Data

Protocol Buffers (protobuf) is a binary serialization format developed by Google. Where JSON encodes data as human-readable text, protobuf encodes data as compact binary. This single difference has cascading implications for payload size, parsing speed, type safety, and schema enforcement.

The Schema-First Approach

Unlike JSON, where you simply start writing key-value pairs, protobuf requires defining a schema first. You describe your data structures in a .proto file using Protocol Buffer Language, a language-agnostic schema definition language.

The schema definition:

syntax = "proto3";

message User {
  string id = 1;
  string name = 2;
  string email = 3;
  double balance = 4;
  bool is_verified = 5;
  int32 kyc_level = 6;
}

message Order {
  string id = 1;
  string user_id = 2;
  double total = 3;
  string status = 4;
  int64 created_at = 5;
}

Each field has a name and a type, as in any structured data format. But it also has a field number: the small integer after the equals sign. This field number is the key to protobuf's efficiency.

Binary Encoding: Why Field Numbers Matter

When protobuf encodes data to binary, field names don't appear in the output. Instead, only the field number and the encoded value are written. Field 1 (id) becomes a tag byte indicating "field 1, type string" followed by the string's length and bytes. Field 4 (balance) becomes a tag byte indicating "field 4, type 64-bit float" followed by eight bytes of IEEE 754 double-precision float.

No "id": string, "balance": string, quotation marks, colons, or braces. Just field tags and values in a compact binary stream.

The same user object that occupies approximately 100 bytes in JSON occupies approximately 35 bytes in protobuf. For a 1000-field enterprise API response called millions of times per day, this difference translates directly to reduced bandwidth consumption and infrastructure cost.

Parsing binary is also fundamentally faster than parsing text. A binary parser reads a fixed-length tag, determines the type and length of the following value, reads that value, and moves to the next field. A JSON parser must tokenize a text stream character by character, handle escape sequences, infer types from value format, and construct a dynamic object from parsed key-value pairs.

On constrained devices or in high-throughput server-to-server communication, this parsing speed difference is meaningful.

Code Generation: The Contract Comes Alive

The .proto schema file is the input to the protoc compiler. This compiler generates data classes in any supported language from the same schema definition.

The same user.proto file generates:

• A User class in Go for the backend server

• A User class in Dart for the Flutter client

• A User class in Python for the data processing service

• A User class in TypeScript for the web frontend

Every generated class has typed fields, serialization/deserialization methods, and equality comparison. There's no manual JSON parsing, type casting, or risk of field name typos. The compiler guarantees that every language's representation of a User is identical.

When the schema changes — a new field is added or a field is removed, for example — every team regenerates their classes. If the change is breaking (a required field removed or a type changed in an incompatible way), the compiler reports errors in every affected codebase. The problem is caught before any code reaches production.

Schema Evolution Rules

Protobuf's field number system enables backward-compatible schema evolution. Because fields are identified by number rather than name, the following changes are safe:

• Adding a new field with a new number is always safe. Existing clients ignore fields they don't recognize. New clients receive the new field.

• Removing a field by marking it as reserved is safe. Existing encoded data that contains the removed field is simply ignored when decoded. The field number must be marked reserved to prevent its reuse.

• Renaming a field is safe. Names aren't encoded. Only the number matters at the binary level.

• Changing a field's type in incompatible ways is unsafe and breaks existing encoded data.

This evolution model means protobuf schemas can grow over time without coordinated updates across all clients and servers.

Trade-offs

Protobuf's efficiency comes with costs that make it inappropriate for all contexts.

Binary data isn't human-readable. You can't open a protobuf response in a browser's developer tools and see what it contains. Debugging requires either decoding the binary with the schema or using specialized tools.

Protobuf also requires tooling. Every consumer of a protobuf-encoded API needs the schema and a protobuf library to decode it. For public APIs consumed by unknown third parties, this is a significant barrier. JSON requires nothing: every programming environment can parse it with built-in libraries.

Schema changes require coordination. When a schema changes, every consumer must update. For internal systems where you control all consumers, this is manageable. For public APIs, it requires versioning and migration strategies.

gRPC: Remote Procedure Calls at Scale

gRPC combines Protocol Buffers with HTTP/2 and Remote Procedure Call semantics to produce a framework for service-to-service communication that is faster, more structured, and more powerful than REST for specific use cases.

Remote Procedure Calls: The Core Concept

A Remote Procedure Call (RPC) framework makes calling a function on a remote server feel like calling a local function. Instead of constructing an HTTP request, serializing a body, parsing a response, and handling status codes, you call a function with typed arguments and receive a typed return value. The network communication is abstracted away.

// Without RPC (manual REST)
const response = await http.post('/users', headers: {...}, body: json.encode(data));
const user = User.fromJson(json.decode(response.body));

// With RPC (gRPC)
final user = await userService.createUser(CreateUserRequest(name: "John", email: "john@example.com"));

The second form is simpler, type-safe, and requires no knowledge of HTTP methods, endpoints, or serialization formats.

The Four Communication Patterns

gRPC's most significant advantage over REST is its support for four distinct communication patterns, all defined in the same .proto schema and accessible through the same generated client.

Unary RPC is the familiar request-response pattern. One request and one response. It's equivalent to a REST API call.

Client ——— LoginRequest ——→ Server
Client ←—— LoginResponse —— Server

Server Streaming RPC sends one request and receives a continuous stream of responses. The server pushes messages as they become available without the client needing to request each one.

Client ——— WatchBalanceRequest ——→ Server
Client ←— BalanceResponse ———————— Server (balance: 500,000)
Client ←— BalanceResponse ———————— Server (balance: 495,000)
Client ←— BalanceResponse ———————— Server (balance: 1,000,000)
[Stream stays open, server pushes on every change]

Client Streaming RPC sends a stream of messages to the server and receives one response at the end. The server processes all received messages and responds once.

Client ——— DocumentChunk 1 ——→ Server
Client ——— DocumentChunk 2 ——→ Server
Client ——— DocumentChunk 3 ——→ Server
Client ←————— UploadResponse —— Server (all chunks processed)

Bidirectional Streaming RPC allows both client and server to send streams of messages simultaneously, in any order.

Client ——— ChatMessage ——→ Server
Server ←— ChatMessage ——— Client
Client ——— ChatMessage ——→ Server
Server ←— ChatMessage ——— Client  (server-initiated)
[Both sides communicate freely and simultaneously]

Why HTTP/2 and Protobuf Make gRPC Efficient

gRPC's efficiency comes from the combination of its two underlying technologies working together.

HTTP/2's multiplexed persistent connections mean many concurrent gRPC calls, including long-running streaming calls, share a single connection. There's no connection setup overhead per call. Multiple streams proceed in parallel without blocking each other.

Protocol Buffer's binary encoding means payloads are compact and parsing is fast. A high-frequency service-to-service call that would transmit 100 bytes of JSON transmits 35 bytes of protobuf. At thousands of calls per second between microservices, this difference is significant.

The generated clients eliminate all serialization and deserialization code. The schema enforces that client and server agree on the contract. Breaking changes are caught by the compiler.

The Organizational Contract

In organizations using gRPC at scale, .proto files live in a dedicated repository separate from any individual service. This repository is the single source of truth for every service contract.

When an engineer wants to add a new field to an API, they open a pull request in the proto repository. Engineers from every affected team review it. The change is discussed, refined, and approved before any implementation begins. When it merges, every team regenerates their clients. Changes that break existing behavior are caught in code review, not in production.

This governance model transforms API evolution from a coordination problem into a code review process.

gRPC's Limitations

gRPC doesn't work natively in web browsers. Browsers can't directly make HTTP/2 requests with the necessary control required for gRPC. A proxy layer (gRPC-Web) is required to translate between gRPC-Web's browser-compatible format and standard gRPC. This adds infrastructure complexity and limits gRPC's applicability for browser-based clients.

gRPC also requires HTTP/2. Environments that don't support HTTP/2 can't use gRPC.

Binary encoding makes debugging harder as well. Inspecting gRPC traffic requires specialized tools and access to the proto schema.

For public APIs consumed by third-party developers, gRPC's tooling requirements are a higher barrier than REST's universally accessible JSON over HTTP.

The Complete Comparison

How to Choose: The Engineering Decision Framework

No single communication approach is universally best. Each exists because it solves specific problems better than the alternatives. The engineering decision involves matching the tool to your requirements.

When to Use REST

Use REST when the API is public or consumed by third parties. REST's universal accessibility makes it the only reasonable choice for public APIs. Any developer in any language can call a REST API with standard HTTP tools. There are no schema files, generated clients, or special libraries.

REST is also a good fit when caching is a priority. REST GET responses can be cached at every layer: CDN, reverse proxy, and browser. For content that doesn't change frequently, REST with proper cache headers can serve millions of requests without hitting the origin server.

It's also solid when the operation is simple request-response. If you're building straightforward CRUD operations with no streaming requirements and no complex data relationships, REST is simpler to implement, document, and debug than any alternative.

And finally use REST when developer experience for the consumer matters. REST APIs are immediately accessible in a browser. They can be tested with curl. Every developer already understands them.

When to Use GraphQL

Use GraphQL when multiple client types have significantly different data needs. A mobile app that needs minimal data for a list view and richer data for a detail view, alongside a desktop app that needs comprehensive data, are ideal GraphQL consumers. Each queries exactly what it needs.

GraphQL also works well for complex interconnected data with many relationships. Social graphs, product catalogs with deeply nested attributes, or content management systems with rich content relationships: GraphQL's ability to traverse relationships in a single query is a genuine advantage.

It's also a good choice for frontend teams that need to iterate quickly. When the frontend can evolve its data requirements without backend changes, development velocity increases. New screens, new data combinations, no new endpoints needed.

And finally, GraphQL works well if you're comfortable with the operational complexity. GraphQL requires query complexity protection, custom caching strategies, and more sophisticated error handling. These are worth the effort when the data fetching advantages are real.

When to Use WebSockets

Use WebSockets when both the client and server need to send messages at any time. Genuine bidirectional real-time communication where either party can initiate a message at any moment.

WebSockets also work great for chat, collaboration, and games. Live chat applications, collaborative document editing, multiplayer real-time games are the canonical WebSocket use cases.

And WebSockets is a solid choice when low-latency messaging is critical. The minimal framing overhead and persistent connection make WebSockets the lowest-latency option for frequent message exchange.

When to Use Server-Sent Events

Use SSE when the server needs to push updates but the client only reads. Notification feeds, live dashboards, streaming AI responses, real-time analytics, or any scenario where the server has a continuous stream of data to deliver and the client only consumes.

SSE also works well when you value simplicity over full bidirectionality. SSE is significantly simpler to implement and operate than WebSockets for one-directional use cases. Automatic reconnection is built in. It works over plain HTTP.

When to Use gRPC

Use gRPC when multiple internal services share the same contract. When several teams build services that call each other, a .proto schema enforced by the compiler prevents contract drift. Everyone generates their clients from the same source of truth.

gRPC also works well for high-frequency service-to-service communication. Two microservices exchanging thousands of calls per second benefit from protobuf's compact binary encoding and HTTP/2's persistent multiplexed connections.

It's also a solid choice for large payloads that are consumed by many internal systems. An internal enterprise API with hundreds of fields called by dozens of internal applications benefits enormously from protobuf's size reduction. Less bandwidth, less parsing overhead, and compiled contract enforcement.

gRPC also works great when low-bandwidth networks matter. For mobile applications in markets where network conditions are variable or constrained, protobuf's binary encoding reduces payload size by 3 to 10 times compared to JSON. The difference between a 15 kilobyte response and a 3 kilobyte response is the difference between a 3-second load and a sub-second load on a 2G connection.

And finally, use gRPC when streaming is a core requirement and you want one framework. gRPC's four communication patterns (unary, server streaming, client streaming, and bidirectional) cover every scenario without requiring separate WebSocket infrastructure alongside your API.

The Hybrid Reality

Most sophisticated systems use multiple approaches, each where it genuinely wins:

A Large Engineering Organization

Public REST API
  External developers, partners, open integrations
  JSON over HTTPS. OpenAPI documentation.
  CDN caching for frequently accessed resources.

Internal gRPC Network
  Service-to-service communication
  Auth service, payment service, notification service,
  fraud detection: all communicating with typed contracts
  over efficient binary protobuf on HTTP/2.

Real-Time Layer
  WebSockets for bidirectional features (live chat, collaboration)
  SSE for one-directional feeds (notifications, live dashboards)
  gRPC streaming for real-time data with typed contracts

Mobile API
  REST for standard operations (profile, settings, history)
  gRPC for high-frequency or large payload calls
  SSE for notification streaming

There's no architectural purity requirement. Each layer uses what fits its requirements. The discipline is in making these choices deliberately rather than by habit or default.

Conclusion

The history of how clients and servers communicate is the history of engineers discovering the limitations of existing tools and building better ones.

HTTP/1.1 gave us a universal request-response protocol that built the web. Its text-based format and sequential connection model worked well for the web of the 1990s and 2000s. As applications became more complex and performance expectations rose, its limitations became bottlenecks.

HTTP/2 rebuilt the transport layer with binary framing and multiplexing, eliminating head-of-line blocking at the HTTP level, compressing headers, and enabling server push. HTTP/3 took this further by replacing TCP with QUIC, addressing the remaining head-of-line blocking at the transport level and making connection establishment faster.

JSON became the dominant data format because of its human readability and universal support. Protocol Buffers emerged as an alternative for contexts where JSON's verbosity and lack of schema enforcement create real problems: internal services, high-frequency communication, constrained networks, and teams needing compile-time contract enforcement.

REST codified HTTP's architectural strengths into a style that made APIs universally accessible and HTTP-native. Its success wasn't purely technical: it aligned with what developers already understood and what the HTTP ecosystem already supported. Its limitations in data fetching efficiency and real-time communication opened the door for GraphQL and streaming alternatives.

GraphQL solved REST's overfetching and underfetching problems by inverting control: the client specifies exactly what it needs. WebSockets solved REST's inability to support genuine bidirectional real-time communication. Server-Sent Events provided a simpler real-time option for one-directional streaming. gRPC combined Protocol Buffers, HTTP/2, and RPC semantics into a framework that excels at typed service-to-service communication at scale.

Understanding all of these tools, along with why each was built, what problem it solves, and where it struggles, is what enables you to make deliberate architectural decisions rather than defaulting to whatever is most familiar.

The right communication approach is always the one that fits the specific requirements of the system you're building: the clients consuming it, the data being exchanged, the network conditions it operates in, the teams building and maintaining it, and the operational complexity you are prepared to manage.

That clarity of fit is what engineering judgment looks like in practice.

In fact, it is thanks to firmware that:

• Printers work
• Defibrillators work
• Car radios works
• and more …

Based on the examples above, you probably already have some idea of what firmware is. But you don't have a clear definition of it.

In order to truly understand what firmware is, we must first understand software, then hardware, and finally we can jump into firmware.

In this tutorial, I'll explain each topic with an analogy. By doing so, everyone can understand.

It doesn't matter if you're just getting into technology!

With that, I would like to make sure that everyone understands what software, hardware, and firmware is. I will not elaborate on the technical terms.

In this article we will explore:

• What exactly is software?
• What exactly is hardware?
• What exactly is firmware?

What is Software?

_Photo by [Pexels](https://www.pexels.com/@kubra-dogu-80605500?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels" rel="noopener">Kübra Doğu from <a href="https://www.pexels.com/photo/food-wood-dawn-coffee-9222655/?utm_content=attributionCopyText&utm_medium=referral&utmsource=pexels" rel="noopener)

Imagine, if you will, that you have no idea what to make for dinner today.

Luckily, you find an old cookbook you have and decide to cook one of the recipes in it.

The cookbook has many recipes. Each of these recipes has its own instructions.

In the cookbook, a recipe can be seen as a set of steps (or instructions) that together make a meal.

Photo by Luis Quintero on Pexels

You can also create your own meals based on your experience in cooking many different recipes, right?

Software is no different.

Software programs can be seen as a set of instructions that work together to form a program.

 global  _main
    extern  _printf

    section .text
_main:
    push    message
    call    _printf
    add     esp, 4
    ret
message:
    db  'Hello, World', 10, 0

Then, applications are a big sets of instructions that perform specific tasks.

Operating systems are big sets of instructions that coordinate software and hardware resources.

• Cookbook = software
• Dinner cookbook = type of software (application or operating system)
• Recipe = program

You need a recipe to make dinner. You must follow each step in the cookbook to create a recipe.

Once you have completed all the steps, your dinner is ready.

You need software to accomplish a particular task. A computer has to follow all instructions for the software to function.

Therefore, the software is running either while the instructions are being followed or after they have been completed.

What is Hardware?

_Photo by [Pexels](https://www.pexels.com/@elevate?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels" rel="noopener">ELEVATE from <a href="https://www.pexels.com/photo/chef-preparing-vegetable-dish-on-tree-slab-1267320/?utm_content=attributionCopyText&utm_medium=referral&utmsource=pexels" rel="noopener)

In order to make dinner, you need a series of steps from the cookbook that tell you how to make a particular meal.

You also need various tools to cook with - like pots and pans, knives, and the food itself. This is like hardware.

So a cookbook gives you instructions that allow you to cook.

For hardware to work, it needs software (a set of instructions) to tell it what to do.

Software then, gives instructions to hardware that lets it work.

• Meal prep tools = hardware
• Recipe = software

Without software, you can't make hardware work.

Without a recipe, you won't know what to do with your various tools and ingredients to make a meal.

Like hardware without software, you can also eat a meal with just uncooked food. Gross.

In the same way that there are various types of meals, there are also various types of computer hardware.

For example:

• CPU
• RAM
• GPU
• and so much more…

Comparison of processes

What is Firmware?

_Photo by [Pexels](https://www.pexels.com/@ella-olsson-572949?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels" rel="noopener">Ella Olsson from <a href="https://www.pexels.com/photo/close-up-photo-of-chocolate-mousse-3026810/?utm_content=attributionCopyText&utm_medium=referral&utmsource=pexels" rel="noopener)

A program is a set of instructions read by a computer.

Let's say you just want to make a snack or dessert. You probably don't need as many ingredients as when making a dinner for your family, right?

Let's say you want software that runs on a microwave. You don't need all the hardware the computer has to make the microwave work, right? You just need that specific to making the microwave work.

Or say you want software running on a printer. You don't need all the hardware the computer has to make the printer work right? Just the software for the printer.

• Microwave software = firmware
• Printers software = firmware

So this means that firmware is nothing more than software, but in a hardware device. Not in a computer.

Firmware lets very specific hardware complete very specific tasks.

Wrapping up

Thanks for reading! Now you know more about:

• Software
• Hardware
• Firmware

Microprocessor photo by Pok Rie from Pexels

Learn the answers to these questions and more from a group of experienced developers and technical course creators.

We just published a video on the freeCodeCamp.org YouTube channel where experienced developers answer the top questions asked by people new to the software field.

You will get wisdom from the following people:

• Gerald Versluis, Senior Software Engineer at Microsoft
• Mama Samba Braima Nelson, Software Engineer
• Gwendolyn Faraday, Principal Software Engineer at Relevize
• Ania Kubow, Software Developer and Course Creator
• Marius Espejo, Senior Software Engineer
• Ave Coders, ML Engineer
• Jack Herrington, Principal Web Developer
• Nana Janashia (TechWorld with Nana), DevOps Engineer & Trainer
• Gift Egwuenu, Frontend Developer & Content Creator
• Jesse (codeSTACKr), Developer Advocate
• James Perkins, Developer Advocate
• Gio (ProgramWithGio), Full-Stack Web Developer
• Delia Lazarescu, Software Engineer at Google
• Islem (CoderOne), Full-Stack Developer and Content Creator

Watch the video below or on the freeCodeCamp.org YouTube channel (1-hour watch).

Software is synonymous with the terms program and application.

Software can describe an operating system like Windows 10 or macOS, or their underlying processes. Software can also describe things like Microsoft Word or Adobe Photoshop.

The term software is often contrasted with hardware.

Computer hardware, is physical or "hard", and is difficult to change once it's produced.

Software runs on computer hardware, and is "soft", or easy to change in comparison to hardware.

Related Tech Terms:

• Hardware Definition
• Macintosh Definition
• PC Definition

By the end of the project, you will code your own LED light, have the knowledge to manipulate the LED to on/off in intervals you choose, and learn basic principles of hardware.

The Elegoo Uno starter kit has all the hardware as well as instructions you need to make a simple LED Light. The LED Light is the first project offered with their kit.

Elegoo Uno comes with many other projects and takes you through from beginner to advanced projects. Each project in the box advances your skills in a simple, easy to follow way.

Components You'll Need

Elegoo Uno R3

Elegoo Uno R3

The Elogoo Uno R3 is a microcontroller board. Microcontrollers are embedded inside devices to control the actions and features of a product. They are compact integrated circuits designed to control operations.

The microcontroller that is included in the Elogoo Uno R3 has 14 digital input/output pins, 6 analog inputs, a USB connection, a power jack, and a reset button. This board has everything you need to support the microcontroller. Simply plug in the USB Cable to turn the microcontroller on.

USB Cable

USB Cable

You need a USB Cable to connect the Elegoo Uno R3 to your computer and turn it on. USB stands for Universal Serial Bus. The USB is used to connect your computer to devices such as digital cameras, printers, scanners, and external hard drives.

In our project, we will use a USB cable to connect our microcontroller to our computer.

LEDs

LED Light

LED stands for light emitting diode. It has a positive and negative lead. The longer side is the positive lead.

How to assemble the components

In this project, we are only going to make the LED blink.

First, we need to plug in the USB Cable to the board and then the computer.

Then we need to plug in the LED to GND (GND is the reference point in an electrical circuit from which voltages are measured, and is a common return path for electric current) and the 13 input on the board.

Code to make the LED flash on/off:

After the microcontroller board is plugged into the computer and the LED is on the board itself, we need to write some simple code to make the LED blink.

// the setup function runs once when you press reset or power the board

void setup() {
  // initialize digital pin LED_BUILTIN as an output.
  pinMode(LED_BUILTIN, OUTPUT);
}

// the loop function runs over and over again forever
void loop() {
  digitalWrite(LED_BUILTIN, HIGH);   // turn the LED on 
  delay(1000);                       // wait for a second
  digitalWrite(LED_BUILTIN, LOW);    // turn the LED off
  delay(1000);                       // wait for a second
}

The code above basically turns the LED on for 1 second and then turns it off for one second.

This function is in a continuous loop. The digitalWrite is a function that takes in 2 parameters, LED_BUILTIN and HIGH || LOW . The loop basically takes in the LED, and then turns the volt to HIGH which turns it on. Then after 1 second it turns the same LED off by turning the volt to LOW .

Here's the Final Product:

The goal of this little LED Light Coding project was to introduce you to elementary principles of how hardware and software can be combined. I hope you enjoyed it!

Kanban was invented in the Japanese automotive industry in the first half of the 20th century. Inspired by how supermarkets stock their shelves based on demand, Toyota's goal was to reduce inventory and to improve the flow throughout their whole production system.

In his book Kanban: Successful Evolutionary Change for Your Technology Business, David Anderson described how to apply the Kanban principles to software development. These principles are:

• Start with what you do now
• Agree to pursue incremental, evolutionary change
• Respect the current process, roles, responsibilities, and titles

What does that mean for agile software development?

In my training courses, I ask the attendees what they already know about agile software development. Common answers are: "Work in Sprints", "There's a product owner", "Manage user stories in a backlog." People are influenced by the arguably most popular agile framework today, Scrum.

Scrum comes with its own predefined roles, events and artifacts. Scrum requires you to follow the rules defined in the Scrum Guide, if you want to call what you're doing Scrum. Kanban is different.

Kanban starts with the process that you follow in your company now. Visualize the steps on a Kanban board. They can include everything you do, from idea to delivery.

Each step becomes the title of a column on the board.

To track your day to day work, it's best to break it down into small items. Maybe user stories that can be implemented in, at most, 2 days. Write each item on a stickie note, and hang it on the board. You can use the vertical order on the board for prioritization.

The cards move from left to right. The people doing the work pull items that have been finished by the previous process step. When they have capacity to do so. So the developers in the example pull the Upload Image card into Dev when they have the capacity to implement it.

Pursuing incremental, evolutionary change

So you have created a Kanban board that shows your process? You are making your work visible, which is a great start!

To get the benefits of Kanban, you need to do some more things. You need to:

• Limit work in process and queues
• Observe and improve flow
• Collaborate effectively

Limit work in process and queues

Limiting work in process means: you set a maximum for the number of items you work on. That's called the Work-in-Process limit, or in short WiP limit. Here's the Kanban board with a WiP limit for some process steps.

The developers can work on 5 items at a time. At most. If their column contains 5 items, they are not allowed to pull items any more. This has two consequences.

First of all: it encourages people to finish their work, instead of starting more work. Started work that isn't finished bears risks. How happy will your customers be when you can't release your software as planned? Because you started to work on all these great ideas, but you didn't go through with them?

The second consequence of limiting work in process: bottlenecks become visible. When a process step starts work that it can't finish, people will feel it immediately. Because the next process step won't be able to pull items.

Apart from limiting the work in process, you should limit queue sizes, too. In the board above, they are shown as the dotted lines between the columns. It works the same way as limiting the work in process.

In summary: Stop starting, start finishing is the motto. From concept to delivery in as little time as possible.

Observe and improve flow

Observing a bottleneck can be painful at first. But at least you know where the major problems are in your process. And Kanban encourages you to improve the flow by removing the bottleneck. A consistent flow enables you to deliver more reliably, and that's good for all stakeholders, including developers.

To observe the flow, you record the time at which a card enters a process step. And the time at which you complete the process step. So you know how much time the card spends in each step, and in each queue between the steps.

Based on the data, you can set up metrics that help you to improve the flow. Common metrics include:

• Cycle time: the time a card takes from the moment a team starts working on it (i.e. Dev) to delivery (i.e. Release). Improving this metric can help you improve your time to market.
• Throughput: the number of cards that move through the system in a given time. Improving this metric can help you improve the performance of your delivery organization.

A common way to get an overview of how many cards are in which process step over time is a cumulative flow chart. Ideally, the number of cards in each step but the last stays roughly the same over time. The number of released cards should mount. When the chart deviates from that, you might have a bottleneck.

Collaborate effectively

The Kanban metrics are a powerful tool to analyze and improve what you're doing. But they are worthless without the people doing the work. Everybody involved in the process steps should be open to the transparency that Kanban creates.

People should work together constructively to remove bottlenecks, instead of blaming individuals. Look at the current state regularly. Are there any bottlenecks? Is there too much or too little work available for a certain process step? Is throughput sufficient? Are there other sources of discontent? What needs to be improved?

Agree on experiments that try out small changes to the system. Realize the changes. Later, look if the experiments worked out as expected. To be able to implement the changes, management support is often crucial.

When to use Kanban

Kanban is very flexible. It can be used in combination with Scrum, which is called Scrumban. It can be used outside of product development. You can even use it to plan a trip or organize what you do in your free time.

I found it especially helpful when working in Scrum Sprints is not possible or is difficult. Example: two companies where one is customer, and the other is supplier, and a hand-off is inevitable. Another example is when you're working on a product that involves both software and hardware, and multiple engineering disciplines are involved.

Kanban can also be used inside your company when development works in an agile fashion, but not all of the rest of the company does. It can be used to facilitate the cooperation between strategic planning and software development.

Don't think that just because you have trouble implementing Scrum, there's no way to become more agile. Kanban starts with what you do now. And if you take it seriously, it will help you improve. One small step at a time.

To learn more about agile software development, visit my online course. To keep up with what I’m doing or drop me a note, follow me on dev.to, LinkedIn or Twitter. Or visit my GitHub project.

• What is agile?
• What are some concepts you should know?
• Stories
• Epics
• Sprints

What is agile?

Agile is a software development methodology that stems from the idea of breaking up large amounts of work into smaller pieces. This gives product managers, developers, and any stakeholder a better understanding of the work.

Historically, software development was a slow process where major changes to requirements could put large strains on teams.

When following the agile methodology, the smaller chunks of work help teams become more flexible, and dare I say agile. And in the process it helps them deliver features faster and respond to changes quicker.

Example Jira project board from atlassian.com/software/jira

These ideas have been broken up into different frameworks of approaching this methodology. Two of the common ones are Scrum and Kanban.

For this walkthrough, most of these concepts follow along the Scrum framework, but there are certainly concepts that apply to both and others.

What are some concepts you should know?

I'd argue half of being productive as a developer in an agile world is simply understanding the terms. Typically the project manager runs the show, so if you can be on the same page with what they're talking about, it will make the process a lot easier.

There are books, courses, and certifications based around learning the nuances of the agile methodology. I'm not going to go deep into some of the philosophical aspects or some of the deeper parts, but I'm going to cover a good set of key concepts that will help you hit the ground running when you start your new job with an agile team.

Stories

A story is typically the smallest defined piece of work. This usually comes in the form of a new ticket that you create in the project tool you're using whether it's Jira or even Github Issues.

Expressing stories

When working on a project, you'll probably run into a variety of ways people express stories. But a good guideline is to work through the concept of the word "story" itself and explain the work that needs to be done in that way.

For instance, if you would like to provide the ability for the people who use your website to share a blog post on Twitter, you may want to write the story as: As a reader, I want to share the post I just read to Twitter.

Creating a new story in Jira

Using that pattern of "as a [person], I want to [action]" helps provide context as what state somebody may be in when visiting their site and what they're trying to achieve. This can be particularly helpful if you're developing features for people who are logged in that are different from guests.

Details and requirements

While the title of a story is an important representation of the work, you'll also want to provide additional specifics.

At a minimum, that should be done by adding a thorough description and a set of acceptance criteria that can help give the developer context and requirements. Depending on the team, this can also include tools like tags or categorizations that make it easier for the team to visualize groups of work.

Providing a strong set of requirements helps both the developer working on the story and the person reviewing it have a measurement to determine if it's actually complete. Without it, everyone's just guessing.

A good way to phrase these are: verify [requirement]. Back to my example of sharing a post on Twitter, maybe some of the requirements to that story would be:

• Verify when clicking the share button a new tweet is created
• Verify the tweet includes a link to the current blog post

Amount of work or level of difficulty

Each story is represented by a number of points. Those points are a way to express how much effort a team of developers expects one story to be. That effort can mean a variety of things though whether it's simply how difficult the team expects the work to be or the amount of risk or uncertainty a particular story has.

One way teams represent this is with the fibonacci sequence, where the amount of points can be 1, 2, 3, 5, 8, etc. Where a negligible text update might be 1 point, adding a new form to a page could be 3 points.

Typically you'll want to avoid pointing stories too high, as you get above 5 points, there's more than likely a way you can break up the work to make it more manageable. While you could easily create a massive 13 point story to accomplish all aspects of a feature, it usually makes sense to tackle the work in smaller, more focused chunks.

Either way, these points all add up together to give your team a rough estimate of how much work a group of stories would take to complete.

Epics

While stories have a goal of defining a bite-sized piece of work, epics are a way to group those pieces of work together to represent a feature.

Defining stories as a feature

A good way to explain this is with another example. If you're working on an application that requires the integration of authentication, you may want to create a new epic simply called "Authentication".

Inside that epic, you could find stories like:

• As a guest, I want to sign into the application with my email address
• As an authenticated user, I want to change my password
• As the security team, I want to prevent spam and abuse of user authentication

Example of an Authentication epic in Jira

With your epic defined, you're giving your team a path to calling a feature complete while also understanding the entire scope of that work. This is important when it comes to planning out the work to be done.

Defining your stories in your epic gives you a sense of how much work something takes, but it doesn't help you figure out how long it would take, which is where sprints come in.

Sprints

Sprints are a way of planning out how the work will actually get done. While similar to epics in that they are a way to group chunks of work, sprints typically represent a period of time in which a particular chunk of work will be done.

Time per sprint

A common way of defining a sprint is two weeks of work. During those two weeks, your team will have a particular velocity, or average amount of work you can complete, for an individual sprint. This velocity is represented by a number of points that is a sum of the average velocity of each of the developers working on that sprint.

Points per sprint

Though many fiercely argue you shouldn't use time to represent that velocity, points will roughly translates to an average amount of time of work for each developer. While 1 point for an experienced developer could be 1 hour, that same 1 point could mean 3 hours for a less experienced developer.

Example project roadmap from atlassian.com/software/jira

But once you have this number of points that your team averages in a sprint, you'll know how many story points you can expect to plan to be completed. This planning goes sprint to sprint as you spread out a group of stories or an epic so you can predict when a feature will be complete.

How agile fits with your team

Try to remember that the Agile methodology through Scrum, Kanban, or any other framework is just that – a framework. While it's probably a good idea to follow the process when you first start out, listen to your team and try to mold it to your own experiences.

Each team works a little bit differently and forcing a process onto that team can cause more harm than good, but there will always be a learning curve for any process. Fight the eye rolls until everyone gets the hang of it and have frequent retrospectives to see what works and what doesn't.

At the end of the day, the processes your team follows should mostly be invisible, working for you instead of against you. Find what works best for your team and share your experiences for others to learn!

What's your teams process?

Share with me on Twitter!

Believe it or not, some organizations still rely on legacy software to carry out operations even though newer and more versatile options are available. We know that “old is gold”, but legacy applications cannot glitter forever. As such, these old and outdated programs have become hard to maintain.

Recently, the U.S. Government Accountability Office (GAO) gave a report that pointed out the most critical federal legacy systems that need to be modernized. This is because they are based on antiquated programming languages, are prone to security loopholes, and are difficult to maintain.

Are you in the same shoes?

In this article, I’m going to talk frankly about the challenges you'll face when maintaining legacy software applications, and how you can overcome them.

Legacy systems are inefficient

Software maintenance is important – it helps enhance the efficiency of the product and narrows the margin for error. Without sufficient maintenance, an application can become inefficient and difficult to operate.

First of all, maintaining a legacy system can be difficult because the code used is old when compared to the code used in any modern software. Old code usually tends to be bulky, lengthy, and incompatible with most modern systems.

For example, JavaScript arrow functions, which were introduced in ES6 in 2015, offer developers a way to write shorter and cleaner function syntax, which is easier to maintain.

let total = (x, y) => x + y;

/* This arrow function is a shorter form of:
let total = function(x, y) {
  return x + y;
};
*/

console.log(total(5, 2) ); 
// 7

High maintenance costs

Another challenge facing most legacy systems is the high cost of maintenance, which may be out of reach for most enterprises.

For example, according to the GAO report mentioned above, the U.S. government planned to spend more than $90 billion in 2019 on IT services, most of which went towards maintaining aging systems.

Furthermore, as technology is evolving, compliance is becoming a major issue for the protection of consumer-facing applications. Achieving compliance with legacy systems is time-consuming and expensive. It will not happen as quickly compared to new applications who are often compliant by default. It also requires a lot of testing to ensure a legacy infrastructure is compliant with the given regulations.

The cost of maintaining a legacy system often increases with time, as it slowly becomes obsolete due to technological enhancements. Also, modification of an existing system is a risky venture, which requires a lot of time and resources.

Lack of sufficient skill sets

For legacy software to be maintained, you need a developer conversant with its operations. However, most developers are future-proofing their applications with new technologies. So, getting someone who can work with an old system can be a challenge.

In some cases, you might need to re-train developers on how the legacy system works, which increases a company’s operating costs.

Furthermore, managing and controlling changes occurring in the software can be difficult. A lot of time and effort is required to keep the systems operational, which is expensive and time-consuming.

Incompatibility with other IT solutions

Currently, there are modern tools that can be used to enable fast and smooth maintenance of software. However, most legacy IT infrastructure is incompatible with such solutions, which complicates their maintenance.

If the features in a legacy system are not compatible with those of new IT solutions, developers may find it difficult to integrate them into their environments.

The difficulty of introducing new features to legacy systems also adds to the challenges in their maintenance. Since most legacy systems break easily, trying to restructure and make them more maintainable may not work as expected.

Solutions to the challenges

To compete favorably in today’s dynamic IT landscape, legacy technologies need modernization. Updated legacy applications lead to more user productivity, reduced maintenance costs, and more helpful experiences.

According to recent research by Avanade, modernizing IT systems can lead to a revenue growth of about 14%. Therefore, choosing different software modernization options could lead to significant benefits to your business.

It’s important to note that software should not be declared obsolete just because it’s old. Some of the ‘old’ software may still contain rich features, which can be useful to the optimal functioning of an application.

Therefore, to overcome the challenges of maintaining aging software, developers can opt to refactor the source code of the system. This way, they can use clean and modern code that is reusable and easy to debug.

In refactoring, you alter your software system to enhance its internal structure. But you don't interfere with the external behavior of the code. This way, the features of the software are optimized due to the code's internal improvements.

When refactoring legacy code, updates and modifications should be sufficiently tested to avoid breakages and poor functioning of the application. For example, regression tests can be done to ensure that everything is working as desired.

Additionally, where resources permit, developers can opt to rewrite the entire source code of the software, while employing modern programming approaches.

If you want to continue maintaining your legacy code without breaking it, you can use any of the following three methods:

1. Identifying change points in the code
2. Isolating your code
3. Wrapping the code

Let's talk about each of the methods.

Identify change points in the code

As pointed out earlier, maintaining legacy code can be challenging. Sometimes the issue can be caused by a section that has been poorly programmed. Therefore, you can overcome this by identifying a location that can allow you to change the application's behavior without altering the source code.

For example, let's say you have the following JavaScript code in a legacy application that connects to a database:

export class DataConnection {
     //some code here

  connector() {
    // some code to connect to database
  }
}

If you want to run some tests on the above code but the the connector() method is causing problems, you can identify where to modify the code behavior without affecting the source code.

In this case, you can extend the DataConnection class and stop it from establishing a connection to an actual database:

class FakeConnection extends DataConnection {

    connector() {
    // solve the issues of making calls to DB

    console.log("Establishing a connection")
  }
}

Consequently, after modifying the code behavior without affecting the source code, you can run tests on the code and maintain it without any problems.

Isolate your code

Another technique that can allow you to maintain your legacy code easily is to isolate and make any changes on a different environment. You just need to identify an insertion point where you can call that changed code from the existing legacy code.

Here is an example:

class BooksData {
  // some code here

  addBooks(books) {
    for (let book of books) {
      book.addDate()
    }

    // some code here

    booksRecords.getNumberOfBooks().add(books)
  }

  // some code here
}

Let's say you want to optimize the books reference in the above legacy code, but addBooks() is giving you problems.

So, you can isolate the code in another new method, like newBooks().

Then you can run tests on this new method successfully because it is separate from the rest of the code. Afterwards, you can include a call to the new method in the existing, non-changed code. This way, there'll be minimal changes and minimal risks to the legacy code.

Here it is:

class BooksData {
  // some code here

  newBooks(books) {
    // some smart and testable logic to optimize books
  }

  addBooks(books) {
    const newBooks = this.newBooks(books)

    for (let book of newBooks) {
      book.addDate()
    }

    // some code here

     booksRecords.getNumberOfBooks().add(books)
  }

  // some code here
}

Wrap the code

If you want to make changes that should take place before or after the existing code, wrapping it can also be another solution.

You can achieve this by giving the old method you intend to wrap a new name, adding a new method with the same name and signature just like the old method, and calling the new method from the new method. Lastly, you should place the new logic before or after the previous method call.

With the new logic, you can run tests or make any changes you want – without affecting the source code.

For example, here is the code we used previously:

class BooksData {
  // some code here

  addBooks(books) {
    for (let book of books) {
      book.addDate()
    }

    // some code here

    booksRecords.getNumberOfBooks().add(books)
  }

  // some code here
}

Here is how to solve the problem through wrapping:

class BooksData {
  // some code here

  addBooks(books) {
    // some smart logic to get books
    this.addMoreNewBooks(moreBooks)
  }

  addMoreNewBooks(books) {
    for (let book of books) {
      book.addDate()
    }

    // some code here

    booksRecords.getNumberOfBooks().add(books)
  }

  // some code here
}

Conclusion

Although modernizing antique software is complicated, demanding, and risky, the results are usually worth the risk. Continuing to rely on legacy IT systems is the same as continuing to use the post office to send an urgent message, while an email could do the trick at the click of the button.

Furthermore, as a programmer, do you equip yourself with modern coding skills? Or, do you still rely on old approaches?

For example, in the exciting world of JavaScript programming, we’ve witnessed a surge of frameworks, like React and Angular, which are defining the future of the language. Spending some time learning about them could prevent you from falling into obsoleteness.

Do you agree?

When I decided to teach myself how to code almost four years ago I had never heard of, let alone thought about, the software development life cycle.

As a brand new developer I was focused on learning the technologies that would help me land that coveted first developer job, not the nuances of how those teams operated.

When I did learn of them, I thought they would be useless to me because I considered myself a web developer not a software developer.

I've since learned that this couldn't be further from the truth and these principles/practices play a large role in my day-to-day activities (whether I realize it or not).

I'm fortunate enough to see how the code I write, the features I build, and the bugs I inadvertently introduce (more than I care to admit) affect the end user and their experience. That experience has helped shape how I think about the process of building products and solving problems for my users.

I've had some time to think about the differences (and similarities) each of these approaches offer. At their core, each is focused on delivering high quality software as efficiently and as cost effectively as possible.

Professionally, I've only used one or two of these methodologies. But I still find value in at least a basic understanding of all of them.

All of these methodologies follow a series of phases similar to this diagram:

Requirement Analysis -> Design -> Implementation -> Testing -> Evolution

So, here are the software development life cycle methods (in no particular order):

• Lean
• Agile
• Waterfall
• Iterative
• Spiral
• Dev Ops

Let's dig in to the differences and similarities of each method.

Lean

The Lean methodology relies heavily on and is comprised of seven principles.

In no specific order they are:

1. Eliminate Waste
2. Amplify Learning
3. Decide As Late As Possible
4. Deliver As Fast As Possible
5. Empower The Team
6. Build Integrity
7. See/Optimize The Whole

Each principle has a specific purpose with benefits that compliment each other.

Eliminating waste (extra features, incomplete work, managerial engagements, etc) creates more value for the customer which, in turn, enhances satisfaction.

Amplifying learning allows teams to reinvest in their ability to deliver products to customers.

Deciding as late as possible refers to all major decisions, giving teams an option based or a set based approach. This allows teams to gather facts rather than opinions to help influence decisions when made.

Delivering as fast as possible is self explanatory - build the product as quickly as possible to deliver it to customers for evaluation/iteration.

In a typical scenario, the managers dole out assignments/work to the developers. In the Lean methodology developers "teach" managers how to listen to the "people in the trenches" thus influencing the decisions/choices of management.

This helps teams feel more empowered to speak up about ides and solutions.

Making integrity a rule instead of an exception means that the customer is confident in the system being built. The customer knows the system is being built to withstand the appropriate amount of growth and "stretching" if need be.

I like to think of the integrity part along the same lines as sitting in a chair. When you sit in the chair you believe it was constructed with the best material that will hold you up every time you sit in it for the life of the chair. The customer needs to that same confidence in the product being built.

Lastly, seeing and optimizing the whole refers to the entirety of the system being built. By optimizing for the whole we look at software not as a sum of many components, but as one large entity that is optimized for efficiency.

This means that during development, the product is broken into manageable pieces and that inadvertent bugs are not only discovered but resolved swiftly.

Agile

This is the "fail fast" approach to building software.

It places emphasis on small, incremental releases with on-going release cycles. With each iteration teams strive to identify and address small issues before they become big problems.

This also means that the teams must engage stakeholders (people/organizations that the code can ultimately affect such as managers, technical leads, CTOs, and customers) to get their feedback.

If you're a freelance, your stakeholder(s) would be your customers - ultimately you need to ensure their satisfaction with the work before moving on.

Agile is technically an offshoot of the Lean methodology with some notable differences - mainly it prioritizes customer satisfaction from the outset and allows teams to respond quickly to customer feedback.

Although it is beyond the purview of this article, there is another more complex framework within Agile that is called SCRUM. This methodology is used for large, extremely complex projects and has even been used outside of software development.

Waterfall

The waterfall methodology is, by most accounts, the oldest one in the list. It was never meant to be a model for software development and got its start in the construction and manufacturing worlds.

This approach is simple in its structure - finish all parts of a phase before moving on to the next phase with more momentum building towards the project finish as stages are completed. Each stage's beginning (except for the first) and completion is contingent on the previous stage's completion/transfer of information.

Under the waterfall approach each stage has its own rigid project plan that finishes off with testing for previously completed work. It should be noted that this approach is not recommended for larger/longer lasting projects because of the aforementioned rigidity.

Think about the genesis of this methodology and you'll understand it more. It came from the construction/manufacturing world where it is common to complete one phase at a time. During the building of a house you wouldn't start putting in the plumbing before the frame has been put up.

That's not the way software development works generally. As we all know it sometimes becomes necessary to revisit a phase that was previously thought to be finished.

Iterative

This is known as the "repetitive approach" or the "make it better the next go around" approach because of the different opportunities it provides to improve the product with each cycle iteration.

I'm biased (as we all are :D) but this happens to be my favorite life cycle for development. I believe it works best for my current situation both in my freelance and career path because it allows me to constantly "move forward while making things better."

With the iterative approach, teams implement a solution, test that solution, evaluate its effectiveness/throughput, and then pinpoint further areas for improvement. This happens for each cycle (iteration) of the development process.

With each version released comes another iteration until the final product is completed and ready for rollout to users.

One of the great features of the iterative approach is you and your team get a working version of software early on in the development process. This can be especially useful to show to stakeholders to gauge their response/feedback.

One of the big drawbacks to this approach is it can consume a large amount of resources very quickly. Imagine all of the people, hours, bug fixes and wages that go into each iteration of the development cycle and you'll get a good picture of the resource usage.

Within this approach is a subset of principles developed by Rational Software Corporation (bought by IBM) called the Rational Unified Process (R.U.P.) which consists of 4 phases:

• Inception
• Elaboration
• Construction
• Transition (product release)

This set of principles is meant to be flexible and tailored to the needs of each team using it.

Spiral

The spiral methodology is likely the most flexible out of the six. It's a methodology built on risks - identifying and negating them. Risk (identification & aversion) drives every decision in this model. It is broken into four sub-phases:

• Planning (objectives)
• Risk Analysis (identify possible roadblocks)
• Develop & Test (current & next version)
• Evaluation (review of current phase & plan for next phase)

Each iteration of each phase begins with planning for the next phase. This way potential risks are identified before they are encountered. This also allows for a plan of action when said risks arise.

During phases teams also work to mitigate these risks and their impact on future iterations of the spiral development.

As the development process continues, each of these four sub-phases is repeated in spiral fashion. This allows for multiple rounds of refinement for each sub-phase until completion.

Dev Ops

If you do a quick search, you will find no shortage of information on this development life cycle method. It is the new kid on the block that brings software development and information-technology operations teams into the same fold.

These teams work in conjunction to provide small, but impactful, updates to products that come at a frequent pace. In turn, this creates a continuous feedback and improvement loop that drives development.

This particular methodology is known for automating the manual parts of development as well (think deployment).

The overall goal of this methodology is, like most others, the shorten the development life cycle and provide quality products.

One of the drawbacks to this methodology is the significant mindset and culture changes within an organization. Teams that may have been accustomed to working on many things find their tasks narrowed to only one or two.

For example, a general purpose developer may find they are now being tasked with only the testing portion or the end-user experience portion.

Bringing It All Home

_Photo by [Unsplash](https://unsplash.com/@clever_visuals?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Clever Visuals on <a href="https://unsplash.com/s/photos/bright-idea?utm_source=unsplash&utm_medium=referral&utmcontent=creditCopyText)

I hope you can now see the importance and the benefits of each of these methodologies. Each of these possess their own strengths and weaknesses.

They are, at their most basic level, a set of guidelines and principles that seek to deliver high quality, efficient work to stakeholders.

When I first started learning to code I didn't have a mentor. By sharing what I've learned I hope to help those who are learning to code when/where they can.

I want to share as much information and experience as I possibly can with other developers. If you are teaching yourself to code or if you're a seasoned developer I hope this article helps, even if in a small way.

Check out my blog where I frequently post articles about web development.

While you're there why not sign up for my newsletter? You can do that at the top right of the main blog page. I like to send out interesting articles (mine and others), resources, and tools for developers every now and then.

If you have questions about this article or just in general my DMs are open -- come say hi on Twitter or any of my other social media accounts which you can find below the newsletter sign up on the main page or on my profile here :)

Have an awesome day and happy coding, friend!

Skills-based Volunteer Organizations:

• FindCollabs
• Give Camp
• Volunteer Match
• Catchafire
• Anyone Can Have A Website

Building a website:

• How to build and deploy a website without writing any code for free
• Wix
• Square Space
• WordPress
• Imcreator.com

Donor and Volunteer Management Systems:

• Cause Signal
• The Data Bank
• Donor Snap
• Donor Perfect
• E Tapestry
• Z2 Systems
• Reg Packs
• Sumac
• Volgistics

Inventory Management Systems:

• Ordoro
• Unleashed Software
• EZ Office Inventory

E-Learning platforms:

• Dokeos
• E Front Learning
• Moodle
• Sakai Project

Community Management:

• CiviCRM
• Total Community Manager

Electronic Forms:

• Typeform
• Google Forms

If your website takes longer than 3 seconds to load, you could already be losing nearly half of your visitors.

Yes this is a fact, proven by several research studies. Long loading times can have a devastating impact on your application’s conversion rates. On the other hand, optimising page load time leads to noticeable improvements in customer experience, conversion rates, SEO and ultimately, your product’s success.

So let’s say that you have recently built a website or a frontend application using a modern JS framework (Angular, React, VueJS etc.). How can we make sure that performance is not going to hold it back from success?

First things first. We have to somehow gather some data. When it comes to measuring a frontend application’s performance, the tools I trust the most are:

• Google chrome’s Lighthouse
• Speedcurve

Both tools can help you keep track of all the major performance KPIs (Pagespeed Index, Time to Interactive, First Contentful Paint, etc.).

Lighthouse is included in your Chrome’s dev-tools, and by analysing your website/web-app, it can give you some really useful hints on how to boost it up.

Chrome’s lighthouse can give you some really helpful hints

With Speedcurve you can have all these KPIs, plus the ability to monitor them across time.

So now that we are able to measure our success, let’s move on with optimising the parts of our website that play the biggest role.

Images

Images are a key part of every website. On average, images make up for more than 60% of data that are loaded on web pages. Being such a critical component of almost all websites, image optimisation is, in my opinion, the most important, and perhaps the lowest hanging fruit.

1. Resize your images and make them responsive.

The most important thing to check is that you are not using a bigger resolution than the one you really need. So you have to resize your images to fit exactly the requirements of your layout.

Responsive layouts need responsive images

Furthermore, you have to make sure that your images are as responsive as your layout is. There is a great tool that I have been using lately, which can help you generate all the different versions of the images that you might need and also generate the HTML5 code that can utilise them. It’s called Responsive Image Breakpoints Generator. I usually prefer generating 8–10 different images.

Of course you can use the generated HTML5 code in any frontend app or website. In addition, if you are into gulp, you could automate this process with the gulp-responsive plugin.

2. Make sure they are lazy loaded.

Lazy loading basically means that we defer the loading of images that are not required immediately. Typically, any image that is not visible to the users on their current viewport, can be loaded at a later point in time, when the image enters or is about to enter the viewport.

No matter which framework you are using, you can find a plugin that can lazy-load the images for you (e.g. v-lazy-image in VueJS), however you could build your own implementation too. Just make sure that you are utilising the modern way to detect when an element enters or exits the browser’s viewport, the IntersesectionObserver API.

Bonus: Use a CDN for image delivery

If you have already optimised the size and the number of images that your website loads, and especially if it’s going to be available globally, you could also use a content delivery network (CDN) to serve them.

In a few words, a CDN caches your images on its globally distributed network of servers. So if a user from Australia requests an image from your website, instead of retrieving that image from your server in Europe, the CDN is going to deliver it from another one, closer to that user in Australia. This cuts down the round trip time needed to load the image.

CSS, JS and HTML

All modern frameworks optimise your code during the production build time (code-splitting, tree-shaking, minification, etc.). What you can do on top of that?

1. Optimise the main HTML document

HTML is the backbone of nearly every web app. When it comes to referencing resources within your HTML document there are a few best practices you should follow. It is recommended to:

• Place CSS references at the top of your HTML document’s header in order to ensure progressive rendering.
• Place JavaScript attributes at the bottom of your HTML body and prefer async script loading. This will prevent any <script> tags from blocking the HTML rendering process.

2. Make sure you only load what you need

Lazy loading components and modules

Angular, React, and VueJS all provide you with lazy-loading features. You only have to split your code properly, according to your own needs and load only the modules that you need, as soon as you really need them. For instance, if you have an e-commerce app, you have to make sure that the Basket module or the Payments module are not loaded when the user is in the home page.

Compression & Caching

In general, for all the assets that are essential to your frontend (images & code) you should apply compression and cache them properly.

File compression will make your app’s assets a bit lighter and decrease the round trip time needed to serve them. One of the most commonly used file compression methods is Gzip, an excellent method for shrinking code chunks, documents, images and audio files.

Brotli is another file compression algorithm, and it’s growing in popularity. This open source algorithm is regularly updated by software engineers from Google and other organisations. It has proven itself to compress files at a much better ratio than other existing methods.

You can enable your preferred compression method on nginx, apache or whichever server you are using, by modifying their configuration files (enabling brotli on nginx or enabling brotli on apache).

When it comes to caching, the most commonly used caching technique (also recommended by Lighthouse) is called Leverage Browser Caching. Again, you can enable it by modifying your server’s configuration files (enabling Leverage Browser Caching).

Summary

When it comes to websites and frontend applications, performance is a huge topic and we should take it seriously.

I hope this article helped you understand and enabled you to tackle some of the most important things that we need to take care of when we want to improve an application’s performance.

For a detailed checklist, make sure that you check out the Front-End-Performance-Checklist.

Cheers! ?

If you’ve ever wondered what Kafka, Heron, real-time streaming, SQS or RabbitMQ are all about, then this article is for you. I’ll discuss in detail why we need a queue for today’s modern software architecture, what are some common technologies used, and how queues are commonly used in the industry. If you enjoy this article, I have a course on scaling distributed systems where I discuss these topics in more detail.

OK, let’s get into it!

A Dummy’s Guide to Queues

First and foremost, why do you need a queue/message broker?

The story of how a queue saved the lemonade stands

Imagine that you are running a lemonade ? stand, and you built out a nifty little web-app that keeps track of how often your clients return to your lemonade stand.

Your web-app has an endpoint, say yourlemonade.com/traffic, and every time you click on a button, the traffic count goes up by 1. Beautiful.

As the traffic to your lemonade stand increases, you click the button more and more. Well, since you’re living in a relatively small neighborhood, you only get 10–20 people per day. Your sales proceed as usual, the web-app handles the traffic just fine, and everything is fine and dandy. Perfect.

Your lemonade stand web-app

The nightmare of a booming business

Now that your lemonade stand has made a name for itself, people from across the city are flocking in to get a taste of your famous lemonade. And on a beautiful Sunday morning, the local news decided to promote your stand, and the traffic EXPLODES.

As you can imagine, the traffic to your lemonade stand increases from 10–20 people per day to 10,000 per day. You’re tapping the traffic button furiously, which in turns triggers a call to yourlemonade.com/traffic, and your web app keeps incrementing the amount of traffic.

Lemonade stand business runs into bottleneck

Unfortunately, your web-app is hosted on an 8-bit, 128MB RAM server in your house garage. With the booming business and increased traffic, your web-app cannot handle the scale of traffic any longer.

Eventually, your server dies. ☠️

With that, your entire web app is brought down. You can’t keep track of traffic anymore. People are rushing in, orders are piling up, yet your web-app is down and you can’t handle any transactions until you can start logging the traffic again.

Traffic brought down your web-app, and business too.

What do you do?

Queue to the rescue

A moment of brilliance strikes you, what if I place a box in front of the counter where each client can just drop a note saying they were there?

Every time a client walks through the door and places an order, you ask them politely to drop their order sheets in a small box placed in front of the payment counter. Outstanding! You have essentially introduced a mechanism to keep track of arrivals while still allowing your business to function as usual.

This is what we call asynchronous processing, and, welcome to the world of queues. ?

Hero to the rescue!

When you start out building software, much like the lemonade stand I mentioned above, it is common for a task to

1. call a service, then
2. wait for the service to finish, and then
3. move on to the next task.

This is what’s called synchronous processing. Asynchronous processing, on the other hand, allows a task to call a service, and move on to the next task while the service processes the request at its own pace. That’s why a queue is a beautiful, elegant way to unblock your systems because it puts a layer in front of your services and allows them to tackle the tasks at their own pace.

If a queue is so powerful, why don’t we just put it in front of everything?

_Image courtesy of [imgflip.com](https://imgflip.com/i/2xylc5" rel="noopener" target="blank" title=")

As anyone who has dabbled in distributed systems can attest to, scaling a distributed system is extremely tricky and complicated. There are a few things to know about queues that might make a queue an unattractive proposition for your system.

Some questions I’d ask before deciding if a queue is the right solution for you:

• Is your service having issues due to high traffic? If it’s not, maybe you should look into what the bottleneck is before jumping into queues. As Donald Knuth famously said, premature optimization is the root of all evil.
• Do you have in-house expertise in managing a queue? Or do you need to potentially hire a team to do it for you? Maintenance costs, like scaling the queue, can skyrocket if you’re not careful. There are services like Amazon SQS (Simple Queueing Service) which offer a managed solution (i.e. you don’t need to maintain anything on your own).
• Is it possible to have duplicate entries in the queue? If so, is that acceptable?
• Do you need to keep a record of all transactions, in case a queue goes down?
• In the case that a queue goes down, does the queue need to be able to replay all the entries? What are your backup options?

There are many more concerns that might be specific to your use case, but hopefully, I’ve made my point that adding a queue isn’t as easy as snapping your fingers.

How Queues Are Used in Modern Architecture

Queues are ubiquitous in today’s modern distributed systems architecture — adopted across various industries for different use cases, and there are more novel use cases every day.

Here are some of the real-world use cases for queues:

Real-time streaming

When MapReduce came around, it was a huge phenomenon in the industry because it allowed mere mortals to process petabytes of data in a reasonable amount of time, anywhere from days to hours. This might seem absurd today when data’s available in almost seconds, but pre-MapReduce, it wasn’t easy to extract usable data from extremely large data sets.

The appetite for data analytics has grown, and we’re now looking at processing data within hours, and sometimes, milliseconds.

To achieve low-latency analytics and performance in a continuous fashion, the concept of real-time streaming was conceived.

A useful example here is to think of ads: ads on Twitter, for example, are shown to millions of people per day. Yet in order to make sure users don’t see the same ads multiple times within a set period of time, Twitter needs to somehow know the last time a user was exposed to a certain ad.

If we had relied on MapReduce to perform this action, it would not even be considered a solution because it’ll take upwards of hours to process all that data. Instead, real-time streaming allows us to process ad impressions as they arrive. This is all made possible because of queues that allow data to be continuously streamed and processed in real time.

Some technologies you’ll often hear about in real-time streaming use cases are Kafka, Kafka streams, Redis, Spark Streaming (which is different from Spark) and so on.

Event-driven architecture

Queues are used as a critical component of an event-driven architecture, or colloquially known as Pub(lisher)-Sub(scriber). Event-driven architecture is, according to Wikipedia:

Event-driven architecture (EDA), is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.

I’d like to think of this as subscribing to a newsletter: as a producer of a newsletter, you know who’s subscribed to your newsletter and who’s not. You write the content, and then you send it to your subscribers.

On the other hand, as a subscriber, you might be subscribed to multiple newsletters, but you don’t know who the other subscribers are. But you don’t really care about that. This is a really nice feature because you can now write software that listens to a bunch of events and only responds to the ones you’re interested in.

RabbitMQ and Amazon SQS (Simple Queuing Service) are some of the technologies often used for these types of use cases.

Distributed, fault-tolerant, scalable infrastructure

Distributed systems are prone to errors, and a queue is one of several ways to increase resiliency in the architecture. In a microservice architecture (or service-oriented architecture), multiple microservices communicate with each other through queues as shared interfaces.

When a microservice fails unexpectedly, a queue is still able to accept messages. This essentially provides a buffer for our microservice to recover. Once the microservice comes back online, it can pick up the messages from the queue and process them again.

Think of it as your mailbox. While you’re out on vacation in Hawaii, the mailperson will still deliver your mail into the mailbox. Once you return from vacation, you can pick up the mail and process them at your leisure.

Thank you for reading! I hope you’ve learned a thing or two about distributed queues from my article. If you enjoyed reading this, please leave a clap, and feel free to join my newsletter here where I write about software and technical interviews!

Resources I Recommend

To further your understanding of queues and various topics mentioned above, I would highly recommend these resources below. Or join my course on scaling distributed systems to learn more about queues :)

• Designing Data-Intensive Applications: Awesome book for learning about scaling distributed systems! Highly recommended.
• Kafka the Guide: I used this book as a reference guide, and enjoyed it for the high-level description.
• Kafka Streams: This is an informative article from Confluent that talks in some high-level detail about Kafka’s implementation of stream processing.
• Elements of Programming Interviews: Great for solving coding problems.
• Cracking The Coding Interview: Great for covering foundational CS coding problems.
• Daily Coding Problem.com: This is a free-to-try website that offers free daily coding problems. You can sign up for interesting daily coding challenges, and you can pay for solutions if you’d like. If you use my referral link (dailycodingproblem.com/zhiachong), you get $10 off!

(FYI, I share more resources on my website: zhiachong.com where I’ve personally tried and tested, and recommend for software engineers of all levels.)

Cheers!

Installing the Ethereum node client on a machine can be a tedious process. There is a simpler way this can be done using a Docker client. This is a guide for running the GETH (Ethereum-Go) node client from inside a Docker container. GETH is the GoLang implementation of the Ethereum protocol. There is an image available to pull from the Docker hub repository that can run the environment.

The GoEthereum website lists the following available images with descriptions.

• ethereum/client-go:latest is the latest develop version of Geth
• ethereum/client-go:stable is the latest stable version of Geth
• ethereum/client-go:{version} is the stable version of Geth at a specific version number
• ethereum/client-go:release-{version} is the latest stable version of Geth at a specific version family

The following ports are opened by default when running from the container.

• 8545 TCP, used by the HTTP based JSON RPC API
• 8546 TCP, used by the WebSocket based JSON RPC API
• 30303 TCP and UDP, used by the P2P protocol running the network
• 30304 UDP, used by the P2P protocol's new peer discovery overlay

The Docker client software must be installed on the machine you are going to run the container from. Containers can only run if you have the Docker client installed. Depending on your operating system, the correct version of the client will be needed.

There are separate versions for Windows, Linux and the MacOS. The container can even be run on a Linux instance running on AWS, like a typical Linux installation. Once the Docker client is installed, the underlying platform doesn’t matter. The commands will be the same for all.

Getting The Image

Open a terminal on Linux or MacOS, or PowerShell command prompt from Windows. In the CLI prompt, type the following command:

docker pull ethereum/client-go

This pulls the Docker image from the hub repository where it was uploaded by the Ethereum developers. Once you have issued this command, the following verbose or similar should be displayed:

I have already pulled the image, so the verbose may look different. When you issue the pull command it will always download the latest image available, which is good practice.

Running The Node

Now you can start the node by issuing the following command:

We want to run the node with the flag options -i and -t to display information from our container. The -p indicates the use of a port number, in this case 30303. Likewise, the command can be run without the flags and it will simply use the default ports and settings from inside the container.

The following information should appear from the terminal.

The INFO line that shows the config reveals what the node client software has installed. The node client is running the latest (as of this posting) version of the Ethereum software which is Constantinople that is a user activated hard fork at block height 7280000.

When running in JSON-RPC API:

Take note that running the option rpcaddr “0.0.0.0” is not secure, since you are opening up your node to all traffic. If your ETH wallet were unlocked, a hacker can get to your node in this way and take your coins. I don’t cover security in this article, but you can read more about that here (securing your GETH node’s RPC ports). Always abide to safe and best practices.

If the node displays the following in the INFO line, there will be a problem:

config=”{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: Engine: ethash}”

The Constantinople: indicates the software was not updated. There is also no line for ConstantinopleFix, which appears in the correct configuration.

Persistent Data

For persistent blockchain data, Docker data volumes should be used with the option -v. The /path/on/host should be replaced with the location you specify. For this the following command must be used:

Checking Node Status

You can check the container’s status using the following command:

docker ps

This will display the Container ID with the image name, status and ports used.

#These are the commands to run from the Docker CLI to run the Ethereum Go node client

#GETTING THE IMAGE

docker pull ethereum/client-go


#RUNNING THE NODE

docker run -it -p 30303:30303 ethereum/client-go

#RUNNING NODE USING API

docker run -it -p 8545:8545 -p 30303:30303 ethereum/client-go --rpc --rpcaddr "0.0.0.0"

#Note, warning about using --rpcaddr "0.0.0.0" in a live environment. It is an insecure way of opening your node.
#There are different ways to secure your ports, but this is one thing to take note of if you plan to use the API.


#PERSISTENT DATA

docker run -it -p 30303:30303 -v /path/on/host:/root/.ethereum ethereum/client-go

Running GETH from a Docker Container

Take note that this does not automatically mine ETH. That is a different process. For getting quick access to the Ethereum blockchain, this is the purpose for running GETH.

For full code source, visit: https://github.com/Play3rZer0/GETHDocker.git

Here I’ll tell you my story how I completed 100 days of code, how I started, how I keep it going, and how I finally finished it.

What I have produced in my 100 days of code

Starting Point

About a year ago I actually heard a lot about the 100 days of code challenge, but I had never thought of doing it. The very sound of one hundred days felt so intimidating for me back then that I was not sure that I could actually finish it.

Then one day I read this article, and after reading so many inspiring stories from people doing 100 days of code on Twitter, I finally decided that the time for me to start was now or never. And I finally started doing my 100 days of code, about 100 days ago from when I am writing this article.

Managing and Planning my work

First things first, I planned on what coding I would do and I decide that I would contribute to the GitHub open-source community. I would spend each day coding for about 1 hour and I would try to keep it to that timeframe — because I have got my me-time too ;)

My main challenge for this was actually finding out what to do every single day. For that, I always prepared two kinds of notes on my phone:

The first type of note was a list of my progress in 100 days of code, each and every day. Not only that, but this note also served as my note on what to do the next day. I always planned on what I was going to do at night. Then the next day I would know what to work on, so I would not waste my time thinking of what to do when I should have been coding.

made it into two notes since there is actually a limit in android Memo App

My second note is was list of what open source things I wanted to make, something like a bucket list or a wish list, but instead it is a code list.

I named this open source wish list

Finding an idea for what to do was also a difficult task to do every day. That’s why I always spent my days reading Medium articles to find inspiration for what I should be exploring.

Since my expertise is in Android mobile application development, I started most of my repositories based on Android applications with Kotlin. Then after I started getting used to it, I finally started making repositories based on other things such as Serverless with go and nodejs, Flutter and Rust.

Nearing the day that I finished my 100 days of code, I also made another note which listed my leftover tasks that usually had something to do with refactoring.

refactor job, everyone loves clean code

Keeping myself going

100 days is quite a long time. You might wonder if there were actually days where I was thinking of quitting.

Yes, there actually were — there have been several days where I doubted myself. Was I actually doing the right thing or was I just wasting my time doing this thing?

But I always kept myself going because of everything I have learned. Living in this Technology Era as a Software Engineer, I must always strive to learn new things and never stop. The moment I stop learning is the moment I stop growing as a Software Engineer. And seeing the stories about how people are finishing 100 days of code and even 360 days of code made me feel challenged since there were so many people out there doing it. If they could do it and even do more, why couldn’t I finish it?

It’s very important to push yourself as much as you’re able. But don’t forget to limit yourself and give yourself a break sometimes. If you reach your limit, you can then call it a day and continue on tomorrow. There were also days when all I did was refactoring and making a new repository. It is always important to keep up, but do not forget to give yourself a break.

And also, the most important part of keeping myself going: Consistency.

Completing my 100 days of code

When I finally finished my 100 days of code, the first two things I felt were feelings of accomplishment and relief.

People may ask me why I did it. I didn’t gain things from it — no profit, no one is looking at my mediocre GitHub repositories, no forks or stars. But of course I learned a lot of things from a technical point of view. Also, most of the people that saw me doing this 100 days of code acknowledged me for being consistent and continuing to do it.

One thing I always believe is that people may betray you, but hard work will not betray you. Keep sharpening yourself. At the end of the day, you will see yourself reaping the good things that you have sown.

Feel free to check out my GitHub! I will be very thankful for any issue opened or any PR. I have produced several android components which are usable and tested.

So then, this ends my story on my first 100 days of code. Where is your story?

“Some people, when confronted with a problem, think ‘I know, I’ll use regular expressions.’ Now they have two problems.” -Jamie Zawinski

For some people, using regular expressions can be a problem. But it doesn’t have to be a problem for you. This article is a full course on Regular Expressions.

1. Introduction

Regular Expressions, or just RegEx, are used in almost all programming languages to define a search pattern that can be used to search for things in a string.

I’ve developed a free, full video course on Scrimba.com to teach the basics of regular expressions.

This article contains the course in written form. But if you would prefer to watch the video version with interactive lessons, you can check it out on Scrimba. The sections in this article correspond to the sections in the Scimba course.

This course follows along with the RegEx curriculum at freeCodeCamp.org. You can check that out for coding challenges and to earn a certificate.

These lessons focus on using RegEx in JavaScript, but the principles apply in many other programming languages you might choose to use. If you don’t already know basic JavaScript, it could be helpful if you cover it a bit first. I also have a basic JavaScript course that you can access on Scrimba and on the freeCodeCamp.org YouTube channel.

So let’s get started! You’ll be saving the day in no time. ?

_From [https://xkcd.com/208/](https://xkcd.com/208/" rel="noopener" target="blank" title=")

2. Using the Test Method

To match parts of strings using RegEx, we need to create patterns that help you to do that matching. We can indicate that something is a RegEx pattern by putting the pattern between slashes /, like so /pattern-we-want-to-match/.

Let’s look at an example:

// We want to check the following sentencelet sentence = "The dog chased the cat."

// and this is the pattern we want to match.let regex = /the/

Notice how we use /the/ to indicate that we are looking for “the” in our sentence.

We can use RegEx test() method to tell if a pattern is present in a string or not.

// String we want to testlet myString = "Hello, World!";

// Pattern we want to findlet myRegex = /Hello/;

// result is now truelet result = myRegex.test(myString);

3. Match Literal Strings

Let’s now find Waldo.

let waldoIsHiding = "Somewhere Waldo is hiding in this text.";let waldoRegex = /Waldo/;

// test() returns true, so result is now also truelet result = waldoRegex.test(waldoIsHiding);

Note that in this example waldoRegex is case sensitive, so if we were to write /waldo/ with a lowercase ‘w’, then our result would be false.

4. Match a Literal String with Different Possibilities

RegEx also has OR operator which is | character.

let petString = "James has a pet cat.";

// We can now try to find if either of the words are in the sentencelet petRegex = /dog|cat|bird|fish/;

let result = petRegex.test(petString);

5. Ignore Case While Matching

So far, we have looked at patterns when the case of the letters mattered. How can we make our RegEx patterns to be case insensitive?

To ignore case we can do it by adding the i flag at the end of a pattern, like so /some-pattern/i.

let myString = "freeCodeCamp";

// We ignore case by using 'i' flaglet fccRegex = /freecodecamp/i;

// result is truelet result = fccRegex.test(myString);

6. Extract Matches

When we want to extract the matched value we can use match() method.

let extractStr = "Extract the word 'coding' from this string.";

let codingRegex = /coding/;

let result = extractStr.match(codingRegex);

console.log(result);

// Terminal will show: // > ["coding"]

7. Find More Than the First Match

Now when we know how to extract one value and it’s also possible to extract multiple values using theg flag

let testStr = "Repeat, Repeat, Repeat";

let ourRegex = /Repeat/g;

testStr.match(ourRegex); // returns ["Repeat", "Repeat", "Repeat"]

We can also combine theg flag with thei flag, to extract multiple matches and ignore casing.

let twinkleStar = "Twinkle, twinkle, little star";

let starRegex = /twinkle/ig;// writing /twinkle/gi would have the same result.

let result = twinkleStar.match(starRegex);

console.log(result);

// Terminal will show: // > ["Twinkle", "twinkle"]

8. Match Anything with Wildcard Period

In RegEx . is a wildcard character that would match anything.

let humStr = "I'll hum a song";

let hugStr = "Bear hug";

// Looks for anything with 3 characters beginning with 'hu'let huRegex = /hu./;

humStr.match(huRegex); // Returns ["hum"]

hugStr.match(huRegex); // Returns ["hug"]

9. Match Single Character with Multiple Possibilities

Matching any character is nice, but what if we want to restrict the matching to a predefined set of characters? We can do by using [] inside our RegEx.

If we have /b[aiu]g/, it means that we can match ‘bag’, ‘big’ and ‘bug’.

If we want to extract all the vowels from a sentence, this is how we can do it using RegEx.

let quoteSample = "Beware of bugs in the above code; I have only proved it correct, not tried it.";

let vowelRegex = /[aeiou]/ig;

let result = quoteSample.match(vowelRegex);

10. Match Letters of the Alphabet

But what if we want to match a range of letters? Sure, let’s do that.

let quoteSample = "The quick brown fox jumps over the lazy dog.";

// We can match all the letters from 'a' to 'z', ignoring casing. let alphabetRegex = /[a-z]/ig;

let result = quoteSample.match(alphabetRegex);

11. Match Numbers and Letters of the Alphabet

Letters are good, but what if we also want numbers?

let quoteSample = "Blueberry 3.141592653s are delicious.";

// match numbers between 2 and 6 (both inclusive), // and letters between 'h' and 's'. let myRegex = /[2-6h-s]/ig;

let result = quoteSample.match(myRegex);

12. Match Single Characters Not Specified

Sometimes it’s easier to specify characters that you don’t want to watch. These are called ‘Negated Characters’ and in RegEx you can do it by using ^.

let quoteSample = "3 blind mice.";

// Match everything that is not a number or a vowel. let myRegex = /[^0-9aeiou]/ig;

let result = quoteSample.match(myRegex);// Returns [" ", "b", "l", "n", "d", " ", "m", "c", "."]

13. Match Characters that Occur One or More Times

If you want to match a characters that occurs one or more times, you can use +.

let difficultSpelling = "Mississippi";

let myRegex = /s+/g;

let result = difficultSpelling.match(myRegex);// Returns ["ss", "ss"]

14. Match Characters that Occur Zero or More Times

There is also a * RegEx quantifier. This one matches even 0 occurrences of a character. Why might this be useful? Most of the time it’s usually in combination with other characters. Let’s look at an example.

let soccerWord = "gooooooooal!";

let gPhrase = "gut feeling";

let oPhrase = "over the moon";

// We are trying to match 'g', 'go', 'goo', 'gooo' and so on. let goRegex = /go*/;

soccerWord.match(goRegex); // Returns ["goooooooo"]

gPhrase.match(goRegex); // Returns ["g"]

oPhrase.match(goRegex); // Returns null

15. Find Characters with Lazy Matching

Sometimes your pattern matches can have more than one outcome. For example, let’s say I’m looking for a pattern in a word titanic and my matched values must begin with a ‘t’ and end with an ‘i’. My possible results are ‘titani’ and ‘ti’.

This is why RegEx has a concepts of ‘Greedy Match’ and ‘Lazy Match’.

Greedy match finds the longest possible match of the string that fits the RegEx pattern, this is a default RegEx match:

let string = "titanic";

let regex = /t[a-z]*i/;

string.match(regex);// Returns ["titani"]

Lazy match finds the shortest possible match of the string that fits the RegEx pattern and to use it we need to use ?:

let string = "titanic";

let regex = /t[a-z]*?i/;

string.match(regex);// Returns ["ti"]

16. Find One or More Criminals in a Hunt

Now let’s have a look at a RegEx challenge. We need to find all the criminals (‘C’) in a crowd. We know that they always stay together and you need to need to write a RegEx that would find them.

let crowd = 'P1P2P3P4P5P6CCCP7P8P9';

let reCriminals = /./; // Change this line

let matchedCriminals = crowd.match(reCriminals);

You can find me walking through the solution in this Scrimba cast.

17. Match Beginning String Patterns

RegEx also allows you to match patterns that are only at the beginning of a string. We’ve already talked about ^ creating a negating set. We can use the same symbol to find a match only at the beginning of a string.

let calAndRicky = "Cal and Ricky both like racing.";

// Match 'Cal' only if it's at the beginning of a string. let calRegex = /^Cal/;

let result = calRegex.test(calAndRicky); // Returns true

let rickyAndCal = "Ricky and Cal both like racing.";

let result = calRegex.test(rickyAndCal); // Returns false

18. Match Ending String Patterns

What about matching a pattern at the end of a string? We can use $ for that.

let caboose = "The last car on a train is the caboose";

// Match 'caboose' if it's at the end of a string.let lastRegex = /caboose$/;

let result = lastRegex.test(caboose); // Returns true

19. Match All Letters and Numbers

Earlier in parts 10 and 11 I showed you how we can match ranges of letters and numbers. If I asked you to write a RegEx that matches all the letters and numbers and ignore their cases you probably would have written something like /[a-z0-9]/gi and that’s exactly right. But it’s a bit too long.

RegEx has something called ‘Shorthand Character Classes’, which is basically a shorthand for common RegEx expression. For matching all letters and numbers we can use \w and we also get underscore _ matched as a bonus.

let quoteSample = "The five boxing wizards jump quickly.";

// Same as /[a-z0-9_]/gi to match a-z (ignore case), 0-9 and _let alphabetRegexV2 = /\w/g;

// The length of all the characters in a string// excluding spaces and the period. let result = quoteSample.match(alphabetRegexV2).length;

// Returns 31

20. Match Everything But Letters and Numbers

If we want to do the opposite and match everything that is not a letter or a number (also exclude underscore _), we can use \W

let quoteSample = "The five boxing wizards jump quickly.";

// Match spaces and the periodlet nonAlphabetRegex = /\W/g;

let result = quoteSample.match(nonAlphabetRegex).length;

// Returns 6

21. Match All Numbers

Ok, what about if you want only numbers? Is there a shorthand character class for that? Sure, it’s \d.

let numString = "Your sandwich will be $5.00";

// Match all the numberslet numRegex = /\d/g;

let result = numString.match(numRegex).length; // Returns 3

22. Match All Non-Numbers

Would you like the opposite and match all the non-numbers? Use \D

let numString = "Your sandwich will be $5.00";

// Match everything that is not a numberlet noNumRegex = /\D/g;

let result = numString.match(noNumRegex).length; // Returns 24

23. Restrict Possible Usernames

So far so good! Well done for making it this far. RegEx can be tricky as it’s not the most easily readable way to code. Let’s now look at a very real-life example and make a username validator. In this case you have 3 requirements:

• If there are numbers, they must be at the end.
• Letters can be lowercase and uppercase.
• At least two characters long. Two-letter names can’t have numbers.

Try to solve this on your own and if you find it difficult or just want to check the answer, check out my solution.

24. Match Whitespace

Can we match all the whitespaces? Of course, we can use a shorthand for that too and it’s \s

let sample = "Whitespace is important in separating words";

// Match all the whitespaceslet countWhiteSpace = /\s/g;

let result = sample.match(countWhiteSpace);

// Returns [" ", " ", " ", " ", " "]

25. Match Non-Whitespace Characters

Can you guess how to match all non-whitespace characters? Well done, it’s \S!

let sample = "Whitespace is important in separating words";

// Match all non-whitespace characterslet countWhiteSpace = /\S/g;

let result = sample.match(countWhiteSpace);

26. Specify Upper and Lower Number of Matches

You can specify the lower and upper number of pattern matches with ‘Quantity Specifiers’. They can be used with {} syntax, for example {3,6}, where 3 is the lower bound and 6 is the upper bound to be matched.

let ohStr = "Ohhh no";

// We want to match 'Oh's that have 3-6 'h' characters in it. let ohRegex = /Oh{3,6} no/;

let result = ohRegex.test(ohStr); // Returns true

27. Specify Only the Lower Number of Matches

When we want to specify only the lower bound, we can do it by omitting the upper bound, for example to match at least three characters we can write {3,}. Notice that we still need a comma, even when we don’t specify the upper limit.

let haStr = "Hazzzzah";

// Match a pattern that contains at least for 'z' characterslet haRegex = /z{4,}/;

let result = haRegex.test(haStr); // Returns true

28. Specify Exact Number of Matches

In the previous section I mentioned that we need a comma in {3,} when we specify only the lower bound. The reason is when you write {3} without a comma, it means that you are looking to match exactly 3 characters.

let timStr = "Timmmmber";

// let timRegex = /Tim{4}ber/;

let result = timRegex.test(timStr); // Returns true

29. Check for All or None

There are times when you might want to specify a possible existence of a character in your pattern. When a letter or a number is optional and we would use ? for that.

// We want to match both British and American English spellings // of the word 'favourite'

let favWord_US = "favorite";let favWord_GB = "favourite";

// We match both 'favorite' and 'favourite' // by specifying that 'u' character is optionallet favRegex = /favou?rite/; // Change this line

let result1 = favRegex.test(favWord_US); // Returns truelet result2 = favRegex.test(favWord_GB); // Returns true

30. Positive and Negative Lookahead

‘Lookaheads’ are patterns that tell your JS to lookahead to check for patterns further along. They are useful when you’re trying to search for multiple patterns in the same strings. There 2 types of lookaheads — positive and negative.

Positive lookahead uses ?= syntax

let quit = "qu";

// We match 'q' only if it has 'u' after it. let quRegex= /q(?=u)/;

quit.match(quRegex); // Returns ["q"]

Negative lookahead uses ?! syntax

let noquit = "qt";

// We match 'q' only if there is no 'u' after it. let qRegex = /q(?!u)/;

noquit.match(qRegex); // Returns ["q"]

31. Reuse Patterns Using Capture Groups

Let’s imagine we need to capture a repeating pattern.

let repeatStr = "regex regex";

// We want to match letters followed by space and then letterslet repeatRegex = /(\w+)\s(\w+)/;

repeatRegex.test(repeatStr); // Returns true

Instead of repeating (\w+) at the end we can tell RegEx to repeat the pattern, by using \1. So the same as above can be written again as:

let repeatStr = "regex regex";

let repeatRegex = /(\w+)\s\1)/;

repeatRegex.test(repeatStr); // Returns true

32. Use Capture Groups to Search and Replace

When we find a match, it’s sometimes handy to replaced it with something else. We can use replace() method for that.

let wrongText = "The sky is silver.";

let silverRegex = /silver/;

wrongText.replace(silverRegex, "blue");

// Returns "The sky is blue."

33. Remove Whitespace from Start and End

Here’s a little challenge for you. Write a RegEx that would remove any whitespace around the string.

let hello = "   Hello, World!  ";

let wsRegex = /change/; // Change this line

let result = hello; // Change this line

If you get stuck or just want to check my solution, feel free to have a look at the Scrimba cast where I solve this challenge.

34. Conclusion

Congratulations! You have finished this course! If you’d like to keep learning more, feel free to checkout this YouTube playlist, that has a lot of JavaScript projects you can create.

Keep learning and thanks for reading!

You are now ready to play regex golf. ?

_From [https://xkcd.com/1313/](https://xkcd.com/1313/" rel="noopener" target="blank" title=")