Whether you’re browsing the web or managing a corporate network, the role of proxies is critical in maintaining security and efficiency. This article will help you understand what proxies are and how they can enhance your online experiences.

What We'll Cover:

• What is a Proxy?

• Benefits of Forward Proxies

• Understanding Reverse Proxies

• Other Proxy Types

• Conclusion

What is a Proxy?

A proxy server serves as an intermediary between your private network and the public internet.

Think of it as a middleman that manages communications between your devices and the internet. When you send a request to access a website, the proxy server receives it and forwards it to the intended destination, acting on your behalf.

In simpler terms, a proxy server provides a layer of security and privacy by masking your internet activities. It helps ensure that all your online requests are routed appropriately while protecting your network from threats like hackers or malicious sites.

This is especially useful for large networks, where direct internet access can expose vulnerabilities and security risks.

Benefits of Forward Proxies

Forward proxies offer a multitude of advantages that can enhance network performance and security.

Firstly, they help regulate internet traffic. By controlling the flow of data, you can prevent harmful websites from accessing your network. Also, forward proxies conceal individual IP addresses and present a single interface to the outside world, enhancing your privacy.

Another key benefit of forward proxies is the ability to monitor and log user activity. Organisations can track website visits and the duration of each session, offering insights into user behaviour and accountability.

They also offer an opportunity to bypass restricted content. In highly regulated environments, proxies help in accessing content that might otherwise be restricted.

Last but not least, forward proxies improve speed and efficiency by caching frequently accessed websites. This means these websites load more quickly as they're retrieved from the cache instead of being retrieved from the internet each time.

Understanding Reverse Proxies

Reverse proxies work in the opposite way by managing the traffic coming into a network rather than the traffic going out. They're particularly useful in protecting servers, enhancing security by creating a single point of entry to the network. This limits direct exposure of servers to potential threats, as external users interact with the reverse proxy rather than the server itself.

A significant benefit of reverse proxies is load balancing. In complex networks, incoming traffic can overwhelm servers, leading to downtimes. Reverse proxies distribute this traffic evenly, preventing any single server from being overloaded. This ensures smooth operations and maximises server uptime.

Reverse proxies can also protect against Distributed Denial of Service (DDoS) attacks by acting as a buffer. They intercept and block malicious traffic before it reaches the servers, providing an extra layer of security. Reverse proxies also conceal server IP addresses, making it harder for hackers to target specific servers directly.

Other Proxy Types

There are even more proxy solutions depending on your specific network needs.

Residential proxies provide anonymous browsing by routing traffic through real IP addresses assigned by Internet Service Providers (ISPs) to actual households. This makes the traffic appear highly legitimate, significantly reducing the chances of detection or blocking by target websites.

They are particularly effective for web scraping, account management, and accessing geo-restricted content because websites treat them as genuine users. But they tend to be more expensive due to the scarcity and operational complexity of maintaining real residential IP pools. Despite the cost, they're often the preferred choice when reliability and stealth are critical.

ISP proxies, also known as static residential proxies, combine the advantages of both residential and datacenter proxies. They're hosted on servers but use IP addresses assigned by ISPs, which gives them the appearance of residential traffic while maintaining high speed and stability.

These proxies are ideal for long-running sessions, automation workflows, and large-scale scraping operations where consistency is important. Businesses often rely on ISP proxies when they need both performance and trustworthiness without frequent IP rotation. They strike a balance between cost, speed, and legitimacy, making them a versatile option.

Datacenter proxies are generated from cloud servers or data centers rather than real residential networks. They're known for their high speed, low latency, and cost-effectiveness, making them suitable for tasks that require rapid data extraction or bulk operations.

But because they originate from identifiable server ranges, websites can more easily detect and block them compared to residential or ISP proxies. They're best used for non-sensitive scraping tasks, testing environments, or scenarios where scale and speed are prioritized over stealth. Many teams use them as a first layer before switching to more sophisticated proxy types if needed.

Mobile proxies route traffic through IP addresses assigned to mobile devices via cellular networks such as 4G or 5G. These IPs are highly trusted by websites because mobile carriers use techniques like carrier-grade NAT, where many users share the same IP, making blocking less effective.

As a result, mobile proxies offer the highest level of anonymity and are extremely effective at bypassing strict anti-bot and anti-scraping mechanisms. They're commonly used for social media automation, ad verification, and accessing mobile-specific content. While they're typically the most expensive option, their success rate in difficult environments often justifies the investment.

Conclusion

Proxies  –  be it forward or reverse  – represent a crucial piece of today’s network security and efficiency puzzle. Forward proxies protect client devices by regulating outgoing internet traffic and masking individual identities, while reverse proxies safeguard servers by controlling incoming traffic and offering load balancing.

By leveraging these proxy solutions, you can ensure enhanced network security and improved functionality. Whether you’re a business looking to protect server data or a user interested in anonymous browsing, choosing the right proxy solution can make a significant difference in maintaining a secure and efficient digital presence.

Join my Applied AI newsletter to learn how to build and ship real AI systems. Practical projects, production-ready code, and direct Q&A. You can also connect with me on LinkedIn.

For many technical jobs it is important to understand computer networking. We just posted a massive 12-hour course that will give you a deep dive into computer networking.

Here are the sections covered in this comprehensive course:

Course Overview & Methodology

• Fundamental Concepts and Networking Stack

• Orientation: Curriculum and Prerequisite (DSA)

• Introduction to the Instructor: Shrathir Sharma

• Course Access (YouTube/Udemy) and Target Audience

• Teaching Methodology: Raw Pen & Paper Style

• Core Modules: IPv4, Error Control, and Flow Control

• Core Modules: Transport, Media Access, and Routing

• Bonus Module: Cybersecurity

Networking Basics

• Defining a Computer Network

• Why Networks Interact: Resource Sharing

• 5 Components of Data Communication

• 4 Metrics for Network Effectiveness

• Transmission Modes: Simplex, Half-Duplex, Full-Duplex

• Types of Connections: Point-to-Point vs. Multi-Point

Topology & Architecture

• Introduction to Topology Layouts

• Mesh Topology and Link Calculations

• Advantages and Disadvantages of Topologies

• Star, Bus, and Ring Topology Details

• The OSI Model Framework

• Layered Architecture and Peer-to-Peer Protocols

Binary & IP Addressing Foundations

• Review of Lecture Zero

• Binary Number Representation & Conversion

• Binary Weights and Octet Conversions

• Introduction to IPv4 Logical Addressing

• Network ID vs. Host ID and IANA Authority

Classful vs. Classless Addressing

• Telephone Network Analogy for IP Classes

• Class A, B, and C Breakdown

• Classful Wastage and the Need for Classless (CIDR)

• Implementation: Fixing Bits for Classes A-E

• IP Address Space Distribution

• Hexadecimal and Decimal IP Representations

IPv4 Addressing Deep Dive

• Class A Details: Reserved Addresses & 127.0.0.1 Loopback

• Calculating Valid Hosts and Reserved All-Zeros/All-Ones

• Loopback Testing & Troubleshooting Connectivity

• Class B Details: Network Ranges & Host Capacity

• Class C Details: Network/Host Ratios

• Class D (Multicasting) and Class E (Experimental)

• IP Conversion Practice: Hexadecimal to Decimal

• Common Pitfalls: "Addresses" vs. "Valid Hosts"

Subnetting & VLSM

• Introduction to Subnetting: Why We Divide Networks

• Disadvantages of Subnetting: Wastage and Cost

• How to Subnet: Borrowing Bits from Host ID

• Subnet Identification: Calculating Subnet IDs and DBAs

• Working with Weights: Identifying Specific Subnets

• Subnet Masks vs. Network Masks

• Designing a Subnet Mask for Specific Requirements

• Variable Length Subnet Masking (VLSM) Strategy

• Determining Subnet IDs using Bitwise AND Operations

• Routing Tables: Matching Destination IPs to Interfaces

• CIDR: Classless Inter-Domain Routing & Slash Notation

• Rules for Valid CIDR Blocks

• Supernetting: Merging Multiple Blocks

Error Control & Detection

• Introduction to Error Control: Noise vs. Security

• Single Bit Error vs. Burst Errors

• Redundant Bits and Block Coding Logic

• Hamming Distance: Calculating Difference Between Strings

• Minimum Hamming Distance for Detection and Correction

• Simple Parity: Even vs. Odd Parity Methods

• 2D Parity: Detecting and Correcting Single Bit Errors

• Limitations of 2D Parity for Multi-Bit Errors

• Cyclic Redundancy Check (CRC): Divisor & Remainder Logic

• Checksum: One's Complement Summation Method

Flow Control & Layered Architecture

• Network Delays: Transmission vs. Propagation

• Queuing and Processing Delays

• Data Encapsulation: Headers and Trailers

• The Need for Flow Control: Avoiding Receiver Overwhelm

• Stop and Wait Protocol: Core Mechanism

• Using Timers and Sequence Numbers

• Calculating Efficiency and Round Trip Time (RTT)

• Throughput: Effective Bandwidth Relationship

• Sliding Window Concept: Improving Efficiency

• Go-Back-N (GBN) Protocol: Sender/Receiver Windows

• Selective Repeat (SR) Protocol: Out-of-Order Handling

• Cumulative vs. Independent Acknowledgments

Network Layer: IP Header & Routing

• IPv4 Header Format Overview

• Type of Services (TOS): Priority and DTRC Bits

• Time to Live (TTL): Preventing Infinite Loops

• Protocol Field and Header Checksum

• IP Options: Strict vs. Loose Source Routing

• TCP Header Structure: Ports, Sequence, and Ack

• Wrap Around Time and Segment Lifetime

• Advertisement Window (Flow Control)

• TCP Control Flags: URG, ACK, PSH, RST, SYN, FIN

• SYN Flooding Attack (DDoS)

• Congestion Control Policy: Slow Start & Avoidance

• TCP Timers: Time-Wait, Keep-Alive, Persistent

• UDP Header and Best-Effort Delivery

• Comparison: TCP vs. UDP

Media Access & Application Support

• Multiple Access: Random vs. Controlled Access

• Pure Aloha vs. Slotted Aloha Throughput

• CSMA (Carrier Sense): Persistent Methods

• Polling, Reservation, and Token Passing

• Routing: Flooding vs. Dynamic Routing

• Distance Vector (Bellman-Ford) vs. Link State (Dijkstra)

• Circuit Switching vs. Packet Switching

• Email Protocols: SMTP, POP3, IMAP4

• Domain Name System (DNS) Hierarchy & Queries

• FTP (File Transfer) and HTTP (Web Services)

• Support Protocols: ARP and ICMP Error Reporting

• Final Summary: OSI Model Layers 1-7

Watch the full course on the freeCodeCamp.org YouTube channel (12-hour watch).

In this article, we’ll look at RTT, jitter, and packet loss as parts of a single timing system rather than separate metrics. You’ll start by understanding RTT and why it changes over time. Then you’ll learn how jitter emerges as an extra delay relative to a baseline. Finally, you’ll see how TCP uses this timing information to decide when delay turns into packet loss, with a focus on real protocol behaviour such as TLS and post-quantum TLS handshakes.

The goal is simple: to understand how timing turns into decisions.

Table of Contents

1. RTT (Round Trip Time)

   • Baseline RTT

   • Why Baseline RTT Matters

2. What is Jitter?

   • What Jitter Actually Means

   • Where Jitter Comes From

   • What Jitter Tells Us

3. How TCP Learns RTT and Jitter

   • TCP Does Not Know RTT in Advance

   • SRTT (Smoothed RTT)

   • RTTVAR (RTT Variance)

   • Why TCP Needs Both

4. How TCP Decides Packet Loss

   • TCP Never Sees a Packet Being Dropped

   • Retransmission Timeout (RTO)

   • Delay vs Packet Loss

   • How Jitter Turns Into Packet Loss

5. Conclusion

RTT (Round Trip Time)

Before talking about delay, jitter, or packet loss, we need to clearly understand what RTT is. If RTT is not clear, everything that comes after it becomes confusing.

RTT stands for Round Trip Time. It is the total time taken for a packet to travel from a client to a server and for the response to return to the client.

Assuming you send a packet and receive the reply after 50 milliseconds, then the RTT is 50 ms. That is the basic definition.

But here is the important part: RTT is not a fixed number. It changes all the time. Even when you communicate with the same server, RTT can change from one packet to the next. This happens because the network is always changing, and this can be because of any of these reasons:

• Packets waiting in queues

• Temporary congestion

• Scheduling inside routers

• Background traffic on the same path

Let’s assume you connect to Google, and you send a packet now. In practice, RTT can be measured using simple tools. One common way is the ping command, which sends a packet and measures how long it takes for the reply to return.

ping -n 1 google.com

And here, you get the reply after 50 ms.

RTT = 50 ms

That value is real, but it is incomplete. A single RTT measurement does not tell us whether the network path itself takes 50 ms, or whether the path is faster and the packet experienced a small temporary delay along the way.

For example, the actual path delay might be 49 ms, with an additional 1 ms spent waiting in a queue. From a single RTT value, there is no way to separate these effects. RTT only makes sense after multiple measurements.

So, let’s measure multiple RTT values.

Now you can see a pattern. From the measurements:

min RTT ≈ 18 ms
max RTT ≈ 19 ms
average RTT ≈ 18 ms

That is why RTT is not something you magically know from one packet. It is something you observe over time.

But now, at this point, an important question comes up: Which of these RTT values represents the real network path?

When RTT is measured repeatedly, the values are not consistent. Some RTT measurements are small, some are larger, and some suddenly jump due to temporary network conditions.

You may see something like this: small, small, small, small, BIG. That is why we need a reference point, which we can call the stable point. Without it, every RTT value looks equally confusing, and we cannot tell whether a packet was slow because the path itself is slow or because something temporary happened.

Baseline RTT

That stable point is the baseline RTT. Baseline RTT means the minimum RTT observed over time. This represents the RTT without temporary effects.

For example, in the above example, our repeated measurements show minimum RTT ≈ 18 ms. 18 ms becomes our baseline RTT. You can think of baseline RTT as the fastest possible RTT for a given path, representing the calm state of the network where packets do not wait in queues, there is no congestion, and no retransmissions occur.

In other words, baseline RTT reflects what the network is capable of when nothing unusual is happening. The best happy case that usually excludes temporary effects.

Why Baseline RTT Matters

Once we have a baseline RTT, individual RTT measurements stop feeling random. We can see when an RTT is close to the baseline, when it is higher than expected, and when extra delay has been introduced by temporary network conditions.

Without a baseline RTT, each RTT value stands alone, and comparison becomes guesswork. With a baseline in place, RTT values gain meaning, variation becomes visible, and we are finally able to reason about what causes RTT to increase, which naturally leads to jitter.

This is the point where we are finally ready to talk about what causes RTT to increase, which leads naturally to jitter.

What is Jitter?

Now comes Jitter. Once we have a baseline RTT, something important becomes clear. Most RTT values are not equal to the baseline. They are usually higher.

So the next natural question is: If baseline RTT shows the calm network, what is causing the RTT to increase in the other measurements?

That extra part is what we call jitter.

What Jitter Actually Means

Jitter is the extra delay added on top of the baseline RTT. In simple words, baseline RTT shows what the network can do when nothing is wrong, while jitter describes what happens when the network becomes busy, and packets experience extra delay.

So every observed RTT can be thought of like this: Observed RTT = Baseline RTT + Extra Delay

Example

Baseline RTT = 18 ms. That extra delay is jitter.

There are two important points to remember. First, jitter is always positive because a packet can be delayed but can never arrive faster than the baseline RTT. Second, baseline RTT acts as the reference point, which means jitter only exists relative to that baseline. Without a baseline, jitter has no meaning.

Where Jitter Comes From

Jitter appears when packets do not move immediately through the network.

This usually happens because of:

• Packets waiting in queues

• Routers delaying packets before forwarding

• Temporary congestion on links

• Retransmissions after drops

These effects are not constant. They come and go. Because of that, jitter is irregular and bursty. Sometimes it is very small, and at other times it can suddenly become large.

What Jitter Tells Us

At this stage, jitter is still just an observation. It tells us how unstable the network timing is and how often packets experience extra delay. We are not making decisions yet. We are only describing what the network is doing.

How TCP Learns RTT and Jitter

We have seen that:

• RTT changes over time

• Baseline RTT gives us a reference

• Jitter explains extra delay

So, up to this point, we have only been observing the network. But now a new question appears: If RTT keeps changing, and if delay and jitter exist, who is actually watching all this? More importantly, who decides when waiting is normal and when waiting becomes a problem?

This is the point where TCP enters the picture. TCP is the component that observes these timing changes and uses them to decide what to do next.

TCP Does Not Know RTT in Advance

TCP does not start with any knowledge of how long the path is, how stable the network will be, or how much delay to expect. It learns all of this dynamically while the connection is running.

TCP learns everything while the connection is running, only by looking at time. Every time TCP sends data and receives an acknowledgement, it gets one RTT sample. Over time, these samples are used to build expectations.

To make sense of these timing samples, TCP maintains two internal values that summarize what it has learned so far.

SRTT (Smoothed RTT)

SRTT is the RTT that TCP expects most of the time. It is not the minimum RTT, and it is not a simple average. It is a smoothed value that represents the normal RTT that the TCP has learned from recent history. This means recent RTT measurements matter more, while older RTT measurements gradually matter less.

Because of this, SRTT does not jump because of a single delayed packet. Instead, it adapts gradually as network conditions change.

For example, suppose TCP observes these RTT samples (in ms): 48, 50, 49, 51, 50.

Then TCP smooths these values into a stable expectation, such as: SRTT ≈ 50 ms.

You can think of SRTT as: The RTT that TCP believes is reasonable for this connection, based mostly on recent history. It is a memory-weighted average, biased toward recent RTT values.

RTTVAR (RTT Variance)

RTTVAR tells TCP how much RTT is changing. Now compare two situations.

Stable RTT

RTT samples: 49, 50, 51, 50

Because these values are close to each other, the variation is small, RTTVAR remains low, and TCP feels confident about its timing estimates.

Unstable RTT

RTT samples: 50, 52, 90, 48

Here, the sudden jump increases variation, causes RTTVAR to rise, and makes TCP less confident about its timing.

Why TCP Needs Both

SRTT alone is not enough for TCP to make reliable timing decisions. If TCP only knew that the RTT is around 50 ms, it would still have no way to tell whether delays are stable or whether sudden spikes are common.

RTTVAR fills this gap by capturing how much RTT changes over time. While SRTT tells TCP what RTT to expect under normal conditions, RTTVAR tells TCP how confident it should be in that expectation.

At this stage, TCP is still learning, not judging. It is building a timing model of the network.

So far, the network produces RTT variation, baseline RTT provides a calm reference, jitter explains extra delay, and TCP observes all of this using SRTT and RTTVAR.

TCP now has expectations. Only after this point does TCP start making decisions. And one of those decisions is packet loss.

How TCP Decides Packet Loss

Now that TCP has learned what RTT usually looks like and how much it varies, it has to answer one important question: How long should I wait before assuming a packet is gone?

This is where packet loss comes in.

TCP Never Sees a Packet Being Dropped

TCP does not see routers, queues, or links, and it does not know where packets go. TCP only sees time. It sends data and then waits.

If the acknowledgment arrives in time, everything is fine. If it does not, TCP must decide what to do next.

Retransmission Timeout (RTO)

To make this decision, TCP uses the Retransmission Timeout, or RTO. RTO is not random. It is computed from what TCP has already learned about network timing.

Conceptually, RTO is calculated as:

RTO=SRTT+max(𝐺,4×RTTVAR)

Here, SRTT sets the expected delay, while RTTVAR adds extra margin to account for jitter. As a result, RTO represents how long TCP is willing to wait, based on how uncertain the network timing is.

Suppose TCP has learned that the SRTT is 50 ms and the RTTVAR is 5 ms. In that case, the RTO becomes 70 ms.

RTO = 50 + 4 × 5
RTO = 70 ms

Now TCP behavior is simple:

• ACK arrives at 60 ms → delay

• ACK arrives at 75 ms → packet loss

The network is the same, and the packet is the same. Only the arrival time changes, and that alone leads to a different decision.

Delay vs Packet Loss

TCP logic is simple. If a packet arrives before the RTO expires, it is treated as delay. If it does not arrive before the RTO, TCP declares it lost.

This leads to an important rule: packet loss is a timing decision, not a certainty. The packet may still arrive later, but once the RTO expires, TCP has already acted.

How Jitter Turns Into Packet Loss

This behaviour connects directly to jitter. As long as jitter stays within the RTO window, packets are delayed but not considered lost. When jitter becomes large enough that delays cross the RTO boundary, TCP interprets delay as packet loss.

So packet loss does not always mean a packet disappeared. Often, it means the network timing has become too unpredictable.

Conclusion

At this point, everything connects: RTT, jitter, and packet loss are not separate network metrics. They describe different parts of the same timing process.

• RTT shows how long communication usually takes.

• Baseline RTT gives a stable reference.

• Jitter explains why delays change.

• Packet loss appears when that variation exceeds what the protocol can tolerate.

Once this flow is clear, network behavior stops feeling random. It becomes a matter of timing, uncertainty, and decisions.

Your IP address, location, and basic network details are visible to that server.

In many cases, this is fine. But there are situations where you may want more control over how your requests travel across the internet. This is where proxies come in.

A proxy acts as an intermediary between you and the internet.

Instead of your device connecting directly to a website, it sends the request to a proxy server. The proxy then forwards the request on your behalf and sends the response back to you.

From the website’s point of view, it’s the proxy that is making the request, not you.

Proxies are used for privacy, security, performance, testing, automation, and access control. They are common in companies, data centers, scraping systems, and even home networks.

To understand why proxies matter, it helps to first understand how internet requests normally work.

What We’ll Cover

• How internet requests work without a proxy

• Types of proxies

• Proxies vs VPNs

• Using a proxy in Python

• Proxy Use Cases

• How proxies affect performance and reliability

• How proxies are detected and blocked

• Security considerations when using proxies

• Conclusion

How Internet Requests Work Without a Proxy

When you type a website address into your browser, your computer resolves the domain name to an IP address using DNS. It then opens a connection directly to that server.

Your IP address is included as part of the network connection so the server knows where to send the response.

The server can log your IP address, infer your location, detect your network provider, and apply rules based on that information. Some websites restrict access by country.

Others rate-limit or block traffic from specific IP ranges. In automated systems, repeated requests from the same IP are often flagged as suspicious.

Without a proxy, all of this traffic is directly tied to your device or server. There is no separation layer.

Types of Proxies

Proxies come in several forms, each designed for different scenarios.

Forward proxies are the most common. These are used by clients to access external resources. Corporate networks often use forward proxies to control employee internet access.

Reverse proxies work in the opposite direction. They sit in front of servers rather than clients. Websites use reverse proxies to load balance traffic, terminate TLS, and protect backend systems.

Transparent proxies operate without explicit client configuration. They intercept traffic at the network level. These are often used by ISPs or enterprise networks.

Residential, datacenter, and mobile proxies differ based on where their IP addresses come from. Residential and mobile proxies appear like real user devices, while datacenter proxies come from cloud providers.

Proxies vs VPNs

Proxies and VPNs are often confused, but they solve different problems. A proxy usually works at the application level. You configure a browser, script, or tool to use a proxy, and only that traffic goes through it.

A VPN works at the operating system or network level. Once connected, all traffic from your device is routed through the VPN tunnel by default. This includes browsers, apps, and background services.

Another difference is encryption. Most VPNs encrypt traffic between your device and the VPN server. Many proxies don’t, unless you’re using HTTPS or a secure proxy protocol.

People sometimes compare proxies to a free VPN, especially when the goal is hiding an IP address. While both can change your apparent location, a proxy is usually more lightweight and task-specific. A VPN is better when you want system-wide privacy, but it comes with more overhead and less fine-grained control.

For developers and automation systems, proxies are often preferred because they are easier to rotate, cheaper at scale, and simpler to integrate into code.

Using a Proxy in Python

Using a proxy in Python is straightforward, especially with popular libraries like requests. Below is a simple example that sends an HTTP request through a proxy.

To get a proxy URL, you can either build your own proxy using open-source solutions like SquidProxy or buy a third-party service that charges per GB of traffic. Here is a list of popular proxy providers.

import requests  # Import the requests library to make HTTP requests

# Proxy URL with authentication details
# Format: protocol://username:password@host:port
proxy_url = "http://username:password@proxy_host:proxy_port"


# Define proxy settings for both HTTP and HTTPS traffic
# Requests will route all outgoing traffic through this proxy
proxies = {
   "http": proxy_url,
   "https": proxy_url
}

# Make a GET request to httpbin.org, which returns the IP address
# This helps verify whether the request is going through the proxy
response = requests.get(
   "https://httpbin.org/ip",  # Test endpoint that echoes the client IP
   proxies=proxies,          # Apply the proxy configuration
   timeout=10                # Fail the request if it takes more than 10 seconds
)

# Print the response body
# If the proxy is working, the IP shown here will be the proxy's IP, not yours
print(response.text)

In this example, the requests library sends the outbound request to the proxy instead of directly to the website. The website sees the proxy’s IP address. The response shows which IP was used, making it easy to verify that the proxy is working.

This same pattern applies to APIs, scrapers, and internal tools. More advanced setups rotate proxies per request or per session.

Proxy Use Cases

One of the most common reasons to use a proxy is IP masking. By routing traffic through a proxy, your real IP address is hidden from the destination server. This is useful for privacy, security testing, and bypassing IP-based restrictions.

Proxies are also used for geographic routing. If a service behaves differently in different countries, a proxy located in a specific region lets you see what users there experience.

In automation and scraping systems, proxies are essential. Sending thousands of requests from a single IP is a fast way to get blocked. Rotating proxies distribute traffic across many IPs, reducing detection.

Companies use proxies to monitor, filter, and log outbound traffic. This helps with compliance, security, and performance optimisation.

How Proxies Affect Performance and Reliability

Adding a proxy introduces an extra network hop, which can increase latency. A well-located, high-quality proxy can still be fast, but performance depends heavily on proxy capacity and distance.

Proxies can also improve performance in some cases. Caching proxies store responses and serve them locally for repeated requests. This reduces load on upstream servers and speeds up access.

Reliability depends on proxy health. If a proxy goes down, all traffic routed through it fails. This is why production systems often use proxy pools and health checks to automatically switch between proxies.

How Proxies Are Detected and Blocked

Websites often try to detect proxy usage. They analyse IP reputation, request patterns, headers, and behavioural signals. Datacenter proxies are easier to detect because their IP ranges are well-known.

Some proxies leak information through headers that reveal the original client IP. Poorly configured proxies are especially easy to spot.

To reduce detection, systems rotate IPs, randomise headers, simulate real browser behaviour, and use residential or mobile proxies. Detection and evasion is an ongoing arms race between websites and proxy users.

Security Considerations When Using Proxies

Not all proxies are trustworthy. When you route traffic through a proxy, that proxy can see your requests and responses. This means sensitive data should only be sent over encrypted connections.

Public or free proxies often log traffic, inject ads, or behave unpredictably. For serious use cases, dedicated or private proxies are safer.

In corporate environments, proxies are part of the security model. They enforce policies, block malicious destinations, and provide audit logs. In these cases, the proxy is a defensive tool rather than a privacy tool.

Conclusion

A proxy is a simple but powerful concept. By inserting an intermediary between a client and the internet, proxies change how requests appear, how traffic is controlled, and how systems scale.

They are used for privacy, testing, automation, compliance, and performance. While they are often mentioned alongside VPNs, proxies offer more targeted control and flexibility, especially for developers and infrastructure teams.

Understanding how proxies work at a request level helps you decide when to use them, how to configure them safely, and how to design systems that rely on them. Whether you are building a scraper, testing geo-specific behavior, or managing outbound traffic, proxies remain a core building block of the modern internet.

Hope you enjoyed this article. Find me on Linkedin or visit my website.

In recent times, there has been a steady rise in styling tools that you can use to elevate the visual appeal of your websites. Such tools include CSS frameworks, animation libraries, icon libraries, and typography libraries. These tools offer customization flexibility, responsiveness, and consistency.

One awesome thing about these styling tools is that they bundle stylistic effects in a file which you can access via a Content Delivery Network (CDN).

In this article, we’ll be looking extensively at CDNs, how they work, their different hosting methods, the differences between them, their pros and cons, and the best use-cases of the methods for your project.

Let's dive straight into it!

What we’ll cover:

1. What is a CDN?

   • How does a CDN work?

   • Real-world analogy to explain CDNs

2. Why are CDNs Important?

3. Different Ways to Use a CDN

   • Remote Hosted CDN

   • Locally Hosted CDN

   • Hybrid Hosted CDN

4. Conclusion

What is a CDN?

A CDN, or Content Delivery Network, is a system of distributed servers that delivers web content (like images, stylesheets, scripts and other resources) to users, reliably and efficiently.

How does a CDN work?

A CDN’s primary function is to cache and serve both static and dynamic web content to users. It achieves this by using the following:

• Origin Server: This is the main server where all content is originally hosted.

• Edge Servers: These are servers distributed in different geographical locations to serve web content to users closer to them.

• Caching: This is a way of storing content on edge servers to reduce repeated requests to the origin server.

• DNS Routing: This is the mechanism that reroutes users to the nearest edge servers based on their location.

This is what happens when a user clicks on a CDN link:

• If trying to access the resource for the first time, the request hits the origin server.

• The origin server sends the resource to the user as a response and also sends a copy to the edge server located geographically closest to the user.

• The edge server caches the copy.

• When the user wants to access the resources again, the edge server (not the origin server) sends over the cached copy.

Real-World Analogy to Explain CDNs

To further explain how the CDN works, I’ll give an analogy to make it clearer. Imagine having an account with a bank that has its headquarters in New York (origin server).

You don’t expect customers who live far away from New York to be scampering to the headquarters any time they encounter issues. Instead, the bank provides branches (edge servers) in different locations to cater to the needs of their customers. Customers can easily walk into any branch nearest to them and get their issues or transactions sorted.

The branches have every customer's account information and transaction logs (cached data). Every branch of that bank delivers the same services and can satisfy their customers regardless of their distance from the headquarters.

The distribution of branches in different locations helps reduce the traffic that would have caused delays if the bank had just the headquarters as the only option. On contacting the bank's customer service on an issue, you would most likely be redirected to the nearest bank branch to you (DNS Routing)!

Why are CDNs Important?

There are many reasons why CDNs many websites use CDNs these days. Some of the key benefits are:

1. Improved Website Performance: CDNs can compress files and optimize images automatically, which helps speed up load time.

2. Efficient Resource Storage: With CDNs, styling resources are stored and managed properly. Resources are also stored in files matching their content types.

3. Better Search Engine Optimization (SEO): Using CDNs directly speeds up load time which in turn impacts search engine rankings. Google considers site speed a key metric that allows web pages to show up higher in search engines.

4. Better User Experience: Users prefer faster and more responsive websites to slow and unresponsive ones. With a better user experience, a website is sure to receive more engagement and lower bounce rates.

Different Ways to Use a CDN

There are three ways you can access CDN resources in your project:

• Remote Hosting

• Local Hosting

• Hybrid Hosting

Remote Hosted CDN

Remote CDN links allow developers to access styling resources from a third-party server by simply linking to the CDN in their HTML files via the link or script tag.

Bootstrap, for example, has two primary CDN links – one for CSS stylesheet and another for JavaScript (handles dynamic interaction like dropdowns, pop-overs, and so on).

To use a Bootstrap stylesheet in your project, you need to add this single line - https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css

And for the JavaScript: https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.bundle.min.js

How to find remote CDN links

The best way to identify thte remote CDN links of your desired styling tool is to visit their official documentation site and look for the direct links there.

Advantages of a remote CDN:

1. Easy to Use: You don’t need to download, manage, or upload files with CDN links. All you need to do is insert a single line of code in your HTML file and you are good to go.

2. Global Caching: whenever you visit a website that uses CDN links, it downloads the resources on loading the page and then saves it in your browser cache. On subsequent visits to the same website or other sites that use the same CDN, it fetches from the cache and displays these resources more quickly. This is one of the biggest advantages of using CDN links, as it improves website load time.

3. Optimized Global Delivery: CDNs are built to deliver content to users around the world by serving files from edge servers closest to users. This helps reduce the time it takes data to transfer across a network, also known as latency, and boosts performance for international users.

4. Reduced Server Load: Due to the CDN fetching data from an external source, the load on your server is reduced, which is helpful for high-traffic websites.

5. Real-Time Updates: Companies that own CDN links carry out periodic bug fixing, security patching, and feature updates, which can be beneficial for your project. These updates are reflected as soon as they are released.

Disadvantages of a remote CDN:

1. Customization Limitations: Styling components in remote CDNs are standard and unmodified. To modify them, you would have to override the specific styles in your local file which can introduce complexities.

2. No Control over Updates: When automatic updates are made, they can cause problems in your web applications. If the changes being introduced include drastic changes, it can affect your website’s layout or behavior in a big way.

3. Dependency on Third-Party Availability: If the CDN service experiences hitches like downtime or slowness, it can lead to broken styles thus, affecting your site’s performance negatively.

4. Privacy and Security Concerns: Links referencing an external source can pose serious security concerns, as they can be used to track users and get vital information. It is important to include only trusted CDN link sources in your web project to avoid breaches.

Locally Hosted CDN

These are CDN resources downloaded from a remote CDN and saved within your project folder or hosted on a local server. This approach allows you to have full control over the resources.

How to host CDN resources locally:

Hosting locally is straightforward and easy. All you have to do is:

• Access the resource by navigating to the CDN link URL (for example, https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css).

• Copy the code found in the URL.

• Create a file with the appropriate file extension (.css, .js) within your project folder.

• Paste and save.

• Reference the file in your HTML document.

If you follow these steps, you should be able to localize the CDN resource and access its styling locally.

Advantages of a locally hosted CDN:

1. Full Control over Files: With resources resident within your project folder/server, you’re in full control of your files as there will be no unexpected outages, changes, or updates that might break your site.

2. Offline Availability: Local hosting ensures that your styling resources are always available, especially for users with little or no network. This is perfect for building Progressive Web Applications (PWAs).

3. Customization: There’s be no need for overrides, as your styling can be modified from within your project files.

4. Security: By localizing CDN resources, you reduce the risk of potential third-party attacks on your project to the barest minimum.

Disadvantages of a locally hosted CDN:

1. No Global Caching: There is no benefit of global caching when resources are hosted locally. This will result in slower load time for first-time visitors.

2. Increased Server Load: With files residing locally, the load on the server increases especially as traffic increases. This approach puts a burden on the server and its capacity has to be considered.

3. Manual Updates: While hosting locally gives you control over updates, you’ll need to manually track and apply updates to your stylesheets when necessary. Also, missing security updates could make your site vulnerable.

4. Regional Performance Impact: If your server is located in a specific region, users from faraway places may encounter slower load times because the content has to travel greater distances.

Hybrid Hosted CDN

This approach involves the combination of using both the remote link and local hosting of CDN resources. A hybrid approach – which involves using remote CDNs for core libraries and local hosting for custom stylesheets – may strike the perfect balance between performance and control.

Best Approach to Use

The decision between remote and local hosting of of your CDN styling resources depends on factors such as project performance needs, user base, and security. Your choice should depend on the suitability of the approach to your project and should elevate the performance levels.

Best use-cases for a remote CDN:

1. Global User Base: If your website is to be accessed by a large, globally distributed audience, using the remote option would work best due to its performance and caching advantage.

2. Fast Integration: In a situation where you want to develop and deploy a project in the shortest period of time, using the remote CDN link is quick and easy.

3. Low-Traffic Website: Small projects such as portfolio sites and blogs are better served using a remote CDN link so as not to put a strain on the server. It also leads to easier implementation.

Best use-cases for a locally hosted CDN:

1. High Security Needs: For applications that require tight security due to the sensitivity of their operations, hosting the CDN resources locally will reduce third-party risks and vulnerabilities.

2. Offline Applications: For web applications that work offline, localizing styling resources would be the best option.

3. Customization Requirements: If you need to create your tailored styling versions, hosting them locally is the best option.

Conclusion

In this guide, you’ve learned what a CDN is, how you can host your CDN, and some of the benefits, drawbacks, and best use-cases for each approach.

Remote CDNs provide speed, convenience and reduced server load, while local hosting offers greater control, security, and customization options.

Ultimately, the best approach depends on your specific use case, audience, and priorities.

You can connect with me on LinkedIn or X for more frontend-related posts and articles.

See you on the next one!

In this article, I want to share with you my findings. The whole process was fun and gave me ideas on how to use these tools to tackle similar problems in the future.

What is NFS?

Network File System (NFS) is a distributed file system protocol that allows a user to access files over a computer network.

Please note that this is not a full tutorial on NFS. For that, please take a look at the following tutorial.

In this article, we will focus only on detecting access to a shared drive using several techiques as well setting up two servers and one client.

Also, I do use a different OS to set up both the server and the client, so instructions on how to do the task change a little bit.

How to Set Up a NFS Server and Client

My lab setup has one NFS server and two clients:

On my setup, I will have three computers talking to each other. One of them will be the NFS server and the other two will be a client.

How to Configure the Server

I will prepare my OrangePI machine to be the NFS server. Do so, I will enter the following commands:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install nfs-kernel-server -y
sudo systemctl enable nfs-kernel-server.service --now

Next step is to tell the server we want to share.

For that, we will edit the /etc/exports file (sudo vi /etc/exports history):

/data *(ro,all_squash,async,no_subtree_check)

Please check the man page to understand what these options mean.

In a nutshell, export /data:

• Is read-only

• Maps IDs to anonymous ID

• This option allows the NFS server to violate the NFS protocol and reply to requests before any changes made by that request have been committed to stable storage

• This option disables subtree checking. It's the default.

Now it is time to activate our shared directories:

root@orangepi5:~# sudo exportfs -a
root@orangepi5:~# sudo showmount -e
Export list for orangepi5:
/data (everyone)

I did something similar to the other host, raspberrypi:

root@raspberrypi:~# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
/var/log/suricata *(ro,all_squash,async,no_subtree_check)
root@raspberrypi:~# showmount -e
Export list for raspberrypi:
/var/log/suricata *

How to Configure the Client

First thing is to confirm we can indeed see the shared mount points from our server:

(tutorials) [josevnz@dmaf5 SpyOnNfs]$ sudo showmount -e orangepi5
Export list for orangepi5:
/data raspberrypi,dmaf5

Data is shared with two machines – just what we expected.

Now, there are several ways to mount this drive. One of them is manually, another one is at startup, and the last one, my preferred one, is on demand.

How to Set Up the AutoMount Client on Fedora Linux

First we set the service:

sudo dnf install -y autofs
sudo systemct enable autofs.service --now

Then we set this up, so we end mounting remote /data into local /misc/data. For that, sdd the following line to your /etc/auto.master:

[root@dmaf5 ~]# vi /etc/auto.misc
# After editing the file, adding our entry to the last line of the file ...
[root@dmaf5 ~]# cat /etc/auto.misc
#
# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# Details may be found in the autofs(5) manpage

cd              -fstype=iso9660,ro,nosuid,nodev :/dev/cdrom

data            -ro,soft,rsize=16384,wsize=16384 orangepi5:/data
suricata        -ro,soft,rsize=16384,wsize=16384 raspberrypi:/var/log/suricata

Restart the service one more time:

[root@dmaf5 ~]# systemctl enable autofs.service --now

And the smoke test:

[root@dmaf5 ~]# ls -l /misc/data
total 0
drwxrwxr-x. 1 root 1001 48 Apr  7 17:57 nexus
[root@dmaf5 ~]# ls /misc/suricata
certs       eve.json.7  files            http.log    stats.log.1     suricata.log.2        suricata-start.log.3  tls.log.4
core        fast.log    http-data.log    http.log.1  stats.log.2     suricata.log.3        suricata-start.log.4  tls.log.5
eve.json    fast.log.1  http-data.log.1  http.log.2  stats.log.3     suricata.log.4        suricata-start.log.5  tls.log.6
eve.json.1  fast.log.2  http-data.log.2  http.log.3  stats.log.4     suricata.log.5        suricata-start.log.6  tls.log.7
eve.json.2  fast.log.3  http-data.log.3  http.log.4  stats.log.5     suricata.log.6        suricata-start.log.7
eve.json.3  fast.log.4  http-data.log.4  http.log.5  stats.log.6     suricata.log.7        tls.log
eve.json.4  fast.log.5  http-data.log.5  http.log.6  stats.log.7     suricata-start.log    tls.log.1
eve.json.5  fast.log.6  http-data.log.6  http.log.7  suricata.log    suricata-start.log.1  tls.log.2
eve.json.6  fast.log.7  http-data.log.7  stats.log   suricata.log.1  suricata-start.log.2  tls.log.3

Now we are ready to play with our service.

How to Create a Python Program that Reads Files into the NFS Server

For our example, we want to determine if a Python application is reading data from this directory. This script has two features:

• Performs a one time read view of a file. This will teach us how to capture this type of scenerarios, when a file is not opened all the time.

• And the script also follows updates on a file periodically.

Here is how our test script looks like in action:

./scripts/test_script.py \
--quick_read /misc/data/nexus/log/jvm.log \
--follow /misc/suricata/eve.json \
--verbose
...
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,890 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,890 <dependency_failed type='unique_concrete_method' ctxk='java/io/ByteArrayOutputStream' x='java/io/ByteArrayOutputStream write ([BII)V' witness='sun/security/ssl/HandshakeOutStream' stamp='66511.855'/>
2023-09-10 14:48:22,890 <dependency_failed type='unique_concrete_method' ctxk='java/io/ByteArrayOutputStream' x='java/io/ByteArrayOutputStream write ([BII)V' witness='sun/security/ssl/HandshakeOutStream' stamp='66511.855'/>
...
# Ctrl-C to exit

The code, written in Python, is pretty simple:

#!/usr/bin/env python
"""
Simple script to simulate light activity on NFS drives
Author Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import concurrent
import os
import time
from concurrent.futures import ThreadPoolExecutor, ALL_COMPLETED
from pathlib import Path
from argparse import ArgumentParser
import logging

logging.basicConfig(format='%(asctime)s %(message)s', encoding='utf-8', level=logging.DEBUG)


def forever_read(the_file: Path, verbose: bool = False):
    for line in continuous_read(the_file=the_file):
        if verbose:
            logging.warning(line.strip())


def continuous_read(the_file: Path):
    """
    Continuously read the contents of file
    :param the_file:
    :return:
    """
    with open(the_file, 'r') as file_data:
        file_data.seek(0, os.SEEK_END)
        while True:
            line = file_data.readline()
            if not line:
                time.sleep(0.1)
                continue
            yield line


def quick_read(the_file: Path, verbose: bool = False):
    """
    Red the whole file and close it once done
    :param verbose:
    :param the_file:
    :return:
    """
    with open(the_file, 'r') as file_data:
        for line in file_data:
            if verbose:
                logging.warning(line.strip())


if __name__ == "__main__":
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        '--verbose',
        action='store_true',
        default=False,
        help='Enable verbose mode'
    )
    PARSER.add_argument(
        '--quick_read',
        type=Path,
        required=True,
        help='Read a file once'
    )
    PARSER.add_argument(
        '--follow',
        type=Path,
        required=True,
        help='Read a file continuously'
    )
    OPTIONS = PARSER.parse_args()
    try:
        with ThreadPoolExecutor(max_workers=3) as tpe:
            futures = [
                tpe.submit(forever_read, OPTIONS.follow, OPTIONS.verbose),
                tpe.submit(quick_read, OPTIONS.quick_read, OPTIONS.verbose)
            ]
            concurrent.futures.wait(futures, return_when=ALL_COMPLETED)
    except KeyboardInterrupt:
        pass

Now, let's go over how we can see if our script is indeed accessing an NFS partition.

Common steps

First we need to learn where to look for. So on the machine, check for NFS in /etc/fstab (for mount points that are available since the machine was rebooted):

[root@dmaf5 ~]# rg -e 'rsize=' /etc/fstab

Then on the AutoMount files:

[root@dmaf5 ~]# rg -e 'rsize=' /etc/auto*
/etc/auto.misc
17:data            -ro,soft,rsize=16384,wsize=16384 orangepi5:/data
18:suricata        -ro,soft,rsize=16384,wsize=16384 raspberrypi:/var/log/suricata

The regular expressions are not exact science, but you get the idea what to look for next.

How to Use the tools

We need to confirm if there was access to any of the following partitions mounted over NFS:

• /misc/data

• /misc/suricata

Next, I will show you a set of tools that will make the task easier, each one of them with their own strength and limitations.

Starting with lsof and ripgrep combined.

How to Use Lsof and rg for Capturing and Filtering

[josevnz@dmaf5 docs]$ lsof -w -b| rg -e '/misc/data|/misc/suricata'
python    36509                 josevnz    3   unknown                           /misc/suricata/eve.json
python    36509 36510 python    josevnz    3   unknown                           /misc/suricata/eve.json
python    36509 36511 python    josevnz    3   unknown                           /misc/suricata/eve.json

I passed the -b option to lsof to avoid it from getting stuck, in case the NFS handle is stale.

A few things about lsof:

• If you are using Autofs, you should know than mount points eventually get un-mounted to save bandwidth. This can be problematic when trying to catch the access of a file that is only opened once.

• The short-lived read didn't show up because the filehandle was closed after we inspected the process.

• If you want to monitor ALL the processes on this machine, you may need to run as root. You can only inspect your own processes without special privileges.

Still, lsof is a great tool to investigate.

Next strategy involves monitoring from the beginning, to catch the elusive short-read. We will use strace.

How to Use strace

sudo dnf install -y strace
(tutorials) [josevnz@dmaf5 SpyOnNfs]$ strace -f ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json 2>&1| rg -e '/misc/data|/misc/suricata'
execve("./scripts/test_script.py", ["./scripts/test_script.py", "--quick_read", "/misc/data/nexus/log/jvm.log", "--follow", "/misc/suricata/eve.json"], 0x7ffd9ae29738 /* 46 vars */) = 0
execve("/home/josevnz/virtualenv/tutorials/bin/python", ["python", "./scripts/test_script.py", "--quick_read", "/misc/data/nexus/log/jvm.log", "--follow", "/misc/suricata/eve.json"], 0x7ffe269dbf88 /* 46 vars */) = 0
[pid 38241] openat(AT_FDCWD, "/misc/suricata/eve.json", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 38242] openat(AT_FDCWD, "/misc/data/nexus/log/jvm.log", O_RDONLY|O_CLOEXEC <unfinished ...>

The openat(AT_FDCWD) entries give away the two files our script is reading from NFS. But as you can tell this approach has some caveats:

• We are filtering the output. It is best to save the output to a file with 'tee' and then search there

• It requires starting the process with strace from the beginning. Yes, you could do a 'strace -p $PID' to attach later to the process, but you risk missing short-lived reads

Is there a different way? Time to move on to the next tool, tshark and see how to use a network capture to confirm access to the share.

How to Use tshark

We can also capture the network traffic and filter out only NFS. It is not perfect, but it may be sufficient.

First, find out which network interface is used to communicate with the NFS server. In my case it is easy – they all connected using a wired private network:

[josevnz@dmaf5 docs]$ ip --oneline address|rg -e 'eno|wlp'
3: eno1    inet 192.168.68.70/22 brd 192.168.71.255 scope global dynamic noprefixroute eno1\       valid_lft 4568sec preferred_lft 4568sec
4: wlp4s0    inet 192.168.1.95/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp4s0\       valid_lft 3423sec preferred_lft 3423sec
4: wlp4s0    inet6 fe80::ac40:5365:7f09:a5d2/64 scope link noprefixroute \       valid_lft forever preferred_lft forever

For this example it is eno1 with IP address '192.168.68.70'. Then capture the traffic, and with some luck we will get the file path:

[root@dmaf5 ~]# tshark -i eno1 -Y "nfs"
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eno1'
 ** (tshark:42326) 16:02:47.417145 [Main MESSAGE] -- Capture started.
 ** (tshark:42326) 16:02:47.417286 [Main MESSAGE] -- File: "/var/tmp/wireshark_eno1rEGxiu.pcapng"
   13 1.601197994 192.168.68.70 → 192.168.68.60 NFS 450 V4 Call GETATTR FH: 0x90ba4ee1  ; V4 Call GETATTR FH: 0x90ba4ee1
   14 1.601374466 192.168.68.70 → 192.168.68.60 NFS 258 V4 Call GETATTR FH: 0x90ba4ee1
   15 1.601395155 192.168.68.70 → 192.168.68.60 NFS 258 V4 Call GETATTR FH: 0x90ba4ee1
   16 1.602155254 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 13) GETATTR
   17 1.602368826 192.168.68.60 → 192.168.68.70 NFS 554 V4 Reply (Call In 13) GETATTR  ; V4 Reply (Call In 14) GETATTR
   19 1.602515091 192.168.68.70 → 192.168.68.60 NFS 274 V4 Call READ StateID: 0xa902 Offset: 57552896 Len: 12288
   20 1.602557170 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 15) GETATTR
   22 1.603156327 192.168.68.60 → 192.168.68.70 NFS 1730 V4 Reply (Call In 19) READ
   66 4.611124808 192.168.68.70 → 192.168.68.60 NFS 642 V4 Call GETATTR FH: 0x90ba4ee1  ; V4 Call GETATTR FH: 0x90ba4ee1  ; V4 Call GETATTR FH: 0x90ba4ee1
   67 4.611301059 192.168.68.70 → 192.168.68.60 NFS 258 V4 Call GETATTR FH: 0x90ba4ee1
   68 4.611809385 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 66) GETATTR
   69 4.611887552 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 66) GETATTR
   71 4.611976479 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 66) GETATTR
   72 4.620685968 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 67) GETATTR
   74 5.017200005 192.168.68.70 → 192.168.68.60 NFS 250 V4 Call GETATTR FH: 0x9419c00c
   75 5.017804843 192.168.68.70 → 192.168.68.59 NFS 242 V4 Call GETATTR FH: 0x314e720f
   76 5.017838787 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 74) GETATTR
   77 5.018131217 192.168.68.70 → 192.168.68.60 NFS 326 V4 Call OPEN DH: 0x90ba4ee1/
   78 5.018711408 192.168.68.60 → 192.168.68.70 NFS 386 V4 Reply (Call In 77) OPEN StateID: 0x9984
   79 5.018855699 192.168.68.59 → 192.168.68.70 NFS 310 V4 Reply (Call In 75) GETATTR
   81 5.018980434 192.168.68.70 → 192.168.68.59 NFS 262 V4 Call GETATTR FH: 0xecd332cc
   82 5.019934959 192.168.68.59 → 192.168.68.70 NFS 310 V4 Reply (Call In 81) GETATTR
   83 5.020032853 192.168.68.70 → 192.168.68.59 NFS 262 V4 Call GETATTR FH: 0x261d4440
   84 5.020734032 192.168.68.59 → 192.168.68.70 NFS 310 V4 Reply (Call In 83) GETATTR
   85 5.020874175 192.168.68.70 → 192.168.68.59 NFS 330 V4 Call OPEN DH: 0xc9b4831b/

This is great, there is activity against two NFS servers, 192.168.68.59 and 192.168.68.60. But, is there a way to see the name of files?

tshark has a way to spit information by field. The problem is that NFS has lots of them:

[root@dmaf5 ~]# for field in $(tshark -G fields| cut -d'        ' -f3|rg -e '^nfs\.'); do echo "-e $field"; done|head -n 10
Running as user "root" and group "root". This could be dangerous.
-e nfs.unknown
-e nfs.svr4
-e nfs.knfsd_le
-e nfs.nfsd_le
-e nfs.knfsd_new
-e nfs.ontap_v3
-e nfs.ontap_v4
-e nfs.ontap_gx_v3
-e nfs.celerra_vnx
-e nfs.gluster

So, let's capture them into a variable (also need to enable some options):

[root@dmaf5 ~]# fields=$(for field in $(tshark -G fields| cut -d'       ' -f3|rg -e '^nfs\.'); do echo "-e $field"; done)
[root@dmaf5 ~]# tshark -i eno1 --enable-protocol nfs -o nfs.file_name_snooping:true -o nfs.file_full_name_snooping:true -T fields -E header=y -E separator=, -E quote=d $fields
Running as user "root" and group "root". This could be dangerous.
nfs.unknown,nfs.svr4,nfs.knfsd_le,nfs.nfsd_le,nfs.knfsd_new,nfs.ontap_v3,nfs.ontap_v4,nfs.ontap_gx_v3,n...

I managed to get the filename only once, then after interrupting and restarting the program I got no luck.

And yet no sign of the file name. The file handle was in the contents but this is not very useful if you want a quick way to see what was accessed.

Is there an easier way to do this? Sysdig may offer some answers.

How to Use Sysdig

While trying to find the elusive mount points, I stumbled into Sysdig:

Sysdig instruments your physical and virtual machines at the OS level by installing into the Linux kernel and capturing system calls and other OS events. Sysdig uses DTrace to get access to the system kernel.

Sysdig also makes it possible to create trace files for system activity, similarly to what you can do for networks with tools like tcpdump and Wireshark.

I decided to use the latest version (0.33.1) for Fedora 37 where my script is running):

sudo dnf install -y https://github.com/draios/sysdig/releases/download/0.33.1/sysdig-0.33.1-x86_64.rpm
# Wait a little bit, as a kernel module needs to be compiled and prepared...
Installed:
  bison-3.8.2-3.fc37.x86_64                    dkms-3.0.11-1.fc37.noarch          elfutils-libelf-devel-0.189-3.fc37.x86_64  flex-2.6.4-11.fc37.x86_64            kernel-devel-6.4.13-100.fc37.x86_64 
  kernel-devel-matched-6.4.13-100.fc37.x86_64  libzstd-devel-1.5.5-1.fc37.x86_64  m4-1.4.19-4.fc37.x86_64                    openssl-devel-1:3.0.9-1.fc37.x86_64  sysdig-0.33.1-1.x86_64              
  zlib-devel-1.2.12-5.fc37.x86_64

How easy is to probe out the script so it is indeed accessing the NFS mounted directories? Let's print three fields of interest and the name of the accesed file:

# `sysdig -l` will output every single field you can capture
[root@dmaf5 ~]# sysdig -p"%proc.cmdline,%fd.name" proc.name contains python and fd.name contains /misc
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
...

What if you want to capture all the data, and filter later? One way to do it is capturing to a file:

# Capture for one minute...
[root@dmaf5 ~]# timeout --preserve-status 1m sysdig -w /tmp/sysdig.dump
[root@dmaf5 ~]# ls -lh /tmp/sysdig.dump
-rw-r--r--. 1 root root 32M Sep 10 19:03 /tmp/sysdig.dump

And then replay the contents, with filtering (replay doesn't need elevated privileges):

[root@dmaf5 ~]# sysdig -r /tmp/sysdig.dump -p"%proc.cmdline,%fd.name" proc.name contains python and fd.name contains /misc|sort -u
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json

Sysdig supports scripting, using the LUA language. For example, it has a very convenient version of lsof:

[root@dmaf5 ~]# sysdig -cl|rg lsof
lsof            List (and optionally filter) the open file descriptors.

So let's use it:

[root@dmaf5 ~]# sysdig -c lsof|rg misc
automount           52410   52410   root    8       directory   /misc
automount           52410   52413   root    8       directory   /misc
automount           52410   52414   root    8       directory   /misc
automount           52410   52415   root    8       directory   /misc
automount           52410   52418   root    8       directory   /misc
automount           52410   52421   root    8       directory   /misc
python              75840   75840   josevnz 3       file        /misc/suricata/eve.json
python              75840   75841   josevnz 3       file        /misc/suricata/eve.json
python              75840   75842   josevnz 3       file        /misc/suricata/eve.json

What I liked about this tool:

• Can work with older kernels (like 4.xx)

• Has a powerful expression language for filtering

• Easy to learn and well documented

• You can write your own scripts if you know LUA

Before finishing up let's look at one more tool, BPF.

How to Use BPF probe

Originally Berkeley Packet Filter, is a kernel and user-space observability scheme for Linux.

The BPF is a very powerful tool, and this short article won't even scratch the surface.

Yes, this is huge. I'm learning this myself.

I found that the bcc repository has lots of ready to use scripts that we could use to track our NFS access, and even check for performance (you can find more examples here, and on the BPF Performance Book repository).

But it is more interesting to write tools yourself that monitor pretty much anything you want. For this tutorial, I will use some ready to use programs that use the traces to capture useful information.

As a first step, we will need to install a high level interpreter for our scripts. Again, on my Fedora Linux machine:

[josevnz@dmaf5 ~]$ sudo dnf install -y bpftrace.x86_64 bcc-tools.x86_64
# And check if the kernel has btf enabled
[josevnz@dmaf5 ~]$ ls -la /sys/kernel/btf/vmlinux
-r--r--r--. 1 root root 5635179 Sep 12 04:21 /sys/kernel/btf/vmlinux

On a separate terminal run again the NFS test script:

. ~/virtualenv/tutorials/bin/activate
cd SpyOnNfs/
./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose

You can trace all the files opened by a program, like top:

18:59:20 loadavg: 1.20 1.00 0.74 1/1175 28520

TID     COMM             READS  WRITES R_Kb    W_Kb    T FILE
28520   clear            2      0      60      0       R xterm-256color
28203   python           7      0      56      0       R eve.json
28347   filetop          2      0      15      0       R loadavg
824     systemd-oomd     2      0      8       0       R memory.swap.current
824     systemd-oomd     2      0      8       0       R memory.low
...

But it doesn't print the full path. It's more useful to ask a NFS snoop and see if one of our files shows up:

[josevnz@dmaf5 SuricataLog]$ sudo /usr/share/bcc/tools/nfsslower 1
# Commented out some warnings ...
Tracing NFS operations that are slower than 1 ms... Ctrl-C to quit
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
19:02:25 python         28202  R 1460    62150       1.96 eve.json
19:02:28 python         28202  R 2446    62151       2.09 eve.json
19:02:31 python         28202  R 970     62154       1.99 eve.json
19:02:34 python         28202  R 3335    62155       2.43 eve.json
19:02:37 python         28202  R 4564    62158       1.84 eve.json
19:02:40 python         28202  R 5876    62162       1.89 eve.json
19:02:43 python         28202  R 4504    62168       1.61 eve.json
19:02:46 python         28202  R 3131    62173       1.92 eve.json

This is much better. Also, we can see than the latency is almost two milliseconds.

We can also monitor mount/ umount operations:

[josevnz@dmaf5 SuricataLog]$ sudo /usr/share/bcc/tools/mountsnoop 
# Commented out some warnings ...
2 warnings generated.
COMM             PID     TID     MNT_NS      CALL
mount.nfs        29012   29012   4026531841  mount("orangepi5:/data", "/misc/data", "nfs", MS_RDONLY, "sloppy,soft,rsize=16384,wsize=16384,vers=4.2,addr=192.168.68.59,clientaddr=192.168.68.68") = 0

This is good as well, we can see the activity over NFS we wanted to confirm.

Next Steps

You learned several tools and as you may have guessed, you can use them to snoop on more than just opened files on NFS.

It is always useful to know more than one tool. Sysdig has a special mention for being very versatile, powerful and yet easy to use. Also, it can be extended with scripts written in the LUA language.

BPF is another alternative and will give you incredible access to the kernel calls. Be prepared to spend time reading and learning how to use the tools.

The code for the scripts used on this tutorial can be obtained from my GitHub repository: SpyOnNfs.

But what do these acronyms mean?

Well, TCP stands for Transmission Control Protocol and UDP stands for User Datagram Protocol

Ok, but what are they? Why are they useful? Why is it thanks to these protocols that the internet works?

In this article, we'll start with a simple analogy to help you understand the ideas behind TCP and UDP. Then I'll explain how they work in plain English. Next we'll discuss why these protocols are important, and we'll understand their key differences.

What are TCP and UDP? Explained with an Analogy

Photo by Suzy Hazelwood from Pexels: https://www.pexels.com/photo/shallow-focus-of-letter-paper-1157151/

You can think about TCP and UDP in a similar way as you think about a mail service.

Just like a mail service sends letters to individuals, TCP and UDP send packets to computers.

Packets are just formatted units of data in bytes.

TCP works more like a registered mail service. With registered mail, the sender sends a parcel to a recipient, and they're notified when the parcel is delivered. The sender can also track the parcel if it is lost or delayed.

On the other hand, UDP is like a regular mail service. The sender sends a parcel without any confirmation of delivery. There is also no tracking or guarantee that the parcel is going to be delivered.

How Do TCP and UDP Work?

Both the UDP and TCP are protocols used to ensure that data is reliably and securely transmitted between devices over a network.

TCP creates a connection between the sender and receiver and then data is transmitted between packets.

TCP also ensures that all packets are delivered in order. Additionally, it has mechanisms for resending lost packets and managing flow control.

Meanwhile, UDP sends packets without establishing a connection. Because of this, packets can be lost or arrive out of order as there is no mechanism to request retransmission.

TCP vs UDP – What's the Difference?

Both UDP and TCP were developed starting in the 1980s.

TCP is important in programming because it provides a reliable and secure way for devices to communicate with each other over a network.

Without TCP, it would be difficult for devices to communicate securely, and many of the applications that we use today would not exist.

For example email communication, file transfers, and online transactions would all be very difficult without TCP.

TCP is preferred for secure, ordered, and reliable data transmission, as well as for delivering large amounts of data with minimal delay and mitigating network congestion.

UDP is also important for connections that require a lot broadband where security is not an issue.

For example video streaming services, online gaming platforms, and IoT devices can all use UDP to transmit data.

Conclusion

With these protocols, the internet can operate safely and devices can communicate with each other effectively.

These concepts are key to understand if you're interested in Computer Networking. If you want to learn more about networking, you can read this tutorial.

The internet has become an integral part of our lives, as it solves one of the most critical problems we have to handle: data transfer. We're constantly either receiving or sending data to someone – whether it's a social media app, news app, or whatever type it may be, there's some form of data transfer.

Because of this, it's super important to learn networking if you're learning mobile app development. In this article I'll be explaining how to build a super simple mobile app that fetches data from internet and renders it on the app.

How to Create the Project

Navigate to the folder where you want to create your project in the terminal and run the following command:

git clone https://github.com/5minslearn/Flutter-Boilerplate.git

Navigate to the Flutter-Boilerplate folder and run the flutter pub get command to install the dependencies.

cd Flutter-Boilerplate/
flutter pub get

That's it. We've got our dependencies installed.

Open the project in Visual Studio Code by running the code ./ command in the terminal.

Create a Flutter app from boiler plate

Start your emulator/connect your device and press F5 in VS Code to run your app.

At the moment, the app will just contain an empty screen as shown in the below screenshot.

Flutter app with empty screen

Let's build our networking app.

Where to Get the Data

This is the most obvious question. If we were to fetch something from the internet and render, we need an API server exposing the data we need. But, most people cannot afford to do that for learning purposes. To overcome this, many people are offering free API services.

You can consume the data from their API services for learning purposes. But, we cannot validate the originality of the data, as most of them will be random.

In this tutorial, we'll be using the API exposed by https://sampleapis.com/. They expose an API endpoint that lists Coding Resources. The URL of the endpoint is https://api.sampleapis.com/codingresources/codingResources.

In our app, we'll fetch the data from this endpoint and list them in our app.

Install the Dependencies

Let's install the dependencies we need to build this app. They are:

1. The http package
2. The url_launcher package

We'll use the http package to make a call to the API endpoint. And we'll use the url_launcher package to open a URL in an external browser.

Open the pubspec.yml file and add the following two packages in the dependencies section:

  http: ^0.13.6
  url_launcher: ^6.1.11

Add dependencies

If you're using VS Code as your IDE, the dependencies will be installed automatically by saving this file. For the other IDEs, run flutter pub get on your project root folder to install the dependencies.

How to Fetch the Data from the API

We got our dependencies ready. Let's make a request to our API and get the data.

Import the http package in the lib/main.dart file.

import 'package:http/http.dart' as http;

Initialize a list object in the _MyHomePageState class by adding the following code:

List _resources = [];

Let’s write a method that makes a call to our API endpoint and decode them into a JSON object.

  void _fetchResources() async {
    final response = await http.get(Uri.parse(
        'https://api.sampleapis.com/codingresources/codingResources'));
    if (response.statusCode == 200) {
      final data = json.decode(response.body) as List;
      setState(() {
        _resources = data;
      });
    } else {
      throw Exception('Failed to load resources');
    }
  }

Paste the above code into the MyHomePageState class. In the above method, we’re making a call to the API endpoint (https://api.sampleapis.com/codingresources/codingResources). From the response, we’re validating if it's successful in receiving the data by checking if its status code is 200 (and throwing an error if it’s not). We then decode the received data and save it in our app’s state.

You may notice an error after pasting the above code at the json.decode part. In order to decode JSON, we need import a convert package in our file. Add the following code at the top of the file:

import 'dart:convert';

The error should be gone now.

We have a method that makes a call to the API endpoint and saves the data in the state. Our next step is to trigger this method when we open the app.

We can do that by overriding the initState method.

  @override
  void initState() {
    super.initState();
    _fetchResources();
  }

Quoting from the Flutter documentation,

"initState is called when this object is inserted into the tree. The framework will call this method exactly once for each State object it creates."

In the above code, we called our method _fetchResources() in the initState() method.

How to Build the UI

We got the list of items whenever we open the app. Our next step is to render them on the UI.

Copy the below code and replace it with the build method of the _MyHomePageState class.

@override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: Text(widget.title),
        ),
        body: _resources.isEmpty
            ? const Center(
                child: CircularProgressIndicator(),
              )
            : ListView.builder(
                itemCount: _resources.length,
                itemBuilder: (context, index) {
                  final resource = _resources[index];
                  return InkWell(
                      onTap: () => {},
                      child: Card(
                          child: ListTile(
                        title: Text(resource['description']),
                        subtitle: Text(resource['url']),
                        leading: const CircleAvatar(
                            backgroundImage: NetworkImage(
                                "https://images.unsplash.com/photo-1547721064-da6cfb341d50")),
                        trailing: Text(resource['types'].join(', ')),
                      )));
                }));
  }

Let’s understand the above code.

We show a loader if our state has empty values. If it has some values in it, we’ll iterate through as a ListView builder and we render a card widget for each item, displaying the description, url, and types of the resource.

That’s it.

Run the app by pressing F5 and you should be able to see the following:

Loading the resources

Showing the resources

That’s awesome, isn’t it?

Let's Fix the Missing Part

But I feel there’s one small thing that’s missing at this point.

We’re able to see the list of the resources. But, we’re not able to view those resources. Some resources have a short link that we can easily remember and type. But, a few of them have a long URL which would be hard for a typical human being to remember. Let’s add a small enhancement that when we click on a resource, its link should be opened in our default browser.

This is very simple to implement in Flutter. This is the reason we added the url_launcher package at the beginning of this tutorial.

Import the url_launcher package in your app like this

import 'package:url_launcher/url_launcher.dart';

Add the following method in the _MyHomePageState class:

_launchURL(String url) async {
    if (await canLaunch(url)) {
      await launch(url);
    } else {
      throw 'Could not launch $url';
    }
  }

The above method accepts a URL, validates the link, and opens it in the browser.

We have to call this method on tapping the card. We can achieve that by calling this method in the onTap property of the InkWell widget.

Here’s the code for it:

onTap: () => {_launchURL(resource['url'])},

Let’s run our app and test this.

You were likely disappointed on tapping a card – I certainly was while working on this.

You should have seen this error:

Error on opening a url: Exception has occurred. "Could not launch https://www.youtube.com/bocajs"

Though the URL is right, why is our system not opening this in a browser?

What are Intent Actions?

Well, now we have to learn about intent actions.

Quoting it from Android Developers documentation,

"An Intent provides a facility for performing late runtime binding between the code in different applications. Its most significant use is in the launching of activities, where it can be thought of as the glue between activities. It is basically a passive data structure holding an abstract description of an action to be performed."

This basically means that when we hand over something to the external app, we have to declare that in our app. For Android, we have to define it in AndroidManifest.xml and for iOS most of these configurations go into the Info.plist file.

Add the queries block in the following code to your AndroidManifest.xml file.

<manifest>
    <application>
        ...
    </application>
    <queries>
        <intent>
            <action android:name="android.intent.action.VIEW" />
            <category android:name="android.intent.category.BROWSABLE" />
            <data android:scheme="https" />
        </intent>
    </queries>
</manifest>

Uninstall the app from your mobile and run the app again.

Hopefully you should be able to see the link opened in the browser.

Link opened on a browser

Conclusion

In this article, you’ve learned about networking in Flutter. We made a request to an API, rendered the list, and opened the URL in the browser.

This repo has my code. You can use it for your reference.

To learn more about Flutter, subscribe to my email newsletter on my site (https://5minslearn.gogosoon.com) and follow me on social media.

But before talking about VPCs directly, I'm going to take a minute or two to refresh your memories on the basics of TCP/IP networking architecture and NAT addressing.

But feel free to skip the next bit if you're already dug in on all that. When that's behind us, though, I'll show you what building VPCs in AWS looks like.

A Quick TCP/IP Primer

Ok. So TCP stands for Transmission Control Protocol and IP stands for Internet Protocol. If those sound a bit broad – almost as if they're trying to describe the totality of the internet – it's because just about everything we do on the internet is indeed controlled by these half century-old TCP/IP protocols.

This article comes from my Securing Your AWS EC2 Instances course. If you'd like, you can follow the video version here:

For our purposes, remember that every network-connected device must have a unique IP address. Given that, mathematically, there can be no more than four billion 32-bit IPv4 addresses, and that there are already far more than four billion network-connected devices on the internet, something had to change.

A typical IPv4 address:
192.168.2.45

The 128-bit IPv6 protocol was eventually introduced to allow trillions of unique addresses. We'll never run out of those.

A typical IPv6 address:
fd42:e265:3791:64f9:216:3eff:fe54:fcfe/64

But before IPv6, another brilliant solution was introduced: NAT networking.

What is Network Address Translation (NAT)?

The NAT protocol sets aside three network segments for use in private networks only. Using NAT, your home can have 15 or 20 devices – including laptops, smartphones on WiFi, network printers, routers, and maybe a smart fridge or two – but between them, they'll use up only a single public IP address.

How does that work? Well, your internet service provider will assign that one public IP to the modem they send you. But that modem will act as a DHCP server and assign each local device a private IP address from one of those three network segments.

A typical NAT architecture

The DHCP server will translate all requests moving between your local devices and internet-based services in a way that leaves those internet services thinking that your device's IP is actually the single public one.

But incoming network packets are actually delivered right to your device using its local NAT address. The system really is brilliant. And it added decades of life to the IPv4 system.

But once we have NAT in place anyway, it can actually do a whole lot more than just translate local addressing. Which brings us to why we're talking about this stuff in the context of EC2 instance security in the first place.

You see, NAT allows for very sophisticated network segmenting. By carefully configuring addressing and routing rules, you can turn a single local network into a multi-layered, highly-secured environment for mission-critical enterprise deployments.

Here's an illustration of a replicated environment using public and private subnets from AWS documentation.

How bastion hosts might fit into an AWS VPC

The only connection with the outside world, such a private subnet, would have flows through a bastion host and a NAT gateway living in an adjacent public subnet. Forcing everything through those two devices lets you precisely control traffic.

The bastion host provides a jump box allowing admins to safely open remote SSH sessions on instances running in your private subnets. And the NAT gateway allows services running on your private instances outbound access to, for example, pull software updates. Both bastion hosts and NAT gateways will incur regular usage costs, by the way.

How to Optimize VPC Networks

Now I'll show you how all this works within the AWS ecosystem. From the VPC dashboard I can click Create VPC. Right away I've got a decision to make: do I want to build just a simple VPC where I can host some resources, or something more complicated?

If I go with VPC only, I'll just need to give the VPC a name, select my addressing environments and I'm good to go.

Part of the VPC configuration interface in AWS

But we're here to test drive the VPC and more option. As you can see, that'll open up an infrastructure preview to the right that shows us exactly what will be created based on the current selections, and what they'll look like.

Right now, as you can see, we'll get one public and one private subnet in each of two availability zones, and appropriate route tables and network connections to make everything work.

Note how AWS automatically assigns logical names to new subnets

Note how each object is automatically given a name that properly reflects the naming structure I specified. All this automatically conforms with security and availability best-practices

I can choose to specify a dedicated tenancy for new instances launched into this VPC – although, as I mentioned earlier, that probably won't be relevant for all that many cases.

The number of availability zones you configure will reflect the depth of fault tolerance your application needs. The more zones, the less chance your application goes down.

Of course, by that same token, the more zones, the more instances you'll need to run and the more you'll spend. You can see that editing this value will have an instance impact on the subnet settings to the right.

You can similarly control the number of public and private subnets. The options available will reflect the number of availability zones you selected above. Basically, the UI makes it pretty much impossible to do something stupid – which is something I appreciate.

Adjusting the subnets within the UI will automatically update the configuration to the right. Selecting a NAT gateway will generate a new object with all the appropriate connectivity built-in.

One last really nice tool is the fields that let us fine tune the address allocation for our subnets. This could be important if you're planning to deploy, say, large numbers of virtual containers into a couple of public subnets but nothing more than a handful of database servers in your private subnets. You'll probably need a lot more addresses in the former and fewer in the latter.

Once you fire up your VPC, it'll take just a few seconds for all the parts to fall into place. When that's done, you'll be able to navigate to the Your VPCs dashboard to confirm what's now available.

Wrapping Up

We covered a lot of ground in this article. You learned how VPCs can be designed to employ sophisticated routing topologies that expose and block resources to efficiently meet your operational and security needs.

Thanks for reading!

Or, in sightly more accurate terms, it requires the combination of sophisticated engineering and the existence of key standards, including network ports, the domain Name System (DNS) and routing policies. And we're going to learn how some of that works in this article.

Besides IP addresses (identifiers that could look something like 192.168.2.54 or e80::1164:3e06:6716:7a0b/64), network traffic can be routed between devices or even individual services running on a device using TCP and UDP ports.

This article comes from The Complete LPI Security Essentials Exam Study Guide. You can also follow along using this video:

Routing Network Traffic Using Network Ports

Ports are numbers used to identify specific applications or services running on a device. When data is transmitted over the network, the source and destination ports are included in the data packet to ensure that the data is sent to the correct application or service.

There are 65,535 TCP and UDP ports in total, each of which can be assigned to a specific application or service for communication over the Internet.

The first 1024 ports are reserved for well-known services and are typically assigned to specific applications or services by the Internet Assigned Numbers Authority. For example, HTTP (Hypertext Transfer Protocol) uses port 80, while HTTPS (HTTP Secure) uses port 443.

The remaining ports (above 1024) are called ephemeral ports and can be used by any application or service that needs to communicate over the Internet. When a device initiates a connection, it selects an available ephemeral port to use for the duration of the connection.

Devices can also be identified by unique hardware Media Access Control addresses. MAC addresses – assigned at the link-layer of the OSI model – are used to identify devices on a local network segment and to provide a way for data to be transmitted directly to the correct device. Here's a typical MAC address:

40:b0:72:d4:29:ab

What is the Domain Name System (DNS)?

The Domain Name System is a system used to translate human-readable domain names into IP addresses. This allows users to access websites and other Internet resources using easy-to-remember domain names instead of IP addresses.

DNS also provides other important functions, such as mapping IP addresses to multiple domain names and directing traffic to the correct server based on the location of the user.

Forward DNS refers to the process of converting a domain name, such as "www.example.com", into an IP address, such as "192.0.2.1". Forward DNS is used by clients to resolve domain names to IP addresses, so that they can access websites, email servers, and other network services.

Reverse DNS, on the other hand, is the process of converting an IP address into a domain name. Reverse DNS is used to identify the domain name associated with a particular IP address.

This information can be used for security purposes, such as identifying the source of an incoming network connection, or for tracking the origin of spam or malicious traffic.

Public DNS servers are operated by organizations such as Google, Cloudflare, and OpenDNS, and are available for use by anyone on the Internet. Public DNS servers are often used as an alternative to the DNS servers provided by Internet Service Providers, as they can offer faster resolution times, better security features, and increased privacy.

Hardware Routing Devices

From a hardware perspective, traffic routing is managed through a loosely related family of networking devices. It should be noted that a single device will sometimes come with features normally identified with more than one category.

• Switches allow multiple devices on a network to communicate with each other by forwarding data packets to the intended recipient.
• Routers connect multiple networks together and forward data packets between them. Routers use routing tables to determine the best path for the data to take.
• Access Points allow wireless devices to connect to a wired network. They act as a bridge between the wireless devices and the network router.

It's useful to bear in mind that a "default router" is the gateway IP address used by devices on a network to communicate with devices on a different network. The default router is typically the IP address of the network router.

Wrapping Up

Understanding the software and hardware tools used to connect all of our many devices across digital networks can help you troubleshoot when things go wrong. But it's also a critical component for understanding the problems – and solutions – involved with network security.

This article and the accompanying video are excerpted from my Complete LPI Security Essentials Exam Study Guide course. And there's much more technology goodness available at bootstrap-it.com.

One of the major mechanisms you use to do that is an access control list (ACL). An ACL is a set of rules for allowing or denying access to certain resources. Resources in this case may be files, networks, or devices.

In this article, we'll talk about what access control lists really are, and how you can use them. We're going to deal with:

• Filesystem ACLs and Network ACLs

• Firewalls and stateful packet filtering

• ACLs in Cloud Networking (Azure NSG, AWS SG, AWS NACL)

• ACLs in DNS (BIND9)

• ACLs in core networking (Cisco ACL types, Huawei ACL types)

Prerequisites

To understand this article, you need a basic understanding of networking, firewalls, and cloud computing. You may particularly need to understand basic of IP addressing and DNS concepts.

Types of Access Control Lists

When we talk about ACLs, many people just think of networks. But in fact, there are two types of ACLs:

• File system ACLs

• Networking ACLs

Filesystem ACLs help operating systems know what the user access privileges are for different files or directories in the system. NFSv4 ACLs and POSIX ACLs are examples of filesystem ACL types.

Networking ACLs are applied on interfaces and you use them to allow or deny traffic from certain sources or to certain destinations. This is what I'll be focusing on in this article.

Structure Of An ACL Rule

An ACL is like a group of rules identified by a name or number. An ACL rule usually has a priority number, the criteria (source address, destination address, and so on) and the allow/deny statement.

Cisco ACL structure

Firewalls and ACLs

A firewall is a security device or software that monitors the traffic going in and out of a device or network, and filters out unwanted or malicious traffic.

Until stateful packet inspection, ACLs were the major mechanism through which firewalls worked. With ACLs, packets are allowed and denied based on properties specified in the rules.

ACLs are stateless. You must create an inbound rule and a corresponding outbound rule, or else packets from one side might be blocked.

With stateful packet inspection (also known as dynamic packet filtering), you could then create security policies for a type of traffic. The firewall would establish a session whenever a packet is allowed, so that any response to that packet is allowed even though there was no specific policy to allow it.

This makes things easier and more efficient than using ACLs that are uni-directional. But it also means that more computing resources are utilized by the firewall and the network is slowed down.

Now, firewalls are a lot more complex than that with deep packet inspection (DPI), Intrusion Detection System (IDS)/Intrusion Prevention System (IPS) capabilities, and even antivirus capabilities, but those are outside the scope of this article.

Let's explore some networking situations where ACLs are used.

ACLs in Cloud Networking

The major cloud service providers (CSPs) provide forms of ACLs or firewall capabilities for their customers to use in their cloud infrastructure.

For example, in Microsoft Azure, we have what is called Network Security Groups (NSG) and in AWS, we have Security Groups (SG) and the Network Access Control List (NACL). These are all implementations of ACL-like security.

AWS

An AWS security group determines what traffic is allowed to and from the resources attached to that security group. It consists of a list of inbound and outbound rules, and is stateful.

Default AWS Security Group

An AWS Network Access Control List is another list of rules but at the subnet level. The rules consist of the rule number, type, protocol, port range, source, destination and allow/deny fields. A NACL can be applied more than one subnet, but a subnet cannot be attached to more than one NACL.

Inbound rules for AWS NACL

Azure

An Azure Network Security Group is a kind of firewall feature that works both at the subnet level and the network interface card (NIC) of the resources in your VNet. It is basically also a list of ACL rules consisting of priority number, name, port, protocol, source and destination.

Here, you can use IP addresses, service tags, or application security groups (ASGs) in the source and destination fields. NSGs are stateful.

Both the Azure NSG and the AWS NACL rules are very similar to the ACL rules used in core networking. Also, you cannot really refer to AWS Security Groups and Azure NSGs as ACLs because they're not stateless.

Azure NSG

ACLs in DNS

DNS servers help resolve domain names to IP addresses. If they accept and respond to requests from every device around them, it will impact their performance and make them susceptible to DDoS attacks. So, DNS administrators use ACLs to determine who can send DNS requests to the servers.

For example, in a BIND9 server, such an ACL will be defined in the named.conf file, and would look like this:

An ACL in BIND9

ACLs in Core Networking

This is a bit more complex than the other contexts we discussed above. ACLs on network devices are configured on the interfaces, and are used in many different scenarios. There are also different types of ACLs. By network devices, I mean devices like routers, switches, firewalls, access controllers, and so on.

Generally, these ACLs are identified by their names or ACL numbers, and their rules follow the format:

permit/deny criteria

For Cisco devices, there are two major types of IPv4 ACLs:

• Standard access lists

• Extended access lists

Standard ACLs

These ACLs permit or deny traffic based on only the source IP address.

R1(config)#access-list 10 permit 192.168.17.0 0.0.0.255

The rule above tells the router to permit packets from the 192,168,17,0/24 subnet. Note that 0.0.0.255 is not a subnet mask. It is a wildcard that tells the device to which extent it must match the address you entered. 255 means any number goes while 0 means it must match exactly.

So here, the network part 192.168.17 must be exactly the same in whatever packet, while the last octet (the host part) can be whatever. You can learn more about IP addressing here.

Extended ACLs

These ACLs permit or deny traffic based on what is known in networking as the 5-tuple (source address, destination address, source port, destination port, transport layer protocol).

R2(config)#access-list 100 permit tcp 10.1.1.0 0.0.0.255 host 10.2.2.2 eq 80

The command above tells the router to permit any packet using the TCP transport layer protocol, coming from the 10.1.1.0/24 network to port 80 (HTTP) of the host, 10.2.2.2.

The term 5-tuple in networking probably originated from mathematics. A tuple means a record/row. 5-tuple means a row with five columns – an ordered list of 5 elements.

The five elements we're mostly concerned with in networking when dealing with packets are the IP addresses (source and destination), port numbers (source and destination), and transport layer protocol. So, they're usually referred to as 5-tuple.

ACL numbers 1 - 99 and 1300 - 1999 denote standard ACLs while numbers 100 - 199 and 2000 - 2699 denote extended ACLs.

Many other vendors follow this pattern, but Huawei doesn't.

For Huawei devices, there are 5 types of IPv4 ACLs:

• Basic ACLs ( ACL numbers 2000 - 2999)

• Advanced ACLs (ACL numbers 3000 - 3999)

• Layer 2 ACLs (ACL numbers 4000 - 4999)

• User-defined ACLs (ACL numbers 5000 - 5999)

• User ACLs (ACL numbers 6000 - 6999)

Basic ACL: Permits or denies traffic based on source address. The ACL number ranges from 2000 - 2999.

Advanced ACL: Permits or denies traffic based on the 5-tuple (source IP address, destination IP address, source port, destination port, and protocol type).

Layer 2 ACL: Permits or denies traffic based on information in the frame header (source MAC address, destination MAC address, layer 2 protocol type).

User-defined ACL: Permits or denies traffic based on packet headers, offsets, character string masks, and user defined character strings.

User ACL: Permits or denies traffic based on source and destination IP addresses or user control list (UCL) groups, source and destination ports, and IPv4 protocol types.

acl 3500                                                
 rule 0 deny tcp source 10.1.1.0 0.0.0.255 destination 192.168.0.9 0 destination-port eq 80                                                                
 rule 5 allow tcp source 10.1.1.0 0.0.0.255 destination 192.168.0.9 0 destination-port eq telnet

Implicit deny

It is also important to note that even if you do not add any rule at the end of your ACL, the last rule there is always a deny rule. It is not shown, so it is implicit. But it is there. It denies any packet that does not match any rule in your ACL.

A Few Things To Know

ACL rules are executed sequentially, so if you have rule 3 and rule 5, rule 3 gets executed first.

It is always a good practice to create rules at intervals (rule 10, rule 20, rule 30) rather than just serially (rule 1, rule 2, rule 3). The reason is that you may want to add a rule in-between two existing rules, and you want the system to execute it in that particular order. It saves stress if there was space for that from the beginning.

Conclusion

Access control is critical to security. Digitally, ACLs have been the go-to mechanism for quick and easy access control. Though other methods like role-based access control (RBAC) and attribute-based access control (ABAC) have emerged, ACL still has its place in access control.

Thanks for reading. If you enjoyed this article, please share it so others can see it too.

You can also connect with me on LinkedIn.

All the code samples for this course are in JavaScript, but the networking concepts you'll learn here apply generally to all coding languages. If you're not familiar with JavaScript yet, you can check out my JS course here.

I've included all the learning material you'll need here in this article, but if you would like a more hands-on experience, you can take the interactive version of this course with coding challenges on Boot.dev here.

I've also published a free video version of this course on the freeCodeCamp YouTube channel:

If you like this video, you can check out my other tutorials on my Boot.dev YouTube channel here.

All that said, let's jump into learning about HTTP!

Table of Contents

1. Why HTTP?

2. What is DNS?

3. What Are URIs?

4. Async/Await

5. Error Handling

6. HTTP Headers

7. What is JSON?

8. HTTP Methods

9. URL Paths and Parameters

10. What is HTTPs?

Why HTTP?

Communicating on the web

Instagram would be pretty terrible if you had to manually copy your photos to your friend's phone when you wanted to share them. Modern applications need to be able to communicate information between devices over the internet.

• Gmail doesn't just store your emails in variables on your computer, it stores them on computers in their data centers.

• You don't lose your Slack messages if you drop your computer in a lake – those messages exist on Slack's servers.

How does web communication work?

When two computers communicate with each other, they need to use the same rules. An English speaker can't communicate verbally with a Japanese speaker, and similarly, two computers need to speak the same language to communicate.

This "language" that computers use is called a protocol. The most popular protocol for web communication is HTTP, which stands for Hypertext Transfer Protocol.

Interacting with a server

In this course, a lot of the code samples will interact with the PokeAPI. It serves data about Pokémon.

Here's some code that retrieves a list of Pokémon from the PokeAPI:

const pokemonResp = await getItemData()

logPokemons(pokemonResp.results)

async function getItemData() {
  const response = await fetch('https://pokeapi.co/api/v2/pokemon/', {
    method: 'GET',
    mode: 'cors',
    headers: {
      'Content-Type': 'application/json'
    }
  })
  return response.json()
}

function logPokemons(pokemons) {
  for (const pokemon of pokemons) {
    console.log(pokemon.name)
  } 
}

When you run this code, you'll notice that none of the data that is logged to the console was generated within our code! That's because the data we retrieved is being sent over the internet from our servers via HTTP. Don't worry, we'll explain more about that later.

HTTP Requests and Responses

At the heart of HTTP is a simple request-response system. The "requesting" computer, also known as the "client", asks another computer for some information. That computer, the "server" sends back a response with the information that was requested.

We'll talk about the specifics of how the "requests" and "responses" are formatted later. For now, just think of it as a simple question-and-answer system.

• Request: "What are the items in the Fantasy Quest game?"

• Response: A list of the items in the Fantasy Quest game

HTTP powers websites

As we discussed, HTTP, or Hypertext Transfer Protocol, is a protocol designed to transfer information between computers.

There are other protocols for communicating over the internet, but HTTP is the most popular and is particularly great for websites and web applications.

Each time you visit a website, your browser is making an HTTP request to that website's server. The server responds with all the text, images, and styling information that your browser needs to render its pretty website.

HTTP URLs

A URL, or Uniform Resource Locator, is essentially the address of another computer, or "server" on the internet. Part of the URL specifies how to reach the server, and part of it tells the server what information we want - but more on that later.

For now, it's important to understand that a URL represents a piece of information on another computer that we want access to. We can get access to it by making a request, and reading the response that the server replies with.

How to Use URLs in HTTP

The http:// at the beginning of a website URL specifies that the http protocol will be used for communication.

Other communication protocols use URLs as well, (hence "Uniform Resource Locator"). That's why we need to be specific when we're making HTTP requests by prefixing the URL with http://

Requests and Responses Review

• A "client" is a computer making an HTTP request

• A "server" is a computer responding to an HTTP request

• A computer can be a client, a server, both, or neither. "Client" and "server" are just words we use to describe what computers are doing within a communication system.

• Clients send requests and receive responses

• Servers receive requests and send responses

JavaScript's Fetch API

In this course, we'll be using JavaScript's built-in fetch API to make HTTP requests.

The fetch() function is made available to us by the JavaScript language running in the browser. All we have to do is provide it with the parameters it requires.

How to Use Fetch

const response = await fetch(url, settings)
const responseData = await response.json()

We'll go in-depth on the various things happening in this standard fetch call later, but let's cover some basics for now.

• response is the data that comes back from the server

• url is the URL we are making a request to

• settings is an object containing some request-specific settings

• The await keyword tells JavaScript to wait until the request comes back from the server before continuing

• response.json() converts the response data from the server into a JavaScript object

See if you can spot the issue with this code snippet:

const pokemonResp = getItemData()

logPokemons(pokemonResp.results)

// the bug is above this line

async function getItemData() {
  const response = await fetch('https://pokeapi.co/api/v2/pokemon/', {
    method: 'GET',
    mode: 'cors',
    headers: {
      'Content-Type': 'application/json'
    }
  })
  return response.json()
}

function logPokemons(pokemons) {
  for (const pokemon of pokemons) {
    console.log(pokemon.name)
  } 
}

Hint: We're not waiting for the data to be sent back across the network.

Web Clients

A web client is a device making requests to a web server. A client can be any type of device but is often something users physically interact with. For example:

• A desktop computer

• A mobile phone

• A tablet

In a website or web application, we call the user's device the "front-end".
A front-end client makes requests to a back-end server.

Web Servers

Up to this point, most of the data you have worked with in your code has simply been generated and stored locally in variables.

While you'll always use variables to store and manipulate data while your program is running, most websites and apps use a web server to store, sort, and serve that data so that it sticks around for longer than a single session, and can be accessed by multiple devices.

Listening and serving data

Similar to how a server at a restaurant brings your food to the table, a web server) serves web resources, such as web pages, images, and other data. The server is turned on and "listening" for inbound requests constantly so that the second it receives a new request, it can send an appropriate response.

The server is the back-end

While the "front-end" of a website or web application is the device the user interacts with, the "back-end" is the server that keeps all the data housed in a central location. If you're still confused, check out this article comparing front-end and back-end development.

A server is just a computer

"Server" is just the name we give to a computer that is taking on the role of serving data across a network connection.

A good server is turned on and available 24 hours a day, 7 days a week. While your laptop can be used as a server, it makes more sense to use a computer in a data center that's designed to be up and running constantly.

What is DNS?

Web Addresses

In the real world, we use addresses to help us find where a friend lives, where a business is located, or where a party is being thrown.

In computing, web clients find other computers over the internet using Internet Protocol or IP addresses.

An IP address is a numerical label that serves two main functions:

1. Location Addressing

2. Network Identification

Domain names and IP Addresses

Each device connected to the internet has a unique IP address. When we browse the internet, the domains we navigate to are all associated with a particular IP address.

For example, this Wikipedia URL points to a page about miniature pigs: https://en.wikipedia.org/wiki/Miniature_pig
The domain portion of the URL is en.wikipedia.org. en.wikipedia.org converts to a specific IP address, and that IP address tells your computer exactly where to communicate with that Wikipedia page.

Cloudflare is a tech company that provides a cool public HTTP server that we can use to look up the IP address of any domain. Take a look at this sample code:

async function fetchIPAddress(domain) {
  const resp = await fetch(`https://cloudflare-dns.com/dns-query?name=${domain}&type=A`, {
    headers: {
      'accept': 'application/dns-json'
    }
  })
  const respObject = await resp.json()
  for (const record of respObject.Answer) {
    return record.data
  }
  return null
}

const domain = 'api.boot.dev'
const ipAddress = await fetchIPAddress(domain)
if (!ipAddress) {
  console.log('something went wrong in fetchIPAddress')
} else {
  console.log(`found IP address for domain ${domain}: ${ipAddress}`)
}

To recap, a "domain name" is part of a URL. It's the part that tells the computer where the server is located on the internet by being converted into a numerical IP address.

We'll cover exactly how an IP address is used by your computer to find a path to the server in a later networking course. For now, it's just important to understand that an IP address is what your computer is using at a lower level to communicate on a network.

Deploying a real website to the internet is actually quite simple. It involves only a couple of steps:

1. Create a server that hosts your website files and connect it to the internet

2. Acquire a domain name

3. Connect the domain name to the IP address of your server

4. Your server is accessible via the internet!

As we discussed, the "domain name" or "hostname" is part of a URL. We'll get to the other parts of a URL later.

For example, the URL https://example.com/path has a hostname of example.com. The https:// and /path portions are not part of the domain name -> IP address mapping that we've been learning about.

Using the URL API in JavaScript

The URL API is built into JavaScript. You can create a new URL object like this:

const urlObj = new URL('https://example.com/example-path')

And then you can extract just the hostname:

const urlObj.hostname

DNS Review

So we've talked about domain names and what their purpose is, but we haven't talked about the system that's used to do that conversion.

DNS, or the "Domain Name System", is the phonebook of the internet. Humans connect to websites through domain names, like Boot.dev.

DNS "resolves" these domain names to find the associated IP addresses so that web clients can load the resources for the specific address.

How does DNS Work?

We'll go into more detail on DNS in a future course, but to give you a simplified idea of how it works, let's introduce ICANN. ICANN is a not-for-profit organization that manages DNS for the entire internet.

Whenever your computer attempts to resolve a domain name, it contacts one of ICANN's "root nameservers" whose address is included in your computer's networking configuration.

From there, that nameserver can gather the domain records for a specific domain name from their distributed DNS database.

If you think of DNS as a phonebook, ICANN is the publisher that keeps the phonebook in print and available.

Subdomains

We learned about how a domain name resolves to an IP address, which is just a computer on a network - often the internet.

A subdomain prefixes a domain name, allowing a domain to route network traffic to many different servers and resources.

For example, the Boot.dev website is hosted on a different computer than our blog. Our blog, found at blog.boot.dev is hosted on our "blog" subdomain.

What are URIs?

We briefly touched on URLs earlier, but now let's dive a little deeper into the subject.

A URI, or Uniform Resource Identifier, is a unique character sequence that identifies a resource that is (almost always) accessed via the internet.

Just like JavaScript has syntax rules, so do URIs. These rules help ensure uniformity so that any program can interpret the meaning of the URI in the same way.

URIs come in two main types:

• URLs

• URNs

We will focus specifically on URLs in this course, but it's important to know that URLs are only one kind of URI.

URLs have quite a few sections, some of which are required, others not.
Let's use the URL API to parse a URL and print all the different parts. We'll learn more about each part later, for now, let's just split and print a URL.

function printURLParts(urlString) {
  const urlObj = new URL(urlString)
  console.log(`protocol: ${urlObj.protocol}`)
  console.log(`username: ${urlObj.username}`)
  console.log(`password: ${urlObj.password}`)
  console.log(`hostname: ${urlObj.hostname}`)
  console.log(`port: ${urlObj.port}`)
  console.log(`pathname: ${urlObj.pathname}`)
  console.log(`search: ${urlObj.search}`)
  console.log(`hash: ${urlObj.hash}`)
}

const fantasyQuestURL = 'http://dragonslayer:pwn3d@fantasyquest.com:8080/maps?sort=rank#id'
printURLParts(fantasyQuestURL)

Further dissecting a URL

There are 8 main parts to a URL, though not all the sections are always present. Each piece plays a specific role in helping clients locate the specified resource.
The 8 sections are:

• The protocol is required

• Usernames and passwords are optional

• A domain is required

• The default port for a given protocol is used if one is not provided

• The default ( / ) path is used if one isn't provided

• A query is optional

• A fragment is optional

Don't get too hung up on memorizing this stuff

Because names for the different sections are often used interchangeably, and because not all the parts of the URL are always present, it can be hard to keep things straight.

Don't worry about memorizing this stuff! Try to just get familiar with these URL concepts from a high level. Like any good developer, you can just look it up again the next time you need to know more.

The Protocol

The "protocol", also referred to as the "scheme", is the first component of a URL. Its purpose is to define the rules by which the data being communicated is displayed, encoded, or formatted.

Some examples of different URL protocols:

• http

• ftp

• mailto

• https

For example:

• http://example.com

• mailto:noreply@fantasyquest.app

Not all schemes require a "//"

The "http" in a URL is always followed by ://. All URLs have the colon, but the // part is only included for schemes that have an authority component.

As you can see above, the mailto scheme doesn't use an authority component, so it doesn't need the slashes.

URL Ports

The port in a URL is a virtual point where network connections are made. Ports are managed by a computer's operating system and are numbered from 0 to 65,535.

Whenever you connect to another computer over a network, you're connecting to a specific port on that computer, which is being listened to by a specific piece of software on that computer. A port can only be used by one program at a time, which is why there are so many possible ports.

The port component of a URL is often not visible when browsing normal sites on the internet, because 99% of the time you're using the default ports for the HTTP and HTTPS schemes: 80 and 443, respectively.

Whenever you aren't using a default port, you need to specify it in the URL. For example, port 8080 is often used by web developers when they're running their server in "test mode" so that they don't use the "production" port "80".

URL Paths

In the early days of the internet, a URL's path often was a reflection of the file path on the server to the resource the client was requesting.

For example, if the website https://exampleblog.com had a web server running within its /home directory, then a request to the https://exampleblog.com/site/index.html URL might expect the index.html file from within the /home/site directory to be returned.

Websites used to be very simple. They were just a collection of text documents stored on a server. A simple piece of server software could handle incoming HTTP requests and respond with the documents according to the path component of the URLs.

These days, it's not always about the filesystem

On many modern web servers, a URL's path isn't a reflection of the server's filesystem hierarchy. Paths in URLs are essentially just another type of parameter that can be passed to the server when making a request.

Conventionally, two different URL paths should denote different resources. For example, different pages on a website, or maybe different data types from a game server.

Query parameters

Query parameters in a URL are not always present. In the context of websites, query parameters are often used for marketing analytics or for changing a variable on the web page. With website URLs, query parameters rarely change which page you're viewing, though they often will change the page's contents.

That said, query parameters can be used for anything the server chooses to use them for, just like the URL's path.

How Google uses query parameters

1. Open a new tab and go to google.com

2. Search for "hello world"

3. Take a look at your current URL. It should start with https://www.google.com/search?q=hello+world

4. Change the URL to say https://www.google.com/search?q=hello+universe

5. Press "enter"

You should see new search results for the query "hello universe". Google chose to use query parameters to represent the value of your search query. It makes sense - each search result page is essentially the same page as far as structure and formatting are concerned - it's just showing you different results based on the search query.

Async/Await

You're probably already familiar with synchronous code, which means code that runs in sequence. Each line of code executes in order, one after the next.

Example of synchronous code:

console.log("I print first");
console.log("I print second");
console.log("I print third");

Asynchronous or async code runs in parallel. That means code further down runs at the same time that a previous line of code is still running. A good way to visualize this is with the JavaScript function setTimeout().

setTimeout accepts a function and a number of milliseconds as inputs. It waits until the number of milliseconds has elapsed, and then it executes the function it was given.

Example of asynchronous code:

jsconsole.log("I print first");
setTimeout(() => console.log("I print third because I'm waiting 100 milliseconds"), 100);
console.log("I print second");

Try altering the waiting times in the async code below to get the messages to print in the proper order:

const craftingCompleteWait = 0
const combiningMaterialsWait = 0
const smeltingIronBarsWait = 0
const shapingIronWait = 0

// Don't touch below this line

setTimeout(() => console.log('Iron Longsword Complete!'), craftingCompleteWait)
setTimeout(() => console.log('Combining Materials...'), combiningMaterialsWait)
setTimeout(() => console.log('Smelting Iron Bars...'), smeltingIronBarsWait)
setTimeout(() => console.log('Shaping Iron...'), shapingIronWait)

console.log('Firing up the forge...')

await sleep(2500)
function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms))
}

Expected order:

1. Firing up the forge..

2. Smelting Iron Bars...

3. Combining Materials...

4. Shaping Iron...

5. Iron Longsword Complete!

Why do we want async code?

We try to keep most of our code synchronous because it's easier to understand, and therefore often has fewer bugs. But sometimes we need our code to be asynchronous.

For example, whenever you update your user settings on a website, your browser will need to communicate those new settings to the server. The time it takes your HTTP request to physically travel across all the wiring of the internet is usually around 100 milliseconds. It would be a very poor experience if your webpage were to freeze while waiting for the network request to finish. You wouldn't even be able to move the mouse while waiting!

By making network requests asynchronously, we let the webpage execute other code while waiting for the HTTP response to come back. This keeps the user experience snappy and user-friendly.

As a general rule, we should only use async code when we need to for performance reasons. Synchronous code is simpler.

Promises in JavaScript

A Promise in JavaScript is very similar to making a promise in the real world. When we make a promise, we are making a commitment to something.

For example, I promise to explain JavaScript promises to you. My promise to you has 2 potential outcomes: it is either fulfilled, meaning I eventually explained promises to you, or it is rejected, meaning I failed to keep my promise and didn't explain promises.

The Promise Object represents the eventual fulfillment or rejection of our promise and holds the resulting values. In the meantime, while we're waiting for the promise to be fulfilled, our code continues executing.

Promises are the most popular modern way to write asynchronous code in JavaScript.

How to Declare a Promise

Here is an example of a promise that will resolve and return the string "resolved!" or reject and return the string "rejected!" after 1 second.

const promise = new Promise((resolve, reject) => {
  setTimeout(() => {
    if (getRandomBool()) {
      resolve("resolved!")
    } else {
      reject("rejected!")
    }
  }, 1000)
})

function getRandomBool(){
  return Math.random() < .5
}

How to Use a Promise

Now that we've created a promise, how do we use it?

The Promise object has .then and .catch that make it easy to work with. Think of .then as the expected follow-up to a promise, and .catch as the "something went wrong" follow-up.

If a promise resolves, its .then function will execute. If the promise rejects, its .catch method will execute.

Here's an example of using .then and .catch with the promise we made above:

promise.then((message) => {
    console.log(`The promise finally ${message}`)
}).catch((message) => {
    console.log(`The promise finally ${message}`)
})

// prints:
// The promise finally resolved!
// or
// the promise finally rejected!

Why are Promises useful?

Promises are the cleanest (but not the only) way to handle the common scenario where we need to make requests to a server, which is typically done via an HTTP request. In fact, the fetch() function we were using earlier in the course returns a promise!

I/O, or "input/output"

Almost every time you use a promise in JavaScript it will be to handle some form of I/O. I/O, or input/output, refers to when our code needs to interact with systems outside of the (relatively) simple world of local variables and functions.
Common examples of I/O include:

• HTTP requests

• Reading files from the hard drive

• Interacting with a Bluetooth device

Promises help us perform I/O without forcing our entire program to freeze up while we wait for a response.

Promises and the "await" keyword

We have used the await keyword a few times in this course, so it's time we finally understand what's going on under the hood.

The await keyword is used to wait for a promise to resolve. Once it has been resolved, the await expression returns the value of the resolved Promise.

Example with .then callback

promise.then((message) => {
  console.log(`Resolved with ${message}`)
}).

Example of awaiting a promise

const message = await promise
console.log(`Resolved with ${message}`)

The async keyword

While the await keyword can be used in place of .then() to resolve a promise, the async keyword can be used in place of new promise() to create a new promise.

When a function is prefixed with the async keyword, it will automatically return a promise. That promise resolves with the value that your code returns from the function. You can think of async as "wrapping" your function within a promise.

These are equivalent:

New Promise()

function getPromiseForUserData(){
  return new Promise((resolve) => {
    fetchDataFromServerAsync().then(function(user){
      resolve(user)
    })
  })
}

const promise = getPromiseForUserData()

Async

async function getPromiseForUserData(){
  const user = await fetchDataFromServer()
  return user
}

const promise = getPromiseForUserData()

.then() vs await

In the early days of web browsers, promises and the await keyword didn't exist, so the only way to do something asynchronously was to use callbacks.

A "callback function" is a function that you hand to another function. That function then calls your callback later on. The setTimeout function we've used in the past is a good example.

function callbackFunction(){
  console.log("calling back now!")
}
const milliseconds = 1000
setTimeout(callbackFunction, milliseconds)

While even the .then() syntax is generally easier to use than callbacks without the Promise API, the await syntax makes them even easier to use. You should use async and await over .then and new Promise() as a general rule.

To demonstrate, which of these is easier to understand?

fetchUser.then(function(user){
  return fetchLocationForUser(user)
}).then(function(location){
  return fetchServerForLocation(location)
}).then(function(server){
  console.log(`The server is ${server}`)
});

const user = await fetchUser()
const location = await fetchLocationForUser(user)
const server = await fetchServerForLocation(location)
console.log(`The server is ${server}`)

They both do the same thing, but the second example is so much easier to understand! The async and await keywords weren't released until after the .then API, which is why there is still a lot of legacy .then() code out there.

Error Handling

When something goes wrong while a program is running, JavaScript uses the try/catch paradigm for handling those errors. Try/catch is fairly common, and Python uses a similar mechanism.

First, an error is thrown

For example, let's say we try to access a property on an undefined variable. JavaScript will automatically "throw" an error.

const speed = car.speed
// The code crashes with the following error:
// "ReferenceError: car is not defined"

Trying and catching errors

By wrapping that code in a try/catch block, we can handle the case where car is not yet defined.

try {
  const speed = car.speed
} catch (err) {
  console.log(`An error was thrown: ${err}`)
  // the code cleanly logs:
  // "An error was thrown: ReferenceError: car is not defined"
}

Bugs vs Errors

Error handling via try/catch is not the same as debugging. Likewise, errors are not the same as bugs.

• Good code with no bugs can still produce errors that are gracefully handled

• Bugs are, by definition, bits of code that aren't working as intended

What is Debugging?

"Debugging" a program is the process of going through your code to find where it is not behaving as expected. Debugging is a manual process performed by the developer.

Examples of debugging:

• Adding a missing parameter to a function call

• Updating a broken URL that an HTTP call was trying to reach

• Fixing a date-picker component in an app that wasn't displaying properly

What is Error Handling?

"Error handling" is code that can handle expected edge cases in your program. Error handling is an automated process that we design into our production code to protect it from things like weak internet connections, bad user input, or bugs in other people's code that we have to interface with.

Examples of error handling:

• Using a try/catch block to detect an issue with user input

• Using a try/catch block to gracefully fail when no internet connection is available

In short, don't use try/catch to try to handle bugs

If your code has a bug, try/catch won't help you. You need to just go find the bug and fix it.

If something out of your control can produce issues in your code, you should use try/catch or other error-handling logic to deal with it.

For example, there could be a prompt in a game for users to type in a new character name, but we don't want them to use punctuation. Validating their input and displaying an error message if something is wrong with it would be a form of "error handling".

async/await makes error handling easier

try and catch are the standard way to handle errors in JavaScript, the trouble is, the original Promise API with .then didn't allow us to make use of try and catch blocks.

Luckily, the async and await keywords do allow it - yet another reason to prefer the newer syntax.

.catch() callback on promises

The .catch() method works similarly to the .then() method, but it fires when a promise is rejected instead of resolved.

Example with .then and .catch callbacks

fetchUser().then(function(user){
  console.log(`user fetched: ${user}`)
}).catch(function(err){
  console.log(`an error was thrown: ${err}`)
});

Example of awaiting a promise

try {
  const user = await fetchUser()
  console.log(`user fetched: ${user}`)
} catch (err) {
  console.log(`an error was thrown: ${err}`)
}

As you can see, the async/await version looks just like normal try/catch JavaScript.

HTTP Headers

An HTTP header allows clients and servers to pass additional information with each request or response. Headers are just case-insensitive key-value pairs that pass additional metadata about the request or response.

HTTP requests from a web browser carry with them many headers, including but not limited to:

• The type of client (for example Google Chrome)

• The Operating system (for example Windows)

• The preferred language (for example US English)

As developers, we can also define custom headers in each request.

Headers API

The Headers API allows us to perform various actions on our request and response headers such as retrieving, setting, and removing them. We can access the headers object through the Request.headers and Response.headers properties.

How to Use the Browser's Developer Tools

Modern web browsers offer developers a powerful set of developer tools. The Developer Tools are a front-end web developer's best friend. For example, using the dev tools you can:

• View the web page's JavaScript console output

• Inspect the page's HTML, CSS, and JavaScript code

• View network requests and responses, along with their headers

The method for accessing dev tools varies from browser to browser. If you're on Chrome, you can just right-click anywhere within a web page and click the "inspect" option. Follow this link for more info on how to access dev tools.

The network tab

While all of the tabs within the dev tools are very useful, we will be focusing specifically on the Network tab in this chapter so we can play with HTTP headers.

The Network tab monitors your browser's network activity and records all of the requests and responses the browser is making, including how long each of those requests and responses takes to fully process.

If you navigate to the Network tab and don't see any requests appear, try refreshing the page.

Why are headers useful?

Headers are useful for several reasons, from design to security. But most often headers are used as metadata or data about the request.

So, for example, let's say we wanted to ask for a player's level from a game's server. We need to send that player's ID to the server so it knows which player to send back the information for. That ID is my request, it's not information about my request.

A good example of a use case for headers is authentication. Often times a user's credentials are included in request headers. Credentials don't have much to do with the request itself, but simply authorize the requester to be allowed to make the request in question.

What is JSON?

JavaScript Object Notation, or JSON, is a standard for representing structured data based on JavaScript's object syntax.

JSON is commonly used to transmit data in web apps using HTTP. The HTTP fetch() requests we have been using in this course have been returning data as JSON data.

JSON Syntax

Because we already understand what JavaScript objects look like, understanding JSON is easy. JSON is just a stringified JavaScript object. The following is valid JSON data:

{
    "movies": [
        {
            "id": 1,
            "genre": "Action",
            "title": "Iron Man",
            "director": "Jon Favreau"
        },
        {
            "id": 2,
            "genre": "Action",
            "title": "The Avengers",
            "director": "Joss Whedon"
        }
    ]
}

How to Parse HTTP Responses as JSON

JavaScript provides us with some easy tools to help us work with JSON. After making an HTTP request with the fetch() API, we get a Response object. That response object offers us some methods that help us interact with the response.

One such method is the .json() method. The .json() method takes the response stream returned by a fetch request and returns a Promise that resolves into a JavaScript object parsed from the JSON body of the HTTP response.

const resp = await fetch(...)
const javascriptObjectResponse = await resp.json()

It's important to note that the result of the .json() method is NOT JSON. It is the result of taking JSON data from the HTTP response body and parsing that input into a JavaScript Object.

JSON Review

JSON is a stringified representation of a JavaScript object, which makes it perfect for saving to a file or sending in an HTTP request.

Remember, an actual JavaScript object is something that exists only within your program's variables. If we want to send an object outside our program, for example, across the internet in an HTTP request, we need to convert it to JSON first.

It's not just used in JavaScript

Just because JSON is called JavaScript Object Notation doesn't mean it's only used by JavaScript! JSON is a common standard that is recognized and supported by every major programming language.

For example, even though Boot.dev's backend API is written in Go, we still use JSON as the communication format between the front-end and backend.

By the way, this course has been about interacting with back-end servers from a front-end perspective. But if you're curious about how you can become a back-end engineer, check out this guide I've put together. For reference, it takes most people between 6-18 months to learn enough to get their first back-end job.

Common JSON use-cases

• In HTTP request and response bodies

• As formats for text files. .json files are often used as configuration files.

• In NoSQL databases like MongoDB, ElasticSearch, and Firestore

How to Pronounce JSON

I pronounce it "Jay-sawn", but I've also heard people pronounce it "Jason", like the name.

How to Send JSON

JSON isn't just something we get from the server, we can also send JSON data.
In JavaScript, two of the main methods we have access to are JSON.parse(), and JSON.stringify().

JSON.stringify()

JSON.stringify() is particularly useful for sending JSON.

As you may expect, the JSON stringify() method does the opposite of parse. It takes a JavaScript object or value as input and converts it into a string. This is useful when we need to serialize the objects into strings to send them to our server or store them in a database.

Here's a snippet of code that sends a JSON payload to a remote server:

async function sendPayload(data, headers) {
  const response = await fetch(url, {
    method: 'POST',
    mode: 'cors',
    headers: headers,
    body: JSON.stringify(data)
  })
  return response.json()
}

How to Parse JSON

The JSON.parse() method takes a JSON string as input and constructs the JavaScript value/object described by the string. This allows us to work with the JSON as a normal JavaScript object.

Seeing as JSON objects have a tree-like structure, it can be helpful to know how to traverse them recursively if needs be.

const json = '{"title": "Avengers Endgame", "Rating":4.7, "inTheaters":false}';
const obj = JSON.parse(json)

console.log(obj.title)
// Avengers Endgame

XML

We can't talk about JSON without mentioning XML. Extensible Markup Language, or XML is a text-based format for representing structured information, just like JSON - it just looks a bit different.

XML is a markup language like HTML, but it's more generalized in that it does not use predefined tags. Just like how in a JSON object keys can be called anything, XML tags can also have any name.

<root>
  <id>1</id>
  <genre>Action</genre>
  <title>Iron Man</title>
  <director>Jon Favreau</director>
</root>

The same data in JSON form:

{
  "id": "1",
  "genre": "Action",
  "title": "Iron Man",
  "director": "Jon Favreau"
}

Why use XML?

XML and JSON both accomplish similar tasks, so which should you use?

XML used to be used for the same things that today JSON is used for. Configuration files, HTTP bodies, and other data-transfer use cases can work just fine using JSON or XML. Here's my advice: generally speaking, if JSON works, you should favor it over XML these days. JSON is lighter-weight, easier to read, and has better support in most modern programming languages.

There are some cases where XML might still be the better, or maybe even the necessary choice, but those cases will be rare.

HTTP Methods

HTTP defines a set of methods that we use every time we make a request. We have used some of these methods in previous exercises, but it's time we dive into them and understand the differences and use cases behind the different methods.

The GET method

The GET method is used to "get" a representation of a specified resource. You are not taking the data away from the server, but rather getting a representation, or copy, of the resource in its current state.

A get request is considered a safe method to call multiple times because it doesn't alter the state of the server.

How to Make a GET Request using the Fetch API

The fetch() method accepts an optional init object parameter as its second argument that we can use to define things like:

• method: The HTTP method of the request, like GET.

• headers: The headers to send.

• mode: Used for security, we'll talk about this in future courses

• body: The body of the request. Often encoded as JSON.

Example GET request using fetch:

await fetch(url, {
  method: 'GET',
  mode: 'cors',
  headers: {
    'sec-ch-ua-platform': 'macOS'
  }
})

Why do we use HTTP methods?

As we touched on before, the primary purpose of HTTP methods is to indicate to the server what we want to do with the resource we're trying to interact with.

At the end of the day, an HTTP method is just a string, like GET, POST, PUT, or DELETE. But by convention, backend developers almost always write their server code so that the methods correspond with different "CRUD" actions.

The "CRUD" actions are:

• Create

• Read

• Update

• Delete

The bulk of the logic in most web applications is "CRUD" logic. The web interface allows users to create, read, update and delete various resources.

Think of a social media site - users are basically creating, reading, updating and deleting their social posts. They are also creating, reading, updating and deleting their user accounts. It's CRUD all the way down!

As it happens, the 4 most common HTTP methods map nicely to the CRUD actions:

• POST = create

• GET = read

• PUT = update

• DELETE = delete

POST Requests

An HTTP POST request sends data to a server, typically to create a new resource. The body of the request is the payload that is being sent to the server with the request. Its type is indicated by the Content-Type header.

How to Add a body

The body of the request is the payload that is being sent to the server with the request. Its type is indicated by the Content-Type header - for us, that's going to be JSON.

POST requests are generally not safe methods to call multiple times, because it alters the state of the server. You wouldn't want to accidentally create 2 accounts for the same user, for example.

await fetch(url, {
  method: 'POST',
  mode: 'cors',
  headers: {
    'Content-Type': 'application/json'
  },
  body: JSON.stringify(data)
})

HTTP Status Codes

Now that we understand how to write HTTP requests from scratch, we need to learn how to ensure that the server is doing what we want.

Earlier in the course, we learned about how to access the browser's developer tools and use those tools to inspect HTTP requests. We can use that same process to check on the requests we are making and verify what they are doing so we can address potential problems.

When looking at requests, we can check on the Status Code of the request to get some information if the request was successful or not.

• 100-199: Informational responses. These are very rare.

• 200-299: Successful responses. Hopefully, most responses are 200's!

• 300-399: Redirection messages. These are typically invisible because the browser or HTTP client will automatically do the redirect.

• 400-499: Client errors. You'll see these often, especially when trying to debug a client application

• 500-599: Server errors. You'll see these sometimes, usually only if there is a bug on the server.

Here are some of the most common status codes, but you can see a full list here if you're interested.

• 200 - OK. This is by far the most common code, it just means that everything worked as expected.

• 201 - Created. This means that a resource was created successfully. Typically in response to a POST request.

• 301 - Moved permanently. This means the resource was moved to a new place, and the response will include where that new place is. Websites often use 301 redirects when they change their domain name, for example.

• 400 - Bad request. A general error indicating the client made a mistake in their request.

• 403 - Unauthorized. This means the client doesn't have the correct permissions. Maybe they didn't include a required authorization header, for example.

• 404 - Not found. You'll see this on websites quite often. It just means the resource doesn't exist.

• 500 - Internal server error. This means something went wrong on the server, likely a bug on their end.

You don't need to memorize them

You need to know the basics, like "2XX is good", "4XX is a client error", and "5XX is a server error". That said, you don't need to memorize all the codes, they're easy to look up.

Let's check some status codes!

The .status property on a Response object will give you the code. Here's an example:

async function getStatusCode(url, headers) {
  const response = await fetch(url, {
    method: 'GET',
    mode: 'cors',
    headers: headers
  })
  return response.status
}

PUT Method

The HTTP PUT method creates a new resource or replaces a representation of the target resource with the contents of the request's body. In short, it updates a resource's properties.

await fetch(url, {
   method: 'PUT',
   mode: 'cors',
   headers: {
   'Content-Type': 'application/json'
   },
   body: JSON.stringify(data)
})

POST vs PUT

You may be thinking PUT is similar to POST or PATCH, and frankly, you'd be right. The main difference is that PUT is meant to be idempotent, meaning multiple identical PUT requests should have the same effect on the server.

In contrast, several identical POST requests would have additional side effects, such as creating multiple copies of the resource.

HTTP Patch vs PUT

You may encounter PATCH methods from time to time. While it is not nearly as common as the other methods, like PUT, it's important to know about it and what it does. The PATCH method is intended to partially modify a resource.

Long story short, PATCH isn't nearly as popular as PUT, and many servers, even if they allow partial updates, will still just use the PUT method for that.

HTTP Delete

The DELETE method does exactly as you would expect: it's conventionally used to delete a specified resource.

// This deletes the location with ID: 52fdfc07-2182-454f-963f-5f0f9a621d72
const url = 'https://example-api.com/locations/52fdfc07-2182-454f-963f-5f0f9a621d72'

await fetch(url, {
  method: 'DELETE',
  mode: 'cors'
})

URL Paths and Parameters

The URL Path comes right after the domain (or port if one is provided) in a URL string.

In this URL, the path is /root/next: http://testdomain.com/root/next.

What paths meant in the early internet

In the early days of the internet, and sometimes still today, many web servers simply served raw files from the server's file system.

For example, if I wanted you to be able to access some text documents, I could start a web server in my documents directory. If you made a request to my server, you would be able to access different documents by using the path that matched my local file structure.

If I had a file in my local /documents/hello.txt, you could access it by making a GET request to http://example.com/documents/hello.txt.

How paths are used today

Most modern web servers don't use that simple mapping of URL path -> file path. Technically, a URL path is just a string that the web server can do what it wants with, and modern websites take advantage of that flexibility.

Some common examples of what paths are used for include:

• The hierarchy of pages on a website, whether or not that reflects a server's file structure

• Parameters being passed into an HTTP request, like an ID of a resource

• The version of the API

• The type of resource being requested

RESTful APIs

Representational State Transfer, or REST, is a popular convention that HTTP servers follow. Not all HTTP APIs are "REST APIs", or "RESTful", but it is very common.

RESTful servers follow a loose set of rules that makes it easy to build reliable and predictable web APIs. REST is more or less a set of conventions about how HTTP should be used.

Separate and agnostic

The big idea behind REST is that resources are transferred via well-recognized, language-agnostic client/server interactions.

A RESTful style means the implementation of the client and server can be done independently of one another, as long as some simple standards, like the names of the available resources, have been established.

Stateless

A RESTful architecture is stateless. This means that the server does not need to know what state the client is in, nor does the client need to know what state the server is in.

Statelessness in REST is enforced by interacting with resources instead of commands. Keep in mind, this doesn't mean the applications are stateless - on the contrary, what would "updating a resource" even mean if the server wasn't keeping track of its state?

Paths in REST

In a RESTful API, the last section of the path of a URL should specify which resource is being accessed. Then, as we talked about in the "methods" section, depending on whether the request is a GET, POST, PUT or DELETE, the resource is read, created, updated, or deleted.

For example, in the PokeAPI:

• https://pokeapi.co/api/v2/pokemon/

• https://pokeapi.co/api/v2/location/

The first part of the path specifies that we're interacting with an API rather than a website. The next part specifies the version, in this case, version 2, or v2.

Finally, the last part denotes which resource is being accessed, be it a location or pokemon.

URL Query Parameters

A URL's query parameters appear next in the URL structure but are not always present - they're optional. For example:

https://www.google.com/search?q=boot.dev

q=boot.dev is a query parameter. Like headers, query parameters are key / value pairs. In this case, q is the key and boot.dev is the value.

The Documentation of an HTTP Server

You may be wondering:

How the heck am I supposed to memorize how all these different servers work???

The good news is that you don't need to. When you work with a backend server, it's the responsibility of that server's developers to provide you with instructions, or documentation that explains how to interact with it.

For example, the documentation should tell you:

• The domain of the server

• The resources you can interact with (HTTP paths)

• The supported query parameters

• The supported HTTP methods

• Anything else you'll need to know to work with the server

The server has control

As we mentioned earlier, the server has complete control over how the path in a URL is interpreted and used in a request. The same goes for query parameters.

Not all servers support query parameters for every type of request, it just depends, so you'll need to consult the docs.

Multiple Query Parameters

We mentioned that query parameters are key/value pairs - that means we can have multiple pairs.

http://example.com?firstName=lane&lastName=wagner

In the example above:

• firstName = lane

• lastName = wagner

The ? separates the query parameters from the rest of the URL. The & is then used to separate every pair of query parameters after that.

For example, make this request to limit the number of Pokémon returned from the PokeAPI to 2:

https://pokeapi.co/api/v2/location/?limit=2

What is HTTPs?

Hypertext Transfer Protocol Secure or HTTPS is an extension of the HTTP protocol. HTTPS secures the data transfer between client and server by encrypting all of the communication.

HTTPS allows a client to safely share sensitive information with the server through an HTTP request, such as credit card information, passwords, or bank account numbers.

HTTPS requires that the client use SSL or TLS to protect requests and traffic by encrypting the information in the request. HTTPS is just HTTP with extra security.

HTTP vs HTTPS

HTTPS keeps your messages private, but not your identity

We won't cover how encryption works in this course, but we will in later courses. For now, it's important to note that while HTTPS encrypts what you are saying, it doesn't necessarily protect who you are. Tools like VPNs are needed for remaining anonymous online.

HTTPS ensures that you're talking to the right person (or server)

In addition to encrypting the information within a request, HTTPS uses digital signatures to prove that you're communicating with the server that you think you are.

If a hacker were to intercept an HTTPS request by tapping into a network cable, they wouldn't be able to successfully pretend they are your bank's web server.

Assuming that a server supports HTTPs, you use it by simply changing the protocol on your request URL: https://boot.dev

Want to put what you've learned into practice with a project?

Check out this project guide where you'll build a web crawler in JavaScript from scratch. It will have you using the Fetch API and parsing JSON data like a pro! You don't need to build the project, but it's a great way to practice what you're learned.

Congratulations on making it to the end!

If you're interested in doing the interactive coding assignments and quizzes for this course you can check out the Learn HTTP course over on Boot.dev.

This course is a part of my full back-end developer career path, made up of other courses and projects if you're interested in checking those out.

If you want to see the other content I'm creating related to web development, check out some of my links below:

• Lane on Twitter

• Lane on YouTube

• Lane's Personal Site

This post relies on basic knowledge of computer networks. Be sure to check my previous post about the five layers model if you need a refresher.

What is Wireshark?

Wireshark is a sniffer, as well as a packet analyzer.

What does that mean?

You can think of a sniffer as a measuring device. We use it to examine what’s going on inside a network cable, or in the air if we are dealing with a wireless network. A sniffer shows us the data that passes through our network card.

But Wireshark does more than that. A sniffer could just display a stream of bits - ones and zeroes, that the network card sees. Wireshark is also a packer analyzer that displays lots of meaningful data about the frames that it sees.

Wireshark is an open-source and free tool, and is widely used to analyze network traffic.

Wireshark can be helpful in many cases. It might be helpful for debugging problems in your network, for instance – if you can’t connect from one computer to another, and want to understand what’s going on.

It can also help programmers. For example, imagine that you were implementing a chat program between two clients, and something was not working. In order to understand what exactly is being sent, you may use Wireshark to see the data transmitted over the wire.

So, let’s get to know Wireshark.

How to Download and Install Wireshark

Start by downloading Wireshark from its official website:

https://www.wireshark.org/#download

Follow the instructions on the installer and you should be good to go.

How to Sniff Traffic with Wireshark

Launch Wireshark, and start by sniffing some data. For that, you can hit Ctrl+K (PC) or Cmd+K (Mac) to get the Capture Options window. Notice that you can reach this window in other ways. You can go to Capture->Options. Alternatively, you can click the Capture Options icon.

I encourage you to use keyboard shortcuts and get comfortable with them right from the start, as they'll allow you to save time and work more efficiently.

So, again, I’ve used Ctrl+K (or Cmd+K) and got this screen:

The Capture Options window in Wireshark (Source: Brief)

Here we can see a list of interfaces, and I happen to have quite a few. Which one is relevant? If you’re not sure at this point, you can look at the Traffic column, and see which interfaces currently have traffic.

Here we can see that Wi-Fi 3 has got traffic going through it, as the line is high. Select the relevant network interface, and then hit Enter, or click the button Start.

Let Wireshark sniff the network for a bit, and then stop the sniff using Ctrl+E / Cmd+E. Again, this can be achieved in other ways – such as going to Capture->Stop or clicking the Stop icon.

Consider the different sections:

Wireshark's sections (Source: Brief)

The section marked in red includes Wireshark’s menu, with all kinds of interesting options.

The main toolbar is marked in blue, providing quick access to some items from the menu.

Next, marked in green, is the display filter. We will get back to it shortly, as this is one of the most important features of Wireshark.

Then follows:

The Packet List Pane

The packet list pane is marked in orange. It displays a short summary of each packet captured.

(Note: the term Frame belongs to a sequence of bytes in the Data Link layer, while a Packet is a sequence of bytes from the Network layer. In this post I will use the terms interchangeably, though to be accurate, every packet is a frame, but not every frame is a packet, as there are frames that don't hold network layer data.)

As you can see in the image above, we have a few columns here:

NUMBER (No.) – The number of the packet in the capture file. This number won’t change, even if we use filters. This is just a sequential number – the first frame that you have sniffed gets the number 1, the second frame gets the number 2, and so on.

Time – The timestamp of the packet. It shows how much time has passed from the very first packet we have sniffed until we sniffed the packet in question. Therefore, the time for packet number 1 is always 0.

Source – The address where this packet is coming from. Don’t worry if you don’t understand the format of the addresses just yet, we will cover different addresses in future tutorials.

Destination – The address where this packet is going.

Protocol – The protocol name in a short version. This will be the top protocol – that is, the protocol of the highest layer.

Length – The length of each packet, in bytes.

Info – Additional information about the packet content. This changes according to the protocol.

By clicking on packets in this pane, you control what is displayed in the other two panes which I will now describe.

The Packet Details Pane

Click on one of the captured packets. In the example below I clicked on packet number 147:

Selecting a specific packet changes the packet details pane (Source: Brief)

Now, the packet details pane displays the packet selected in the packet list pane in more detail. You can see the layers here.

In the example above, we have Ethernet II as the second layer, IPv4 as the third layer, UDP as the fourth layer, and some data as a payload.

When we click on a specific layer, we actually see the header of that layer.

Notice that we don’t see the first layer on its own. As a reminder, the first layer is responsible for transmitting a single bit – 0 or 1 – over the network (if you need a refresher about the different layers, check out this post).

The packet bytes pane in Wireshark (Source: Brief)

Below the packet details pane, we have the packet bytes pane. It displays the data from the packet selected in the packet list pane. This is the actual data being sent over the wire. We can see the data in hexadecimal base, as well as ASCII form.

How to Use the Display Filter

Wireshark has many different functions, and today we will focus on one thing – the display filter.

As you can see, once you start sniffing data, you get a LOT of traffic. But you definitely don’t want to look at everything.

Recall the example from before – using Wireshark in order to debug a chat program that you’ve implemented. In that case, you would like to see the traffic related to the chat program only.

Let’s say I want to filter only messages sent by the source address of frame number 149 ( 192.168.1.3 ). I will cover IP addresses in future posts, but for now you can see that it consists four numbers, delimited by a dot:

The display filter in Wireshark (Source: Brief)

Now, even if you don’t know how to filter only packets sent from this IP address, you can use Wireshark to show you how it’s done.

For that, go to the right field we would like to filter – in this case, the source IP address. Then right click -> and choose filter -> Apply as Filter.

Applying a display filter (Source: Brief)

After applying the filter, you only see packets that have been sent from this address. Also, you can look at the display filter line and see the command used. In this way, you can learn about the display filter syntax (in this example, it is ip.src for the IP source address field):

Applying a display filter (Source: Brief)

Now, try to filter only packets that have been sent from this address, and to the address 172.217.16.142 (as in Frame 130 in the image above). How would you do that?

Well, you could go to the relevant field – in this case, the IP destination address. Now, right click -> Apply as Filter -> and select ...and Selected:

Applying a display filter (Source: Brief)

If you look at the display filter line after applying this filter:

Applying a display filter (Source: Brief)

You can also learn that you can use the && operand in order to perform and. You could also write the word and, instead, and get the same result.

Applying multiple conditions using && or and (Source: Brief)

How to Use Wireshark to Research the Ping Utility

Ping is a useful utility to check for remote servers’ connectivity.

This page explains how to use ping in Windows, and this page explains how to do that in OSX.

Now, we can try to ping <address> using the command line. By default, ping sends 4 requests and waits for a pong answer. If we want it to send a single request, we could use -n 1:

Using the command line to ping Google (Source: Brief)

You can see that Google has responded. The time it took for the message to return was 92 milliseconds. We will learn about the meaning of TTL in future posts.

Ping is useful to determine whether a remote service is available, and how fast it is to reach that service. If it takes a very long time to reach a reliable server such as google.com, we might have a connectivity problem.

Try it yourself

Now, try to use Wireshark to answer the following questions:

1) What protocol does the ping utility use?

2) Using only Wireshark, compute the RTT (Round Trip Time) – how long it took since your ping request was sent and until the ping reply was received?

Next, run the following command:

ping -n 1 -l 342 www.google.com

3) What is the main difference between the packet sent by this command, and the packet sent by the previous command? Where in Wireshark can you see this difference, inspecting the packets?

4) What is the content (data) provided in the ping request packet? What is the content provided in the ping response packet?

Let's solve it together

So the first question is:

What protocol does the ping utility use?

To answer that question, start sniffing in Wireshark, and simply run the ping command. Stop the sniff, and consider the packets pane:

Sniffing while running ping (source: Brief)

Wireshark marks the packets as Echo (ping) request and Echo (ping) reply.

Considering these packets, we can see they consist of Ethernet for the Data Link layer (though that may differ from one network to another), IPv4 as the Network layer, and then ICMP as the protocol for Ping itself. So the answer we found is: ICMP.

Next question:

Using only Wireshark, compute the Round Trip Time

Looking at the captured packets, we can see the Time column, and subtract the time of the Pong packet ( 7.888... ) from the time of the Ping packet ( 7.796...).

So in this case the RTT was: 92 ms. Of course, the value can be different when you run the ping utility.

What is the main difference between the packet sent by this command, and the packet sent by the previous command?

For question number 3, we are asked to run the following command:

ping -n 1 -l 342 www.google.com

Looking at the first run of ping, we can see the length of the packets are 74 bytes:

Sniffing while running ping (source: Brief)

Observing the packets sent after running ping with the -l 342 argument, we can see that the value is bigger:

Sniffing while running ping (source: Brief)

So the main difference is the amount of bytes sent as the data.

Question number four:

What is the content (data) provided in the ping request packet?

What is the content provided in the ping response packet?

Click on the request packet to observe the data sent:

Observing the data sent by the ping utility (source: Brief)

The answer for the ping request is a through w, over and over again.

Regarding the ping response – it is the same as the request.

Summary

Wireshark is a wonderful tool for anyone working with Computer Networks. It can help you understand how protocols work and also help you debug applications or network issues.

As you have seen, you can learn how things work by simply running Wireshark in the background while using them and then inspect the traffic. With this tool under your belt, the sky is the limit.

In future tutorials, we will also rely on our knowledge of Wireshark and use it to further understand various concepts in computer networks.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

• Computer Networks Playlist - on my Brief channel.
• Wireshark's website.

Think of when you are using a mobile app to order food from a restaurant. The restaurant's menu, pricing, and ordering information could all be stored in a database that is managed by a back-end application.

To enable the mobile app to access the data, the app must make an API request to the back-end application. The request will include information such as the restaurant's location, the menu items, and the desired order.

The back-end application will then respond with the requested information. The mobile app can then use the data to create a user-friendly interface for ordering food.

API requests can also be used to update the database with new data, providing a way for the application to save and store new information. You can also set up with in-app subscription events. For example, when a user's subscription is about to expire, an API request is sent to a notification system to alert the them.

In this tutorial, you’ll learn how to make GET, POST, PUT and DELETE requests to APIs in a React Native app using the FetchAPI. You can access the full code from this tutorial here.

Prerequisites

To follow this tutorial, you’ll need just two things:

• A basic understanding of React Native

• Expo Snack

Expo Snack is an online development environment for React Native which essentially lets you run React Native apps on your web browser. This removes the hassle of setting up your local React Native environment from scratch.

Setting up the app

Go to Expo Snack to initialize a new React Native project, then go to the App.js file and clear the file’s content.

Start by importing React, useState, and useEffect hooks from React as well as the Text, View, and StyleSheet components from React Native.

import React,{useState, useEffect} from 'react';
import { Text, View, StyleSheet } from 'react-native';

Next, define an App component function. Inside the function body, we set the data state to an empty array, the loading state to true, then we return a simple “Hello World” text for now.

export default function App() {
  const [data, setData] = useState([]);
  const [loading, setLoading] = useState(true);
  return (
    <View>
      <Text>Hello World</Text>
    </View>
  );
}

Later on, when we retrieve data from JSONPlaceholder API (see next section), we’ll populate out the data state with it and render the data from the App component as text.

The API we’ll be working with

The API we’ll be fetching our data from is JSONPlaceholder. This is a free fake API for testing and prototyping purposes.

The API comes with 6 common resources which you can read, edit, update or delete by making API requests.

Array of 100 post objects

So here each post is an object with four properties: userId, id, title and body. There are 100 objects in the array.

You can:

• create a new post by making a POST request to this API (that is, add a new object)

• read a specific post by making a GET request to this API (that is, read an object)

• update an existing post by making a PUT request to this API (that is, modify an object)

• delete an existing post by making a DELETE request to this API (that is, remove an object)

Let’s start with making GET requests in our React Native app.

How to Make a GET Request in React Native

An API GET request is a type of API request used to retrieve data from a server. The request is sent via a HTTP GET method and data is returned in the form of a JSON or XML object.

Let’s make a GET request in our React Native app. First, store the URL you want to make the request to inside a variable (paste this code under state variable declaration):

const url = "https://jsonplaceholder.typicode.com/posts"

Next, use the fetch method to execute the API request to the URL. Wrap the fetch inside of useEffect (this hook allows us to perform side effects in our code, for example API calls):

useEffect(() => {
  fetch(url)
    .then((resp) => resp.json())
    .then((json) => setData(json))
    .catch((error) => console.error(error))
    .finally(() => setLoading(false));
}, []);

So what’s happening here inside useEffect? First we make a GET call to the URL. Once the data is returned, we parse it to JSON with resp.json(), and in the next then block we call setData to update the state with the post.

In the event of any error, the catch method will run (which logs the error to the console). Finally, we set the loading state to false.

Finally, we render the post gotten from the API:

<View style={styles.container}>
  {loading ? (
    <Text>Loading...</Text>
  ) : (
    data.map((post) => {
      return (
        <View>
          <Text style={styles.title}>{post.title}</Text>
          <Text>{post.body}</Text>
        </View>
      );
    })
  )}
</View>;

So if the API request is still in progress, we show the “Loading” text on our app. Once the data is retrieved, we loop through the array and render the title and text of each post.

To style it up a bit, paste the following stylesheet underneath App:

const styles = StyleSheet.create({
  container: {
    flex: 1,
    justifyContent: "center",
    backgroundColor: "#ecf0f1",
    padding: 8,
  },
  title: {
    fontSize: 30,
    fontWeight: "bold",
  },
});

Here’s the result when the posts are returned from the API.

Final look

As you can see, we successfully fetched the list of posts from the API and rendered each of them in our React Native app.

Now if you want to retrieve a specific resource (like a specific post) as opposed to a collection of resources like a list of posts, all you need to do is add the resource ID to the URL like so:

const url = "https://jsonplaceholder.typicode.com/posts/1"

If we make a GET request to the above API endpoint, the server will only gives us back the post with an ID of 1.

Let’s move on to other types of requests.

How to Make a POST Request in React Native

An API POST request is a type of API request that is used to create or update a resource on a web server.

It sends data to the server in the form of a request body which usually contains information such as the title, description, and other relevant details. If the data is accepted, the server responds with a success code and the resource is created or updated.

When making a POST request with the FetchAPI, you must specify the method as ‘POST’. Here’s an example of a POST to the fake server from our React Native app:

fetch("https://jsonplaceholder.typicode.com/posts", {
  method: "POST",
  headers: {
    Accept: "application/json",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    userId: 55,
    id: 101,
    title: "Post title",
    body: "Post body",
  }),
})
  .then((response) => response.json())
  .then((responseData) => {
    console.log(JSON.stringify(responseData));
  })
  .done();

The value of the body property must always be a JSON object with JSON string values (hence the JSON.stringify call).

Once the server is done processing the request, it sends back a response to tell you if the resource was created on the server or not (and why it failed).

How to Make a PUT Request in React Native

If a POST request is used to create a new resource on a server, a PUT request is used to update a specific resource on that server.

In a PUT request, you need to specify the ID of the resource you want to update on the server as well as the new values. Here’s an example which updates the title and body of post one on the server:

fetch("https://jsonplaceholder.typicode.com/posts/1", {
  method: "PUT",
  body: JSON.stringify({
    userId: 55,
    id: 101,
    title: "New Post title",
    body: "New Post body",
  }),
})
  .then((response) => response.json())
  .then((responseData) => {
    console.log(JSON.stringify(responseData));
  })
  .done();

Similar to a POST request, the server sends back a response to tell you if the resource was updated on the server or not (and why it failed).

How to Make a DELETE Request in React Native

As you might have guessed, a DELETE request is used to delete a specific resource from a server.

In a DELETE request, you only specify the ID of resource you want to delete on the server:

fetch("https://jsonplaceholder.typicode.com/posts/1", {
  method: "DELETE",
  headers: {
    Accept: "application/json",
    "Content-Type": "application/json",
  },
})
  .then((response) => response.json())
  .then((responseData) => {
    console.log(JSON.stringify(responseData));
  })
  .done();

Once the server is done with processing the request, it sends back a response to tell you if the resource was deleted on the server or not (and why it failed).

How to Integrate Other Third-Party APIs to React Native

Third-party APIs are created and maintained by organizations other than the main developer of the application. They are used to provide access to certain external data sources so that developers can incorporate them into their applications.

With the widespread use of React Native for developing mobile applications, third-party APIs can be easily integrated to create powerful and feature-rich applications.

One of the main benefits of using third-party APIs is that they are often updated frequently, which means that the developers can quickly access the latest features and data sources. This can be especially useful when developing mobile applications, as they often require access to the latest features and data sources.

Additionally, you can also use third-party APIs to add features that are hard or impossible to create in-house as it requires substantial human, financial, and time resources.

For example, there are complex APIs for Facebook login, payment processing, weather report, integrating in-app purchases infrastructure, and so on.

Conclusion

This tutorial walked you through the steps of making API requests in React Native using the built-in Fetch library.

I hope you enjoyed this as much as I did writing it. Have a great week.

Even more specifically, one amazing thing about the Internet is its ability to handle errors.

What do I mean by errors? When a packet or a frame is received by a machine, we say it contains an error if the data that had been sent is not the data that was received. For instance, a single 1 was mistakenly received as a 0 after its transmission.

This can happen due to many different reasons. Perhaps there was some disturbance in the wire where the data was transmitted – say, a child rode her bicycle over the wire. Perhaps there was some collision in the air as many people transmitted at once. Maybe it was a device's error.

Regardless of the specific reason, you still get valid data on the Internet. Without handling errors, you may read the last sentence and instead of errors read errbbb. Weird, isn't it? So how does the Internet handle errors?

There are two main approaches for handling errors – detection, and correction. We shall start by describing detection, and then talk about correction.

What is Error Detection?

When dealing with error detection, we are looking for a boolean result – True, or False. Is the frame/packet valid, or not. That is all. We don’t want to know where the error occurred. If the frame is invalid, we will simply drop it.

So when the receiver receives a frame, they will determine whether an error has occurred. If the frame is valid, they will read it. If the frame contains errors - the receiver will drop it.

_Error Detection: we only want to know if the frame/packet is valid or not. (Source: Brief)_

One method for error detection is using a checksum. A common implementation of a checksum is called CRC – Cyclic Redundancy Check.

In this post we will not trouble ourselves with the mathematical implementation of CRCs in the real world (if you're interested, check out Wikipedia). Rather, we'll simply try to understand the concept. To do so, let’s implement a very simple checksum mechanism ourselves.

Consider a protocol for transmitting 10-digit phone numbers between endpoints. This protocol is extremely simple: each packet includes exactly 10 bytes, each one representing a digit. For example, a packet might include the following digits:

5551234567

_A packet with a payload of 10 digits (Source: Brief)_

For simplicity's sake, we will omit the headers of the packet and focus solely on the payload.

Now, we will add a checksum. Say that we add all the digits. So in this example, we would calculate 5 + 5 +5 +1+… all the way through 7. We would get 43. This would be our checksum value.

Now, the sender won’t only send the phone number, but also the checksum value right after it. In this example, the sender would send:

_The packet's data is followed by a checksum. (Source: Brief)_

Now, as the receiver, you can do the same thing. You will read the phone number, and calculate the checksum. You will add the digits, and get 43.

Since you've received the correct result (that is, your calculation based on the data matches the checksum value sent in the packet), you can assume that the frame is valid.

_The sender compares their calculated checksum value and the checksum in the packet. If the values match, the packet is assumed to be valid (Source: Brief)_

What happens in case of an error? 🤔

Let’s say, for instance, that the digit 2 was replaced by an 8. Now, even though the sender sent the same stream as before ( 555123456743 ), you, as the receiver, see something a bit different:

_A packet containing an error (Source: Brief)_

Now, you are calculating the checksum, adding all the digits. You get 49. Since this value is different from the checksum value specified in the original frame, 43, the frame is considered to be invalid and you drop it.

_The sender compares their calculated checksum value and the checksum in the packet. If the values don't match, the packet is assumed to be invalid (Source: Brief)_

Are there problems with this method? 🤔

Yes, there are. Consider, for example, what happens if there are two errors – and instead of the original stream ( 555123456743 ), you receive the following:

_A packet received with two errors, resulting in the stream 456123456743 (Source: Brief)_

What happens when you add the digits?

Even though the digits are not the same as the original packet, the checksum will remain correct, and the frame will be regarded as valid.

_Despite the errors, the checksum value happens to be correct, resulting in a false assumption that the packet is valid (Source: Brief)_

Real checksum functions, such as CRCs, are of course much better implemented than the one in our example – but in extremely rare cases, such problems may occur.

Notice that using this kind of method, error detection, we don’t know where the problem occurred, but only whether the frame is valid or not. If the checksum value is invalid, we assume that the frame is invalid and drop it.

What is Error Correction?

As mentioned earlier, detection is not the only way to handle errors. Another approach might be to find the error and correct it. How can we do that?

An extremely simple way would be to transmit the data many times – let’s say, three times. For example, the stream 5551234567 would be transmitted as follows:

_Sending the same data multiple times (Source: Brief)_

So we basically sent the data three times.

Now, in case of an error in one digit, the receiver can look at the other two digits, and choose the one that appears two times out of three.

So, for instance, if we had a problem and 2 was replaced with an 8, the receiver would get this stream:

_An error in one of the occurrences of the data (Source: Brief)_

Now, as a receiver, you can say: “I have 2, 8, 2… so it was probably 2 in the original message”.

Is this problematic? Well, in some rare cases, we might get the same error twice. So it is possible, even though unlikely, that two of the original twos have been received as eights.

So while the sender sent this stream:

_Sending the same data multiple times (Source: Brief)_

The first 2 was mistakenly read as an 8, and also the second 2 was received as an 8:

_Two identical errors; Rare, but possible (Source: Brief)_

Now, it looks as if the original message included an 8, and not a 2.

What can you do in order to lower the probability of such scenario?

The most simple solution would be to simply send the data even more times. Let’s say, five times. So now we duplicate all the data, and send it 5 times in total…

_Sending the data five(!) times (Source: Brief)_

Now, say that two errors occurred, and again two of the 2 digits were replaced with 8s.

_Two identical errors; Rare, but possible (Source: Brief)_

Clearly, it is very unlikely to get the same error twice, but even in this case, we still get 2 three times, so as the receiver you can tell, with a high probability, that the original message contained a 2, rather than an 8.

What's the Overhead?

Now would be a good time to introduce the term overhead. When we say overhead, we basically mean data or time needed to convey the actual message. Let’s first understand what this term means in general, and then consider it in the context of handling errors.

Let’s say that I have a lesson to teach in my university. My goal is to teach the lesson itself, which is also called the payload in that context – that is, the actual data or message I would like to convey.

In order to teach the lesson, or to convey the payload, I first have to physically get to the university – so I get out of my home, walk to the bus station, wait for the bus, take the bus, get off the bus, walk to the building, wait for the lesson to start – and only then do I actually get to teach the lesson.

This entire process is overhead that I have to pay in order to deliver the payload, in this case – to teach the lesson.

_Overhead and Payload are two extremely important terms in Computer Networks (Source: Brief)_

The same applies in computer networks. Our payload is the data, and there is always some overhead associated with sending it.

Back to Handling Errors

In the context here – sending the data three times, as suggested earlier, means that for every byte of payload we have two bytes of overhead. If we send the data five times, then for every byte of payload, we have four bytes of overhead. That’s a LOT!

Consider error detection, on the other hand. In our example protocol for sending phone numbers, how much overhead did we have?

Recall that for every ten-digit phone number, that is ten bytes, we included a two-digit checksum value. In other words, we had two bytes of overhead for ten bytes of payload. It is clear that in our example, error detection yields much smaller overhead in comparison to error correction.

_In the sample protocol, for every ten-digit phone number (ten bytes of payload), we included a two-digit checksum value (two bytes of overhead) (Source: Brief)_

There are better ways to achieve error correction with high accuracy than to simply send the data so many times, but they are more complicated and out of scope for this post. Even with very complicated error correction techniques, they still require lots of overhead when compared to error detection.

Also, notice that except for the bytes sent as overhead in case of error correction, error detection is much simpler.

Error Correction vs Error Detection – Which is Better?

We already concluded that error detection is simpler, and with a smaller payload compared to error correction.

So, when would we prefer error correction?

One case might be when we have a one-way link. That is, a network where we can only transfer data in one direction.

For example, say you have a secret agent that you need to send a message to. The agent knows that they need to look up to the sky at exactly midnight, and they will see a series of flashes indicating the secret message.

The secret agent cannot reply, or their location and identity will be revealed. In addition, you don’t want to send the message over and over again, as not to draw much attention, and to make it harder for someone to intercept the message.

In this case, you definitely want your agent to receive the exact message that you’ve sent. Consider a case where you want to send them the message “do not place the bomb”.

_A sensitive message for a secret agent (Source: Brief)_

Of course, you don’t want to risk the unfortunate scenario of the agent reading the message as “do now place the bomb”, due to an error.

_An error may change the meaning of the message substantially (Source: Brief)_

If you use error detection, the agent might be aware that the message they received is invalid in case of an error, but they won’t be able to tell you that they need you to send the message again. As you want the agent to be able to read your message correctly and without sending any data back to us, error correction is preferred.

So, one-way link is one case where we prefer error correction. What about other cases?

Sometimes you just can’t send the data again, perhaps because it has been erased from the memory of your machine. That is, the data is deleted right after it has been sent. In this case, you'd clearly prefer error correction, as sending the data again, as we would do with error detection, is just impossible.

Also, if sending the data again is possible, but extremely expensive, error correction may be preferable.

For example, if you send a message to the moon, say, with a spaceship – it might be really expensive to send it over again in case of an error. Using error correction, you send the data only once and the receiver should be able to deal with it, even if an error occurred.

_Cases where correction is preferred (Source: Brief)_

In general, we prefer error correction when retransmitting the data is costly or impossible.

When would we prefer error detection?

Well, in case we can retransmit the data, we usually prefer error detection since it comes with very little overhead compared to error correction. Especially, when sending the data is relatively cheap.

For example, on the Internet, if an error occurs when you send a frame, no problem – you can simply send it again!

For example, when I covered the Ethernet protocol in a previous post, I mentioned that Ethernet protocol uses change detection, namely CRC32 – that is, 32 bits (or 4 bytes) of a checksum for every frame.

Note that it doesn’t mean that error detection is simply better. It just better fits the Internet than error correction. As mentioned before, error correction is preferable in other cases.

Wrapping Up

In this tutorial, we discussed various methods for handling errors. We looked at error detection, where we only know whether a frame is valid or not. We also considered error correction, where the receiver can restore the correct value of an erroneous frame. We also introduced the term overhead.

We then understood why we use error detection on the Internet, rather than error correction. Stay tuned for more posts in this series about Computer Networks 💪🏻

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional Resources

• Computer Networks Playlist - on my Brief channel
• CRC - Wikipedia
• The Complete Guide to Ethernet Protocol