This handbook will take you through every aspect of IPv4, from understanding IP addresses to examining packet headers and fragmentation. You'll learn:

• How IP addresses work and their different formats

• Network addressing schemes from fixed-length to CIDR

• Special IPv4 addresses and their uses

• The structure and purpose of every field in the IPv4 header

• How IPv4 handles packet fragmentation across different networks

Whether you're a network engineer, software developer, or IT professional, understanding IPv4 is crucial for working with modern computer networks.

What we’ll cover:

1. Background

2. Understanding IP Addresses

3. Network ID and Host ID

4. How to Determine Network vs. Host Portions

   • Fixed-Length Approach

   • What are the disadvantages here? 🤔

5. Classful Addressing

   • IP Address Assignment

   • What are the disadvantages here? 🤔

6. CIDR: Classless Interdomain Routing

   • Real-world Example

7. Subnet Masks

8. Interim Summary – IPv4 Addresses

9. Test Yourself

   • Converting Between Prefix Notation and Subnet Masks

   • Working Backwards with Subnet Masks

   • Non-Byte-Aligned Prefixes

   • Determining Network Membership

10. Special IPv4 Addresses

    • The "This Host" Address: 0.0.0.0

    • "This Network" Addresses

    • Broadcast Addresses

    • Loopback Addresses: 127.0.0.0/8

    • Summary of Special IPv4 Addresses

11. IPv4 Header

    • The Header Structure

    • IPv4 Header – Interim Summary

12. IPv4 Fragmentation

    • Why Fragmentation Is Needed

    • How Fragmentation Works in IP

    • Identification Field

    • Fragment Offset

    • More Fragments and Don't Fragment Flags

    • Fragmentation Example

    • IPv4 Fragmentation – Summary

13. Summary – IPv4

    • Addressing and Network Structure

    • IPv4 Header Structure

    • Fragmentation

    • Final Words

14. About the Author

15. Additional References

Quick notes before we start

1. You can find more content about computer networks on my YouTube channel: Computer Networks Playlist

2. I am working on a book about Computer Networks! Are you interested in reading the initial versions and providing feedback? Send me an email: gitting.things@gmail.com

Background

IP stands for "Internet Protocol", so IPv4 is Internet Protocol version 4. It was described in RFC 791 by IETF, published in September 1981, and first deployed for production in 1982 on SATNET (the Atlantic Packet Satellite Network), which was an early satellite network that formed an initial segment of the Internet.

IPv4 is connectionless and operates in a best-effort delivery model. This means it doesn't guarantee delivery, correct ordering of packets, or the validity of the data. It's designed to be fast and flexible.

Understanding IP Addresses

IP addresses are hierarchical, logical addresses that power most internet connections today. Each consists of 4 bytes, or 32 bits. They're usually written in dotted decimal notation, for example:

Test yourself – Does the following address represent a valid IP address?

No. Since the dots separate different bytes, each value must be between 0 and 255. Since the number 392 is bigger than 255, it cannot be represented in a single byte.

Network ID and Host ID

IP addresses have two parts: a network identifier (or network ID) that belongs to all hosts in the network and a host identifier (or host ID) that identifies the specific host in this network.

The network identifier will be the same for all hosts in the network, and is also called a "prefix". For example, consider a network identifier of 201.22.3. Given that this is the network prefix, the following addresses:

201.22.3.15
201.22.3.91

Are part of the same network, as they share the same prefix. The first address belongs to host number 15 in this network, and the second belongs to host number 91.

This address has a different prefix, or a different network identifier, and thus belongs to a different network:

201.22.14.50

In the examples above, there's a network identifier consisting of 3 bytes, or 24 bits, and a host identifier consisting of 1 byte, or 8 bits.

How to Determine Network vs. Host Portions

A question arises: how do you know which bits are part of the network ID, and which are part of the host ID? Several approaches have evolved over time to address this challenge.

Fixed-Length Approach

Let's consider this solution: For every IP address, the first, most-significant byte would represent the network ID, and the remaining three, least-significant bytes would represent the host ID. This way it's really easy to read IP addresses. For example for this address:

20.12.1.92

You know that it describes network 20, and the host 12.1.92 inside that network. Any IP address that doesn't start with 20, such as 22.1.2.3, would reside in a different network, and any IP address that starts with 20, like 20.1.2.3, would be within the same network.

What are the disadvantages here? 🤔

With only one byte (8 bits) to represent the network ID, you only have 2^8, or 256, different networks. Of course, there are far more networks than that in the real world. Even in the early days of the internet, universities and large companies each needed their own network identifiers.

In general, using a fixed length for the network ID and a fixed length for the host ID is not flexible enough. If you decide that the two most-significant bytes will represent the network ID and the two least-significant bytes will represent the host ID, you can represent up to 2^16, or 65,536 networks, which is also not enough. Furthermore, some networks, such as those of large companies, might require more than 65,536 host IDs.

Classful Addressing

The solution lies in providing some flexibility. Consider another approach called "classful addressing". In this approach, the number of bits dedicated for the network ID changes from one address to another, and you can tell the network ID by looking at the first, most-significant byte of the address.

• Any address starting with a number between 1 and 127 belongs to "Class A", meaning that its network ID consists of 1 byte, leaving 3 bytes for the host ID.

• Any address starting with a number between 128 and 191 belongs to "Class B", which means that its network ID is 2 bytes long, and its host ID is also 2 bytes long.

• Any address starting with a number between 192 and 223 belongs to "Class C", so it has 3 bytes of a network ID, and 1 byte of host ID.

You can see the full representation of this approach in the table below:

For example, what class does this address belong to?

(1) 130.12.204.5

Since it starts with 130, which is between 128 and 191, it belongs to "Class B". This means that its network ID is 130.12, and its host ID is 204.5. Let's mark it as "address number 1".

Do this address and the following address (2) belong to the same network?

(2) 130.90.2.40

No, since they have different network identifiers, they are not within the same network.

What class does the following address belong to?

(3) 200.1.1.9

It belongs to class C, as the value of its first byte, 200, is between 192 and 223. This means that its network identifier is 200.1.1, and any address starting with this prefix will reside within the same network. This specific address describes host 9 within this network.

To complete the picture, addresses starting with a value between 224 and 239 belong to "Class D" – that is, multicast addresses – addresses that belong to multiple devices. Addresses starting with a value between 240 and 255 were reserved for future use. Addresses starting with 0 are special addresses.

IP Address Assignment

In the early internet, IPv4 addresses were assigned to organizations by the Internet Assigned Numbers Authority (IANA). As the internet grew, this responsibility was distributed to five Regional Internet Registries (RIRs) that handle address allocation for different geographic regions. Large organizations would receive blocks of addresses based on their needs, with address classes determining the size of these blocks.

What are the disadvantages here? 🤔

While classful addressing allows for more flexibility compared to the fixed-length approach, even this approach isn't flexible enough.

Consider this scenario: A small startup company with just two founders needs a network identifier. Which class would they need?

Getting a class A or class B would be excessive, so they might get a class C – allowing 256 addresses. This is more than currently needed, but allows some expansion. What happens if the startup grows to more than 256 employees (and devices)?

At this point, they would need to get a class B address, giving no less than 65,536 addresses, when all they need is a bit over 256 addresses. This means wasting more than 60,000 addresses.

This became a real problem in the early 1990s as the internet was growing faster. The need for more IP addresses became apparent, and there was an impending exhaustion of the IPv4 address space. Cases where 60,000 addresses were wasted could no longer be tolerated.

CIDR: Classless Interdomain Routing

One of the measures to handle this shortage of addresses was to abandon classful addressing in 1993 and switch to another approach called CIDR – Classless Interdomain Routing. This approach is still used today.

CIDR allows for flexibility when choosing the network ID and the host ID. It lets network administrators create subnets of precisely the right size, rather than being limited to Classes A, B, or C.

Let's start with a simple example. In CIDR notation, we add a suffix indicating how many bits are used for the network portion:

(4) 200.8.3.1/16

This slash notation specifies how many bits describe the network ID. In example (4) above, the first 16 bits (or 2 bytes) are used for the network ID. So, in this case, 200.8 is the network identifier, and 3.1 is the host identifier. The fact that 200.8 is the network ID means that all addresses from 200.8.0.0 through 200.8.255.255 are in this network.

Consider these additional addresses:

(5) 200.2.13.5
(6) 200.8.21.6

Given this address prefix of 16 bits, or 2 bytes, which of these addresses belong to the same network as example (4) (200.8.3.1/16)?

The first address (5) (200.2.13.5) does not belong to this network, as its first 16 bits – 200.2, are different from the first 16 bits of the example address.

The second address (6) (200.8.21.6) does belong to the same network as that of the example address.

Real-world Example

In practice, an ISP might receive a large block like 104.16.0.0/12 from the RIR. This gives them control of all addresses from 104.16.0.0 to 104.31.255.255. The ISP can then allocate smaller subnets to customers, such as giving a small business a /24 subnet with 256 addresses, or a larger company a /20 subnet with 4,096 addresses.

Subnet Masks

Another way to express the network prefix is by using a subnet mask, like so:

255.255.0.0

When converted to binary, 255 in decimal equals eight 1s in binary – so all bits are on. So if you translate this mask into binary, you get:

11111111 11111111 00000000 00000000

In other words, 16 bits are on, which means a network prefix of 16 bits. Both conventions (CIDR notation and subnet masks) are used very frequently.

With CIDR, an address can reside in different networks given different network prefixes, or subnet masks. If you consider the same example address with a different prefix, say that of 8 bits – both additional addresses would belong to the same network, as they all share the first 8 bits – 200.

How would you present a network prefix of 8 bits as a subnet mask? You need the first 8 bits to be on, so that means 255 in decimal, and the remaining bits are off, resulting in this subnet mask:

255.0.0.0

What happens if you use a network prefix of 24 bits? First, how would you express that as a subnet mask? You need 24 bits to be on, so that is 3 times 8 bits to be on, resulting in:

255.255.255.0

Now, neither of the additional addresses reside within the same network as the example address, as they don't share its network ID of 200.8.3.

Note that network prefixes do not have to represent full bytes. For example, you can use a network prefix of 12 bits, or 11 bits, or 22 bits. When the prefix length isn't a multiple of 8, the subnet mask will have a value other than 0 or 255 in one of its positions.

This addresses the issue regarding the startup company. If a startup has 300 employees, they'd need to get a 23-bits network ID, leaving 9 bits for hosts within their networks. This means 2^9, or 512 addresses, which should be sufficient.

Interim Summary – IPv4 Addresses

In this section, you've learned about IPv4 addresses. IP addresses are hierarchical, logical addresses that consist of 4 bytes. IP addresses have two parts: a network identifier that belongs to all hosts in the network, and a host identifier which identifies the specific host in the network.

You've explored various options for determining the network identifier and the host identifier:

1. Fixed-length approach – too rigid and limited

2. Classful addressing approach – better but still wasteful

3. CIDR (Classless Interdomain Routing) – flexible and efficient

CIDR provides much more flexibility and helps overcome the significant problem of IPv4 address shortage. However, CIDR is only one part of addressing the shortage of IPv4 addresses, with other solutions including NAT (Network Address Translation) and eventually, IPv6.

The next section will explore special IPv4 addresses and then examine the header of IPv4 packets.

Test Yourself

Now practice the concepts you've learned and make sure you feel comfortable with them.

Take a moment to try answering the following questions before checking the answers.

Converting Between Prefix Notation and Subnet Masks

How would you represent a network prefix of 16 bits, written like this /16, as a subnet mask?

You need 16 bits that are on. When 8 bits are on you get 255 in decimal, so you'd use:

255.255.0.0

Given this network prefix, do these addresses belong to the same network?

Yes, they do, as they share the same most-significant 16 bits, or two bytes

Does this address belong to the same network as that of the previous addresses?

Yes, it does. Again, it shares the same two most-significant bytes.

What about this one? Does it belong to the same network as the previous addresses?

No, as the first two bytes are not 42.31 – this is a different network. So this address describes host 1.2, within the network 42.32.

Working Backwards with Subnet Masks

Let's try the other way around. You have this subnet mask:

255.255.255.0

How would you express it using a network prefix?

You have three occurrences of 255, which means three times 8 bits that are on, so overall you have 24 bits that are on. So you can also write /24. This means 3 bytes.

Given this subnet mask, do addresses (1) and (3) above belong to the same network?

They do, as they both have the same most-significant three bytes – network 42.31.93.

What about addresses (1) and (2)?

Given this network prefix, they don't belong to the same network. The first address belongs to network 42.31.93, and the second address belongs to network 42.31.1.

Non-Byte-Aligned Prefixes

Network prefixes do not have to align to 8 bits, or full bytes. Let's say you have a network prefix of 14 bits. How would you convert that to a subnet mask?

Well, the first byte is clear: you have 8 bits on, so the first byte is 255. What about the next one?

In binary, you'd want to have six additional 1s, and then 2 0s – so in binary you'd write:

11111100

Converting to decimal, this binary number represents 252. So your subnet mask is:

255.252.0.0

Another way to make this conversion: You know that eight 1s in binary represent 255 in decimal. You also know that 11 in binary is 3, so you can simply subtract 3 from 255 and get 252.

Next, try the other way around. You have the following subnet mask:

255.255.224.0

How many bits represent the network prefix?

The first two bytes are clear: you have 16 bits. Converting the third byte to binary: 224 in decimal is 11100000 in binary. This means you have an additional three 1s, so you can write the subnet mask above as a prefix of /19 bits – 16 bits for the two 255 bytes, and 3 additional bits for the 224 byte.

Determining Network Membership

Let's consider the following addresses:

Are they part of the same network? 🤔

It depends on the subnet mask.

If the network prefix is /8, then they are part of the same network, as they share the same network ID.

On the other hand, if the network prefix is /16, then they have different network IDs, and thus don't belong to the same network. But what happens with prefixes in between? Will they reside in the same network for a prefix of /9? /14?

The way to approach this question is to convert the second byte of these addresses to binary. For the first address, this byte is 24, which in binary is:

00011000

For the second address, the second byte is 23, which in binary is:

00010111

You can see that the most significant 4 bits within the second byte are identical. If you add the first 8 bits of the address, you see that the most significant 12 bits of these addresses are the same.

So, if you have a network prefix of /11, do these addresses belong to the same network?

Yes, they do – their most significant 11 bits are identical.

What about /13?

No, with this network prefix, they don't share the same network identifier, as their 13th bit is different.

This practice should help you feel comfortable with subnet masks and network prefixes. In the next section, you'll learn about special IP addresses and then examine the header of IP packets.

Special IPv4 Addresses

Now that you're comfortable with IP addresses and subnet masks, let's explore some IP addresses that have special meanings.

The "This Host" Address: 0.0.0.0

The address 0.0.0.0 means "this host" and is used in two scenarios:

First, when a machine boots up and doesn't yet have an IP address. IP addresses are logical addresses that need to be assigned to a machine. Prior to this assignment, a device has no IP address at all. If the device needs to communicate at this stage, it may use this special address, 0.0.0.0.

Second, when writing network applications that need to listen for incoming connections on all network interfaces. For example, if a machine has two interfaces – one with the IP address 1.1.1.1, and another with the address 2.2.2.2 – listening on the address 0.0.0.0 means accepting connections regardless of which network interface receives them.

"This Network" Addresses

Another class of special addresses are those starting with zeros, where the zeros mean "this network."

For example, if you have a machine with the address:

12.34.55.55

And a network prefix of 16 bits, this machine can send a packet to another device on the network using its full address, for example 12.34.66.66, or alternatively use the special zeros notation and send the packet to:

0.0.66.66

This means "send a packet to the host 66.66 on this network." Of course, the recipient must also know the relevant network prefix to correctly interpret this address.

Broadcast Addresses

The address 255.255.255.255, where all bits are set to 1, is the address of all hosts in the local network – the broadcast address. This is similar to the broadcast address in Ethernet (FF:FF:FF:FF:FF:FF). In both cases, all bits are set to 1.

Using a proper network identifier where the host identifier is all set to 1s can be used to send a broadcast packet to remote networks. For example, consider a network 12.34.0.0/16 and another network with the network ID of 12.35.0.0/16. If a machine at 12.34.55.55 wants to send a packet to all devices in the other network, it could use the destination address: 12.35.255.255.

Even though this is allowed according to the IP specification (RFC), in practice this feature is often disabled as it can create security vulnerabilities.

Loopback Addresses: 127.0.0.0/8

All addresses in the network 127.0.0.0/8 (that is, all addresses that start with 127) are loopback addresses. Packets sent to any of these addresses are not put onto the physical network but are processed locally within the operating system. This is extremely useful for development and debugging.

For example, when developing a simple chat program, you need two clients that exchange data. One approach would be to use two different physical computers, but this is tedious – you'd need to write a message on one computer, check the other computer to see if it was received, then write a message on the second computer, and go back to the first to validate receipt.

A much simpler approach is to use a loopback address. Both clients can run on the same machine and connect with one another. You can run two different client programs on the same physical computer and exchange messages between them without needing an additional machine.

For instance, you might use the address 127.0.0.1, with one client listening on port 1337 and the other on port 1338. When client A sends a packet to client B, this packet never leaves your network card but remains within the operating system. Client B receives the packet from the loopback interface as if it had been received from the physical network.

After debugging is complete, your client code doesn't need to change – the only difference is that they will communicate using real IP addresses instead of the loopback address.

Summary of Special IPv4 Addresses

To summarize the special IPv4 addresses you've learned about:

In the next section, you'll learn about the structure of the IPv4 header.

IPv4 Header

Now that you understand IP addresses, subnets, and special addresses, it's time to examine the IPv4 header structure in detail.

The Header Structure

The diagram above shows the header of IPv4 as defined in RFC 791. Let's examine each field:

Version (4 bits)

The header starts with the Version field, which consists of four bits. For an IPv4 packet, the version is 4, so this field will always carry the value of 4 (or 0100 in binary).

❓ Why does the header start with the Version field? 🤔

(Note – when I start a sentence with the ❓mark – it’s a question addressed at you, and I encourage you to try and answer it before reading on).

The reason is that the remaining fields may differ according to the version. If a network device reads an IP packet and the version field carries the value of 4, it will expect the remainder of the packet to follow the IPv4 structure. If it carries another value, such as 6, the remaining fields are different, as in IPv6.

Internet Header Length (IHL) (4 bits)

This field indicates the length of the header itself.

❓ Why do we need to specify the length? 🤔

Unlike Ethernet, where the header size is fixed, the IPv4 header length can vary because of optional fields. For an IP packet without special options, the header consists of 20 bytes, which is the most common case.

The IHL field doesn't specify the length in bytes directly but in units of 4-byte words. So to specify a length of 20 bytes, the value would be 5 (5 × 4 = 20). This encoding allows the field to use only 4 bits while specifying header lengths up to 60 bytes (when IHL = 15).

A common IPv4 packet therefore begins with the byte 0x45 in hexadecimal, meaning it's version 4 of the IP protocol, and the header is 20 bytes long.

Type of Service (TOS) (8 bits)

The idea behind this field is that not all packets are equally important. You may want to give priority to some packets over others.

For example, packets carrying real-time data (like voice or video conferencing) are more time-sensitive than packets carrying, say, email or file downloads. If a router is currently experiencing high load, it should ideally prioritize time-sensitive packets.

The Type of Service field allows senders to indicate the priority of their packets. However, on the public internet, this field is often ignored by routers because any sender can set any priority value. In most cases, this field carries the value of 0.

Total Length (16 bits)

This field specifies the total length of the IP packet, including both the header and the payload (data).

❓ Why is this needed to specify the length? 🤔

Unfortunately, the IP layer doesn’t necessarily know if some of the bytes in the packet are actually a padding of the second layer. I described this in detail in a previous post, where I showed that in Ethernet protocol, in some cases, the receiving Ethernet entity cannot tell which bytes belong to the payload and which bytes are simply padding. The IP layer needs to know precisely which bytes belong to the actual packet, hence the Total Length field.

❓What is the maximum size of an IPv4 packet? 🤔

Since this field is 16 bits long, an IPv4 packet may contain a maximum of 2^16-1 bytes, or 65,535 bytes, including the header. The minimum size is 20 bytes, consisting of just the header without options or payload.

Fragmentation Fields (32 bits)

The next four bytes are dedicated to fragmentation control. I’ll cover these fields in a separate section, as they involve a complex topic deserving special attention.

Time to Live (8 bits)

Despite its name, this field doesn't actually measure time but rather the maximum number of routing hops a packet can traverse before being discarded.

To understand its purpose, consider this scenario: If Machine A sends a packet to Machine B through a series of routers, but there's a routing loop where Router 2 sends to Router 3, which sends to Router 4, which sends back to Router 2, the packet could circulate indefinitely, consuming bandwidth and never reaching its destination.

The TTL field prevents this by setting a limit on how many hops a packet can take:

1. The sender sets an initial TTL value (often 64 or 128)

2. Each router that handles the packet decrements the TTL by 1

3. If a router receives a packet with TTL = 1, it decrements it to 0 and discards the packet

4. The router then sends an ICMP "Time Exceeded" message back to the original sender

This doesn't solve the underlying problem of routing loops, but it prevents packets from circulating forever.

In IPv6, this field is renamed "Hop Limit," which more accurately describes its function.

Protocol (8 bits)

This field describes the payload of the IPv4 packet. For example:

• A value of 6 means the payload is TCP

• A value of 17 means the payload is UDP

This helps the receiving system know which protocol handler should process the packet's contents. It's similar to the Type field in Ethernet, which specifies the protocol of the layer encapsulated within the Ethernet frame.

Header Checksum (16 bits)

This is a 16-bit checksum used to verify the validity of the header only (that is, excluding the payload). The sender computes this value based on the fields of the header, and the receiver also computes it to validate that the header was received correctly.

❓The checksum must be recalculated by each router. Why is that? 🤔

Because the TTL field changes at each hop. For example, if a packet starts with TTL = 7, each router will:

1. Verify the current checksum based on TTL = 7

2. Decrement TTL to 6

3. Calculate a new checksum based on TTL = 6

4. Forward the packet with the new checksum

If the checksum verification fails, the device drops the packet. This prevents packets with corrupted headers (which might have incorrect destination addresses, for instance) from being forwarded.

Source and Destination Addresses (32 bits each)

These fields contain the source and destination IPv4 addresses, respectively. Each is 4 bytes (32 bits) long, as you learned in the previous sections on IPv4 addressing.

Options (Variable Length)

Most IPv4 packets don't include options, but when present, they can provide additional functionality:

• Record Route: Each router that handles the packet adds its own address to this option, creating a trace of the packet's path

• Source Routing: Allows the sender to specify the route the packet should take:

  • Strict Source Routing: The entire route must be followed exactly

  • Loose Source Routing: Certain routers must be traversed, but the exact path between them is flexible

Padding

In some cases, the header ends with padding bytes (usually 0s).

❓Why does the IPv4 header have padding?🤔

As explained before, the IHL field specifies the header length in 4-byte units, so the total header length must be a multiple of 4 bytes. If options make the header length not divisible by 4, padding bytes (usually 0) are added to reach the next multiple of 4.

For example, if you have 3 bytes of options, you would need 1 byte of padding to make the total header length a multiple of 4 bytes.

IPv4 Header – Interim Summary

You've now learned about the structure of the IPv4 header, with the exception of the fragmentation fields which I’ll cover in the next section.

The IPv4 header efficiently packs all the necessary routing and control information into a compact structure, typically 20 bytes long (without options). This design allows for fast processing by routers while providing the flexibility needed for internet communication. It is amazing how prominent IPv4 is, even so many years after its publication.

In the next section, you'll learn about IPv4 fragmentation.

IPv4 Fragmentation

In the previous section, you learned about most of the IPv4 header structure, with the exception of 32 bits dedicated to fragmentation. This topic deserves special attention, as it reveals important aspects of how IP packets travel across different networks.

Why Fragmentation Is Needed

To understand what fragmentation is and why it's needed, consider the following network scenario:

In this diagram, you have two different networks where Machine A resides in one network and Machine B resides in another. A router forwards packets between these two networks.

These two networks have different Maximum Transmission Units (MTUs). MTU refers to the maximum size of a frame that can be transmitted in a network. For example:

• Machine B is connected to an Ethernet network with an MTU of 1500 bytes

• Machine A is connected to a different network with an MTU of 2000 bytes

Different MTUs stem from the different protocols and hardware that different networks have. Ethernet has an MTU of 1500 bytes. This maximum size was chosen because RAM was expensive back in the late 1970s when Ethernet was planned, and a receiver would need more RAM if a frame could be bigger. Other networks were devised at different times where RAM prices might have been lower, or just have other considerations that affect the MTU.

Now, consider this scenario: Machine A wants to send a packet to Machine B. This packet is 1800 bytes long. From A's perspective, there's no problem since its network supports packets of this size. Machine A transmits the packet.

When the router receives this packet, it faces a problem: it cannot simply forward the packet to B's network because the packet is too big for the network's MTU. The router must fragment the packet – splitting it into smaller chunks of up to 1500 bytes, which will then be reassembled by Machine B.

How Fragmentation Works in IP

Let's examine the scenario further. The router needs to take an IP packet of 1800 bytes and split it into two fragments, each consisting of up to 1500 bytes. If Machine A sends another packet of 1800 bytes to Machine B, the router will have to split that one too – resulting in four different fragments that will be reassembled into two separate packets.

When Machine B receives these fragments, it must ensure that it reassembles fragment #1 together with fragment #2 of packet A, and fragment #1 with fragment #2 of packet B – and not, for instance, fragment #1 of packet A with fragment #2 of packet B. It must also reassemble the fragments in the correct order – so structure a packet that consists of #1#2 and not #2#1.

Identification Field

First, focus on making sure Machine B reassembles fragments of the same packet (for example, fragment #1 and fragment #2 of packet A in the example above, rather than fragment #1 of packet A and fragment #2 of packet B). This is achieved using the identification field of IPv4. Fragments belonging to the same packet will have the same identification value. For example, both fragments of packet A might have identification set to 100, and both fragments of packet B might have identification of 200.

It's important to note that sharing identification values isn't sufficient for fragments to belong to the same packet. Fragments of the same packet must also share:

• The same source IP address

• The same destination IP address

• The same protocol value (indicating whether the payload is TCP, UDP, and so on)

Fragment Offset

Since IP is a connectionless protocol, there's no guarantee that fragments will arrive at Machine B in the correct order. Fragment #2 of packet A may arrive before fragment #1. To handle this issue, each fragment carries an Offset field, which denotes the offset from the beginning of the original packet.

The Offset field consists of 13 bits, which means it can carry values from 0 to 8191 (2^13-1). This poses a potential problem, as the maximum size of an IP packet can be 65,535 bytes (since the Total Length field of the IP header consists of 16 bits).

To address this limitation, the value encoded in the Offset field is actually multiplied by 8 (2^3). This means the minimum size of a fragment is 8 bytes, with the exception of the last fragment.

❓Why do IP packets carry an offset in bytes divided by 8, instead of just a sequential fragment number?🤔

While using sequence numbers might seem simpler, it would create problems when packets need to be fragmented multiple times.

For example, if Computer A sends a packet to the first router, which fragments it into pieces of 1480 bytes and 320 bytes, and then these fragments are sent to another router that needs to fragment them again into even smaller pieces, how would you number them?

With byte offsets, the solution is straightforward – if the first fragment has an offset of 0 and the next one has an offset of 1480, then if we need to split them into maximum 800-byte fragments, we'd have:

• First fragment: 800 bytes with offset 0

• Second fragment: 680 bytes with offset 800

• Third fragment: 320 bytes with offset 1480

More Fragments and Don't Fragment Flags

When Machine B receives a fragment, it needs to know whether this is an entire packet by itself or if it should expect additional fragments. For this purpose, each IP fragment carries a More Fragments (MF) bit that is set to 1 for every fragment that is not the last fragment of the packet. For the last fragment, it's set to 0.

In case the packet consists of a single fragment – the MF bit will be set to 0, and the offset field will also hold the value 0 (that is, 13 bits of 0s).

Another bit related to fragmentation is the Don't Fragment (DF) bit. When this flag is turned on, intermediate devices should not fragment the original packet, even if it exceeds the MTU. Instead, they should drop it and typically send an ICMP "Fragmentation Needed" message back to the source.

In our example, if Machine A sets the Don't Fragment bit to 1, the router would drop the packet, and notify Machine A about it.

Note that right after the identification field and before the DF flag, there is a reserved bit set to 0. This bit was reserved in case it is needed in the future, for a reason unknown to the original authors of IPv4.

Fragmentation Example

Consider again our example above – with Machine A residing in a network where the MTU is 2000, and Machine B residing in a network where the MTU is 1500. Machine A sends a packet which is 1800 bytes long.

❓Can you fill the values in these tables?

First Fragment:

Second Fragment:

For our example above, the values of the relevant fragmentation fields in IP would be as follows:

First Fragment:

• Total Length: 1500 (including 20 bytes of IP header, so 1480 bytes of payload)

• Identification: 1337 (arbitrary value)

• Don't Fragment bit: 0 (off, to allow further fragmentation if needed)

• More Fragments bit: 1 (on, as this is not the last fragment)

• Offset: 0 (it's the first fragment)

Second Fragment:

• Total Length: 340 (including 20 bytes of IP header, so 320 bytes of payload – together with the first fragment, we get to 1800 bytes of payload)

• Identification: 1337 (same as first fragment, indicating they belong together)

• Don't Fragment bit: 0 (off, to allow further fragmentation if needed)

• More Fragments bit: 0 (off, as this is the last fragment)

• Offset: 185 (1480/8 = 185, or 0xB9 in hexadecimal)

IPv4 Fragmentation – Summary

You've now learned about the final part of the IPv4 Header: fragmentation. Fragmentation is necessary to allow packets to travel across networks with different MTUs. The IPv4 header includes several fields specifically designed to support fragmentation:

• Identification (16 bits): Identifies which fragments belong together

• Flags (3 bits): Including the "More Fragments" and "Don't Fragment" flags

• Fragment Offset (13 bits): Indicates where in the original packet this fragment belongs

With this knowledge, you now understand every bit and byte of the IPv4 header and how IP packets can traverse networks with different characteristics.

Summary – IPv4

In this comprehensive guide to IPv4, you've learned about the fundamental building blocks of Internet communications. Let's recap the key concepts we covered:

Addressing and Network Structure

• IPv4 addresses are 32-bit numbers typically written in dotted decimal notation

• Networks can be identified using various methods:

  • Fixed-length approach (historically)

  • Classful addressing (A, B, C, D, E classes)

  • CIDR (modern approach allowing flexible network sizes)

• Special addresses serve specific purposes:

  • 0.0.0.0 for "this host"

  • 127.0.0.0/8 for loopback

  • 255.255.255.255 for broadcast

IPv4 Header Structure

• The header contains crucial fields for packet routing and processing:

  • Version and IHL for header interpretation

  • Type of Service for traffic prioritization

  • Total Length for packet size

  • Various fields for fragmentation control

  • TTL to prevent infinite routing loops

  • Protocol to identify the encapsulated protocol

  • Checksum for error detection

  • Source and destination addresses

Fragmentation

• Allows IPv4 packets to traverse networks with different MTUs

• Uses three key fields:

  • Identification to group fragments

  • Flags to control fragmentation

  • Fragment Offset to reassemble packets

Final Words

While IPv4 has limitations, particularly its address space constraints, its elegant design and robust features have allowed it to remain the backbone of the Internet for over four decades. Understanding IPv4 provides essential context for working with modern networks and helps in transitioning to newer protocols like IPv6.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Gitting Things Done (in English) and Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

• Computer Networks Playlist - on my Brief channel

However, have you ever attempted to troubleshoot your network by applying the OSI Model? Through the use of a bottom-to-top methodology that is based on the Open Systems Interconnection (OSI) architecture, we will uncover the complexities of network troubleshooting, providing you with the knowledge and tools that are essential for effectively addressing a wide variety of networking difficulties.

What is the OSI Model (Open Systems Interconnection)?

The Open Systems Interconnection (OSI) model is a conceptual framework that categorizes the functions of network communications into seven distinct levels. To put it simply, the OSI standardizes how various computer systems can communicate with one another.

seven layers of the OSI model

How to Troubleshoot a Website by Applying the OSI Model Principles

Consider the following example of troubleshooting a website hosted on your server that is not working. We'll use Linux as our operating system. I believe that the divide and rule is a better technique for debugging.

The OSI model is one method for efficiently breaking down an issue so that you can methodically simplify the environment in order to discover a solution and conquer it.

Physical Layer

As I previously stated, when it comes to debugging, it is usually preferable to begin from the bottom. The physical layer is the bottom layer in the OSI Model. The key components in this layer consist of ethernet cables, hubs, and switches. At this level, you should check the power supply and the status of devices, as well as examine interface statistics.

• The "ifconfig" tool provides a detailed overview of all the ethernet cards present in your system.
• In addition, you have a choice of using the "IP link show" commands. If the result shows "down," it suggests that layer1 is not functioning.
• Sometimes, ethernet connections may be physically connected to the server but not activated by default. To enable, use the command below.

IP link set eth0 up

• If you're looking for more detailed information, the ethtool utility can be quite helpful. This utility provides the ability to query and modify settings. It allows you to adjust parameters such as speed, port, auto-negotiation, PCI locations, and checksum offload.

Data Link Layer

The data link layer enables the transmission of data between two devices that are connected to the same network. There are two components in this layer. The first component is the medium access control (MAC) layer, which includes the operation of hardware addressing and access control.

The second layer is the logical link layer, which enables the creation of a logical connection between different media. A common issue in this layer is the inability of two servers to establish connectivity. Tools such as ping, traceroute, arp, macof, and Wireshark are utilized for testing the data link layer.

This may help in verifying correct transmission and reception of data frames among devices within the same network group.

Network Layer

The network layer's job is to make it easy for data to move between two networks. Network devices that work at Layer 3 of the OSI model are routers. A router's main job is to make it easier for networks to talk to each other. Working with IP addresses is part of this layer.

In this stage, you should mostly look for problems with IP addresses. You can type "ip -br address show" to see the address. You can see if your network card has been given an IP address. You might not be getting dynamic IP addresses from DHCP if you use it to get them.

One common problem that often comes is the lack of an upstream gateway for a specific route or the absence of a default route. When an IP packet is transmitted to a different network, it needs to be directed to a gateway for additional processing.

Understanding the routing of packets to their final destinations is crucial for the gateway. The routing table contains the list of gateways for various routes and can be managed using the “ip route” commands. We can also check connectivity by sending pings to the default gateway or beyond gateway.

Transport Layer

Protocols like Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) are used by the transport layer to control network traffic between systems and make sure that data flows efficiently.

The transport layer is in charge of sending data packets, looking for errors, controlling the flow of data, and putting them in the right order. You may run into problems in this layer, like ports that aren't listening. Your service might not start because the port is already being used. You can see what ports are open by running "commad "netstat -antlp | grep "LISTEN"".

One problem that often occurs is related to remote connectivity. Consider a scenario where your local system is unable to establish a connection with a distant port, specifically HTTP on port 80. The telnet command tries to create a TCP connection with the specified host and port. This capability is ideal for conducting remote TCP connectivity testing.

To check a remote UDP port, you can utilize the "netcat" utility.

Session Layer

This layer is responsible for facilitating the initiation and termination of communication between the two devices (for example: authentication). The period of time during which communication is initiated and terminated is referred to as the session.

In this layer you should be investigating credentials, certificates of the servers, the session ID and cookies of the clients

Presentation Layer

The presentation layer of the OSI model is responsible for formatting and transforming data in a way that allows it to be presented to the user.

SSL or TLS encryption methods are key parts of this layer. Here, you should be examining for encryption and decryption issues.

Application Layer

The system takes input from the user and transmits output back to the user. The Bellow Protocols function at this level.

You should verify the configuration files on your server for any wrong settings. Additionally, it is essential to look at the log files on the servers to get more detailed information about the issues.

• File Transfer Protocol (FTP)
• Simple Mail Transfer Protocol (SMTP)
• Secure Shell (SSH)
• Internet Message Access Protocol (IMAP)
• Domain Name Service (DNS)
• Hypertext Transfer Protocol (HTTP).

Conclusion

Troubleshooting network issues in Linux can be a daunting task, but by applying the principles of the OSI model, you can systematically diagnose and resolve problems with greater efficiency.

Starting from the bottom layer and working your way up, we've explored various tools and techniques tailored to each level of the OSI model.

Beginning with the physical layer, we inspected hardware components and used tools like ifconfig and ip link show to verify connectivity. Moving up to the data link layer, we focused on MAC addresses and used utilities like ping and Wireshark for testing. At the network layer, we delved into IP addressing and routing, employing commands such as ip route and ping to diagnose issues.

Transitioning to the transport layer, we addressed TCP and UDP related problems, utilizing commands like netstat and telnet to check for open ports and establish connections. Further up the stack, we discussed the importance of session management and encryption at the session and presentation layers respectively.

Finally, at the application layer, we examined specific protocols like FTP, SMTP, SSH, and HTTP, emphasizing the significance of configuration files and log analysis in resolving issues.

During a recent project update meeting, my team talked about how we were going to use serialization to send data back and forth from this application.

An engineer who was looking to get more into software projects told me that they were unfamiliar with the term.

It's easy to miss essential processes like these that don’t come up until you dive into more extensive projects. This was the case for this person, as it was for me at one point.

So I wanted to write about it. I helped my colleague learn about serialization that day, and you’re going to learn about it today.

What is Serialization?

Serialization is the process in which one service takes in a data structure, such as a dictionary in Python, wraps it up, and transmits it to another service for reading. That’s the simple definition.

Imagine that I need to send a message to someone. So I write down the text on an already assembled puzzle. I take apart the pieces, add some instructions on how to reassemble the puzzle, and send it along.

The message recipient then gets the pieces of the puzzle, puts them all back together, and now they have my message.

Basic serialization flow of events

The technical definition is a bit more fun. To wit, serialization is the process of converting a data object into a byte stream, and saving the state of the object to be stored on a disk or transmitted across a network. This cuts down the storage size needed and makes it easier to transfer information over a network.

Serialization Process

Marshaling and Serialization - what are the differences?

The process of marshaling might come to mind. Marshaling is the process of transforming the memory representation of an object into a suitable form for transmission.

Although marshaling and serialization are loosely synonymous, there is a crucial difference. For example, when creating a Golang program to read JSON data into a Golang data structure, you might use marshaling to translate JSON key values into Golang key values.

The difference is that marshaling might be used to translate data. In contrast, serialization sends or stores data in a byte stream and reassembles it in its original form. Both do serialization, but there is a difference in intent in these two processes.

You can see this struct I created for interacting with Twitter data below as an example of marshaling in action. In Golang, you can give hints called tags, easily converting this object into JSON data using Golang's built-in marshaling service.

Golang Struct using JSON tags

What is Endianness?

I’d also like to touch on the subject of endianness lightly. Endianness is a term used to describe the order of bytes in memory.

You can think of memory as a block where bites of data are stored. For serialization to work, the byte stream needs to transfer data types regardless of the changing endianness from one system to another.

You can see the little and big-endian differences below. It is essential that the endianness matches from one system to another or be converted somehow, as not all systems order their bits the same way.

Little and big-endian Courtesy of https://pvs-studio.com/en/blog/lessons/0019/

Use Cases for Serialization

Our use case takes full advantage of these features. We plan to take in some information from the hardware we’re scanning, package up that information into a byte stream, and send it along with the network to another service that will reconstruct the data.

The process of reversing the serialization process and reconstructing the data back into its original form is called deserialization.

There are other use cases for this. For example, REST APIs or messaging protocols such as AMQP can use serialization to compress and send data.

AMQP is a messaging protocol where you send messages to an AMQP broker, and the receiving service is “listening” to this broker for a message. Backend engineers might know this well, as this is often used for sending data back and forth within distributed systems.

Many programming languages include the ability to spin up some serialization easily. So it is a language-agnostic topic.

Serialization Example

Let’s give a quick example. This code uses the library kombu to send messages via AMQP. We’re using this to send messages from one software package to another over a network. This code is for a service sending a message to an AMQP broker:

def send_message(self, payload, sender_serializer):
...
    try:
        producer.publish(
            {'payload': message},
            ...
            serializer = 'json',
            ...
        )
        return

Take note of the publish method. We are passing in the serialization method as an argument so that the library knows how to serialize the data we are passing in.

The data message is converted into a stream of bytes, which, if you look at it, just looks like a long string of letters and numbers, and we send the message.

The corresponding service will use the same serialization method to reconstruct the data in its original state. This is a significant feature as we are creating a suite of tools that need to be able to send messages to each other for them to work.

Serialization Data Formats

I use JSON for serialization whenever the task at hand calls for it. However, you can also use a few others.

JSON has a lot of overhead, but the human readability makes it ideal for me. You can also use Protobufs, YAML, or XML. Those are just some of the data object formats you can use.

Conclusion

I’m glad I got this out of my system. I got to stop thinking about this, and, hopefully, someone learned something from it.

Serialization becomes essential when you’re putting together your communication pipeline. It’s good to know about this topic to feel confident approaching whatever tool you are using with the proper background knowledge.

-George

TCP, which stands for Transmission Control Protocol, and UDP, or User Datagram Protocol, are part of the internet protocol suite. TCP and UDP are different methods to send information across the internet.

But even knowing what they stand for, it's hard to know which protocol you should use, or why you would use one over the other.

In this article, we'll go over computer networking basics, the differences between TCP and UDP, when each is used, and more.

Computer Networking Basics

Before diving into how TCP and UDP work, it's helpful to know the basics about how the internet works.

Generally speaking, the internet is a network of connecting devices. Each device, whether it's your smartphone or a server, communicate through the internet protocol suite.

The internet protocol suite is a collection of different protocols, or methods, for devices to communicate with each other. Both TCP and UDP are major protocols within the internet protocol suite:

Source

Each device that's connected to the internet has a unique IP address. And whenever two devices communicate over the internet, they're likely using either TCP or UDP to do so.

Here's a brief comparison between the two:

Source

For an even higher-level overview of how the internet works, check out this five minute video:

What is TCP?

TCP, or Transmission Control Protocol, is the most common networking protocol online. TCP is extremely reliable, and is used for everything from surfing the web (HTTP), sending emails (SMTP), and transferring files (FTP).

TCP is used in situations where it's necessary that all data being sent by one device is received by another completely intact.

For example, when you visit a website, TCP is used to guarantee that everything from the text, images, and code needed to render the page arrives. Without TCP, images or text could be missing, or arrive in the incorrect order, breaking the page.

TCP is a connection-oriented protocol, meaning that it establishes a connection between two devices before transferring data, and maintains that connection throughout the transfer process.

To establish a connection between two devices, TCP uses a method called a three-way handshake:

Source

For example, to read this article on your device, your device first sent a message to the freeCodeCamp News server called an SYN (Synchronize Sequence Number).

Then the freeCodeCamp News server sends back an acknowledgement message called a SYN-ACK.

When your device receives the SYN-ACK from the server, it sends an ACK acknowledgment message back, which establishes the connection.

Once a TCP connection is established between two devices, the protocol guarantees that all data is transmitted.

Going back to the example of your device and freeCodeCamp News, once the three-way handshake is complete, the News server can start sending all the data your device's web browser needs to render this article.

All devices break up data into small packets before sending them over the internet. Those packets then need to be reassembled on the other end.

So when the freeCodeCamp News server sends the HTML, CSS, images, and other code for this article, it breaks everything into small packets of data before sending them to your device. Your device then reassembles those packets into the files and images it needs to render this article.

TCP ensures that these packets all arrive to your device. If any packets are lost along the way, TCP makes it easy for your device to let the server know it's missing data, and for the server to resend those packets.

Once your device receives all the data it needs to render the article, TCP automatically terminates the connection between the two devices with a method similar to the three-way handshake, this time using FIN and ACK packets.

What is UDP?

UDP, or User Datagram Protocol, is another one of the major protocols that make up the internet protocol suite. UDP is less reliable than TCP, but is much simpler.

UDP is used for situations where some data loss is acceptable, like live video/audio, or where speed is a critical factor like online gaming.

While UDP is similar to TCP in that it's used to send and receive data online, there are a couple of key differences.

First, UDP is a connectionless protocol, meaning that it does not establish a connection beforehand like TCP does with its three-way handshake.

Next, UDP doesn't guarantee that all data is successfully transferred. With UDP, data is sent to any device that happens to be listening, but it doesn't care if some of it is lost along the way. This is one of the reasons why UDP is also known as the "fire-and-forget" protocol.

A good way to think about these differences is that TCP is like a conversation between two people. Person A asks person B to talk. Person B says sure, that's fine. Person A agrees and they both start speaking.

UDP is more like a protester outside with a megaphone. Everyone who is paying attention to the protester should hear most of what they're saying. But there's no guarantee that everyone in the area will hear what the protester is saying, or that they're even listening.

UDP vs TCP — Source

Which is Faster – TCP or UDP?

In general, UDP is the faster protocol.

UDP is much simpler, and doesn't try to establish a connection between devices before sending data, or verify that all the data even arrived. It simply sends out data to any device that requests it, and keeps doing that until the other device disconnects or there is no more data left to send.

Think drinking from a hose rather than sipping from a bottle. You'll quench your thirst either way, but will probably end up with a damp shirt using the former method.

_Not a hose, but still pretty accurate. Also imagine that the TCP bottle keeps asking if you received water while you drink from it. Source_

But being faster doesn't mean that UDP is the better protocol overall. It just means that it's better in certain situations.

As mentioned earlier, TCP is necessary in situations where it's vital that all data packets are sent in order, and that all packets arrive. The web just wouldn't function without TCP.

And while TCP is slower because of the way it establishes connections, and due to the checks for missing packets, it can still be blazing fast. Because they're on the web and use HTTP, sites like YouTube or Netflix all use TCP to send data to your devices.

TCP also allows for buffering, so your browser can request and load more data as you watch, allowing for smooth playback and for you to skip ahead to other parts of the video.

UDP is the better choice for live video and audio or online games where speed is more important than potential data loss.

When you make a call over Google Meet or Zoom, your video and audio are being transmitted over UDP. If some packets are lost along the way, it'll just appear as a bit of lag or clipped video/audio.

If you play video games, you might think that the way TCP ensures all data packets arrive at the other device would make it the ideal choice. But in reality, all the checking and resending data that TCP does just adds latency.

Game developers have found other clever ways to ensure that player input and state are as accurate as possible. If you're interested in reading more about why UDP is preferred for online gaming, check out this article.

FIN

I hope this article helped you understand some of the nuances between TCP and UDP. And if someone asks which is faster, tell 'em what you read here: "UDP is faster, but..."

And if you like what you read, let me know over on Twitter.

Most commonly ping refers to the act of sending a packet or signal to another device and listening for a response. This is usually done to measure the speed of the network, or to determine the status of a computer or server.

The term ping was coined in the early 1980s by Mike Muuss, who chose the word because of its similarities to the way sonar works and sounds. Muuss also developed the first ping program to diagnose network issues.

A version of the ping program should be installed by default on all modern operating systems.

To use ping, just open the command line for your system and type ping followed by an IP address or a host name, then press enter. To exit the ping utility, just press CTRL + C.

For example, if you run ping www.freecodecamp.org, you'll see something like this:

Each line shows information for each probe or ping packet that's sent out. Some of the post important information is at the end of the line. This shows the time in milliseconds it took to send out a packet and receive it back from the other server or computer.

And when you stop the ping utility, you'll see a summary of all the ping attempts:

Ping can also refer to the latency or response time of a network itself. This definition is usually used in online gaming, where ping represents the response time between the gaming client (a console or PC), and the game's servers.

In this context, high ping (>= 150ms) means that there will be a large delay between a player's action in the game and the game's response. And low ping (around 20-50ms) means that the time between a player's action and the game's response is minimal.

Usually when you visit a website, your browser sends a request to a server. The server then sends you back all the files (HTML, CSS, images, and so on) for your browser to render the website.

But if the server has a problem, you wouldn't be able to get all those files, and can't visit the site.

In a peer-to-peer network, a bunch of computers connect to each other and all act as small servers. If one computer in a peer-to-peer network goes offline, the other computers can fill in for it.

Several years ago, Spotify was one of the largest peer-to-peer networks. Back then, they leveraged P2P networking as a way to provide their service using their customer's bandwith. Now Spotify uses central servers that they control.

P2P can also be applied to other things like payments. In this context, it means that the payment gets sent directly to the other person. But the payment might still pass through a company's central servers, unlike a P2P network.

For example, if you send $20 to your friend with a P2P payments app like Venmo, they will receive the money instantly. Your friend can then transfer the money from Venmo to their bank account, or send it to someone else.

But if you use a traditional money wiring service, you will need your friend's bank information to send the money directly to their account. Also, the transfer might have fees, and take several days.

Let me break that down a bit. When I say, within a single home or office, I mean all the devices that are connected through either a physical or wireless connection to a network router. That router might be a WiFi access point or the modem your internet service provider (ISP) gave you.

By organizing I mean each device is given an identifying address, and its access to the internet beyond your local network is defined.

And by protecting I mean that, generally, traffic requests directed at your devices from external networks will be scanned and filtered to help prevent unauthorized and potentially dangerous access.

Based in part on content from my Linux in Action book, I'll try to explain how all that works.

IPv4 addressing

Here's how that might look. The Router in this image has a public IP address of 183.23.100.34 to which all incoming and outgoing traffic is associated.

At the same time, the router acts as a Dynamic Host Configuration Protocol (DHCP) server, assigning private IP addresses to all the PCs, laptops, smartphones, and servers in the house. The devices will use those addresses whenever they talk to each other.

A typical local area network (LAN) topography

Notice how all the local devices are described as using something called "NAT IP addresses." NAT stands for Network Address Translation, and it's the method used for organizing devices within a private LAN.

But why? What's wrong with giving all devices the same kind of public IP address the router has?

In the beginning, there was IPv4. IPv4 addresses are 32-bit numbers made up of four 8-bit octets separated by dots. Here's what that might look like:

192.168.1.10

Subnet notation

Because it’s critically important to make sure systems know what kind of subnet a network address is on, we need a standard notation that can accurately communicate which octets are part of the network and which are available for devices.

There are two commonly used standards: Classless Inter-Domain Routing (CIDR) notation and netmask.

Using CIDR, one network might be represented as 192.168.1.0/24. The /24 tells you that the first three octets (8×3=24) make up the network portion, leaving only the fourth octet for device addresses. The second network (or subnet), in CIDR, would be described as 192.168.2.0/24.

These same two networks could also be described through a netmask of 255.255.255.0. That means all 8 bits of each of the first three octets are used by the network, but none of the fourth.

Understanding private networks

In theory, the IPv4 protocol allows for around four billion unique addresses, ranging from 1.0.0.0 to 255.255.255.255.

But even if all four billion of those addresses were practically available, it still wouldn't come close to covering each of the billions of cell phones, billions of laptop and desktop computers, and billions more network-connected cars, appliances, and Internet of Things devices that are already out there. To say nothing of the billions more that're coming soon.

So network engineers set aside three ranges of IPv4 addresses to be used exclusively in private networks. Devices using any address from those ranges will not be directly reachable from the public internet and will not be able to access internet resources. These are the three ranges:

Between 10.0.0.0 and 10.255.255.255
Between 172.16.0.0 and 172.31.255.255
Between 192.168.0.0 and 192.168.255.255

Remember what the "T" in NAT stood for? It was "Translation." What that means is that a NAT-enabled router will take the private IP addresses used in traffic requests between the LAN and the internet and translate them to the router's own public address. The router, true to its name, will then route those requests to their appropriate destinations.

This simple redesign of network addressing saved many billions of addresses for use with devices - like cell phones - that weren't part of a private network. All those laptops, PCs, and so on running in all those homes and offices would conveniently (and seamlessly) share their routers' public IPs.

Problem solved? Well, not quite. You see, even with all that efficient use of addresses, there still won't be enough for the explosion of public-facing devices coming online. To manage that problem, more network engineers came up with the IPv6 protocol. Here's what an IPv6 address might look like:

2002:0df6:0001:004b:0100:6c2e:0370:7234

That looks nasty, doesn't it? And it looks like it's a much bigger number than that wimpy IPv4 example from before.

Yup and yup. I've gotten pretty good at remembering some kinds of IPv4 addresses, but I've never even tried to "download" one of these monsters.

For one thing, it's hexadecimal, meaning it uses the numbers between 0 and 9 and the first six letters of the alphabet (a-f)! Besides that, there are eight octets rather than four, and the address is 128-bit rather than 32-bit.

All of which means that, once the protocol is fully implemented, we won't be at risk of running out of addresses for a very, very long time (meaning: forever). And what that means is that, from the perspective of address allocation, there's no longer any need for private NAT networks.

Although, for security considerations, you'll still want to give your devices some protection within your LAN.

There's much more administration goodness in the form of books, courses, and articles available at my bootstrap-it.com.

Get to know the Open Systems Interconnection model

Overview

Throughout this series, we will be tackling the basics such as:
(Part 1) How does DNS work?
(Part 2) Network Stack, OSI Model [You are here!]
(Part 3) HTTP Methods and Formats
(Part 4) Client Identification
(Part 5) Basic/Digest Authentication
(Part 6) HTTPS working with SSL/TLS

OSI Model

The Open Systems Interconnection (OSI) Model is a standardized model for telecommunication in computer systems. It does not regard the underlying technology, but instead the layers involved in communication. Let us explore the different layers within the OSI Model:

Typical 5-layered OSI Model

1. Application Layer

This layer allows applications to communicate over the network once the connection has been established, such as from the Web Browser (Application) to the Server. Examples of protocols in this layer include HTTP and TELNET.

HyperText Transfer Protocol (HTTP)

A set of rules for transferring files over the Internet. For example, when you enter the URL into the browser, the browser sends an HTTP request for the webpage. The host would then return the webpage, together with all the elements that are within, such as images, text, videos, styling fonts, etc.

2. Transport Layer

This layer is responsible for the host-to-host communication of messages. Examples of protocols in this layer include TCP and UDP.

Transmission Control Protocol (TCP)

The most common connection-oriented protocol. It defines how to establish and maintain a network conversation. It is responsible for establishing a connection (called a socket) between the client and the host in a 3-way handshake.

The user requesting the data will send a SYN data packet to the server, requesting synchronization. The server will then respond with a SYN-ACK to the user, indicating that it has acknowledged the data packet, and would like to connect as well. The connection is hence established when the user sends the last ACK to the server.

TCP is the most common due to its elegance, in which it is able to offer the following:

Connection-oriented communication
Establish a handshake protocol between end-points to ensure connection before data is exchanged, and transmit as a data stream (data packets).

Reliability
Using checksums, it ensures that the data packets transmitted and received are the same. If there are missing/corrupted packets, it will request for re-transmission of the data packets by sending a NACK message to the sender.

Order
The data packets are numbered and transmitted. As such, TCP will ensure that the received packets are re-ordered before delivering the application.

Flow Control
The rate of data transmission is regulated to improve efficiency while preventing buffer overruns/underruns, where data is sent faster than the receiver is able to process it, and vice versa.
The mechanics behind it are explained below in the TCP Slow Start section.

Multiplexing
Basically, it is able to send over multiple streams of information concurrently over the same socket. These are done through different ports on the socket. We will discuss the differences between Multiplexing and Pipelining further along in the article.

User Datagram Protocol (UDP)

While similar to TCP, it is a connection-less protocol. It is the complete opposite of TCP, making it unreliable and unordered. Dropped packets will not be re-transmitted, causing gaps in the data.

However, that makes it best for time-sensitive applications, such as voice calls over the internet (VoIP). This is because it does not require the 3-way handshake before transmitting, making it fast. In addition, dropped data packets are not a problem in VoIP, as the human ear is very good at handling the short gaps that are typical with dropped packets.

3. Network Layer

This layer is responsible for providing data routing paths for network connections. Basically, it moves data packets across the network with the most logical path.

Internet Protocol (IP)

Defines the structure of the data packets, as well as labeling it with the source and destination information.

The source and destination information are in the form of IP Addresses, in which can be in the form 104.16.121.127(IPv4), or 2001:db8:0:1234:0:567:8:1(IPv6).

4. Link/ Physical Layer

This layer is the root of the OSI model, where information is transmitted either in the Local Area Network (LAN) for the Link Layer, and a physical signal such as electrical, mechanical medium in the form of code words or symbols in the Physical Layer.

Visualising Routes

Using tracert google.com, the route can be traced from the client-side (your computer) to the host (google.com).

From above, you can see the route starting from my device 192.168.1.254 to the router 10.243.128.1, before passing through the Internet Service Provider (ISP) located in Portugal, and so forth.

Complementary Layers

TCP/IP Model

TCP will request for re-transmission of dropped data packets, and re-order them

IP is only responsible for the structure of the data packet. As such, it will not make amends if the data packet is corrupted, or dropped. This is where TCP comes into play, numbering the data packets before sending over to the client. At the client’s side, TCP will request for re-transmission of lost/corrupted packets, and then rearrange the packets of data.

HTTP/TCP Model

As we have mentioned earlier, HTTP can now make requests via the connection made by TCP Handshake. But how do they complement each other?

HTTP Persistent Connections
This would allow multiple HTTP request/response on a single TCP connection, as opposed to opening a new connection upon every request/response.

Sample response for Persistent Connection

This is done through the HTTP Header, where Connection: Keep-Alive. On default, the connection will only close upon another response where Connection: Close is sent after 30 seconds of idle.

TCP Slow Start
As mentioned before, TCP supports flow control. This is done through TCP Slow Start, which is a form of prevention for network congestion.

The sender has a congestion window (CWND) and the receiver has a receiver window (RWND). If the data is larger than the congestion/receiver window, there would be a buffer under/overrun respectively.

To prevent that, the sender will begin by sending a data packet with a small congestion window (CWND = 1), to slowly probe the receiver for its receiver window.

The receiver will respond with an acknowledge, prompting the sender to double the data packets each time until no acknowledge is received. At this point, the optimum number of data packets has been discovered, allowing other congestion control algorithms to keep the connection at this speed.

Working Together
Hence, TCP Slow Start is able to figure out the optimum number of data packets to send before the connection is closed. This will allow the amount of data sent from the host to the client to be optimized without the risk of buffer overrun (data is sent faster than it can be received).

Other HTTP Features

HTTP Pipelining

This feature in version HTTP/1.1 allows multiple requests to be sent at once on the same socket, without waiting for a response. However, it has been replaced by TCP Multiplexing in the newer version of HTTP/2.

The key difference is that although both allow for multiple requests all at once on the same socket, Pipelining would still require responses to be sent in order. It means that if the items requested are in the order (A, B, C), the client would not receive item C if item B has not been delivered properly.

In Multiplexing, the order does not matter. This would allow quicker delivery time.

These methods are best used for the idempotent method, which are methods that respond independently of the number of times requested — for example, requesting a web page multiple times will respond to the same web page.

Parallel Connections

Ever opened a webpage and seen multiple components of the webpage (video bar, thumbnails, buttons) load simultaneously?

_Multiple components loading simultaneously | Photo courtesy of [Cloudflare Mobile SDK](https://www.cloudflare.com/products/mobile-sdk/" rel="noopener" target="blank" title=")

This is made possible with Parallel Connections, where there is more than one TCP Connection established at the same time, allowing these components to load concurrently instead of one after another.

However, although it might seem to load faster, it might be held back by the client’s limited bandwidth. If all Parallel Connections are competing for the limited bandwidth, each component will load proportionately slower, resulting in zero advantage in total loading speed.

Conclusion

With the OSI Model, we can easily understand the big picture of networks, and how they interact with each other from hardware to software.

In general, it is a great teaching tool as well as a reference for troubleshooting. The model is also useful for design, as it investigates the functions at every layer, forcing one to ponder over the design layer by layer.

What I have gone through so far is the OSI 5-Layer Model, whereas there is also the OSI 7-Layer Model which also deals with Identification, Authentication and Data Encryption.

This is Part 2 of the HTTP Introductions Series. You can read the first article about the importance of DNS Servers in Part 1. Let’s explore the structure of HTTP Requests next in Part 3!

Hi! I’m Cher Don, currently pursuing a Major in Data Science. I’m the CTO of Paralegal Bot, and you can find my website below. Thanks for reading!

Piqued;
_Quality Content We offer the best content for difficult to grasp concepts. We've been there, and felt the same you do…_www.piqued.co

A network visualization tutorial with Gephi and Sigma.js

Here’s a preview of what we’ll be making today: the programming languages influence graph. Check out the link to explore the “design influence” relationships between over 250 programming languages past and present!

Your turn!

In today’s hyper-connected world, networks are an ubiquitous aspect of modern life.

Take the start of my day so far — I used London’s transport network to travel into town. Then I went into a branch of my favourite coffee shop and used my Chromebook to connect to their Wi-Fi network. Next, I logged in to the various social networking sites I frequent.

It’s no secret that some of the most influential companies of the last few decades owe their success to the power of networks.

Facebook, Twitter, Instagram, LinkedIn and other social media platforms rely on the small-world properties of social networks. This lets them connect their users with each other (and advertisers) effectively.

Google owes much of its current success to their early dominance of the search engine market — enabled in part through their ability to return relevant results with the help of their Page Rank network algorithm.

Amazon’s efficient distribution network allows them to offer same-day delivery in some major cities.

Networks are also super-important in fields such as Artificial Intelligence and Machine Learning. Neural networks are a very active field of research. Many feature detection algorithms, essential in Computer Vision, rely heavily on using networks to model different parts of images.

A wide range of scientific phenomena can also be understood in terms of network models. This includes quantum mechanics, biochemical pathways, and ecological and socio-economic systems.

Given their undeniable importance, then, how can we better understand networks and their properties?

The mathematical study of networks is known as “graph theory”, and is one of the more accessible branches of mathematics. This article aims to provide an introduction, assuming little prior knowledge or experience.

We’ll be using Python 3.x and some awesome open-source software called Gephi to put together a network visualization of how a range of programming languages past and present are linked by influence.

But first…

What exactly is a network?

The examples described above give us some clues. Transport networks are made up of destinations connected by routes. Social networks are made up of individuals, connected through their relationships to one another. Google’s search engine algorithms evaluate the “rank” of different webpages by looking at which pages link out to others.

More generally, a network is any system that can be described in terms of nodes and edges, or in colloquial terms, “dots and lines”.

An example of nodes (languages) connected by edges (design influence)

Some systems are readily abstracted in this manner. Social networks are perhaps the most obvious example. Computer filesystems are another — folders and files are linked by their “parent” and “child” relationships.

But the real power of networks comes from the fact that many, many systems can be abstracted and modelled in network terms, even if at first it isn’t obvious how.

Representing networks

We need to go a little beyond pen-and-paper sketches to analyze and describe networks mathematically. How can we turn pictures of dots and lines into numbers we can crunch?

One solution is to draw up an adjacency matrix to represent our network.

Matrices are one of those concepts that might sound a little intimidating if you’re not familiar with them, but fear not. Think of them as grids of numbers which can be used to perform many calculations all at once. Here’s an example below:

      Python Java Scala C#
Python     0    1     0  0
Java       0    0     0  1
Scala      0    1     0  0
C#         0    1     0  0

In this matrix, the intersection of each row and column is either 0 or 1, depending on whether or not the respective languages are linked. You can check this against the illustration above!

For most purposes, the adjacency matrix is a good way of representing a network mathematically. From a computational perspective, however, it can sometimes be a bit cumbersome.

For instance, with even a relatively modest number of nodes (say 1000), there will be a much larger number of elements in the matrix (e.g., 1000² = 1,000,000).

Many real-world systems yield sparse networks. In these networks, most nodes only connect to a small proportion of all the others.

If we represented a 1000-node sparse network in computer memory as an adjacency matrix, we’d have 1,000,000 bytes of data stored in RAM. Most will be zeros. There’s got to be a more efficient way of going about this.

An alternative approach is to work with edge lists instead. These are exactly what they say they are. They are simply a list of which node pairs link to each other.

For example, the programming languages network above can be represented as follows:

Java, Python
Java, Scala
Java, C#
C#, Java

For larger networks, this is a much more computationally efficient means of representing them. It is of course possible to generate an adjacency matrix from an edge list (and vice versa). It’s not like we have to pick one or the other.

Another means of representing networks are adjacency lists. This lists every node followed by the nodes it links to. For example:

Java: Python, Scala, C#
C#: Java

Collecting data, making connections

Any network model and visualisation will only be as good as the data used to construct it. This means, as well as ensuring the data is both accurate and complete, we also need to justify a means of inferring edges between nodes.

In many respects, this is the critical step. Any subsequent analysis and inferences made about the network depend on being able to justify the “linkage criterion”.

For example, in social network analysis, you might link people based upon whether they follow one another on social media. In molecular biology, you might link genes based upon their co-expression.

Often, the method used to link nodes will allow for weights to be assigned to the edges, giving a measure of “strength”.

For instance, in the context of online retail, you could link products based upon how often they are purchased together. Products that are frequently bought together would be linked by a higher weighted edge than products which are only sometimes bought together. Products that are bought together no more often than would be expected by chance wouldn’t be linked at all.

As you might imagine, the methods for linking nodes to one another can be as sophisticated as you like.

However, for this tutorial we’ll be using a simpler means of connecting programming languages. We’re gonna rely on the accuracy of Wikipedia.

For our purposes, this should be fine. Wikipedia’s success is testament that it must be doing something right. The open-source, collaborative method by which articles are written should ensure some degree of objectivity.

Also, its relatively consistent page structure makes it a convenient playground for trying out web-scraping techniques.

Another bonus is the extensive, well-documented Wikipedia API, which makes information retrieval easier still. Let’s get started.

Step 1 — Installing Gephi

Gephi is available on Linux, Mac and Windows. You can download it here.

For this project, I was using Lubuntu. If you’re on Ubuntu/Debian, then you can follow the steps below to get Gephi up and running. Otherwise, the installation process will likely be much the same as whatever you’re familiar with.

Download the latest version (at the time of writing this was v.0.9.1) of Gephi for your system. When it’s ready, you’ll need to extract the files.

cd Downloads
tar -xvzf gephi-0.9.1-linux.tar.gz
cd gephi-0.9.1/bin./gephi

You may need to check your version of the Java JRE. Gephi requires a recent version. On my relatively fresh install of Lubuntu, I simply installed the default-jre, and everything worked from there.

apt install default-jre
./gephi

There’s one more step before you’re ready to get underway. In order to export the graph to the Web, you can use the Sigma.js plugin for Gephi.

From Gephi’s menu bar, choose the “Tools” option, and select “Plugins”.

Click on the “Available Plugins” tab and select “SigmaExporter” (I also installed JSON Exporter, because it’s another useful plugin to have around).

Hit the “Install” button and you’ll be walked through the process. You’ll need to restart Gephi once you’re done.

Step 2 — Writing the Python script

This tutorial will use Python 3.x, plus a few modules to make life easier. Using the pip module installer, run the following command:

pip3 install wikipedia

Now, in a new directory, create a file called something like script.py, and open it up in your favourite code editor/IDE. Below is an outline of the main logic:

1. First, you’ll need a list of programming languages to include.
2. Next, go through that list and retrieve the HTML of the relevant Wikipedia article.
3. From this, extract a list of programming languages that each language has influenced. This will be a rough-and-ready linkage criterion.
4. While you’re at it, it’d be nice to grab some metadata about each language.
5. Finally, you’ll want to write all the data you’ve collected to a .csv file

The full script can be found in this gist.

Import some modules

In script.py, start by importing a few modules which will make things easier:

import csv
import wikipedia
import urllib.request
from bs4 import BeautifulSoup as BS
import re

OK — begin by making a list of nodes to include. This is where the Wikipedia module comes in handy. It makes accessing the Wikipedia API super-easy.

Add the following code:

pageTitle = "List of programming languages"
nodes = list(wikipedia.page(pageTitle).links)
print(nodes)

If you save and run this script, you’ll see it prints out all the links from the “List of programming languages” Wikipedia article. Nice!

However, it’s always sensible to manually inspect any automatically collected data. A quick glance will reveal that, as well as many actual programming languages, the script has also picked up a few extra links.

For example, you might see “List of markup languages”, “Comparison of programming languages” and others in there.

Although Gephi lets you remove nodes you’d rather not include, it wouldn’t hurt to “clean” the data before proceeding. If anything, this will save time later on.

removeList = [
    "List of",
    "Lists of",
    "Timeline",
    "Comparison of",
    "History of",
    "Esoteric programming language"
    ]

nodes = [i for i in nodes if not any(r in i for r in removeList)]

These lines define a list of substrings to be removed from the data. The script then goes through the data, removing any elements that contain any of the unwanted substrings.

In Python, this requires just one line of code!

Some helper functions

Now you can start scraping Wikipedia to build up an edge list (and collect any metadata). To make this easier, first define a few functions.

Grabbing HTML

The first function uses the BeautifulSoup module to get hold of the HTML for each language’s Wikipedia page.

base = "https://en.wikipedia.org/wiki/"

def getSoup(n):
    try:
        with urllib.request.urlopen(base+n) as response:
            soup = BS(response.read(),'html.parser')
            table = soup.find_all("table",class_="infobox vevent")[0]                return table
     except:
         pass

This function uses the urllib.request module to get hold of the HTML for the page at “https://en.wikipedia.org/wiki/” + “programming language”.

This is then passed to BeautifulSoup, which reads and parses the HTML into an object we can use to search for information.

Next, use the find_all() method to extract the HTML element you’re interested in.

Here, this will be the summary table at the top of each programming language article. How can these be identified?

The easiest way is to visit one of the programming language pages. Here, you can simply use the browser’s Developer Tools to inspect the elements of interest.

The summary table has the HTML tag <table> and the CSS classes "infobox" and "vevent", so you can use these to identify the table in the HTML.

Specify this with the arguments:

• "table" and
• class_="infobox vevent"

find_all() returns a list of all elements that match the criteria. In order to actually specify the element you’re interested in, add the index [0]. If the function is successful, it returns the table object. Otherwise, it returns None.

The data we’re after is in this HTML element!

With any automated data collection procedure, it’s always important to handle exceptions thoroughly. If not, then in the best case scenario the script crashes and you’ll need to start over.

In the worst case, you’ll end up with a data set riddled with inconsistencies and errors. This will make it a nightmare to work with down the line.

Retrieve metadata

The next function uses the table object to look for some metadata. Here, it searches the table for the year the language first appeared.

def getYear(t):
    try:
        t = t.get_text()
        year = t[t.find("appear"):t.find("appear")+30]
        year = re.match(r'.*([1-3][0-9]{3})',year).group(1)
        return int(year)
    except:
        return "Could not determine"

This short function takes the table object as its argument, and uses BeautifulSoup’s get_text() function to produce a string.

The next step is to create a substring called year. This takes the 30 characters after the first appearance of the word "appear". This string should contain the year the language first appeared.

In order to extract just the year, use a regular expression (courtesy of the re module) to match any characters that begin with a digit between 1 and 3, and are followed by three digits.

re.match(r'.*([1-3][0-9]{3})',year)

If this is successful, the function returns year as an integer. Otherwise, it returns a sad-looking “Could not determine”. You might wish to scrape further metadata — such as paradigm, designer or typing discipline.

Collecting links

One more function for you — this time, you’ll feed in the table object for a given language, and hopefully receive out a list of other programming languages.

def getLinks(t):
    try:
        table_rows = t.find_all("tr")
        for i in range(0,len(table_rows)-1):
            try:
                if table_rows[i].get_text() == "\nInfluenced\n":
                    out = []
                    for j in table_rows[i+1].find_all("a"):
                        try:
                            out.append(j['title'])
                        except:
                            continue
                    return out
            except:
                continue
        return
    except:
        return

Woah, look at all that nesting… What is actually going on here then?

This function makes use of the fact that the table objects have a consistent structure. The information in the table is stored in rows (the relevant HTML tag is <tr> ). One of these rows will contain thetext"\nInfluenced\n"`. The first part of the function finds which row this is.

Once this row has been found, you can then be pretty sure the next row contains links to each of the programming languages influenced by the current one. Find these links using find_all("a") — where the argument "a" corresponds to the HTML tag <a>.

For each link j, append its ["title"] attribute to a list called out. The reason to be interested in the ["title"] attribute is because this will match exactly the language’s name as stored in nodes.

For example, Java is stored in nodes as "Java (programming language)", so you need to use this exact name throughout the data set.

If successful, getLinks() returns a list of programming languages. The rest of the function deals with exception handling, in case something should go wrong at any stage.

Collecting the data

At last, you’re almost ready to sit back and let the script do its thing. It will collect the data and store it in two list objects.

edgeList = [["Source,Target"]]
meta = [["Id","Year"]]

Now write a loop that will apply the functions defined earlier to every item in nodes, and store the outputs in edgeList and meta.

for n in nodes:
    try:
        temp = getSoup(n)
    except:
        continue
    try:
        influenced = getLinks(temp)
        for link in influenced:
            if link in nodes:
                edgeList.append([n+","+link])
                print([n+","+link])
    except:
        continue
    year = getYear(temp)
    meta.append([n,year])

This function takes each language in nodes and attempts to retrieve the summary table from its Wikipedia page.

Then, it retrieves all the languages the table lists as having been influenced by the language in question.

For each language that also appears in the nodes list, append an element to edgeList in the form of ["source,target"]. In this way, you’ll build up an edge list to feed into Gephi.

For debugging purposes, print each element added to edgeList — just to be sure everything’s working as it should. If you were being extra thorough, you could add print statements to the except clauses, too.

Next, get the language’s name and year, and append these to the meta list.

Writing to CSV

Once the loop has run, the final step is to write the contents of edgeList and meta to comma separated value (CSV) files. This is easily done with the csv module imported earlier.

with open("edge_list.csv","w") as f: 
    wr = csv.writer(f)
    for e in edgeList:
        wr.writerow(e)

with open("metadata.csv","w") as f2:
    wr = csv.writer(f2)
    for m in meta:
        wr.writerow(m)

Done! Save the script, and from the terminal run:

$ python3 script.py

You should see the script printing out each source-target pair as it builds up the edge list. Make sure your internet connection is steady, and sit back while the script does its magic.

Step 3 — Graph building with Gephi

Hopefully you got Gephi installed and running earlier. Now you can create a new project and use the data you gathered to build a directed graph. This will show how different programming languages have influenced one another!

Start by creating a new project in Gephi, and switch to the “Data Laboratory” view. This provides a spreadsheet-like interface for handling data in Gephi. The first thing to do is import the edge list.

• Click “Import spreadsheet”.
• Choose the edge_list.csv file generated by the Python script. Ensure that Gephi knows to use the commas as the separator.
• Choose “Edge List” from the List type.
• Click “Next” and check that you are importing both Source and Target columns as strings.

This should update the Data Lab with a list of nodes. Now, import the metadata.csv file. This time, make sure to choose “Nodes list” from the List type.

Switch over to the “Preview” tab, and see how the network looks.

Ah… It’s just a little bit… monochrome. And messy. Like a plate of spaghetti. Let’s fix this.

Making it pretty

There are all sorts of ways you can work on the presentation, and here’s where a little bit of creative freedom comes in. With network visualisations, there are essentially three things to take into consideration:

1. Positioning There are several algorithms which can generate layout patterns for a network. A popular choice is the Fruchterman-Reingold algorithm, which is available in Gephi.
2. Sizing The size of nodes in a graph can be used to represent some interesting property. Often, this is a centrality measure. There are many ways of measuring centrality, but they all reflect the “importance” of a given node, in terms of how well-connected it is to the rest of the network.
3. Coloring It is also possible to use color to show some property of a node. Often, color is used to indicate community structure. This is broadly defined as a “group of nodes which are more connected with each other than with the rest of the graph”. In a social network, this can reveal friendship, family or professional groups. There are several algorithms which can detect community structure. Gephi comes with the Louvain method built-in.

To make these changes, you will need to calculate some statistics. Switch to the “Overview” window. Here you will see a panel on the right. It should contain a “Statistics” tab. Open this, and you will see a range of options.

Gephi comes with many inbuilt statistical capabilities. For each of them, clicking “Run” will generate a report that will reveal insights about the network.

Some useful ones to know include:

• Average degree The average language is connected to about four others. The report also shows a degree distribution graph. This reveals that most languages have very few connections, while a small proportion have many. This suggests that this is a scale-free network. Much research has been done on scale-free networks, and the processes that generate them.
• Diameter This network has a diameter of 12 — meaning this is the “widest” number of connections between any two languages. The average path length is just under four. This means that, on average, any two languages are separated by four edges. These figures give a measure of the “size” of the network.
• Modularity This is a score that shows how “compartmentalized” the network is. Here, the modularity score is about 0.53. This is relatively high, suggesting there are distinct modules within this network. Again, this indicates something interesting about the underlying system. Languages tend to fall into distinct “influence groups”.

Anyhow, to modify the appearance of the network, head over to the left panel.

In the “Layout” tab, you can select which layout algorithm to use. Hit “Run” and watch the graph shift about in real-time! See which layout algorithm you think works best.

Above the Layout tab is the “Appearance” tab. Here, you can play with different settings for the node and edge colors, sizes and labels. These can be configured based upon attributes (including the stats you get Gephi to calculate).

As a suggestion, you could:

• Color the nodes by their Modularity attribute. This colors them according to their community membership.
• Size the nodes by their Degree. Better connected nodes will appear larger than less connected ones.

However, you should experiment and come up with a layout you like best.

Once you’re happy with the appearance of your graph, it is time to move on to the final step — exporting to Web!

Step 4 — Sigma.js

Already you have built a network visualisation that can be explored in Gephi. You could choose to take a screenshot, or save the graph in SVG, PDF or PNG format.

However, if you installed the Sigma.js plugin earlier, then why not export the graph to HTML? This will create an interactive visualisation that you can host online, or upload to GitHub and share with others.

To do this, select “Export > Sigma.js template…” from Gephi’s menu bar.

Fill in the details as required. Make sure to choose which directory you export the project to. You can change the title, legend, description, hover behavior and many other details. When you’re ready, click “OK”.

Now, if you navigate to the directory you exported the project to, you will see a folder containing all the files generated by Sigma.js.

Open up index.html in your favorite browser. Ta-da! There’s your network! If you know a little CSS and JavaScript, you can dive into the various generated files to tweak the output as you wish.

And that concludes this tutorial!

Summary

• Many systems can be modelled and visualised as networks. Graph theory is a branch of math that provides tools to help understand network structures and properties.
• You used Python to scrape data from Wikipedia to build a programming languages influence graph. The linkage criterion was whether a given language was listed as an influence on another’s design.
• Gephi and Sigma.js are open-source tools that allow you to analyze and visualize networks. They allow you to export the network in image, PDF or Web formats.

Thanks for reading — I look forward to any comments or questions you might have! For a fantastic resource to learn more about graph theory, see Albert-László Barabási’s interactive online book.

The full code for this tutorial can be found here.