After 15 years in law and other roles, I’d experienced a number of jobs, countries, companies, and career paths. None of them compared to the joy and thrill I get from coding.

The downside? Picking up new coding skills can be confusing, frustrating, and time consuming. And it’s easy to forget some of the minute but important details.

So I wrote this handbook. It's intended to get you started on coding Solidity ASAP. It follows the Pareto Principle (aka the 80/20 rule) by focusing on the 20% of the info that will cover 80% of your needs.

I started assembling these concepts when I was learning Solidity, as part of my role at Chainlink Labs. I applied many of the self-learning techniques that I learned when transitioning to coder at the age of 38.

This is the resource I wish I had. It is designed to give beginner and intermediate developers solid mental models to stack as you get deeper into the language (mental models massively accelerate effective learning).

I will keep this handbook updated, but I could really use your help! Just tweet me @ZubinPratap to let me know if I need to update this handbook.

I’d like to acknowledge my amazing colleagues Kevin Ryu, Andrej Rakic, Patrick Collins, and Richard Gottleber for the invaluable guidance and input on this handbook.

Table of Contents

1. Who is this handbook for?

2. Essential prior knowledge

3. What is Solidity?

4. What is a smart contract?

5. How to declare variables and functions in Solidity?

6. Variable scope in Smart Contracts

7. How visibility specifiers work

8. What are constructors?

9. Interfaces and abstract contracts

10. Smart contract example #2

11. What is contract state?

12. State mutability keywords (modifiers)

13. Data locations – storage, memory, and stack

14. How typing works

15. Solidity data types

16. How to declare and initialize arrays in Solidity

17. What are function modifiers?

18. Error handling in Solidity - require, assert, revert

19. Inheritance in Solidity

20. Inheritance with constructor parameters

21. Type conversion and type casting in Solidity

22. How to work with floating point numbers in Solidity

23. Hashing, ABI encoding and decoding

24. How to call contracts and use the fallback function

25. How to send and receive Ether

26. Solidity libraries

27. Events and logs in Solidity

28. Time logic in Solidity

29. Conclusion and further resources

Who Is This Handbook For?

This handbook is for people who are interested in exploring the vision behind “Web3”, and who wish to acquire in-demand skills that are essential to realizing that vision.

Do not memorize it! Read it and then use it as a “desktop reference” companion. As you learn any new language, you will find that concepts, idioms, and usage can get a bit confusing or your memory fades with time. That’s ok! That’s what this handbook is designed to help you with.

Over time I may add some more advanced topics to this, or create a separate tutorial. But for now this handbook will get you most of the outcomes you require to build your first several Solidity dApps.

This handbook assumes you have at least a few months of experience with programming. By programming I mean at the very least you’ve written in JavaScript or Python or some compiled language (since HTML and CSS aren't actually "programming" languages, it won't be enough to know only them).

The only other requirements are that you are curious, committed, and not putting arbitrary deadlines on yourself.

As long as you have a laptop, and a browser with an internet connection, you’ll be able to run the Solidity code. You can use Remix in your browser to write the code in this handbook. No other IDE required!

Essential Prior Knowledge

I have also assumed that you know the basics of blockchain technology, and in particular you understand the basics of Ethereum and what smart contracts are (hint: they’re programs that run on blockchains and hence provide special trust-minimized benefits!).

You are unlikely to need them to understand this handbook. But practically speaking, having a browser wallet like Metamask and understanding the difference between Ethereum contract accounts and externally owned accounts will help you get the most out of this handbook.

What is Solidity?

Now, let’s start with understanding what Solidity is. Solidity is an object-oriented programming language influenced by C++, JavaScript and Python.

Solidity is designed to be compiled (converted from human readable to machine readable code) into bytecode that runs on the Ethereum Virtual Machine (EVM). This is the runtime environment for Solidity code, just like your browser is a runtime environment for JavaScript code.

So, you write Smart Contract code in Solidity, and the compiler converts it into bytecode. Then that bytecode gets deployed and stored on Ethereum (and other EVM-compatible blockchains).

You can get a basic introduction to the EVM and bytecode in this video I made.

What is a Smart Contract?

Here is a simple smart contract that works out of the box. It may not look useful, but you’re going to understand a lot of Solidity from just this!

Read it along with each comment to get a sense of what’s going on, and then move on to some key learnings.

We will get to some of the details like what public and view mean shortly.

For now, take seven key learnings from the above example:

1. The first comment is a machine readable line ( // SPDX-License-Identifier: MIT) that specifies the licensing that covers the code.

SPDX license identifiers are strongly recommended, though your code will compile without it. Read more here. Also, you can add a comment or “comment out” (suppress) any line by prefixing it with two forward slashes "// ".

2. The pragma directive must be the first line of code in any Solidity file. Pragma is a directive that tells the compiler which compiler version it should use to convert the human-readable Solidity code to machine readable bytecode.

Solidity is a new language and is frequently updated, so different versions of the compiler produce different results when compiling code. Some older solidity files will throw errors or warnings when compiled with a newer compiler version.

In larger projects, when you use tools like Hardhat, you may need to specify multiple compiler versions because imported solidity files or libraries that you depend on were written for older versions of solidity. Read more on Solidity’s pragma directive here.

3. The pragma directive follows Semantic Versioning (SemVer) - a system where each of the numbers signifies the type and extent of changes contained in that version. If you want a hands-on explanation of SemVer is check out this tutorial - it is very useful to understand and it’s used widely in development (especially web dev) these days.

4. Semicolons are essential in Solidity. The compiler will fail if even a single one is missing. Remix will alert you!

5. The keyword contract tells the compiler that you’re declaring a Smart Contract. If you’re familiar with Object Oriented Programming, then you can think of Contracts as being like Classes.

If you’re not familiar with OOP then think of contracts as being objects that hold data - both variables and functions. You can combine smart contracts to give your blockchain app the functionality it needs.

6. Functions are executable units of code that encapsulate single ideas, specific functionality, tasks, and so on. In general we want functions to do one thing at a time.

Functions are most often seen inside smart contracts, though they can be declared in the file outside the smart contract’s block of code. Functions may take 0 or more arguments and they may return 0 or more values. Inputs and outputs are statically typed, which is a concept you will learn about later in this handbook.

7. In the above example the variable qtyCups is called a “state variable”. It holds the contract’s state - which is the technical term for data that the program needs to keep track of to operate.

Unlike other programs, smart contract applications keep their state even when the program is not running. The data is stored in the blockchain, along with the application, which means that each node in the blockchain network maintains and synchronizes a local copy of the data and smart contracts on the blockchain.

State variables are like database “storage” in a traditional application, but since blockchains need to synchronize state across all nodes in the network, using storage can be quite expensive! More on that later.

How to Declare Variables and Functions in Solidity

Let’s break down that HotFudgeSauce Smart Contract so we understand more about each little piece.

The basic structure/syntax to defining things in Solidity is similar to other statically typed languages. We give functions and variables a name.

But in typed languages we also need to specify the type of the data that is created, passed as input or returned as output. You can jump down to the Typing Data section in this handbook if you need to understand what typed data is.

Below, we see what declaring a “State Variable” looks like. We also see what declaring a function looks like.

The first snippet declares a State Variable (I’ll explain what this is soon, I promise) called qtyCups. This can only store values that are of type uint which means unsigned integers. “Integer” refers to all whole numbers below zero (negative) and above zero (positive).

Since these numbers have a + or - sign attached, they’re called signed integers. An unsigned integer is therefore always a positive integer (including zero).

In the second snippet, we see a familiar structure when we declare functions too. Most importantly, we see that the functions have to specify a data type for the value that the function returns.

In this example, since get() returns the value of the storage variable we just created, we can see that the returned value must be a uint.

public is a visibility specifier. More on that later. view is a State-Mutability modifier. More on that below too!

It’s worth noting here that state variables can also be of other types - constant and immutable. They look like this:

Constants and immutable variables have their values assigned once, and only once. They cannot be given another value after their first value is assigned.

So if we made the qtyCups state variable either constant or immutable, we would not be able to call the increment() or decrement() functions on it anymore (in fact, the code wouldn’t compile!).

Constants must have their values hardcoded in the code itself, whereas immutable variables can have their values set once, generally by assignment in the constructor function (we’ll talk about constructor functions very soon, I promise). You can read more in the docs here.

Variable Scope in Smart Contracts

There are three scopes of variables that Smart Contracts have access to:

1. State Variables: store permanent data in the smart contract (referred to as persistent state) by recording the values on the blockchain.

2. Local Variables: these are “transient” pieces of data that hold information for short periods of time while running computations. These values are not stored permanently on the blockchain.

3. Global variables: these variables and functions are “injected” into your code by Solidity, and made available without the need to specifically create or import them from anywhere. These provide information about the blockchain environment the code is running on and also include utility functions for general use in the program.

You can tell the difference between the scopes as follows:

1. state variables are generally found inside the smart contract but outside of a function.

2. local variables are found inside functions and cannot be accessed from outside that function’s scope.

3. Global variables aren’t declared by you - they are “magically” available for you to use.

Here is our HotFudgeSauce example, slightly modified to show the different types of variables. We give qtyCups a starting value and we dispense cups of fudge sauce to everyone except me (because I’m on a diet).

How Visibility Specifiers Work

The use of the word “visibility” is a bit confusing because on a public blockchain, pretty much everything is “visible” because transparency is a key feature. But visibility, in this context, means the ability for one piece of code to be seen and accessed by another piece of code.

Visibility specifies the extent to which a variable, function, or contract can be accessed from outside the region of code where it was defined. The scope of visibility can be adjusted depending on which portions of the software system need to access it.

If you’re a JavaScript or NodeJS developer, you’re already familiar with visibility – any time you export an object you’re making it visible outside the file where it is declared.

Types of Visibility

In Solidity there are 4 different types of visibility: public, external, internal and private.

Public functions and variables can be accessed inside the contract, outside it, from other smart contracts, and from external accounts (the kind that sit in your Metamask wallet) - pretty much from anywhere. It’s the broadest, most permissive visibility level.

When a storage variable is given public visibility, Solidity automatically creates an implicit getter function for that variable’s value.

So in our HotFudgeSauce smart contract, we don’t really need to have the get() method, because Solidity will implicitly provide us identical functionality, just by giving qtyCups a public visibility modifier.

Private functions and variables are only accessible within the smart contract that declares them. But they cannot be accessed outside of the Smart Contract that encloses them. private is the most restrictive of the four visibility specifiers.

Internal visibility is similar to private visibility, in that internal functions and variables can only be accessed from within the contract that declares them. But functions and variables marked internal can also be accessed from derived contracts (that is, child contracts that inherit from the declaring contract) but not from outside the contract. We will talk about inheritance (and derived/child contracts) later on.

internal is the default visibility for storage variables.

The 4 Solidity Visibility specifiers and where they can be accessed from

The external visibility specifier does not apply to variables - only functions can be specified as external.

External functions cannot be called from inside the declaring contract or contracts that inherit from the declaring contract. Thus, they can only be called from outside the enclosing contract.

And that’s how they’re different from public functions – public functions can also be called from inside the contract that declare them, whereas an external function cannot.

What are Constructors?

A constructor is a special type of function. In Solidity, it is optional and is executed once only on contract creation.

In the following example we have an explicit constructor and it accepts some data as a parameter. This constructor parameter must be injected by you into your smart contract at the time you create it.

Solidity constructor function with input parameter

To understand when the constructor function gets called, it’s helpful to remember that a Smart Contract is created over a few phases:

• it is compiled down to bytecode (you can read more about bytecode here). This phase is called “compile time”.

• it gets created (constructed) - this is when the constructor kicks into action. This can be referred to as “construction time”.

• Bytecode then gets deployed to the blockchain. This is “deployment”.

• The deployed smart contract bytecode gets run (executed) on the blockchain. This can be considered “runtime”.

In Solidity, unlike other languages, the program (smart contract) is deployed only after the constructor has done its work of creating the smart contract.

Interestingly, in Solidity, the finally deployed bytecode does not include the constructor code. This is because in Solidity, the constructor code is part of the creation code (construction time) and not part of the runtime code. It is used up when creating the smart contract, and since it’s only ever called once it is not needed past this phase, and is excluded in the finally deployed bytecode.

So in our example, the constructor creates (constructs) one instance of the Person smart contract. Our constructor expects us to pass a string value which is called _name to it.

When the smart contract is being constructed, that value of _name will get stored in the state variable called name (this is often how we pass configuration and other data into the smart contract). Then when the contract is actually deployed, the state variable name will hold whatever string value we passed into our constructor.

Understanding the Why

You might wonder why we bother with injecting values into the constructor. Why not just write them into the contract?

This is because we want contracts to be configurable or “parameterized”. Rather than hard code values, we want the flexibility and reusability that comes with injecting data as and when we need.

In our example, let’s say that _name referred to the name of a given Ethereum network on which the contract is going to be deployed (like Rinkeby, Goerli, Kovan, Mainnet, and so on).

How could we give that information to our smart contract? Putting all those values in it would be wasteful. It would also mean we need to add extra code to work out which blockchain the contract is running on. Then we'd have to pick the right network name from a hard-coded list which we store in the contract, which takes up gas on deployment.

Instead, we can just inject it into the constructor, at the time we are deploying the smart contract to the relevant blockchain network. This is how we write one contract that can work with any number of parameter values.

Another common use case is when your smart contract inherits from another smart contract and you need to pass values to the parent smart contract when your contract is being created. But inheritance is something we will discuss later.

I mentioned that constructors are optional. In HotFudgeSauce, we didn’t write an explicit constructor function. But Solidity supports implicit constructor functions. So if we don’t include a constructor function in our smart contract, Solidity will assume a default constructor which looks like constructor() {}.

If you evaluate this in your head you’ll see it does nothing and that’s why it can be excluded (made implicit) and the compiler will use the default constructor.

Interfaces and Abstract Contracts

An interface in solidity is an essential concept to understand. Smart Contracts on Ethereum are publicly viewable and therefore you can interact with them via their functions (to the extent the Visibility Specifiers allow you to do so!).

This is what makes smart contracts “composable” and why so many Defi protocols are referred to as “money Legos” - you can write smart contracts that interact with other smart contracts that interact with other smart contracts and so on…you get the idea.

So when you want your smart contract A to interact with another smart contract B, you need B’s interface. An interface gives you an index or menu of the various functions available for you to call on a given Smart Contract.

An important feature of Interfaces is that they must not have any implementation (code logic) for any of the functions defined. Interfaces are just a collection of function names and their expected arguments and return types. They’re not unique to Solidity.

So an interface for our HotFudgeSauce Smart Contract would look like this (note that by convention, solidity interfaces are named by prefixing the smart contract’s name with an “I”:

That’s it! Since HotFudgeSauce had only three functions, the interface shows only those.

But there is an important and subtle point here: an interface does not need to include all the functions available to call in a smart contract. An interface can be shortened to include the function definitions for the functions that you intend to call!

So if you only wanted to use the decrement() method on HotFudgeSauce then you could absolutely remove get() and increment() from your interface - but you would not be able to call those two functions from your contract.

So what’s actually going on? Well, interfaces just give your smart contract a way of knowing what functions can be called in your target smart contract, what parameters those functions accept (and their data type), and what type of return data you can expect. In Solidity, that’s all you need to interact with another smart contract.

In some situations, you can have an abstract contract which is similar to but different from an interface.

An abstract contract is declared using the abstract keyword and is one where one or more of its functions are declared but not implemented. This is another way of saying that at least one function is declared but not implemented.

Flipping that around, an abstract contract can have implementations of its functions (unlike interfaces which can have zero implemented functions), but as long as at least one function is unimplemented, the contract must be marked as abstract:

You may (legitimately) wonder what the point of this is. Well, abstract contracts cannot be instantiated (created) directly. They can only be used by other contracts that inherit from them.

So abstract contracts are often used as a template or a “base contract” from which other smart contracts can “inherit” so that the inheriting smart contracts are forced to implement certain functions declared by the abstract (parent) contract. This enforces a defined structure across related contracts which is often a useful design pattern.

This inheritance stuff will become a little clearer when we discuss Inheritance later. For now, just remember that you can declare an abstract smart contract that does not implement all its functions - but if you do, you cannot instantiate it, and future smart contracts that inherit it must do the work of implementing those unimplemented functions.

Some of the important differences between interfaces and abstract contracts are that:

• Interfaces can have zero implementations, whereas abstract contracts can have any number of implementations as long as at least one function is “abstract” (that is, not implemented).

• All functions in an interface must be marked as “external” because they can only be called by other contracts that implement that interface.

• Interfaces cannot have constructors, whereas abstract contracts may.

• Interfaces cannot have state variables where abstract contracts may.

Smart Contract Example #2

For the next few Solidity concepts we’ll use the below smart contract. This is partly because this example contains a smart contract that is actually used in the real world. I've also chosen it because I have a clear bias to Chainlink Labs since I work there (😆) and it’s awesome. But it’s also where I learned a lot of Solidity, and it’s always better to learn with real-world examples.

So start by reading the code and the comments below. You’ve already learned 99% of what you need to understand the contract below, provided you read it carefully. Then move on to key learnings from this contract.

This smart contract gets the latest USD price of 1 Eth, from a live Chainlink price feed oracle (see the oracle on etherscan). The example uses the Goerli network so you don’t end up spending real money on the Ethereum mainnet.

Now to the 6 essential Solidity concepts you need to absorb:

1. Right after the pragma statement we have an import statement. This imports existing code into our smart contract.

This is super cool because this is how we reuse and benefit from code that others have written. You can check out the code that is imported on this GitHub link.

In effect, when we compile our smart contract, this imported code gets pulled in and compiled into bytecode along with it. We will see why we need it in a second…

2. Previously you saw that single-line comments were marked with //. Now you're learning about multiline comments. They may span one or more lines and use /* and */ to start and end the comments.

3. We declare a variable called priceFeed and it has a type AggregatorV3Interface. But where does this strange type come from? From our imported code in the import statement - we get to use the AggregatorV3Interface type because Chainlink defined it.

If you looked at that Github link, you’d see that the type defines an interface (we just finished talking about interfaces). So priceFeed is a reference to some object that is of type AggregatorV3Interface.

4. Take a look at the constructor function. This one doesn’t accept parameters, but we could have just as easily passed the ETH/USD Price Feed’s oracle smart contract’s address 0xD4a33860578De61DBAbDc8BFdb98FD742fA7028e to it as a parameter of type address. Instead, we are hard-coding the address inside the constructor.

But we are also creating a reference to the Price Feed Aggregator smart contract (using the interface called AggregatorV3Interface).

Now we can call all the methods available on the AggregatorV3Interface because the priceFeed variable refers to that Smart Contract. In fact, we do that next… 5. Let's jump to the function getLatestPrice(). You’ll recognize its structure from our discussion in HotFudgeSauce, but it’s doing some interesting things.

Inside this getLatestPrice() function we call the latestRoundData() function which exists on the AggregatorV3Interface type. If you look at the source code of this method you’ll notice that this latestRoundData() function returns 5 different types of integers!

Calling methods on another smart contract from our smart contract

In our smart contract, we are commenting out all 4 values that we don’t need. So this means that Solidity functions can return multiple values (in this example we are returned 5 values), and we can pick and choose which ones we want.

Another way of consuming the results of calling latestRoundData() would be:( ,int price, , ,) = priceFeed.latestRoundData() where we ignore 4 out of 5 returned values by not giving them a variable name.

When we assign variable names to one or more values returned by a function, we call it “destructuring assignment” because we destructure the returned values (separate each one out) and assign them at the time of destructuring, like we do with price above.

Since you’ve learned about interfaces, I recommend you take a look at Chainlink Labs’ GitHub repo to examine the implemented latestRoundData() function in the Aggregator contract and how the AggregatorV3Interface provides the interface to interact with the Aggregator contract.

What is Contract State?

Before we proceed any further, it’s important to make sure that the terminology that we’re going to see a lot is comprehensible to you.

“State” in computer science has a well-defined meaning. While it can get very confusing, the crux of state is that it refers to all the information that is “remembered” by a program as it runs. This information can change, update, be removed, created and so on. And if you were to take a snapshot of it at various times, the information will be in different “states”.

So the state is just the current snapshot of the program, at a point in time during its execution - what values do its variables hold, what are they doing, what objects have been created or removed, and so on.

We have previously examined the three types of variables - State Variables, Local Variables, and Global variables. State variables, along with Global variables give us the state of the smart contract at any given point in time. Thus, the state of a smart contract is a description of:

1. what values its state variables hold,

2. what values the blockchain-related global variables have at that moment in time, and

3. the balance (if any) lying in the smart contract account.

State Mutability Keywords (Modifiers)

Now that we have discussed state, state variables, and functions, let’s understand the Solidity keywords that specify what we are allowed to do with state.

These keywords are referred to as modifiers. But not all of them permit you to modify state. In fact many of them expressly disallow modifications.

Here are Solidity modifiers you will see any real-world smart contract:

Note that in Solidity version 0.8.17, there were breaking changes that enabled the use of payable as a data type. Specifically we now allowed to convert the address data type to a payable address type by doing a type conversion that looks like payable(0xdCad3a6d3569DF655070DEd06cb7A1b2Ccd1D3AF).
What this does is makes a given ethereum address payable, after which we can send Eth to that address.

Note that this use of payable is a type conversion, and not the same as the function modifier, though the same keyword is used. We will cover the address type later, but you can read about this here. | | virtual | functions | This is a slightly more advanced topic and is covered in detail in the section on Inheritance. This modifier allows the function to be “overridden” in a child contract that inherits from it. In other words, a function with the keyword virtual can be “rewritten” with different internal logic in another contract that inherits from this one. | | override | functions | This is the flip side to the virtual modifier. When a child contract “rewrites” a function that was declared in a base contract (parent contract) from which it inherits, it marks that rewritten function with override to signal that its implementation overrides the one given in the parent contract. If a parent’s virtual function is not overridden by the child, the parent's implementation will apply to the child. | | indexed | events | We will cover events later in this handbook. They are small bundles of data “emitted” by a smart contract typically in response to noteworthy events happening. The indexed keyword indicates that one of the pieces of data contained in an event should be stored in the blockchain for retrieval and filtering later. This will make more sense once we cover Events and Logging later in this handbook. | | anonymous | events | The docs say “Does not store event signature as topic” which probably doesn’t mean a lot to you just yet. But the keyword does indicate that it’s making some part of the event “anonymous”. So this will make sense once we understand events and topics later in this handbook. |

Note that variables that are not storage variables ( i.e. local variables declared and used inside the scope of a given function) do not need state modifiers. This is because they’re not actually part of the smart contract’s state. They’re just part of the local state inside that function. By definition then, they’re modifiable and don’t need controls on their modifiability.

Data Locations – Storage, Memory, and Stack

On Ethereum and EVM-based chains, data inside the system can be placed and accessed in more than one “data location”.

Data locations are part of the fundamental design and architecture of the EVM. When you see the words “memory”, “storage” and “stack”, you should start thinking “data locations” - that is, where can data be stored (written) to and retrieved (read) from.

Data location has an impact on how the code executes at run time. But it also has very important impacts on how much gas gets used during deployment and running of the smart contract.

The use of gas requires a deeper understanding of the EVM and something called opcodes - we can park that discussion for now. While useful, it is not strictly necessary for you to understand data locations.

Though I’ve mentioned 3 data locations so far, there are 2 other ways in which data can be stored and accessed in Smart Contracts: “calldata”, and “code”. But these are not data locations in the EVM’s design. They’re just subsets of the 3 data locations.

Let’s start with storage. In the EVM’s design, data that needs to be stored permanently on the blockchain is placed in the relevant smart contract’s “storage” area. This includes any contract “state variables”.

Once a contract is deployed and has its specific address, it also gets its own storage area, which you can think of as a key-value store (like a hash table) where both the keys and the values are 256 bit (32 byte) data “words”. And “words” has a specific meaning in computer architecture.

Because storage persists data on the blockchain permanently, all data needs to be synchronized across all the nodes in the network, which is why nodes have to achieve consensus on data state. This consensus makes storage expensive to use.

You’ve already seen examples of storage variables (aka contract state variables) but here is an example taken from the Chainlink Verifiable Random Number Consumer smart contract

Storage data location. Putting data in the contract's storage layout.

When the above contract is created and deployed, whatever address that is passed into the contract’s constructor becomes permanently stored in the smart contract’s storage, and is accessible using the variable vrfCoodinator. Since this state variable is marked as immutable, it cannot be changed after this.

To refresh your memory from the previous section on keywords, where we last discussed immutable and constant variables, these values are not put in storage. They become part of the code itself when the contract is constructed, so these values don’t consume as much gas as storage variables.

Now let’s move to memory. This is temporary storage where you can read and write data needed during the running of the smart contract. This data is erased once the functions that use the data are done executing.

The memory location space is like a temporary notepad, and a new one is made available in the smart contract each time a function is triggered. That notepad is thrown away after the execution completes.

When understanding the difference between storage and memory, you can think of storage as a kind of hard disk in the traditional computing world, in the sense that it has “persistent” storage of data. But memory is closer to RAM in traditional computing.

The stack is the data area where most of the EVM’s computations are performed. The EVM follows a stack based computation model and not a register based computation model, which means each operation to be carried out needs to be stored and accessed using a stack data structure.

The stack’s depth - that is the total number of items it can hold - is 1024, and each item in the stack can be 256 bits (32 bytes) long. This is the same as the size of each key and value in the storage data location.

You can read more about how the EVM controls access to the stack data storage area here.

Next, let's talk about calldata. I have assumed that you have a basic understanding about Ethereum smart contract messages and transactions. If you don’t, you should first read those links.

Messages and transactions are how smart contract functions are invoked, and they contain a variety of data necessary for the execution of those functions. This message data is stored in a read-only section of the memory called calldata, which holds things like the function name and parameters.

This is relevant for externally callable functions, as internal and private functions don’t use calldata. Only “incoming” function execution data and function parameters are stored in this location.

Remember, calldata is memory except that calldata is read-only. You cannot write data to it.

And finally, code is not a data location but instead refers to the smart contract's compiled bytecode that is deployed and stored permanently on the blockchain. This bytecode is stored in an immutable ROM (Read Only Memory), that is loaded with the bytecode of the smart contract to be executed.

Remember how we discussed the difference between immutable and constant variables in Solidity? Immutable values get assigned their value once (usually in the constructor) and constant variables have their values hard-coded into the smart contract code. Because they’re hardcoded, constant values are compiled literally and embedded directly into the smart contract’s bytecode, and stored in this code / ROM data location.

Like calldata, code is also read-only - if you understood the previous paragraph you’ll understand why!

How Typing Works

Typing is a very important concept in programming because it is how we give structure to data. From that structure we can run operations on the data in a safe, consistent and predictable way.

When a language has strict typing, it means that the language strictly defines what each piece of data’s type is, and a variable that has a type cannot be given another type.

In other words, in strictly typed languages:

But in JavaScript, which is not typed, b=a would totally work - this makes JavaScript “dynamically typed”.

Similarly, in statically typed languages you cannot pass an integer into a function that expects a string. But in JavaScript we can pass anything to a function and the program will still compile but it may throw an error when you execute the program.

For example take this function:

As you can imagine, this can produce some pretty hard-to-find bugs. The code compiles and can even execute without failing, though it produces unexpected results.

But a strongly typed language would never let you pass the string “2” because the function would insist on the types that it accepts.

Let’s take a look at how this function would be written in a strongly typed language like Go.

How typing works in syntax, using Golang for illustration purposes

Trying to pass a string (even if it represents a number) will prevent the program from even compiling (building). You will see an error like this:

./prog.go:13:19: cannot use "2" (untyped string constant) as int value in argument to add

Go build failed.

Try it out for yourself!

So types are important because data that seems the same to a human can be perceived very differently by a computer. This can cause some pretty weird bugs, errors, program crashes and even big security vulnerabilities.

Types also give developers the ability to create their own custom types, which can then be programmed with custom properties (attributes) and operations (behaviors).

Type systems exist so that humans can reason about the data by asking the question “what is this data’s type, and what should it be able to do?” and the machines can do exactly what is intended.

Here is another example of how data that looks the same to you and me may be interpreted in hugely different ways by a processor. Take the sequence of binary digits (that is the digits can only have a value of 0 or 1, which is the binary system that processors work with) 1100001010100011.

To a human, using the decimal system that looks like a very large number - perhaps 11 gazillion or something.

But to a computer that is binary, so it’s not 11 anything. The computer sees this as a sequence 16 bits (short for binary digits) and in binary this could mean the positive number (unsigned integer) 49,827 or the signed integer -15,709 or the UTF-8 representation of the British Pound symbol £ or something different!

A sequence of bits can be interpreted by a computer to have very different meanings (source)

So all this explanation is to say that types are important, and that types can be “inbuilt” into a language even if the language does not strictly enforce types, like JavaScript.

JavaScript already has inbuilt types like numbers, strings, booleans, objects, and arrays. But as we saw, JavaScript does not insist on types being stuck to the way a statically typed language like Go does.

Now back to Solidity. Solidity is very much a statically typed language. When you declare a variable you must also declare its type. Going further, Solidity will simply refuse to compile if you try to pass a string into a function that expects an integer.

In fact Solidity is very strict with types. For example, different types of integers may also fail compilation like the following example where the function add() expects an unsigned integer (positive) and will only add to that number thus always returning a positive integer. But the return type is specified as an int which means it could be positive or negative!

So even though the input and output are 256-bit integers, the fact that the function only receives unsigned integers makes the compiler complain that the unsigned integer type is not implicitly convertible to the signed integer type.

That’s pretty strict! The developer can force the conversion (called type casting) by rewriting the return statement as return int256(a + 10). But there are issues to consider with that sort of action, and that’s out of scope for what we’re talking about here.

For now, just remember that Solidity is statically typed, which means that the type of each variable must be expressly specified when declaring them in the code. You can combine types to form more complex, composite types. Next, we can discuss some of these inbuilt types.

Solidity Data Types

Types that are built into the language and come with it “out of the box” are often referred to as “primitives”. They’re intrinsic to the language. You can combine primitive types to form more complex data structures that become “custom” data types.

In JavaScript, for example, primitives are data that is not a JS object and has no methods or properties. There are 7 primitive data types in JavaScript: string, number, bigint, boolean, undefined, symbol, and null.

Solidity also has its own primitive data types. Interestingly, Solidity does not have “undefined” or “null”. Instead, when you declare a variable and its type, but do not assign a value to it, Solidity will assign a Default Value to that type. What exactly that default value is depends on the data type .

Many of Solidity’s primitive data types are variations of the same ‘base’ type. For example the int type itself has subtypes based on the number of binary digits that the integer type can hold.

If that confuses you a bit, don’t worry - it isn’t easy if you’re not familiar with bits and bytes, and I’ll cover integers a bit more shortly.

Before we explore Solidity types, there is another very important concept that you must understand - it is the source of many bugs, and “unexpected gotchas” in programming languages.

This is the difference between a value type and reference type, and the resulting distinction between data in programs being “passed by value” vs “passed by reference”. I’ll go into a quick summary below but you may also find it useful to watch this short video to strengthen your mental model before proceeding.

Pass by reference vs pass by value

At an operating system level, when a program is running, all data used by the program during its execution is stored in locations in the computer’s RAM (memory). When you declare a variable, some memory space is allocated to hold data about that variable and the value that is, or eventually will be, assigned to that variable.

There is also a piece of data that is often called a “pointer”. This pointer points to the memory location (an “address” in the computer’s RAM) where that variable and its value can be found. So the pointer effectively contains a reference to where the data can be found in the computer’s memory.

So when you pass data around in a program (for example when you assign a value to a new variable name, or when you pass inputs (parameters) into a function or method, the language’s compiler can achieve this in two ways. It can pass a pointer to the data’s location in the computer’s memory, or it can make a copy of the data itself, and pass the actual value.

The first approach is “pass by reference”. The second approach is “pass by value”.

Solidity’s data type primitives fall into two buckets - they’re either value types, or they’re reference types.

In other words, in Solidity, when you pass data around, the type of the data will decide whether you’re passing copies of the value or a reference to the value’s location in the computer’s memory.

Value Types and Reference Types in Solidity

In Solidity’s “value types”, integers are of two categories - uint is unsigned (positive integers only, so they have no plus or minus signs) and int is signed (could be positive or negative, and if you wrote them down, they’d have a plus or minus sign).

Integer types can also specify how many bits long they are - or how many bits are used to represent the integer.

An uint8 is an integer represented by 8 binary digits (bits) and can store up to 256 different values (2^8=256). Since uint is for unsigned (positive) integers, this means it can store values from 0 to 255 (not including 1 to 256).

However when you have signed integers, like an int8, then one of the bits is used up to represent whether it's a positive or negative number. That means we have only 7 bits left, and so we can only represent up to 2^7 (128) different values, including 0. So an int8 can represent anything from -127 to +127.

By extension, an int256 is 256 bits long and can store +/- (2^255) values.

The bit lengths are multiples of 8 (because 8 bits makes a byte) so you can have int8, int16, int24 etc all the way to 256 (32 bytes).

Addresses refer to the Ethereum account types - either a smart contract account or an externally owned account (aka “EOA”. Your Metamask wallet represents an EOA). So an address is also a type in Solidity.

The default value of an address (that is the value it will have if you declare a variable of type address but don’t assign it any value) is 0x0000000000000000000000000000000000000000 which is also the result of this expression: address(0).

Booleans represent either true or false values. Finally, we have fixed size byte arrays like bytes1, bytes2 … bytes32. These are arrays of fixed length that contain bytes. All these types of values are copied when they’re passed around in the code.

For “reference types”, we have arrays, which can have a fixed size specified when they’re declared, or dynamically sized arrays, which start off with a fixed size, but can be “resized” as the number of data elements in the array grows.

Bytes are a low-level data type that refer to the data that is encoded into binary format. All data is eventually reduced to binary form by the compiler so that the EVM (or, in traditional computing, the processor) can work with it.

Storing and working with bytes is often faster and more efficient compared to other data types that are more human readable.

You may be wondering why I’ve not referred to strings in either types of data in the picture above. That’s because in Solidity, strings are actually dynamically-sized arrays, and the arrays store a sequence of bytes (just binary numbers) that are encoded in the UTF-8 encoding format.

They’re not a primitive in Solidity. In JavaScript they’re referred to as primitives, but even in JavaScript strings are similar (but not the same as) to arrays and are a sequence of integer values, encoded in UTF-16.

It is often more efficient to store a string as a bytes type in a smart contract, as converting between strings and bytes is quite easy. It is therefore useful to store strings as bytes but return them in functions as strings. You can see an example below:

Other than Solidity strings, the bytes data type is a dynamically sized byte array. Also, unlike its fixed-size byte array cousin, it's a reference type. The bytes type in Solidity is a shorthand for “array of bytes” and can be written in the program as bytes or byte[].

If you’re confused by bytes and byte arrays…I sympathise.

The underlying gory details of strings and byte arrays are not too relevant for this handbook. The important point for now is that some data types are passed by reference and others are passed by copying their values.

Suffice it to say that Solidity strings and bytes without a size specified are reference types because they’re both dynamically sized arrays.

Finally, among Solidity’s primitives, we have structs and mappings. Sometimes these are referred to as “composite” data types because they’re composed from other primitives.

A struct will define a piece of data as having one or more properties or attributes, and specifying each property’s data type and name. Structs give you the ability to define your own custom type so that you can organize and collect pieces of data into one larger data type.

For example you could have struct that defines a Person as follows:

You can instantiate or initialize a Person struct in the following ways:

Mappings are similar to hashtables, dictionaries or JavaScript objects and maps, but with a little less functionality.

A mapping is also a key-value pair, and there are restrictions on the kinds of data types you can have as keys, which you can read about here. The data types associated with a mapping’s keys can be any of primitives, structs, and even other mappings.

Here is how mappings are declared, initialized, written to and read from - the below example is from the Chainlink Link Token Smart Contract source code.

Declaring and using the Mappings type in Solidity

If you try to access a value using a key that doesn’t exist in the mapping, it will return the default value of the type that is stored in the mapping.

In the above example, the type of all values in the balances mapping is uint256, which has a default value of 0. So if we called balanceOf() and passed in an address that does not have any LINK tokens issued to it, we’d get a value of 0 back.

This is reasonable in this example, but it can be a bit tricky when we want to find out whether or not a key exists in a mapping.

Currently there is no way to enumerate what keys exist in a mapping (that is, there is nothing equivalent to JavaScript’s Object.keys() method). Retrieving using a key will only return the default value associated with the data type, which does not clearly tell us whether or not the key actually exists.

There is an interesting “gotcha” with mappings. Unlike other languages where you can pass key-value data structures as an argument to function, Solidity does not support passing mappings as arguments to functions except where the functions visibility is marked as internal. So you could not write an externally or publicly callable function that would accept key-value pairs as an argument.

How to Declare and Initialize Arrays in Solidity

Solidity comes with two flavors of arrays, so it is useful to understand the different ways in which they can be declared and initialized.

The two main types of arrays in Solidity are the fixed-size array and the dynamic-sized array.

To refresh your memory, fixed-size arrays are passed by value (copied when passed around in the code) and dynamic-sized arrays are passed by reference (a pointer to the memory address is passed around in the code).

They’re also different in their syntax and their capacity (size), which then dictates when we would use one versus the other.

Here’s what a fixed-size array looks like when declared and initialized. It has a fixed capacity of 6 elements, and this cannot be changed once declared. The memory space for an array of 6 elements is allocated and cannot change.

A fixed-size array can also be declared by just declaring a variable and the size of the array and the type of its elements with the following syntax:

Contrast that with a dynamically-sized array that is declared and initialized as follows. Its capacity is unspecific and you can add elements using the push() method:

You can also declare and initialize the value of an array in the same line of code.

These arrays are available in storage. But what if you needed only temporary in-memory arrays inside a function? In that case there are two rules: only fixed size arrays are allowed, and you must use the new keyword.

Clearly, there are several ways to declare and initialize arrays. When you’re wanting to optimize for gas and computations you want to carefully consider which type of arrays are required, what their capacity is, and if they’re likely to grow without an upper bound.

This also influences and is influenced by the design of your code - whether you need arrays in storage or whether you need them only in memory.

What are Function Modifiers?

When writing functions, we often receive inputs that need some sort of validation, checking, or other logic run on those inputs before we go ahead with the rest of the “business” logic.

For example, if you’re writing in pure JavaScript, you may want to check that your function receives integers and not strings. If it’s on the backend you may want to check that the POST request contained the right authentication headers and secrets.

In Solidity, we can perform these sort of validation steps by declaring a function-like block of code called a modifier.

A modifier is a snippet of code that can run automatically before or after you run the main function (that is, the function that has the modifier applied to it).

Modifiers can be inherited from parent contracts too. It is generally used as a way to avoid repeating your code, by extracting common functionality and putting it in a modifier that can be reused throughout the codebase.

A modifier looks a lot like a function. The key thing to observe about a modifier is where the _ (underscore) shows up. That underscore is like a “placeholder” to indicate when the main function will run. It reads as if we inserted the main function where the underscore is currently.

So in the modifier snippet below, we run the conditional check to make sure that the message sender is the owner of the contract, and then we run the rest of the function that called this modifier. Note that a single modifier can be used by any number of functions.

How function modifiers are written, and the role of the underscore symbol

In this example, the require() statement runs before the underscore (changeOwner()) and that’s the correct way to ensure that only the current owner can change who owns the contract.

If you switched the modifier’s lines and the require() statement came second, then the code in changeOwner() would run first. Only after that would the require() statement run, and that would be a pretty unfortunate bug!

Modifiers can take inputs too - you’d just pass the type and name of the input into a modifier.

Modifiers are a great way to package up snippets of logic that can be reused in various smart contracts that together power your dApp. Reusing logic makes your code easier to read, maintain and reason about – hence the principle DRY (Don’t Repeat Yourself).

Error Handling in Solidity - Require, Assert, Revert

Error handling in Solidity can be achieved through a few different keywords and operations.

The EVM will revert all changes to the blockchain’s state when there is an error In other words, when an exception is thrown, and it’s not caught in a try-catch block, the exception will “bubble up” the stack of methods that were called, and be returned to the user. All changes made to the blockchain state in the current call (and its sub-calls) get reversed.

There are some exceptions, in low-level functions like delegatecall, send, call, and so on, where an error will return the boolean false back to the caller, rather than bubble up an error.

As a developer there are three approaches you can take to handle and throw errors. You can use require(), assert() or revert().

A require statement evaluates a boolean condition you specify, and if false, it will throw an error with no data, or with a string that you provide:

We use require() to validate inputs, validate return values, and check other conditions before we proceed with our code logic.

In this example, if the function’s caller does not send at least 1 ether, the function will revert and throw an error with a string message: “you must pay me at least 1 ether!”.

The error string that you want returned is the second argument to the require() function, but it is optional. Without it, your code will throw an error with no data - which is not terribly helpful.

The good thing about a require() is that it will return gas that has not been used, but gas that was used before the require() statement will be lost. That’s why we use require() as early as possible.

An assert() function is quite similar to require() except that it throws an error with type Panic(uint256) rather than Error(string).

An assert is also used in slightly different situations– where a different type of guarding is required.

Most often you use an assert to check an “invariant” piece of data. In software development, an invariant is one or more pieces of data whose value never changes while the program is executing.

In the above code example, the contract is a tiny contract, and is not designed to receive or store any ether. Its design is meant to ensure that it always has a contract balance of zero, which is the invariant we test for with an assert.

Assert() calls are also used in internal functions. They test that local state does not hold unexpected or impossible values, but which may have changed due to the contract state becoming “dirty”.

Just as require() does, an assert() will also revert all changes. Prior to Solidity’s v0.8, assert() used to use up all remaining gas, which was different from require().

In general, you’d likely use require() more than assert().

A third approach is to use a revert() call. This is generally used in the same situation as a require() but where your conditional logic is much more complex.

In addition, you can throw custom-defined errors when using revert(). Using custom errors can often be cheaper in terms of gas used, and are generally more informative from a code and error readability point of view.

Note how I improve the readability and traceability of my error by prefixing my custom error’s name with the Contract name, so we know which contract threw the error.

In the above example, we use revert once with a custom error that takes two specific arguments, and then we use revert another time with only a string error data. In either case, the blockchain state is reverted and unused gas will be returned to the caller.

Inheritance in Solidity

Inheritance is a powerful concept in Object Oriented Programming (OOP). We won't go into the details of what OOP is here. But the best way to reason about inheritance in programming is to think of it as a way by which pieces of code “inherit” data and functions from other pieces of code by importing and embedding them.

Inheritance in Solidity also allows a developer to access, use and modify the properties (data) and functions (behaviour) of contracts that are inherited from.

The contract that receives this inherited material is called the derived contract, child contract or the subclass. The contract whose material is made available to one or more derived contracts is called a parent contract.

Inheritance facilitates convenient and extensive code reuse – imagine a chain of application code that inherits from other code, and those in turn inherit from others and so on. Rather than typing out the entire hierarchy of inheritance, we can just use a a few key words to “extend” the functions and data captured by all the application code in the inheritance chain. That way child contract gets the benefit of all parent contracts in its hierarchy, like genes that get inherited down each generation.

Unlike some programming languages like Java, Solidity allows for multiple inheritance. Multiple inheritance refers to the ability of a derived contract to inherit data and methods from more than one parent contract. In other words, one child contract can have multiple parents.

You can spot a child contract and identify its parent contract by looking for the is keyword.

If you were to deploy only Contract B using the in-browser Remix IDE you’d note that Contract B has access to the getName() method even though it was not ever written as part of Contract B. When you call that function, it returns “A” , which is data that is implemented in Contract A, not contract B. Contract B has access to both storage variables A_NAME and B_NAME, and all functions in Contract A.

This is how inheritance works. This is how Contract B reuses code already written in Contract A, which could have been written by someone else.

Solidity lets developers change how a function in the parent contract is implemented in the derived contract. Modifying or replacing the functionality of inherited code is referred to as “overriding”. To understand it, let’s explore what happens when Contract B tries to implement its own getName() function.

Modify the code by adding a getName() to Contract B. Make sure the function name and signature is identical to what is in Contract A. A child contract’s implementation of logic in the getName() function can be totally different from how it’s done in the parent contract, as long as the function name and its signature are identical.

The compiler will give you two errors:

1. In Contract A, it will indicate that you are “trying to override non-virtual function” and prompt you by asking if you forgot to add the virtual keyword.

2. In Contract B, it will complain that the getName() function is missing the override specifier.

This means that your new getName in Contract B is attempting to override a function by the same name in the parent contract, but the parent’s function is not marked as virtual – which means that it cannot be overridden.

You could change Contract A’s function and add virtual as follows:

Adding the keyword virtual does not change how the function operates in Contract A. And it does not require that inheriting contracts must re-implement or override it. It simply means that this function may be overridden by any derived contracts if the developer chooses.

Adding virtual fixes the compiler’s complaint for Contract A, but not for Contract B. This is because getName in Contract B needs to also add the override keyword as follows:

We also add the pure keyword for Contract B’s getName() as this function does not change the state of the blockchain, and reads from a constant (constants, you’ll remember, are hardcoded into the bytecode at compile time and are not in the storage data location).

Keep in mind that you only need to override a function if the name and the signature are identical.

But what happens with functions that have identical names but different arguments? When this happens it’s not an override, but an overload. And there is no conflict because the methods have different arguments, and so there is enough information in their signatures to show the compiler that they're different.

For example, in contract B we could have another getName() function that takes an argument, which effectively gives the function a different “signature” compared to the parent Contract A’s getName() implementation. Overloaded functions do not need any special keywords:

Don’t worry about the abi.encodepacked() method call. I’ll explain that later when we talk about encoding and decoding. For now just understand that encodepacked() encodes the strings into bytes and then concatenates them, and returns a bytes array.

We discussed the relationship between Solidity strings and bytes in a previous section of this handbook (under Typing).

Also, since you’ve already learned about function modifiers, this is a good place to add that modifiers are also inheritable. Here’s how you’d do it:

You might wonder which version of a function will be called if a function by the same name and signature exists in a chain of inheritance.

For example, let's say there is a chain of inherited contracts like A → B → C → D → E and all of them have a getName() that overrides a getName() in the previous parent contract.

Which getName() gets called? The answer is the last one – the “most derived” implementation in the contract hierarchy.

State variables in child contracts cannot have the same name and type as their parent contracts.

For example, Contract B below will not compile because its state variable “shadows” that of the parent Contract A. But note how Contract C correctly handles this:

It’s important to note that by passing a new value to the variable author in Contract C’s constructor, we are effectively overriding the value in Contract A. And then calling the inherited method C.getAuthor() will return ‘Hemingway’ and not ‘Zubin’!

It is also worth noting that when a contract inherits from one or more parent contract, only one single (combined) contract is created on the blockchain. The compiler effectively compiles all the other contracts and their parent contracts and so on up the entire hierarchy all into a single compiled contract (which is referred to as a “flattened” contract).

Inheritance with Constructor Parameters

Some constructors specify input parameters and so they need you to pass arguments to them when instantiating the smart contract.

If that smart contract is a parent contract, then its derived contracts must also pass arguments to instantiate the parent contracts.

There are two ways to pass arguments to parent contracts - either in the statement that lists the parent contracts, or directly in the constructor functions for each parent contract. You can see both approaches below:

In Method 2 in the ChildTwo contract, you’ll note that the arguments passed to the parent contracts are first supplied to the child contract and then just passed up the inheritance chain.

This is not necessary, but is a very common pattern. The key point is that where parent contract constructor functions expect data to be passed to them, we need to provide them when we instantiate the child contract.

Type Conversion and Type Casting in Solidity

Sometimes we need to convert one data type to another. When we do so we need to be very careful when converting data and how the converted data is understood by the computer.

As we saw in our discussion on typed data, JavaScript can sometimes do strange things to data because it is dynamically typed. But that’s also why it’s useful to introduce the concept of type casting and type conversions generally.

Take the following JavaScript code:

There are two ways of converting the variable a into an integer. The first, called type casting, is done explicitly by the programmer and usually involves a constructor-like operator that uses ().

Now let's reset a to a string and do an implicit conversion, also known as a type conversion. This is implicitly done by the compiler when the program is executed.

In Solidity, type casting (explicit conversion) is permissible between some types, and would look like this:

In this example we converted an integer with a size of 256 bits (since 8 bits makes 1 byte, this is 32 bytes) into a bytes array of size 32.

Since both the integer value of 2022 and the bytes value are of length 32 bytes, there was no “loss” of information in the conversion.

But what would happen if you tried to convert 256 bits into 8 bits (1 byte)? Try running the following in your browser-based Remix IDE:

Why does the integer 2022 get converted to 230? That’s clearly an undesirable and unexpected change in the value. A bug, right?

The reason is that an unsigned integer of size 256 bits will hold 256 binary digits (either 0 or 1). So a holds the integer value ‘2022’ and that value, in bits, will have 256 digits, of which most will be 0, except for the last 11 digits which will be... (see for yourself by converting 2022 from decimal system to binary here).

The value of b on the other hand will have only 8 bits or digits, being 11100110. This binary number, when converted to decimal (you can use the same converter - just fill in the other box!) is 230. Not 2022.

Oopsie.

So what happened? When we dropped the integer’s size from 256 bits to 8 bits we ended up shaving the first three digits of data (11111100110) which totally changed the value in binary!

This, folks, is information loss.

So when you’re explicitly casting, the compiler will let you do it in some cases. But you could lose data, and the compiler will assume you know what you’re doing because you’re explicitly asking to do it. This can be the source of many bugs, so make sure you test your code properly for expected results and be careful when explicitly casting data to smaller sizes.

Casting up to larger sizes does not result in data lost. Since 2022 needs only 11 bits to be represented, you could declare the variable a as type uint16 and then up-cast it to a variable b of type uint256 without data loss.

The other kind of casting that is problematic is when you’re casting from unsigned integers to signed integers. Play around with the following example:

Note that a , being a signed integer of size 16 bits holds -2022 as a (negative integer) value. If we explicitly type cast it to an unsigned integer (only positive) values, the compiler will let us do it.

But if you run the code, you’ll see that b is not -2022 but 63,514! Because uint cannot hold information regarding the minus sign, it has lost that data, and the resulting binary gets converted to a massive decimal (base 10) number - clearly undesirable, and a bug.

If you go further, and un-comment the line that assigns the value of c, you’ll see the compiler complain with 'Explicit type conversion not allowed from "int16" to "uint256"'. Even though we are up-casting to a larger number of bits in uint256, because c is an unsigned integer, it cannot hold minus sign information.

So when explicitly casting, be sure to think through what the value will evaluate to after you’ve forced the compiler to change the type of the data. It is the source of many bugs and code errors.

There is more to Solidity type conversions and type casting and you can go deep into some of the nitty gritty in this article.

How to Work with Floating Point Numbers in Solidity

Solidity doesn’t handle decimal points. That may change in the future, but currently you cannot really work with fixed (floating) point numbers like 93.6. In fact typing int256 floating = 93.6; in your Remix IDE will throw an error like : Error: Type rational_const 468 / 5 is not implicitly convertible to expected type int256.

What’s going on here? 468 divided by 5 is 93.6, which seems a weird error, but that’s basically the compiler saying it cannot handle floating point numbers.

Follow the suggestions of the error, and declare the variable’s type to be fixed or ufixed16x1.

You’ll get an “UnimplementedFeatureError: Not yet implemented - FixedPointType” error.

So in Solidity, we get around this by converting the floating point number to a whole number (no decimal points) by multiplying by 10 raised to the exponent of the number of decimal places to the right of the decimal point.

In this case we multiply 93.6 by 10 to get 936 and we have to keep track of our factor (10) in a variable somewhere. If the number was 93.2355 we would multiply it by 10 to the power 4 as we need to shift the decimal 4 places to the right to make the number whole.

When working with ERC tokens we will note that the decimal places are often 10, 12, or 18.

For example, 1 Ether is 1*(10^18) wei, which is 1 followed by 18 zeros. If we wanted that expressed with a floating point, we would need to divide 1000000000000000000 by 10^18 (which will give us 1), but if it was 1500000000000000000 wei, then dividing by 10^18 will throw a compiler error in Solidity, because it cannot handle the return value of 1.5.

In scientific notation, 10^18 is also expressed as 1e18, where 1e represents 10 and the number after that represents the exponent that 1e is raised to.

So the following code will produce a compiler error: “Return argument type rational_const 3 / 2 is not implicitly convertible to expected type…int256”:

The result of the above division operation is 1.5, but that has a decimal point which Solidity does not currently support. Thus Solidity smart contracts return very big numbers, often up to 18 decimal places, which is more than JavaScript can handle. So you'll need to handle that appropriately in your front end using JavaScript libraries like Ethersjs that implement helper functions for the BigNumber type.

Hashing, ABI Encoding and Decoding

As you work more with Solidity, you’ll see some strange-sounding terms like hashing, ABI encoding, and ABI decoding.

While these can take some effort to wrap your head around, they’re quite fundamental to working with cryptographic technology, and Ethereum in particular. They’re not complex in principle, but can be a bit hard to grasp at first.

Let’s start with hashing. Using cryptographic math, you can convert any data into a (very large) unique integer. This operation is called hashing. There are some key properties to hashing algorithms:

1. They’re deterministic - identical inputs will always produce an identical output, each time and every time. But the chance of producing the same output using different inputs is extremely unlikely.

2. It is not possible (or computationally infeasible) to reverse engineer the input if you only have the output. It is a one-way process.

3. The output’s size (length) is fixed - the algorithm will produce fixed-size outputs for all inputs, regardless of the input size. In other words the outputs of a hashing algorithm will always have a fixed number of bits, depending on the algorithm.

There are many algorithms that are industry-standard for hashing but you’ll likely see SHA256 and Keccak256 most commonly. These are very similar. And the 256 refers to the size - the number of bits in the hash that is produced.

For example, go to this site and copy and paste “FreeCodeCamp” into the text input. Using the Keccak256 algorithm, the output will (always) be 796457686bfec5f60e84447d256aba53edb09fb2015bea86eb27f76e9102b67a.

This is a 64 character hexadecimal string, and since each character in a hex string represents 4 bits, this hexadecimal string is 256 bits (32 bytes long).

Now, delete everything in the text input box except the “F”. The result is a totally different hex string, but it still has 64 characters. This is the “fixed-size” nature of the Keccak265 hashing algorithm.

Now paste back the “FreeCodeCamp” and change any character at all. You could make the “F” lowercase. Or add a space. For each individual change you make, the hash hex string output changes a lot, but the size is constant.

This is an important benefit from hashing algorithms. The slightest change changes the hash substantially. Which means you can always test whether two things are identical (or have not been tampered with at all) by comparing their hashes.

In Solidity, comparing hashes is much more efficient than comparing the primitive data types.

For example, comparing two strings is often done by comparing the hashes of their ABI-encoded (bytes) form. A common helper function to compare two strings in Solidity would look like this:

We’ll address what ABI encoding is in a moment, but note how the result of encodePacked() is a bytes array which is then hashed using the keccak256 algorithm (this is the native hashing algorithm used by Solidity). The hashed outputs (256 bit integers) are compared for equality.

Now let's turn to ABI encoding. First, we recall that ABI (Application Binary Interface) is the interface that specifies how to interact with a deployed smart contract. ABI-encoding is the process of converting a given element from the ABI into bytes so that the EVM can process it.

The EVM runs computation on bits and bytes. So encoding is the process of converting structured input data into bytes so that a computer can operate on it. Decoding is the reverse process of converting bytes back into structured data. Sometimes, encoding is also referred to as “serializing”.

You can read more about the solidity built-in methods provided with the global variable abi that do different types of encoding and decoding here. Methods that encode data convert them to byte arrays (bytes data type). In reverse, methods that decode their inputs expect the bytes data type as input and then convert that into the data types that were encoded.

You can observe this in the following snippet:

I ran the above in Remix, and used the following inputs for encode(): 1981, 0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC, [1,2,3,4].

And the bytes I got returned were represented in hexadecimal form as the following:

0x00000000000000000000000000000000000000000000000000000000000007bd0000000000000000000000003c44cdddb6a900fa2b585dd299e03d12fa4293bc000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000030000000000000000000000000000000000000000000000000000000000000004

I fed this as my input into the decode() function and got my original three arguments back.

Thus, the purpose of encoding is to convert data into the bytes data type that the EVM needs to process data. And decoding brings it back into the human readable structured data that we developers can work with.

How to Call Contracts and Use the Fallback Function

Depending on the design of the smart contract and the visibility specifiers present in it, the contract can be interacted with by other smart contracts or by externally owned accounts.

Calling from your wallet via Remix is an example of the latter, as is using Metamask. You can also interact with smart contracts programmatically via libraries like EthersJS and Web3JS, the Hardhat and Truffle toolchains, and so on.

For the purposes of this Solidity handbook, we will use Solidity to interact with another contract.

There are two ways for a smart contract to call other smart contracts. The first way calls the target contract directly, by using interfaces (which we discussed previously). Or, if the Target contract is imported into the scope of the calling contract, it directly calls it.

This approach is illustrated below:

In Remix you can deploy Target first, and call count() to see that the default value for the count variable is 0, as expected. This value will be decremented by 1 if you call the decrement() method.

Then you can deploy TargetCaller and there are two methods you can call, both of which will decrement the value of count in Target.

Note that each of these two methods access the Target contract using a slightly different syntax. When interacting by using the ITarget interface, the first method takes in Target’s address whereas the second method treats Target as a custom type.

This second approach is only possible when the Target contract is declared in, or imported into, the same file as the TargetCaller. Most often, you will interact with smart contracts deployed by third parties, for which they publish ABI interfaces.

Each time you call these methods, the value of count in Target will decrease by 1. This is a very common way to interact with other smart contracts.

The second way to do it is by using the “low level” call syntax that Solidity provides. You use this when you also want to send some ether (value) to the target contract. Will talk about sending value in the next section, but for now just replace the code in Remix with the following:

You’ll note that decrement() now takes an argument, and the interface and the Target contract are updated with this new input data.

Next note that TargetCaller implements a new function that calls decrement() with a new syntax, explained below.

In the next section we will see examples of these low level ways of calling a target smart contract to send Ether to it.

But what happens when you call a contract and it doesn’t actually have the function you tried to call?

This can happen maliciously to exploit how Solidity works on the EVM. Or, more commonly, it can happen accidentally. This occurs when, for example, there is an error in the Interface and the compiler cannot match the function and parameters you sent with any that are actually contained in the contract. What happens then?

For these situations, many contracts employ a special function called the fallback function. The function looks like a normal function but it doesn’t need the function keyword. If you want it to also handle situations where your contract is sent some ether, you must also mark it payable. But this is not the recommended way to enable your contract to receive payments.

Let’s take a look by repurposing our previous Target, ITarget and TargetCaller and adding a fallback function as follows:

Once we deploy a fresh instance of Target, we can call count() and see that it is set to the default value of zero.

Next we can deploy TargetCaller and call the callFallback() method which internally calls nonExistentFunction().

It’s worth noting that the interface says that nonExistentFunction() is available but the actual Target contract does not have any such function. This is why the Target fallback function gets triggered and the value of count is now incremented by 1.

The purpose of the fallback function is to handle calls to the contract where no other function is available to handle it. And if the fallback is marked payable, the fallback function will also enable the smart contract to receive Ether (though that is not the recommended use for fallback). We will cover this in the next section.

How to Send and Receive Ether

To send Ether to a target contract from your smart contract, you need to call the target contract using one of the following three in-built Solidity methods: transfer, send or call.

transfer will throw an exception when it fails, and send and call will return a boolean that you must check before proceeding. Of these three, transfer and send are no longer recommended for security reasons, though you can still use them and they'll work.

Smart Contracts cannot receive Ether except in the following scenarios:

• They implement a payable fallback or payable receive special function, or

• Forcibly when the calling contract calls selfdestruct and forces a target contract to accept all its remaining ether. The calling contract is then deleted from the blockchain. This is a separate topic and is often used maliciously by exploiters.

It is generally recommended that you use a receive() function if you want your smart contract to receive Ether. You can get away by just making your fallback function payable, but recommended practice is to use a receive() function instead.

If you rely only on the fallback function, your compiler will grumble at you with the following message: “Warning: This contract has a payable fallback function, but no receive ether function. Consider adding a receive ether function.”

If you have both receive and fallback, you may legitimately wonder how Solidity decides which function to receive Ether with. This design decision also tells you what these functions are designed to do.

Receive is meant to receive ether. And fallback is meant to handle situations where the contract has been called but, as we discussed in the previous section, there is no matching method in the contract that can handle the call.

Solidity matches the method that was intended to be called by checking the msg.data field in the transaction sent by the caller. If that field is a non-empty value, and that value does not match any other function declared in the called contract, then the fallback method is triggered.

If msg.data is empty, then it will check if there is a receive function that has been implemented. If so, it will use that to accept the Ether. If no receive exists, it will default to the fallback function. The fallback is therefore the...fallback (default) method when nothing else makes sense.

The receive function is the better way to enable your contract to receive Ether. You can use the fallback function for any scenario where your smart contract is called but there is nothing to “handle” that call.

Here is a super handy logic tree that shows what receive and fallback are intended to handle.

(credit: Solidity By Example)

Going back to our example where we explored the fallback function, we can add a receive function to Target as follows:

We already saw how callFallback will change the count value in Target. But if we deploy a fresh instance of Target, we can now send it 10 wei, as shown below, because it now has a payable receive function. Prior to sending 10 wei (or any other amount) Target has a balance of zero, as shown below.

Hitting the Transact button with empty calldata (msg.data) will change the balance as shown in the image below. We can check count to see that it is incremented by 5, which is the logic in the receive function.

Sending Wei to the Target Contract and observing the updated balance

If we call callFallback and give it the address of the new Target instance, we will note that it increments only by 1. If we include some wei, that will increase the balance of Target as well.

So any transfer of Ether to a smart contract would require the receiving smart contract to have payable functions that can receive it. At the very minimum, the receiving smart contract would need a payable fallback function, though a payable receive function is the better approach for receiving Ether payments.

Solidity Libraries

In any programming language, a library refers to a collection of helper and utility functions that are designed to be reusable across multiple code bases. These functions solve specific, recurring programming problems.

In Solidity, libraries serve the same purpose, but have some special attributes.

First, they are stateless - that is, they don't store data (other than constants because these do not change the blockchain’s state). They also can't receive value (which means they cannot have payable receive or fallback functions).

They also cannot inherit from other contracts or libraries, nor can libraries have child (derived) contracts.

All functions declared in a library must not be abstract - that is, they must all have concrete implementations.

Since Solidity libraries are stateless, none of the methods in them can modify the blockchain’s state. This means all methods inside libraries are pure or view functions.

Another interesting attribute of Solidity libraries is that they don't need to be imported into your smart contract. They can be deployed as standalone contracts and then called via their interface in all consuming smart contracts - just as you would an API service in the traditional engineering world.

However, this is true only where the library contains public or external methods. Then that library can be deployed as a standalone contract with its own Ethereum address, and becomes callable to all consuming smart contracts.

If the libraries contain only internal methods, then the EVM simply “embeds” the library code into the smart contract that uses the library (because internal functions cannot be accessed from other smart contracts).

Libraries in Solidity have advantages that go beyond code reuse. Deploying a library one time on the blockchain can save on future gas costs by avoiding repeated deployment or importing of the library's code.

Let's look at a simple library and then dissect the code to understand how to use the library’s code.

This library has two methods that operate on the int data type. The first argument is called self for reasons that will become clear shortly. One method takes a number and then multiplies it by a constant value that is stored in the library’s code. The second method takes in two numbers and adds them.

Now let’s see how we can use this in a consuming smart contract.

The first thing to note is that there are two ways of using the WeirdMath library.

You can use it by either:

1. Invoking the library’s name followed by the function you want to call, or

2. calling the function directly on the data type you want the function to operate on. This data type must be identical to the type of the self parameter in the library’s function.

The first approach is demonstrated by method 1 in the code snippet where we invoke the library with WeirdMath.add(num1, num2);.

The second approach uses the Solidity using keyword. The expression return num1.add(num2); applies the WeirdMath library’s add function to the num1 variable. This is the same as passing it in as self, which is the first argument to the add function.

Events and Logs in Solidity

Smart Contracts can emit events. The events contain pieces of data that you the developer specify.

Events cannot be consumed by other smart contracts. Instead, they are stored on the blockchain as logs, and can be retrieved via the APIs that read from the blockchain.

This means that your application (most commonly your front end application) can “read” logs that contain the event’s data from the blockchain. In this way your user interface can respond to events on the blockchain.

This is how application user interfaces are updated to respond to on-chain events. Since these logs on the blockchain can be queried, logs are a cheap form of storage, as discussed previously in the discussion on storage areas.

Events emitted by a Smart Contract can be inspected using the relevant blockchain explorer, because everything on a public blockchain is publicly viewable. But if the smart contract’s bytecode has not been verified, the event data may not be human readable (it will be encoded). The events of verified smart contracts will be human readable.

Nodes and other blockchain clients can listen for (subscribe to) specific events. At the heart of it, this is how Chainlink Oracles work - decentralized oracle nodes listen to events from smart contracts, and then respond accordingly. They can even pull data off from events, run complex and resource-intensive computations off-chain, and then submit cryptographically verifiable computation results back onto the blockchain.

Other network APIs and indexing services like subgraphs are made possible because of the ability to query blockchain data via the events emitted by smart contracts.

Here is what an Event looks like in Solidity:

An event is first declared, and its arguments and their data types are specified. Any piece of data that has the indexed keyword is indexed by the EVM so that queries on blockchain logs can use the indexed param as a filter. This makes retrieving logs faster.

An event can store up to 4 indexed parameters - depending on whether it is anonymous or non-anonymous. Indexed event parameters are also referred to as “Topics” in the Solidity world.

Most events are non-anonymous, meaning that they include data on the name and arguments of the Event.

Non-anonymous events allow the developer to specify only 3 topics because the first topic is reserved to specify the Event signature in ABI-encoded hexadecimal form. You can read more about anonymous and non-anonymous topics here.

You can also explore Events on the relevant blockchain explorer (like etherscan.io).

You can approach this from one of two entry points. You can look at the contract’s address directly and then go to the Events Tab (which will show you events emitted by that contract only). Or you can go to a transaction hash and examine all the events emitted by all contracts that were touched by that transaction.

For example, below is a screenshot of the Chainlink VRF Coordinator smart contract’s events on Ethereum mainnet.

Inspecting the Chainlink VRF Coordinator Contract's events on etherscan

The contract tab has a green tick mark which means the contract is verified, and so the Event name and arguments are human readable. Take a moment to study this image as it contains a lot of information! If you’d like to study it directly on etherscan, click here.

This Chainlink VRF Coordinator contract responds to requests for cryptographically verifiable random numbers, and supplies the asking smart contract with random numbers (referred to as “random words”).

If you want to learn what the computer science meaning of “word” is, check out my colleague and I addressing this question in this Chainlink 2022 Hackathon video).

When the VRF Coordinator contract fulfills the request for random numbers, it emits a RandomWordsFulfilled event. That event contains 4 pieces of data, of which the first one, requestID, is indexed.

Solidity events contain three categories of data:

1. The address of the contract that emitted the event.

2. The topics (indexed event parameters used for filtering queries of the logs).

3. Non-indexed parameters, referred to as “data”, and they are ABI-encoded and represented in hexadecimal. This data needs to be ABI-decoded, in the way described in the section on ABI encoding and decoding.

When working in Remix you can also inspect the Events in the console as shown below:

Inspecting Event data in the Remix Browser IDE

You can also programmatically access the events using the contract receipts object in EthersJS. Using the code snippet we used above in the SimpleStorage contract, we can access the events using EthersJS and Hardhat using the following JavaScript:

You can also use libraries such as the EtherJs library in your front end application to listen to events and filter historical events. Both of these are useful when your application needs to respond to events on the blockchain.

Time Logic in Solidity

Time in Solidity is specified in relation to each block that gets added to the blockchain.

The global variable block.timestamp refers to the time, in milliseconds, when the block was generated and added to the blockchain. The milliseconds count refers to the number of milliseconds that have passed since the beginning of the Unix Epoch (in computing, this is 1 January 1970).

Unlike Web2 references to timestamps in milliseconds, the value may not increment with every millisecond.

A block often contains several transactions, and since block.timestamp refers to the time at which the block was mined, all transactions in a mined block will have the same timestamp value. So the timestamp really refers to the block’s time, and not so much as to when the caller may have initiated the transaction.

Solidity supports directly referring to the following units of time: seconds, minutes, hours, days, and weeks.

So we could do something like uint lastWeek = block.timestamp - 1 weeks; to calculate the timestamp of exactly 1 week before this current block was mined, down to milliseconds. That value would be the same as block.timestamp - 7 days;.

You can also use this to calculate future expiry dates, where, for example, you may want an operation to be possible between now and next week. You could do this with uint registrationDeadline = block.timestamp + 1 weeks; and then we could use the registrationDeadline as a validation or guard in a function as follows:

In this function we register the voter only if the current block’s timestamp is not past the registration deadline.

This logic is used extensively when we want to make sure certain operations are performed at the right time, or within an interval.

This is also one of the ways in which Chainlink Automation, a decentralized way to automate the execution of your smart contract, can be configured. The Chainlink decentralized oracle network can be configured to automatically trigger your smart contracts, and you can run a wide variety of automations by checking conditions, including time related conditions. These are used extensively for airdrops, promotions, special rewards, earn-outs and so on.

Conclusion and Further Resources

Congrats! You made it through this epic journey. If you’ve invested the time digesting this handbook, and run some of the code in the Remix IDE, you are now trained in Solidity.

From here on, it’s a question of practice, repetition and experience. As you go about building your next awesome decentralized application, remember to revisit the basics and focus on security. Security is especially important in the Web3 space.

You can get good information on best practices from OpenZeppelin’s blogs and the Trail of Bits resources, amongst others.

You can also get more practical hands on experience by doing the full end-to-end full-stack blockchain developer course that my colleague Patrick Collins published on freeCodeCamp (it’s free!).

There are other resources like blockchain.education and freeCodeCamp’s own upcoming Web3 curriculum that can consolidate your learnings.

In any event, this handbook can be your “desktop companion” for that quick refresh on basic concepts, regardless of what level of experience you’re at.

The important thing to remember is that Web3 technology is always evolving. There is a crying need for developers willing to tackle complex challenges, learn new skills, and solve important problems that come with decentralized architecture.

That could (and should) be you! So just follow your curiosity and don’t be afraid to struggle along the way.

And once again, I intend to keep this handbook updated. So if you see anything that’s not quite right, is outdated, or unclear, just mention it in a tweet and tag me and freeCodeCamp - a big shout out to you all for being part of keeping this handbook fresh.

Now…go be awesome!

Post Script

If you are serious about a career change to code, you can learn more about my journey from lawyer to software engineer. Check out episode 53 of the freeCodeCamp podcast and also Episode 207 of "Lessons from a Quitter". These provide the blueprint for my career change.

If you are interested in changing your career and becoming a professional coder, please reach out here. You can also check out my free webinar on Career Change to Code if that is what you're dreaming of.

• How I taught myself new skills

• How did I know that learning to code at 37 was not “too late”

• How I prepared for big tech coding interviews

• How I analyzed and minimized the risk of career change

• How I worked out that software engineering was “right” for me

• What languages I focused on

• Whether being a FANG/FAMGA software engineer is right for everybody (hint: tempting to think so but I saw plenty of evidence of it not being right for some people’s goals)

Note: I think “FAANG/FAMGA” is limiting, and prefer to refer to “big tech”since there are many highly prestigious companies other than the usual 4-5 everyone obsesses about.

Why I Wrote This Article

Each of those questions I listed above deserve their own article, even though I know our contemporary culture prefers “tweet-sized tips”. However, meaningful skills can’t be internalized from a few hundred characters.

So today I am going to share with you my answer to one of those questions – the approach I adopted when:

1. I made that first jump in careers from lawyer to coder at 38, and

2. How I prepared for big tech interviews at the ripe ol’ age of 39, with less than 2 years of coding behind me.

If you want me to write on any of the other questions I always get, please let me know! I’m going to put my contact details in this post somewhere, and the only way to find it will be to go through the article. 😊 That is also to encourage you to carefully read this article rather than skimming for a “quick tip”.

Uncovering the Real Goal

I hold the following opinions: getting coding interviews is harder than learning to code. Doing well at the interviews is often just as hard as getting the interviews. Behavioral interviews are hard if you don’t have solid experience – but your competition does.

When I made the switch I had zero coding background. Later, when I set my sights on Big Tech, I knew I’d be competing with PhDs, people who’d been coding since their teens (often 20+ years), and folks who'd achieved a lot more technically than I had in my one brief year’s experience as a developer.

And I had the added challenge of applying for big tech from outside the United States.

So I had to develop a plan that went well beyond just “learning to code”.

First, let me explain why I concluded that “learning to code” is the easy part, even though I’d tried and failed to learn to code 4 times, between 2012 and 2018.

This insight came to me in late 2018, when my startup was struggling. I had lost tens of thousands of dollars building my startup. I had no income for over 2 years.

And I decided to pull out over $40K from my mortgage. Why? I got into a top-rated bootcamp in San Francisco.

I left my family behind, and relocated to the US. I was meant to be there for over 14 weeks, but I withdrew from this top-rated bootcamp in week 1, and returned to Australia.

I had been super excited about the adventure (and scared from all the debt). But I developed grave doubts about the bootcamp strategy. I noticed that my instructors and the program were designed to help people "learn to code" rather than "becoming a coder".

From my experience on the hiring side in 3 other industries and 4 countries, I knew this was a mistake.

Learning to code is just a form of “literacy”. And literacy is not skill.

I was living proof: each of the 4 times I focused on “learning to code” I succeeded in a narrow sense. Whether it was HTML or Java or writing a simple Android app from a book, I always succeeded at learning how to read and write the basics. But I had no idea how to actually build anything useful. I was completely hopeless at applying my knowledge – I had no real “skill”.

In this century we don't get hired for what we know. We get hired for skill.

I quickly saw several reasons why a coding bootcamp was the wrong strategy for me.

This expensive bootcamp would likely give me some basic skills, the kind that may even get me an “entry level” job. But I could see things were going to be rushed, standardized, and focused on "getting out the other end".

I didn't want to "tick a box". I wanted skill. Competence. Confidence.

Moreover, bootcamps seemed to prepare everyone for entry level "junior developer" jobs.

I was 37 and I didn’t want to settle for an “entry-level job” mindset. Plus I don't believe anyone with 3 + years of experience is "junior" even if they're a total newbie in the field.

Then I discovered that several instructors and teaching assistants at the bootcamp were former students who hadn’t yet found jobs. They had never changed careers – many had never had a "career" as such. The job-search counselors had never even been on the interviewing side in tech.

How was I going to learn to do something from people who’d never done what I needed to do?

And then there was geography. Students based in San Francisco had advantages, whereas people coming from other parts of the US did not find work that easily and ran out of money in the long months after graduating and before getting their first job. Heck, I was based in Australia...how was this going to work for me?

I looked at my explicit goal. It was not to "learn to code" – it was to build a career that fulfilled me.

Plus, when researching bootcamps, I could see a path to learning “part time”. That seemed like a much more sustainable strategy to me because I could find a job and learn at night and on weekends. After over 2 years of no income, I needed to have cashflow in order to keep fear at bay, and focus on becoming a professional coder.

As the legendary businessman Harvey Firestone said:

“Having a surplus is the greatest aid to business judgment that I know.”

Having income while studying would give me the confidence to make better judgments. Better judgments are important for a career that is fulfilling in the long term.

I had no doubt that I would "learn to code" if I did 3-4 months in the bootcamp.

But would I learn enough for a good team to pay me money for my skills? I no longer believed bootcamps and online coding websites would help me achieve that.

Learning to code would not get me far. I had to be good enough to beat my competitors who had coding degrees, prior experience and networks. I wanted a career in code.

So I left the bootcamp, wrote off about $9,000 and returned to Australia. Sure, I had gained rudimentary coding literacy to pass the entrance test for the bootcamp. But I was very far from competent.

This analysis may be hard to understand if you’re new to the workplace. Another way to truly understand it is to notice that many of us play music, but aren’t in a band. As my mentor said:

"Michael Jordan didn’t want to learn to play basketball. He wanted to be in the NBA. Huge difference."

How to Use Your Advantages & Disadvantages

That one insight made all the difference. Within 8 months, in 2019, I got all 4 developer jobs I applied for. I simply followed the program I’d developed with the help of my (non-technical!) coach.

Don't be misled though. I had unexpected advantages. I benefited from 2 major advantages that initially appeared to be disadvantages.

This was not my first career change, and for almost a decade, I had been on the hiring side in previous careers. Today I’ve been on the hiring side in engineering too, and the patterns are very similar.

My biggest advantage was that I was no longer thinking about the job market challenge as a candidate. I was thinking about this from the perspective of the hiring manager. That had a significant impact on my plan because I had experience of how hiring managers think – their constraints, priorities, values, business needs, dislikes, red flags...

My experience (age?) helped me understand what impact I could deliver to a team and organization, and to identify the right people that I could learn from, who could help me with ideas, guidance, suggestions and referrals. What was a disadvantage was now an advantage – a career changer's advantage.

On the point of advantages, I want to call out something important.

The easiest thing in the world is to dismiss others for having “killer” advantages. It's easy to shrug and say "Of course – that’s why they succeeded". It’s easy to not notice that everyone has very compelling and serious disadvantages.

For me, my “disadvantages” were:

• Geography – I don’t live in the US or one of the other tech hubs

• No formal computer science qualifications

• Zero technical background

• A mortgage and financial responsibilities

• My “age” – learning new skills when you’re pushing 40 is harder than when you’re 25

• Cultural and social expectations, judgment, negativity

• Recruiters and bosses being younger than I was and uncertain of how to deal with me

• Compared to other candidates, people viewed my change as very, very “risky”

• I was going to earn less in engineering than I would as a lawyer

Every day I hear from people who let any one of these things stop them. While I cannot comment on whether or not they’re real reasons, I can say that if we argue for our limitations, we get to keep them.

Clinging on to our disadvantages does not help us overcome them.

With the help of my coach and a ton of mindset/psychology training, I was able to dig into my disadvantages and convert a few of them (not many!) into major advantages. And that’s when I realized that my experience with how interviewing, recruitment, and resourcing works would help me a lot in formulating a strategy.

Priority one: career change

When learning to code and intentionally setting the goal of becoming a professional coder, I found that I kept switching to short-term thinking and obsessing about my first role.

I wanted my first role to be glorious, to prove all my critics wrong, to pay me a gazillion dollars and save me from having to deal with self doubt and struggle for the rest of my life.

But I had to train myself to view things differently. My first role must be one that makes me learn and grow, and set me up for future success no matter what happened. It needed to pay fair market rates, but I was happy to take a slightly lower salary if the team was fantastic and the growth/learning was solid. It did not need to be my dream job.

I was very explicit in the tradeoffs I’d make:

• Team matters more than brand

• Team matters more than money

• Brand matters more than money (as it would set me up for future opportunities)

• Money matters more than stock (this was for my first dev role, as this would make my continued learning plan sustainable, even though stock may give me more financial upside)

• But, learning mattered more than brand or money – as learning more would save me time which is better than saving me money

• I would not trade off between team and learning. I needed both (but I was more likely to get reliable signals on the team than I would on the actual learning/growth I’d get on the job)

“Nailing it” or being comfortable was not a priority. My priority was to successfully change careers.

Accepting a rubbish coding job (and there are plenty of them…) would not be a “successful” career change for me. But equally I didn’t have to get into big tech (ever) for my career change to be “successful”. This was very personal – success to me meant loving the work I do and learning a lot doing it. Period.

How I Made a Customized Plan

After analyzing my advantages and "convertible" disadvantages, the next thing I needed to develop was a customized plan. I needed one that was tailored to me, that I could believe in.

I needed a plan that took into account my specific context, which includes my temperament, experience, beliefs, values, goals and skills.

Note that I’ve still not talked about the technical interviews, the algorithms and data structures, and so on. When developing my plan, I had to focus on all the many pieces that had nothing to do with my coding skills or technical capability.

It would also need to account for my psychological “runway” – how much time was I prepared to invest in this career change, before I’d give up, lose hope, or change my mind? I could not answer that without understanding how long it was going to take me to learn the minimum required skills.

To understand that, I needed to research and analyze what the minimum set of engineering skills that the market would value were.

And to understand that, I needed to analyze the dozens of engineering domains in the market, and which ones would suit my temperament, interest/passion and advantages. And from that analysis I’d need to pick the domains that I would focus on and exclude all others.

I'd have to find the overlap between my interests, my ability and what the market valued.

Again, my experience on the hiring side of the market gave me some (small but important) advantages. I knew that pure technical skill would not be enough – that’s just the starting point. the “invitation to the dance”.

I also knew that good teams don’t just hire for raw technical skill, they hire for essential non-technical attributes as well. What those traits are depend on the technical domain, the team culture, the current composition of the team, and so on.

Yes, you're right. A custom plan is extremely multi-dimensional, and getting things right is only moderately helpful, whereas getting things wrong can result in a giant loss of direction and waste of time.

Since I was almost halfway through my career, I was determined not to repeat past mistakes. I was going to be:

• Highly specific in my goals.

• Highly intentional in my choices and actions.

• Focused on what I wanted, paying no attention to the things I did not want.

• Willing to change myself, my habits and negative beliefs, so that I could change the world around me.

• Ready to focus on building a rewarding and fulfilling career rather than just “getting the next job”.

• Ready to focus on delivering value for my future team rather than a “what’s in it for me” mindset.

• Willing to play the long game – think in 5-10-25 year horizons rather than the next few weeks.

I must confess, doing these things consistently was much harder than I anticipated. I slipped up a lot, especially on the first three. But since I had my custom plan written down, I let that be my guide and the sole source of truth for what I needed to do.

My plan required me to focus on fundamental programming skills, and then narrow it down to the segment that I felt matched my long-term goals and skills. For me, this was web development. This meant completely and ruthlessly avoiding all the “shiny new things” and doing things like Python or Java.

And tutorials and endless videos were not going to make me beat the competition. I calculated that the minimum set of skills I’d need to develop for my city would require 900-1100 hours of focused coding, practicing the right things in the right sequence.

Preparing the plan took several weeks. I kept refining and strengthening it and didn’t rush it artificially. I am very inspired by Abraham Lincoln (another lawyer who changed careers!) and he once said “Give me six hours to chop down a tree and I will spend the first four sharpening the axe”.

I was so tempted to dive right into my plan and get “busy” but I’d learned that being busy was so not the same as being effective. Once the plan felt as complete as it could be with the information I had at my disposal, I turned to executing on that plan, with total focus.

This meant a lot of sacrifice, and many, many days of self doubt, battling the temptation to switch, and learning to manage my energy. I developed some amazing habits during this time, but they were only obvious to me in hindsight. During those 6 months of work, I was constantly assailed by uncertainty, fear and occasional loss of hope.

Later on, I adapted this planning process to create a plan for Big Tech and especially for Google. That plan took another 500-600 hours of intentional study that was completely different from my plan to become a developer. More on that later.

Early Results, and then…Google SWE

Another lesson I’d learned from 4 years of trial, error, and failure was that I was very likely to change plans midway, switch resources, courses or focus.

This is a much more serious problem than we realize because every time we switch focus or plans we throw out the hard work we’ve done, go back to square 1 and…start…all…over…again.

Imagine if you were driving from A to B and kept taking U-turns and going back and restarting. You’d never get anywhere.

But I’d made myself a promise (it’s easier to keep a single promise than to keep a bunch of them!): I was going to complete my plan and then decide whether I would keep going. This time I wasn’t going to stop until I completed my plan.

I didn’t have to like doing it. I just had to like the possibilities that lay at the end of doing it.

My plan had a specific time when I would start interviewing even if I didn’t feel ready for interviews. But to get to that stage I had to get good at generating interview opportunities.

Again, my past experience in other careers helped me. I applied all my learnings over 18 years and secured 4 interviews within a few weeks, and got all 4 offers even though there were plenty of candidates with more experience, skill, and qualifications than I had.

It wasn’t because I was better at coding. I don’t see how that could be, given that I’d only been in the game for a few months.

I believe I got all 4 offers because during the interview process I demonstrated that I was a better candidate for the hiring manager. This approach was critical in how I presented myself.

Getting the offers is great, but I had an unexpected problem. Since my plan required me to be highly intentional about the kind of work I’d pursue, all 4 roles I’d applied for were the ones I really believed would be a fantastic start to my new career. How was I going to choose?

Yes it’s a great problem to have, but that doesn’t make the decision easy!

To resolve this problem intelligently, I learned to ask myself a very important question:

Do I know this or do I simply think this?

Very often we make decisions based on assumptions and beliefs that are completely untested, and we mistake our opinions or fantasies for reality. What we actually know is way less than what we believe without truly knowing.

Being intentional required me to focus on what I know rather than what I wished, or merely thought. I had to either find evidence to back my thoughts, or disregard thoughts for what I knew.

That framework developed my analytical powers and helped me choose the right first role in early 2019. I turned 39 around that time.

Till today I continue to use this know vs think framework in my personal decision making as well as in my engineering decision making. I find it to be an excellent framework for analyzing tradeoffs in complex decisions.

Looking back, I learned a TON about sticking to my plan, revisiting my goals, self-awareness and practicing being intentional.

A few months into my new role, I noticed that a lot of lawyers were starting to reach out and ask me how I did this. Curiously, many of them were people who insisted I was making a huge mistake and that wanting to learn to code at this stage was immature and foolhardy.

Now they wanted to “learn to code”. And that made me think.

What else have others said is “not possible”? Do they know that or do they just think that?

And my goals hadn’t changed. For me learning, growth, and team were still more important than brand or money. But I was almost 40 and I also wanted to explore life in a way that I’d never had the courage to do in the first half of my career.

So I decided to set myself a stretch goal: I was going to find out what being a software engineer at Big Tech was like. I’d worked in huge companies before and I knew that it isn’t for everyone – that’s the reason I went into startup and smaller companies in the first place.

But wouldn’t I learn and grow SO much by being surrounded by some of the “best” people in their field, from engineering to product and sales? Did I know this? Or did I just think it?

I did the research and found that on average people were happy in Big Tech. But I also found that most people were not as intentional as I was. So I confined my research to people who were very intentional about their careers. And they (almost universally) said they grew a lot from Big Tech, even if they decided to leave. Leaving Big Tech was also intentional, in pursuit of their end goals.

So I decided to try for big tech (including 3 of the FAMGA companies) and several others, in Silicon Valley, New York, and Seattle. I was still based in Australia, so this was a massive challenge.

I re-designed my plan. Several of the steps were the same but the coding curriculum would have to be massively overhauled. I also needed to find out how US big tech recruitment is done, and be worthy of a referral.

About 7 months later, I started generating interviews. I worked very hard to prove myself worthy of referrals for those 7 months, and people offered to give me referrals based on my efforts and proven commitment.

I got referred to Meta (it was called Facebook then) but I didn't get the interview as my skills were not the right match. This was a massive learning for me as I thought I’d been careful to only apply for roles where my skills matched – and I was wrong.

That was when I realized that sometimes the position description is written in ways that mean one thing for the hiring company but means very different things outside that company. This is because different organizations use the same words to mean different things. Both the hiring side and the candidate side can be unaware of this!

All these learnings added up. Within 3 months I got offers from 2 big tech companies, and missed out on a third at the final interview because I just didn’t know how to write a filesystem from scratch (I didn’t understand much about the Linux world at all!).

And then I got an offer from Google.

Again, I was faced with a very difficult decision. Google is a cultural force that is very hard to be objective about. But I really wanted to stay true to my goals, my plan, and my intentions.

Trying to separate what I knew from what I thought was EXTREMELY hard when it came to Google. But I was absolutely sure about one thing: the team I’d interviewed with were amazing people.

And this is where I believe luck matters. No matter what people say about skills, brains, smarts and all that stuff, luck and "magic" have a role in life.

My interviewers from Google were friendly, kind, cheerful and highly focused engineers. They were not looking to prove I sucked. They wanted to help me prove I was good. They answered my questions with enthusiasm, and I felt like collaborators from the first minute.

Is this a Google thing? Maybe. But later, when training to be a technical interviewer at Google, I witnessed a very wide variety of interviewer/hiring manager styles and beliefs. I saw candidates with great skill losing their nerve, struggle to communicate their process, and so on. And I came to appreciate the role of chance in all this.

So yes – I was lucky that I had the kind of interviewers I did, and that on interview day I happened to know how to work through the code.

This is also where my ultra-focused preparation on type of work and relevant skills paid off. There are a very large number of engineering roles at Big Tech that I would never have been suited to (like the one at Meta). Even with my prior experience in other careers, I had no idea how HUGE the engineering world is, and how many flavors and types there are, and how hard it is to tell them apart.

By forcing myself to be highly intentional and not randomly and blindly applying for big tech engineering roles, I improved my chances in small but important ways. I dug deep into each role and researched them carefully by speaking to friends in the industry (again – my age and experience was an asset because I had built relationships over 15+ years never expecting that they’d be so useful later on!).

For each offer I got, I had researched the role deeply, and prepared for the technical interviews well. On interview day the stars were aligned, and things worked out. Though I don’t think I “nailed” my interviews, I did well enough to communicate that I was a suitable candidate for the team’s needs.

Which leads me to the question: How did I prepare for the Big Tech interviews?

How to Prepare for Big Tech Interviews

Answer: In two phases that took me over 500 hours to execute.

Phase 1: Learn the Realities and Competitive Landscape

If I wanted to tackle Big Tech, from another country, with less than a year of industry experience, and 15+ years in an unrelated career, with no Computer Science degree, I needed to have a very clear view of the realities, especially the competitive landscape.

This means there was room for informed hope, but no room for dreamy-eyed, pie-in-the-sky magical and wishful thinking.

Hard truths (see my initial YouTube Videos on this here). Hard realities. Hard work.

I had to fully internalize and appreciate the following:

• Coding is just the starting point. Not the ending point. Learning to code was the first step of many, one small piece of a much larger puzzle. In other words, it is necessary, but not sufficient to get a coding job (and especially at big tech).

• My biggest enemy would be my own mind. It would either let the negativity of others get me down, or it would turn against me and undermine my own confidence. I had to build the habits that would help me have a resilient mindset. I focused on recovering from setbacks rather than avoiding them.

• My competitors would likely not be career changers. Or if they were, they’d be from closely related fields like computer engineering, mechanical engineering, or electronics engineering. The vast majority would have technical qualifications, maybe even PhDs (this turned out to be true!), and several years of industry experience.

• As an outlier and a “wildcard”, the hardest part would be getting the interviews. Learning algorithms and data structures would be easier. And “cracking the coding interview” (whatever that means…) would be easier too. Why? Code is deterministic – identical code generally produces identical results. But life is not deterministic, and getting interviews is highly subjective. In the job market, identical actions do not produce identical results.

• I needed to shape myself into the kind of person seasoned engineers would want to work with

• I assumed most of my competitors would have at least 3 -5 years of experience. I couldn't catch up with them, let alone overtake them. Instead, I needed to outperform them on non-technical skills and compare favorably (if not outperform them) on the technical stuff.

• I had to communicate better than the others. If I didn’t know something, I’d need to say so and then communicate how I’d solve it if given the right time and opportunity. I also had to communicate to show interviewers that I understood their business needs and wasn’t just focusing on my selfish dreams.

• Which means I had to really work on understanding what the hiring team valued, was looking for, wanted, and needed.

• I could not control my competitors (their skill, their performance, how much they knew, and so on), or what my interviewer was thinking, wanting, what they valued, or whether they liked career changers or not. I could not control most things. I could only optimize my effort, my focus, my psychology, and how much learning I extract out of each experience, good or bad. I could only control my choices and actions. So focusing on anything outside that would be a waste of precious energy.

• I had to accept the role of luck. Jordan, Tendulkar, Federer – they’ve all had bad days. I would too. Or maybe I'd do great, but on the day someone else would do better. Or someone else would just be a better fit for what the team needed. No harm, no foul. I’ve made those hard decisions on the interviewing side countless times myself, so I know how these things work.

• If I was successful at more than 1 offer I would need to pre-think and pre-agree with myself about what signals and factors I would use to decide (I’d learned from my experience of choosing between my first 4 offers!).
  If you’re wondering how I knew how to do all this…I didn’t. Not all at once. What you’re reading is a summary in hindsight. But I had to work most of this out in “real time” based on “first principles thinking”. And I worked with my mentor to tighten them up as I went along. It took a lot of time and I was so impatient to “start coding”. But …I knew what advice Abraham Lincoln would give me…

Phase 2: How I Chose My Learning Resources

I know everyone thinks there is a secret “magic bullet” out there. Some blog, video, resource, tutorial, podcast, PDF cheatsheet…_something…_that will unlock the entire “secret” and make us instantly learn things.

Nope.

I’ll bang on about this till the day I die – information is a commodity.

Learning is hard, but it’s made much harder because there is too much free information.

We all fall into the trap of thinking there is some missing piece of information. There isn’t.

Why? Because no matter where you live, what language you speak, what color your eyes, skin, or hair are, what gender you identify with – all resources are going to teach you things that “work”. They’re all “the same” at a very fundamental level.

They have to be, because that’s how computers work.

If you and I write an identical function in JavaScript or Python or Java, we are going to get identical results. That’s how computers work – they’re deterministic algorithms.

But life (and interviewing) is most decidedly not deterministic. Identical effort, grades, skill, intelligence will not produce identical results.

Again, I had to teach myself to learn. I had to teach myself to shift my attention away from resources/blogs/websites/courses and put my attention on building solid mental models, identifying relevant skills, drilling down on concepts rather than code implementations, applying what I already knew in new ways, reasoning, problem solving, and communicating my reasoning while I reasoned.

Now you’re going to be surprised by the resources I used for my Google and other Big Tech interviews.

Yes, I used Leetcode, Algoexpert, InterviewCake and Jenny’s CS Lectures and perhaps a few others. But I didn’t complete a single one of them.

This wasn’t because I switched or lost focus. I was being intentional. I realized that they all teach the same thing, but with slightly different styles and content. So I used these resources to learn concepts and mixed and matched all these resources based on my analyses of interview patterns.

My reasoning was simple. Having been on the hiring side, I know that every year the quality of candidates (at good companies) improves. I personally think that focusing on the company is a huge mistake – we should focus on the team, people and kind of work.

But the world works a certain way, and because of that, everyone rushes to the big companies. This increases competition which makes it harder for hiring managers to assess candidates.

The only way for hiring managers to deal with this is to raise the bar, making it harder for candidates. The overall number of candidates keeps increasing, but the “pool” of candidates that get invited to the interview stays within a narrow range – usually 2 -10 people. Regardless of how many hundreds apply that’s about the number that will even get a chance to interview.

Therefore, the number of candidates that don’t hear back or meet with rejection increases, especially in bull markets.

If the competition is increasing and the resources on the internet are also increasing, but the shortlisted number stays more or less constant, then “learning more” cannot be the solution. Everyone is “learning more”, so relative to the competition there is no change.

I also realized that Big Tech would have lists of interview questions (this is “efficient” as interviewing is very time consuming, and so it makes sense to save time by having a question bank that interviewers could use). Naturally, they would not use these questions if they were “leaked” – that would defeat the interview process.

So, logically, hiring managers will not ask questions that are available on Leetcode or Algoexpert or other sites. This produces a kind of “arms race” – the more questions are made publicly available, the more questions in the question bank are changed. That results in more innovation and variance in questions and hiring strategies.

This left me only one choice. I had to learn to solve problems using mental models and by classifying them. Chances are I was never going to be asked to sort a Linked List or to implement Dijkstra’s shortest path algorithm. Instead, I would need to know how to apply these sorts of algorithms to “real world”, practical problems.

Often real world problems don’t look, sound, or smell like the practice questions we study. Practice questions and competitive coding questions tend to be “neatly” packaged up with clear constraints.

But as an interviewer, I wanted to know how the candidate thinks, reasons, analyses, interprets information, and collaborates. Solving the problem is a bonus. Often they have the right solution but run out of time – but ask great questions and clearly know how to tackle the problem. These candidates can still get the offer.

Later, as an engineer at Google, I could always tell if someone knew how to solve something even if they were unable to solve it in time. Equally, it’s immediately obvious when a candidate does not know how to solve something (and that's OK – we are all learning).

By adopting my approach of understanding problem types and solutions rather than specific code implementation, I could focus on learning to reason rather than learning to write specific algorithms.

This approach meant that I completed less than 40% of Algoexpert (and back then it had half the questions it has now). I also did maybe 50-60 questions from Leetcode and most of them weren’t the “hard” ones.

I figured that “hard” questions would probably show up in 45 minute interviews about 20% of the time which means 80% of the time they’d be easy or medium questions. So it made sense to optimize for the 80% given I was still relatively new to engineering and the hard problems would get in the way of understanding easy and medium ones.

I used these resources to recognise patterns rather than just “complete” and get a certification. That’s why I didn’t complete any of them. And no, I didn’t use “Cracking the coding interview” either.

Along the way I also started a systematic process of understanding system design questions. I documented system design preparation in a long form blog on the essential concepts for system design interview questions.

I also decided to focus on one language: JavaScript. It wasn’t the best for interviewing (seasoned engineers discourage it on Quora and elsewhere) but I didn’t see how it mattered. The purpose of the interview was not to test my choice of languages – it was to test my ability to think abstractly and solve complex computer science-y problems.

The language is just a tool (another core belief I hold). In fact, using an untyped language like JS would give me opportunities to talk about its limitations or strengths, which is an opportunity to demonstrate that I understand the tradeoffs in language choices. That way, I could showcase broader knowledge and insight without actually having to implement it all in code.

But a lot of the resources I learned from used Java or C++. These languages are the dominant languages at Google. So being forced to read these languages and understand the principles forced me to not focus too much on “spitting out the code” and more on reasoning through the code so that I could write it.

That was my entire plan. Practice, Pattern Recognition, Mental Models/First Principle Thinking, System Design, doing fewer things really well, and focusing on getting interviews rather than just learning more code.

How to Stand Out During Interviews

As I mentioned, we all have advantages and disadvantages. And we all think our disadvantages are special and huge and our advantages are common, unremarkable, and probably not of much use.

This is not true. Logically, if we all think our disadvantages are severe, then we should all succumb to them. Yet some people overcome them. Only to find that others had it even worse and overcame those.

It’s way better to focus on what we can do well. For me, I truly believed that I could add a lot of value to the team. I don’t care about being the smartest or the best. But I care about being an outstanding learner, and maintaining my growth mindset at all costs.

And so I tried to use this to stand out as best I could. I learned to ask really good questions of recruiters, interviewers, and hiring managers.

But there was a deeper reason for this. Asking good questions was my way of interviewing the company. Like I said, I didn’t want to repeat the mistakes of the first half of my career. So asking really good questions was important for me to assess whether the company was a fit for me and not just the other way around.

Since I had decided that I was going to value the team and the learning over all else, I never asked about the compensation until the recruiter raised it. It was going to be lower than what I earned as a lawyer anyway!

Instead I focused very hard on learning about the team, its dynamics, its group beliefs and values, how the manager solved problems (especially people problems), what the team was interested in, what the company division was interested in, how its balance sheet was doing, what its strategy was, what its resource allocation and budgeting looked like, and so on.

All these things were things I'd learned in other industries, as an individual contributor, as a manager, executive, founder and so on.

And all of these things also showed that I was being highly intentional. I was genuinely interested in the team, and the company, its product and its future. This wasn’t just another job I was applying for. This was active and personal…not passive and opportunistic.

And I believe that helped me stand out. Maybe not for all the roles I interviewed for, but for many of the offers I got.

When I was on the hiring side, I’d always prefer candidates who were genuinely interested in the role, the people, the product and the company. Those that showed up just “to get a job” didn’t have the kind of energy and drive we wanted.

Interview Planning and Strategy

The last aspect of my roadmap required me to deeply understand the different types of coding interview processes.

This included technical and non-technical interviews, the format of interviews, the way companies organize them, run them, plan them, staff them, evaluate them, and weigh them. But it also required me to understand my strengths and weaknesses.

When going for Big Tech in the US I was aiming for 2-3 interviews per month – a huge goal given that I was not even in the US and was in a timezone that was 17 hours ahead of the west coast.

I had to plan and structure interviews at weird hours so I could accommodate my day job as a developer and also my study time. Some interviews took 6 hours, some took 10 or more. Some were “pair programming for a day” type interviews.

All this took a ton of planning, and mental training. I had to be careful to get the right sleep, the right exercise, maintain my mindset and confidence, deliver on my day job, be there for my family, study and maintain focus on my goals.

For this, I had to be honest with myself at what I was good at. For example, I am not a morning person. But I can endure late nights. So I structured interviews, work, sleep, or even exercise accordingly.

There were some interviews that were scheduled at 2am or after, and I wouldn’t sleep before (as I really am not good at waking up on time!). So I’d do a full workout at 1am to get my energy and focus up, then interview, then sleep till 10 am, then go to work and manage my schedule accordingly.

I would also be careful to plan interviews so that I wasn’t doing two of them back to back unless they were very similar and time bound. For example, doing a take-home test and doing a timed test in the same week require different planning from doing a take-home and a live-coding interview – all while managing work and family.

To schedule the interviews appropriately I’d work closely with the recruiter and be very transparent with them. This had two advantages: I gained credibility and trust with the recruiter for being collaborative and communicative, and they also got to see that I had other opportunities going, which increased the value of my candidacy. Competition is a good thing.

Concluding Thoughts

I am sure many of you were expecting this article to give “insider” tips and a specific set of languages, and DSA questions to learn. I believe I’ve given you something much better. Rather than giving you fish, I am showing you how to fish.

Apart from the ethics of it, insider tips are of limited value, especially in Big Tech. In huge companies things can be vastly different from team to team and city to city. You need to understand the principles of recruitment and career development, rather than just the specific languages and algorithms. Assuming that all interviews are the same is a big mistake.

And as for our obsession with data structures and algorithms…managing your career is the ultimate algorithm. Your mind is the ultimate data structure. Learn how to work with both and you’ll always do well, despite the occasional failures. Powerful ideas aren’t grandiose – they’re elegantly minimal.

If you have read this article you will have noticed some of the things I linked to include multiple ways to reach me. You can also get invited to my webinars, mini-courses and newsletters if you want to go beyond just “learning to code” and learn how to build a career that is right for you.

Perhaps the most important message I can leave you with is that it is a mistake to obsess about big tech. Yes, they’re great to be part of, but if we believe they’re the only thing that is right for us we will miss out on all the other amazing opportunities.

Big Tech has glamor because of cultural trends these days. Sure, it’s nice to work for a great organization, but plenty of great organizations are not well known. Plus what is “great” for one person is pain for another.

Your number one priority is to be happy, fulfilled, and live the life that you want. This is not derived from a company. This comes from the people you spend time with (especially colleagues) and the kind of work you do. Big Tech companies have their share of bad managers, teammates, and work, just like any other company.

If you build skill, stack a great plan on top of the right mindset, and train yourself to set the right goals, you can achieve much more than you dreamed of – with or without Big Tech.

Post Script

If you would like to learn more about my journey from lawyer to software engineer, check out episode 53 of the freeCodeCamp podcast and also Episode 207 of "Lessons from a Quitter". These provide the blueprint for my career change.

If you are interested in teaching yourself to code, changing careers and becoming a professional coder, or becoming your own technical co-founder, please reach out here. You can also check out my free webinar on Career Change to Code if that is what you're dreaming of.

In this article I'm going to share with you ten important lessons I learned during this career transition.

I finished law school in Bangalore, India, a year before Google listed on the Nasdaq. That was a long time ago. The world was very different. I think Facebook was still “thefacebook”.

I started my career as a litigator in the courts in India. Two years later I switched to corporate law, and then ended up moving to Australia to join one of the world’s largest law firms, in their Melbourne office.

Several years later, after surviving the Global Financial Crisis of 2008, I became an Australian citizen and moved “in-house” – the term for when lawyers move from working at a law firm to working internally at one specific company.

I loved being a lawyer, because it was filled with smart, ambitious people and it paid well. But there was always a part of me that longed to code, to create, and to contribute to the internet’s repository of useful tools.

To me, programming a computer is such a magical act. To write text, and have machines follow your commands – there is something so powerful, so limitless about that. It fired my imagination.

But I really did not view myself as “mathematical” or “nerdy” or a “brainiac”. Every bit of social messaging around hackers, coders, programmers, and technical founders (which by 2012 was a status symbol!) reinforced the myth that programming was the domain of “wickedly smart” math wizards.

I unwittingly reinforced that myth to myself – I tried to teach myself to code THREE times. In 2014, in 2015 and in 2017. And all three times I quit because I tried to jump too high, set myself up for failure, and then assumed I was not smart enough (when actually, I had just tried to run before I’d learned to walk).

Programming felt beyond my reach. Just. Too. Hard. I had no clarity on where to begin, what bits to learn, and most frustratingly, what bits to ignore.

Instead, I started a tech company and hired a team of coders. One of them became my technical co-founder. He quit a year into it, and I was left with partnerships and contracts with local governments that I had to honour.

Faced with the choice of being defeated or learning to code, I chose to try again – to learn to code.

I was 37. In 2020 COVID hit, and just as it was getting very serious, I signed my contract to be an an engineer at Google. I had just turned 39.

It felt odd to be “starting” over. But it also feels incredibly fun to be a beginner again. In fact, I can actually truly relate to what Steve Jobs said at his famous 2005 Stanford Commencement Address:

“The heaviness of being successful was replaced by the lightness of being a beginner again, less sure about everything. It freed me to enter into one of the most creative periods of my life.”

I had left a successful legal career and then a less than successful startup journey to be a beginner again. As a coder and coach, I am being more creative than I’ve ever had the privilege of being.

This is not about Google or any other fabled tech company. This is about you. It doesn’t matter what your goal is. I want you to know that transformations are not “special” or “miraculous” – they just feel that way.

The people I coach have gained an understanding of how I “found” my roadmap to becoming a coder, in a world where there is way too much information and consequently way too little clarity. But there isn’t enough time here to go into that.

Today I want to help you by whittling down my 10 most important principles for people wanting to transform their lives and achieve their goals.

You’ll see that this is true of coding, or anything else. But since I am now a self-taught engineer, working at Google, it is particularly relevant to others who are coders or aspiring coders.

At the end of this piece, I’ve put down some ways to reach out to me if you have any questions.

5 Things NOT to Do When Changing Careers

I am starting with the “don’ts” because it’s easier to identify things we do that hold us back, than to develop new habits. Tackling the don’ts first will give you quick wins that will build your confidence – and from there you can build up to the “Dos”.

DON'T Look at the Mountain Top

For a long time, you’re going to be climbing a steep mountain. It’s going to feel long in the moment, but it’s a short period in the context of Life.

The mountaintop will not get closer no matter how often you look at it. But the more often you look at it, the more you’ll get discouraged because somewhere, we are all in a hurry. Things take time. And as Jeff Bezos says, you can’t skip steps.

If you lose your optimism you lose confidence, and then you lose energy and momentum.

Instead, just focus on the process of learning. No matter how much code you write, you are always learning new tools, techniques, practices, languages, frameworks and so on. And if you rush yourself into feeling “competent” you will disappoint yourself, because at first you’re going to get suck.

That’s normal. You can’t skip the novice stage when you’re learning, and if you try to skip it, you will get frustrated and lose your passion.

The process of learning is to do a little bit consistently, rather than a ton of stuff in an inspired burst. 60 minutes every day for 3 months is far more effective than blitzing it for 20 hours on a weekend and then not doing anything for a month.

So don’t focus on “becoming a good engineer” or “getting that job”. Instead, focus on learning and getting your repetitions up until things start to feel comfortable.

DON’T Mistake Doubts for Evidence

You’re going to have doubt and fear with you every step of the way. If you think of it as a shadow, you may stop paying so much attention to it.

Doubts have a way of convincing us they’re real. They spotlight our weaknesses and over-emphasise our failures. And then, without realising it, we take them as evidence of our ability.

This one takes practice. And since you’re never going to be doubt-free no matter how much of an expert you are, you should practice making space for your doubts while not paying too much attention to them.

No matter what your doubts tell you, there is only one thing you know for certain: if your doubt is right, it is right only right now. Tomorrow it could be wrong. If not tomorrow, the day after that.

Your doubts don’t prove anything about tomorrow. Chances are your doubts are not right at all – they’re just insecurities and exhaustion.

Rather than fight doubt, it’s better to do some vigorous 10 minute exercise or take a nap. If doubts disappear (and you’ve experienced that!) then they can’t be all that real.

DON’T Measure Reality Against your Secret Wishes

All of us have secret hopes and fantasies. We hope we will be recognised as brilliant and our boss will promote us without us having done much, or that we can get super-fit bodies with 2 months of going to the gym, or that our first 3 blogs will get a bazillion comments and shares on social media. We secretly wish for success, without too much effort, and very quickly.

Maybe. Statistically, some people will get that lucky, just as a dead clock will tell you the right time twice a day.

But if you’re not aware of this secret wish and its power over your mind, you will feel disappointment. And you will not find the energy and discipline to keep going.

The truth is pretty much any goal that is beyond your current skills takes a lot of time, energy, focus, and discipline. And it requires you to maintain those practices well past the point you were secretly hoping for instant gratification. In fact you could go for months without much positive reinforcement from the world around you.

So measuring your results against your secret wishes is a recipe for giving up. Instead measure your results against your previous results.

I use a simple technique for this – my personal motto is to be 1% better every day. As long as you’re getting a little bit better each day (maybe you just read a blog, or did some deliberate practice for 30 minutes), then you would have improved yourself. These micro-improvement don’t stack – they compound.

Measure yourself against your past results, not your secret fantasies and you will feel tremendous motivation to keep going.

DON’T Take Big Decisions on Bad Days

You’re going to have a lot of bad days. Don’t negotiate with that – accept it and don’t put too much emotional weight on it. Even when you’re the best at what you do, you’re going to have bad days.

So you must expect a lot more of them, a lot more frequently, when you’re learning complex new skills. You must fully expect that you will have such bad days that the only rational-feeling thing to do is to quit.

You will have such bad days that you cannot see how things can get better. You will have such horrible days that you will assume the way you feel right then is how you’re going to feel forever unless you give yourself the relief of quitting.

I went through this so many times. Not just in coding – but in every single job I’ve ever had. This is why people respond so well to having a coach. A coach is “an external mind” that can see more clearly than you when you’re lost, struggling, confused, or discouraged.

The threat of quitting on a bad day is so high that I developed a technique to get around this. Rather than constantly arguing and negotiating with myself in my head, and confusing myself to exhaustion, I simply follow one rule: is this a bad day? Oh yes. Ok, no decisions today.

Rather than quit, take a break. When I was learning to code, I would sometimes stop studying and coding for a week if I was dangerously close to giving up. I figured I’d rather lose 1 week of progress than quit altogether. Most often, in 2-3 days my internal “state” would change and I’d be back at it, harder than ever.

In fact I found a rule from an olympic gymnast that I adopted for myself. I even wrote about it when I found it, as a way to ensure I followed her advice.

You can give up any time you like….as long as you’re having a good day. But you cannot give up when you’re having a bad day.

DON’T Assume Your Results Reflect Your Ability

It is easy to assume that your results reflect your ability. In a sense, that is true – they either reflect your effort or they reflect your current ability. And that’s the most important distinction to make.

If it’s not a failure of effort, then it is a shortcoming in your current skills. And that’s great, because skills can always be improved. It’s just hard because we secretly wish it would be faster, easier, and more glamorous to get to success!

Most importantly, it is critical not to let your internal dialogue switch from “I don’t know how to do X” to “I cannot do X”.

We often unwittingly confuse lack of knowledge or skill with lack of ability or potential. This is particularly easy to do when we’re surrounded by people who are a whole lot better than we are.

I felt this so many times. As a junior lawyer and then 15 years later as a junior engineer. I assumed other people were talented or gifted and I was not. As I grew in experience, other “juniors” viewed me as talented or gifted. It was just experience and learning.

There is a line from T. Harv Ecker that I really found to be true in my various careers:

“If you’re not doing as well as you’d like, all that means is there’s something you don’t know.”

Also, as a close cousin to this principle, follow John Wooden’s advice:

"Never let what you cannot do get in the way of what you can. "

There will be many weeks when you’re grappling with something that you simply don’t know how to do. It won’t come easy. It won’t come quick. And you may feel very stuck.

But there will always be something – maybe it’s something small – that you can do to take one step forward. By always moving forward, even if it's a tiny step, you will keep your momentum. And usually those small steps are the small learnings you need to stack up to solve the bigger problem that has you so badly stuck.

5 Things to DEFINITELY Do When Changing Careers

DO Work on Your Growth Mindset

If you practice the top 5 Don’ts, you will automatically start building a growth mindset. At first you may not have a growth mindset – it’s more likely to be that you will gradually remove the unproductive “fixed mindset” habits that we all have.

If you aren’t familiar with the concept of the growth mindset, take a look at this TED talk by Carol Dweck. She has even published her research in a bestselling book. This is a foundational worldview that will help in everything you do.

Note that I say “work” on your growth mindset. This is not a weekend hack. It is a lifestyle. It requires constant practice and self-awareness.

In my experience, the most powerful impact of practicing a growth mindset is that I asked radically different questions when I was failing or struggling.

For example, instead of asking myself “why can’t I figure this out?” I trained myself to ask “how can I find someone or something to help me figure this out”.

I remember when I was learning JavaScript promises, and I was really struggling to get the concept. At first I sank into depression because I read a few blogs and I just couldn’t grasp some of the intricacies and mental models. I felt drained and frightened, overwhelmed and very alone.

I watched this TED talk and I immediately asked myself – do I know any JavaScript developers? I went to LinkedIn and searched my contacts (I was too shy to ask total strangers on Twitter – a sign of my fixed mindset!).

I found a guy I’d met at a startup event, who was self taught and had built a JavaScript webapp. I reached out to him and it turned out his office was 15 minutes from my house. Two days later he spent his lunch break showing me how to use promises. It felt easy after that.

Carol Dweck was right – it wasn’t that I didn’t know JavaScript promises, it was that I didn’t know them...yet.

DO Make Friends with the Impostor Syndrome

Impostor Syndrome is when you secretly think that you’re not good enough to do what you’re doing and that you will get found out and exposed as a fraud, or an impostor. I wouldn’t say it’s common. I’d say it’s almost universal.

When onboarding at Google, I heard this topic come up so many times. There are tons of blogs, resources, and internal guidance on how to deal with it. Engineers feel it everywhere. Heck, the co-founder of Atlassian did a TED talk on it.

So, it’s a form of doubt, and we already covered how to deal with doubt in the DON’Ts section.

Impostor Syndrome has some added layers to it, because it typically arises in professional settings. That’s why it gets special treatment here in the DOs section.

Professional contexts produce more fear of embarrassment because we have all this baggage about performance, and promotions, and being reviewed by bosses. All that extra baggage gives it more power over us.

As you start to get successful in your coding journey, make friends with Impostor Syndrome and joke about it – it will oddly make your mindset more growth oriented, because by owning that you don’t know something, you open yourself to learning it.

If you hide it, you spend a lot of energy on soothing yourself that you could otherwise use to learn the thing you don’t know yet.

DO Practice Reframing Failure and Setbacks as Moments of Learning

We all hate the feeling of failing. We also can’t avoid it. What do you do when you can’t avoid something you dislike intensely?

You attach a different meaning to it.

If you classify setbacks and disappointments as failures, then you will emotionally respond to them as failures. If you classify setbacks and disappointments as information about what you need to learn, then you respond to them as lessons.

Let me be clear – this is not easy to do! Like everything else I’m writing about here, this takes practice. Just like coding.

As clichéd as it is, the emotional impact of the glass is half full versus the glass is half empty debate is very real. In fact marketers know this. They don’t tell you your fruity yoghurt is 20% fat. They tell you it’s 80% fat-free.

You’re absolutely going to fail, and if you learn from it then you have to ask yourself – did I fail or did I just learn something?

Put all your attention on understanding what you learned from that failure, analyse, deconstruct, and internalise it. That way, you won’t notice failing so much.

DO Hold Yourself Accountable by Making Fewer Decisions

As a lawyer, I can tell you that our minds are experts at arguing for both sides, depending on what suits it. If you ask your mind why it’s OK to skip a run today, it will tell you. If you ask your mind why running is important, it will tell you that too.

How do we get anything done with a mind that changes with the wind?

We don’t ask the mind. We simply commit to one action. If we ever discuss this action with our mind, then we will get sucked into a negotiation with, or rather against, ourselves. This wastes a lot of energy.

Have you ever gone around in circles in your head wondering whether you really need to do the assignment now, or whether you can finish watching the movie on Netflix and then find time for it later? We all have.

It’s much easier if we say, from 8 to 10pm I do nothing but X. Until I do X I will not do Y. But once I do X I will reward myself with Y, and not get distracted by Z.

If it sounds like a lot of effort, I can promise you that it’s way less effort than having the same internal debate every few days and then beating yourself up for it later.

Commit once and save yourself the headache of constantly negotiating with yourself. This is the best way I’ve found to keep myself accountable, because I don’t need to keep checking in. I just check against one rule that is not up to my mind to decide.

DO Focus on the Insight Rather than the Implementation

This one is important for coders and is a little less meta than the other points.

Ultimately, most people reading this are looking to code professionally or for economic gain of some sort. It is easy to be in a hurry when progress has taken longer than you secretly wished, or planned for.

As a result, we tend to try and learn “tricks” and “shortcuts”. These are tactically useful as they will help us make some progress. But they also come at a cost – those shortcuts may not help us apply the principles to unfamiliar contexts.

For example, if you’re learning how to use recursion, it’s tempting to memorise the approach rather than build your mental intuition around it. But if you can’t see how the same technique can be used to solve other problems or a “class” of problems, then you’re missing out on the real power of the tool.

This can then catch up with you in interviews or at the workplace. In interviews, you may fail to recognise the underlying “class” of problems.

For example, when I was first teaching myself data structures I once failed to recognise a problem about HR reporting structures as a tree/graph traversal problem. That actually happened during a mock interview! I had practiced graph traversals and memorised implementations but hadn’t fully internalised the concept.

Similarly, without understanding concepts you cannot communicate effectively with team members or people who come to you to learn. This will set you back at the workplace because effective engineers need to communicate abstract concepts well.

At a more general level, adopting this “DO” means that you won’t fake it. It will force you to be patient and persistent in developing your understanding, rather than a hack around true comprehension.

No matter what skill you’re learning in life, having clarity of concepts is much more scalable than knowing just one implementation of the concept.

Conclusion

On a departing note – these are my top 5 Dos and Don’ts. If you make them yours I can guarantee that you will make progress, and come up with your own - that is extremely important as your own framework can only emerge from confidence and practice.

But having your own framework will make you a real weapon, because it’s reusable across any learning context.

I really hope that these lessons are of value to you. I truly believe that if one person can do it, I can too, and if you believe that, you will be able to do it, too.

Post Script

If you would like to learn more about my journey from lawyer to software engineer, check out episode 53 of the freeCodeCamp podcast and also Episode 207 of "Lessons from a Quitter". These provide the blueprint for my career change.

If you are interested in teaching yourself to code, changing careers and becoming a professional coder, or becoming your own technical co-founder, please reach out here. You can also check out my free webinar on Career Change to Code if that is what you're dreaming of.

I wrote this in-depth guide when preparing for my FAANG software engineering interviews. It covers the essential software system design concepts you need to reason your way through distributed systems, and it helped me get into Google as an engineer, after more than 15 years of being a corporate lawyer.

This is not an exhaustive treatment, since System Design is a vast topic. But if you're a junior or mid-level developer, this should give you a strong foundation.

From there, you can dig deeper with other resources. I've listed some of my favourite resources at the very bottom of this article.

I've broken this guide into bite-sized chunks by topic and so I recommend you bookmark it. I've found spaced learning and repetition to be incredibly valuable tools to learn and retain information. And I've designed this guide to be chunked down into pieces that are easy to do spaced repetition with.

1. Section 1: Networks & Protocols (IP, DNS, HTTP, TCP etc)

2. Section 2: Storage, Latency & Throughput

3. Section 3: Availability

4. Section 4: Caching

5. Section 5: Proxies

6. Section 6: Load Balancing

7. Section 7: Consistent Hashing

8. Section 8: Databases

9. Section 9: Leader Election

10. Section 10: Polling, Streaming, Sockets

11. Section 11: Endpoint Protection

12. Section 12: Messages & Pub-Sub

13. Section 13: Smaller Essentials

Let's get started!

Section 1: Networks and Protocols

"Protocols" is a fancy word that has a meaning in English totally independent of computer science. It means a system of rules and regulations that govern something. A kind of "official procedure" or "official way something must be done".

For people to connect to machines and code that communicate with each other, they need a network over which such communication can take place. But the communication also needs some rules, structure, and agreed-upon procedures.

Thus, network protocols are protocols that govern how machines and software communicate over a given network. An example of a network is our beloved world wide web.

You may have heard of the most common network protocols of the internet era - things like HTTP, TCP/IP etc. Let's break them down into basics.

IP - Internet Protocol

Think of this as the fundamental layer of protocols. It is the basic protocol that instructs us on how almost all communication across internet networks must be implemented.

Messages over IP are often communicated in "packets", which are small bundles of information (2^16 bytes). Each packet has an essential structure made up of two components: the Header and the Data.

The header contains "meta" data about the packet and its data. This metadata includes information such as the IP address of the source (where the packet comes from) and the destination IP address (destination of the packet). Clearly, this is fundamental to being able to send information from one point to another - you need the "from" and "to" addresses.

And an IP Address is a numeric label assigned to each device connected to a computer network that uses the Internet Protocol for communication. There are public and private IP addresses, and there are currently two versions. The new version is called IPv6 and is increasingly being adopted because IPv4 is running out of numerical addresses.

The other protocols we will consider in this post are built on top of IP, just like your favorite software language has libraries and frameworks built on top of it.

TCP - Transmission Control Protocol

TCP is a utility built on top of IP. As you may know from reading my posts, I firmly believe you need to understand why something was invented in order to truly understand what it does.

TCP was created to solve a problem with IP. Data over IP is typically sent in multiple packets because each packet is fairly small (2^16 bytes). Multiple packets can result in (A) lost or dropped packets and (B) disordered packets, thus corrupting the transmitted data. TCP solves both of these by guaranteeing transmission of packets in an ordered way.

Being built on top of IP, the packet has a header called the TCP header in addition to the IP header. This TCP header contains information about the ordering of packets, and the number of packets and so on. This ensures that the data is reliably received at the other end. It is generally referred to as TCP/IP because it is built on top of IP.

TCP needs to establish a connection between source and destination before it transmits the packets, and it does this via a "handshake". This connection itself is established using packets where the source informs the destination that it wants to open a connection, and the destination says OK, and then a connection is opened.

This, in effect, is what happens when a server "listens" at a port - just before it starts to listen there is a handshake, and then the connection is opened (listening starts). Similarly, one sends the other a message that it is about to close the connection, and that ends the connection.

HTTP - Hyper Text Transfer Protocol

HTTP is a protocol that is an abstraction built on top of TCP/IP. It introduces a very important pattern called the request-response pattern, specifically for client-server interactions.

A client is simply a machine or system that requests information, and a server is the machine or system that responds with information. A browser is a client, and a web-server is a server. When a server requests data from another server then the first server is also a client, and the second server is the server (I know, tautologies).

So this request-response cycle has its own rules under HTTP and this standardizes how information is transmitted across the internet.

At this level of abstraction we typically don't need to worry too much about IP and TCP. However, in HTTP, requests and responses have headers and bodies too, and these contain data that can be set by the developer.

HTTP requests and responses can be thought of as messages with key-value pairs, very similar to objects in JavaScript and dictionaries in Python, but not the same.

Below is an illustration of the content, and key-value pairs in HTTP request and response messages.

source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages

HTTP also comes with some "verbs" or "methods" which are commands that give you an idea of what sort of operation is intended to be performed. For example, the common HTTP methods are "GET", "POST", "PUT", "DELETE" and "PATCH", but there are more. In the above picture, look for the HTTP verb in the start line.

Section 2: Storage, Latency & Throughput

Storage

Storage is about holding information. Any app, system, or service that you program will need to store and retrieve data, and those are the two fundamental purposes of storage.

But it's not just about storing data – it's also about fetching it. We use a database to achieve this. A database is a software layer that helps us store and retrieve data.

These two primary types of operations, storing and retrieving, are also variously called 'set, get', 'store, fetch', 'write, read' and so on. To interact with storage, you will need to go through the database, which acts as an intermediary for you to conduct these fundamental operations.

The word "storage" can sometimes fool us into thinking about it in physical terms. If I "store" my bike in the shed, I can expect it to be there when I next open the shed.

But that doesn't always happen in the computing world. Storage can broadly be of two types: "Memory" storage and "Disk" storage.

Of these two, the disk storage tends to be the more robust and "permanent" (not truly permanent, so we often use the word "persistent" storage instead). Disk storage is persistent storage. This means that when you save something to Disk, and turn the power off, or restart your server, that data will "persist". It won't be lost.

However, if you leave data in "Memory" then that usually gets wiped away when you shut down or restart, or otherwise lose power.

The computer you use everyday has both these storage types. Your hard disk is
"persistent" Disk storage, and your RAM is transient Memory storage.

On servers, if the data you're keeping track of is only useful during a session of that server, then it makes sense to keep it in Memory. This is much faster and less expensive than writing things to a persistent database.

For example, a single session may mean when a user is logged in and using your site. After they log out, you may not need to hold on to bits of data that you collected during the session.

But whatever you do want to hold on to (like shopping cart history) you will put in persistent Disk storage. That way you can access that data the next time the user logs in, and they will have a seamless experience.

Ok, so this seems quite simple and basic, and it's meant to be. This is a primer. Storage can get very complex. If you take a look at the range of storage products and solutions your head will spin.

This is because different use-cases require different types of storage. They key to choosing the right storage types for your system depends on a lot of factors and the needs of your application, and how users interact with it. Other factors include:

• the shape (structure) of your data, or

• what sort of availability it needs (what level of downtime is OK for your storage), or

• scalability (how fast do you need to read and write data, and will these reads and writes happen concurrently (simultaneously) or sequentially) etc, or

• consistency - if you protect against downtime using distributed storage, then how consistent is the data across your stores?

These questions and the conclusions require you to consider your trade-offs carefully. Is consistency more important than speed? Do you need the database to service millions of operations per minute or only for nightly updates? I will be dealing with these concepts in sections later, so don't worry if you've no idea what they are.

Latency

"Latency" and "Throughput" are terms you're going to hear a lot of as you start to get more experienced with designing systems to support the front end of your application. They are very fundamental to the experience and performance of your application and the system as a whole. There is often a tendency to use these terms in a broader sense than intended, or out of context, but let's fix that.

Latency is simply the measure of a duration. What duration? The duration for an action to complete something or produce a result. For example: for data to move from one place in the system to another. You may think of it as a lag, or just simply the time taken to complete an operation.

The most commonly understood latency is the "round trip" network request - how long does it take for your front end website (client) to send a query to your server, and get a response back from the server.

When you're loading a site, you want this to be as fast and as smooth as possible. In other words you want low latency. Fast lookups means low latency. So finding a value in an array of elements is slower (higher latency, because you need to iterate over each element in the array to find the one you want) than finding a value in a hash-table (lower latency, because you simply look up the data in "constant" time , by using the key. No iteration needed.).

Similarly, reading from memory is much faster than reading from a disk (read more here). But both have latency, and your needs will determine which type of storage you pick for which data.

In that sense, latency is the inverse of speed. You want higher speeds, and you want lower latency. Speed (especially on network calls like via HTTP) is determined also by the distance. So, latency from London to another city, will be impacted by the distance from London.

As you can imagine, you want to design a system to avoid pinging distant servers, but then storing things in memory may not be feasible for your system. These are the tradeoffs that make system design complex, challenging and extremely interesting!

For example, websites that show news articles may prefer uptime and availability over loading speed, whereas online multiplayer games may require availability and super low latency. These requirements will determine the design and investment in infrastructure to support the system's special requirements.

Throughput

This can be understood as the maximum capacity of a machine or system. It's often used in factories to calculate how much work an assembly line can do in an hour or a day, or some other unit of time measurement.

For example an assembly line can assemble 20 cars per hour, which is its throughput. In computing it would be the amount of data that can be passed around in a unit of time. So a 512 Mbps internet connection is a measure of throughput - 512 Mb (megabits) per second.

Now imagine freeCodeCamp's web-server. If it receives 1 million requests per second, and can serve only 800,000 requests, then its throughput is 800,000 per second. You may end up measuring the throughput in terms of bits instead of requests, so it would be N bits per second.

In this example, there is a bottleneck because the server cannot handle more than N bits a second, but the requests are more than that. A bottleneck is therefore the constraint on a system. A system is only as fast as its slowest bottleneck.

If one server can handle 100 bits per second, and another can handle 120 bits per second and a third can handle only 50, then the overall system will be operating at 50bps because that is the constraint - it holds up the speed of the other servers in a given system.

So increasing throughput anywhere other than the bottleneck may be a waste - you may want to just increase throughput at the lowest bottleneck first.

You can increase throughput by buying more hardware (horizontal scaling) or increasing the capacity and performance of your existing hardware (vertical scaling) or a few other ways.

Increasing throughput may sometimes be a short term solution, and so a good systems designer will think through the best ways to scale the throughput of a given system including by splitting up requests (or any other form of "load"), and distributing them across other resources etc. The key point to remember is what throughput is, what a constraint or bottleneck is, and how it impacts a system.

Fixing latency and throughput are not isolated, universal solutions by themselves, nor are they correlated to each other. They have impacts and considerations across the system, so it's important to understand the system as a whole, and the nature of the demands that will be placed on the system over time.

Section 3: System Availability

Software engineers aim to build systems that are reliable. A reliable system is one that consistently satisfies a user's needs, whenever that user seeks to have that need satisfied. A key component of that reliability is Availability.

It's helpful to think of availability as the resiliency of a system. If a system is robust enough to handle failures in the network, database, servers etc, then it can generally be considered to be a fault-tolerant system - which makes it an available system.

Of course, a system is a sum of its parts in many senses, and each part needs to be highly available if availability is relevant to the end user experience of the site or app.

Quantifying Availability

To quantify the availability of a system, we calculate the percentage of time that the system's primary functionality and operations are available (the uptime) in a given window of time.

The most business-critical systems would need to have a near-perfect availability. Systems that support highly variable demands and loads with sharp peaks and troughs may be able to get away with slightly lower availability during off-peak times.

It all depends on the use and nature of the system. But in general, even things that have low, but consistent demands or an implied guarantee that the system is "on-demand" would need to have high availability.

Think of a site where you backup your pictures. You don't always need to access and retrieve data from this - it's mainly for you to store things in. You would still expect it to always be available any time you login to download even just a single picture.

A different kind of availability can be understood in the context of the massive e-commerce shopping days like Black Friday or Cyber Monday sales. On these particular days demand will skyrocket and millions will try to access the deals simultaneously. That would require an extremely reliable and high-availability system design to support those loads.

A commercial reason for high availability is simply that any downtime on the site will result in the site losing money. Also, it could be really bad for reputation, for example, where the service is a service used by other businesses to offer services. If AWS S3 goes down, a lot of companies will suffer, including Netflix, and that is not good.

So uptimes are extremely important for success. It is worth remembering that commercial availability numbers are calculated based on annual availability, so a downtime of 0.1% (i.e. availability of 99.9%) is 8.77 hours a year!

Hence, uptimes are extremely high sounding. It is common to see things like 99.99% uptime (52.6 minutes of downtime per year). Which is why it is now common to refer to uptimes in terms of "nines" - the number of nines in the uptime assurance.

In today's world that is unacceptable for large-scale or mission critical services. Which is why these days "five nines" is considered the ideal availability standard because that translates to a little over 5 minutes of downtime per year.

SLAs

In order to make online services competitive and meet the market's expectations, online service providers typically offer Service Level Agreements/Assurances. These are a set of guaranteed service level metrics. 99.999% uptime is one such metric and is often offered as part of premium subscriptions.

In the case of database and cloud service providers this can be offered even on the trial or free tiers if a customer's core use for that product justifies the expectation of such a metric.

In many cases failing to meet the SLA will give the customer a right to credits or some other form of compensation for the provider's failure to meet that assurance. Here, by way of example, is Google's SLA for the Maps API.

SLAs are therefore a critical part of the overall commercial and technical consideration when designing a system. It is especially important to consider whether availability is in fact a key requirement for a part of a system, and which parts require high availability.

Designing HA

When designing a high availability (HA) system, then, you need to reduce or eliminate "single points of failure". A single point of failure is an element in the system that is the sole element that can produce that undesirable loss of availability.

You eliminate single points of failure by designing 'redundancy' into the system. Redundancy is basically making 1 or more alternatives (i.e. backups) to the element that is critical for high availability.

So if your app needs users to be authenticated to use it, and there is only one authentication service and back end, and that fails, then, because that is the single point of failure, your system is no longer usable. By having two or more services that can handle authentication, you have added redundancy and eliminated (or reduced) single points of failure.

Therefore, you need to understand and de-compose your system into all its parts. Map out which ones are likely to cause single points of failure, which ones are not tolerant of such failure, and which parts can tolerate them. Because engineering HA requires tradeoffs and some of these tradeoffs may be expensive in terms of time, money and resources.

Section 4: Caching

Caching! This is a very fundamental and easy-to-understand technique to speed up performance in a system. Thus caching helps to reduce "latency" in a system.

In our daily lives, we use caching as a matter of common-sense (most of the time...). If we live next door to a supermarket, we still want to buy and store some basics in our fridge and our food cupboard. This is caching. We could always step out, go next door, and buy these things every time we want food – but if its in the pantry or fridge, we reduce the time it takes to make our food. That's caching.

Common Scenarios for Caching

Similarly, in software terms, if we end up relying on certain pieces of data often, we may want to cache that data so that our app performs faster.

This is often true when it's faster to retrieve data from Memory rather than disk because of the latency in making network requests. In fact many websites are cached (especially if content doesn't change frequently) in CDNs so that it can be served to the end user much faster, and it reduces load on the backend servers.

Another context in which caching helps could be where your backend has to do some computationally intensive and time consuming work. Caching previous results that converts your lookup time from a linear O(N) time to constant O(1) time could be very advantageous.

Likewise, if your server has to make multiple network requests and API calls in order to compose the data that gets sent back to the requester, then caching data could reduce the number of network calls, and thus the latency.

If your system has a client (front end), and a server and databases (backend) then caching can be inserted on the client (e.g. browser storage), between the client and the server (e.g. CDNs), or on the server itself. This would reduce over-the-network calls to the database.

So caching can occur at multiple points or levels in the system, including at the hardware (CPU) level.

Handling Stale Data

You may have noticed that the above examples are implicitly handy for "read" operations. Write operations are not that different, in main principles, with the following added considerations:

• write operations require keeping the cache and your database in sync

• this may increase complexity because there are more operations to perform, and new considerations around handling un-synced or "stale" data need to be carefully analyzed

• new design principles may need to be implemented to handle that syncing - should it be done synchronously, or asynchronously? If async, then at what intervals? Where does data get served from in the mean time? How often does the cache need to be refreshed, etc...

• data "eviction" or turnover and refreshes of data, to keep cached data fresh and up-to-date. These include techniques like LIFO, FIFO, LRU and LFU.

So let's end with some high-level, and non-binding conclusions. Generally, caching works best when used to store static or infrequently changing data, and when the sources of change are likely to be single operations rather than user-generated operations.

Where consistency and freshness in data is critical, caching may not be an optimal solution, unless there is another element in the system that efficiently refreshes the caches at intervals that do not adversely impact the purpose and user experience of the application.

Section 5: Proxies

Proxy. What? Many of us have heard of proxy servers. We may have seen configuration options on some of our PC or Mac software that talk about adding and configuring proxy servers, or accessing "via a proxy".

So let's understand that relatively simple, widely used and important piece of tech. This is a word that exists in the English language completely independent of computer science, so let's start with that definition.

Source: https://www.merriam-webster.com/dictionary/proxy

Now you can eject most of that out of your mind, and hold on to one key word: "substitute".

In computing, a proxy is typically a server, and it is a server that acts as a middleman between a client and another server. It literally is a bit of code that sits between client and server. That's the crux of proxies.

In case you need a refresher, or aren't sure of the definitions of client and server, a "client" is a process (code) or machine that requests data from another process or machine (the "server"). The browser is a client when it requests data from a backend server.

The server serves the client, but can also be a client - when it retrieves data from a database. Then the database is the server, the server is the client (of the database) and also a server for the front-end client (browser).

Source: https://teoriadeisegnali.it/appint/html/altro/bgnet/clientserver.html#figure2

As you can see from the above, the client-server relationship is bi-directional. So one thing can be both the client and server. If there was a middleman server that received requests, then sent them to another service, then forwards the response it got from that other service back to the originator client, that would be a proxy server.

Going forward we will refer to clients as clients, servers as servers and proxies as the thing between them.

So when a client sends a request to a server via the proxy, the proxy may sometimes mask the identity of the client - to the server, the IP address that comes through in the request may be the proxy and not the originating client.

For those of you who access sites or download things that otherwise are restricted (from the torrent network for example, or sites banned in your country), you may recognize this pattern - it's the principle on which VPNs are built.

Before we move a bit deeper, I want to call something out - when generally used, the term proxy refers to a "forward" proxy. A forward proxy is one where the proxy acts on behalf of (substitute for) the client in the interaction between client and server.

This is distinguished from a reverse proxy - where the proxy acts on behalf of a server. On a diagram it would look the same - the proxy sits between the client and the server, and the data flows are the same client <-> proxy <-> server.

The key difference is that a reverse proxy is designed substitute for the server. Often clients won't even know that the network request got routed through a proxy and the proxy passed it on to the intended server (and did the same thing with the server's response).

So, in a forward proxy, the server won't know that the client's request and its response are traveling through a proxy, and in a reverse proxy the client won't know that the request and response are routed through a proxy.

Proxies feel kinda sneaky :)

But in systems design, especially for complex systems, proxies are useful and reverse proxies are particularly useful. Your reverse proxy can be delegated a lot of tasks that you don't want your main server handling - it can be a gatekeeper, a screener, a load-balancer and an all around assistant.

So proxies can be useful but you may not be sure why. Again, if you've read my other stuff you'd know that I firmly believe that you can understand things properly only when you know why they exist - knowing what they do is not enough.

Section 6: Load Balancing

If you think about the two words, load and balance, you will start to get an intuition as to what this does in the world of computing. When a server simultaneously receives a lot of requests, it can slow down (throughput reduces, latency rises). After a point it may even fail (no availability).

You can give the server more muscle power (vertical scaling) or you can add more servers (horizontal scaling). But now you got to work out how the income requests get distributed to the various servers - which requests get routed to which servers and how to ensure they don't get overloaded too? In other words, how do you balance and allocate the request load?

Enter load balancers. Since this article is an introduction to principles and concepts, they are, of necessity, very simplified explanations. A load balancer's job is to sit between the client and server (but there are other places it can be inserted) and work out how to distribute incoming request loads across multiple servers, so that the end user (client's) experience is consistently fast, smooth and reliable.

So load balancers are like traffic managers who direct traffic. And they do this to maintain availability and throughput.

When understanding where a load balancer is inserted in the system's architecture, you can see that load balancers can be thought of as reverse proxies. But a load balancer can be inserted in other places too - between other exchanges - for example, between your server and your database.

The Balancing Act - Server Selection Strategies

So how does the load balancer decide how to route and allocate request traffic? To start with, every time you add a server, you need to let your load balancer know that there is one more candidate for it to route traffic to.

If you remove a server, the load balancer needs to know that too. The configuration ensures that the load balancer knows how many servers it has in its go-to list and which ones are available. It is even possible for the load balancer to be kept informed on each server's load levels, status, availability, current task and so on.

Once the load balancer is configured to know what servers it can redirect to, we need to work out the best routing strategy to ensure there is proper distribution amongst the available servers.

A naive approach to this is for the load balancer to just randomly pick a server and direct each incoming request that way. But as you can imagine, randomness can cause problems and "unbalanced" allocations where some servers get more loaded than others, and that could affect performance of the overall system negatively.

Round Robin and Weighted Round Robin

Another method that can be intuitively understood is called "round robin". This is the way many humans process lists that loop. You start at the first item in the list, move down in sequence, and when you're done with the last item you loop back up to the top and start working down the list again.

The load balancer can do this too, by just looping through available servers in a fixed sequence. This way the load is pretty evenly distributed across your servers in a simple-to-understand and predictable pattern.

You can get a little more "fancy" with the round robin by "weighting" some services over others. In the normal, standard round robin, each server is given equal weight (let's say all are given a weighting of 1). But when you differently weight servers, then you can have some servers with a lower weighting (say 0.5, if they're less powerful), and others can be higher like 0.7 or 0.9 or even 1.

Then the total traffic will be split up in proportion to those weights and allocated accordingly to the servers that have power proportionate to the volume of requests.

Load-based server selection

More sophisticated load balancers can work out the current capacity, performance, and loads of the servers in their go-to list and allocate dynamically according to current loads and calculations as to which will have the highest throughput, lowest latency etc. It would do this by monitoring the performance of each server and deciding which ones can and cannot handle the new requests.

IP Hashing based selection

You can configure your load balancer to hash the IP address of incoming requests, and use the hash value to determine which server to direct the request too. If I had 5 servers available, then the hash function would be designed to return one of five hash values, so one of the servers definitely gets nominated to process the request.

IP hash based routing can be very useful where you want requests from a certain country or region to get data from a server that is best suited to address the needs from within that region, or where your servers cache requests so that they can be processed fast.

In the latter scenario, you want to ensure that the request goes to a server that has previously cached the same request, as this will improve speed and performance in processing and responding to that request.

If your servers each maintain independent caches and your load balancer does not consistently send identical requests to the same server, you will end up with servers re-doing work that has already been done in as previous request to another server, and you lose the optimization that goes with caching data.

Path or Service based selection

You can also get the load balancer to route requests based on their "path" or function or service that is being provided. For example if you're buying flowers from an online florist, requests to load the "Bouquets on Special" may be sent to one server and credit card payments may be sent to another server.

If only one in twenty visitors actually bought flowers, then you could have a smaller server processing the payments and a bigger one handling all the browsing traffic.

Mixed Bag

And as with all things, you can get to higher and more detailed levels of complexity. You can have multiple load balancers that each have different server selection strategies! And if yours is a very large and highly trafficked system, then you may need load balancers for load balancers...

Ultimately, you add pieces to the system until your performance is tuned to your needs (your needs may look flat, or slow upwards mildly over time, or be prone to spikes!).

We've talked about VPNs (for forward proxies) and load-balancing (for reverse proxies), but there are more examples here.

Section 7: Consistent Hashing

One of the slightly more tricky concepts to understand is hashing in the context of load balancing. So it gets its own section.

In order to understand this, please first understand how hashing works at a conceptual level. The TL;DR is that hashing converts an input into a fixed-size value, often an integer value (the hash).

One of the key principles for a good hashing algorithm or function is that the function must be deterministic, which is a fancy way for saying that identical inputs will generate identical outputs when passed into the function. So, deterministic means - if I pass in the string "Code" (case sensitive) and the function generates a hash of 11002, then every time I pass in "Code" it must generate "11002" as an integer. And if I pass in "code" it will generate a different number (consistently).

Sometimes the hashing function can generate the same hash for more than one input - this is not the end of the world and there are ways to deal with it. In fact it becomes more likely the more the range of unique inputs are. But when more than one input deterministically generates the same output, it's called a "collision".

With this in firmly in mind, let's apply it to routing and directed requests to servers. Let's say you have 5 servers to allocate loads across. An easy to understand method would be to hash incoming requests (maybe by IP address, or some client detail), and then generate hashes for each request. Then you apply the modulo operator to that hash, where the right operand is the number of servers.

For example, this is what your load balancers' pseudo code could look like:

request#1 => hashes to 34
request#2 => hashes to 23
request#3 => hashes to 30
request#4 => hashes to 14

// You have 5 servers => [Server A, Server B ,Server C ,Server D ,Server E]

// so modulo 5 for each request...

request#1 => hashes to 34 => 34 % 5 = 4 => send this request to servers[4] => Server E

request#2 => hashes to 23 => 23 % 5 = 3 => send this request to servers[3] => Server D

request#3 => hashes to 30 => 30 % 5 = 0 => send this request to  servers[0] => Server A

request#4 => hashes to 14 => 14 % 5 = 4 => send this request to servers[4] => Server E

As you can see, the hashing function generates a spread of possible values, and when the modulo operator is applied it brings out a smaller range of numbers that map to the server number.

You will definitely get different requests that map to the same server, and that's fine, as long as there is "uniformity" in the overall allocation to all the servers.

Adding Servers, and Handling Failing Servers

So - what happens if one of the servers that we are sending traffic to dies? The hashing function (refer to the pseudo code snippet above) still thinks there are 5 servers, and the mod operator generates a range from 0-4. But we only have 4 servers now that one has failed, and we are still sending it traffic. Oops.

Inversely, we could add a sixth server but that would never get any traffic because our mod operator is 5, and it will never yield a number that would include the newly added 6th server. Double oops.

// Let's add a 6th server
servers => [Server A, Server B ,Server C ,Server D ,Server E, Server F]

// let's change the modulo operand to 6
request#1 => hashes to 34 => 34 % 6 = 4 => send this request to servers[4] => Server E

request#2 => hashes to 23 => 23 % 6 = 5 => send this request to servers[5] => Server F

request#3 => hashes to 30 => 30 % 6 = 0 => send this request to  servers[0] => Server A

request#4 => hashes to 14 => 14 % 6 = 2 => send this request to servers[2] => Server C

We note that the server number after applying the mod changes (though, in this example, not for request#1 and request#3 - but that is just because in this specific case the numbers worked out that way).

In effect, the result is that half the requests (could be more in other examples!) are now being routed to new servers altogether, and we lose the benefits of previously cached data on the servers.

For example, request#4 used to go to Server E, but now goes to Server C. All the cached data relating to request#4 sitting on Server E is of no use since the request is now going to Server C. You can calculate a similar problem for where one of your servers dies, but the mod function keeps sending it requests.

It sounds minor in this tiny system. But on a very large scale system this is a poor outcome. #SystemDesignFail.

So clearly, a simple hashing-to-allocate system does not scale or handle failures well.

A popular solution - consistent hashing

Unfortunately this is the part where I feel word descriptions will not be enough. Consistent hashing is best understood visually. But the purpose of this post so far is to give you an intuition around the problem, what it is, why it arises, and what the shortcomings in a basic solution might be. Keep that firmly in mind.

The key problem with naive hashing, as we discussed, is that when (A) a server fails, traffic still gets routed to it, and (B) you add a new server, the allocations can get substantially changed, thus losing the benefits of previous caches.

There are two very important things to keep in mind when digging into consistent hashing:

1. Consistent hashing does not eliminate the problems, especially B. But it does reduce the problems a lot. At first you might wonder what the big deal is in consistent hashing, as the underlying downside still exists - yes, but to a much smaller extent, and that itself is a valuable improvement in very large scale systems.

2. Consistent hashing applies a hash function to incoming requests and the servers. The resulting outputs therefore fall in a set range (continuum) of values. This detail is very important.

Please keep these in mind as you watch the below recommended video that explains consistent hashing, as otherwise its benefits may not be obvious.

I strongly recommend this video as it embeds these principles without burdening you with too much detail.

If you're having a little trouble really understanding why this strategy is important in load balancing, I suggest you take a break, then return to the load balancing section and then re-read this again. It's not uncommon for all this to feel very abstract unless you've directly encountered the problem in your work!

Section 8: Databases

We briefly considered that there are different types of storage solutions (databases) designed to suit a number of different use-cases, and some are more specialized for certain tasks than others. At a very high level though, databases can be categorized into two types: Relational and Non-Relational.

Relational Databases

A relational database is one that has strictly enforced relationships between things stored in the database. These relationships are typically made possible by requiring the database to represented each such thing (called the "entity") as a structured table - with zero or more rows ("records", "entries") and and one or more columns ("attributes, "fields").

By forcing such a structure on an entity, we can ensure that each item/entry/record has the right data to go with it. It makes for better consistency and the ability to make tight relationships between the entities.

You can see this structure in the table recording "Baby" (entity) data below. Each record ("entry) in the table has 4 fields, which represent data relating to that baby. This is a classic relational database structure (and a formalized entity structure is called a schema).

source: https://web.stanford.edu/class/cs101/table-1-data.html

So the key feature to understand about relational databases is that they are highly structured, and impose structure on all the entities. This structure in enforced by ensuring that data added to the table conforms to that structure. Adding a height field to the table when its schema doesn't allow for it will not be permitted.

Most relational databases support a database querying language called SQL - Structured Query Language. This is a language specifically designed to interact with the contents of a structured (relational) database. The two concepts are quite tightly coupled, so much so that people often referred to a relational database as a "SQL database" (and sometimes pronounced as "sequel" database).

In general, it is considered that SQL (relational) databases support more complex queries (combining different fields and filters and conditions) than non-relational databases. The database itself handles these queries and sends back matching results.

Many people who are SQL database fans argue that without that function, you would have to fetch all the data and then have the server or the client load that data "in memory" and apply the filtering conditions - which is OK for small sets of data but for a large, complex dataset, with millions of records and rows, that would badly affect performance. However, this is not always the case, as we will see when we learn about NoSQL databases.

A common and much-loved example of a relational database is the PostgreSQL (often called "Postgres") database.

ACID

ACID transactions are a set of features that describe the transactions that a good relational database will support. ACID = "Atomic, Consistent, Isolation, Durable". A transaction is an interaction with a database, typically read or write operations.

Atomicity requires that when a single transaction comprises of more than one operation, then the database must guarantee that if one operation fails the entire transaction (all operations) also fail. It's "all or nothing". That way if the transaction succeeds, then on completion you know that all the sub-operations completed successfully, and if an operation fails, then you know that all the operations that went with it failed.

For example if a single transaction involved reading from two tables and writing to three, then if any one of those individual operations fails the entire transaction fails. This means that none of those individual operations should complete. You would not want even 1 out of the 3 write transactions to work - that would "dirty" the data in your databases!

Consistency requires that each transaction in a database is valid according to the database's defined rules, and when the database changes state (some information has changed), such change is valid and does not corrupt the data. Each transaction moves the database from one valid state to another valid state. Consistency can be thought of as the following: every "read" operation receives the most recent "write" operation results.

Isolation means that you can "concurrently" (at the same time) run multiple transactions on a database, but the database will end up with a state that looks as though each operation had been run serially ( in a sequence, like a queue of operations). I personally think "Isolation" is not a very descriptive term for the concept, but I guess ACCD is less easy to say than ACID...

Durability is the promise that once the data is stored in the database, it will remain so. It will be "persistent" - stored on disk and not in "memory".

Non-relational databases

In contrast, a non-relational database has a less rigid, or, put another way, a more flexible structure to its data. The data typically is presented as "key-value" pairs. A simple way of representing this would be as an array (list) of "key-value" pair objects, for example:

// baby names
[
    { 
        name: "Jacob",
        rank: ##,
        gender: "M",
        year: ####
    },
    { 
        name: "Isabella",
        rank: ##,
        gender: "F",
        year: ####
    },
    {
      //...
    },

    // ...
]

Non relational databases are also referred to as "NoSQL" databases, and offer benefits when you do not want or need to have consistently structured data.

Similar to the ACID properties, NoSQL database properties are sometimes referred to as BASE:

Basically Available which states that the system guarantees availability

Soft State means the state of the system may change over time, even without input

Eventual Consistency states that the system will become consistent over a (very short) period of time unless other inputs are received.

Since, at their core, these databases hold data in a hash-table-like structure, they are extremely fast, simple and easy to use, and are perfect for use cases like caching, environment variables, configuration files and session state etc. This flexibility makes them perfect for using in memory (e.g. Memcached) and also in persistent storage (e.g. DynamoDb).

There are other "JSON-like" databases called document databases like the well-loved MongoDb, and at the core these are also "key-value" stores.

Database Indexing

This is a complicated topic so I will simply skim the surface for the purpose of giving you a high level overview of what you need for systems design interviews.

Imagine a database table with 100 million rows. This table is used mainly to look up one or two values in each record. To retrieve the values for a specific row you would need to iterate over the table. If it's the very last record that would take a long time!

Indexing is a way of short cutting to the record that has matching values more efficiently than going through each row. Indexes are typically a data structure that is added to the database that is designed to facilitate fast searching of the database for those specific attributes (fields).

So if the census bureau has 120 million records with names and ages, and you most often need to retrieve lists of people belonging to an age group, then you would index that database on the age attribute.

Indexing is core to relational databases and is also widely offered on non-relational databases. The benefits of indexing are thus available in theory for both types of databases, and this is hugely beneficial to optimise lookup times.

Replication and Sharding

While these may sound like things out of a bio-terrorism movie, you're more likely to hear them everyday in the context of database scaling.

Replication means to duplicate (make copies of, replicate) your database. You may remember that when we discussed availability.

We had considered the benefits of having redundancy in a system to maintain high availability. Replication ensures redundancy in the database if one goes down. But it also raises the question of how to synchronize data across the replicas, since they're meant to have the same data. Replication on write and update operations to a database can happen synchronously (at the same time as the changes to the main database) or asynchronously .

The acceptable time interval between synchronising the main and a replica database really depends on your needs - if you really need state between the two databases to be consistent then the replication needs to be rapid. You also want to ensure that if the write operation to the replica fails, the write operation to the main database also fails (atomicity).

But what do you do when you've got so much data that simply replicating it may solve availability issues but does not solve throughput and latency issues (speed)?

At this point you may want to consider "chunking down" your data, into "shards". Some people also call this partitioning your data (which is different from partitioning your hard drive!).

Sharding data breaks your huge database into smaller databases. You can work out how you want to shard your data depending on its structure. It could be as simple as every 5 million rows are saved in a different shard, or go for other strategies that best fit your data, needs and locations served.

Section 9: Leader Election

Let's move back to servers again for a slightly more advanced topic. We already understand the principle of Availability, and how redundancy is one way to increase availability. We have also walked through some practical considerations when handling the routing of requests to clusters of redundant servers.

But sometimes, with this kind of setup where multiple servers are doing much the same thing, there can arise situations where you need only one server to take the lead.

For example, you want to ensure that only one server is given the responsibility for updating some third party API because multiple updates from different servers could cause issues or run up costs on the third-party's side.

In this case you need to choose that primary server to delegate this update responsibility to. That process is called leader election.

When multiple servers are in a cluster to provide redundancy, they could, amongst themselves, be configured to have one and only one leader. They would also detect when that leader server has failed, and appoint another one to take its place.

The principle is very simple, but the devil is in the details. The really tricky part is ensuring that the servers are "in sync" in terms of their data, state and operations.

There is always the risk that certain outages could result in one or two servers being disconnected from the others, for example. In that case, engineers end up using some of the underlying ideas that are used in blockchain to derive consensus values for the cluster of servers.

In other words, a consensus algorithm is used to give all the servers an "agreed on" value that they can all rely on in their logic when identifying which server is the leader.

Leader Election is commonly implemented with software like etcd, which is a store of key-value pairs that offers both high availability and strong consistency (which is valuable and an unusual combination) by using Leader Election itself and using a consensus algorithm.

So engineers can rely on etcd's own leader election architecture to produce leader election in their systems. This is done by storing in a service like etcd, a key-value pair that represents the current leader.

Since etcd is highly available and strongly consistent, that key-value pair can always be relied on by your system to contain the final "source of truth" server in your cluster is the current elected leader.

Section 10: Polling, Streaming, Sockets

In the modern age of continuous updates, push notifications, streaming content and real-time data, it is important to grasp the basic principles that underpin these technologies. To have data in your application updated regularly or instantly requires the use of one of the two following approaches.

Polling

This one is simple. If you look at the wikipedia entry you may find it a bit intense. So instead take a look at its dictionary meaning, especially in the context of computer science. Keep that simple fundamental in mind.

Polling is simply having your client check on a server by sending it a network request and asking for updated data. These requests are typically made at regular intervals like 5 seconds, 15 seconds, 1 minute or any other interval required by your use case.

Polling every few seconds is still not quite the same as real-time, and also comes with the following downsides, especially if you have a million plus simultaneous users:

• almost-constant network requests (not great for the client)

• almost constant inbound requests (not great for the server loads - 1 million+ requests per second!)

So polling rapidly is not really efficient or performant, and polling is best used in circumstances when small gaps in data updates is not a problem for your application.

For example, if you built an Uber clone, you may have the driver-side app send driver location data every 5 seconds, and your rider-side app poll for the driver's location every 5 seconds.

Streaming

Streaming solves the constant polling problem. If constantly hitting the server is necessary, then it's better to use something called web-sockets.

This is a network communication protocol that is designed to work over TCP. It opens a two-way dedicated channel (socket) between a client and server, kind of like an open hotline between two endpoints.

Unlike the usual TCP/IP communication, these sockets are "long-lived" so that its a single request to the server that opens up this hotline for the two-way transfer of data, rather than multiple separate requests. By long-lived, we meant that the socket connection between the machines will last until either side closes it, or the network drops.

You may remember from our discussion on IP, TCP and HTTP that these operate by sending "packets" of data, for each request-response cycle. Web-sockets mean that there is a single request-response interaction (not a cycle really if you think about it!) and that opens up the channel through which two-data is sent in a "stream".

The big difference with polling and all "regular" IP based communication is that whereas polling has the client making requests to the server for data at regular intervals ("pulling" data), in streaming, the client is "on standby" waiting for the server to "push" some data its way. The server will send out data when it changes, and the client is always listening for that. Hence, if the data change is constant, then it becomes a "stream", which may be better for what the user needs.

For example, while using collaborative coding IDEs, when either user types something, it can show up on the other, and this is done via web-sockets because you want to have real-time collaboration. It would suck if what I typed showed up on your screen after you tried to type the same thing or after 3 minutes of you waiting wondering what I was doing!

Or think of online, multiplayer games - that is a perfect use case for streaming game data between players!

To conclude, the use case determines the choice between polling and streaming. In general, you want to stream if your data is "real-time", and if it's OK to have a lag (as little as 15 seconds is still a lag) then polling may be a good option. But it all depends on how many simultaneous users you have and whether they expect the data to be instantaneous. A commonly used example of a streaming service is Apache Kafka.

Section 11: Endpoint Protection

When you build large scale systems it becomes important to protect your system from too many operations, where such operations are not actually needed to use the system. Now that sounds very abstract. But think of this - how many times have you clicked furiously on a button thinking it's going to make the system more responsive? Imagine if each one of those button clicks pinged a server and the server tried to process them all! If the throughput of the system is low for some reason (say a server was struggling under unusual load) then each of those clicks would have made the system even slower because it has to process them all!

Sometimes it's not even about protecting the system. Sometimes you want to limit the operations because that is part of your service. For example, you may have used free tiers on third-party API services where you're only allowed to make 20 requests per 30 minute interval. if you make 21 or 300 requests in a 30 minute interval, after the first 20, that server will stop processing your requests.

That is called rate-limiting. Using rate-limiting, a server can limit the number of operations attempted by a client in a given window of time. A rate-limit can be calculated on users, requests, times, payloads, or other things. Typically, once the limit is exceeded in a time window, for the rest of that window the server will return an error.

Ok, now you might think that endpoint "protection" is an exaggeration. You're just restricting the users ability to get something out of the endpoint. True, but it is also protection when the user (client) is malicious - like say a bot that is smashing your endpoint. Why would that happen? Because flooding a server with more requests than it can handle is a strategy used by malicious folks to bring down that server, which effectively brings down that service. That's exactly what a Denial of Service (D0S) attack is.

While DoS attacks can be defended against in this way, rate-limiting by itself won't protect you from a sophisticated version of a DoS attack - a distributed DoS. Here distribution simply means that the attack is coming from multiple clients that seem unrelated and there is no real way to identify them as being controlled by the single malicious agent. Other methods need to be used to protect against such coordinated, distributed attacks.

But rate-limiting is useful and popular anyway, for less scary use-cases, like the API restriction one I mentioned. Given how rate-limiting works, since the server has to first check the limit conditions and enforce them if necessary, you need to think about what kind of data structure and database you'd want to use to make those checks super fast, so that you don't slow down processing the request if it's within allowed limits. Also, if you have it in-memory within the server itself, then you need to be able to guarantee that all requests from a given client will come to that server so that it can enforce the limits properly. To handle situations like this it's popular to use a separate Redis service that sits outside the server, but holds the user's details in-memory, and can quickly determine whether a user is within their permitted limits.

Rate limiting can be made as complicated as the rules you want to enforce, but the above section should cover the fundamentals and most common use-cases.

Section 12: Messaging & Pub-Sub

When you design and build large-scale and distributed systems, for that system to work cohesively and smoothly, it is important to exchange information between the components and services that make up the system. But as we have seen before, systems that rely on networks suffer from the same weakness as networks - they are fragile. Networks fail and its not an infrequent occurrence. When networks fail, components in the system are not able to communicate and may degrade the system (best case) or cause the system to fail altogether (worst case). So distributed systems need robust mechanisms to ensure that the communication continues or recovers where it left off, even if there is an "arbitrary partition" (i.e. failure) between components in the system.

Imagine, as an example, that you're booking airline tickets. You get a good price, choose your seats, confirm the booking and you've even paid using your credit card. Now you're waiting for your ticket PDF to arrive in your inbox. You wait, and wait, and it never comes. Somewhere, there was a system failure that didn't get handled or recover properly. A booking system will often connect with airline and pricing APIs to handle the actual flight selection, fare summary, date and time of flight etc. All that gets done while you click through the site's booking UI. But it doesn't have to send you the PDF of the tickets until a few minutes later. Instead the UI can simply confirm that your booking is done, and you can expect the tickets in your inbox shortly. That's a reasonable and common user experience for bookings because the moment of paying and the receipt of the tickets does not have to be simultaneous - the two events can be asynchronous. Such a system would need messaging to ensure that the service (server endpoint) that asynchronously generates the PDF gets notified of a confirmed, paid-for booking, and all the details, and then the PDF can be auto-generated and emailed to you. But if that messaging system fails, the email service would never know about your booking and no ticket would get generated.

Publisher / Subscriber Messaging

This is a very popular paradigm (model) for messaging. The key concept is that publishers 'publish' a message and a subscriber subscribes to messages. To give greater granularity, messages can belong to a certain "topic" which is like a category. These topics are like dedicated "channels" or pipes, where each pipe exclusives handles messages belonging to a specific topic. Subscribers choose which topic they want to subscribe to and get notified of messages in that topic. The advantage of this system is that the publisher and the subscriber can be completely de-coupled - i.e. they don't need to know about each other. The publisher announces, and the subscriber listens for announcements for topics that it is on the lookout for.

A server is often the publisher of messages and there are usually several topics (channels) that get published to. The consumer of a specific topic subscribes to those topics. There is no direct communication between the server (publisher) and the subscriber (could be another server). The only interaction is between publisher and topic, and topic and subscriber.

The messages in the topic are just data that needs to be communicated, and can take on whatever forms you need. So that gives you four players in Pub/Sub: Publisher, Subscriber, Topics and Messages.

Better than a database

So why bother with this? Why not just persist all data to a database and consume it directly from there? Well, you need a system to queue up the messages because each message corresponds to a task that needs to be done based on that message's data. So in our ticketing example, if 100 people make a booking in 35 minutes, putting all that in the database doesn't solve the problem of emailing those 100 people. It just stores a 100 transactions. Pub/Sub systems handle the communication, the task sequencing and the messages get persisted in a database. So the system can offer useful features like "at least once" delivery (messages won't be lost), persistent storage, ordering of messages, "try-again", "re-playability" of messages etc. Without this system, just storing the messages in the database will not help you ensure that the message gets delivered (consumed) and acted upon to successfully complete the task.

Sometimes the same message may get consumed more than once by a subscriber - typically because the network dropped out momentarily, and though the subscriber consumed the message, it didn't let the publisher know. So the publisher will simply re-send it to the subscriber. That's why the guarantee is "at least once" and not "once and only once". This is unavoidable in distributed systems because networks are inherently unreliable. This can raise complications, where the message triggers an operation on the subscriber's side, and that operation could change things in the database (change state in the overall application). What if a single operation gets repeated multiple times, and each time the application's state changes?

Controlling Outcomes - one or many outcomes?

The solution to this new problem is called idempotency - which is a concept that is important but not intuitive to grasp the first few times you examine it. It is a concept that can appear complex (especially if you read the wikipedia entry), so for the current purpose, here is a user-friendly simplification from StackOverflow:

In computing, an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters.

So when a subscriber processes a message two or three times, the overall state of the application is exactly what it was after the message was processed the first time. If, for example, at the end of booking your flight tickets and after you entered your credit card details, you clicked on "Pay Now" three times because the system was slow ... you would not want to pay 3X the ticket price right? You need idempotency to ensure that each click after the first one doesn't make another purchase and charge your credit card more than once. In contrast, you can post an identical comment on your best friend's newsfeed N number of times. They will all show up as separate comments, and apart from being annoying, that's not actually wrong. Another example is offering "claps" on Medium posts - each clap is meant to increment the number of claps, not be one and only one clap. These latter two examples do not require idempotency, but the payment example does.

There are many flavours of messaging systems, and the choice of system is driven by the use-case to be solved for. Often, people will refer to "event based" architecture which means that the system relies on messages about "events" (like paying for tickets) to process operations (like emailing the ticket). The really commonly talked about services are Apache Kafka, RabbitMQ, Google Cloud Pub/Sub, AWS SNS/SQS.

Section 13: Smaller Essentials

Logging

Over time your system will collect a lot of data. Most of this data is extremely useful. It can give you a view of the health of your system, its performance and problems. It can also give you valuable insight into who uses your system, how they use it, how often, which parts get used more or less, and so on.

This data is valuable for analytics, performance optimization and product improvement. It is also extremely valuable for debugging, not just when you log to your console during development, but in actually hunting down bugs in your test and production environments. So logs help in traceability and audits too.

The key trick to remember when logging is to view it as a sequence of consecutive events, which means the data becomes time-series data, and the tools and databases you use should be specifically designed to help work with that kind of data.

Monitoring

This is the next steps after logging. It answers the question of "What do I do with all that logging data?". You monitor and analyze it. You build or use tools and services that parse through that data and present you with dashboards or charts or other ways of making sense of that data in a human-readable way.

By storing the data in a specialized database designed to handle this kind of data (time-series data) you can plug in other tools that are built with that data structure and intention in mind.

Alerting

When you are actively monitoring you should also put a system in place to alert you of significant events. Just like having an alert for stock prices going over a certain ceiling or below a certain threshold, certain metrics that you're watching may warrant an alert being sent if they go too high or too low. Response times (latency) or errors and failures are good ones to set up alerting for if they go above an "acceptable" level.

The key to good logging and monitoring is to ensure your data is fairly consistent over time, as working with inconsistent data could result in missing fields that then break the analytical tools or reduce the benefits of the logging.

Resources

As promised, some useful resources are as follows:

1. A fantastic Github repo full of concepts, diagrams and study prep

2. Tushar Roy's introduction to Systems Design

3. Gaurav Sen's YouTube playlist

4. SQL vs NoSQL

I hope you enjoyed this long-form guide!

If you would like to learn more about my journey from lawyer to software engineer, check out episode 53 of the freeCodeCamp podcast and also Episode 207 of "Lessons from a Quitter". These provide the blueprint for my career change.

If you are interested in teaching yourself to code, changing careers and becoming a professional coder, or becoming your own technical co-founder, please reach out here. You can also check out my free webinar on Career Change to Code if that is what you're dreaming of.

You’ve probably heard people talk about cryptocurrency and encryption algorithms, about the end of "intermediaries" and so on.

It's easy to assume that cryptocurrency (eg: Bitcoin, Ripple, Ethereum, Litecoin, etc.) are the same as blockchain. They're not.

Cryptocurrencies are a clever application of a much cleverer technology – the Blockchain.

In this post, I will cover some of the basic concepts of the blockchain so you understand what it is, how it must be conceptualized, and what can be built on top of it.

But as with all things, they make more sense if you understand why they were invented, before you get into what they do. That context will help you grasp what problem the blockchain was designed to solve.

Why use blockchain?

Great question. So glad you asked. Let's sit back and do a small thought experiment.

What happens if you and your best friend independently and separately conduct the same petition campaign? Let’s say it’s for the “Free the Hamsters” cause.

Let's say you conduct it in an identical sequence across the same suburb, but come up with different sets of signatures on the petition. Which version of the signed petition is the “source of truth”?

You would need to trace back your separate trails, one signature at a time, to locate the last discrepancy. And then you'd have to work further back to identify the first result that diverged between your signature sheets. Prior to that root divergence, all other signatures on the two lists should match up.

You then know that prior to that divergence. Both lists are in accord, so those signatures represent the minimum number of people who signed to support freeing the hamsters.

While that may work well for hamsters and small suburban surveys, it doesn’t work so well in the digital world. Or voting, banking, financial transactions, transferring land title, discharging contractual obligations etc. You need independent and “trusted third parties” to verify a chain of events, and solemnly reassure you that the "chain of custody" was unbroken.

A "chain of custody" can sometimes also be called the "provenance" – they both mean the same thing: the sequence of historical events concerning the data in question.

That’s why you have governments having the final say on your identity, and votes need to be physically counted and recounted by hundreds of volunteers, and clerks in dingy offices maintain ledgers and certificates to confirm whether or not you own your farm/white picket-fenced bungalow.

That is why you need financial intermediaries to ensure that when you buy your collector's item Darth Vader doll, using a credit card, the money (value) is “removed” from your account and “put into” to the seller’s account.

This is technically called the “double spending problem” – how do you ensure that you’re not spending the same money twice? Without someone to do this, you could spend money and at the same time continue to hold on to that money.

So it’s a big problem really – modern life requires that we rely on, trust in and pay for “trusted” third party intermediaries to ensure that value (money) does actually digitally “change hands”. That is why Visa and MasterCard exist, and why PayPal and others link with your bank accounts.

At the heart of the blockchain's why is this problem: how do you know a sequence of events has not been tampered with to alter the current state?

This is where the blockchain fits in. Clear, so far?

How Blockchain works

For the sake of communicating a concept with simplicity, I may take liberties with some of the technical, under-the-hood aspects of this technology. My aim is to get you to understand what it is and have a mental model of how it works. For that, I may need to be a little loose with precision in order to improve the odds of comprehension, especially for non-native English speakers.

It is essential to remember that the blockchain is a technology – mathematically complex software code to be specific. And Bitcoin (or Ethereum or any of the other cryptos on offer) are just applications of that technology.

So the key principles are:

• Blockchains are ‘mined” (produced through the expenditure of effort, like in gold mining) by powerful and resource-hungry computers – called nodes, that are on the same network.

• Chains of digitally encrypted and timestamped records of transactions are lumped into “blocks”, which are maintained on a “ledger” by each node. As transactions are added to a block, and blocks are linked together linearly and chronologically as "chains". Then the entire record/ledger gets synchronised across the network of nodes such that all the block "chains" on the nodes should tell an identical story of the history of any given transaction. Thus we get "block + chain = blockchain". It's a long, complicated linked list.

• Each block in a chain has its own id - a cryptographic hash that is unique and specific to each block. That hash is also stored in the next block in the chain, causing a link. A block can store thousands of transactions and the tiniest change in that block's data would result in a new hash. So if a hash changes but the next block has a different hash, then we know some data in the previous block was tampered with.

• As hundreds become thousands of nodes (and more are added all the time), each node has to “agree” on the history of the blocks/ledger – this is called "critical consensus". One of the ways in which consensus is achieved is through the cryptographic hash we talked about earlier.

• Where there are discrepancies in the ledger (for example the hash of a block doesn't match with the next block's reference to the previous block's hash), the ledger with the longest chain of valid transactions embedded will be the “correct” one – the source of truth. Any nodes working on other (shorter versions) of the chain switch to the longer one. This maintains the critical consensus (this bit is hugely oversimplified, but sufficient for now).

• Any naughty interception or change to one ledger (again, for example, where a block's hash doesn't tally) would immediately create a discrepancy with all the other versions. It would also have a shorter block “history” to corroborate it, which makes that tampered version a suspicious character in the blockchain network where length matters (ahem).

• Replicating that discrepancy across all the versions of the ledger – the entire blockchain network – is such a huge task that it is computationally impractical, and would only happen if the bad guys suddenly had control over the majority of the nodes mining blockchain and changed them all rather rapidly. This sort of coordinated attack on the majority of the nodes on the network is often called the 51% attack.

Interestingly, Satoshi Nakamoto says in the original Bitcoin white paper,

“As such, the verification is reliable as long as honest nodes control the network, but is more vulnerable if the network is overpowered by an attacker.”

However, elsewhere he/she/the organisation (we don't know who "Satoshi" is) calmly points out that to modify past transactions in blocks, across the entire network of nodes, would require the attacker to re-do the chain of custody in those blocks, and all the blocks added after that. Then they'd have to run like mad to catch up with, and surpass, the work of nodes that aren’t under the bad guy’s control (so that they can re-write the ledger, so-to-speak).

And because of this, the “probability of a slower attacker catching up diminishes exponentially as subsequent blocks are added”.

The sheer programmatic complexity, pace and volume of nodal activities make it hard for counterfeiters/attackers to catch up with, let alone outrun, the new blocks mined constantly.

That does make sense. It’s like the lie you tell one family member about why you couldn’t attend their kid’s flute recital. And then you have to madly chase everyone else in the family and ensure you’ve told them all the same lie so that when the original person you’ve lied to brings it up, everyone is aware of this lie and plays along. Sounds exhausting.

To wrap up, the defining characteristic of a blockchain is that it is a distributed ledger across many, many nodes and it is extremely computationally intensive (expensive) to add nodes to that network.

Thus each ledger must be "aware" of all the transactions, and must have an agreed version (which will have the longest “chain of custody” behind it) across the entire network to which the next transaction will be added.

As Satoshi Nakamoto declares in the original Bitcoin white paper, “ The only way to confirm the absence of a transaction is to be aware of all transactions.”

Importantly, the blockchain "dis-intermediates" trust – so we don’t need to pay “trusted third parties” transaction fees for being trustworthy and keeping us, and the counter-parties we deal with, honest. The blockchain programmatically ensures truth (provenance) of the history of transaction in it.

So why should we care?

Well, by getting rid of the need for “trusted intermediaries” any intermediary that charges a modest fee for giving us the gift of certainty needs to find a new job. And that impact banks who traditionally offer such assurance services.

It also means that we can program “smart contracts” between promisor and promisee that automatically recognise (digitally) whether that promise has been delivered or not.

This has enabled a truly tech savvy artist like Imogen Heap to sell her music directly to her listening public, and collect her dues directly from them rather than losing the bulk of earnings to record labels, managers and other “trusted intermediaries”.

It will likely change the way Intellectual Property is protected, accessed, shared, distributed, and developed on the internet.

It could even mean that Uber’s fleet of drivers transact directly with people who want a ride rather than rely on Uber to coordinate and control the flow of information and money.

It may mean that I could directly send you small amounts of money for virtually no fees (micro-transactions). It could mean that the millions of unbanked people in the world who have smartphones can start to transact well beyond their traditional physical-world boundaries.

Wonderfully enough, governments are looking beyond just cryptocurrency when it comes to deploying this technology – to record land ownership, for example.

In effect, we could create a world of true peer-to-peer digital transactions for the transfer of value that is distributed, horizontal, removes the need to rely on trust, and above all requires extraordinary computational power to tamper with. These transactions could be between people, machines and devices.

It could therefore offer a new security paradigm for the protection of data collected by and transferred through the “internet of things”.

I personally believe that the complexity of the modern world is obscured behind intuitive touchscreens. Blockchain technology will quickly become embedded in our technological universe without us being fully aware of it – just like we have been using yeast recombinant DNA for synthetic insulin production since the 1970s.

The changes and cost savings will be sweepingly referred to as technological changes, like that "interweby thing” or some other vague, all-inclusive phrase.

One catch: it'll work as long as we can trust that a "trust-less system" that is coded and engineered by humans (whom we trust?) will further the cause of trust-less-ness in an untrusting and untrustworthy world. You may need to read that sentence several times.

Wrapping up

OK – you should now be reasonably aware of the basics of blockchain. But there is a lot more to learn if you're interested.

You can debate whether blockchain is useful or over-hyped, revolutionary or boring. But it's hard to ignore that it is pretty cool as a concept.

Here is a really fantastic video by Anders Brownworth that explains the whole thing with a mockup blockchain. I strongly recommend you watch it.

And as a learning exercise, you can build your own blockchain right in your browser or your command line. Here's a quick tutorial for how to build your own blockchain.

If you have any comments about this article or think I could have explained parts of this better, Tweet at me at @ZubinPratap

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

But first, let me give you some context. I was writing a bunch of code involving third party APIs (Stripe's recurring billing and subscription APIs, to be specific), and had written a wrapper class and some server route-handlers to respond to requests from the front-end web app. The entire application is React +TypeScript + Node, with a Koa server.

As part of this, I was trying to handle the following errors:

1. Errors thrown by Stripe's API

2. Errors thrown by my wrapper class, especially when fetching user data from the database

3. Errors in route-handlers that arise from a combination of the above.

During development, my most common errors were incomplete data in the server requests and incorrect data passed to Stripe.

To help you visualize the flow of data, let me give you some background on the server-side code. Typically this is what the function call chain looked like:

Route-Handler -> Stripe Wrapper -> Stripe API

The first function being called would be in the Route-Handler, then in the Stripe Wrapper class, inside which the Stripe API method would be called. So the call stack has Route-Handler at the bottom (first called function) and the Stripe API method on the top (last called function).

The problem was that I did not understand where to put my error handling. If I did not put an error handler in the server code, then node would crash (literally, exit execution!) and the front end would receive an error HTTP response (typically a HTTP 5xx err0r). So I put a few try-catch handlers inside the various methods being called, and added logging statements inside the catch block. That way I could debug the error by tracking the logs.

An example of the calling logic:

 function stripeAPI(arg){
    console.log('this is the first function')
    if(!arg) throw new Error('no arg!')
    // else
    saveToDb()
}

function stripeWrapper(){
    console.log('this is the second function, about to call the first function')
    try{
        stripeAPI()
    } catch(err) {
//         console.log(' this error will not bubble up to the first function that triggered the function calls!')
    }
}

function routeHandler(){
    console.log('this is the third  function, about to call the second function')
    stripeWrapper()
}


function callAll(){
    try{
       routeHandler() 
       return 'done'
    } catch (err){
       console.log('error in callAll():', err)
       return ' not done '
    }

}


callAll()

The problems?

1. If I didn't log the error, I lost the error! In the above snippet, note that even though I've called first() without the required arguments, the error defined in the definition of first did not get thrown! Also, there is no saveToDb() method defined... and yet this was not caught! If you run this code above, you will see it returns 'done' - and you've got no idea that your database wasn't updated and something had gone wrong! ☠️☠️☠️

2. My console had way too many logs, repeating the same error. It also meant that in production, there was excessive logging... ?

3. The code looked ugly. Almost as ugly as my console.

4. Others who worked with code found it confusing and a debugging nightmare. ?

None of these are good outcomes, and all are avoidable.

The concepts

So, let's get some basics out of the way. I'm sure you know them but some people may not, and let's not leave them behind!

Some basic terminology:

Error - also known as an 'exception', is when something goes wrong in the node code, and the program exits immediately. Errors, if not handled, will cause the program to come to a screeching halt, and ugly messages are spewed into the console, with a long and generally hideous error stack trace message.

Throw - the throw operator is how the language handles an error. By using throw you generate an exception using the value you put after the operator. Note that the code after throw does not get executed - in that sense it is like a return statement.

Error - there is a JavaScript object called Error. An error gets 'thrown' in order to help the programmer know something needs to be handled. Think of it as a little ticking bomb ? that gets thrown from one function to another inside a chain of function calls. Technically, you can throw any data, including JavaScript primitives as an error, but it's generally a good idea to throw an Error object.

You typically construct the Error object by passing in a message string like so: new Error('This is an error'). But simply creating a new Error? object is unhelpful as that's only half the job. You've got to throw it so it can be caught. That's how it becomes useful.

Languages generally come with a standard set of errors, but you can create a custom error message with the new Error('this is my error message') constructor, and your error message should help you work out what's going on. More on Node errors.

Catch - this is what you do when someone throws something at you, right? You'd probably do it reflexively even if someone threw you one of these... ?!

The catch statement in JavaScript lets you handle an error ? that gets thrown. If you don't catch the error, then the error 'bubbles up' (or down, depending on how you view the call stack) until it reaches the first called function and there it will crash the program.

In my example an error thrown by the Stripe API will bubble up all the way to my Route-Handler function, unless I catch it somewhere along the way and deal with it. If I don't handle the error, Node will throw an uncaughtException error and then terminate the program.

Let's return to my example:

Call stack

Route-Handler -> Stripe Wrapper -> Stripe API

Error path

Stripe API (? _thrown here) -> API Wrapper (� �_not caught) -> _Route-Handler (_� �still not caught) -> ccrraashh ???

We want to avoid app crashes as it can cause your data to corrupt, your state to be inconsistent, and your user to think your app sucks. So handling errors thoughtfully requires many levels of analysis.

There are some detailed guides to error handling in JavaScript and one of my favourites is here, but I will summarize my key leanings for you here.

Try-Catch statement

Use these to gracefully handle errors, but be careful about where and when. When errors are caught and not handled properly they are lost. That 'bubbling up' process happens only up until the error encounters a catch statement. If there is a catch statement in the call chain that intercepts the error then the error won't crash the app, but not handling the error will hide it! Then it gets passed as an argument to catch and it requires you to handle it there.

try{
// code logic
} catch (error) {
// handle the error appropriately
}

So it's very important to catch and handle the error at a point where it makes the most logical sense for you when you have to debug it. It's tempting to think that you must catch it at the very first place it comes up (the last function called that sits right on the top of the call stack), but that isn't true!

Route-Handler -> Stripe Wrapper (don't catch here!) -> Stripe API

If I put my try-catch in the Stripe Wrapper which directly invokes Stripe's API, then I don't have information on where my Stripe Wrapper function was called. Maybe it was the handler, maybe it was another method inside my wrapper, maybe it was in another file altogether! In this simple example it's obviously called by Route-Handler, but in a real world app, it could be called in multiple places.

Instead, it makes sense for me to put the try-catch in the Route-Handler, which is the very first place where the function calls begin that resulted in the error. That way you can trace the call stack (also called unwinding the call stack) and drill down into the error. If I send bad data to Stripe it will throw an error, and that error will pass through my code until I catch it.

But when I catch it I need to handle it properly, or I could inadvertently conceal this error. Handling errors usually means deciding whether I need my front end user to know something has gone wrong (their payment didn't work, for example), or is it just an internal server error (for example, Stripe could not find the product ID I passed) that I need to handle gracefully without tripping up my front end users and crashing the Node code. If I added things to the database that are not correct, then I should clean up those false writes now.

When handling the error, it is a good idea to log it so I can monitor the app for bugs and failures in production and debug efficiently. So at the very, very least, handling would include logging the error in the catch statement. But...

 function stripeAPI(arg){
    console.log('this is the first function')
    if(!arg) throw new Error('no arg!')
    // else
    saveToDb()
}

function stripeWrapper(){
    console.log('this is the second function, about to call the first function')
    try {
        stripeAPI()
    } catch(err) {
        console.log('Oops!  err will not bubble up to the first function that triggered the function calls!')
    }
}

function routeHandler(){
    console.log('this is the third  function, about to call the second function')
    stripeWrapper()
}


function callAll(){
    try {
       routeHandler() 
       return 'done'
    } catch (err){  
       console.log('error in callAll():', err)
       return ' not done '
    }

}


callAll()

...as you can see above, if I catch it and log it in the middle level (my Stripe Wrapper class), it won't reach routeHandler or callAll, and my app will not know something went wrong. callAll still returns done and the only evidence something went wrong was in the log statement: 'Oops! err will not bubble up to to first function that triggered the function calls!'. Had we not put a log statement there the error would have vanished without a trace.

This is 'error hiding' and it makes debugging a pain. If I add a try-catch but don't do anything in the catch statement, I will prevent my program from crashing. But I also end up 'hiding' the problem! It usually leads to inconsistent state - parts of my server code thinks everything is OK, and tells my front end that. But another part of my server code had indicated something was wrong!

In this simple example, it's easy to unravel, but think of deeply nested called across your entire application - what a nightmare!

If you absolutely need to handle the error in the middle of your call stack, then be sure to re-throw the error appropriately. That means ending your catch statement with another throw error operation. That way the error will get thrown again and continue to 'bubble up' towards the first function (bottom of the call stack) that triggered the call chain where it can be properly handled again.

Here's what it looks like, adding just one small re-throw in the stripeWrapper() function. Run the code and see the difference in outcome because callAll() now gets passed the error!

function stripeWrapper(){
    console.log('this is the second function, about to call the first function')
    try{
        stripeAPI()
    } catch(err) {
        console.log('Oops!  err will not bubble up to to first function that triggered the function calls!')

        throw err  // add this to re-throw!

    }
}

function callAll(){
    try{
       routeHandler() 
       return 'done'
    } catch (err){  // catches the re-thrown error and prints it to console!
       console.log('error in callAll():', err)
       return ' not done '
    }

}

Since you threw the error in the middle stage, it went to the outer boundary, and got caught there. The code returns not done and you can investigate why the error says 'no arg'. You can also then see that it never executed saveToDb(), as the error threw before that code could be executed! That could be a good thing in cases where you're saving things to the database assuming that there were no errors until that point. Imagine saving things to the database that should never have been saved - that's dirty data in the database now! ???

So, don't do what I did in my early days of programming and simply log the error at every step in the call stack and re-throw it. It just means you will get multiple logs for each error as it passes through the call stack! Only intercept the error at a place where you can most efficiently and usefully handle it, ideally once in a given chain of calls.

In general, it really helps if you place your try catch statement at the outermost (first calling) function that lies at the bottom of the call stack. You can identify this as the place the error will bubble up to just before throwing an uncaughtException error. That's a good place to catch, log, and handle it.

To see the difference in handling when you don't use the try-catch simply modify callAll() to look like this:

function callAll(){
    routeHandler()  

    // this won't run!
    console.log('This function is not contained inside a try-catch, so will crash the node program.')
}

callAll()

You'll note that the console.log statement never runs here because the program crashes when routeHandler() finishes executing.

Rules of Thumb ???

So let's summarize some quick rules that will cover 90+% of your needs:

1. Do not litter your code with try-catch statements

2. Try as much as possible to catch only once in a given chain of function calls

3. Try and place that catch at the outermost boundary - the first function that starts the chain of function calls (bottom of the call stack)

4. Do not leave your catch statement empty as a way to stop your program from crashing! If you don't handle it, chances are it will lead to inconsistent state between your front end and back end. This can be dangerous and lead to a horrible user experience ?!

5. Do not use a catch statement only in the middle of the call stack, and not at the outer boundary. This will cause the error to get 'hidden' in the middle of your code where it isn't going to help you debug or manage data properly. Others who work with your code will find where you live and cut off your internet connection.

6. Catch it where you need to know, and where you can meaningfully do all the things necessary to clean things up.

Stripe API (? thrown here) -> API Wrapper (? passing through) -> Route-Handler (? caught, handled, logged) -> ???

Thanks for reading!

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

(Banner photo by Thomas Smith on Unsplash)

I learned by doing projects (often freeCodeCamp ones!). But I needed to make sure I didn't lose my hard work, and that others could see the hard work I was putting in, so every project had to go on Github. The more complete projects I had on Github, the easier it would be for recruiters. But the steps required to set up a project, initialise a repo, and sync with Github were really annoying and repetitive, so I decided to solve the problem.

Bad news: this isn't going to be a big, fancy, detailed and technically sexy post. It's going to be very un-sexy.

Good news: you don't need to be a shell scripting god(dess) to do it.

So my typical project setup work flow usually goes like this:

1. Go to my ../projects folder and run mkdir project-of-some-name to create a folder with the name project-of-some-name.

2. cd into that project folder and do git init to initialise a local git repo in there.

3. run touch README.MD to create the README file, open it and add some basic descriptions, including links to the resources / tutorials I was implementing in that project. Save the file.

4. run git add . and then git commit -m ' ...some initial commit message...

5. open a browser, go to Github, login, create a new (remote) repository, copy the url, return to my terminal, make sure I was in the correct project folder project-of-some-name...then run the git scripts needed to set up the remote repo as the 'upstream' repo and connect my local repo to it. Then, finally, I can run a git push and my local commit would get pushed up

6. lie down and take a nap, exhausted from this repetitive process.

Admittedly, this was my process, but I liked to stay organised and always be able to access my projects so I can refer to them.

Since automation is a great way to practice your coding skills, I decided to write a small shell script that automates these horrible and repetitive steps. The script is at the bottom of this post, and be warned - it's not sophisticated or fancy. But it sure gets the job done, and I don't need to log in to Github and fool around will all those steps!

Before you copy the script, you need to know how to run it on your Mac. So below are the steps you need to implement to be able to use the script to automate your setup workflow.

1. I keep my scripts in my root/home folder, in a sub-folder called scripts. I suggest you do the same or similar. To get to the root/home folder, in your terminal type cd ~ because the tilda ( ~ )is the symbol for the home folder. In your Mac Finder app it shows up as the one with a house icon. So all my scripts are stored in ~/scripts

2. This matters because to run a shell script from any directory in the terminal, you have to type out the full path. In my case I have to type out ~/scripts/git-script.sh to run the script. But we're getting ahead of ourselves.

3. copy the code chunk at the bottom of this post and then open a text editor, paste it in and then save it as [filename].sh. The .sh is the extension for shell scripts. Save the file in the directory you want to save it at - again I recommend ~/scripts as the folder to save your scripts in.

4. Navigate to that folder in your terminal. To be safe run ls in the terminal to check that you can see the script is there. If it's not you're in the wrong folder or step 3 didn't successfully complete.

5. Make the shell script executable. To do that you type the following in the terminal: chmod +x <<the-correct-filename.sh>>. This is the unix way to make a shell script "executable". I'm not confident I fully understand what that means, other than it's needed to make any shell scripts you write executable, so don't ask me and I won't lie to you.

6. navigate to your projects folder and make a new folder that you intend to house your project. Effectively, you've got to do this: mkdir - create a project-of-some-name inside the folder where you keep all your projects. So your project will eventually be placed inside my-computer/my-projects/project-of-some-name. cd into this folder and then type pwd to get the full path. Copy that - you will need to paste it shortly. It should look like my-computer/my-projects/project-of-some-name

7. open your terminal again, and then type ~/scripts/<<the-correct-filename.sh>>``. The script runs! You will be guided through some input... The main steps are:

   what do you want to call your Github repo (don't use spaces- 'my-awesome-project' is good. Don't use 'my awesome project' as the repo name.

   Enter a description that shows up in the Github repo's description. For this it's safe to use spaces.

Enter the project path you got in step 6, the one that you get after typing pwd in the terminal and getting something like my-computer/my-projects/project-of-some-name

enter your Github username (not email address) and then your Github password. Be careful as you type as these values don't show up on the screen.

....that's it. The script will set up your git repo locally inside my-computer/my-projects/project-of-some-name and then create a README.MD (blank) and then commit it locally, then set up a remote repo in Github (log you in via API) etc and then push everything up!

finally, you will see that the terminal you were interacting with has changed the currently active directory to your project folder. It will now be at my-computer/my-projects/project-of-some-name and you can type in ls and see the README.MD file. If you then type git status you will see your local repo's status (the state of your local project) and if you type in git remote it will show you the Github url for your project!

Done! Happy Coding!

Annnd.....finally......here is the script! I've commented each step so you can reason your way through it.

# Make executable with chmod +x <<filename.sh>>

CURRENTDIR=${pwd}

# step 1: name of the remote repo. Enter a SINGLE WORD ..or...separate with hyphens
echo "What name do you want to give your remote repo?"
read REPO_NAME

echo "Enter a repo description: "
read DESCRIPTION


# step 2:  the local project folder path
echo "what is the absolute path to your local project directory?"
read PROJECT_PATH

echo "What is your github username?"
read USERNAME

# step 3 : go to path 
cd "$PROJECT_PATH"


# step 4: initialise the repo locally, create blank README, add and commit
git init
touch README.MD
git add README.MD
git commit -m 'initial commit -setup with .sh script'


# step 5 use github API to log the user in
curl -u ${USERNAME} https://api.github.com/user/repos -d "{\"name\": \"${REPO_NAME}\", \"description\": \"${DESCRIPTION}\"}"

#  step 6 add the remote github repo to local repo and push
git remote add origin https://github.com/${USERNAME}/${REPO_NAME}.git
git push --set-upstream origin master

# step 7 change to your project's root directory.
cd "$PROJECT_PATH"

echo "Done. Go to https://github.com/$USERNAME/$REPO_NAME to see." 
echo " *** You're now in your project root. ***"

Thanks for reading!

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

Make yourself compelling by removing all their objections.

At the end of the day, when you're interviewing for roles, your job is to make yourself seem like the obvious choice. You do this by anticipating and removing any objections that the hiring manager may have to hiring you - so you need to have all the abilities of the other applicants, and then communicate your own abilities on top.

Here are the strategies that I've personally observed and directly used in my own careers over the years. I encourage you to think deeply about them and incorporate them into your approach.

This is part 2 of 2. In this part I will cover strategies that make you more compelling if you're worried about how your previous career will look when you interview for dev roles.

In part 1 I covered the big advantages that having one or more previous careers offers when you switch to a career in dev.

Experience helps you focus

The more experience you have, the more likely you are to have clarity into your own temperament, interests, skills and preferences. We tend to ignore or overlook these things when looking for work. I understand that if you're desperate for a job and running out of options, then you do not have the luxury of being picky. But many of us have not been targeted because not because of urgency but because we assume a scatter-gun approach is better. It's not. Being targeted increases your success rate, your effectiveness and your overall job satisfaction.

Also, being able to be clear on why you want that specific role goes a very long way in convincing the hiring manager that you're the right person for the job. As a hiring manager I only interview people with the required technical skills. I choose on the basis of their fitness for the role, which includes the clarity the candidate has about the role itself, and why they think they would enjoy the role. Use your experience to be clear on why you want the job, and let this clarity shine through in your cover letters, interviews and conversations.

Experience helps you play to your strengths

Everyone has strengths and weaknesses, and successful invest more in developing their strengths than weaknesses. This is because you need to be outstanding only in a few things to be an outstanding contributor. So you might as well develop things that you're already good at, and aim to be outstanding.

Having previous experience in other roles helps career changers understand their strengths, contextualise why those strengths matter in a target role, and articulate those reasons in a way that makes sense to the hiring manager. By mapping your strengths to a role and the organisation's needs you automatically show that you've put a great deal more thought into how you can help the organisation achieve its goals. After all they're hiring you not so you can have a job, but so that the company can achieve its goals. Show them that you have strengths that will help them do that, and you're already ahead of the competition who just talk about their strengths without mapping them to the organisation's goals.

Tell a better story

The benefit of having multiple perspectives on business and roles is that you become a better story teller. By story I don't mean cooking up things - I mean telling _your_s story better. Weaving a narrative on your choices is a powerful opportunity - career changers can be mysterious and confusing to others, because most people don't have the interest, courage or freedom to change careers and sometimes this causes confusion as to whether you're serious or just flighty. If you have a powerful story, you can excite and inspire the hiring manager, and show that you're different, courageous, focused, dedicated, and willing to take chances on yourself. These are very desirable qualities in an employee. You want to show how there is a clear and compelling logic to your decisions even if it doesn't look like that from the outside looking in. The way to do that is by telling powerful stories that show how your unusual experience, mindset, interests and strengths combined in unique ways to bring you to the present day. A good story also shows self-awareness which is fast becoming a critical skill in the workplace, and in modern teams.

Demonstrate business acumen

As you prepare for interviews for dev roles, use your previous experience to guide your research into the role and the target company. Understand its business context, not just the engineering set-up. Then show how the role will achieve the broader business goals. No one hires a developer just so they can have software. The software is is a part of the product, and the product is part of business strategy, and the strategy is part of the business goals, and the goals are a function of perceived and actual market needs, and the market's needs are linked to the economy and so on. See the big picture, understand where you fit in a smaller picture, and show how you understand the importance of your work for the business. That will set you miles ahead of developers who only talk about CI/CD as an essential part of the modern software development workflow without understanding why the company should care. If you can explain why the company should care about what you do, you're more likely to get better funding too.

Experience makes you a better team player

While this is true for all experience, having diverse experiences shows that you're adaptable. Being adaptable is critical in being able to fit in, and work effectively with, teams. Being a better team player solves a huge risk for employers - a lot of productivity is lost when people don't gel well, or collaborate well. And this is not just true within the functional team, but across functions throughout the organisation. Career changes have a naturally broader intuition about the expanded nature of "teams" in the modern workplace. Communicate this, and you will set the interviewer's mind at ease.

Experience makes you understand the hiring manager's needs

By having experienced different roles and contexts before, you actually know a lot more than you realise. You already have a sense of what makes people effective in their roles - from your own direct performance and those of others in your team. Now all you got to do is ask yourself why those skills or behaviours mattered to your previous bosses. You will start to see a pattern, and this pattern is the clue to understanding what your interviewer is probably going to look for, perhaps subconsciously. Showing that you have maturity and experience to see this makes you stand out as a leader, and clued-in candidate. You get it. Show that you "get it" and you will stand out among the applicants.

On practicing with rejection

No one likes to be rejected. But no matter what you do, the odds are that you will be rejected more often than you succeed. And that's OK because you only need one acceptance at a time, and all rejections fade away. But the learnings from those rejections are hugely valuable - provided you take the time to investigate the reasons, understand and internalise them, and clearly commit to making small or big changes necessary to ensure you grow from the rejection. You should never be rejected for the same reason twice, or you're just wasting your energy doing the same thing , and expecting different results. To get better, turn each rejection into a lesson, and that way you will accumulate the skills to be an excellent interviewee. Often less capable candidates beat the others to get a role because they understand what the hiring manager is looking for, and communicate what matters.

So good luck, and remember to use your previous experience as an advantage!

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

Looking back I see how I've grown exponentially with each career change. But here's the unexpected professional advantage : each new 'job' benefited enormously from my previous experience, even though most people (especially non-career changers) thought it was a bad idea. As it turns out all the people who had strong views about it had never done it themselves.

Career changers have some very significant advantages that are often overlooked, and almost always underplayed

Right after I decided to try and land a dev job late last year, I also decided to document the strategies and plans that actually worked for me when I taught myself to code - by publishing a course on Udemy targeted at people (especially those that long to get into a tech role) who want to teach themselves this vital skill of the new world. As part of that process, and based on feedback from a number of colleagues and recruiters for dev roles, I realised that us career changers have some very significant advantages that are often overlooked, and almost always underplayed.

This is part 1 of 2. In this part I will cover the big advantages of having one or more previous careers offers when you switch to a career in dev. In part 2 I cover strategies that make you more compelling if you're worried about how your previous career will look when you interview for dev roles.

Experience

OK, so this is a no-brainer. But a shallow analysis of this will only mean that you don't actually leverage this asset properly. Why is experience in an unrelated, or semi-related field of any use? Most people would tell you that you need direct and "relevant" experience. And that is definitely the case when it comes to satisfying the specifics of the Job Description. But anyone that has actually reflected on their role will acknowledge that the real experience of the job is often quite different from the sanitized words in a JD. The difference is the same difference as a colouring in book before you colour it in and after. So here are the main reasons why experience really does matter, even if it is an unrelated field (potentially even more so then as it gives you depth and breadth of life experience).

1. Maturity. Dealing with teams teaches you soft skills, hard skills, and subconscious skills at dealing with personalities, temperaments, cultures, habits and mindsets.

2. Insight. other roles gives you insight into other functions in an organisation. For example, if you're implementing a billing system you will have a better awareness of critical aspects of the user's experience if you understand how a sales person uses billing data to manage the sales pipelines and increase customer acquisition or reduce churn. If you were previously in sales or marketing or had your own startup, you would really have valuable skills when it comes to designing the system and selecting SaaS products.

3. Context. Have worked in another team, in another role, in another organisation also helps you contextualise the drivers, habits, behaviours and motivations that influence how teams interact. Product, engineering, marketing, finance, they all have drivers and unique pressures and motives. Truly understanding the context of your co-workers and collaborators outside your narrow function makes you hugely valuable because it helps you cooperate better, which in turn encourages others to cooperate with you. No matter how skilled a dev you are, if you annoy the heck out of the folks who are driving revenue, you will struggle to be taken seriously, and that will impact the quality of your work life.

4. Cultural adaptiveness. The ability to understand context, have insight, and deal with coworkers maturely adds up to an overall skills that I call your cultural adaptability. This just isn't about different ethnic, regional or anthropological cultures. This is about team-cultures too, the unique dynamics that an organisation's culture produces in its teams. Being adaptive in this way makes you highly effective, and effective team players are more valuable as employees than talented but ineffective members. Being great at your role is much, much more than pure technical skill or intellectual horse power.

5. Communication. While a lot is written about improving communication in the engineering culture, the fact is that relatively poor communicators can still be highly effective if they have a broadened vocabulary in the context of their business. You may not have to know or care what Net Present Value or EBITDA is. But understanding these things will make you a better inter-team communicator. In exactly the same way that it helps if your finance person understands what a firewall, or API call is. Knowing some of the technical language spoken by other team members is flattering to them, useful to you, and beneficial for the organisation. And since these may be very boring to you (or not), the best way to pick up the vernacular is to have been in or close to those functions in a previous life, so you know not just what they mean but why they're important to your colleague.

6. Prioritization. Effective team members prioritise in a way that benefits the team and also the larger organisation. If you have previous experience in other domains, you will understand how something that is a low priority to you has an outsized impact on another function, and vice versa. This sensitivity to the co-worker's perspective and professional pressures is a huge asset when it comes to building rapport, trust, camaraderies, and very important - organisational influence.

7. Organisational Dynamics. This is the name for its healthy form. It's unhealthy twin is known as sh!tty politics. But it's a fact of life. You do not and should not have to participate. But it really helps to be able to detect it, foresee it, recognise it, sidestep it, and handle it. On rainy days it always helps to see a car speeding towards that puddle next to you, so you can avoid being splashed, right? In all its forms, the ability to navigate and deal constructively with Organisational Dynamics is a tremendous skill as it reduces stress, improves productivity, builds trust and credibility and delivers successful outcomes for the teams involved. If you've been only in one career or domain all your life you will become adept at recognising its forms in your specific function and in your context. But recognising it when it happens elsewhere but is coming soon to a colleague near you, is a huge strategic advantage as you can anticipate and prepare for it appropriately.

Thanks for reading!

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

I made a ton of mistakes and even quit a several times. So much so that I published a course on Udemy about how not to quit coding.

At the end of a long, painful, and often tedious journey, I attained my goals. And as it happened, even found work, first freelance, then as a developer at a small, growing startup.

I never intended to become a professional dev, and it was pure luck that the curriculum I designed for myself happened to give me a huge advantage (hence the Udemy course).

Like almost everyone else who teaches themselves to code, I started with a lot of self-doubt. I started timidly - trying it out in tiny bites. The easiest ways to get started were with sites like Codecademy. I then discovered freeCodeCamp, and Courseera. And from there on I kept learning the theory and fundamentals of programming.

I did several of the courses twice, as I quickly forgot what I "learned". This turns out to be normal - like any new skill (think of walking, or driving) it takes a lot of practice and time. So I grew comfortable with not being able to memorise. I focused instead on understanding. I developed my knowledge. These resources (always used for free) gave me a sense of confidence in my understanding of the fundamentals. Thanks to these resources, I overcame my doubt, started to enjoy it, and felt like I was making progress.

There was one problem. I couldn't build much. Or rather, I was easily overwhelmed when I tried to to build anything more than a basic, static HTML site. I rarely could go much beyond what I learned on these sites. I was not yet able to take on "real" projects. I was not ready to solve code related problems on my own. I didn't know enough to even Google for answers correctly or efficiently.

So, I turned to other resources, and I will return to them in a moment. Before I tell you what I did that resulted in me getting 4 job offers, let me tell you why I happened to get those offers...

It wasn't just because I had the knowledge. That is easily established without an interview - most firms give you a coding test anyway. If you don't know enough to make it past that, you're not going to get the interview, let alone an offer.

As one interviewer put it: "We don't hire for knowledge. We hire for skill." And that seems perfectly logical and intuitively obvious - there is an important difference between knowledge and skill. Skill is applied knowledge.

Here's the catch : how do you showcase your skills? A certification from online courses showcases knowledge. It does not demonstrate ability. Managers expect you to know the theory and syntax. They expect you to write a for loop easily, and to be able to read a switch-case statement and know what it does. But what they want is for you to be able to put that knowledge to work, and build things with it. You're hired to get a job done, and that's really what you need to prove you can do.

Which brings me to the nasty shock many people who have "learned to code" face when they struggle to find work. They're shocked that all their learning and effort does not get them much interest in the job market. Think of it this way - the finest department stores still need to showcase their wares in the shop windows right? Empty shop windows look bare and uninviting. They do not inspire confidence.

A shop window that has certificates of quality, and standards but is bare of goods is not going to bring in customers, no matter how good the goods.

So think about the resources you've used to learn and ask yourself this : what have you got to show for it? If it's nothing more tangible than a certificate, then consider what a potential hiring manager is going to assess you on, and how are you going to stand out from others?

See what I mean?

So this hiring manager said I was the least "experienced" candidate who'd applied, but I was very compelling. Why? I had (without meaning to) built a solid showcase. And they didn't need to look at my CV.

So now it's time to tell you what I did. I took a number of courses that were long and intense, but which were project-based. Because they were long and intensive, and project-based I had to use Github to store my code as I went along. And it turned out that my Github profile was what impressed the hiring managers - they loved that I had a lot of projects there.

It turns out that most applicants have weak Github profiles.

And that's an opportunity for you. Anyone can make claims in a CV. It's not terribly hard to make it to the end of a course and get a certificate. But you cannot argue with projects sitting in your Github repo. Especially if they're documented with a decent README file and installation instructions (this bit I didn't do so well ?).

Personally, I found that Udemy was the best for this - I did some great courses from Stephen Grider, Colt Steele, Andrew Mead etc. All of them had 20 -30 hours of video (which, if you code along and experiment, will take you 50+ hours). It's easiest to follow along if you've already done the basics on Freecodecamp, Codecademy etc. After a year or so of this, I had, through slow and patient persistence, made it through a bunch of courses, and accidentally built up a pretty good bunch of projects. I was even familiar and skilled enough to work on full stack projects that I dreamed up, and tackle things like Google Actions, Alexa, AR etc that were not related to the courses I'd taken. But importantly, I now had a body of work that demonstrated my ability and commitment to code. I had tangible evidence that any hiring manager could rely on, in addition to me passing the tests they gave me.

So, if I had to summarise the best approach to teaching yourself to code if you want a job as a dev, it would be this:

Projects, not certificates, will teach you you enough to do it professionally and showcase your skills and knowledge.

Keep in mind this post is targeted at new, beginner devs. If you want to see a talk version of this content, check out this video below from juniordev.io's Melbourne chapter. It's a fantastic group and I encourage you to think about participating in similar meetups in your city, to build your own understanding of conditions in your local market.

Good luck, and don't be afraid to go the extra mile. That's where the successful people are.

I really, truly believe your most precious resources are your time, effort and money. Of these, the single most important resource is time, because the other two can be renewed and recovered. So if you’re going to spend time on something make sure it gets you closer to this goal.

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

When we are inspired to set goals, what inspires us are the outcomes not the process. We are never eager to do 50 crunches at the gym — we are eager to walk around with flat abs.

Our wonderful human imagination makes us fall in love with the destination and overlook the path.

This is a good thing. Without this, none of us would start. But the flip side is that because of this most of us quit, or lose momentum. When the reality of the path to our dream is experienced, day in and day out, it’s very easy to slow down and eventually stop. Maybe we don’t want the dream after all. Or, more specifically, we want the result, not the work.

Right?

Sadly, this natural human habit is what stands in the way of many us learning new skills, especially complex skills like coding. Setting outcome goals is what drives people to action, but without process goals, the outcome is seriously at risk.

This is how I set my coding goals the first time I tried to learn to code: I am going to learn HTML this month. Then next month I’m going to do JavaScript.

Sound familiar? Exciting outcome, no process.

That is actually a risky, ineffective way to set your coding goals. Here’s why.

In your head that target date seems reasonable, achievable, and importantly, encouraging. It’s motivating to think you will have results that soon. The problem is that a deadline-based goal like that is arbitrary. Learning a new skill takes a certain amount of time. Some are faster than others, but everyone has to put a minimum amount of time that they need to acquire that skill. The skill doesn’t care which month you want to finish by. It only demands that you put in enough time to learn it.

Therefore, setting a deadline-based outcome goal is incomplete — it ignores some vital math about learning.

For example, if you want to learn HTML and CSS in 30 days (outcome), it is absolutely achievable. But**,** what does that actually mean? How many hours, on average, does a person need to learn HTML and CSS? And then, how many hours a day or week (process) do you need to achieve that goal? If it takes 8 hours a day, 5 days a week, are you in a position to do that? Which metric really decides success: the outcome or the process? (hint: it’s not outcome).

Simply put: knowing how much time it takes to achieve a goal and how much time you can commit to regularly will decide when you you can expect to achieve that goal by.

It’s like a time and distance navigation problem — if you need to travel 100 kilometres, you cannot know when you will get to your destination unless you know how fast you’re traveling. No matter how sincere you are, you cannot do 100 kilometres in 1 hour if you’re walking. Setting that sort of goal is absurd and will result in frustration if you fooled yourself into thinking it’s possible.

So when you set your coding goals, work out the distance between where you are and the skill level you want, how much time will it take in total, and most importantly how much time you can guarantee you will devote to it every week. Divide total time by weekly time and you get the number of weeks it’s going to take you. That is your deadline.

Most learning resources online make this easy. freeCodeCamp.org makes it super easy — the curriculum tells you clearly how much time a certification takes! Sites like Udacity and Coursera also tell you this, but it’s buried under other course data. Typically they tell you you can complete a course in X weeks if you spend a minimum of Y hours a week.

Even on Udemy this can be calculated. For example, my course is about 3 hours long. You can finish it in a day, if you spend 4 hours (including thinking and assignments). If you spend an hour a day you cannot finish it in less than 3 days. For more technical courses, where you learn by coding along, I would multiply the hours of video by at least 3 to get an estimate of time required to complete the course.

Put this way, it’s pretty clear that deadline-based outcome goals for learning are arbitrary if you don’t focus on the process goals of daily or weekly commitment. “When by” is useless without knowing “how much every day”.

And this one shift in approach will greatly improve your odds of success. This way you will know if something is going to take you 6 or 12 or 18 months. No nasty surprises — and you can build a solid plan around that, and prepare yourself for the journey.

Ultimately, every place is walking distance if you have the time. It’s no different for learning new skills. It takes the time it takes. You just got to be ready to show up and do the work until the math gets you to your destination.

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

Looking for a technical co-founder? I was too. For many years. It was a difficult journey, because the prevailing “wisdom” is you need to go out and find a technical co-founder because all the successful startups had them (not true, by the way). But what happens when you're at the end of your runway, and your choice is to learn to code, or quit?

Technical co-founders are supposed to give you the stability, the complement of essential skills, and the accountability that are not possible being a solo founder. Of course, no one is tracking the countless examples where having a rubbish co-founder made your journey immeasurably harder, or thwarted your ability to progress altogether. But of course, what “they” mean is that you don’t need just a technical co-founder, you need the right technical co-founder, duh!

Well, of course.

None of this is particularly helpful advice. It’s like saying you need to be awake to have good ideas (seems obvious and intuitive, but not always true).

My experience as a non-technical (solo) founder

Here is what I’ve personally experienced in the many years I spent looking for technical co-founders:

1. I spent a ton of time browsing forums, LinkedIn, and contact lists looking for people who met the minimum criteria

2. I spent a great deal of time not being able to validate much, test much, progress much because I didn’t have anything to validate, test or progress other than a well-honed pitch

3. I met a lot of people, and most were not interested in entrepreneurship, or didn’t have the necessary work ethic (aka masochistic streak)

4. I met a lot of people who were interested but for all the wrong motivations (get rich quick, glory, fame…)

5. I met a few people who had the right motivations and (as far as I could tell) the right skills, but who did not have the mentality to withstand the brutality of starting up

6. I met a very few people who had experienced starting up and had the skills, but none of them were enthused by my concepts (statistical inevitability)

I knew diddly squat about software, though I wanted to start a technology company. So, here are my mistakes from back then:

1. I had no clue about the fundamental, most basic aspects of software and its design

2. I grossly underestimated the complexity involved (didn’t know how much I didn’t know)

3. I grossly underestimated time involved

4. I grossly overestimated the people I approached to be technical co-founders

5. I significantly (but not knowingly) overstated my role in the initial period — “hustling” and “business development” were my skills, and I did not appreciate that some of the most successful startup companies spent a third of their time on those things, and most of their time building the product and responding to customer needs

For close to 4 years I told myself “I do not need to learn to code. My talents are better used elsewhere”. Sound familiar?

It’s only partly true. As someone with very limited resources, my talents needed to be used on whatever gave me the best chance at succeeding. I had some spare cash I could spend on developers. I had some time I could spend managing them, and that time was mainly created by cutting down on social activity, sleep and disallowing myself weekends. I had valuable experience I could use in putting together a business plan. I had strong social skills and communication skills I could use to pitch to prospects, and also to potential co-founders.

I did all these things, and inched closer to my goals. But it took far too much time. Of course progress is always slow, certainly slower than we would like. But we only slow ourselves down further by not looking at the situation objectively. Even when I did have co-founders (who eventually quit because it was too hard or their life circumstances changed) I found that managing their work ethic, expectations and moods took a great deal of my time and energy. That’s ok — but no one budgets for that.

You see, as aspiring founders, our greatest enemy is anything that causes us to lose time. With every week that passes without results you’re more likely to quit. And we never truly know what our time-cost is when we choose a course of action. And we never know when we are victims of the sunk-cost fallacy.

Looking back, it cost me 4 years and quite a lot of money. And at the end of that, the only way to start again was to repeat that expenditure of time, effort and money, doing the same thing — putting together a plan, and then desperately looking for a technical co-founder.

Here we go again…

A simple time-math

In 2014 I read a blog by Sam Altman, the president of YCombinator. In it, Sam says some of the most profound truths I’ve ever disregarded. Here is the tweet that I dug up for fun.

The first 2 or three times I read his piece I made really sensible sounding arguments as to why it did not apply to me. I was wrong and it cost me money, but worse, it cost me a lot of time (I made back the money).

He basically posits that it’s faster (much faster) to learn to program enough to build your prototype than it is to find a reliable, dependable co-founder who is a good fit, and will go the distance. Not just faster, but the odds of progress are vastly higher.

It’s obvious. Finding a good co-founder, technical or otherwise, is a long shot — like finding the right partner for life — and requires some degree of luck. Learning to code a bit is faster, needs no luck and therefore has a higher success rate.

In fact you can stop reading this blog right here if you like. Read his. It’s better. The only reason I’m writing mine is to share direct, personal experiences that confirm what he said. It’s telling that to date his blog has only had ~8,500 views — of which a dozen are mine. That is much less than the number of aspiring non-technical founders out there.

A dating analogy

In high school, I remember being told that if you’re desperate for the affections of someone, you will act in ways that compromise yourself — your standards, your values and your best interests. You will settle for people, behaviours and situations that aren’t right for you.

It’s exactly the same with looking for co-founders. As time went on and my fear and desperation increased, I found myself compromising — reducing my standards. Negotiating against myself. Making excuses for others. Settling.

Over time I made bad decisions and bad compromises. Fortunately, none of those bad decisions resulted in actual co-founding relationships.

My point is that I was prepared to make bad compromises, just to make progress. That is a bad start to something that you may have to spend the next 10–20 years of your life on.

The technical stuff doesn’t end at launch

It’s tempting to be tactical and say I just need to get to launch. That is not a sustainable plan. There is a difference between planning to “make it up when I get to that bridge” and having to do that because life left you no choice.

I learned the hard way that my need for technical help grew after launch. I thought the hard yards were getting to launch. Boy, was I wrong. Things break. Bugs emerge. Features don’t work as expected. Users have strong views on things. Iteration is the way to achieve product-market fit. And it has to be rapid, well coordinated and systematic. Data helps, and lot of valuable data comes post-launch!

That is why paying for developers isn’t sustainable unless you got lots of funding. And you’re not likely to get lots of funding before you even have a product. It’s possible, but not for most founders.

So what are you going to do when 4 weeks after launch things are breaking, users are reporting unexpected bugs, and the server crashed, or the app store has changed some policy? You spend more money. And beg the developers to hurry up. Meanwhile you’re doing your darnedest to find users, pitch, sell, etc.

You’re spending your time on things, for sure, and they’re important. But given a choice between fixing a bug / adding a features your users are clamoring for, and pitching your business plan to a potential seed funder, the best use of your time is product, not the pitch. And you cannot do it because you don’t know how. So you exert yourself on things of second-order consequence because you cannot exert yourself on things of paramount importance.

Developing technical empathy

As I mentioned on the podcast, I was (mortifyingly) one of those people who insisted that “it is a simple, quick prototype”. I totally, utterly, woefully, lacked any concept of what the development process is like.

Quincy, the founder of freeCodeCamp and the one who runs the podcast, summed it up very well:

It gives you empathy for the developer experience, and helps you make meaningful time-estimates, not only in terms of what is possible, but also what is straightforward, and what is complex. [paraphrased]

Imagine if someone who has no clue about your work comes to you and insists that something that takes a week should take 2 days — wouldn’t you want to knock them on the head, and just turn away in disgust?

I’m seriously embarrassed by all the times I did this (insist that it’s a simple app, not knock someone on the head).

Worse still, why would they take me seriously? Had I really showed them respect and commitment by at least trying to learn a little bit of their craft? From their point of view, I was hiding behind my skills and the reasonable excuse that coding is “not the best use of my time”.

Here is another sinister side effect of not being knowledgeable enough on the technical stuff. I could never evaluate the relative skills of the people I spoke to. I had to go on faith, trust or recommendations. I had no way to assess their fitness for the very purpose I needed them to fulfill.

Looking back, I could have saved myself a ton of money and months of effort, while building a skill that extends my runway almost indefinitely — had I learned to code earlier.

As Sam Altman says:

“When people like this say “I’ll do whatever it takes to make this business successful” (which they almost always say), I say something like “Why not learn to hack?”

Why not indeed. Do whatever it takes. Especially if it helps your startup “not die”.

Engineering is not the be-all-end-all

Not for one moment do I think coding is the answer to everything. If you are among those who have an interested and totally reliable technical co-founder, classmate, colleague, sibling etc, then yes it’s not the best use of your time — why? Because you’ve got a great resource in that other person. Then learning to code is duplicating skillsets.

But when you don’t have that skillset, learning a little of it is the best use of your time, if it saves you a lot of time in the long run. Here is the math I apply:

priority = probability of outcome in a given unit of time

So:

Find a co-founder in 6 months and start build in 7th month : 50% probability

Learn enough code in 6 months and start build in 7th month : 90%

This entire article would be totally obvious if it said that coders need to learn marketing and communication skills to pitch. Coders need to get out of the building and talk to their customers and not just code. This is now considered “obvious” advice.

So why isn’t the reverse just as obvious?

Give yourself credibility

Engineers are like the really good looking girl at the bar. They get “hit on” all the time. They get approached all the time. I don’t know directly, but I’m guessing that gets tired fast, and cynicism is just another “you’re going to love my startup idea” away.

You know what’s refreshing to someone you’re chatting up at a bar? Interest and awareness about them. The same goes for coders. If you’re aware enough of their world, and interested enough in the detail of their skills, they will respond and, at the very least, help.

This bit I do know from personal experience. Ever since I’ve learned to code I have many more engineers happy to give me advice, guide me, correct me, and even dive into my ideas with me. It’s not easier to find the right co-founder, but that has nothing to do with expertise, and more to do with their interests, priorities and life-circumstances.

And now what?

And now, for the first time in my life, I’m in a position where I can experiment with my ideas. Earlier it cost me time and money. Now it costs me a little time, and even then less time than finding developers, negotiating scope, supervising work, reviewing work, testing work. And that time is an investment as I keep improving the skill even if the idea pans out as commercially unviable.

I’m not a great coder. I don’t think I need to be (maybe in 5 years I will revise this view). But I know enough to build my own prototypes, and understand what is involved in building a viable product. And I know enough to take a call on which bits to outsource, how to describe what I want, not get taken for a ride, assess the output, and team up with other hackers to get results. I may never be a professional developer, and that’s fine. That is not what this is about.

But I have become my own technical co-founder. There may come a day when the best use of my time really is the non-technical stuff. But that day will come once I’ve built something that’s growing. I believe I’ve increased my chances of finding that something simply because I can run many more cheap, low-stress, experiments that do not involve me spending money or begging others for help.

All this in less than 12 months. Think about it. Maybe it really is the best use of your time if you want to be a founder.

I really, truly believe your most precious resources are your time, effort and money. Of these, the single most important resource is time, because the other two can be renewed and recovered. So if you’re going to spend time on something make sure it gets you closer to this goal.

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

Google Home Devices that run the Google Assistant

Hello, world!

Problem statement

I’ve been getting into Actions on Google quite a bit of late — enjoying myself thoroughly — My learning has been primarily designing small “party tricks” that I can amuse visitors with. I’ve been doing the same with Alexa too, but since I’m a lot more familiar with the Google Cloud Platform and Firebase especially, I have prototyped more on Google.

Actions and working with the Google Assistant often requires server side logic to handle the interaction with the desired level of personalisation or customization. This is called “fulfillment”.

The development cycle for fulfillment can be a bit tedious because you need to push your server code to the cloud each time, to see if it’s working. Every time we make changes that we want to test, we need to push code to Firebase and give it 30–60 seconds to propagate. It’s like web dev before hot reloads — ugh! So 2011.

Summary of the Architecture

I won’t go into the details of building Actions for the Assistant, as there is plenty out there to help you with that. But a high level overview of the system now will help you understand the problem statement that inspired this blog.

Here is a diagram from Google that explains how Actions/ Assistant interactions are processed.

A user’s interaction gets picked up by the device, converted to text that is parsed by the device, and passed to a DialogFlow agent which is a NLUengine. That then works out the “intent” of the user’s statement (“phrase” or “utterance”). That intent then gets matched to code that “fulfills” the intent of the user, then returns a response that is (hopefully) appropriate and acceptable.

If you have studied the picture, you will see that a request is made in the last column, which denotes Firebase. What it really denotes is a back-end server that has a HTTP webhook that receives ‘requests’ for fulfillment and then works out how to respond appropriately. Architecturally, the Dialogflow NLU agent sends a POST request to the server webhook, which then gets processed by the server using SDKs.

The back end server that processes requests and responses is easily hosted on Firebase Cloud Functions (of course, Google makes it easy to use their stack — that’s just good business!).

Also, a locally run server will generate a webhook endpoint like http://localhost:3000, which is not accessible to Google Assistant’s POST requests.

Solution Statement — local fulfillment dev server!

There are three steps to this solution:

1. Create the back-end fulfillment server as an Express server.

2. Handle Dialogflow requests and responses as an Express endpoint. The default Firebase Cloud Functions server users the Actions on Google Nodejs Client Library which has built-in functionality to receive HTTP POST requests from the Dialogflow agent. But that won’t work outside the Firebase environment (i.e. on our local development server). So we need to deploy our fulfillment server an an Express app.

3. use the Ngrok Node package to create a temporary HTTP endpoint that tunnels HTTP requests through to our local:3000 (or whatever port you use) server.

I will not go into the steps of setting up your basic code for a simple nodejs fulfillment server — the Dialogflow/ Actions on Google docs help you with all that. But I do provide here the snippets that show you how to convert that Dialogflow App into an Express App, and what your package.json needs to look like.

Let’s start with the basic setup for the Dialogflow node.js client. We will call it the Dialogflow App.

const {
  dialogflow,
  BasicCard
} = require("actions-on-google");


// Instantiate the Dialogflow client.
const app = dialogflow({ debug: true });


// Handlers go here..
app.intent("Default Welcome Intent", conv => {
   // handler for this intent
});

app.intent("Say_Something_Silly", conv => {
   // handler for this intent
});


module.exports = app;

This is the app that ‘handles’ the intents. The above code is just scaffolding. Export the app, and import it in functions/index.js.

index.js is the entry point in our functions folder, which contains the cloud functions we push up to Firebase Cloud Functions. In this file we create the Express App, and import the DialogflowApp object, and then pass it in to the Express route that will receive the HTTP POST requests from Dialogflow. Note that we need the body-parser npm package as the HTTP requests are JSON.

"use strict";

const express = require("express");
const bodyParser = require("body-parser");
const functions = require("firebase-functions");


// clients
const dialogFlowApp = require("./DialogflowApp");
const expressApp = express().use(bodyParser.json());

// EXPRESS APP fulfillment route (POST). The entire dialogFlowApp object (incl its handlers) is the callback handler for this route.
expressApp.post("/", dialogFlowApp);


//  EXPRESS APP test route (GET)
expressApp.get("/", (req, res) => {
  res.send("CONFIRMED RECEIPT OF GET.");
});



/*
 *   LOCAL NGROK SERVER LOGIC. ENSURE that you "export IS_LOCAL_DEV=true" in terminal prior to start
 */
if (process.env.IS_LOCAL_DEV) {
  const PORT = 8000;
  expressApp.listen(PORT, () =>
    console.log(`*** SERVER RUNNING LOCALLY ON PORT ${PORT} ***`)
  );
} else {
  console.log("*** NOT LOCALLY SERVED - OR - LOCAL ENV VAR NOT SET  ****");
}

//EXPORT either of the following two endpoints:  one express app, one dialogflow app
exports.fulfillmentExpressServer = functions.https.onRequest(expressApp);
exports.dialogflowFirebaseFulfillment = functions.https.onRequest(dialogFlowApp);

The key parts of this code are that we create a POST route that takes, as the handler-callback, our DialogflowApp object. I created a GET route just to make quick browser GET requests to test the endpoint is working. But Dialogflow only uses the POST route.

Note that I’ve made two exports here. One is the Express app and the other is the dialogflow App itself. This creates to Firebase functions with two endpoints which are identified by the property attached to the exportsobject. One endpoint will be <……/fulfillmentExpressServer> and the other will be <……/dialogflowFirebaseFulfillment>.

I can use either of these HTTP endpoints for fulfillment, once I’m done developing locally and have pushed the final code up to Firebase Cloud Functions.

NGROK for the local development server tunneling

There is some funny looking code in there on line 26. On my Mac terminal, I use export IS_LOCAL_DEV=true before I start the server locally. That codeblock on line 26 basically starts the server listening locally, which is notneeded when we push the code up to Cloud Functions — it is for the local server only.

"dependencies": {
    "actions-on-google": "^2.0.0",
    "body-parser": "^1.18.3",
    "express": "^4.16.4",
    "firebase-functions": "^2.2.0"
  },
"devDependencies": {
    "ngrok": "^3.1.1"
  },
"scripts": {
    "lint": "eslint .",
    "serve": "firebase serve --only functions",
    "shell": "firebase experimental:functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log",
    "tunnel": "ngrok http 8000",
    "dev": "nodemon index.js"
  },

Which brings me to the above configuration in package.json. In the scripts property, you can see one called tunnel. Note the port number 8000 there. That can be set to anything you like, but make sure that the code in index.jsthat sets the PORT constant (line 27 in the Gist) is the same port number.

Before you run the code, check the following:

1. You’ve set your environment variable and the code matches it — in my case I used IS_LOCAL_DEV=true

2. Your port numbers match in index.js and the tunnel script

Then you open two terminal windows and run the following:

npm run dev and npm run tunnel

In the terminal that ran the tunnel (ngrok) you will see something like this:

Note the two ‘Forwarding’ addresses. They’re identical, except one is https. Either works. That is your endpoint, assuming your POST route is / and not /<some path>. If you’ve added a path to the POST handler then you need to add that path to the Forwarding address.

Note that every time you run ngrok it generates a new, temporary URL for you. Therefore, you need to update your Dialogflow fulfillment webhook every time you run npm run tunnel.

And there you go. No more pushing every little change in code to Firebase Cloud Functions and waiting a minute or two before testing. Develop on the fly, and with nodemon, your server restarts and you can keep testing as you code!

And if you are a code newbie, and find this overwhelming, that is natural. You’re jumping too far ahead. I made that mistake too, and it cost me too much.

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

_source:[https://www.freepik.com](https://www.freepik.com/free-photos-vectors/school" rel="noopener" target="blank" title=")

When it comes to learning to code, attrition is a huge problem. The majority of people quit. And, as Steve Jobs said, quitting is the ‘rational’ thing to do when something is really, really hard. And if you don’t love it, you’re not going to see it through.

But here is a twist: we love things we are good at. Before we get good, we like things that we are “getting the hang of”. Think back to the last time you were getting the hang of something new, and you will remember that feeling of: ‘Yeah — I can do this! I’m gonna get there!’

Unfortunately, most people do not enjoy their journey to learn to code, even though it becomes one of the most addictive things you will ever do, once you start to get the taste of success. The trick is to start tasting small victories, early, and often.

My story

But I did not know all this 4 years ago. I distinctly remember my first, confused attempts to code in 2014. I didn’t even know what an IDE was. It didn’t help that I was 34, had zero background and didn’t know a single programmer.

I tried to learn to code. Not once. Twice. And both times I gave up. I am a very hard worker, very focused, and hungry for a challenge. And I was defeated both times.

I found the challenge too hard. Until …hang on, I am jumping ahead.

Let me be clear and reiterate — it wasn’t coding that was hard. It was learning to code that was hard. And this is a very critical distinction. It’s not the subject that is hard, it’s the process of learning it.

It took me 3 years of giving up, trying again, needlessly wasting dollars, reading all sorts of seemingly promising things, receiving conflicting advice and trying to heed both, bleary eyed nights and frustrated days.

In late 2017, I launched a little startup product, at huge personal expense. I was afraid that I would not raise funding in time, and my co-founder bowed out. I had hired developers — and that couldn’t last long. One day I browsed through my diary, and I saw a pattern.

I was going about the learning all wrong.

I made a note of all the things I felt defeated me. The list had little to do with coding, but everything to do with my choices, behavior, priorities, and … here is the big one**…my misconceptions**.

Less than a year later, I was doing well at hackathons, holding my own with those that had gone to expensive US Coding Bootcamps, helping other beginners start coding, and finally, doing what I’d dreamt of doing in 2014 — programming almost every day. I had gone past “getting the hang of it” to… “wow…not daunted by it!”.

And the best part? I knew that if I failed at the startup, the next one would be enormously easier because I had learned enough to achieve a key goal: I had become my own “technical cofounder”. I was able to prototype my own ideas.

No more trying to persuade others to “build me a simple prototype”.

I love coding so much, and I can feel myself getting better every day. I love that I have barely scratched the surface — programming is a gift that keeps on giving!

And I also now know that I could have achieved my objective with less than half the effort and expense.

But I am now in danger of forgetting what it was like back in 2014, or even 2016, and what I thought, felt, worried about, and struggled with in those early, early days. And I also now know that I could have achieved my objective with less than half the effort and expense. Humans are so great at compressing decades of information into learnings that others can use — for example, every autobiography is decades of learning in a few hundred pages. Podcasts can give you in an hour what it took people years to learn.

And so I resolved to use my diary notes to create a complete course that helps anyone, anywhere, prepare for the coding journey. There are plenty of courses that teach you how to code. But there is very little information that prepare you with what you need to know before you dive into learning.

I’m super pleased to announce that I’ve launched the course on Udemy! And if you’re a FreeCodeCamp reader, you’ll find in this post a clue on how to get a special promo code!

Who is the course for?

It is for people, especially those who have spent time in other careers, that are wondering if they can learn to code (you can), are interested in what it would take (a lot, but less than you expect if you learn from my mistakes!), have tried and given up (this is the majority!), are confused or overwhelmed by the tsunami of choices and decisions (too much noise, not enough guidance), or have started coding and want to make sure you achieve your outcomes with minimum wasted effort.

Oh, and if you want to save time, effort and money while you learn to code, then this course is for you. Because I lost over 8 months of time (more than 500 hours of time) and wasted over $1,100 in subjects that I did not need for my specific goal.

I designed the course to help you find the shortest path to your specific coding goals**.* Remember that old saying ‘People don’t just buy a drill — they buy the ability to hang the picture up on the wall*’? That is 100% true.

You don’t want to just learn to code. You want to create something with code. You want to solve problems, build a business, achieve a flow state, help others, sell things, learn things.

There is no shortage of drills (learn-to-code resources). There is a shortage of guidance on how to find the drill, where to get it, how to use it on your specific wall, for your specific picture.

And here is something you won’t hear from other courses — I even ask you to consider quitting if your objective can be achieved through a means other than learning to program.

Time is your most precious non-renewable resource.

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

.