We just posted a course on the freeCodeCamp.org YouTube channel that will teach you how to be an exceptional engineering manager. Ankita Kulkarni teaches this course. She is an experienced engineering manager and has mentored hundreds of people to become a better manager.

The transition from a developer to an engineering manager can be daunting, but with the right guidance and understanding, it's a journey worth embarking on. This course offers a unique perspective on what it truly means to be a leader in the tech world, highlighting the differences, challenges, and rewards of this role.

The course delves into various aspects of engineering management, focusing on the evolution from an individual contributor to a leader who inspires and guides a team. It addresses several key areas:

The Essence of Leadership: Discover the fundamental differences between being a software engineering manager and a technical lead, and learn what it takes to be a great leader. This includes understanding the roadmap to successful leadership and the skills necessary to guide a team effectively.

Skill Development for Career Advancement: Uncover the pivotal skills that once mastered, can significantly enhance your career trajectory. This section is particularly insightful for those looking to make a tangible impact in their professional journey.

Communication and Feedback: Learn the art of effective communication and feedback within a team setting. As a manager, your role in shaping the team's dynamics and performance is crucial, and this course provides practical tips on how to achieve this.

Personal Growth and Reflection: Ankita shares her personal experiences, offering a candid look at what she wishes she knew before stepping into the role of an engineering manager. This section is invaluable for those contemplating a similar transition.

What sets this course apart is its holistic approach to engineering management. It's not just about technical skills or project management; it's about understanding the human element in a tech environment. The course emphasizes the importance of empathy, communication, and team dynamics, which are often overlooked but critical for a successful leadership role.

This course is ideal for individual contributors who are curious about engineering leadership, as well as current managers seeking to refine their skills and strategies.

Watch the full course on the freeCodeCamp.org YouTube channel (2-hour watch).

In the fast-paced world of technology, efficient user management is crucial for maintaining a secure and well-organized Linux environment. This article serves as a comprehensive guide to user management in Linux, focusing on the needs of CTechCo, a hypothetical technology company.

By understanding the various aspects of user management, which includes creating, modifying, and deleting user accounts, implementing user authentication. By following user management best practices, CTechCo can ensure the security and productivity of its Linux systems.

Table Of Contents

• What are Users in Linux?
• Types of Users in Linux
• User Account Properties
• How to Create Users
• How to Delete Users
• How to Modify User Accounts
• Password Management
• Group Management
• User Authentication
• Best Practices for User Management in Linux
• Principle of Least Privilege
• User Permissions
• Monitoring and Auditing
• User Training
• Conclusion

What are Users in Linux?

In a Linux system, users refer to individuals or entities that interact with the operating system by logging in and performing various tasks. User management plays a crucial role in ensuring secure access control, resource allocation, and system administration.

A user in Linux is associated with a user account, which consists of several properties defining their identity and privileges within the system. These properties are a username, UID (User ID), GID (Group ID), home directory, default shell, and password.

Each user account possesses these unique properties listed above.

Type of Users in Linux

Linux supports two types of users: system users and regular users.

System users are created by the system during installation and are used to run system services and applications.

Regular users are created by the administrator and can access the system and its resources based on their permissions.

Let's meet CTechCo's diverse workforce, consisting of individuals who interact with the Linux system through user accounts. Meet John, a developer; Lisa, a system administrator; and Sarah, a marketing manager. They each have unique usernames such as "johndoe," "lisasmith," and "sarahsmith," respectively. These usernames act as their identification within the Linux system.

How to Create Users

CTechCo's system administrator, Alex, needs to create user accounts for John, Lisa, and Sarah. Alex initiates the process using the useradd command. For example, to create John's account, Alex executes the command below:

useradd -u 1002 -d /home/john -s /bin/bash john

This command creates John's account with uid (-u) as 1002, the home directory (-d) as /home/john and sets (-s) /bin/bash as his default shell.

Similarly, Alex will create a user account for Lisa and Sarah using same format

Alex can verify the new user account by running the id command:

id john

This will display the user ID and group memberships for the user "john".

uid, gid, and groups information for john user

User Account Properties

Within CTechCo's Linux environment, user accounts possess various properties that define their characteristics and access privileges. Let's explore these properties in the context of our use case.

1. Username: Each user is assigned a unique username that serves as their identifier within the Linux system. For example, John's username is "john".
2. UID (User ID) and GID (Group ID): Every user account is associated with a UID and GID. The UID is a numerical value assigned to the user, while the GID represents the primary group to which the user belongs. For instance, John's UID may be 1002, and his primary group's GID is 1002 as well.
3. Home Directory: Each user has a designated home directory where their personal files and settings reside. John's home directory is /home/john.
4. Default Shell: The default shell determines the command interpreter used when a user logs in. It defines the user's interactive environment. In our case, John's default shell is set to /bin/bash, which is a popular shell in Linux.
5. Password: User accounts require passwords to authenticate and access the system. CTechCo's users, including John, must create strong passwords to ensure security.
6. Group: The group membership determines which system resources the user can access, as well as which users can access the user's files.

Alex could take a look at the users on their Linux by running the cat /etc/passwd command. The users will be displayed in this format:

john:x:1002:1002:,,,:/home/john:/bin/bash

Here's what each of the fields in the format above represents:

• john: This is the username of the user account.
• x: This field contains the encrypted password of the user. It is replaced with an 'x' character to indicate that the password is stored in the /etc/shadow file for security reasons.
• 1002: This is the UID (User ID) of the user account, which is a unique numerical identifier assigned to the user by the system.
• 1002: This is the GID (Group ID) of the user account, which represents the primary group membership of the user.
• ,,,: This is the GECOS field, which stands for "General Electric Comprehensive Operating System". This field is used to store additional information about the user, such as their full name or contact information. In this case, the field is empty, as no additional information was provided while creating the user account.
• /home/john: This is the home directory of the user account, which is the location where the user's files and personal data are stored.
• /bin/bash: This is the default shell for the user account, which is the command interpreter used to process commands entered by the user in the terminal. In this case, the default shell is Bash, which is the most commonly used shell in Linux.

How to Delete Users

Let's assume that Lisa has left CTechCo. To remove her account and associated files, Alex has to utilize the userdel command. For instance, to delete Lisa's account, Alex runs:

sudo userdel lisa

This will delete the user account for lisa, along with their home directory and any files or directories owned by the user.

How to Modify User Accounts

As CTechCo's workforce evolves, the IT team may need to make adjustments to user accounts. Let's explore how they can modify user accounts to accommodate changing needs and permissions.

For example, John (the developer), is assigned additional responsibilities within the company. To reflect this change, the IT team can modify John's account using the usermod command. Let's consider the following scenario:

How to Modify User Groups in Linux

CTechCo creates a new group called development to manage access to development-related resources. To add John to the development group, the following command can be used:

sudo usermod -aG development john

This command adds John to the development group, granting him the necessary access privileges.

How to Change Default Shell in Linux

In a case where John prefers to use a different shell other than the default /bin/bash shell. The IT team can modify his account accordingly. For example, to change John's default shell to /bin/zsh, the following command can be used:

sudo usermod -s /bin/zsh john

This command updates John's account to use the new default shell — /bin/zsh.

You can run the cat /etc/passwd again to see that the shell for john has changed from /bin/bash to /bin/zsh.

Group Management

Effective group management is crucial for controlling access to resources within CTechCo's Linux environment. Let's explore how the IT team can create and manage groups to ensure proper access control.

How to Create a New Group in Linux

To create a new group, such as the marketing group, the following command can be used:

sudo groupadd marketing

The command above creates the marketing group, which can be used to grant specific permissions and access to marketing-related resources.

To view the group you just added, run the command below:

cat /etc/group

This returns all the groups on your Linux machine and when you scroll to the bottom, you can find the most recent groups.

You can also use the command to return a specific group (marketing in our case):

cat /etc/group | grep marketing

How to Assign Users to Groups in Linux

Once a group is created, users can be added to it. For example, to add Sarah (the marketing manager) to the marketing group, the following command can be used:

sudo usermod -aG marketing sarahsmith

This command adds Sarah to the marketing group, enabling her to access the resources associated with that group.

Password Management

Ensuring strong password management practices is essential for maintaining the security of user accounts within CTechCo's Linux environment. Let's explore how the IT team can enforce password policies and manage user passwords effectively.

Setting Password Policies: The IT team can establish password policies to enforce strong passwords, including complexity requirements, password expiration, and account lockouts. These policies can be configured in the /etc/login.defs file.

Changing User Passwords: Users should be encouraged to change their passwords periodically. They can do so using the passwd command. For example, John can change his password with the following command:

sudo passwd john

This command prompts John to enter his current password and then allows him to set a new, secure password.

User Authentication

User authentication is a crucial aspect of user management in Linux, ensuring that only authorized users can access the system. CTechCo can employ various authentication mechanisms to safeguard their Linux environment.

Password-Based Authentication

Password-based authentication is the most common method for user authentication in Linux. When users log in, they provide their username and corresponding password to authenticate their identity.

For example, John logs into the system by entering his username and password at the login prompt. The system then verifies the provided password against the stored password hash associated with John's account.

SSH Key-Based Authentication

Secure Shell (SSH) key-based authentication provides a more secure alternative to password-based authentication. Users generate a public-private key pair, where the public key is stored on the server and the private key is kept securely on the user's device.

With SSH key-based authentication, users like Lisa, a system administrator at CTechCo, can authenticate without entering a password. Instead, the server verifies the user's identity based on the possession of the private key.

To configure SSH key-based authentication for Lisa, the following steps can be taken:

1. Generate an SSH key pair on Lisa's machine using the ssh-keygen command.
2. Copy the public key to the server's /home/lisasmith/.ssh/authorized_keys file.
3. Configure the server to allow SSH key-based authentication.

Best Practices for User Management in Linux

To ensure the security and efficiency of user management in Linux, CTechCo can follow several best practices. These practices minimize security risks and enhance the overall management process.

Principle of Least Privilege

The principle of least privilege (PoLP) is a fundamental concept in user management. It states that users should only be granted the minimum privileges necessary to perform their tasks effectively.

CTechCo can apply the PoLP to limit user access and mitigate the impact of potential security breaches. For example, John is granted administrative privileges using the sudo command only when required for specific tasks. By running the following command, John can execute commands with elevated permissions:

sudo command

User Permissions

CTechCo's IT team can assign appropriate permissions to users and groups to control access to files, directories, and resources. They can use the chmod command to set permissions for files and directories, such as read, write, and execute permissions for the owner, group, and others.

For instance, to grant read and write permissions to the owner and read-only permissions to the group and others, the following command can be used:

chmod 640 filename

To view the permissions for the file, you can use the ls -l command. This will display the file's permissions in the following format:

-rw-r--r-- 1 username username 0 Apr 5 11:24 filename.txt

In the format above, the first three characters represent the file's permissions for the owner of the file.

The second three characters represent the permissions for members of the file's group.

The last three characters represent the permissions for all other users.

In this case, the owner of the file has read and write permissions, while members of the group and all other users only have read permissions.

Monitoring and Auditing

CTechCo can implement monitoring and auditing mechanisms to track user activities and identify potential security breaches. They utilize tools like auditd to collect and analyze system logs, enabling them to detect suspicious activities and take appropriate actions.

For example, the IT team can configure auditd to monitor critical system files and directories, as well as user logins and administrative commands.

Also, to view system logs in Linux, Alex can use the tail command. For example, to view the last 10 lines of the system log file, you can use the following command:

sudo tail /var/log/syslog

User Training

CTechCo recognizes the importance of user training in maintaining a secure Linux environment. They can conduct regular training sessions to educate users about password security, best practices for data handling, and awareness of social engineering attacks.

Additionally, they can encourage users to report any suspicious activities or security incidents promptly, fostering a culture of security awareness and responsibility.

By adhering to these best practices, CTechCo can ensure a robust user management process and minimizes security risks in their Linux environment.

Conclusion

Managing users in a Linux environment is essential for maintaining a secure and organized system. In the context of CTechCo, we have explored various aspects of user management and authentication such as:

• The concept of users in Linux, types and their roles within the system.
• User account properties, such as usernames, UIDs, GIDs, home directories, default shells, and passwords.
• User management tasks, including creating, deleting, and modifying user accounts with the use of commands like useradd, userdel, and usermod.
• How group management works using the groupadd and usermod commands.
• User authentication mechanisms, including password-based authentication and SSH key-based authentication.
• Best practices for user management, such as following the principle of least privilege.
• The use of the sudo command for elevated permissions.
• User permissions and access control configured through the chmod command.
• Monitoring and auditing user activities using tools like auditd.
• User training and awareness programs to promote password security and data handling best practices.

We began by understanding the concept of users in Linux, including their roles and importance within the system. We discussed the different types of users, such as regular users and system users, and their respective account properties, including usernames, UIDs, GIDs, home directories, default shells, and passwords.

Moving on to user management, we covered the process of creating, deleting, and modifying user accounts. We saw how the useradd, userdel, and usermod commands can be used to perform these operations. Additionally, we explored group management, where the groupadd command is used to create groups and the usermod command is utilized to assign users to groups.

User authentication mechanisms were also discussed. We examined password-based authentication, where users provide their username and password for verification. Additionally, we explored the more secure SSH key-based authentication, which relies on public-private key pairs.

We the highlighted some best practices that CTechCo could follow like the principle of least privilege, granting users only the necessary privileges for their tasks. They can utilize the sudo command for elevated permissions when required. User permissions, configured through the chmod command, are implemented to control access to files and directories. Monitoring and auditing mechanisms, such as using the auditd tool, are employed to track user activities and detect potential security breaches. Furthermore, user training and awareness programs can be conducted to educate users about password security, data handling best practices, and social engineering awareness.

By incorporating these best practices, CTechCo's IT team can ensure a secure user management process, minimizing security risks and maintaining a well-structured Linux environment.

As always, I hope you enjoyed the article and learned something new. If you want, you can also follow me on LinkedIn or Twitter.

In it, we'll cover everything from digital security, privacy, serverless platforms, the internet of things (IoT), technology research, and much more at a high level.

If you're a manager or decision maker at your company, someone who is plotting your career or educational trajectory, or just a curious person, then this book is for you.

Here's a link to the physical book if you'd like to grab a copy.

Table of Contents

• Chapter 1: Understanding Digital Security
• Chapter 2: Understanding Digital Privacy
• Chapter 3: Understanding the Cloud
• Chapter 4: Understanding Digital Connectivity
• Chapter 5: Understanding the Business of Technology Research
• Chapter 6: Where Hot Trends Go to Die
• Chapter 7: Compute Platforms
• Chapter 8: Security and Privacy

Chapter 1: Understanding Digital Security

Whatever your connection to technology, security should play a prominent role in the way you think and act.

Technology, after all, amplifies the impact of everything we do with it. The things we say and write using communication technologies can be read and heard by many, many more people than would be possible without. The ability to conveniently connect with people and collaborate on projects of all kinds is much greater.

The tasks we can perform are, through the magic of automation, almost limitless. The scope of information we can instantly access through the simplest and least expensive devices towers far beyond anything the greatest scholars could have hoped to see in a lifetime just a few decades ago.

All that means that criminals and other individuals unconstrained by moral conscience will have yet more powerful tools to compromise the data you create and consume, and steal or damage the property you acquire. So you've got a strong interest in learning how to protect yourself, your property, and that of the people and organizations around you.

This chapter will present a brief overview of what's at stake in the technology security domain. We'll define the kinds of threats we face and discuss the key tools at our disposable for pushing back against those threats.

If you're interested in digging deeper into the topic, my LPI Security Essentials book is entirely devoted to giving you the full picture.

Hacking? What's Hacking?

Defining computer hacking in a way that doesn't anger someone, somewhere, is like talking about politics at work. Be prepared for long, awkward silences and possibly violence.

You see, purists might insist that the term hacking should apply exclusively to individuals focused on forcibly re-purposing computer hardware for non-standard purposes. Others reserve the title for people who bypass authentication controls to break into networks for criminal or political purposes. And how about those who wear the title as a sign of their practical expertise in all things IT? (And then, of course, there are crackers.)

But this is my book, so I'm going to use the term any way I want.

I therefore decree that hacking is all about plans the bad guys have for your digital devices. Specifically, their plans to get in without authorization, get out without being noticed, and (sometimes) take your stuff with them when they leave.

Using the term this way gives us a useful way to organize a discussion of some common and particularly scary threats.

How Hackers Get In

The trick is to find a way through your defences (like passwords, firewalls, and physical barriers). In most cases, passwords probably provide the weakest protection:

• Passwords are often short, use a narrow range of characters, and are easy to guess.
• If a device came with a simple factory default password (like "admin" or "1234") just meant to get you in for the first time, then the odds are pretty good that many users will never get around to trading it in for something better.
• Even strong passwords can be stolen by deceptive phishing email scams ("Click here to login to your bank account..."); social engineering ("Hi, it's Ed from IT. We're having some trouble with your corporate account. Would you mind telling me your password over the phone so I can quickly fix it?"); and keyboard tracking software.

We'll talk more about firewalls later in this chapter. And physical barriers? I think you already know what a locked door looks like. But it's probably worth spending a few moments thinking about other kinds of digital attack.

The big prize is usually getting to your data and making off with copies. But for some, simply destroying the originals can be just as satisfying.

Of course, logging into your devices using stolen passwords is the most straightforward approach. But access can also be achieved by intercepting your data as it travels across an insecure network.

One approach that's commonly used here is known as a man-in-the-middle attack, where data packets can be intercepted in transit and altered without authorized users at either end knowing anything's wrong.

Properly encrypting your network connections (and avoiding unsafe public networks altogether) is an effective protection against this kind of threat. We'll talk more about encryption a bit later.

If the hardware you're using has an undocumented "back door" built in, then you're pretty much toast whatever you do. We'll talk more about back doors later in the book.

But for now, I'll just note that there have been no shortage of factory-supplied laptops, rack servers, and even high-end networking equipment that's been intentionally designed to include serious access vulnerabilities. Be very careful where you purchase your compute devices.

If the attackers find a way into your physical building (sometimes posing as employees of a delivery company), they could quietly plug a tiny listening device into on unused ethernet jack on your network. That'll give them a nice platform to watch and even influence all your activities from the inside.

Protecting your physical infrastructure and carefully monitoring network activity is your best hope against that kind of intrusion.

Even if your home or office is all fortressed up, there's no guarantee that data moving around on mobile devices (like smartphones or laptops) won't find its way into the wrong hands.

And even if you've been careful to use only the best passwords for those devices, the data drives themselves can still be easily mounted as external partitions on a thief's own machine. Once mounted, your files and account information will now be wide open.

The only way to protect your mobile devices from this kind of threat is to encrypt the entire drive using a strong passphrase.

What Hackers Are After

Now that entire economies are run on computers directly connected to public networks, there's money and value to be had through well-planned corporate, academic, or political espionage efforts... and through old fashioned, traditional theft.

Whether the goal is building up a military or commercial competitive advantage, completely destroying the competition, or just getting your hands on "free" money, illegally accessing other people's data has never been easier.

So what are hackers likely to be after?

All the important financial and other sensitive information you'd prefer they didn't have. Including, it should be noted, the kind of information you use to identify yourself to banks, credit card companies, and government agencies.

Once the bad guys have got important data points like your birth date, home address, government-issued ID numbers, and some basic banking details, it's usually not hard to present themselves as though they're you, completely taking over your identity in the process.

Digital attacks can also be used as blackmail to force victims to pay to undo the damage they've done.

That's the objective of most ransomware attacks, where hackers encrypt all the data on a victim's computers and refuse to send the decryption keys needed to restore your rightful access unless you send them lots of money. Such attacks have already effectively brought down critical infrastructure like the IT systems powering hospitals and cities.

The very best defence against ransomware is to have full and tested backups of your critical data and a reliable system for quickly restoring it to your hardware. That way, if you're ever hit with a ransomware attack, you can simply wipe out your existing software and replace it with fresh copies, populated with your backed up data.

But you should also beef up your general security settings to make it harder for ransomware hackers to get into your system in the first place.

When their primary goal is to prevent you or your organization from going about its business, hackers can remain at a safe distance and launch a distributed denial of service (DDoS) attack against your web infrastructure.

Historical DDoS attacks have used massive swarms of thousands of illegally hijacked network-connected devices to transmit crippling numbers of requests against a single target service. When large enough, DDoS attacks have managed to bring down even huge enterprise-scale companies using sophisticated defences for hours at a time.

The site hosting one of my favorite online open source collections was hit hard more than a year ago and still hasn't fully recovered.

What Is Encryption?

If your data is unreadable, there's a lot less bad stuff that unauthorized individuals will be able to do with it. But if it's unreadable, there's probably not a whole lot you'll be able to do with it either.

Wouldn't it be nice if there was some way to present your data as unreadable in every scenario except where there's a legitimate reason?

Well waddaya know? There is, and it's called data encryption.

Encrypting Data in Transit

Encryption algorithms encode information in a way that makes it hard, or even impossible, to be read.

A simple (and ancient) example is symbol replacement, where every letter "a" in a message would be replaced with, say, the letter three positions on in the alphabet (which would be "d"). Every "b" would become "e" and so on. "Hello world" would be "khoor zruog". Then people who come across your encoded message would be unable to understand it at a glance.

Of course, it wouldn't take long for a modern computer (or even a smart 8-year-old) to decode that one.

But some very clever cryptologists have been working hard over most of the past century to produce much more effective algorithms.

There are some significant variations of modern cryptography. But the general idea is that people can apply an encryption algorithm to their data and safely transmit the encrypted copy over insecure networks. Then the recipient can apply a decryption key of some sort to the data, restoring the original version.

Encryption is now widely available for many common activities, including sending and receiving emails. You can similarly ensure that the data you request from a website is the same data that's eventually displayed in your browser by checking the lock icon in your browser's address bar. The icon confirms that the website server employs Transport Layer Security (TLS) encryption.

Over the past few years, the Let's Encrypt project (letsencrypt.org) has encouraged millions of new websites to use encryption by provided free encryption certificates and simple-to-use tools to help server administrators install them.

Encrypting Data at Rest

TLS will protect your data when it's out and about, but what'll keep it safe even when it's relaxing in its comfy storage disk? File and drive encryption, that's what.

All operating systems now offer integrated software for encrypting all or part of a storage disk either at installation time or later. Each time you power up an encrypted disk, you'll be prompted to enter the passphrase you created when you enabled encryption.

The thing is that if you forget your passphrase, you're pretty much permanently locked out of your system and the data is as good as gone forever.

But if you don't encrypt your system then, as we noted earlier, anyone who steals the hardware will have easy and instance access to your private information.

It's a tough world out there, isn't it?

What Does a Firewall Do?

You can think of a firewall as a filter.

Just like, say, a water filter is able to block certain impurities, allowing only clean water through, a firewall can inspect every packet of data coming into or leaving your infrastructure, blocking access where appropriate.

Besides not needing to be replaced every few weeks, the big advantage of a firewall over a water filter is that it can be closely configured to permit and refuse entry to exactly match your security and functional needs, and then updated later should your needs change.

Hardware Firewalls

A hardware firewall is a purpose-built physical networking device that's commonly used within enterprise environments.

Such firewalls are installed at the edge of a private network and set to block potentially dangerous incoming traffic, redirect other traffic to remote destinations, or permit traffic to access thousands of hosts within the local network.

Hardware firewalls are sold by specialized companies like Cisco and Juniper, and also general equipment manufacturers like HP and Dell.

Firewalling appliances tend to be very expensive, often costing many thousands of dollars. They're normally only deployed to manage enterprise infrastructure.

Software Firewalls

A software firewall is an application that runs on a regular PC that can perform just about any function that you'd otherwise expect from a hardware firewall.

There are two important differences:

• Firewall software (like the Linux iptables utility) is often free and, while complicated, enjoys the benefits of vast documentation resources. The software can also be installed any old PC that's just lying around, reducing your overall cost to nearly nothing.
• You won't want to use such a firewall within a busy business environment however, since a regular PC probably won't have the compute power necessary to manage high volumes of network traffic. Nor, in most such cases, will it be reliable enough to provide mission-critical services 24/7.

There's another flavor of software firewall that's used as part of consumer-grade operating systems. Such firewalls allow you to better secure your OS by setting rules for what kind of activities you want to allow. These can be especially useful for mobile devices that frequently move from network to network.

Cloud computing platforms - like Amazon Web Services (AWS) and Microsoft's Azure - provide a firewall-like technology for use with the resources you might deploy within their systems. Firewall policies might exist in objects with names like "security group" or "access control list" that can be applied to whichever resource requires them.

Who Does Security Best?

In the not too distant past, you would often hear IT professionals swearing they would never run their IT operations on infrastructure they didn't physically control.

This was common when referring to outsourcing to third party, offsite companies or to cloud computing platforms.

Whether it was because those administrators didn't trust the reliability and security of compute infrastructure run by strangers, or because regulatory restrictions required that sensitive workloads remained local, the sentiment was widely shared. And it made sense.

But the past is a foreign country. Today, it can be forcefully argued, the most secure and reliable environments can be found in the biggest public cloud providers.

Why? They've got the money and incentive to hire the very best engineers, and the money and incentive to build the very best infrastructure.

Beyond that, cloud providers maintain data centers in political jurisdictions around the world, and go to great lengths to ensure their deployments comply with industry and government standards.

Let me illustrate. Remember the DDoS threat we discussed a bit earlier in the chapter?

Well, back in the summer of 2020, an unnamed organization deploying resources on AWS was hit with a DDoS attack peaking at 2.3 Tbps. That is, each and every second, requests hit that organization's public-facing service with 2.3 terabytes of data.

What does "2.3 terabytes" actually mean?

Well, a megabyte is (approximately) one million bytes of information (a PDF version of this book would probably take up six megabytes or so). A gigabyte is one thousand million bytes of information. A terabyte is one thousand thousand million bytes of information.

That would be the equivalent of around 165,000 PDF books. 2.3 terabytes would be the rough equivalent of 380,000 PDF books.

Now try to imagine all the text characters used to fill 380,000 PDF books being thrown at a web service each second.

Got that image in your mind?

Now here's what happened to that web service: Nothing. It just carried on working as though it hadn't a care in the world.

How on earth is that even possible?

Amazon's AWS Shield service simply mitigated the attack. The customer didn't have to do a thing.

That is why moving your workloads to the public cloud doesn't necessarily involve compromising your standards.

Chapter 2: Understanding Digital Privacy

Public service warning: you might find this chapter a wee bit depressing. If you'd prefer some cheering up right now, perhaps skip to "Understanding the Cloud."

For all the many benefits we enjoy from technology - and particularly the technologies that make up the public internet - there are clearly plenty of costs, too. Figuring out how you want to balance the benefits against the costs can take some careful thinking.

Here's a concise and effective way to describe the equation (whose source I've sadly forgotten):

"Select any two of privacy, security, and convenience. But you can't have all three."

In other words, if security is a critical value for you, then you'll need to give up on 24/7 instant access to your money, credit, and personal accounts. That's because that kind of access requires exposing your accounts across public networks at a level that won't permit as much data protection as you might want.

Similarly, what if you just can't live without the convenience of getting news updates and social connectivity through sites belonging to third party businesses that collect and use your personal information? Well, you'll need to "pay for it" by giving up a measure of your privacy.

Of course, most of us will choose some blend of those three elements based on a practical compromise between competing values and needs.

But making a reasonable decision on that blend will require solid information. That's what you'll find through the rest of this chapter.

How Companies Get Your Data

Curious about what kinds of personal and even private data you may be exposing through the course of a normal day on the internet?

The answer is "all kinds".

Perhaps the best way to understand the scope and nature of the problem is to break it down by platform.

Financial Transactions

Take a moment to visualize what's involved in a simple online credit card transaction.

You probably signed into the merchant's website using your email address as an account identifier and a (hopefully) unique password.

After browsing a few pages, you'll add one or more more items to the site's virtual shopping cart. When you've got everything you need, you'll begin the checkout process, entering shipping information, including a street address and your phone number. You might also enter the account number of the loyalty card the merchant sent you and a coupon code you received in an email marketing message.

Of course, the key step involves entering your payment information which, for a credit card, will probably include the card owner's name and address, and the card's number, expiry date, and a security code.

Assuming the merchant infrastructure is compliant with Payment Card Industry Data Security Standard (PCI-DSS) protocols for handling financial information, then it's relatively unlikely that this information will be stolen and sold by criminals.

But either way, it will still exist within the merchant's own database.

To flesh all this out a bit, understand that using your loyalty card account and coupon code can communicate a lot of information about your shopping and lifestyle preferences, along with records of some of your previous activities.

Your account on the site also includes your contact information, and your home location.

All of that information can, at least in theory, be stitched together to create a robust profile of you as a consumer and citizen.

It's for these reasons that I personally prefer using third-party e-commerce payment systems like PayPal because such transactions leave no record of my specific payment method on the merchant's own databases.

Devices

Modern operating systems are built from the ground up to connect to the internet in multiple ways.

They'll often automatically query online software repositories for patches and updates and "ask" for remote help when something goes wrong. Some performance diagnostics data is sent and stored online, where it can contribute to statistical analysis or bug diagnosis and fixes.

Individual software packages might connect to remote servers independently of the OS to get their own things done.

All that's fine.

Except, you might have a hard time being sure whether all the data coming and going between your device and the internet is stuff you're OK sharing.

Can you know that private files and personal information aren't being swept in with all the other data? And are you confident that none of your data will ever accidentally find its way into some unexpected application lying beyond your control?

To illustrate the problem, I'd refer you to devices powered by digital assistants like Amazon's Alexa (figure 2.1) and the Google Assistant ("OK Google").

Since, by definition, the microphones used by digital assistants are constantly listening for their key word ("Alexa..."), everything anyone says within range of the device is registered.

At least some of those conversations are also recorded and stored online and, as it turns out, some of those have eventually been heard by human beings working for the vendor.

In at least one case, an inadvertently-recorded conversation was used to convict a murder suspect.

Figure 2.1: A device with Amazon's Alexa digital assistant.

Amazon, Google, and other players in this space are aware of the issue and are trying to address it. But it's unlikely they'll ever fully solve it.

Remember, convenience, security, and privacy don't work well together.

Now if you think the information from computers and tablets that can be tracked and recorded is creepy, wait 'till you hear about thermostats and light bulbs.

As more and more household appliances and tools are adopted as part of "smart home" systems, more and more streams of performance data will be generated alongside them.

And, as has already been demonstrated in multiple real-world applications, all that data can be programmatically interpreted to reveal significant information about what's going on in a home and who's doing it.

Mobile Devices

Have you ever stopped in the middle of a journey, pulled out your smartphone, and checked a digital map for directions?

Of course you have.

Well the map application is using your current location information and sending you valuable information but, at the same time, you're sending some equally valuable information back.

What kind of information might that be?

I once read about a mischievous fellow in Germany who borrowed a few dozen smartphones, loaded them up on a kids' wagon, and slowly pulled the wagon down the middle of an empty city street. It wasn't long before Google Maps was reporting a serious traffic jam where there wasn't one.

How does the Google Maps app know more about your local traffic conditions than you do?

One important class of data that feeds their system is obtained through constant monitoring of the location, velocity, and direction of movement of every active Android phone they can reach - including your Android phone.

I, for one, appreciate this service and I don't much mind the way my data is used. But I'm also aware that, one day, that data might be used in ways that sharply conflict with my interests. Call it a calculated risk.

Of course, it's not just GPS-based movement information that Google and Apple - the creators of the two most popular mobile operating systems - are getting. They, along with a few other industry players, are also handling the records of all of our search engine activity and the data returned by exercise and health monitoring applications.

In other words, should they decide to, many tech companies could effortlessly compile profiles describing our precise movements, plans, and health status. And from there, it's not a huge leap to imagine the owners of such data predicting what we're likely to do in the coming weeks and months.

Web Browsers

Most of us use web browsers for most of our daily interactions with the internet. And, all things considered, web browsers are pretty miraculous creations, often acting as an impossibly powerful concierge, bringing us all the riches of humanity without even breaking into a sweat.

But, as I'm sure you can already anticipate, all that power comes with a trade-off.

For just a taste of the information your browser freely shares about you, take a look at the Google Analytics page shown in figure 2.2. This dashboard displays a visual summary describing all the visits to my own bootstrap-it.com site over the previous seven days. I can see:

• Where in the world my visitors are from
• When during the day they tend to visit
• How long they spent on my site
• Which pages they visit
• Which site they left before coming to my site
• How many visitors make repeat visits
• What operating systems they're running
• What device form factor they're using (i.e., desktop, smartphone, or tablet)
• The demographic cohorts they belong to (genders, age groups, income groups)

Figure 2.2: The home dashboard of a Google Analytics page displaying visualizations of visitors to a website.

Besides all that, a web server's own logs can report detailed information, in particular, the specific IP address and precise time associated with each visitor.

What this means is that, whenever your browser connects to my website (or any other website), it's giving my web server an awful lot of information. Google just collects it and presents it to me in a fancy, easy-to-digest format.

By the way, I'm fully aware that, by having Google collect all this information about my website's users, I'm part of the problem. And, for the record, I do feel a bit guilty about it.

In addition, web servers are able to "watch" what you're doing in real time and "remember" what you did on your last visit.

To explain, have you ever noticed how on some sites, right before you click to leave the page a "Wait! Before you go!" message pops up? Servers can track your mouse movements and, when they get "too close" to closing the tab or moving to a different tab, they'll display that popup.

Similarly, many sites save small packets of data on your computer called "cookies." Such a cookie could contain session information that might include the previous contents of a shopping cart or even your authentication status. The goal is to provide a convenient and consistent experience across multiple visits. But such tools can be misused.

Finally, like operating systems, browsers will also silently communicate with the vendor that provides them. Getting usage feedback can help providers stay up to date on security and performance problems. But independent tests have shown that, in many cases, far more data is heading back "home" that would seem appropriate.

Website Interaction

Although some of this might be covered by previous sections in the chapter, I should highlight at least a couple of particularly relevant issues. Like, for instance, the fact that websites love getting you to sign up for extra value services.

The newsletters and product updates that they'll send you might we be perfectly legitimate and, indeed, provide great value. But they're still coming in exchange for some of your private contact information.

As long as you're aware of that, I've done my job.

A perfect example is the data you contribute to social media platforms like Twitter, Facebook, and LinkedIn.

You may think you're just communicating with your connections and followers, but it actually goes much further than that.

Take a marvellous - and scary - piece of software called Recon-ng that's used by network security professionals to test for an organization's digital vulnerability. Once you've configured it with some basics about your organization, Recon-ng will head out to the internet and search for any publicly available information that could be used to penetrate your defences or cause you harm.

For instance, are you sure outsiders can't possibly know enough about the software environment your developers work with to do you any damage?

Well perhaps you should take a look at the "qualifications" section from some of those job ads you posted on LinkedIn. Or how about questions (or answers) your developers might have posted to Stack Overflow?

Every post tells a story, and there's no shortage of clever people out there who love reading stories.

Software like Recon-ng can help you identify potential threats, but that only underlines your responsibility to avoid leaving your data out there in public in the first place.

The bottom line? Smile. You're being watched.

Why Companies Want Your Data

Data is money. Some of the biggest and most successful tech companies of the past decade or two made their billions from data. Generally, that'd be from your data.

Of course, the value doesn't all move in one direction.

Big tech companies do, as a rule, provide useful services. Health tracking apps do track and report on your health. Social media companies do (on rare occasion) provide for healthy social interactions. And historical performance data does sometimes help improve customer and technical service.

But businesses exist to generate revenue and, as a rule, the more data they own, the more revenue it can generate.

The more potential customers there are who provide their email and social media account coordinates, the easier it'll be to connect to them with new offers. And the easier it would be for other companies working in overlapping industries to connect to a business's customers as well. The incentive for you to sell your contact list to an interested third party is obvious.

Naturally, legal restrictions and user agreements can sometimes stop such sales of data sets. But not every use-case is necessarily covered by such laws, and not every company is necessarily bound by a strong desire to follow the law.

A delicious case in point would be Canada's Do Not Call list from all the way back in 2004. The law prevented telemarketers from contacting anyone who had adding themselves to the national list. The law required all telemarketers to remove all entries from the list from their own call lists.

The problem was that spammers happily downloaded the Do Not Call lists and, confident that they represented confirmed active accounts, called those specifically. The only law that was effective in this case was the law of unintended consequences.

Your data can also be useful for personalizing the results you get from search engine queries. Of course, you might sometimes enjoy seeing results relating to previous browsing behavior, but don't lose sight of the fact that your behavior is being used as part of a campaign to sell you stuff.

It's not only search engines: smartphone browsing histories are sometimes used by nearby businesses to push customized ads in your direction - sometimes even through automated digital displays on physical billboards and other signage.

Perhaps the biggest value your data can offer is when it's aggregated along with data generated by thousands or millions of other users. Data scientists can stream and parse huge, dynamic data sets to extract significant insights about subtle but significant trends. In many cases, such data is sanitized to remove any personally identifiable information (PII).

We can nicely sum up the 21st Century web application business model with this popular - and accurate - expression:

"If you're not paying for the product, you are the product."

How to Protect Your Data

All that sounds pretty bleak. After all, George Orwell's 1984 was meant to be a warning, not a how-to guide.

What can you do to push back?

Be aware of your environment.

Do you still even notice those terms of service disclosures you "click to sign" before they'll let you use some service or tool?

Some of those disclosures are as long as this chapter - and, if I may say so myself, a whole lot less fun. But the fact is that they contain information that can have a profound impact on both you and your data.

Many agreements describe what data they're likely to collect and what they're planning to do with it. They'll often also offer assurances that they'll never sell your data to third parties - an assurance that they might sometimes even honor in both the letter and the spirit of the law (although there have been famous cases of companies that did neither).

I've never met anyone who has the time and energy to read through those endless disclosures from end to end. But if an organization pays a bunch of lawyers to write something, you can bet it's a serious business.

Be aware of your rights.

Beyond your specific agreement with a technology service provider, the use of your data might be regulated by government legislation.

One example is the European Union's General Data Protection Regulation (GDPR), which controls how organizations must treat any personal data they encounter in the course of their operations.

Another example is the US government's Health Insurance Portability and Accountability Act (HIPAA), which regulates the handling of private information in the health insurance and healthcare industries.

Be aware of your alternatives.

Consider adopting privacy-first tools instead of the more heavily commercial services you're using now.

For instance, the DuckDuckGo.com search engine, whose home page is shown in figure 2.3, doesn't track your search behavior and will return the same results to a particular query for you as for anyone else.

They are a for-profit business, but they earn much of their income through affiliate links that pay them a commission for sales generated through search links - none of which has any impact on your privacy.

Figure 2.3

The Brave browser, as another example, has been shown to send far less undocumented data out to the internet than any other major browser.

To be specific, in early 2020, Douglas Leith of the School of Computer Science & Statistics, Trinity College Dublin, tested six browsers for their risks of revealing unique identifying information about their host computers (scss.tcd.ie/Doug.Leith/pubs/browser_privacy.pdf). He found that Brave clearly offered the greatest privacy protection.

Brave also blocks web page ads by default, which raises a question. Since many web pages earn income exclusively through display ads, does Brave expect content providers to offer their services for free?

The browser provider actually has a business model that includes the content providers: users of the Brave browser can opt to be shown simple and extremely unobtrusive ads from carefully curated advertisers in exchange for micro payments in a cryptocurrency.

The users can then choose to make micro payments to website content providers using those funds as a way to pay for their content through the Brave Rewards program (pictured in figure 2.4).

Figure 2.4

Opting for open source applications can also be an effective privacy strategy.

OpenStreetMap (openstreetmap.org) is an alternative to Google Maps. It might not have all the bells and whistles and built-in connectivity you may be used to, but it's just that connectivity that powers our reservations, isn't it?

If you're not comfortable with the big mobile operating system players (Android and iOS), you could, instead, buy a phone and install one of a number of experimental mobile Linux variations.

Going down this road will likely be bumpy. Expect to run into unexpected configuration and compatibility challenges, and don't expect to find all the convenient apps that you've come to know and love using the big app stores.

See a hole that needs filling? Why not contribute your own innovation by participating in existing open source projects or adding your own solutions to the community?

Chapter 3: Understanding the Cloud

You may not always be aware of it, but you're enjoying the many fruits of the cloud just about every hour of every day. Many of the joys (and horrors) of modern life would be impossible without it.

Before we talk about what it does and where it's taking us, we should explain exactly what it is.

The "cloud" is all about using other people's computers rather than your own. That's it. No, really.

Cloud providers run lots of compute servers (which are just computers that exist to "serve" applications and data in response to external requests), storage devices, and networking hardware. Whenever the impulse takes you, you can provision units of those servers, devices, and networking capacity for your own workloads. When you add millions more users taken by similar impulses, you get the modern cloud.

For many - although not all - applications, there are enormous cost and performance benefits to be realized by deploying to a cloud. And countless applications - whether small, large or smokin' colossal - have found productive homes on one cloud platform or another.

So let's see how it all works and what you might be able to do with it.

Application Server Deployment Models

Over the decades, we've been through a number of models for running server workloads. In a way, all those changes have been the product of just two technologies:

• Networking protocols that permit communication between connected nodes
• Virtualization which permits fast, efficient, and cost-effective use of hardware resources for multiple and parallel uses

Networking, largely because it's now such a stable and well established technology, isn't something we'll focus on here. But we will get back to virtualization a bit later.

Local Data Centers

In the old days, if you wanted to fire up a new server to perform a compute task, you would spend a week or so calculating how much compute power you'd need for your job. You'd then contact the sales reps at a few hardware vendors, wait for them to get back to you with bid tenders, compare the bids and, when you've selected one, wait another couple of weeks for your new hardware to be delivered. Finally you'd put all the pieces together, plug it all in, and start loading software.

The room where your servers ran would need a reliable and robust power supply and some kind of cooling system: like angry children, servers generate a great deal of heat but don't like being hot. You probably wouldn't want to do any other work in that room, since the noise of your servers' powerful internal cooling fans was difficult to ignore.

While locally-deployed servers gave you all the direct, manual control over your hardware that you could need, it came at a cost.

For one thing, opportunities for infrastructure redundancy (and the reliability that comes with it) were limited. After all, even if you regularly backed up your data (and assuming your backups were reliable), they still wouldn't protect you from a facility-wide incident like a catastrophic fire.

You would also need to manage your own networking, something that could be particularly tricky - and risky - when remote clients required access from beyond your building.

By the way, don't be fooled by my misleading use of past tense here ("were limited," "backed up"). There are still plenty of workloads of all sizes happily spinning away in on-premises data centers.

But the trend is, without question, headed in the other direction.

Server Co-Location

Another option for mid-sized to large organizations is to store your own servers in someone else's data center, an arrangement known as co-location.

The hosting company provides the server racks and power, along with all the networking and cooling equipment you'll need. Whenever you need physical access to your servers, they'll always be happy when you drop by to say hello.

This is a convenient way to maintain direct control over your servers while leaving physical security and the larger infrastructure headaches in the hands of specialists. Co-location facilities are often capable of far higher standards of security and reliability then smaller operations would be able to manage on their own.

For security reasons, co-location centers will probably not advertise their services at street level.

But if you want to see what they look like, search for "server hosting co-location" in your city and then use Google Satellite to check out one or two of the addresses that come back. If you see a large, unmarked building with dozens of powerful air conditioning units on the roof, that'll be a data center.

Virtualization

As I hinted earlier, virtualization is the technology that, more than any other, defines the modern internet and the many services it enables.

At its core, virtualization is a clever software trick that lets you convince an operating system that it's all alone on a bare metal computer when it is, in fact, just one of many OSs sharing a single set of physical resources. A virtual OS will be assigned space on a virtual storage disk, bandwidth through a virtual network interface, and memory from a virtual RAM module.

Here's why that's such a big deal: Suppose the storage disks on your server host have a total capacity of two terabytes and you've got 64GB of RAM. You might need 10GB of storage and 10GB of memory for the host OS (or, hypervisor is what some virtualization hosts are called). That leaves you a lot of room for your virtual operating system instances.

You could easily fire up several virtual instances, each allocated enough resources to get their individual jobs done. When a particular instance is no longer needed, you can shut it down, releasing its resources so they'll instantly be available for other instances performing other tasks.

But the real benefits come from the way virtualization can be so efficient with your resources. One instance could, say, be given RAM and storage that, later, proves insufficient. You can easily allocate more of each from the pool - often without even shutting your instance down. Similarly, you can reduce the allocation for an instance as its needs drop.

This takes all the guesswork out of server planning. You only need to purchase (or rent) generic hardware resources and assign them in incremental units as necessary. There's no longer any need to peer into the distant future as you try to anticipate what you'll be doing in five years. Five minutes is more than enough planning.

Now imagine all this happening on a much larger scale: Suppose you've got many thousands of servers running in a warehouse somewhere that are hosting workloads for thousands of customers.

Perhaps one customer suddenly requests another terabyte of storage space. Even if the disk that customer is currently using is maxed out, you can easily add another terabyte from some other disk, perhaps one plugged in a few hundred meters away on the other side of the warehouse.

The customer will never know the difference, but the change can be virtually instant.

Cattle vs Pets

Server virtualization has changed the way we look at computing and even at software development.

No longer is it so important to build configuration interfaces into your applications that'll allow you to tweak and fix things on the fly. It's often more effective for your developers and sysadmins to build a custom operating system image (nearly always Linux-based) with all the software pre-set. You can then launch new virtual instances based on your image whenever an update is needed.

If something goes wrong or you need to apply an change, you simply create a new image, shut down your instance, and then replace it with an instance running your new image.

Effectively, you're treating your virtual servers the way a dairy farmer treats cows: when the time comes (as it inevitably will), you take a old or sick cow out and kill it, and then bring in another (younger) one to replace it.

Anyone who's ever been involved with legacy server room administration would gasp at such a thought! Our old physical machines would be treated like beloved pets.

At the slightest sign of distress, we'd be standing, concerned, at its side, trying to diagnose what the problem was and how it can be fixed. If all else failed, we'd be forced to reboot the server, hoping against hope that it came back up again. If even that wasn't enough, we'd give in and replace the hardware.

But the modularity we get from virtualization gives us all kinds of new flexibility.

Now that hardware considerations have been largely abstracted out of the way, our main focus is on software (whether entire operating systems or individual applications).

And software, thanks to scripting languages, can be automated. So using orchestration tools like Ansible, Terraform, and Puppet, you can automate the creation, provisioning, and full life cycle management of application service instances.

Even error handling can be built into your orchestration, so your applications could be designed to magically fix their own problems.

Virtual Machines vs Containers

Virtual instances come in two flavors. Virtual machines (or VMs) are complete operating systems that run on top of - but to some degree independent of - the host machine.

This is the kind of virtualization that uses a hypervisor to administrate the access each VM gets to the underlying hardware resources, but such VMs are generally left to live whichever way they choose.

Examples of hypervisor environments include the open source Xen project, VMware ESXi, Oracle's VirtualBox, and Microsoft Hyper-V.

Containers, on the other hand, will share not only hardware, but also their host operating system's software kernel. This makes container instances much faster and more lightweight (since their images don't need to include a kernel).

Not only does this mean that containers can launch nearly instantly, but that their file systems can be transported between hosts and shared. Portability means that instance environments can be reliably reproduced anywhere, making collaboration and automated deployment not only possible, but easy.

Examples of container technologies include LXD and Docker. And enterprise container implementations include Google's open source Kubernetes orchestration system.

Public Clouds

Public cloud platforms have elevated the abstraction and dynamic allocation of compute resources into an art form. The big cloud providers leverage vast networks of hundreds of thousands of servers and unfathomable numbers of storage devices spread across data centers around the world.

Anyone, anywhere, can create a user account with a provider, request an instance using a custom-defined capacity, and have a fully-functioning and public-facing web server running within a couple of minutes. And since you only pay for what you use, your charges will closely reflect your real-world needs.

A web server I run on Amazon Web Services (AWS) to host two or three of my moderately busy websites costs me only $50 a year or so and has enough power left over to handle quite a bit more traffic.

The AWS resources used by the video streaming company Netflix, will probably cost a bit more - undoubtedly in the millions of dollars per year. But they obviously think they're getting a good deal and prefer using AWS over hosting their infrastructure themselves.

Just who are all those public cloud providers, I'm sure you're asking?

Well that conversation must begin (and, often, end) with AWS. They're the elephant in every room.

The millions of workloads running within Amazon's enormous and ubiquitous data centers, along with their frantic pace of innovation, make them the player to beat in this race. And that's not even considering the billions of dollars in net profits they pocket each quarter.

At this point, the only serious competition to AWS are Microsoft's Azure which is doing a pretty good job keeping up with service categories and, by all accounts, is making good money in the process; and Alibaba Cloud which is mostly focused on the Asian market at this point.

Google Cloud is in the game, but appears to be focusing on a narrower set of services where they can realistically compete.

As the barrier to entry in the market is formidable, there are only a few others who are getting noticed, including Oracle Cloud, IBM Cloud and, with a welcome change to the naming convention, Digital Ocean.

Private Clouds

Cloud goodness can also be had closer to home, if that's what you're after. There's nothing stopping you from building your own cloud environments on infrastructure located within your own data center.

In fact, there are plenty of mature software packages that'll handle the process for you. Prominent among those are the open source OpenStack (openstack.org) and VMware's vSphere (vmware.com/products/vsphere.html) environments.

Building and running a cloud is a very complicated process and not for the hobbyist or faint of heart. And I wouldn't try downloading and testing out OpenStack - even just to experiment - unless you've got a fast and powerful workstation to act as your cloud hosts and at least a couple of machines for nodes.

You can also have it both ways by maintaining certain operations close to home while outsourcing other operations in the cloud. This is called a hybrid cloud deployment.

Perhaps, as an example, regulatory restrictions require you to keep a backend database of sensitive customer health information within the four walls of your own operation, but you'd like your public-facing web servers to run in a public cloud. It's possible to connect resources from one domain (say, AWS) to another (your data center) to create just such an arrangement.

In fact, there are ways to closely integrate your local and cloud resources. The VMware Cloud on AWS service makes it (relatively) easy to use VMware infrastructure deployed locally to seamlessly manage AWS resources (aws.amazon.com/vmware).

The Value of Outsourcing Your Compute Operations

Why might you want to migrate workloads to the cloud? You might end up saving a lot of money. So there's that.

Of course, it's not going to work out that way for every deployment, but there do seem to be a lot of use cases where it does.

To help you make informed decisions, cloud platforms often provide sophisticated calculators for you to compare the costs of running an application locally as opposed to what it would cost in the cloud. The AWS version of that is here: aws.amazon.com/tco-calculator

Part of the pricing calculus is the way you pay.

The traditional on-premises model involved large up-front investments for expensive server hardware that you hoped would deliver enough value over the next five to ten years to justify the purchase. These investments are known as capital expenses ("Capex").

Cloud services, on the other hand, are billed incrementally (by the hour, or even minute) according to the number of service units you actually consume. This is normally classified as operating expenses (Opex).

Using the Opex model, if you need to run a server workload only once every few days for five minutes at a time in response to an external triggering event, you can automate the use of a "serverless" workload (using a service like Amazon's Lambda) to run only when needed. Total costs: perhaps only a few pennies a month to cover those minutes the service is actually running.

Besides cost considerations, there's a lot more going on in the cloud world that should attract your consideration.

You've already seen how the lag time between the decision to deploy a new server on-premises and its actual deployment (weeks or months) compares to a similar decision/deployment process in a public cloud (a few minutes).

But large cloud providers are also positioned to deliver environments that are significantly more secure and reliable.

As an example, you may remember our story about the DDoS attack from chapter 2 (Understanding Digital Security). That was the incident where the equivalent of 380,000 PDF books worth of data were used to bombard an AWS-hosted web service each second... and the service survived. Are you confident you could do that yourself?

And how about reliability through redundancy? Would your on-premises infrastructure survive a catastrophic loss of your premises? Even if you did the right thing and maintained off-site backups, how long would it take you to apply them to rebuilt, network-connected, and functioning hardware?

The big cloud platforms run data centers across physically distant locations around the world. They make it easy (and in some cases unavoidable) to replicate your data and applications in multiple locations so that, even if one data center goes down, the others will be fine. Can you reproduce that?

Cloud providers also manage content distribution networks (CDNs) allowing you to expose cached copies of frequently-accessed data at edge locations near to wherever on earth your clients live. This greatly reduces latency, improving the user experience your customers will get. Is that something you can do on your own?

One more thought. Most of the big investments into new IT technologies these days are being plowed into cloud ecosystems.

That's partly because the big cloud providers are generating cash far faster than they can hope to spend it. But it's also because they're involved in a live-or-die race to capture new segments of the infrastructure market before the competitions claims them.

The result is that the sheer rate of innovation in the cloud is staggering.

I earn a living keeping a close eye on AWS, and even I regularly miss new product announcements. One of the reasons I avoid including screenshots of the AWS management console in my books and video courses is because their console is updated so often, the images will often be out-of-date before the book hits the streets.

In some cases, this might mean that local deployments will run at a built-in disadvantage simply because they won't have access to the equivalent cutting edge technologies.

The Risks of Outsourcing Your Compute Operations

Having said all that, as with most things in life, choosing between cloud and local isn't always going to be as obvious as I may have made it sound.

There may still be, for instance, laws and rules forcing you to keep your data local. There will also be cases where the math just doesn't work out: sometimes it really is cheaper to do things in your own data center.

You should also worry about platform lock-in. The learning curve necessary before you'll be ready to launch complex, multi-tier cloud deployments isn't trivial. And you can be sure that the way it works on AWS, probably won't be quite the same as what's happening on MS Azure.

The knowledge investment you'll need to make once you make your choice will probably be expensive.

But what happens to that investment if the provider's policies suddenly change in a way that forces you off the platform? Or if they actually go out of business (this could happen: Kodak, Blockbuster Video, and Palm were once big, too)?

And what about getting locked out of your account for some reason? How hard would it be for you to retool and reload everything somewhere else?

Just think ahead and make sure you're making a rational choice.

Chapter 4: Understanding Digital Connectivity

Telephones changed the way we all talked to each other and went about our work (well, the way our great-grandparents did, at any rate). Information could now be communicated instantly, rather than being sent over slow, overland routes.

But that's hardly news to anyone these days. The modern network - best known as the internet - similarly boosted communication, although this time it was the movement of data rather than voice that got a boost.

In the fifty years or so since the birth of the internet, it's been trusted with the movement, storage, and management of more and more of our data. These changes have brought tremendous opportunities, risks, and pressures.

Just getting connected is now a basic necessity.

Managing all of our many connected devices and leveraging the ways we authenticate to extend our identities also present challenges. We'll discuss all that in this chapter.

Connecting to the Internet

These days, after food and shelter, one of the most basic resources of all is internet connectivity.

If you can't access the internet, you'll find it difficult to do your banking, educate yourself, book travel arrangements, or even figure out exactly where you are.

It's not for nothing that widespread, reliable, and relatively fast internet access is critical for a region's general economic development.

Even though the internet was originally built as a decentralized, distributed network of resources, you still need to establish some kind of connection to access it.

The best connections are run by network carriers, known as tier 1 networks. Theses networks can reach all other networks through a peering arrangement that doesn't require payment for IP transit.

You can think of these networks as the backbone of the internet, and their network infrastructure is its structure.

Examples of companies managing tier 1 networks include AT&T and Verizon in the US, Tata Communications (India), and Deutsche Telekom (Germany). Those carriers will resell bandwidth to smaller internet service providers (ISPs) who, in turn, sell access to end users like you and me.

Broadband Options

Individuals looking for broadband access in their homes or small businesses can usually choose between one of four access models:

• Cable. Since they're already in the business of providing data to millions of homes over existing physical connections, cable TV providers can easily transmit internet over the same wires.
• Digital subscriber line (DSL). A family of technologies that permit digital data across copper telephone lines, DSL can provide a roughly similar level of service as cable, but without the need for an underlying cable subscription. In fact, using a "dry copper" connection, you don't even need a telephone landline account.
• Fibre optics. Due to some arcane technical details (including the laws of physics), transmitting digital signals as infrared light can happen faster and require fewer repeaters than comparable electrical cables. A fibre optics internet connection could typically deliver transfer speeds of 10-40Gbit/s - a thousand times faster than currently standard rates using cable or DSL.
• Satellite. Running new cable through densely populated cities is expensive, but companies can quickly make their money back through the many access contracts they'll sign.

But sparsely populated rural regions are much more difficult to service. Partly to fill a rural connectivity gap, a number of companies ambitiously working to launch constellations of thousands of orbiting satellites to provide universal internet coverage.

As of this writing, SpaceX is furthest along with its project, having already successfully launched more than 500 satellites as part of the Starlink system.

Besides those dominant technologies, there have been more than a few alternate connectivity solutions attempted. Some are experimental but promising, and others are a bit more speculative.

Google's Balloon Internet (known officially as Loon LLC), is an attempt to float fleets of high-altitude balloons providing a 1 Mbps signal to anyone within range on the ground.

Loon is designed to provide low-end broadband in hard-to-reach regions where reliable service has been difficult or even impossible. As of 2020, the project seems to be in a late experimental stage.

Broadband over power line (BPL) can take advantage of all the electrical grid that connects power authorities with homes and businesses to provide internet data.

Ultimately, the technology delivers limited bandwidth because line noise causes significant data signal loss. Data carrying power lines can also cause interference with high frequency radio communications.

In the end, relatively low signal quality and strong competition from other technologies mean that BPL will probably never be widely adopted.

Networks using forms of wireless Internet service provider (WISP) can service homes and offices across larger geographic areas without the need to physically wire every building.

Wired connections are installed in an area's center and, where necessary, connected backhauls are installed in elevated areas to strengthen the wireless signals aimed at consumers. Existing radio towers or other tall structures can be used for the backhaul repeaters, making a WISP system relatively inexpensive to install.

Smaller-scaled wireless network co-ops can be shared locally using a neighborhood internet service provider (NISP) (using rooftop antennas) or a wireless mesh network (where network-connected devices act as peer nodes) to efficiently share a single physical connection.

Those systems are primarily designed to serve us where we live and work. But mobile data access is definitely also a thing.

I'm sure you're already familiar with data plans that mobile phone companies can provide alongside their calling and texting services.

Mobile Phone Data Access

Cell connectivity is distributed through geographic areas (known as "cells") from individual radio transmitters spread throughout the cell.

Since the transmitters within each cell will use different radio frequencies than the cells around it, modern wireless technologies permit a seamless, automated "handover" as a user moves between cells.

The technologies used for wireless telephony have changed since the 80s, when what's now known as 1G ("First Generation") cell phones were introduced. To describe the evolution of cell phones in very general terms, we could say that:

• 1G phones carried only voice communications and had a maximum transfer speed of 2.4 Kbps.
• 2G phones could carry Short Message Service (SMS) and Multimedia Messaging Service (MMS) messaging, which could include short videos and images.
• 3G phones had much higher transfer rates (as high as 2 Mbps) than any variant of 2G and was therefore dubbed, "mobile broadband."
• 4G phones could reach speeds as high as 100 Mbps, which permitted HD mobile TV, online gaming, and video conferencing.
• 5G phones - when used on compatible networks - are expected to reach transfer speeds of up to 20 Gbps at a very low latency, permitting fully immersive virtual environments.

Should the 5G rollout be successful (and, at the time of writing, this isn't yet clear), the range and limits of new service categories that could be deployed is not yet known.

When it comes to planning a new venture, it's long been the accepted wisdom that there's no replacement for solid market research.

Without knowing who your customers will be, where they live, and what they like, how can you properly serve them?

Well, now you can add to that list "how reliable and robust is their internet connectivity," because without digital access, they may never find you or be able to consume your service.

Talking to the Internet of Things

Two recent changes are, more than anything else, responsible for the internet of things (IoT) ecosystem: the availability of cheap, embedded, single-board computers (like the Raspberry Pi pictured in figure 4.1), and cheap and always-on internet connectivity.

Figure 4.1

Those tiny single-boards - often smaller than a credit card - are easy to incorporate into just about anything you're planning to manufacture. Such devices cost very little - sometimes just a few dollars a piece - and they're generally built to run fully-powered (and free) Linux distributions.

And network availability means that the vast streams of data generated by all those on-board cameras, sensors, and other peripherals, can be automatically sent back "home" for processing and analysis.

The Dream of IoT

Here are some ways that IoT applications are already actively changing business and consumer practices:

• Inventory control. The very first IoT device was - arguably at least - a Coca-Cola vending machine at Carnegie Mellon University. Back in the early 80s, the machine was modified to digitally report its ongoing inventory.

The simple idea that physical devices can monitor themselves and their surroundings, providing accurate, up-to-the-minute status reports to remote servers lies at the heart of countless modern industrial solutions.

Modern retail, wholesale, logistics, and manufacturing operations now have constant access to their inventories, allowing them to understand trends and anticipate problems.

• Agriculture. Increasingly, modern farming incorporates robotic irrigation, fertilization, planting, and even harvesting technologies. All those robots running around your property are generating data and, from time to time, getting themselves into trouble.

Moving that data "back" to administration servers is critical for keeping track of what's going on, what might need fixing, and how your actual farm is performing.

You can, therefore, expect that each of those devices will be part of someone's IoT.

• Military. Communication is key for military operations. But if even weapons, vehicles, and other equipment can communicate autonomously, and if there are servers dedicated to interpreting and acting on that communication, then you're already way ahead of the game.

Sensors connected to each of hundreds of components for, say, a fighter jet, can contribute to giving planners an unprecedented view of what's really going on.

• Smart cities. When sensors embedded in buildings, roads, public lighting, smartphones, and electrical systems are combined with data coming from traffic cameras and public departments, the potential for data-driven insights is huge.

Properly understood data can help cities manage their resources, utilities, and even traffic more efficiently, and better maintain their physical infrastructure.

• Smart homes. On a far smaller scale than smart cities, home appliances can be connected and monitored and controlled through smartphone apps or remote servers.

Smart home devices already include heating and cooling systems, light bulbs, robotic vacuum cleaners, garage doors, and security systems. These devices can be controlled through phone apps but, in many cases, also through voice controlled devices like Amazon Echo (Alexa).

Conversations about IoT are always just one step away from buzzwordism - where words lose meaning and exaggeration becomes an acceptable lifestyle choice.

Not all IoT stuff is actually IoT. Or, to put it another way, not all IoT is worth talking about.

But here's one good way to categorize a particular technology: if, hour after hour, something generates more data than any human being could possibly absorb, then it's probably an IoT device.

Effectively dealing with all that data can be a problem. And that's not the only potential for trouble in IoT land.

The Nightmare of IoT

In the information technology world, as a general rule, the more active network connections you have in your infrastructure, the greater your risk of being successfully attacked.

That's because successful hacker intrusions usually come through badly configured or unpatched devices. The more public-facing devices you're exposing, the greater the chance one of them will have a serious vulnerability.

What kind of vulnerabilities are we talking about?

Well, the US government's Common Vulnerabilities and Exposures database contains nearly 140,000 individual entries, each one representing a unique software weakness that could allow unauthorized access to and destruction of an IT system.

There are threats impacting all operating systems (Windows, Linux, macOS), all formats (server, PC, smartphone), and all ages (there are active threats going back to the 1990s).

And many hundreds of new entries are added each month.

In that sense, IoT devices are no different than any other kind of computer. But there is one way that they're a whole lot worse.

Because you usually don't normally directly interact with IoT devices on an OS level, and because they're often commodity items that are purchased and deployed by the dozens, or even thousands, you don't instinctively treat them like computers.

Most of us, as an example, are aware that we should create complex and unique passwords for our laptops and WiFi routers.

But your fridge? Just plug it in and it'll be fine!

The problem is that many IoT devices - like "smart" fridges - have their own embedded operating systems and, usually, their own network interfaces.

There's a good chance that anyone driving down your quiet residential street can scan for available networks, and quickly identify the brand of IoT device you're using. They can then assume that you haven't changed the authentication credentials from their factory defaults, and log in to your private network.

What makes things much worse is that many device manufacturers are still shipping their products with authentication credentials using some variation of admin/admin.

That's a big problem.

Leveraging Federated Identities

All this talk about the dangers presented by authentication and credentials should make you curious about how they can be used to generate some good connectivity stuff.

In a single word, that'd be federation.

Identity federation is a technology for linking a single person's identity across multiple network services. Federation is what lets you log in to online gaming or web application sites using, say, your Google account credentials.

The upside of federation is that a single login can be all you'll need as you move between many of the online services you regularly use. That lets you reduce the risk of exposing your password through a vulnerable website.

Of course, it also increases the damage that can come from a serious data breach of the servers used by your federation provider.

Federation can be used to integrate with third party single sign-on (SSO) authentication systems, like Kerberos, the Lightweight Directory Access Protocol (LDAP), and Microsoft's Active Directory (AD). When used with cloud services, SSO systems can securely permit automated as-needed access to private account resources for consumers or processes.

Besides convenience, all this authentication goodness drives opportunities for secure remote collaboration on large, complex projects - itself a fast-growing trend.

Chapter 5: Understanding the Business of Technology Research

Getting a new technology out to consumers will usually require good people and boat loads of resources - including money. Generally, lots of money.

A lot of that money will be spent on research and, more often than not, the hard research needed to translate a great idea into a usable product will be performed by someone whose job title isn't "entrepreneur."

In fact, sometimes the research will be done by individuals who are barely aware that their innovations have any commercial value at all.

If you're here because you want to get the jump on cutting edge technologies, then you may want to keep an eye on the organizations that are known to produce practical research. Knowing who's big in research, who's funding it, and where the big bucks are being spent can give you useful insights into what might be coming next.

From there, you're just a step away from, say, spending time learning the tools that'll come with the new tech or positioning yourself to profit when it finally shows up.

Who Funds Commercial Science and Why?

Once upon a time, major breakthroughs in serious scientific research were the products of private patronages. The Italian Medici family, for instance, famously supported many individuals whose work would prove pivotal, including Leonardo da Vinci and Galileo.

However, the years leading up to the Second World War saw the scope and complexity of research projects growing far beyond the capacity of private support. The war's dependence on unprecedented technological complexity - exemplified by the work of the Manhattan Project building the atom bomb - pushed more and more research under government charge.

Government involvement in research has continued in the generations since the war. Still, it's been estimated that universities and governments are responsible for only 30% of research funding between them, with most of the rest provided by private industry (see en.wikipedia.org/wiki/Funding_of_science).

Let's see how that breaks down.

Taxpayers

Democratic governments, of course, don't spend their own money, of which they traditionally have none.

Their many programs and services are funded by revenues raised, one way or another, from their capital assets and from their populations. In modern nation states, "populations" would mean those individuals and corporations who pay taxes.

Public research and development can be performed within government agencies. According to the terms of some agency mandates, research results must immediately enter the public domain.

But even those who retain rights to their research will often point their work towards businesses and institutions that can use it productively.

The US National Science Foundation (NSF), for instance, uses its $8 billion annual budget to fund "approximately 25 percent of all federally supported basic research conducted by America's colleges and universities" (https://www.nsf.gov/about/).

Other American agencies do much or all of their research in-house. Here are some examples:

• The National Institute of Standards and Technology (NIST) has a mandate to "promote innovation and industrial competitiveness."

One very important part of that mission is maintaining the National Vulnerability Database (NVD), which plays a foundational role in the management of the vulnerability assessment and detection systems protecting our IT infrastructure.

• The US military's Defense Advanced Research Projects Agency (DARPA) collaborates with private and public sector partners to aid in the development of emerging technologies.

Work in recent years has included research into robotics and autonomous vehicles, but you might be more familiar with a DARPA innovation from a few decades ago: the internet.

• The National Institutes of Health (NIH) employs 6,000 research scientists across 27 research institutes and centers.

Their "mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability."

The complete list of US government research agencies (available at en.wikipedia.org/wiki/List_of_United_States_research_and_development_agencies) makes for quite a read. Take a look for yourself.

Naturally, governments of other countries have their own research agencies. One example is Canada's National Research Council (NRC), which has evolved from its military technology origins through the two world wars, to its current focus on partnerships with private and public-sector technology companies.

The NRC now divides its work into four "business lines:"

• Strategic research and development
• Technical services
• Management of science and technology infrastructure
• NRC-Industrial Research Assistance Program (IRAP)

As we mentioned when discussing the NSF, a significant proportion of taxpayer funds directed towards research and development are granted to public and private colleges and universities.

But, from the college perspective, how much academic R&D funding comes from government sources?

A 2016 review of the 20 US colleges that spent the most on R&D found that they each spent between $837 thousand and $2.4 million, and that between approximately 47-87% of their total spending came from government sources of one sort or another (see bestcolleges.com/features/colleges-with-highest-research-and-development-expenditures/).

By contrast, businesses only provided between 2 and 22% of that funding.

Private Charitable Funding

While we're on the subject of academic research, we shouldn't ignore a third source of funding: private endowments.

Some - although not all - permanent endowments were targeted by their donors at research activities. Although the fund capital can't be spent each year, the income that capital generates can.

Harvard university famously - or perhaps infamously - has a total endowment greater than 40 billion dollars. Some of that undoubtedly finds its way to R&D.

Curiously, according to that 2016 study, Harvard's total R&D spending that year - including activities funded by governments (52.1%), businesses (4.7%), and endowments - was just over one million dollars.

Of course, donations support plenty of research outside of academic settings, too. Most serious diseases have associated charitable foundations that exist to raise money for both victim care and medical research.

And many thousands of registered non-profits exist throughout the world supporting non-medical causes, including many that involving technology-related research. The Bill & Melinda Gates Foundation is a particularly well-known example.

Corporations

Technology-oriented companies have a strong interest in getting their hands on innovations before their competition. To improve their chances, many will run their own research labs in-house.

The Bell Telephone Company, for instance - and its successors including American Telephone & Telegraph Company (AT&T) - maintained the active and enormously creative Bell Labs. Bell Labs, under various names, was responsible for many innovations, including the transistor, lasers, and the Unix operating system.

Individual technologists at some companies are often sources of innovation. 3M, for instance, has what they call a "15% Culture," where employees are allowed to use company time and space to pursue research based on their own ideas and interests.

Over the years, the program has generated successful products for the company, including their sticky paper Post-its. In another example, Percy Spencer, working on radar for US defense contractor Raytheon, accidentally discovered that microwaves could cook food.

It should be noted that not all corporate innovation is truly home-grown. A lot of it is actually funded indirectly through government money in the form of tax incentives or credits.

Under such programs, companies may be permitted to use research-related spending (including salary expenses) to reduce the income taxes they would otherwise pay.

Major Fields of Commercial Technology Research

Trying to grasp the full scope of technology development at this point in history is an unforgivable waste of time.

There's serious innovation going on every minute of the day, in every time zone, through countless labs, office towers, warehouses, garages, basements, bedrooms and, of course, invisibly within creative people's minds.

No one's keeping track of it all because it's not possible. Not to mention the fact that much of that innovation happens under a thick shroud of secrecy.

But it's probably worth offering just a couple of examples to give you a feel for where to look.

Quantum Computing (and Why We Should Care)

A cousin of mine with an advanced physics degree from Cambridge University once tried to explain quantum mechanics to me.

He failed. Miserably.

My poor old brain just couldn't absorb it. So don't expect any full, measured descriptions of the underlying science here. Instead, I'll try to show you how experimental compute technologies that depend on the physics might work, and what can be done with them.

The super-quick executive summary version of this is that computers powered by one quantum technology or other will work a lot faster than any of the super-est of super computers we have now. So much faster, in fact, that they may be able to solve problems that would be simply unfeasible using traditional computers (an achievement known as quantum supremacy).

This would mean that some long-held assumptions about the way software works will no longer apply.

For instance, the whole reason why we use the encryption tools we use today to protect sensitive data is because it would take hundreds, even thousands, of hours of high-performance compute time to break an encryption key.

In most cases, it's just not worth the effort and expense.

But if you could easily buy time on a computer that processed operations exponentially faster, then two things would immediately happen:

• Cracking encryption algorithms would become trivial
• Honest folk would have to seriously look for a new way to protect their data

Currently, Google and IBM are among the major companies that have invested heavily in quantum compute research projects.

As well as I can understand it, quantum computers would measure the state of subatomic particles and use that binary measurement to represent a computational value.

The description of that state is known as a qubit, which is effectively the quantum equivalent of traditional computing's bit.

But because a qubit can also exist within what's known as coherent superposition - meaning that its value can exist in a superposition of two possible states - it can be used to represent a more complex range of values.

And that, I'm given to believe, means that such computers will be able to do stuff much, much faster than they can now. If this actually happens, it'll be big.

Energy Technologies

The modern world consumes an awful lot of energy.

We're constantly moving about, controlling our indoor (and in-transit) climate conditions, exchanging information, and expecting that all the world's riches be delivered to our doorsteps. By tomorrow.

But those energy-thirsty activities come with costs, not the least of which are the emissions they leave behind. The search for reliable, steady, and affordable energy sources that can help us find a healthy balance between consumption and emissions is ongoing; and unimaginably expensive.

Small modular nuclear reactors (SMRs) have been the focus of some serious developments in recent years. They appear to promise reliable, steady, and affordable energy in ways that their expensive and complex nuclear predecessors couldn't.

First and second generation reactors were, overall, reliable and steady - and they were clean - but their massive capital costs and large physical footprints made them more than a bit inflexible.

The idea behind SMRs is that highly efficient reactors can be manufactured off-site and delivered on trucks one module at a time for on-site assembly. This makes the per megawatt generation of power far cheaper, and project completion much faster. And this allows the deployment of nuclear power to service smaller markets that previously couldn't consider it as a realistic option.

As the name implies, SMRs are smaller than traditional reactors. They're designed to deliver between 50 and 300 MW of electricity each, compared with the 800 to 1,200 MW outputs that were previously common.

Companies heavily involved in this research include Britain's Rolls-Royce and an American company with historical connections to the US Department of Energy called NuScale Power.

Various governments around the world have also invested in this technology one way or another.

Medical Technology Research

If you think we're spending a lot of money on energy, wait 'till you see how much healthcare costs.

Across the 37 Organisation for Economic Co-operation and Development (OECD) nations, healthcare industry spending accounts for around 10% of the total gross domestic product. That's more than $3,000 a year for every single man, woman, and child.

On the one hand, with all that money being thrown around, there are undoubtedly many business opportunities waiting to be discovered. But there's also a lot of room for new and innovative technologies that can improve the delivery of healthcare while reducing the costs.

Telehealth and telesurgery are two excellent candidates.

Telehealth involves the provision of health services (like patient-doctor consultations) through a telecommunication medium. This might mean having a simple telephone conversation rather than a visit to the office, but it could also incorporate video conferencing tools or even the use of remote diagnostic equipment.

For example, small, remote communities could maintain imaging facilities and technicians even many hundreds of miles away from the nearest medical labs and radiology specialists.

Digital connections can permit distant doctors to view, say, ultrasound results, speak directly with patients, and confidently reach diagnoses. And all without the need for anyone to undertake exhausting and expensive travel.

Telehealth also allows for meaningful patient-doctor contact without the risk of spreading disease.

Telesurgery is an extension of telehealth which can allow some surgical procedures, even when doctors are many miles away from their patients. The technology makes use of high-definition video feeds and purpose-built robotic arms that can be controlled by doctors remotely.

Telesurgery tools have the potential to save money for cash-strapped health systems but, more importantly, they can improve healthcare and save lives.

Chapter 6: Where Hot Trends Go to Die

You're reading this book, so I'll assume you have an active interest in learning about hot technology trends. But you don't want to take every last one of these gadgets and business fads too seriously: some are bound to disappoint.

(Although don't think you can hold me personally responsible for any of my predictions: by the time this book hits the streets I expect to be living on a sun-drenched tropical island under the witness protection plan.)

To get a sense of how fragile the innovation business is, keep in mind the popular wisdom that teaches us how nine out of ten startups will fail.

Now multiply that by the particularly speculative nature of the technology industry in particular, and you'll appreciate how easily things can often go spectacularly wrong.

That's not to say that the folks who dreamed up all the doomed businesses we'll soon discuss were fools or frauds. It's easy for us, enjoying the benefits of historical hindsight, to judge their efforts.

We should be sensitive to how different things must have looked in the heat of the moment. Still, bearing that in mind, there's value to be had from trying to at least understand what went wrong.

So here are some particularly impressive examples from the junkyard of tech history. It can be loads of fun to relive some of the biggest business disasters in history, but there are also important lessons we can apply when assessing this year's crop of "can't-fail" devices.

Market Research Beats Wishful Thinking Every Time

I'm not sure it's possible to reliably count all the individuals and companies that have assured us we've finally reached the age of bypassing choked highways using flying cars while happily chatting with loved ones through video calls.

There have been hybrid wheeled/winged prototypes since soon after the Second World War, and telephones for showing the world how you look in pyjamas have, in theory at least, been available since the early 1970's.

Well, video calls are now easily available through any smart phone or PC. And, 75 years of failure later, the rush to deliver consumer flight hasn't slowed down a bit.

But both technologies are most notable for being so rarely used: flying cars because none have ever hit full production, and video phones because very few people seem interested.

What's been the trouble? There were certainly engineering, safety, and regulatory hiccups over the years. And there's no question that flying car manufacturers would be hard pressed to find a large customer base of drivers who were also qualified pilots (although self-driving/flying versions could, in theory, avoid that issue).

But I suspect a big part of the problem was marketing: no one bothered asking Joe Q. Customer for his thoughts on the matter.

But Marketing Isn't Everything

In the beginning, there were comments from tech insiders about how a new device would be the biggest thing since, well, ever. Then came an unauthorized book leaking intriguing information, some ambitious public claims, and a product launch.

In the end, there was the Segway: a personal transportation device that was too big and fast for sidewalks, too big and slow for roads, and too expensive for most customers. And using it in the rain or snow wasn't much fun at all.

Today you'd probably have to look pretty hard to find a living, breathing Segway anywhere close to your neighborhood. They're sometimes used for police street patrols and touring, but they haven't eliminated the car or revolutionized urban development.

Nor, as far as I can see, did they make the company's investors fantastically wealthy. In fact, the company's manufacturing plant in Bedford, New Hampshire ceased operations in the summer of 2020.

What went wrong?

Well, perhaps the hype was a bit over the top. OK, make that way over the top. It's never a good thing to pump up expectations to the point they can't possibly be met.

There was also the failure to match the tool to an appropriate environment. Where, after all, was it supposed to be used?

But, to be really successful, a new product has to be built on more than clever engineering. It also has to solve a real and pressing problem.

When Too Much Power Isn't a Good Thing

Back in 2013, Google introduced a new consumer compute product they called Glass. This was a sleek headset that could be worn as an attachment to a pair of designer prescription glasses.

When powered on, Glass could accept voice and touch commands to record video of everything the wearer sees, and display data - often with full "awareness" of the wearer's current physical location.

Glass was a single device intended to replace much of the function currently served by smartphones, laptops, and media players.

For the task of integrating our physical world with the endless data that describes it, this was going to be perfect. And then it wasn't.

As more details about Glass became known, questions were raised in the broader tech world. Was it appropriate - or even legal - to silently record videos of other people? Should face recognition software be applied to random pedestrians walking past on the sidewalk without their consent? Was it safe to drive while wearing Glass?

Potential customers had their own questions. Is the product affordable (they started at $1,500)? Is it necessary? Does it fit the vision I have for my public image?

The longer those questions floated around the internet, the more answers came back. Answers, by and large, consisting of a single word: "No."

Google Glass, as a consumer product, slowly faded away and eventually disappeared altogether. The massive media promotion campaign had come up empty.

Which is not to say that the product itself failed completely.

As it turns out, Glass has found considerable success in medical environments where, for instance, it could be used to permit remote surgical experiences. It's also found a home in industrial settings, where front line workers often need instant, hands-free access to relevant schematics and directions.

But it was a long while before all that goodness happened.

Perhaps someone should have slowed things down at some point, saying: Even if it's possible to engineer all of those features into a consumer product, is it necessarily a good idea?

When a Thousand Pieces Don't All Fall Where They're Supposed To

Sometimes measuring success and failure isn't so easy. Take WebTV as a case in point.

Who doesn't own a TV? (Besides me, I mean.) Wouldn't it make sense to create an inexpensive and easy-to-use product that leverages billions of existing home TVs for non-standard but popular uses? How about a device that can turn the TV you already own into a web browser and email portal?

If that doesn't sound so exciting today, back in the mid-90's the idea behind WebTV had its definite charms.

Just imagine the secondary revenue streams this could generate. Wouldn't advertisers climb over each other to pay big bucks to have their products pitched to all those TVs?

Had WebTV managed to deliver on the "easy-to-use" angle, things might have gone differently.

But it turned out that the primary demographic for the device was heavily skewed to older people; who needed a lot of (expensive) customer support coaching through the setup process.

Their inability to keep up with the fast-changing internet browsing standards also made it tough to provide a consistently optimal browsing experience - especially for users sitting ten feet away from the screen on their couches.

How did things actually play out?

On the one hand, within two years of their launch, the company was purchased by Microsoft for more than 400 million dollars, who rebranded the service "MSN TV." In one form or another, they stuck around until long past the death of dial up internet access. So that's a good thing.

But, arguably, they failed to capture nearly as much interest and adoption as they could have.

The real prize was in becoming a dominant portal for internet access. Because the platform was proprietary, the company could effectively have controlled the entire internet experience of hundreds of millions of users.

The potential scope of the product would have dwarfed the modest revenues they actually achieved. So that's not a good thing.

Were they too far ahead of their time? Did they miscalculate by insisting on a closed, proprietary platform? Did they fail to see the monstrous growth in the standalone personal computer (PC) industry coming?

Either way, it wasn't exactly a fairy tale ending.

When Timing Isn't Your Thing

The tech industry moves fast. I'm sure that little nugget of wisdom won't leave any of you wrapped in stunned silence.

But when you think about how much work is needed before you can convert a fresh, new idea into a ready-to-ship product, it's remarkable anything innovative ever gets off the ground.

Bad timing, then, is a risk faced by the people behind pretty much any new technology as it makes its way to market.

By way of example, the existence of strong competition from companies like Nintendo and Sony's PlayStation were probably largely to blame for the premature death of Apple's Bandai Pippin gaming console back in the mid nineties.

Although, the fact that there were never more than 25 game titles that would run on the device, and that, like all Apple products, it was priced much higher than the competition couldn't have helped.

All wasn't dark and foreboding for Apple in those years.

Looking back with what we now know, the strong presence of their iPod digital music player platform was probably what doomed Microsoft's Zune.

With the Zune, Microsoft had the bad luck (or lack of foresight) to get stuck between an iPod device made dominant by its simplicity, and the looming age of the smartphone (where standalone portable music players become irrelevant).

Clearly, as Shakespeare would have said it, "ripeness is all."

But there's another thing about timing: eventually, you'll need to deliver the goods.

There's a limit to how long consumers will wait for that bright new technology that's been on everyone's "must-have" list for too long without making an actual, real-world appearance. Beware empty promises.

You should also keep a critical eye out for good, old fashioned bad business practices - the kind that never seem to go out of style: unrealistic business plans; unfamiliarity with a business' core, underlying fundamentals; and unreasonable, greedy start up costs.

Also, the catastrophic disaster that characterized the dot-com boom and subsequent bust around the start of the 21st Century comes to mind. The take-any-business-model-and-build-it-a-website paradigm looked good, but it was applied far too broadly and often ignored the obvious context in the process.

Don't blindly trust popular trends and buzz phrases.

Up to this point, we've covered some "big-picture" technology topics like privacy and connectivity, following their twisting tentacles wherever they'd lead us.

In the next chapters, we're going to focus on six broad categories, one category per chapter, each with a handful of specific technologies.

These technologies have names that you've probably encountered and that, for the most part, already play important economic roles. Sometimes their significance and impact might be exaggerated, but they're all the real deal.

Chapter 7: Compute Platforms

The way things sit now, if you were somehow allergic to computers, you'd be hard pressed to really banish them from your life, no matter where you found yourself.

Taking a quiet walk in the woods? What about the smartphone in your pocket?

Left your phone in the car? See that cell phone tower just behind those trees?

Odds are good that the tower is more than just an antenna – it could also be hosting an edge computing server.

And don't think that there weren't computers embedded into the under-the-hood workings of the car (or bus) that drove you over.

Allergies aside, if you want to fully grasp the current state of the compute world, it'll be helpful to understand all the places computers can pop up and what they might look like.

In this chapter, we'll enumerate the classes into which modern compute devices can fall, and describe their strengths, weaknesses, and potential.

What Is a Server?

Honestly, I'd been working as a professional system administrator for a while before I could properly answer this question.

The truth is that every server is a computer, and any computer can be a server. The term server simply implies that the device is providing some service to at least one external device (known as a client).

If the printer that's plugged into your desktop computer can be shared by the other computers in your local network then your desktop is a server (a printer server, to be precise).

The WiFi router provided by your internet service provider is, by all definitions, a network server – as it serves network access to its clients.

And the tiny, $5 Raspberry Pi Zero single-board device (figure 7.1) that powers your homemade surveillance camera is a video server – although that won't work without attaching a $7 camera module.

Figure 7.1

But that's not what most people are thinking about when they use the term.

The first time I ever walked into the server room of a mid-sized business, I was hit by the sound of dozens of powerful chassis fans and the heat from hard-working CPUs and fast-spinning disk drives.

Instantly, I knew how folks normally use the word.

(Also, I quickly discovered that heat was a huge problem: the admins were struggling to keep their server room properly cooled and would, over time, end up having to write off some expensive hardware due to heat-related failures.)

So, by "server," we usually mean computers installed within those rack-mounted, stackable cases built to efficiently house and protect highly performant, expensive, and delicate components.

Server racks will normally live in well vented and cooled rooms with easy access to ample electrical power. You may have to search for them, but such rooms will also always contain colorful bundles of cabling, connecting the servers to networks.

As a rule, servers won't usually have displays or even keyboards plugged in, as they're likely to be managed remotely or, even more likely, fully automated and requiring no administration at all.

Server farms belonging to giant cloud providers like Amazon Web Services will have many thousands of commodity computers running within aisle after aisle of vast warehouses.

When one fails, a monitoring panel somewhere will light up and a technician will eventually be dispatched to remove the server, throw it out, and slide a replacement into the newly-available rack.

No tears are shed when we say goodbye to hardworking and devoted old hardware in those places.

What Is Linux?

Speaking of servers, did I mention that they all need some kind of operating system installed? And did I mention further that the vast majority of the servers powering the vast majority of the operations that make the internet and all its functionality possible are running the Linux operating system? Oh, and did you know that the open source Linux operating system is available for free?

I didn't mention all that? My bad.

Well, servers need operating systems. Most servers (well over 90 percent of the virtual machine instances running on Amazon's AWS EC2, for instance) run Linux. And Linux is, indeed, freely available for any use on any server, laptop, desktop, router, embedded system, or supercomputer.

In fact every last one of the world's top 100 supercomputers uses Linux. And the Android smartphone OS? Yup. It's Linux, too.

Strictly speaking, "Linux" is the software kernel that allows a computer user to take control of a computer's physical hardware elements. The kernel translates your keystrokes into a format that will be understood by the drivers controlling your storage drives, memory, network interfaces, displays, and – in fact – keyboard and mouse.

Many thousands of additional software programs are closely associated with Linux, but they're actually part of the user space that hovers "above" the Linux kernel.

Having said all that, Linux, including its broader software ecosystem, dominates the server computing market right now. The fact that you can freely install and fire up as many physical or virtual instances as you like makes Linux very attractive, especially in the scripted workload orchestration world.

Virtualized Linux instances will often be brought to life and then, after completing a task that takes even a few seconds, killed off again.

The versatility and flexibility Linux brings to computing have been the spark of some deeply impressive innovation and creativity.

Part of the Linux versatility is the fact that you can choose from among hundreds of variations (known as distributions).

Are you looking to run enterprise supported servers? Internet of Things (IoT) devices? Security testing machines? Multimedia management? Video or audio production? All of the above? None of the above? There's bound to be a distribution that's a good match for you.

And if the exact specs you need can't be found, feel free to rewrite the kernel itself and create your own distro.

Full disclosure: I know a thing or three about Linux, being the author of Linux in Action (Manning), a coauthor of Ubuntu Bible (Wiley), and the author of the Linux Fundamentals learning path at Pluralsight.

Fuller disclosure: I'm writing this on an Ubuntu Linux workstation in my home, where all of our many devices have been Linux-powered for more than a decade.

What Is Virtualization?

We've already discussed virtualization in some depth as part of Chapter 3 (Understanding the Cloud), so we'll just cover some big-picture conceptual basics here.

In the old days, you'd come up with an idea for a new compute project and submit a proposal with your managers asking for money.

When the project was green lighted, you'd estimate your requirements, solicit bids from hardware vendors, order a new server and, when it finally arrived, load it up with your application software, fire it up, and let the world see what you'd done.

That's the way things usually worked: One project. One server. Lots of waiting time.

But what if you overestimated your compute needs by 50 percent? That'd be a few thousand dollars down the drain. And if an important but lightweight project didn't really need a full, standalone server, you'd often have to buy it anyway.

How about if the project would only have to run for a few months? Spend the money and hope you'll find a new use for the thing once your initial project shut down.

Awkward. Mountains of awkward.

Virtualization is a (mostly) software trick that lets you fool multiple installed operating systems into thinking they're all alone on a physical computer when they're really sharing it with other OS's.

You can provision and run a single virtualization host of one flavor or another and then fill it up with one or a hundred virtual servers.

One of those servers might need a lot of system memory but only a GB or two of storage space. Another one might be heavy on video conversion tasks and storage but is only needed for a half an hour a day. A third could be a 24/7 monitoring system that just needs a quiet place to do its thing without anyone bothering it.

As long as you never push the physical host past its overall resource limits, the virtual machines can all coexist happily together. And when one service is no longer needed, you can reassign its free-up resources to something else.

The ramp-ups and ramp-downs of a typical virtual server's life cycle are fast. For all intents and purposes, they'll generally launch and shut down instantly.

This is possible because the underlying hardware is always running – and because the OS image is small and, usually, optimized for virtual environments.

As we saw back in Chapter 3, cloud-hosted services are all virtualized. As more and more IT infrastructure moves to the cloud, more and more of your online activities will be driven by virtual machines. You won't notice the difference, but every time you search the internet or authenticate to an online account, there's a good chance that it's a container or VM you're connecting to, and not directly to a physical machine.

Where Do You Go to Get Some o' That There Cloud?

Like virtualization, we also talked about the cloud back in chapter 3 - which would make sense, considering that the chapter was called "Understanding the Cloud".

We mentioned how the public cloud market was dominated by AWS and, to a lesser extent, by Microsoft's Azure.

I'll just take a minute or two here to add a quick guide through some of the cloud industry's worst jargon.

Infrastructure as a Service (IaaS) environments give you full access to virtual server instances. The provider will ensure the underlying hardware, networking, and security elements are in place and functioning, while it's your job to manage the OS and other software running on your instance.

Major IaaS players include Amazon's Elastic Compute Cloud (EC2) and Azure's Compute.

Platform as a Service (PaaS) environments hide most or all of the infrastructure administration tasks from you, leaving you with an interface where you can run your own data or code.

One good example is AWS Elastic Beanstalk, which lets you upload your application code from where it'll be automatically deployed to Amazon's cloud. Other providers in this space include Heroku and Salesforce Lightning Platform.

Software as a Service (SaaS) environments expose only an end-user interface, managing all layers of the administration infrastructure invisibly.

Microsoft's Office 365 and Google's G Suite are widely used SaaS office productivity tools. But there's a growing marketplace of SaaS tools providing online software equivalents to many applications that, in years past, could only be used on standalone workstations. Those applications include accounting, computer assisted design (CAD), and graphic design solutions.

Consumption-based pricing or, as it's sometimes known, pay-as-you-go billing, is a cornerstone of the cloud concept. The idea is that you don't have to gamble by investing up-front in infrastructure, but you can pay incremental amounts for units of compute services as you use them.

It might not always come out cheaper in the long-run, but pay-as-you-go definitely makes it easy to test application stacks and experiment with multiple alternative configurations before pulling the trigger on a full deployment.

It also means that – assuming you don't make any dumb configuration mistakes – it's nearly impossible to badly over-provision.

On-demand is also sometimes referred to as self-service. The ability to request instant delivery of compute resources any time of the day/week/year gives you complete control over your organization's application life cycles. You're never at the mercy of other people's schedules and limitations.

Service Level Agreements (SLAs) are legal disclosures published by companies in the business of providing services. Even if the standard of resource reliability provided by the major public cloud platforms is generally excellent, accidents will happen.

When you pay hourly or monthly fees for cloud services, the company's SLA tells you that you should anticipate downtime of a certain number of minutes or hours each month.

As an example, Amazon's SLA sets its EC2 availability rate at 99.99% each month. If, in a particular month, you encounter greater downtime, you might be eligible for service credits or refunds.

Multitenancy is the placement of virtual instances belonging to multiple cloud customer accounts on a single hardware resource.

A multitenancy setup for a server instance will probably be significantly less expensive than a dedicated instance. Choosing a dedicated instance, however, would guaranty that your instance will never be hosted on a physical server alongside an instance from a second account. Security or regulatory considerations might require that you avoid multitenancy.

Migration describes the process involved with moving existing business application and database workloads from local (on-premises) deployments to a cloud provider. Providers often make specialized tools and free tech support available for migrations.

Elasticity describes the ways virtualized cloud resources can be quickly added to meet growing demand or, equally quickly, reduced in response to dropping demand.

Elastic resources are especially well-suited to maintaining application availability and health without incurring unnecessary costs. Elasticity can usually be automated, so applications will respond instantly to changing environments without the need for manual intervention.

What Is "Serverless" Computing?

Serverless computing is no different from server computing. It's just that, even if you squint your eyes real tight, you don't get to see the server.

Or, to put it another way, serverless computing is like running a virtual server instance, but without having to configure its instance settings or log in to set things up.

In other words, you can't run software code of any kind without a computer somewhere processing your commands.

So let's say that serverless is a form of virtualization where everything except your application code is abstracted. In that sense, serverless platforms like Amazon's Lambda and Azure's Functions are a lot like the model used by Amazon Elastic Beanstalk, except that they're so simple to use that they can easily be incorporated into a larger, highly automated multi-tier workload.

What Is Edge Computing?

Latency is the term we use to describe the time it takes for data to travel from a remote server across a network to your computer – or back in the other direction.

Assuming you prefer fast service over slow service (which seems a safe assumption), high latency numbers are a bad thing.

Network engineers can invoke various magical spells – Oops! I mean clever configuration efficiencies – to reduce delays due to latency.

But no matter how many tricks they're hiding in their mysterious black bags, they can't ignore the laws of physics. Even using the best connections and configuration profiles, data still has to physically move across the distances between remote locations.

The only way to reduce this kind of latency is to shorten the distance. I suppose that one way would be for online service providers to very politely ask their customers to sell their houses and move somewhere closer to the servers running in the office (as if real estate prices weren't already high enough in Silicon Valley). Alternatively, though, how about moving the server closer to the customer?

Ah. You've discovered edge computing: the fine art of installing large distributed networks of smaller servers where mirror copies of server data can be stored and, when necessary, fed to any customers in the area who initiate requests.

If you've got enough of those servers spread evenly through the geographic regions where you customers live, then you can significantly reduce the latency they experience.

One kind of edge computing that performs this function is known as a content distribution network (CDN). Cloudflare and Amazon's CloudFront are among the larger CDNs currently in operation.

Edge computing resources like those used by CDNs have also increasingly been used to manage large streams of data from and to IoT devices like the computers embedded in cars.

Placing capable compute devices at the edges of large networks makes it possible to consume and transform such data sets faster than by moving the data all the way back to the more distant cloud.

What Are the Key Compute Form Factors?

Computers, like egos, come in all shapes and sizes. Would you like to carry a rack full of bare metal servers around in your pocket to pay for your shopping? You're probably better off using some kind of mobile payment app on your smartphone.

Size matters. A lot. A device's form factor will determine the dimensions and capacity of its internal components. That means the particular motherboard, memory modules, storage drives, peripheral ports, and power supply you select for a device will be limited by your overall form factor.

The form factor you choose – for either a new project or just for your personal use – will generally be obvious (server racks can be heavy and don't handle travel well). But knowing what's available can make it easier to plan.

Devices Using Video Displays

The term personal computer (PC), these days, is used to describe either desktop or laptop computers.

Laptops, since they're designed to be mobile, are largely self-contained. Desktops, by contrast, generally come with the core compute elements within a box that includes external ports for connecting peripherals like keyboards and monitors.

While you can find computers with power and functionality that's comparable to PCs in very small (credit-card sized) cases, the larger boxes used by desktops allow for easier customization and upgrades.

Gaming consoles – like Sony's PlayStation, Microsoft's Xbox, and the Nintendo Switch – are effectively the equivalent of desktop PCs, except that their software is built on closed systems.

They're "closed" in the sense that their software interface exposes only the functionality the manufacturer wants you to see. Modifying or customizing the OS or internal works of a game console is normally impossible.

A touchscreen device uses the gestures and taps it senses from users as input devices in place of the traditional mouse or keyboard. Touchscreen technologies are the primary inspiration behind smaller form factors for consumers, since there's no need for external input devices.

This, more than just about anything else, has driven the tremendous growth of the tablet and smartphone markets. (It also explains the freakishly agile thumbs of entire generations of young people.)

Devices Without Video Displays

The router that connects devices to a network through either WiFi or ethernet cables contains pretty much the same internal motherboard and network interfaces that you'd find in any other compute device. The big difference is that there's no HDMI, DVI, or VGA video port.

Routers are meant to run autonomously and, when administration is necessary, it'll usually happen through a browser interface across a network.

You launch an admin session with your router by entering its IP address into your browser and authenticating when prompted. In some cases, you can also launch terminal sessions through the Secure Shell (SSH) protocol.

This remote administration model is shared by many display-less device types. Those will include medical (and non-medical) implants or wearables that come with tiny computers built to monitor, report, or even interact with their host environments.

(In the context of wearables, I would at least briefly discuss smart watches here but, for the life of me, I can't figure out why anyone would want one.)

Display-free computers are also embedded in medical devices, appliances, cars, logistics fleets and industrial machinery. All of those embedded computers are components of the growing internet of things.

Chapter 8: Security and Privacy

Let's get some administrivia out of the way right off the top. You'll no doubt recall how security and privacy were covered like a blanket in the first two chapters (named "Understanding Digital Security" and "Understanding Digital Privacy" respectively). So why are we flogging this certifiably dead horse now?

Because it's not dead. Security and privacy are as or more important than anything else in IT. Most of us don't think about them enough, but it's something you can't really overdo.

As an outstanding IT professional I once worked with would have said: "Paranoid is only the beginning." And besides, there are still some urgent and fascinating topics we haven't addressed.

So we'll spend some time exploring how the core security tools (like authentication controls and encryption) can be applied to solve a much wider range of security and privacy problems. And we'll also go face to face with a couple of significant threats that exist thanks to the very devices we've come to love.

Blockchains

The new-technology-hype machine just loved blockchains when they first came to public attention. There were frequent gushing articles in the media about how this was it: blockchains were poised to change the world, ushering in a golden age of endless joy and fluffy fairy unicorns. Rejoice! Salvation is come.

But despite all that, blockchain technologies are, in fact, a big deal. Before we go there, though, just what is this stuff all about?

A blockchain is a distributed string of digital records used to record and validate transactions. The goal is to maintain a reliable and incorruptible public "ledger" of transactions to secure and improve the way financial and commodity operations are recorded.

The blocks in blockchains are actually data packets containing some identifying meta information (including a timestamp) and a cryptographic hash.

The hash – which is produced by software running on the computer that generates the block – is derived from the unique contents of the previous block in the chain which, in turn, was based on the block that preceded it.

Because the contents of one block are dependent on the state of others, no single block can be modified without leaving behind some obvious and easily traceable evidence.

This explains why it's called a chain, because if any one link (block) is altered, the entire chain will break. In effect, a chain will never be trusted unless it maintains the clear consensus of the creators of all its blocks.

Generating the hashes for blockchains is compute-intensive and can incur significant costs in compute power and electricity.

This is by design, since it all but forces blockchains into the hands of distributed communities, rather than individuals or small groups. This decentralization makes chains less vulnerable to attack and adds robust reliability to the data that's being managed.

Blockchains and Cryptocurrency

Like most people, I first heard about blockchains in the context of cyptocurrencies like Bitcoin and Ethereum. Cryptocurrencies are digital assets that can be used as alternatives to fiat money (that is, the kind of virtual and mutually accepted representations of value found in exchange instruments like national currencies).

Using the funds in a cryptocurrency account, I could pay for goods or services while, in many cases, retaining anonymity. Of course, this very anonymity carries significant risks.

Cryptocurrencies have, for instance, been used to support criminal activities. The people behind ransomware attacks will often demand cryptocurrency payments in exchange for the decryption keys that you hope will restore access to your lost data.

And the contents of large cryptocurrency accounts have been effectively lost when controlling servers crashed (or were forced down) or, in at least one case, when the administrator of a currency worth millions of dollars died without sharing his authentication information.

It's worth noting that the relative value of funds in the account itself – when measured against the ability to exchange them for fiat money – has historically been volatile, unpredictably suffering from violent market fluctuations.

Blockchains and Accounting

Blockchains can solve many of the same old problems addressed by traditional accounting practices. Specifically, integrating blockchain verification into a business's financial processes can provide secure transactions and on-demand access to immutable and transparent records.

The ongoing, real-time existence of such records could possibly remove the need for periodic audits and monthly reconciling.

Many of those same features could profoundly change the very nature and value of contracts – a change that could spill over beyond accounting, in to the practice of law.

Blockchains and Insurance

The potential security and privacy features of properly designed blockchains can also create efficiencies and value in the insurance industry.

For one example, having a single blockchain where all the insurers within a particular market can reliably share their customer account information can help reduce claims fraud.

Suspicious behavior and multiple claims for a single event will be more readily visible within a transparent and highly accessible system that includes data from all participating parties.

Being able to reduce administrative duplication can also greatly streamline the processing of legitimate claims.

You'll appreciate this when you consider how a victim's insurer will often process their customer's claim using similar steps to those used by the insurer you're claiming from. But if both companies are able to openly share their data, the process can be unified and, even better, automated.

Perhaps most significantly, the delivery of healthcare can be enhanced and made more efficient if critical personal records can be safely and instantly accessed. And – you guessed it – blockchains can be helpful here, too.

What kinds of automation are we talking about? Well, going back to accident claims, a "smart contract" is software that regularly checks for changes to the status of associated objects. The simple mouse click approval of an insurance appraiser, for instance, could set into motion all the events necessary to pay a claim, notify all related parties, and update existing records.

Maybe – just maybe – insurance isn't as boring as people think.

Multi-Factor Authentication

Passwords are terrible things. Sure, we can't just leave our devices and online accounts open to anyone. But who decided that asking people to memorize long strings of meaningless text (like sIIkdm^&sv234LKi) was the solution?

Sure, you could choose easy-to-remember passwords like mysecret or this clever variation: mysecret22, but anything that's that easy to remember is equally easy to guess. And double that if you're using the same password for multiple accounts. In other words, that kind of protection is just not worth the effort.

There are, by the way, two ways to improve your passwords:

• Use a password vault to generate and safely store insanely complex passwords that you won't need to remember: you can just copy and paste them into any login pages you visit.
• Use long (15-20 character) passwords that incorporate memorable, but unconnected, words. Something like:

house-seventy-warfare-calf.

Mathematically speaking, it's highly unlikely that anyone will have the compute power and time needed to crack that one. And it's not so hard to memorize.

But when it comes to particularly sensitive sites – like the ones where you do your banking – even good passwords aren't good enough. For that reason, more and more organizations are incorporating multi-factor authentication (MFA) into their security profiles.

A website or application configured with MFA requires you to present more than one kind of evidence that you are who you claim to be. One could be based on something you know, and another could be evidence based on something you have.

"Something you know" could be a password, while "something you have" could be a standalone MFA device or an app running on your smartphone.

It'll often work by having the application send a short-lived code via instant message to a preset phone number. You'll be expected to enter the code onto the authentication login page.

Federated Identities

Once you've got the basics of authentication out of the way, through strong passwords and/or MFA, there's the question of authorization. In other words, what resources your logged in account will be able to access.

Individual systems will control users through some kind of access controls. Microsoft Windows, for example, uses Active Directory, Linux has object permissions, and cloud providers like Amazon Web Services can apply roles and policies.

But if you want your users to be able to move between services without having to log in to each service individually, of if you would just prefer not to have to manage authentication at all, you can implement a federated identity.

You've probably already experience federation without even knowing it. Logging into a third party web service using your Google account is one form of federation.

The service integrates its authentication system with a federation provider using an identity technology like Security Assertion Markup Language (SAML) or OAuth. When you accept the terms and log in, the provider will share just enough of your identity information with the third party service to enable an account.

Digital Surveillance

Because it can both protect you from harm and also invade your privacy, surveillance is a two-edged sword. But digital surveillance is a two-edged sword that's a whole lot sharper. Let me explain why that is.

Closed circuit video cameras have been in use within security systems since at least the 1930's, but they really did only one thing: record images that were usually stored locally and then, after a few days or so, overwritten with new recordings.

That was helpful but, to be useful, you would need to physically get to the tape and then laboriously search through, find, and view any images of interest.

Digital surveillance cameras are certainly cheaper than their analog equivalents, much easier to physically hide, and easy to access through networks.

But there's also a lot more you can do with digital video feeds. You can, for instance, configure email alerts whenever the camera detects motion. Or you can redirect a video stream to cloud services (like Amazon's Kinesis) where it can be integrated with your data analytics and machine learning operations or interpreted in near real-time by an object and face recognition service (like Amazon's Rekognition).

All of those tools can be used in the service of both positive and harmful goals. The fact is that there are now countless millions of such cameras deployed around the world that are, in many cases, connected to large-scale surveillance operations. At the very least you should be aware of the potential and risk such technologies present.

Backdoors

A backdoor is a hardware or software-based vulnerability that was intentionally built into a device or the operating system that runs it.

In some cases, the backdoor exists with the full knowledge of the customer, as it was intended to enable remote support or the automated installation of patches and updates. But that's not always the case.

Governments, government-associated companies, and criminal organizations have been caught shipping sensitive compute and networking devices with dangerous backdoors. Such vulnerabilities have been used to bypass encryption protection to monitor communications, steal research data, and harvest authentication information.

Backdoors can take the form of active malware that collects local data and then sends it to remote attack servers, or passively permit remote logins through insecure network environments.

Protecting yourself from backdoors requires defenses on multiple levels, including:

• Careful vetting of potential hardware vendors (taking into account their home countries and associations)
• Regular monitoring of reliable technology information sources for news of new vulnerability discoveries
• Careful monitoring of your devices' network activities
• Regular patching of your networking and compute systems
• Blind, stupid luck in large doses

Chapter 9: Managing Data Storage

We've all been at this 21st Century thing for a while and by now it's pretty clear that data is the big driver of, well, of everything.

Governments build their policies around economic and population data. Scientists build their theories around environmental, physical, and biological data. Businesses build their plans around production, sales, and consumer behavior data.

Data is being generated at rates previously undreamed of. I've read that the sensors on a pair of General Electric GEnx engines on a Boeing 787 Dreamliner generate a terabyte of data each day.

A single network-connected car (like a Tesla) might upload around 100MB of location, performance, and maintenance-related data on any average day.

Multiply that by the millions of such cars that will soon be in use around the world, and multiply that number by the thousands of other devices that are out there, and the scale of the data "problem" should be clear.

Got plans to add your own data to the flood and you feel the need to save and store it, too? You'll need to be able to explain why you need it so you'll know how it should be done. I can't help you with that "why," but I think I can give you some useful thoughts about the "how."

They way you store data will depend on what it looks like as it's produced and how you may need to access it later. Where you store your data will depend on how much of it there is, how deeply you'd be impacted by its loss, and how often you'll need to take it out and play with it. Let's take a look at both of those variables.

Data Storage Formats

Since not all data is created equal, it'll make sense to look for the tools and environments that'll most closely match the work you're planning to do. Here are some options:

Spreadsheets

They may be flashy, colorful, consumer-facing applications, but spreadsheets are no lightweights when it comes to serious data processing.

As we'll see in more detail a bit later, spreadsheets have their limitations. But when it comes to presenting data in visually accessible ways, applying mathematical, statistical, and financial operations to that data, and even integrating remote data sources (like stock market quotes), no other tool comes close.

Spreadsheets can import simple, plain text data from files of just about any size as long as the text can be delimited. That is, breaks between data divisions should be marked by some consistent character.

When you import the data, you can specify the appropriate delimiter. Tabs, hard returns, and commas are common delimiting characters. In fact, the popular acronym CSV stands for comma-separated values.

Here's what a few lines of CSV text might look like. Note that the first row contains column headings. Spreadsheets can easily understand how those should be treated differently.

Year,Volume,Rate,Growth
2015,56,10,15
2020,90,11,(2)
2022,109,8,12

Spreadsheets display their data in cells, which are arranged into horizontal rows and vertical columns.

Functions can be applied to the contents of individual cells or to some or all of the cells in a column or row, and can incorporate values in cells in relative locations.

Data sets within a spreadsheet can be rendered as graphs. Spreadsheets can be used as web forms where users can input data that's saved for future use.

The most popular spreadsheet is probably Microsoft's Excel, which is part of their Microsoft 365 Office Suite. But feature for feature, the open source Calc that comes with the LibreOffice suite is a viable alternative. Google Sheets is a cloud-based spreadsheet solution that may lack some of the feature depth of the others, but is a strong collaboration tool.

Databases

As a rule, databases are not built for visualizing data in attractive and intuitive formats. And, on their own, they're not known for complex mathematical calculations either. But boy, can they handle extra-large data sets and multi-table relationships.

When I say that databases don't really help you visualize your data, that's generally because they're meant to be used "behind" front end applications in multi-tier deployments.

For instance, an e-commerce website will display web pages where users can browse what you've got for sale, add items to a virtual shopping cart, and check out using their payment information.

The web page itself just draws a graphical interface and shows you where to click your mouse, but the information about the items you're selling – including their price and associated images – are probably retrieved from a backend database whenever the page loads.

Similarly, your selections and, eventually, payment information will be written to a different database. The software process that handles your shipping might later consult the payment database for the shipping address.

Databases are there at every stage, but no one will ever actually see them.

Administrating large databases so they're efficient, secure, and reliable takes serious engineering and, in some cases, an enormous amount of money.

Before you build your database deployment, you'll need to know whether your operation requires strong Atomicity, Consistency, Isolation, and Durability (ACID) and support for complex and flexible queries. If it does, you may be looking for a relational database engine like SQL Server, MariaDB, or Amazon's Aurora.

Or perhaps you need speedy performance and would prefer a more flexible schemaless environment (suggesting you'd be better off with a NoSQL solution, like MongoDB or Redis).

SQL, by the way, stands for structured query language – which is an established syntax for using language-like code for interacting with your data.

Counterintuitively, depending on who you ask, NoSQL might not stand for Not SQL. Some prefer to think of it as Not Only SQL instead.

Jupyter Notebook

Don't think you have to consume your data using the same tool that's storing it. It's possible, for example, to import existing data that's stored either locally or at a remote site into an interactive compute environment like a Jupyter Notebook.

The advantage of this kind of setup is that the data can now be addressed within the context of, say, a robust Python programming environment without actually touching – or potentially corrupting – the original source.

The open source JupyterLab is a popular resource for working with large data sets using Python. You can download and build your own JupyterLab or run it remotely through a cloud provider like Amazon's Elastic Map Reduce service, or Microsoft's Azure Notebooks.

For particularly large data sets – especially those that already live in the cloud – an existing cloud platform can make sense.

Data Storage Devices

Although it's not quite this simple, let's say that there are four broad categories of data storage media drives:

• Magnetic tape on open reels, cartridges, or cassettes
• Optical including Compact Disk (CD) and Digital Video Disc (DVD)
• Magnetic media in 2.5 and 3.5 inch drive casing – including spinning hard drives
• Solid state including SSD drives in 2.5 and 3.5 inch drive casing, SD cards, and USB flash drives

A few magnetic tape systems may still exist here and there, but the days of laboriously and slowly copying large data sets to banks of multiple backup tapes – and hoping the backup would actually work – are pretty much over.

Trust me: no one is complaining. CDs and DVDs are headed the same direction. Their maximum capacities are no match for the sheer volume of today's enterprise data needs, and consumers don't make local copies of nearly as many large media files as they once did.

Which leaves spinning magnetic and solid state drives.

Gigabyte for gigabyte, spinning hard drives are probably still a bit less expensive than their solid state equivalents (although the price difference is narrowing), but the performance gains delivered by SSDs are very noticeable.

Some time ago, I realized that I could actually save money by buying smaller capacity SSDs for my personal workstations and laptops instead of larger hard disk drives (HDDs).

Let me explain. The way we use data on our personal computers has changed in recent years. Rather than storing media and software archives locally, we're much more likely to assume they'll be available to stream or download whenever we need them.

For most of us, faster download speeds have made "living in the cloud" easy. So we normally just don't need as much storage space any more. The 500GB SSD drive plugged into my busy workstation is barely half full – even taking into account the dozen or so virtual machines and the many ISO images I've got there. And the drive cost me less than I would have paid for a one or two terabyte HDD.

One of the primary roles of storage is data backup. Rather than physically transferring backups between media, local data archiving – using either SSD or HDD media – generally works by moving archives across networks.

The trick is designing a backup system that automatically provides you with sufficient duplicates of your archives, rotates them through appropriate life cycles (where, eventually, they're retired and destroyed), and all without generating unnecessary network traffic overhead.

Besides backups, you'll also often want to share data among users working throughout your campus.

Two tools for managing both backups and file sharing are network-attached storage (NAS) and storage area networks (SAN). Their similar names suggest they're in the same business. Trust me: they're not.

Network-Attached Storage (NAS)

NAS is a relatively simple and inexpensive way to share files across a local network. It works through a standalone server device that contains multiple storage drives. The drives will normally be configured as a Redundant array of inexpensive disks (RAID) array to provide redundancy and performance benefits.

The NAS device connects to the network over ethernet cabling and uses regular TCP/IP networking. Client machines in the LAN will see the NAS resources through standard file sharing protocols like Server Message Block (SMB) and Network File System (NFS).

NAS solutions can be great for smaller home or office environments, but the fun will quickly fade as you grow. NAS devices themselves are generally not powerful enough to handle too much of a client workload, and working with large files over an ethernet network may slow things down.

Storage Area Network (SAN)

If NAS setups are "relatively simple and inexpensive," SANs are complex and expensive. Not by accident were they designed for large enterprise deployments. As a result of the high end hardware you throw into a NAS system, performance will be great and you'll scale much further.

Rather than ethernet, SANs run through much faster Fibre Channel switches (or, sometimes, the slower iSCSI). They provide block-based storage rather than file systems and are mounted on client machines as local drives.

Data Storage Services

As internet connection speeds have improved, it's become more practical to move at least some data archives to the cloud.

Instead of local backups – which could be lost in a catastrophic event like a fire – data could regularly be saved to online platforms. Once there, you'd have a viable, off-site backup. But, if you wanted, you'd also have access to that data from anywhere on earth. If you work remotely with a distributed team, that can be helpful.

You've probably already own and have even collaborated on documents that live on Dropbox, Microsoft 365, or Google Drive.

For most people, the primary point of interaction for those services is a web browser. But serious data management – or even relatively complex and regular file backups – aren't practical within a browser. So cloud computing providers offer storage and archiving services whose administration can be scripted and automated.

Cloud storage services, like Amazon's Simple Storage Service (S3), provide full archive life cycle management. Data that must remain highly available could, for example, be saved to the S3 Standard storage class.

After a few months – when you're less likely to need the data, but must still retain a copy for regulatory reasons – you could move your archive to the cheaper S3 Glacier class. Data in Glacier is secure and durable, but would take much longer to access.

After a full year you might be able to delete it altogether. Better yet, there are simple ways to automate the way your data moves through its life cycle.

All major cloud providers will have their own comparable data storage services. Naturally, prices and exact service features will differ from one another. And, of course, feature and pricing details will often change.

It may not always be practical to transfer data to the cloud over the internet. Extremely large data sets can take a very long time to upload even using fast internet connections.

Sure, if you're lucky enough to have a fibre optics connection giving you one gigabyte/second, then a one terabyte upload would take only two and a half hours or so (assuming no one else was using the connection).

But what about 100 TBs of data (that'll take you more than ten days)? And what if you only get 100MB/second (more than three months)? If you're uploading jumbo-sized archives weekly or have other uses for your internet connection, then uploading isn't an option.

For such cases, you can still get your data into the cloud, but it'll have to find another ride.

AWS, as it turns out, offers their Snow Family services. Snowball is a large, secure storage device. It can be safely shipped to AWS customers, loaded up with dozens of terabytes of data, and then shipped back. Once back home at Amazon, the data will be directly uploaded to a bucket in the customer's account. Alternatively, Snowballs can be kept on-location and used as edge compute devices.

Snowball's big brother is AWS Snowmobile, a 45-foot long secure shipping container capable of handling Exabyte-scale digital migration.

Snowball's little cousin, AWS Snowcone, is a rugged container the size of a tissue box that can handle eight TB of usable storage, along with the possibility of virtual cloud instances and network connectivity to the AWS cloud. Besides transferring your data, Snowcones can be used as highly mobile edge compute devices in their own right.

Thanks for reading!

I hope you enjoyed the book. If you'd like to pick up a physical copy, you can use this link.

Or if you'd like to check out my other books on penetration testing, software configuration management, and the internet of things, take a look at my author page.

We'll answer these questions and more. You'll get a manager-level crash course in augmented analytics. By the end of this article, you should have a solid understanding of what augmented analytics is, and how you can use it to improve your decision making.

What is augmented analytics?

Augmented analytics is a term used to describe the use of machine learning to automatically analyze data, then provide insights that humans can act on.

It’s sometimes also referred to as “cognitive analytics” or “analytics 2.0.”

How do augmented analytics work?

Augmented analytics rely on machine learning algorithms.

Machine learning is a type of Artificial Intelligence (AI) that allows computers to learn directly from data – without the need for a human to programming them. (That is, there's no need for a human to explicitly tell the computers what to do.)

Machine learning algorithms can automatically detect patterns in data, then use that to make predictions about future events.

What are the benefits of augmented analytics?

There are several benefits of using augmented analytics, including:

1. Automation of tedious and time-consuming tasks. Augmented analytics can automate the more rote aspects of data analysis, freeing managers to focus on the higher-level considerations, including a lot of domain-specific considerations.

2. Improved accuracy. By using machine learning algorithms, augmented analytics can provide more accurate insights than humans alone could. Computers are much better at crunching numbers than humans are. And humans are better at interpreting the underlying meaning of those numbers.

3. Increased speed. Augmented analytics can help managers make faster decisions by providing real-time insights. Imagine automatically refreshing dashboards that you can reference whenever you need to make a decision.

4. Increased scalability. You can easily scale up your augmented analytics to support larger organizations, or incorporate a broader array of data sources.

Which Industries Use Augmented Analytics?

Here are a few examples of industries that have embraced augmented analytics:

Augmented Analytics in Retail

In the retail industry, you can use augmented analytics to automatically analyze customer data and identify trends. If you can anticipate what customers are likely to buy, you can make better decisions about product assortment, pricing, and promotions.

Augmented Analytics in Manufacturing

In the manufacturing industry, managers use augmented analytics to automatically analyze data from sensors and machine data to identify issues and optimize production.

Imagine a factory where you know the exact state of every assembly line, the output of every human worker, or even the power usage of individual machines. Imagine being able to identify bottlenecks or waste through real-time dashboards.

There is a lot of room for applying Augmented Analytics tools to save money. So we see manufacturers like Toyota, Apple, and Airbus all making heavy use of these tools.

Automated Analytics in Healthcare

In the healthcare industry, doctors and managers can use augmented analytics to automatically analyze patient data, identify vital sign trends, and improve patient care.

Healthcare in particular has a lot of room for improvement in understanding medical records, and making better decisions based on available data. Augmented Analytics can give hospitals a big leg up.

Automated Analytics in Financial Services

There is perhaps no more obvious use of automated analytics than in finance. Numbers are an intrinsic part of moving money around between investors and companies deploying that money.

Because finance involves money, it is a magnet for fraudulent activity. Automated Analytics can help banks and other parties prevent fraud. They can also help governments prevent money laundering.

Automated Analytics in Marketing

Marketers can use augmented analytics to automatically analyze customer data to identify trends and improve marketing campaigns.

When you are spending money on advertising, even marginal refinements can improve your bottom line.

In short: Automated Analytics can help managers focus on what they do best, while offloading the grunt work to computers

Thanks for taking time to read my article. I hope you have fun exploring the world of Automated Analytics. If you're looking to harness technology tools, you can learn some new programming and computer science skills here on freeCodeCamp.org.

This article is intended for two audiences: people who need to write self-evaluations, and people who need to provide feedback to their colleagues.

For the purpose of this article I provide specific examples of how to recognize and quantify impact. I hope you can leverage these concepts to write more holistic and thoughtful feedback for your peers.

Key Work From Home Skill: Show Impact by Being Goal Oriented

Here is a list of behaviors tied to impact that you will want to demonstrate (if you are working from home) or that you should evaluate if managing a remote workforce.

1. Completes assignments correctly and on time. Aware of details. Responds effectively to clear direction and follows set procedures.
2. Capable of addressing a wide variety of problems and managing unexpected circumstances.
3. Sees underlying principles, patterns, or themes in an array of related information.

Sample Feedback - Junior Staff

Jennifer assumes full care of procedures, obligations, and activities. She gives prompt answers to inquiries and solves urgent issues, escalating where possible.

Jennifer meets deadlines regularly and yields good results. She also performs well and maintains a positive attitude under pressure.

Jennifer fosters a culture of speed and simplicity on her team and amongst stakeholders (both internal and external) and aligns vision with clear, stretching goals for team members.

She is a respected remote member of the team. She thinks creatively and out of the box and gives guidance to her peers and colleagues that can bring out the best results for our team.

Sample Feedback - Mid-Career Staff

Jennifer completes individual tasks with little or no follow-up and consistently prioritizes tasks through to completion. She regularly exhibits strategic foresight and develops strategic strategy around key objectives.

She leads successfully (both with and without authority) and inspires other stakeholders to do the same.

Jennifer remains versatile by taking into consideration conflicting interests and business objectives, and makes choices on which goals should be prioritized.

Sample Feedback - Senior Member of Team

Jennifer articulates a long-term strategic vision of what our business looks like in terms of global influence and leverages the necessary tools at her disposal for success.

She works successfully and delegates thoughtfully. She ensures that all team members contribute to our goals. She offers advice and instruction on how to handle conflicting interests and priorities.

I have worked with Jennifer for one year and seen her manage her team and resources effectively to deliver results on assignments.

Sample Feedback - Executive

Jennifer is a strong executive. She brings important positive improvements to our company in structure, culture, and organization.

Through her ventures and programs, she generates global influence, and organizational progress that leads to greater profits and impact in the market.

She expresses goals clearly to ensure that the right things are achieved and shows support for long-term strategic projects, successful investments, and supporting members of her team.

Key Work From Home Skill: Show Expertise by Using the Right Skills for the Right Task at the Right Time

Here is a list of behaviors tied to showing expertise that you will want to demonstrate (if you are working from home) or that you should evaluate if managing a remote workforce.

1. Capable of addressing a wide variety of problems and managing unexpected circumstances.
2. Sees fundamental concepts, trends, or themes in a variety of relevant material.
3. Builds rapports by listening to, discussing, and negotiating with internal and external stakeholders.
4. Find ways to collaborate on their projects and career development, and benefit from other perspectives.

Sample Feedback - Junior Staff

John has knowledge and experience that helps his team. He handles operations efficiently with little guidance. He shows a strong ability to think objectively for himself and for our company. He shows critical thought, putting together insight, knowledge and perspective while making decisions or generating results.

He knows how to use various tools, resources and relationships to improve our operations. For example, he builds a long list of target opportunities and is exhaustive in his approach. his efforts lead to more demand for our products and help our sales and revenue accelerate.

John selects suitable methodologies and approaches based on his good judgement and analytical skills and applies methods in his field of expertise to innovate. He frequently suggests new ways to solve challenges. This is valued by the team.

He is an emerging leader, and has shown signs of influencing the team from his work to adopt emerging best practices.

Sample Feedback - Mid-Career Staff

John recognizes the central function of our business, and is able to manage conversations independently and create supporting resources to convey the importance of operating in the market for our organization.

He constantly aims to maximize the efficiency of the deals / projects / products being launched, as described by key results he tracks.

John ensures that all people interested in a project or endeavor are kept informed about changes and plans and concerns. Others' ideas are incorporated and answered without compromising relationships.

He is a team leader. He operates from evidence and data and a deep knowledge base and remains open to innovative strategies for resolving needs or solving problems.

John is a compassionate and thoughtful leader who sets clear and actionable targets and works with his colleagues to achieve goals, build pipeline, and create new business strategies.

Sample Feedback - Senior Member of Team

John has an open mind that everybody appreciates. He seeks feedback from others, both from within our team and from the outside. He knows how to develop an end to end business model and proposal independently which leads to stakeholder satisfaction and customers approval.

This was clearly illustrated in the last deal / project / product in which John was involved, as he handled objections effectively. His peers were left satisfied, and so were our clients.

Considered as a subject matter expert and trusted advisor by clients and internal teams, John has an expertise in leading negotiation discussions and internal meetings which explain the work and vision of our team. John explicitly encourages and persuades others to follow and accomplish goals, and to accept new positions or views.

John has a warm and open approach to communication. He is open and energetic, a deep thinker, and passionate. I find my interactions with him both inspiring and motivating.

He always shows a bias towards action and a willingness to roll up his sleeves to jump into meaningful conversations. His attitude and approach reflect favorably on our organization and how he handles herself has contributed tremendously to our team’s culture.

John is an emerging executive. He integrates executive direction into every decision and consultation. He advocates for and positively represents other programs and services when working with stakeholders and is skilled at attracting and retaining talent to our team.

Sample Feedback - Executive

John leads the team to produce results expertly, while offering a scale-enabling structure and operation. He helps team leaders prioritize work and business opportunities, and offers constructive coaching without micromanaging team members.

John knows how to create a business that's best in class, right for our market. By leading and empowering our company and business unit effectively, John creates major business effects. He is experienced in shaping opinions across levels, both internally and externally, and is competent in handling career growth opportunities for team members.

John is a successful executive because, when appropriate, he intervenes to eliminate roadblocks at all levels of the company, both internally and externally. He exhibits strategic leadership, and is frequently active in initiatives and discussions that drive value beyond his core functional field for our business.

Key Work From Home Skill: Leadership

Here is a list of behaviors that leaders demonstrate and that you will want to emulate (if you are working from home) or that you should evaluate if managing a remote workforce.

1. Lead individuals or programs to achieve desired outcomes and long-term goals
2. Communicate by listening to, discussing, and negotiating with colleagues at all levels.

Sample Feedback - Junior Staff

Sarah explores opportunities for cooperation and learning from other viewpoints on both her core work and in the advancement of her career.

Sarah sets simple, important, demanding, and attainable goals and has aspirations that fit with the organization's objectives.

She makes suggestions that enhance quality and productivity for the team.

Sample Feedback - Mid-Career Staff

Sarah conveys confidence. She demonstrates the capacity to triumph over obstacles to achieve her goals. She encourages and persuades others to seek and accomplish objectives, and takes on new positions or views where appropriate.

In designing practical strategies for achieving goals, she combines other people's ideas and desires, and gets more senior stakeholder recognition and support for those strategies.

Sarah sees the potential in others and takes opportunities to apply and develop that potential.

Sample Feedback - Senior Member of Team

Sarah discovers and develops other influential leaders within the organization, guiding her team's growth and culture.

She interacts and collaborates well with different stakeholders in the business and industry. She connects mission, vision, principles, priorities and tactics to the team's day-to-day work and gets buy-in from other senior stakeholders.

Sample Feedback - Executive

Sarah builds relationships with internal and external leaders to allow business opportunities to come forward, naturally and at accelerated timelines.

She knows the costs and profits of her business unit and promotes talent from within to help improve core operating metrics.

She is a consummate strategist and leads several teams, directly and indirectly managed by other managers.

Conclusion

When evaluating remote staff use the above guidance to think holistically about how these workers are adding value relative to their experience and position.

These sample pieces of feedback can be used to showcase how a remote worker adds value to your clients, your firm, and your culture.

Guides to help you work remotely seem to have swept through the Internet these days.

Do this, avoid that, stay productive, and all those run-of-the-mill tips we’ve already tried out.

So instead of talking about how your remote team needs to schedule out their day effectively or clean their desk, here I am focusing on the hands-on tips you can apply to manage your software development team while working remotely.

First, we need to understand the exact meaning of “remote teams”.

This will allow you to clearly see that having even one individual who’s working separately means you should improve the way in which you manage your team and the process they use for remote team collaboration and task execution.

What are remote teams really?

Today, millions of companies are faced with remote work situations in one form or another. Whether they’re fully-distributed organizations, have a couple of employees working from another part of the world, or just regularly work with external collaborators.

The first thing that comes to your mind when you think of remote teams is something like: different people working from all parts of the world and distinct time zones on a single project and having the same general goals.

In reality, there are many remote teams that have employees in a single country or even in one city. It’s just that everyone works from their own home, without having to go to the office. ? This saves on costs while managing remote teams.

Source

But many companies (software development included) have offices in multiple countries. When your US team members talk to their colleagues from the UK, they’re essentially collaborating remotely. So remote team communication is a must in these cases even if everyone is in the office.

This being said, whether work is done from an office or not is not necessarily a defining factor for remote teams. The problems you could experience lie solely within the communication and work processes.

Best practices for managing remote software development teams across multiple time zones

Since you’re probably used to all the “do this to be productive” rants, I’ll try to tell you what you might not know.

Hire the right people for your remote teams

People are the heart of any team.

So if you hire the right professionals [and humans] for your team, you’re halfway to success.

On the hiring and interviewing process, here are some of the best tips from Shannon Hogue, Global Head of Solutions Engineering, @Karat:

“I always tell hiring managers to start at the review and work back to determine which core competencies should be listed in a job description. Then, to keep things standard across a global network of Interview Engineers, we put those competencies into structured scoring rubrics for each interview.

Here are 3 quick steps for building a structured rubric that can help keep everyone on the same page, be it for hiring or managing:

Identify what competencies are both relevant and important to assess.

For each competency, list observable behavior and results as checkboxes (select all) and/or radio buttons (choose one).

Write down an “algorithm” to help interviewers summarize a completed rubric into a single conclusion.

For example, if Technical Communication is a relevant competency you might list it on the rubric with a specific scale.”

Here’s what Shannon’s example could look like:

Beyond any hiring tip, effective remote work also imposes a question on whether your office workers will adapt to the new lifestyle.

If you’re switching from an office team to a distributed one or even if you want to hire someone who hasn’t worked remotely before, you need to guarantee they’ll fit in the team and work efficiently in your remote team even before day #1.

For software developers the demand is so high that you’ll often bump into this problem:

How do I make sure that who I’m hiring is the perfect fit within our team when they haven’t worked remotely at all? ?

Perhaps tech professionals are some of the easiest to assess beforehand. Check out any of their side projects, if they own a business or do freelancing, and even look at their online presence.

A developer who’s only been putting work for their official job and has no GitHub repositories and projects to boast is clearly not going to be someone you’ll trust right away.

Trust and accountability are the 2 main attributes to consider beyond any hard skills.

These two character traits (which we often ignore because we’re too focused on business results) can predict whether that person will fit within your team culture. And, even more importantly, if they’re willing to stay with your company for many years to come and dedicate themselves to your project.

From then on, managing remote teams is also a matter of how you handle their training.

Onboarding and training

The trial period all HR experts recommend exists for a good reason. It allows you to literally SEE how they work, collaborate, and get along with the rest of the team.

Many managers make the mistake of hiring a developer, letting them start the trial weeks, but not monitoring what they’re actually doing.

Don’t assume that someone who seems skilled and experienced will deliver the same commitment and quality of work to you too. May I say this is the best and only time to be a micromanager and literally oversee all of their actions.

But do offer your help within the first days by making sure they have all resources needed and making the appropriate introductions to your team.

Get them involved in the code writing process for your project as soon as possible so you can get a better feel of how they conduct their work. I’ll also advise you to try a pair programming session during the hiring process too. More on this procedure later in this guide.

Prepare the right work methods

This is largely the manager’s decision. Team feedback is only valid when everyone knows the relevant best practices and benefits of each work method. This doesn’t mean you can't get their input on the methods they’d prefer to use. However, letting them take care of the decision process will only give rise to conflicts and postpone the choice for the time being since there simply will be too many opinions for you to sort out.

“While you do not want to micromanage a remote team, you certainly want to set up clear minimum administrative processes. For a previous remote team I had to manage, this included setting-up very precisely the infamous JIRA workflow.

While it is good to leave a remote team some freedom, work also needs to be clearly organized, so there should be a few processes that have very precise descriptions. In development, the most important of such processes are issues management, and also the build process. This should be described in detail, and followed strictly.

To make sure key processes are respected, there should be set points when work needs to stop until the process is followed. As an example, I gave strict orders to my business team not to look at any incident that is not documented.” - Nicolas de Mauroy, CEO @Open Lowcode

But how close should supervision really be?

Surprisingly – not that in depth.

Keep your devs under your radar only enough to guarantee that all processes run smoothly and you’re able to maintain a trust both ways. Yes, 50% of data breaches across all industries are caused by insiders, but that doesn’t mean you must force everyone to track their time or record their screens.

Developers should really only have to track their time in 2 instances:

1. They/you really want to improve your work processes.
2. They’re getting paid by the hour.

Your role as a manager goes beyond the person who is seemingly in charge of everything. You’re a liaison. That key link between one team member and another, helping your development team collaborate with operations, marketing, design, customer support, and sales while also ensuring all team members are happy and their needs are met.

I’m super disappointed to tell you that only 35% of managers are engaged with their work. How is this going to motivate someone else?

Employees who are supervised by highly-engaged managers are in turn going to be better and happier performers themselves. This will boost your employee retention rates and create that long-lasting company-worker bond most companies can never attain.

This takes me to the importance of knowing when and how to communicate. Both ways.

Craft your remote team collaboration plan

Collaboration goes hand-in-hand with your work method and how you manage tasks. It’s the little things like crafting clear task descriptions and expectations that add up to distinguish a positive collaboration experience from a chaotic one.

Tinjo Thomas, Design Technologist @Coredes Interactive also recommends the following two tweaks you can make:

“1. Always set a due date for each task even though your developers are experts and can make estimates themselves.

2. If you are giving instructions to only one or two people, don't send the message to the whole group. Reach out to every individual in private. People spend a lot of time reading unnecessary conversations on channels.”

On meetings

Meetings are honestly a very controversial topic. Managers love to call meetings. Devs, though, hate most of them but are afraid to voice their thoughts.

To keep meetings relevant, here are some actionable points to keep in mind:

“On a more macro-level, knowledge sharing can happen through a weekly team meeting with a Q&A via video whereby the members of the team can share what they learned from the codebase with each other as well as ask questions. In terms of knowledge sharing through team meetings - you can have several types of meetings.

The overall idea for these is to encourage collaboration in an effort to turn explicit knowledge into tacit knowledge. So you can have brainstorming sessions or meetings geared more towards lessons learned (this can be in the form of a demo or presentation). All done entirely remotely.” - Emil Hajric, CEO @Helpjuice

One huge mistake first-time remote team managers make is filling up entire days with meetings:

“First-time remote team managers need to be quite aware of the differences between collaborative work [like Scrum meetings and software architecture design] and individual focus work [like coding]. It is important to schedule time and ensure everyone can contribute to the collaborative session, but then leave your professionals alone to do what they were hired to do, write great software.” - Pedro Henriques, Co-Founder & CEO @BRIDGE IN

You’ll want to always remember to press that record button during meetings. We’re not necessarily fans of paper trails, but you’ll need them to avoid bothering your team with issues that have already been discussed:

“You should be recording and saving absolutely everything from remote team meetings and strategy sessions. While working remotely, we've noticed a huge uptick in productivity when managers and team members can refer directly to meetings instead of having to play phone tag or email back and forth for simple information. It's a huge boon to workflow across the board and should really become the new normal with the increase in remote team management.” - Alexander M. Kehoe, Co-Founder and Operations Director @Caveni

Keeping up-to-date with all changes is even more important when your team’s distributed. But remote teams have different approaches to how they make sure they don’t just communicate, but also apply and consider every opinion.

Here’s where the real-time vs asynchronous communication battle comes in. Everything’s already been said about this. So the conclusion is:

There’s a right time for each type of communication. A predisposition for certain types of tasks will also make your remote team more likely to go for one method of communication over the other.

Just like the video vs written communication issue.

The Voro team, for instance, has used their office and remote communication tests to conclude that written collaboration worked best for them:

“My #1 tip for managing a remote team is to default to written communication rather than video calls. This forces you to clarify your thinking and communicate succinctly. When we all worked in the same office, it was easy to walk over to someone’s desk to talk through a problem. It would waste a great deal of time to replicate that with a scheduled video call.

Since we are 100% remote, more of our communication has been written, and we try to memorialize most of that in internal documents, so we continue to move fast and share institutional knowledge seamlessly.” - Tomas Hoyos, CEO @Voro

Invest more time in one-on-ones

With one-on-ones, you’re focusing more on sharing the status of specific tasks and individual activities, while every single member on your team gets to make their own suggestions. One-on-ones are commonly held between one team leader and the other members at a time.

Having these twice a month is your best bet to make sure everything gets covered. Holding them weekly is too often as you won’t have time to focus on your other tasks because you’re too busy with meetings.

These one-on-one meetings are also a good opportunity for you to hold reviews and updates on the OKRs. To facilitate the feedback and ideation process, Charles Ahmadzadeh, CTO @Bunch uses checklists during these meetings:

“I like to use checklists with my team. I'll ask each team member to create a checklist each day of what needs to be done before ending the day and before submitting their work. We review these checklists in 1:1s to spot patterns.

I mostly use this to help others around me grow or keep myself accountable to the feedback I receive from the team.

The good thing about them is that they work no matter how senior you are (even astronauts live by checklists). They’re easy to share in 1:1, even if your team is remote, and very simple to use, as long as you’re committed to improving yourself.”

NOTE ON OBJECTIVES AND KEY RESULTS: Team leaders can establish these at a team and individual level by also taking into account suggestions from developers.

For instance, each department or team can have specific targets: to release a feature on time, avoid more than 2 major bugs after a release, keep the error rate low, etc. Individually, each member should take care of one release so that every single team member will know how to handle them.

Here’s an actionable idea: Use TrackJS [or other error tracking tool for your programming language]. Every member can choose a top error and fix it. This allows members to increase their accountability and learn new things to avoid freezing their skills.

Pair programming finally getting the attention it deserves

Now is also the best time to experiment with pair programming. This practice allows two developers to work simultaneously, often on the same tasks. While this is usually done from the same computer, you can switch it up a bit and make use of tools like TeamViewer or screen sharing.

Pair programming also helps distributed teams improve the quality of their code, tighten bonds between colleagues, transfer information and know-how from one dev to another, increase their own accountability, and speed up the entire development process.

Does this make you feel like you’ve been missing out on all its benefits? ? Just keep in mind some best practices if you want to get started with using the pair programming practice on a regular basis.

By using common sense, you probably won’t have both developers working on the same thing for 8 hours straight. Instead, distribute tasks evenly and allot a specific time slot for them to actually work together. Dealing with multiple time zones? Pair programmers within similar work schedules.

Communicate beyond work with your remote teams

We’re all humans. So we can’t work, work, work all the time. How about you implement some activities for people to destress twice a week as part of your remote team management process?

Playing board games online. Having a #watercooler channel on Slack. Finding some creative online team building activities. These are all ideas to help your remote teams unwind and get their mind off anything that’s stressing them out.

Pedro Henriques also shared his thoughts on the importance of implementing this type of downtime while working remotely:

“Remote teams need to deliberately schedule slack time and create informal virtual watercooler moments. In the office, the watercooler or coffee machine is often the place for spontaneous discussions about various topics ranging from a nasty software bug to vent about office politics or even troubles in personal life. These discussions are not superfluous. They’re absolutely critical for team cohesion. Therefore managers should actively promote them, but make sure not to attend.”

Choose the right tools: The basics will do, there are no magical solutions

I’ve seen so many “best tools for remote work” lists the Internet could really crash for good.

Nah, but really, stop looking for the best apps and miraculous software to somehow save your remote teams.

We already know everyone is using Slack and Zoom to collaborate and the ones who don’t use these are probably doing so because of small issues they had or costs they wanted to lower.

For instance, from all apps I’ve tested for this purpose Google Meet was the fastest and most accurate one but guess what? My clients are using Zoom already. No room to impose another tool.

The key point is not to waste weeks trying to find the right tools for your team. Others have already done this. There are thousands of options out there. All of them are copying features from each other so you end up with the same tool in dozens of versions.

So here’s your stack:

Jira - Slack - Zoom - GitHub - InVision - a bug tracking tool - every developer’s code editor

Test these [preferably the free versions first] and only look for alternatives if you have huge issues that are stilting your workflow. My two cents.

Also, some wise words from Tinjo Thomas:

“Don't try to save money on bug tracking or team management tools. You will know why it’s important when things don’t get done as you expected.”

Know a remote team manager who could use these tips? Share this remote team management article with your network to keep improving remote work processes.

Remote work is rapidly growing in all industries. Some professionals might try to push away this new way of working, seeing it as simply a current necessity. They might not think it's fit for a product manager who’s constantly managing team members, strategies, client and partner communication, and upcoming challenges.

But let me tell you this:

Product management can be even more effective when done remotely.

No need to fear going for this career path.

I’ve put together this detailed guide to help you understand the challenges of remote work and its implication on your product management career.

First, let's have a look at a remote product manager’s core duties.

What it means to to be a remote product manager

I’ll just say this from the start:

There’s not much of a difference between what a remote product manager does when compared to someone who works from the office.

In fact, you’re probably already doing part of your work remotely. Like conducting user interviews or holding meetings with your stakeholders.

Your role in remote product management

So what does a remote product manager even do?

Different companies and products have specific needs. Your daily activities as a product manager or owner can be similar though. Remote PMs have the exact same roles as their office-based counterparts.

Here are some of the duties a product manager has:

• Lead the development of the product and its strategy across multiple lifecycles
• Discover user requirements through user interviews
• Develop the positioning and vision of a product
• Use a variety of tools to produce wireframes and mockups, gather feedback, and deliver results
• Establish clear timelines and feedback patterns
• Define quantitative and qualitative metrics and use these to evaluate the success of the product and review delivered work to ensure alignment with the specifications
• Build actionable user stories
• Create and manage the product roadmap
• Maintain consistent client and stakeholder communication
• Partner with other teams to ensure a unified product development and delivery process
• Lead the product team and take full responsibility for its actions
• Create new feature announcements and content

Who hires remote product managers?

To find a new remote job and make the first step towards this ideal lifestyle, you can simply look under the Product category of these remote-exclusive job websites:

• Dynamite Jobs
• DailyRemote
• Working Nomads
• Remote Age
• Remote Global
• Remoters
• Jobspresso
• NODESK
• letsworkremotely

Good news is that more and more companies are starting to consider switching to distributed teams and even accepting remote workers for highly skilled professionals. If you need an overall look at some of the companies who are hiring remote product managers, here’s a list to start with:

• Buffer
• Canva
• MailerLite
• Close.io
• Convergys
• ConvertKit
• Hotjar
• MeetEdgar
• Pearson
• Dell
• DigitalOcean
• Glassdoor
• Pluralsight
• Toptal
• VMware
• Zapier
• Apple
• Automattic

Keep an eye on their job boards to see when a new opportunity pops up.

Note: You can always just ask your current employer to let you work remotely or reach out to companies you’d love to work for and inquire into any available positions.

Why working remotely might not be right for your product management career

Just like the skills of an in-office and remote product management expert are similar, there are a couple of personality traits and habits that will influence whether or not you’re the right fit for working remotely.

To get it out of the way, it’s safe to say that bad communicators will never make it as remote employees. These are those people who can’t take feedback, always have to contradict other team members, and just don’t want to answer your inquiries on time.

Product management is fast-paced. ? Especially in small teams who are commonly the ones looking for remote members anyway. So communication needs to be done in detail and accurately even when you’re not face-to-face with the rest of the team.

There’s no time to nudge people into sharing their thoughts or asking questions when they don’t know how to move forward with a task. Even more so, the product manager should be the one to tie together all team, user, client, and other stakeholder communications.

That’s why most remote product management jobs start with “X years of experience in a cross-functional product design or product management role”. Companies want their product managers to be dependable. Sometimes even more than what they expect from the rest of their team since they’re literally handing over the product to you.

If you’re a constant slacker who’d rather scroll through Reddit for hours than create one more ticket, you’re better off somewhere else.

What are the skills you’ll need to land a remote product management job

A product manager in general needs to have some of the most varied soft and hard skills in a team. Honestly, besides exceptionally good communication and organization skills, the traits and abilities you need to develop are roughly the same as for any office job.

I had a look at 100+ remote product manager jobs on Glassdoor so you don’t have to assume what the top skills for such professionals are. Next are the results that prove my point. ?

Here are some of the most commonly mentioned skills that employers are looking for from their next remote product manager:

• Strong leadership skills
• Organization and prioritization capabilities
• Critical thinking
• Excellent communication and interpersonal skills
• Strong client management abilities
• Ability to manage multiple, simultaneous projects
• Time management and budgeting skills
• Command of diverse product development frameworks, strategies, and/or rapid prototyping solutions
• Ability to troubleshoot customer issues and create detailed bug reports
• Ability to work autonomously
• Passion for working cross-functionally
• Problem-solving skills
• Strong technical understanding of how software products are built
• Ability to collect and structure qualitative and quantitative research that will be used for making product and design decisions
• A clear understanding of key metrics and ways of measuring a product’s success

All in all, the requirements depend on the company you apply to. Some will prefer a strong communicator who is able to work cross-functionally and bring the entire team together. Other employers will have specific needs such as programming language knowledge or stronger experience when it comes to user interviews or focus on a specific stage of a product’s life cycle.

What you’ll need above any of the above skills is product sense. This means you’ll first have to become a subject matter expert and learn all the ins and outs of the product. You’ll then own the entire creative process around generating new ideas, spotting challenges, and creating the whole roadmap along with doing user research, keeping track of metrics, and prioritizing tickets.

Simply, you need a complete product-oriented skillset along with dependable traits that will allow your employer to trust you with their product’s evolution and fixing potential problems.

Where to start your remote product management career preparation

There’s no training in particular that you need to go through separate from your usual product management courses.

However, before jumping into this entirely new work style, I recommend you test it out for a couple of weeks or months. Try a side project or get a part-time job that will have you interact with clients and other team members through multiple time zones. Having this kind of beforehand experience will show you if you actually like collaborating this way, if remote work is suited for you, and where you need to improve.

You might not like using multiple tools to maintain communication or maybe you’re just someone who only feels productive in an office environment. For tiny problems like the latter, try to find an actionable solution. In this case, go to a coworking space daily or set up a fully equipped work desk.

If the issues are huge [like when you’re not productive at all without supervision], there’s really nothing you can do.

If you’re at the beginning of your PM career, being ready for remote work is the least of your worries. You’ll first need to develop and grow your product management skills overall.

Some of these skills include learning how to conduct user research and market analysis, knowing what goes into a product roadmap, understanding how PM frameworks can be used within different types of teams, learning how to set and monitor key metrics, and so many more product-focused skills and roles you’ll put to use for real products.

By far the biggest struggle new PMs always have is worrying about their interview process.

What will I say?

What if I don’t know the answer to a question?

I’m not ready for this job.

All natural concerns for any product manager newbie. There’s so much help out there though. ?

I’m a huge networking fan so the best tip I’d have for you is to connect with experienced product managers who are willing to coach and prepare you for your next interview.

If you’re the shy type, you can always opt for classic LinkedIn messaging instead of face-to-face meetings but bear in mind you’ll have to get over your fears to nail the interview. ?

Remote product management best practices

So you’ve landed a new job or you just want to prepare yourself mentally for what’s to come? I’ve put together my 6 best tips I’ve acquired throughout the years from my own experience and talking to other professionals:

Maintain a regular schedule

One of the top perks of remote work is flexibility to work whenever you want to. Honestly though, as a product manager you might have to mold your schedule according to when the rest of the team is online.

Postponing work [and implicitly communication] can cause serious gaps in productivity. If you don’t know when you have to work on your tasks, when it’s time to dedicate yourself to prioritizing other colleagues’ duties, and when you can still fit in time for feedback and meetings, you’ll end up postponing everything indefinitely when you’re really supposed to be one of the faster responders in the team.

As a PM, others look up to you and expect your input and instructions at all times. So compared to the asynchronous communication that remote companies are used to, sending feedback and holding video conferences in real-time works better for product managers and their teammates.

This takes me to the importance of establishing clear communication patterns and methodologies.

Communicate

You already know the story: communication rules when it comes to remote work. Yet, in this kind of work situation, one-on-one meetings are more insightful than ever. They allow product managers to talk to every single member of a team individually, get feedback, and improve not just their product, but also the employee-company relationship.

Beyond this, you need to understand that working remotely takes extra effort for maintaining those friendly work relations you would in real life.

Managers have a top duty here: to create strong bonds that will eventually build up employee retention.

No ideas? Turn to your team. Hold special opportunities for non-work related activities like get-togethers or just a weekly one-on-one meeting to allow employees to get to know each other better, learn about their hobbies, and make a new friend.

Turn to video tools to bridge the communication gap

Your basic Slack back-and-forth messages or email exchanges create huge information loopholes. They don’t give enough room for clarifying any details and you might end up with results that are totally different from what you’ve expected.

Video, on the other hand, fully replaces your face-to-face office meetings. All remote fun aside, in-person meetings are still an essential part of work for humans as they allow your team members to understand that there are other people who depend on their performance and it’s not just them working from an empty office.

This, in turn, creates accountability, making the entire collaboration process much more effective.

Plus, screen sharing will save the day every single time.

Establish clear review patterns

There’s no such thing as working aimlessly. You need goals and set performance review methods to assess whether the work you and your team puts into a project or product is efficient and spot potential bottlenecks.

To keep everyone involved in this review process, set up a defined timeline for your review tasks and meetings. For instance, decide when and how to hold your daily stand-up meetings. Working remotely you have lots of possibilities for every single review process.

For the stand-ups you can opt for video conferences or use a Slack bot to automate everything and save time since team members can simply write down what they worked on, what they’re taking care of on a given day, and their potential challenges.

The review workflow doesn’t just end here as you’ll probably have to take over and see how and who can help a person with their issues so the product development process can run smoothly.

Address risks ahead of time

While strong communication is a common trait both on-site and distributed teams need to develop, what remote product managers need more than anything are plans. Particularly a highly-detailed action plan for tackling risks before panic installs.

I’m talking here about a common document any team member will access to read and find out how they can manage their own crisis.

Keep your whole team on board

Working remotely is a collective effort. As the person who’s held responsible for any success and failure of your colleagues, you’re in charge of cultivating a desire for remote success within the team.

Here’s the list of everything your team needs to understand and work on before you jump into the world of fully-distributed companies:

• The benefits of remote work
• Communication guidelines
• Risk management techniques
• The importance of feedback and staying connected
• Staying accountable for one’s work
• Get involved in the teams’ overall growth by supporting its cohesion and moving collaboration processes forward
• Developing traits and skills such as discipline, empathy, dependability, and self-organization [in other words, constantly developing themselves on a personal and professional level]

Key takeaways

So yes! You can work remotely as a product manager and be just as effective at your work. No performance limitations here.

Product managers still have the same duties and roles when working from home or an exotic beach in the middle of nowhere.

Daily work is quite similar too but your quality of life is highly improved. Here’s what a day in your life as a remote product manager might look like:

• Take care of urgent tasks or team/client inquiries
• Stay updated with any industry or market changes
• Do some networking ?
• Stand-up meeting time! ?
• Work on the product roadmap
• Take a well-deserved lunch break ?
• Answer some more emails
• Respond to tickets
• Client meeting
• One-on-meeting with one of your teammates
• Sprint retrospective! ?

Do keep in mind that product managers have some of the most diverse days with a constant flux of new challenges and opportunities that need to be covered.

Already a happy remote worker? Share your best tips with fellow product managers or people who are considering this career but have been hesitant thinking it’s not suited for the remote lifestyle.

Plus: seven cheap and simple things you can do to meet your employees’ need for professional development

Train, but not in the way you’d expect

An interesting issue came up in a call last week with Lorraine, one of the CEOs I coach. A promising employee at her company had announced that he was leaving to join a strategy consultancy — not for the pay or the stability, but for the career development. The team member thought the consultancy would build his career more.

“Should I push back?”, Lorraine asked me. “I don’t think career development is something people should expect in a startup.”

You can see Lorraine’s point. Many of the benefits that come with a job at a big company — like long, structured training programs or big budgets to fly off to interesting conferences a few times a year — are expensive. And they’re things that startups that are losing money can’t afford. You can’t ask for that and also expect to become a millionaire from your stock options if the job works out.

So this presents Lorraine with a dilemma. Should she increase her company’s costs, when I already have a limited runway? Or should she resign herself to losing good people who want career development that she can’t provide?

Everyone who thinks about their own future wants to learn new skills and to get more senior jobs in the future. But remember, mature companies have both scale and profitability. It’s much cheaper for them per head to build a structured training program. They can afford to wait longer for the returns. Investors in a startup that may not be around in two years have to think shorter term. But even really confident startups have a little incentive to sink cash into training in their early months.

The route to professional growth in startups is different from the route in big companies

To run the numbers, think about a startup that raises its first finance from friends and family at a valuation of $2m. Spending $20,000 on training promising employees will cost 1% of the company’s value. But if this cost can be put off until the business raises a seed round a year later, at a valuation of $10m, then the training cost will fall to one-fifth of a percent. And if it can be put off until a Series A two years later, when the business is worth $40m, then the cost will be only one-twentieth of a percent of the company’s value. No wonder startup CEOs want to delay spending, no matter how strongly they believe in training and professional development.

Turn your lack of hierarchy into an advantage

But there’s a catch in the lavish programs offered by big business. More training comes with more structure. They’re often also more rigid and hierarchical. If you’re a young and ambitious new employee, good luck debating a business decision with a partner who has been at the organization ten years. In startups, by contrast, many good founder-CEOs follow Paul Saffo’s principle of ‘strong opinions, weakly held.’ They’re willing to consider new ways of doing things, even if those new ways come from someone who joined last week.

Startups have another advantage, too. In a successful venture-backed business, the size of the team can double or triple within a year or two. Then do the same again two years later. Since hiring is so hard, a known insider with proven ambition and determination can have a big advantage.

This presents a choice for employees. For people who think they need to learn a defined body of expertise — especially stuff that requires long, continuous work with an experienced practitioner — a mature company has the advantage. But for those who are open to more opportunistic career development — training on the job, picking up stuff as they go, learning by doing, and simply volunteering for projects — then a startup may have an advantage.

So if you’re a startup CEO like Lorraine, then start by explaining this to everyone you hire before they join. And if an existing employee complains about the career development you offer, then have an honest chat. Help them figure out whether they’ll be happy and fulfilled in this structure. If not, you should put this down to a hiring mistake, and make a mental note to be more upfront in future recruiting.

Make sure both sides understand the bargain that’s being struck

But there’s also stuff you can do, without loading costs on to the business, to make career development less random, more structured, and more helpful to the team. A good start is that employees take the lead on their own career development, not the HR department. If they want the company to invest money in their future, they need to start investing their own time first. They can identify and take advantage of opportunities that are free, like reading blogs, watching TED talks, taking online courses — and proving that they’re more valuable to the business as a result. This gives a company a good reason to think that investing more in them will produce great returns.

So if you’re the CEO of a startup, what can you do to encourage and support this? Here are six approaches that I’ve seen succeed in different companies.

1. Make professional development part of the evaluation process.

Ask people to report what they’re doing outside work to develop their skills. Give recognition and reward to the strongly positive signal of people investing in themselves. When employees know they’re going to be asked about this, they magically start noticing opportunities that might otherwise have floated by.

2. Don’t just evaluate employees’ self-managed professional development.

Support it, too. Make it part of one-on-ones. Use your experience and knowledge to give feedback on which topics people work on, and which content in those areas.

3. Provide low-cost tools with company resources.

Even startups on a shoestring can afford books. Rather than just declare open season and let people buy whatever they want — often a recipe for a shelf full of shiny titles that never get read — invite people to ask others what they’ve found most useful, and buy the books that are upvoted most.

4. Encourage people to share what they’ve learned.

Surgeons have a phrase for it: ‘see one, do one, teach one.’ When someone has just finished studying something, they’re in a great position to give a 30-minute presentation on it to colleagues. This can become a regular fixture, where the company provides the tools and the drinks, and the team provides the learning.

5. Create mentoring relationships inside the company.

There are often things that more experienced people can do in their sleep that newer hires can benefit hugely from observing. To avoid adding a big burden to employees who are already busy, be realistic about how much time you can reasonably ask them to spend on helping others. Start by halving it.

6. Use SAAS platforms for education.

There’s an amazing range of online content available for hundreds of dollars, compared with the thousands of dollars the same content used to cost when taught in offsite training seminars. Learning for twelve minutes a day isn’t just less disruptive than disappearing for a week a year to take a full-time course; it can also produce better learning.

7. Encourage team members to reach out beyond the company to learn.

It’s not just CEOs that benefit from coaching. Almost everyone in the company — sales and marketing, product, engineering, finance — can learn a lot from other people in the industry. As CEO, you can prompt people to identify outsiders they can learn from, and help them figure out how to make the relationship work.

Wrapping up

You may fear that none of these seven suggestions can match what might be offered by McKinsey, Procter & Gamble, or Goldman, Sachs. And you’d be right. But if you spend a couple of hours putting them together, they can combine into a career development program that isn’t expensive but delivers real learning and real personal growth. And that’s something you can tell people about during the hiring process. It’s also something that will draw in talented employees with strong self-motivation and keenness to learn.

About me

I’m Tim, and I run a seed fund out of London that invests in SAAS, platforms, marketplaces, and tools. I’ve backed 50 companies and sat on 19 boards, and I also coach Series A and B CEOs introduced by VC firms in Europe, America, and Asia. Previously, I founded a startup and took it to an IPO on the NASDAQ, and before that worked for The Economist and the Financial Times.

About this publication

Lessons From Startup CEOs is a series of blog posts covering nine of the most important skills I’ve seen in founders who are most successful in scaling their companies. If you’d like to read more of them, please follow.

It’s a rare occasion that companies provide leadership training before you become a manager. A few days or weeks after what was probably one of the happiest days in your recent memory, the day you were offered a position outside of the individual contributor track, you find yourself with a million questions. You feel that you were tricked into signing something without reading the fine print.

That feeling you’re experiencing isn’t new, it’s just that you’ve all but forgotten it. It’s not knowing what you’re supposed to do. It’s being clueless. Because if you think years of writing software trained you to become a manager, research states the contrary. But it’s not the end of the world. Even though your company most likely doesn’t understand the need for formal management training, there is a plethora of information available to you that will make your job easier, and maybe even enjoyable.

When I became a manager I did what I typically do when faced with a challenge I know almost nothing about: I started reading. I’ve read a lot of books, some were good, a few of them were amazing. All of them shaped the way I do my job, and so I thought I would share them for other aspiring or active managers out there.

I’ve curated this list based on several factors:

• The books should cover a broad set of engineering management and leadership topics. It’s easy to find overlapping books. It is much harder to find a diversity of information when you’re very new to leadership.
• They should be written in different eras. The software industry is constantly evolving. It doesn’t make sense to only read about what was happening in the 1980s or 90s.
• Reading order matters a lot. Some books are more specialized than others. The information provided can be thought of as layers that stack on top of each other. If you’re inexperienced, you may start in the middle or the end, and that will basically ruin other books for you.
• Finally, I put a hard limit of 7, just because I think this list is enough to build a foundation layer on top of which you can continue reading and maybe even doing your own research further on.

But enough introduction, let’s see the list :)

Peopleware: Productive Projects and Teams, by Tom DeMarco & Tim Lister

This should be mandatory reading for everyone. Period. No, not just everyone in software, everyone working in a private company should read this book. It’s amazing to me how pretty much all of the problems that people deal with it on a daily basis have already been solved. In the 1980s. If you read just one book from this list, let it be this one.

High Output Management, by Andrew S. Grove

Soon after you make the switch from the individual contributor track to the leadership track you will ask yourself a basic question.

How do I measure my own success?

You used to be able to answer that easily. Now that you’re writing code less and less, and dealing with team dynamics and people problems, how will you or your boss measure your progress and success? This book provides a now universally accepted answer: you are measured through the success of the people reporting to you. That is to say, if your team is successful, then you’re successful. I’m over-simplifying of course but that was my biggest take away from this book.

There are more insightful bits of information such as a walk-through of a production pipeline, how to run meetings, or how to conduct one on ones. This is a classic book that everyone should read even if they’re not interested in leadership roles. It’s that good.

I will say though that this book was written a long time ago. The impression I got was that leaders from that age were more authoritative than what you see in companies these days. This is not a critique, it’s a fair warning. You may read a few chapters and conclude that there’s no way anyone could get away with doing that in a company. There’s a certain nuance to the leadership style described in this book and it is important to understand it.

Influence: Science and Practice, by Robert B. Cialdini

An engineering manager’s job is to ensure their team has everything it needs to succeed. This means managing the interaction between multiple groups of people towards an agreeable outcome.

If you’ve ever tried to convince a friend to move from WhatsApp to Telegram and failed, you’ve made an attempt at exerting influence. You’ll need to do that basically every day, and in my experience, this is a very hard skill to learn. It takes a lot of practice, and there’s really no sandbox mode in which you can fail and it’ll be OK. You will try to talk to someone at some point into doing something, and you will fail, and you or your team will suffer for it.

This book is the definitive guide on how to approach the problem scientifically. A lot of managers seem to think they don’t need to learn how to influence others, particularly their direct reports, as rank is the ultimate influencer. Thinking that will keep you from ever becoming a great leader, in my opinion. Yes, you will probably overrule someone at some point and it will feel terrific while you do it. But if that person ends up hating you for it, you’ve just lost their trust and you will see the consequences of that later on.

Rapid Development: Taming Wild Software Schedules, by Steve McConnell

This is another classic textbook that I wish everyone in software would read. If you’re currently in a company who’s agile and are struggling with unproductive meetings, poor quality code, team members that can’t get along with each other, or stakeholders who push you to leave the office late at night, you will find at least one solution within this book. It’s also a good leadership book as it describes, in what is probably my favorite chapter, a list of things you definitely should not do if you want to have a great team. I still refer to that chapter from time to time, it’s just really good and insightful.

Managing Humans, by Michael Lopp

I loved this book for its humor and insights into some of the biggest tech companies’ cultures. It doesn’t feel like a textbook compared to the others on this list, but that’s what makes it a great fit.

You need to understand that leaders are still human, they’re going to screw up, and the end result may be tragic or hilarious. Or both. That’s a very important lesson to learn, one that took me the longest time, to be honest.

It’s very easy to get caught up in theory while trying to learn how to be a leader. But reality often is so insane that no book can offer you the solution you need. You will end up in situations where you will do everything by the book, and people will still be unhappy. And that’s OK. The stories from Managing Humans will help you understand that.

The Manager’s Path, by Camille Fournier

What makes this book awesome is that it offers a clear and simple description of your responsibilities and goals at every step in the leadership ladder. Most companies don’t even have that information for their own leadership track.

It starts by describing what an entry-level leader should focus on, such as mentoring junior developers. It moves on to engineering managers, senior managers, and even covers what a VP of software should do. I personally have always wondered what a day in a VP’s shoes looks like, and if you have as well, this book is for you.

Beyond just taming your curiosity of what the upper echelons look like, I love this book because it is so realistic. There’s no utopian view of teams and processes here. It’s all ground in the experience of the author. That makes the information within highly applicable in your work environment.

The one minute manager, by Kenneth H. Blanchard

I initially didn’t want to read this after I found it, but I’m glad I changed my mind about it. It’s basically a short story about a manager who is very good at his job, particularly in one area.

I won’t spoil it for you. The information you get out of this story isn’t particularly innovative. There are other books that dive into its topic in more detail. But there’s something really satisfying about it that I can’t really explain. It’s like a GitHub gist that solves a problem in under 100 lines of code.

You can be dismissive of it — there’s no way it could be that simple, right? Or you can read through it carefully and appreciate its simplicity, knowing that it’s not a generalized solution, but still good enough that it’s worth having in your toolbox. It is entirely possible you’ll read this book and think I’m exaggerating, but hey, it’s my list ;)

Radical Candor, by Kim Scott

I can count to 7 just fine, but I really wanted to offer this book as sort of a special addition to my list. I wouldn’t say it’s mandatory reading. Once you’ve gone through all the books above, you should pick this up for a very striking alternative view.

Basically, Radical Candor is a framework on how to relate to people. It’s not really related to software development. You can apply this anywhere, with your friends, with your family, it’s frankly very abstract in that regard.

The author, Kim Scott, has worked both at Google and Apple. She recalls some of her experiences while working there. I found those chapters to be extremely enjoyable, but do take them with a grain of salt. Some of the more recent reports in the media directly contradict some stories from this book. This is why I’m hesitant to recommend it to everyone.

The framework itself makes sense but it’s so hard to actually put in practice. I know that’s not the author’s fault. I wish the world would work in the way she describes, but I don’t think it ever will. Nonetheless striving to achieve even 50% of Radical Candor will make you a better leader. If you have the stomach for 100% then I’d like to meet you in person.

Is that it?

This list is a solid foundation for newbie managers or people who are thinking of becoming team leads or managers. There is so much more to learn. I would’ve appreciated a simple and short list like this myself when I made the step towards leadership. Hopefully you do as well.

The team leader is a key figure in a team of developers. It is a difficult role, involving both technical and social skills. This is the reason why not everyone is tailored for it.

The technical competence usually is not the problem (emphasis on usually). The social skills — let’s call that leadership — are a different story. Technical skills are fundamental, but leadership is not made of code. It is what keeps teams together and motivated.

Bad leadership can compromise not only the success of the product but the team itself. It is not uncommon to see good developers leave because of a bad team leader (a.k.a. the boss). This is not to be taken lightly, and it is a topic worth exploring a bit. So, what is leadership made of?

Leadership is made of respect

Respect is a key point in leadership. A good team leader needs to be respected by developers, first of all like a professional. It is hard to give respect to someone that you consider unqualified. But as stated before, leadership is not only about technical competence. This means that a team leader should also be respected as a person. How to achieve this? The answer is easy: if you want respect, you must give respect.

Respect is a two-way road

Have you ever felt like your boss doesn’t like you? That you are not really part of the team? That everything you do is not enough or correct? That’s the kind of feeling that leads a developer to perform below their capabilities. It is also what brings a developer not to respect the boss. If I feel you don’t like me, probably in the end, I won’t like you either.

Have no bias

Developers are human beings. They have their virtues and vices, they have a life or family outside the office. A team leader should understand and respect developers, both as professionals and as people, all the same way and without bias. Personal feelings should be kept outside the office. It is hard, but a good team leader manages to do it.

Leadership is made of trust

Trust is the foundation of every relationship, and professional ones are no exception. Leadership means trusting your developers.

Your way isn’t my way

Personally, I hate when the boss tells me not only what task to do, but also how to do it in every detail. It’s even worse when they are always shoulder-surfing me, checking what I’m doing — and how I’m doing it. Eventually, I will think that the boss does not trust me. The end result is that I’m more worried about completing the task in the way my boss would like instead of doing it the best possible way (yes, the two things sometimes do not overlap).

To delegate is a win-win

This is why it is important for a team leader to give trust to developers. There is a new feature of the product to implement? Choose a couple of developers and let them handle the feature, from design to implementation to production. Check with them on the status of the work, give suggestions if it is necessary, but let them be free to handle the task.

Delegating the task shows that you trust your developers to do a good job. This lets you focus on the big picture instead of implementation details. They will appreciate it and will try to deliver the best possible work. I call this a win-win.

Leadership is made of rules

Rules are what makes society work. The same is for a team of developers. Anyone can create a set of rules and impose them on the team. Leadership is what makes the team follow the rules because they want to follow them, not because they are obliged to. This is possible only if they trust the leader to create a set of rules based on shared values (trust and respect).

Forced rules are no rules

I don’t know you, but I have hard times following rules I don’t believe in. I cannot do it just because someone imposed them on me, without explanation and without a minimum discussion. I can try to do it, but probably I will bend the ones I find wrong. If everyone on the team bends some rules, the result is that there aren’t rules anymore. Say goodbye to productivity.

Create common ground

Even if you as the team leader have the final say, discuss these rules with the whole team. In this way, there will be a common ground to work on. No more rules bending. These rules can go from coding conventions to the choice of the IDE. It may sound trivial, but it is better to point out even simple things. The team leader should set an example, and should be the first one to follow the rules.

Leadership is made of time

A team leader should value their time; it’s one of the most important things they can give to the team. Avoid unnecessary meetings and save time for your developers. Leadership is being there for a developer asking for your time.

My time is valuable like yours

I can understand that everyone is always busy. But if I ask you to talk — because I need to talk, for some valid reason — and you set up a meeting, I expect that you will attend that meeting. My time is valuable exactly like yours. If you don’t respect my time, this is bad for a couple of reasons. First, if you made me wait for nothing, you wasted my time. I could have done something useful instead. Second, I feel like I’m not worth your time.

Take some time to talk to developers

Set checkpoint meetings to discuss the status of the work. Find a time slot monthly to do a one-on-one with every member of the team, to check how things are going and understand their needs. Always be on time when you schedule a meeting. By the way, remember a simple rule:

• 5 minutes in advance is on time.
• On time is late.
• Late is unacceptable

I said it before, but I can’t repeat it enough: the team leader should always set an example.

Leadership is made of strength

Sometimes things go south. When this happens, a team leader stays strong.

When pieces start falling down…

You as the team leader should be a reference in every situation. Otherwise, when something is falling down, you will trigger a chain reaction that will make the team unable to solve the situation. In the worst case, things go wrong and your developers are blamed because you cannot handle what’s happening. If you cannot handle a difficult situation, chances are you are not suited for the role of team leader.

Don’t panic

Leadership requires the strength to be always a solid reference for the developers, even in an Armageddon-like situation. If your developers see you panicking and losing control of the situation, they will panic too. This is the last thing you want in a difficult situation. Panic doesn’t fix problems. Do yourself and your developers a favour: stay calm. Don’t panic.

Please, don’t panic

Conclusion

The role of the team leader is a complex one, involving both technical and social skills. Technical competence is a must, but it is not enough to be a good team leader. There are aspects that are far more important in the creation of a productive and close-knit team.

As a final note, remember that the trick is to think about what’s best for your team. In the end, the team leader works for the developers, not the other way around.

See you! ?

Working as a front-end developer for nearly two years, I’ve got helpful experience from being part of several web development projects of design/digital agencies.

One obvious but valuable lesson I’ve learnt is that collaborating between each groups with one goal but distinct responsibilities and purposes is not easy. There’re different aspects and levels of difficulties in terms of collaboration, and the specific part of which I’d like to address here is workflow process.

Based on my experience, and with the help from my designer and developer friends, I built a website development workflow designed for small team (5–15 people). The system is composed of Confluence, Jira, Airtable and Abstract. In this article, I’ll share the why and how of the this workflow.

Motivation for building a new workflow

To deliver a customized website without using templates provided by website builders, the minimum talent requirements includes a designer, developer and project manager. After participating in a couple of cases, I had a sense that there was something wrong with the workflow we had: important information was always not aligned both internally between different roles and externally to the client. This inefficient communication was clearly slowing down the development cycle and hurting the team.

So I started to solve this problem.

_Google search workflow great resources: [workflow definition](https://medium.com/eightshapes-llc/system-features-step-by-step-e69c90982630" rel="noopener" target="_blank" title="">Design systems features, style guide resources and <a href="https://www.projectmanager.com/training/define-workflow-process" rel="noopener" target="blank" title=")

I Google searched resources about establishing and improving a workflow. Though I learned a lot from all the great resources, I found nearly none of which was for web development projects in a design/digital agency. It was either a design system or coding guidelines that scoped in design or front-end roles, or a workflow that was built for a team with its own product.

As a result, I decided to cherry pick the parts I needed to solve our problems, and formed a customized workflow for website development.

Problems and goals

Following are the problems I inspected from our existing workflow, and the corresponding improvement goals:

1. Waterfall methodology

waterfall model abstract demo

Problem: Based on my experience, website projects adopt a waterfall approach because clients don’t have a concept of a minimum viable product (MVP). Instead of splitting functionalities from views and modulization, clients tend to think about the site in a traditional page-by-page way, which forces both designers and developers to work page by page in sequence. This causes them to lose a universal perspective across the project. This situation results in lots of back-and-forth redundant revisions between pages.

Goal: Changing the mindset of clients is both arrogant and unrealistic. The goal is to find a way to separate requirements from views as soon as possible and develop in as as modulized a way as possible internally based on page-by-page model.

2. Universal design tokens and components managed by both designers and developers

_design tokens from [Salesforce](https://www.lightningdesignsystem.com/design-tokens/" rel="noopener" target="blank" title=")

Problem: This is a common issue that a lot of articles have shared great solutions to, which mostly propose building a design system that’s managed by style guide/library generators. Though it is great solution, managing an extra site that barely provided edit permission to designers was not appropriate in our situation.

Goal: Except for creating universal design tokens and languages that designers, developers and managers can all understand, build a system that allows everyone to manage the assets in a synchronous way.

3. Accurate, updated progress dashboard

we need an editable and accessible progress dashboard

Problem: Though issue trackers, kanban, and more project management models are useful and practical, most of them failed to act as a straightforward, flexible and friendly progress dashboard. This kind of dashboard would save the team a lot of time because it would prevent team members from actively reporting or asking about the current situation of specific tasks. It also makes managers’ lives easier if they have a clear knowledge of the entire project without too much effort.

Goal: Build a dashboard system that provides edit permission for individuals in charge of specific tasks.

Workflow diagram

Before we dive into the detail introduction of the management tools stack, let’s take a look at the abstract simplified workflow I organized. It’s pretty much just a visualization of a normal workflow that most agencies have, but there’s two points to be noted here.

workflow diagram I designed

1. Developer evaluation

First, when requirements or issues coming from the client are approved and documented by manager, with the exception of sending the task to a designer, they also go to a developer for evaluation. In this process, the developer reviews the specification of the task, checking if there are any rather complicated functions or features included. If it’s positive, the developer could start working on it or notify the designer about the potential problems beforehand.

2. Single source of truth

Also notice that after design deliverable is approved by the client, and before handing the task over to the developer’s hand, it goes through a process of register/modify/delete over design store conducted by the designer. This is because the developer should always be exposed to one and only one source of design store, which contains constantly maintained and updated assets ready for development.

Now we can dive into the management tools stack I prepared and see how the tools help us solve our problems.

The tools stack

After experimenting with various options on the market, the stack I’m proposing here is composed of Confluence, Jira, Airtable and Abstract. In addition to basic introduction and few key application examples, I’ll not cover all the details of using the tools.

_[ABEM](http://atomicdesign.bradfrost.com/" rel="noopener" target="_blank" title="">atomic design and <a href="https://css-tricks.com/abem-useful-adaptation-bem/" rel="noopener" target="blank" title=")

Note: the system assumes that the development team adopts the atomic design methodology and ABEM naming system.

1. Confluence

Role: information and resource center

Though it’s intimidating at first, Confluence provides a powerful workspace that’s easy to organize, and it has tons of features, integration of apps, and customized templates. It’s definitely not a universal solution to all problems, but it’s perfect for documentation of specifications, requirements, meeting notes and more.

Therefore, Confluence in this stack works as an information and resource center, which means every related link and detail about this project should be documented properly in here.

My favorite advantage of Confluence is the ability to customize document templates. This feature makes it really convenience to standardize the workflow.

developer evaluation stage

Example: Component functionality review

I mentioned the developer evaluation process above, which is actually a complicated job. This is because this process includes basic information of the component, a developer’s FSM review (if necessary), FAQ space and more. But the flexibility of the template and tools Confluence provides makes this super easy. Just build a template in configuration settings and you’re good to go.

custom template for component review in Confluence

2. Jira

Role: issue tracking and action type management

Also a member of the Atlassian family, Jira is a super powerful issue tracking and project planning software. My favorite part about it is making customized issue workflows. Since there are tons of great tutorials on how to utilize power of Jira, the only thing I want to point out here is using issue type as mentioned below.

designer update design store

Example: Update developer on changes of design store by issue type

To ensure that developers are building the components based on correct design views, they need to be notified whenever something in the design store is being updated, which includes actions like register, modify and delete. So, as a component is updated, the designer should open an issue with the responsible developer assigned and the correct issue/action type selected.

_[Jira issue types function](https://confluence.atlassian.com/jiraportfolioserver024/configuring-initiatives-and-other-hierarchy-levels-934716034.html" rel="noopener" target="blank" title=")

3. Airtable

Role: component management and progress dashboard

Airtable, a mixture of spreadsheet and database, is the thing that makes this stack work. There’s two amazing features that support my workflow: four types of view transition in single table and related content linking. I’ll showcase two examples of using these two features here.

developer starts working on the task

Example 1: Component management

How do you manage your component library? We chose not to use a style guide generator, because it’s not accessible for designers to edit. Using the Sketch component library wasn’t appropriate either, because it’s has too many limitations if we tried to use it outside the scope of the software itself.

I wouldn’t say Airtable is a perfect solution, but it’s the easiest and most flexible option I could think of. Take a look at the demo template of the component management table here:

component table

Once a newly registered design view that’s ready to be developed programmatically is submitted to the developer, they would asses the view based on the ABEM system, and register it into the table. There are 9 columns in the table, including:

1. Name: naming of the component in ABEM principle

2. Preview: screenshot or exported image of component

3. Linked page: link to the page contains this component

4. Children component: link to children components contains this one

5. Modifier: checks if there’s style variations (ex: — active, — red)

6. Component category: a general category classification (ex: text, hero, sidebar)

7. Development status: status of development progress (pending, assigned, in progress, complete, in revision)

8. Assignee: developer responsible for this component

9. Atomic level: atomic category of this component (atom, molecule, organism)

The best thing here is that you can reference data in both the same and other tables. This connection of dots prevents things from getting messier as the scale grows. Also notice that you can filter, sort and change views easily.

Example 2: Page development status

Since the assumption here is that we’ll inevitably asses development progress page by page, a table template designed for this purpose is necessary. This table can be a progress dashboard for both internal teams and shared with client at the same time.

page list table

Any information about the page, including deadline, InVision prototype link, assignee, and children component can be organized here. Note that it’s very convenient to document and update design, front-end, and back-end development status at the same time.

4. Abstract

Role: single source of truth and design assets version control

Abstract is GitHub for Sketch assets that saves designers from the hell of copying and pasting files. It’s out of this article’s scope to demonstrate details of managing version control flow. The key takeaway here is that Abstract is the design store that acts as the single source of truth. Designers should keep updating master branch to the latest version of confirmed design and then notify developers. On the other hand, developers should only take design assets in the master branch as reference.

_[Abstract branch template](https://www.abstract.com/how-it-works/" rel="noopener" target="blank" title=")

More work to be done

From my own experience, development of the entire project after adopting this new workflow has been at least two times faster than before. It’s not a perfect solution, because it still requires lots of manual labor to update and maintain.

But I think it could be aa helpful reference to website development teams searching for aa better workflow, and hopefully more people can share their workflows in the future!

?中文版連結 (Chinese version) / Originally posted on vinceshao.com

When I started my position as VP of R&D at a growing startup, I thought my biggest challenges would be mainly technical. That is, provide state of the art architecture solutions, conduct code reviews, help solve complex algorithmic challenges and maybe even push some code. That’s because my previous technical leading positions were at small startups, where anyone who can code is needed when dealing with limited resources in a fast paced environment.

A few mistakes straight into the position made me realize very quickly that leading R&D for a team of over 20 people requires a variety of skills. Just to clarify, my team is constructed of five divisions - Front End, Back End, Mobile, Research (mostly machine learning) and QA.

So here are 5 lessons I wish I had known when starting this position:

1. Get to know your people

Understanding what empowers and motivates your team members is powerful. It’s important to remember that every person is different: they need different things, have different communication styles, and focus on different things.

Get to know what motivates each of your team members and what are their passions and career goals. This way you can leverage tasks and responsibilities based on it and maximize productivity and motivation within your team. It’ll also help retain your employees and make them feel more resourceful and self-fulfilled.

I schedule a weekly 1-on-1 meeting with each team leader and a monthly 1-on-1 with the rest of my team. In these meetings, I try to focus mainly on the personal level. Some meetings are very short, and some suddenly take hours. This policy provides me with a constant pulse of my teams’ status and motivation level, allowing me to prioritize who and when needs that extra push and attention. And believe me, there was always someone who needed it.

2. Don’t be the Hero

I truly agree with the saying “You’re only as successful as your team”. As engineers, we are constantly striving to solve complex problems, or in other words, be the hero. As the leader of your team, your job is to have a capable team that can solve any challenge on their own, without you. The more you try and solve for them, the more they’ll rely on you for future challenges.

I found this rule to be very hard to follow since sometimes it feels much more effective to bring out a solution from experience, than have your team research for days on end. However, down the line, it’s proven to be the most valuable lesson I’ve learned. With an independent and capable team, you’ll have much more time and focus to push and improve areas of your R&D that you are mostly capable of.

3. Never be a Bottleneck

The first mistake I made going into the position was to take on a coding task. Coding is my comfort zone which is why I probably fell back to it. Very quickly I was flooded with unexpected top priority issues to deal with, hours and hours of staff, business, and product meetings, not finding almost a single hour to focus on the coding task. Even when I did find some time, we all know coding requires getting “into the zone”, which was hard when constantly getting interrupted. In the end, I was creating a bottleneck in my own team, which almost delayed deployment.

I am still amazed at how many unexpected issues can occur on a daily basis. From HR and external relations to technical and political company challenges. As a leader, you should make sure you’re always available to deal with urgent issues. If you take on tasks such as deployment yourself, you’ll either be risking being a bottleneck or not have enough time to deal with urgent tasks that only you can help solve effectively.

4. Strive to be clear and assertive about estimates

We’ve all been asked questions such as “how long will this take?”, “why is it taking so long?”, etc. Surprisingly, we are often asked these kinds of questions when things are not taking any longer than estimated. We are often asked these questions when our peers either didn’t really like the original estimation or didn’t ask for it in the first place, and now they’re upset, despite nothing going wrong.

Therefore, you must always be assertive about sharing estimates and updating accordingly, even when people don’t ask. It is your job to make it clear, as best you can, what “long” actually is by providing your best view into the timescale of a project, and proactively updating that view when it changes.

Nonetheless, you should also be assertive about getting estimates from your team, and constantly strive to improve their estimation process and instincts. Try to take part in estimation meetings, and don’t be afraid to challenge their input and cut scope toward the ends of projects in order to make important deadlines. Your role in these meetings is to play a tiebreaker and make decisions about which features are worth cutting, and which features are essential to the project’s success.

5. Focus on building processes and strategy

As someone who’s constantly aware of both the high-level business needs, and internal requirements and pains, you’re in the best position for focusing on building external processes.

When I started the position, the first process I focused on was how to conduct code reviews. While this is something that might be expected of a VP of R&D, it is a process my team can probably build better than I can. Since they face the daily challenges of deployment, and understand each other’s styling preferences and coding standards, this was definitely an internal process I could give my team leaders to build while I was focusing on external processes.

Also, by having your team lead such internal processes, you’d increase overall engagement and sense of responsibility that will ultimately lead to more initiatives within your team.

An example of an external process would be the delivery process between the product and R&D team. Each division has its own requirements, culture, and needs. I’d conduct meetings with the VP of Product, and interview product managers and my team leaders to fully understand how to build a process that’s aligned with everyone’s needs and maximize delivery productivity.

Only you have the time and resources to fully understand and see the high-level picture of what’s needed to accomplish such cross-functional external processes.

I have no doubt many more lessons are to be learnt as I continue my journey in my role, so I hope to continue sharing them with you.

If you’ve enjoyed this piece, go ahead, give it a clap ??! You can also share it somewhere online so others can read it too. If you have any questions, feel free to drop me a line in the comments below!

Back when massive open online courses (MOOCs) were new, I started a project to complete the equivalent of an MBA, using free courses. Platforms such as Coursera and edX were making headlines, and when I learned that leading universities (including business schools) were distributing online courses for free, I was inspired to construct a business education rather than pay an exorbitant amount for a degree.

I called the project the “No-Pay MBA” and I blogged about it at www.NoPayMBA.com. I finished my business education in 2016, and wrote a book to guide others interested in doing a self-made MBA (Don’t Pay For Your MBA, HarperCollins, 2017).

Now that I’m working for the MOOC review site Class Central, I’ve been wondering, how would I approach this project today, given the wealth of new courses and tools that are now available?

When I was doing my No-Pay MBA, there were only a few business MOOCs available, so I basically signed up for all of them. Fortunately or unfortunately, so many respected business schools have released free courses — 1,867 business courses are currently listed on Class Central — that it would be impossible to do that today.

Through my studies, I also came to believe that putting together an MBA-level education requires much more than simply ticking through a checklist of course topics. That’s partly because MOOCs differ from on-campus courses in scope, length, and rigor; and partly because creating your own MBA offers a tremendous opportunity to tailor the curriculum to your needs.

My advice: start by learning some foundational business concepts in introductory-level courses, then develop some general business skills, and finally, dive deep in an area of concentration.

Below is a sample course list that follows this advice, drawn entirely from business schools in U.S. News & World Report’s top ten. I cross referenced the U.S. News list with Class Central’s database to find around 90 courses from top ten schools, including the University of Pennsylvania (#3), MIT (#5), Northwestern (#6), the University of California Berkeley (#7), the University of Michigan (tied for #7), Columbia University (#9), and Dartmouth (#10).

I used only courses from these schools not because there aren’t great MOOCs from business schools lower down in U.S. News’ ranking (the full course catalog of the University of Illinois’s iMBA, for example), but simply because A) I had to cut down the number of courses somehow, and B) I wanted to demonstrate that the content for a top-tier MBA is there for the taking.

Note: The curriculum below is a pretty solid MBA program. However, it doesn’t include every topic that might be included in an MBA. Some notable gaps: project management, micro- and macroeconomics, and business ethics. It’s also rather light on operations management.

Another Note: While many of these courses do involve fees, all of them can be audited for free, either in full or in part. If you’re new to MOOCs, you can learn more about them in Class Central’s Beginners’ Guide to Massive Open Online Courses.

Foundational Business Concepts (4 courses)

The four courses in the Business Foundations Specialization from the University of Pennsylvania’s Wharton School of Business are a popular starting place for those new to the world of business. I suggest taking all four.

Introduction to Marketing, University of Pennsylvania via Coursera

Introduction to Financial Accounting, University of Pennsylvania via Coursera

Introduction to Operations Management, University of Pennsylvania via Coursera

Introduction to Corporate Finance, University of Pennsylvania via Coursera

General Business and Management Skills (4–6 courses)

Quantitative modeling, negotiation, project and people management, decision-making, leadership, and communication are important skills regardless of industry. I recommend taking at least one course that covers each of the aforementioned topics, for a total of between four and six courses.

Fundamentals of Quantitative Modeling, University of Pennsylvania via Coursera

Decision-Making and Scenarios, University of Pennsylvania via Coursera

Managing Talent, University of Michigan via Coursera

Leading Teams, University of Michigan via Coursera

Scaling Operations: Linking Strategy and Execution, Northwestern University via Coursera

High Performance Collaboration: Leadership, Teamwork, and Negotiation, Northwestern University via Coursera

Leadership Communication for Maximum Impact: Storytelling, Northwestern University via Coursera

Inspiring and Motivating Individuals, University of Michigan via Coursera

Making Successful Decisions through the Strategy, Law & Ethics Model, University of Michigan via Coursera

Successful Negotiation: Essential Strategies and Skills, University of Michigan via Coursera

Concentration (8–10 courses)

I advise anyone pursuing self-directed education to develop deep knowledge and skills in an area of concentration. This is where you can reap the real value of a self-directed education. Focus on a discrete area, take courses that are rigorous, and put what you learn into practice in real world scenarios.

I’ve listed three possible MBA concentrations that could be constructed from courses offered by the top ten B-schools, excluding pre-packaged options such as the MicroMasters in Supply Chain Management from MIT.

Option 1 — Entrepreneurship

Entrepreneurship is the obvious choice for someone interested in starting a business. Added bonus: if you’re planning on being your own boss you won’t have to defend the value of your self-directed education in a job interview.

Entrepreneurship 1: Developing the Opportunity, University of Pennsylvania via Coursera

Entrepreneurship 2: Launching your Start-Up, University of Pennsylvania via Coursera

Entrepreneurship 3: Growth Strategies, University of Pennsylvania via Coursera

Entrepreneurship 4: Financing and Profitability, University of Pennsylvania via Coursera

Entrepreneurship in Emerging Economies, Harvard University via edX

Becoming an Entrepreneur, Massachusetts Institute of Technology via edX

The Iterative Innovation Process, Massachusetts Institute of Technology via edX

CS50’s Computer Science for Business Professionals, Harvard University via edX

Option 2 — Digital Strategy

With plenty of courses and many opportunities to use your skills in the real world, digital strategy makes for an excellent area of concentration. Being a relatively new field, it may also be easier to get a job in this area without a traditional degree.

What is Social?, Northwestern University via Coursera

The Importance of Listening, Northwestern University via Coursera

Engagement & Nurture Marketing Strategies, Northwestern University via Coursera

Content, Advertising & Social IMC, Northwestern University via Coursera

The Business of Social, Northwestern University via Coursera

Marketing Analytics: Price and Promotion Analytics, University of California, Berkeley via edX

Marketing Analytics: Competitive Analysis and Market Segmentation, University of California, Berkeley via edX

Marketing Analytics: Products, Distribution and Sales, University of California, Berkeley via edX

Marketing Analytics: Marketing Measurement Strategy, University of California, Berkeley via edX

Marketing Analytics, Columbia University via edX

Writing for Social Media, University of California, Berkeley via edX

Content Strategy for Professionals: Engaging Audiences, Northwestern University via Coursera

Content Strategy for Professionals: Managing Content, Northwestern University via Coursera

Content Strategy for Professionals: Expanding Your Content’s Reach, Northwestern University via Coursera

Content Strategy for Professionals: Ensuring Your Content’s Impact, Northwestern University via Coursera

Omnichannel Strategy and Management, Dartmouth via edX

Option 3 — Finance

Finance is a more risky area of concentration, since it can be hard to break into the industry without the right credentials. However, if you already have a foot in the door, there are plenty of finance courses at the MBA level.

Valuation: Alternative Methods, University of Michigan via Coursera

Financial Engineering and Risk Management Part I, Columbia University via Coursera

Financial Engineering and Risk Management Part II, Columbia University via Coursera

Modeling Risk and Realities, University of Pennsylvania via Coursera

Corporate Financial Policy, University of Michigan via edX

Principles of Valuation: Risk and Return, University of Michigan via Coursera

Financial Institutions and Markets, University of Michigan via edX

Financial Decision Rules for Project Evaluation, University of Michigan via edX

Principles of Valuation: Time Value of Money, University of Michigan via Coursera

Valuing Companies, University of Michigan via Coursera

Project Risk Assessment, University of Michigan via edX

The Free Cash Flow Method for Firm Valuation, Columbia University via edX

Thank you for reading!

If software development feels like it is only part of your professional purpose, perhaps you should consider becoming a tech lead. A tech lead could mean different things: a team lead (with no direct reports), or a manager. For example, an engineering manager is a person who is responsible for the team and its projects. That means they are also responsible for peoples’ careers, business growth, deliverables, deadlines, culture, code standards, technical debt, and more.

If you’re a developer, it may not be clear how to get from where you are to a technical leadership position. If your goal is to become a manager soon, you will need to ask yourself why you want this role. Becoming a manager may or may not align with your long-term goals.

I got into software development because I felt more comfortable working with computers than people. But after a while I found myself helping other developers more and more. I enjoyed leading projects and pushing for better code standards. It was an obvious choice for me personally.

For many software engineers, growing as an individual contributor (IC) could be a more appropriate path. Many companies provide IC alternatives to management. These alternatives include a staff engineer, distinguished engineer or fellow engineer. These are very senior technical roles, but no one reports to them as they would to a manager.

the stereotype of a developer: eating pizza, works at night alone, etc, etc

So, do you want to become an engineering manager or another type of team leader? It is important to be honest about what drives you — is it writing code and architecting software? Or, is it helping others to get better results, negotiating deadlines with stakeholders, and convincing your business team that code refactoring is not a waste of time? Your answers to these questions should help you determine which path is more appropriate for your desired outcomes.

If you’re still convinced a technical leadership path is right for you, then you have some work ahead of you. Consider working with your manager or a mentor to have them help you in areas where you are less familiar. Here is an outline of ten key areas of focus:

Stepping up. A true leader can lead without the title or authority. Anyone with a fancy title and enough authority given by the organization chart can give orders. But that’s not what leadership is — it is about what you do.

Therefore, you should start small. Take on more responsibilities during difficult projects. Help your peers by providing feedback in pull requests. Volunteer to present on the project updates. Propose improvements to your team or product workflow. Mentor a colleague.

There are enough opportunities that people either don’t want to see or don’t have enough expertise or confidence to take on. Determine what your colleagues are struggling with, and then step up and do them.

Ownership. When taking on responsibilities, be accountable for everything you do or don’t do. A leader takes responsibility and avoids blaming others for mistakes, missing deadlines, or bugs.

Rather than complaining about a bug someone introduced, just help them fix it and explain how to avoid it in the future. Coming up with excuses doesn’t help anyone. Take the time to deliver what you committed to. If necessary, negotiate a better deadline with your manager. Run a project like your own business and actually care about it.

Recently, one of the tech leads on my team pulled the latest master branch. They saw a big drop in unit test coverage. Rather than complaining, he added missing test coverage. And then presented how to properly check for the coverage and how to write a unit test for complex features. He offered to help if anyone needs it without blaming anyone. The team appreciated that.

Relationships (or politics). Sometimes people misinterpret relationships and call them “politics”. They are the same things. If you don’t want to deal with “politics” then perhaps think again if you want to get into leadership in the first place.

Building meaningful relationships is one of the responsibilities of engineering managers. Management is making things happen through other people. Start building good relationships with other engineering managers. They are your future peers.

There are a few ways to do this, such as presenting at tech talks, doing workshops, and mentoring developers outside of your team. Engineering managers will appreciate the relationships you build through these tasks.

Technical expertise. An engineering manager should be an engineer first. They must have a strong software engineering background and hands-on experience. Becoming one of the strongest engineers on the team is a requirement. A manager who can’t code or doesn’t understand the technical details can’t take part in technical discussions. Once you become a manager, you should always keep your skills sharp enough to be competent at higher level architecture.

Mentorship. Any “really good developer” on the team who’s not a team player is more harmful than helpful. If you’re technically strong, then you should be helping others to get to your level. Pair programming, code reviews, presentations, open source or inner source projects are all great examples of how to get started in mentoring others.

It is rare for someone to come to you and ask you to mentor them. Yet, by branding yourself “the expert” and proactively doing the things mentioned above, people will naturally start coming to you for advice. By helping others you build meaningful relationships and gain people’s respect. Hopefully, they do the same in return and mentor others as well.

Project management. Delivering projects on time is one of the core responsibilities of any leader. If, as a developer, you’re constantly missing deadlines and underestimating tasks, others can’t trust you. You have to be organized and be on top of your tasks.

We all know estimating software projects is hard as there is a lot of uncertainty. However, with the right process, it’s not impossible. Constantly communicate the progress and expectations of the project with your manager or stakeholders.

For example, my team is doing a weekly status report, where the project tech leads have an opportunity to communicate the progress, mention any blockers or raise a major concern of not delivering on time.

Communication. Communicating clearly and concisely is a very important characteristic of any leader. If you can’t explain clearly what you want from your team, then you have failed as a leader before any work even begins.

Communication comes in many forms, including verbal, written and even body language. Always work on improving all of your communication skills.

My team missed a few deadlines because I failed to communicate the requirements clearly and on-time. There were few instances where the lack of communication created confusion on the team who was supposed to do what. I learned that relying on project managers or business stakeholders to explain the project details isn’t working. An engineering manager has to understand the project and then explain it and sell it to the team. And motivate them to want to work on it.

Managing up. Manage your manager (and sometimes their manager). This means constantly communicating with them and managing expectations. Managers rarely like surprises, good or bad. Establish trusting relationships with your manager. Be the go-to person for important and high profile projects, and actually get them done on time and on budget. Then more projects will follow and you can repeat the process.

Conflict and crises. Production issues happen, no matter how many unit or integration tests you have. Yes, you want to minimize the number of bugs your projects have. What matters more is how you handle production issues. A person who starts panicking under pressure is immediately disqualified as a leader in the eyes of others. The team and other managers want to see a calm person who has everything under control, even in the most stressful situations.

A tech lead I used to work with was always calm. There was no conflict or pressure that could make him snap. At least nobody saw him stressed out. When it came to handling a production issue at 3 am, he didn’t disappoint. The issue was fixed in minutes and he showed up to work as if nothing happened.

Another tech lead got so stressed out with the deadline he called in sick on the day we were supposed to launch the feature. He was so anxious, it made everyone else around him uncomfortable to work with him.

Even though these are 2 complete opposites, you can guess which one was more successful as a tech lead.

Vision. For everything they are responsible for, a leader should understand “why”. They are also responsible for ensuring everyone else understands “why” they are working on a project. A leader must explain (often many times) why the project is happening, why the specific people are working on it and how this project fits into the “big picture”. A team has to believe in what they do, only then can they can be effective.

Leadership is not limited to one or two people

Lead the Way Forward, Starting Today

Leadership is not limited to one or two people, so don’t wait for permission, step up today. Be an expert in your field and start helping people when they are stuck. Work on your communication skills, even something minor like technical documentation. Build great professional relationships with your current and potential future peers. Make sure you manage your time wisely and be on top of your projects’ deadlines. And don’t forget that leadership is about people, so genuinely help people grow and do their best job.

You can find me on Twitter https://twitter.com/netxm if you have questions or just want to say “hi”.

Imagine yourselves managing a software development group which consists of very strong programmers. They research and develop one of the most complicated systems known to mankind. They write and write and write code — but eventually, no one wants software versions of that code. No one knows what they actually do day to day. The truth is — no one believes in your group.

This year I became a software development group manager, and I’ve had to deal with 15 employees in 3 teams who, from the moment they arrive until the moment they leave the office — feel transparent.

This is my story, which will illustrate my journey as group lead, from a moment before closing the group, to a well appreciated, needed and a formal authority in the corporation. In this blog post, I will present the group’s background including a few major failure points which occurred in my group — and which can occur in many big corporations.

In my professional past, I developed defense systems as a software engineer, managed autonomous vehicle projects for very big clients, developed autonomous vehicles navigation algorithms and organized events as a data science community manager. Becoming a software group manager, after a very technical professional past, was not an easy task.

The more problems I’ve diagnosed in my group, the more it seemed like a failing startup — there was no money, no client, no motivation. I arrived a moment before shut down.

Before managing my group, I heard of it from many different sources. The main idea was its low productivity and a grand waste of human resources. I had to try and understand — how can such a talented group of software engineers not get their well-deserved recognition? How can they work on such important projects and still seem to be wasting theirs and the company’s time?

The more hours I spent at the office, the more challenges I discovered within the group.

I felt that in order to succeed, I needed to reach out to the people as my first step, before digging further into the projects. I scheduled one-on-one talks with everybody and a separate talk for each team. My intention was to map out my people’s thoughts, strengths and professional interests, in order to be as impartial as I could to gossip.

In the meantime, I started mapping out all the projects the group had been working on at the time, and the picture I got was so scattered and misguided that none of them actually knew what the group’s main objective was.

It was very clear to me that over the years the group has lost its professional identity.

How does a group lose its identity?

When the group was established, it had a very well defined purpose and a great value as it was beneficial for the client. It was in charge of application hosting platforms and applications which were consequently hosted upon these platforms.

After the final version of each of the developed products was released, the spiraling began. The group received projects that had nothing to do with its main purpose, they began supporting technical issues in order to help clients other than the main client, and worst of all — the group stopped releasing versions. Unfortunate managing choices let the group’s already bad reputation deteriorate immensely. Its identity had been completely wiped out.

I immediately devised a plan which consisted of freshening up the teams, teaching proper agile development methodology and strengthening the teams’ leads. On the verge of change, I decided to wait. I had spotted a hostile group member.

It was a programmer who the former group manager put in a team lead position. He got to manage the strongest team of all three. As a team lead, he somehow managed to wreak havoc upon the entire group and be completely unproductive.

I knew I had to deal with the human resources problem before I headed on to solving the other issues.

To sum up some of the damage he caused over the years:

• he set irrelevant assignments to his group members
• disappeared for large amounts of time during the day
• failed to complete his assignments as a programmer and his responsibilities as team lead
• created an atmosphere filled with fear from approaching him
• and a most serious incident — deleted large amounts of code that his team members had been working on for the past year and a half.

I will present one example in greater detail.

During assignment planning, he received an assignment to organize the project’s GIT. Up until that time, the GIT was a great big mess that consisted of two different project versions. Since the team lead (let’s name him Steve for the sake of the example) was the most experienced senior developer, we all agreed it would be most effective if he got the assignment, since it would take him the least amount of time and efforts.

After a month, in which we’d held a few status meetings where he spoke of his progress, we found out that not only did the GIT stay exactly the way it was, but he also ordered his team members to develop upon the older version — thus creating unnecessary integration problems. Everywhere else a worker like that would’ve been immediately fired. Unfortunately, I could not fire him nor take any other disciplinary measure against him.

This story with Steve reminded me of an employee I had as a regional manager for a non-profit organization a few years back. Let’s name her Suzy. As a regional manager, I was in charge of managing my branch managers on the one hand, and on the other hand developing our region’s community and business relations.

Suzy was one of my employees. In time, we realized that being a branch manager lead her to have a strong thirst for power. She took on many responsibilities, treated her branch co-managers as her employees, disrespected her co-workers, and basically bossed everyone around.

No one had the guts to stand up to her for years, because it was a voluntary job, and we didn’t want her to use her rage against the organization. Every action had to be taken very delicately. I’d tried changing the atmosphere, adding bonding activities and a few other approaches. When we saw nothing changed her behavior, I decided to take more drastic measures.

I started actively managing their meetings and assigned more responsibilities to the other managers. I reduced the amount of power she supposedly had, by letting the other managers talk to the contacts she usually contacted. My response to the problem was decentralization, which ultimately led to a healthier work environment.

Going back to Steve — the situation was much more extreme and involved one of the company’s crown jewel projects. It was clear to everyone that Steve, much like Suzy, had a thirst for power. He always requested more people for his team, and it didn’t matter to him what happened next — he just needed to be everyone’s boss. At management meetings, he had to cancel out every idea but his own, and he was always so stubborn he never really listened to his peers.

Since I could not fire him, but could not keep him either, I started looking for a different position for him, far from my group. Simultaneously, I started decentralization. I gave him and his team fewer responsibilities and every new group member was forwarded to one of the other two teams.

Throughout the process, I gave him performance feedback, hoping his behavior would change. His stubbornness and thirst for power prevented any change, and when he felt his power was slipping away he got angry — and so we reached a turning point.

At this point, Steve and I finally had a shared interest. He wanted to seek power elsewhere, and I wanted him to seek power elsewhere. I knew he was one of the group’s many setbacks, but I also knew that dealing with any other setback before I got rid of him would be futile.

Peter Drucker once said: “Management is doing things right, leadership is doing the right things”. This thought is what kept me going in the direction I believed was right. I saw in my mind’s eye an effective software group, and I knew I had to work very hard and convince many people I was going in the right direction — in order to accomplish what I’d started. I chose to fight instead of to comfortably sit in my office and watch the group suffer defeats.

To make a long story short, I got rid of Steve, he moved to another group, and I began mapping the faults and behavioral patterns I was looking forward to dealing with. As we are an R&D group, I started with the technology.

Overcoming a technological gap — a horrible circumstance or a creative adventure?

My next step was to figure out how to overcome technological gaps. When I say technological gaps, I don’t mean switching from Java to Node.js, oh no. We’re talking here about obsolete systems, ancient coding language (worse than hieroglyphs, trust me on that one), an impossible coding environment, no coding methodology, plenty of spaghetti code etc. The systems were implemented in such a way that there was no possible hardware upgrade without just writing everything from scratch.

I began solving each problem separately. I found several solutions to each problem and began prioritizing. For example, the coding environment issue. The system’s coding environment was so old, that even the company that designed it had already been shut down for many years. It was designed in a certain way that even the simple things were in fact very counter-intuitive.

The compilation process, for one, had to be done very carefully. In order to compile one file, a programmer had to find all the files that were affected by it, all the files it might affect and files that are somehow related to it. They needed to find all these files manually, and run them through some other processes, and then when that was finally over with- they needed to wait for a few hours for the compilation to be complete. This was the least of all problems; therefore it was the first issue to attend to.

After massive research and thought, we found an elegant solution, that was immediately applicable. We found a way to automate the previously manual process through a third party application. However, an easy compilation was not the main goal here. My goal was to target all the faulty infrastructures, find and implement a simple solution, yet constantly remind the client we needed more budget to solve the more serious infrastructure issues.

This proved to be a path worth pursuing, since each problem we’ve solved uncovered more issues concerning the architecture and design of the system. This project was doomed the moment the papers were signed, but I hadn’t given up hope.

I started sniffing around, trying to find an interesting project related to our system. I found a project that could solve a very serious problem that we could add to the main system.

This decision provided two things: first, the client understood there was more potential to the group than he had previously thought. And second, that we could add infrastructure upgrades as a part of the new project’s budget.

This had more value than just money — the developers began feeling the new spirit, they felt more needed, and they began working with extra willpower on infrastructure projects, since they knew it was meant to serve a project the client believed in. This led to an effective work environment, where people felt more productive and felt more inclined to share creative solutions.

At this point, you might ask — where is our DevOps team?

Well, to tell you the truth — there was no designated DevOps team, nor was there a QA team, a product manager, a proper project manager and so on. We had to rely on ourselves and to be completely honest — but it was better that way.

In my opinion, distributing DevOps responsibilities among team members is more efficient and leads to better and faster results instead of depending on an outside team to come and rescue us from poor infrastructure. By sharing the responsibilities, I made sure that when integration and deployment time came, my group members would know how to deal with issues. By integration time, they would have come to a point in their research of the code and its behavior where things made sense. Another positive outcome was a shared interest in succeeding, leading to better teamwork and shared efforts.

Each third party application we wrote made the people stronger. They felt committed to the project and wanted to do whatever it took to make sure the system was approachable — for them but also for the next generations.

One of my developers began showing unusual expertise in DevOps related projects. I began quietly grooming him for the position of DevOps team expert. He was always excited to research the system’s code, to write manuals, to suggest improvements, to explore various solutions, to help others. He was a great DevOps mentor, our go-to guy.

Suddenly there was hope. We began solving one infrastructure problem at a time. After building a few relatively small helpful applications, we could embark on an end-to-end solution to something big.

One of our big infrastructure projects was to change the work environment from the old obsolete one to the beloved Visual Studio. This project had been postponed for many years because of low prioritizing. Since we drew infrastructure on our flag — this topic now became the top priority.

This project was the perfect study case. I wanted to prove that my group was both professionally capable and had fuel to see a massive project through. These things were very important to me since the group has never released a version and people outside the group claimed that the programmers were unprofessional.

My thesis was that when a manager creates an environment where every effort is put in order to reach one major goal — people would work hard to reach that one major goal. They will research, they will talk to experts and they will find a way.

The first aspect I was trying to prove, through the new environment platform porting project, was that the group consisted of programmers who had a passion for their jobs. Since they began working on the project, it became very clear in a very short amount of time that they had large amounts of fuel and passion that were just waiting for the right time. They had tons of energy and they put their souls into the project.

My developers began researching, came with different solutions, coded day and night — until we decided on the proper architecture and design. This was not an easy project and we began working on it from point blank. Reviewing the code, throughout the process, made sure that programming skills were not a problem — this cleared out the second aspect of my thesis.

This project’s success had opened the group to a whole new dimension. Their professional self-esteem has begun its restoration process, and they were now very eager to prove themselves worthy to other clients.

I felt improvement, but it wasn’t the group’s time yet. They needed to fill some of the other holes they had fallen into, now and again, over the years. They needed building and empowering. We needed proper marketing, and effective software group marketing comes with, believe it or not, good software products. We needed to work on development methodologies and coding standards, on proper product management and on project milestones before we could show anything to anyone.

Overcoming the technology gap was a difficult step. It demanded many sacrifices from each group member, but it was necessary when it came to leaving the vicious zero-products cycle. It was necessary for improvement and for the upcoming products we were going to sell our future clients.

What is a group’s identity? How do we define our group and how do we create an identity when there is none?

All this and more in my next blog post. Stay tuned :)