I have seen too many founders who are smart, legit, and ambitious get ghosted by investors simply because they couldn't walk through their unit economics.

It's not personal. It's math.

So here it is: Numbers that will either carry your pitch or quietly kill it, explained by someone who has sat through them time and time again, with examples, and no fluff.

Here’s what we’ll cover:

• 1. Burn Rate: How Fast Are You Lighting Your Cash on Fire?

• 2. Cash Runway: How Long Before You Run Out of Cash?

• 3. CAC (Customer Acquisition Cost): How Much Does it Cost to Convince Someone to Pay You?

• 4. Customer Lifetime Value (LTV): How Much is One Customer Worth Over Time?

• 5. Gross Profit Margin: What Do You Actually Keep After Delivering Your Service or Product?

• 6. Monthly / Annual Recurring Revenue (MRR / ARR)

• 7. Churn Rate: How Fast Are Your Users Leaving

• 8. Payback Period: How Long Before You Recover Your CAC?

• 9. Earnings Before Interest, Taxes, Depreciation, and Amortization (EBITDA)

• 10. Valuation: What’s Your Company Worth – and What Supports that Number?

• Real Talk Before You Close That Tab

  • Real Experience

  • Why This Matters

• Conclusion and Final Thoughts

1. Burn Rate: How Fast Are You Lighting Your Cash on Fire?

Burn rate is the speed at which a startup is spending its cash. Basically, how fast are you consuming your venture capital to cover over overhead until you generate positive cash flow from operations? It’s a measure of negative cash flow.

If you’re spending $80K a month to keep the lights on (payroll, AWS, your workspace snacks, and so on), that’s how much cash you’re burning each month. But many startups calculate two different burn rates: gross burn (how much cash you’re spending, ignoring any revenue), and net burn (monthly operating experiences minus any cash you take in each month). Net burn basically measures how fast your cash is shrinking, and it’s often what investors care more about.

Real talk: investors want to know when the plane runs out of fuel before they board. Thats what this metric helps them understand – how fast you’re going through the money you have.

At some point, if a company has a high burn rate, it has to reduce structural costs by cutting expenditures on labor, rent, marketing, and/or capital equipment. The burn rate is an important metric for any company, but it's particularly important for startups that aren't yet generating revenue. It tells managers and investors how fast the company is spending its capital

Watch this video to understand more about Burn Rate.

2. Cash Runway: How Long Before You Run Out of Cash?

Cash runway tells how long a startup can continue to operate at a certain burn rate until they are out of cash. For startups without revenue, you can calculate this by dividing available cash by total monthly expenses. Available cash is defined as the funds that are accessible now or can be accessed at a later time relatively quickly to pay for expenses.

When you’re making this calculation, it’s important to not include any anticipated fundraising and other uncertain sources of capital.

Actively managing cash runway is crucial for startup survival and growth. With a significant percentage of startups failing due to cash shortages, founders need to closely monitor their cash burn rate and runway.

The length of runway needed varies based on factors including the startup’s stage, industry, and milestones. In tighter venture capital markets, startups should plan for longer runways and consider strategies such as increasing revenue, reducing expenses, or raising additional capital. Regularly updating financial models and understanding metrics like the burn multiple can help you make informed decisions to extend your runway and align your growth ambitions with financial stability.

While it’s a simple calculation at face value, a cash runway analysis is nuanced and unique to every startup and can be impacted by a multitude of circumstances.

To calculate this, you just divide your total cash reserves by the amount you’re spending each month. Say you’ve got $250K in the bank and you’re spending $50K/month: 250/50 = 5. So you’ve got 5 months. Not 6. Not “it depends” – 5. That’s your runway.

Investors ask “If we don’t fund you, how long do you survive?” If you don’t know that answer, you're not fundraising – you’re freelancing with hope.

Here is a video that explains cash runway with real world examples and the thought process behind it.

And here’s an article from JP Morgan breaking down cash runway, its importance, and what can you to to maximize it.

Burn Rate vs Runway

So, let’s just make this super clear: burn rate is simply how much you spend each month to run your operation – that is, your negative cash flow. Runway is how many months there are left before your bank balance reaches zero.

So again, why do these numbers matter?

Because burn rate tells you how quickly you need to find more revenue or funding. Runway tells investors whether you are going to still be around by the time they finish their due diligence.

They are not just numbers. They are your survival clock.

Smart founders utilize these metrics to:

• Trim the fat without cutting muscle – know what to focus on and what to let go

• Forecast hiring/fundraising deadlines – know the process and prep for it. Numbers don’t line but they sure can get you ghosted.

• Assure investors you’re not going to come knocking again in 90 days – establishing credibility is key, make an investor realize its not just a hobby, you mean business.

The goal: Extend runway without stalling momentum. Keep the plane in the air, while building a bigger engine.

3. CAC (Customer Acquisition Cost): How Much Does it Cost to Convince Someone to Pay You?

Cost of acquisition refers to the entire cost that a business incurs to obtain a new client or asset. This includes the purchase price, shipping, installation, and marketing costs for the asset acquired. CAC takes into account the total expenditure on all marketing, advertising, and sales for the period, which you then divide by the number of new customers for the period.

In this case, all the upfront costs incurred to purchase a business asset, including equipment or inventory, are part of the cost of acquisition. Cost of acquisition includes:

• Purchase price of the item

• Costs to ship it to its point of use

• Costs to install the item

• Costs to get it up and running (in the case of equipment) or ready for sale (in the case of inventory) condition

• Marketing sales teams salaries

• All sales and consulting marketing expenses geared to get new consumers should all be included

Formula:
CAC = (Total Marketing + Sales Expenses) / Number of New Customers Acquired

Say you spent $10K last month across paid ads, content creation, outbound campaigns, and sales team costs. You onboarded 100 new customers, so your CAC = $100.

But is that good?

It depends on:

• Your pricing model (one-time vs. subscription)

• Your margin (how much of that sale do you actually keep?)

• Your customer retention (how long do they stick around?)

If you’re selling a $20 product once, a $100 CAC is a non-starter. But if that customer brings in $50/month for 12 months, you’ve got a solid return.

Watch for red flags:

• CAC is rising but revenue isn’t

• You’re overly reliant on paid ads (especially if organic/referral is flat)

• You don’t know CAC by channel (averages hide leaks)

A healthy CAC is one that pays itself back quickly and can be improved over time as you optimize funnels and messaging

Here is a video that breaks down CAC for you.

4. Customer Lifetime Value (LTV): How Much is One Customer Worth Over Time?

Customer Lifetime Value is the average monetary value of each customer to your business. LTV takes into account how much a unique customer is expected to spend with your business. It’s an important metric so you know how much new customers are worth to your business over their lifespan as a customer.

Let’s say you charge $25/month. The average customer sticks around 12 months.
LTV = $300.

In this case, if your CAC is $80? You’re in the green. But if it’s $350? You’re basically paying people to hang out (and losing money on them).

Now, let’s connect this to CAC.

Say your CAC is $80. You’re doing fine – your LTV is ~4x CAC. That’s what investors want to see.

Rule of thumb: you want your LTV to be at least 3x your CAC. A 1:1 ratio means you’re barely breaking even, before operational costs and the math stops working at scale. So if you can hit a 3:1 ratio, great – and based off my experience, your business will be much more appealing if it’s closer to 5:1.

And keep in mind that different models can have different thresholds. For example, a SaaS company with low churn can afford higher CACs, while an e-commerce platform might need faster payback. And marketplaces and freemium models may have lower LTV per user, but they can often more easily offset it with volume.

If you don’t know your LTV or can’t defend it with data, it becomes hard to justify spend – and easy for investors to walk.

If you want to know more, this video walks you through the basics.

And here’s a video that beautifully explains the CAC and LTV relationship.

5. Gross Profit Margin: What Do You Actually Keep After Delivering Your Service or Product?

Gross profit margin shows the amount of money a business collects after it pays for all its expenses. It’s usually calculated as a percentage of sales. This specific metric is also referred to as the gross margin ratio.

Companies use gross margin as a measure of how production costs relate to revenue. If a company's gross margin falls because it is making less revenue, it may try to cut labor costs, find cheaper suppliers of materials, or increase prices to increase revenue.

Gross profit margins can also allow a business to measure how efficient a company is, or compare two very differently sized companies that share a common revenue stream or product

If you sell a subscription for $50/month and it costs you $10/month to host, maintain, and support it, your gross margin is 80%.

• Good: SaaS companies often hit 70–90%.

• Bad: If you're below 30%, your "scalable" business will collapse under weight.

Want to know the conceptual math behind this metric and how it differs from Profit Margin? Here is a fantastic video that easily breaks it down.

6. Monthly / Annual Recurring Revenue (MRR / ARR)

Annual recurring revenue (ARR) is revenue a company expects to see from its product and service offerings, calculated over the course of a year. Companies that sell annual subscriptions like using ARR as a sales metric to track what they anticipate making in a year.

ARR tends to be used if companies sell a product or service in the software as a service (SaaS) space, but it can also be useful in terms of streaming services, cell phone bills, and (almost) anything else with a predictable, recurring charge.

ARR is calculated annually, whereas monthly recurring revenue (MRR) is calculated monthly. MRR is useful in that it shows what’s happening on a month-to-month basis. For example, if you change your price in April, you can see the immediate effects of that change in May. MRR also helps track fluctuations in revenue based on outside factors like holiday shopping seasons and economic conditions.

In a nutshell, Monthly / Annual Recurring Revenue = predictable income.

If you’re pulling $20K/month in subscriptions, that’s $240K ARR. Simple.

What investors care about:

• • Is it growing?

    • How fast?

    • And how stable is it?

Here is a founder breaking down the metric and explaining the relationship between MRR/ARR.

7. Churn Rate: How Fast Are Your Users Leaving

The churn rate, also known as attrition rate, represents the rate at which a customer stops doing business with a company. Customer churn is typically expressed as the percentage of service subscribers that discontinue their service subscriptions within a time frame. Churn can also be expressed as the rate at which employees leave their jobs in a given time.

In order for a business to grow its number of clients, its growth rate (which takes into account new customers) must be higher than its churn rate.

The benefit of calculating a churn rate is that it can clarify how well a business is retaining its customers, which is a measure of the quality of service the business is providing and the usefulness of that service.

When a business can see its churn rate increasing from period to period, this suggests that a critical aspect of how it is running the business might be problematic or flawed.

It could be the result of:

• A faulty product(s)

• Bad customer service

• Costs exceed utility to customers

And so on.

The churn rate will indicate to a business that it needs to learn why its customers are leaving, and where it needs to adjust its business. It’s more expensive to attract new customers than it is to retain them, so reducing the churn rate can save a business resources in the future.

Real talk: Say you had 500 users at the start of the month, and you lost 50 by the end of the month. That’s 10% churn – which is high! Annualize that and…ouch. You're not growing. You're replacing.

Make sure you fix this before you fundraise. Or at least explain why churn’s high and what you’re doing to plug the holes.

Here is a video that beautifully explains Churn Rate.

8. Payback Period: How Long Before You Recover Your CAC?

The payback period is a popular tool for determining investment return. People invest money for the purpose of getting it back and generating a positive return on the money they invested. The shorter the payback period, the more beneficial the investment will be.

The payback period does not factor in the time value of money. You can determine it simply by counting the number of years until the principal paid in is returned.

This metric measures how quickly your customer pays you back for the cost of acquiring them. The payback period doesn’t take into account the total profitability of an investment. It’s just concerned with paying the investment back.

There are two common interpretations:

1. Customer-Level Payback: If your CAC is $250 and your customer pays $50/month, it’ll take 5 months to recover the acquisition cost.

2. Investment-Level Payback: You spend $100,000 on a new sales hire, tool stack, or feature. You want to know how long it takes for that investment to generate $100,000 in profit.

Both use the same principle: the shorter the payback period, the less cash you need to float your growth.

If you want a target, aim for 6 months for customer-level payback. Closer to 3-6 is ideal. Long payback periods mean you need deep pockets – or exceptional retention – to stay afloat.

Here’s a video where you can learn more.

9. Earnings Before Interest, Taxes, Depreciation, and Amortization (EBITDA)

EBITDA stands for Earning Before Interest, Taxes, Depreciation, and Amortization. You can think of this as just your company's operating profit if you wanted a very rudimentary way of referring to it.

It’s not flashy. It’s not fun. But it tells investors: “Here’s what we really make once the accounting fog clears.” and “Are we generating real profits from our actual operations?”

EBITDA is what investors look at because it is the best way of comparing apples to apples when considering startups. EBITDA can provide investors a measure of your operational health.

A negative EBITDA for an early stage company isn't going to raise any eyebrows. Just don’t act surprised when someone brings it up. You need to do that math before the pitch. But, if you have a growing early stage company that's moving from negative EBITDA to positive? Now your getting into grown folks business.

Here’s a video that explains the basics of EBITDA.

And here’s another video that explains how investors look at EBITDA and its value in determining your business’s worth.

10. Valuation: What’s Your Company Worth – and What Supports that Number?

Valuation is a focused exercise that determines the value of an asset, investment, or company. So, how much your company’s worth. And the goal is typically to determine whether that value is a fair value.

Valuations can be conducted in one of two ways:

• an absolute valuation, which evaluates a company on its own merits and entirely independently of other factors/companies, or

• a relative valuation, which evaluates the company relative to other similar firms, or assets, in the same sector or industry. This determines if the company, or asset, is worth that much relative to others.

Depending on how the analysis and conclusions are reached, there are a variety of methods and techniques used to develop valuations. And as you’d expect, there’s often significant variability between outputs (or valuations) based on the inputs and context.

While valuations are predominantly quantitatively driven, there’s often a significant subjective influence that come from the assumptions and estimates made along the way. Valuations are also subject to developing situations and events outside of the analysis or the control of the analyst – for example, earnings reports or material news, or economic news – that can result in a change to a valuation stance.

If you’re pre-revenue and you’re saying $30M because a friend raised at that, please stop. Their experience likely has nothing to do with yours.

Valuation = traction + market comps + revenue + momentum + team.

Valuation isn’t just about what you want – it’s about what you can defend.

Startups are typically valued using:

• Comparable Analysis (Comps): What similar companies are worth

• Discounted Cash Flow (DCF): Projecting future cash and discounting it back

• Revenue Multiples: Often 5x–10x for SaaS, but varies wildly

• Precedent Transactions: What investors paid in past rounds for similar startups

But that’s the math.

Here’s the messy truth: Valuation = Traction + Team + TAM (total addressable market) + Timing + Storytelling.

Hard factors:

• MRR/ARR

• Growth rate

• Churn

• CAC:LTV

• Gross margins

Soft factors:

• Founding team’s track record

• Market momentum

• Hype or scarcity

Don’t inflate. Don’t anchor to your friend’s raise. Know your comps. And show why your model is defensible, not just desirable. Inflated numbers make investors run. They don’t correct you – they just ghost you.

There are numerous books written on valuation and each technique could be its own PhD. But my role here is to give you a sneak peak into the metrics.

Here’s a basic video on valuation if you’re interested in a deeper dive.

And here’s a more detailed video course outlining different forms of valuation. Professor Damodaran from New York University is considered to be one of the aces and thought leaders when it comes to valuation. In this video course he explains stepwise and beautifully so you can understand and explore the fascinating world of valuations.

Real Talk Before You Close That Tab

Real Experience

I met a founder once – early days, rough product, but you could tell he actually cared. He wasn't trying to look good. No buzzwords. No "disrupt" talk. Just someone trying to solve something annoying and important.

He walked into the room with a twinkle. Not swagger – just that gentle intensity. We were leaning in.

Then, in the middle of the pitch, someone asked, "So what's your monthly burn?" And I swear to you, he said, "Umm... I think my co-founder has that. I haven't looked in a while."

That was it.

No freak out. No awkward pause. Just... a cluck. Like a window closing in the background.

The product? Still smart. But the moment? Gone.

Nobody was mad. Nobody laughed. We even said thank you. But nobody followed up.

Why? Because it didn't feel like a business. It felt like a maybe.

Why This Matters

I’ve seen so many versions of that same scene play out. It’s never about charisma. It’s not even about the idea, half the time.

It’s about whether the person asking for money actually knows what they’re building. Not the dream, the mechanics. The guts, nuts and bolds of the business. The ugly Excel math nobody brags about on Twitter.

Unfortunately, no simple pitch deck will do that part for you. No co-founder can answer those questions on your behalf.

If it’s your vision, own the math. If it’s your company, learn the cost of keeping it alive.

The rest? The logos, the taglines, the “go-to-market” plans?.... All of that’s just packaging.

And you don’t have to be perfect either. You just have to be in it. Eyes open. Numbers in your head.
Because if you’re asking people to believe in what you’re building, you’d better believe in the scaffolding holding it up.

So yeah, know your CAC. Your LTV. Your margins. Your churn. Not to check some box on an investor’s sheet, but to prove to yourself and the investor that the thing you’re spending your life on…has legs. That it can stand. And run.

And maybe, someday, outlast you. Maybe!

Conclusion and Final Thoughts

I hope this was helpful to you, especially if you’re a founder or aspiring founder trying to build the next big thing. While there a many more ratios and concepts, these are the crux of them.

A lot of other complex ratios and valuations are either built using these metrics or refer them in some way. And each of these metrics could be an article of its own. But I wanted to give you my top 10 run down so that you could get a head start. Numbers are very much a part of the ideation stage itself, and omitting them from your strategy could prove to be a fatal mistake.

I’ll leave you with one last video on How to Start a Start Up with Michael Seibel (Reddit, YC, Twitch) that I hope you find valuable. It lays out, in a crash course format, the mindset of a founder who has been there and done that. The fun fact is that a lot of the themes he speaks of tie in to the metrics here, directly or indirectly.

I hope this gives you a perspective of being on the other side, evaluating your hard work and passion, and I hope it sets you up for success in your next Investor Review.

I look forward to your thoughts, comments, and feedback. If this was helpful, engaging, and informative, do share it – you never know who may need it, or could benefit from it. I wish you all the very best in your funding rounds.

Until then, keep learning, unlearning, and relearning, folks.

But they serve different purposes, feel different to write, and open up different types of job opportunities.

This article will help you understand what each language is, what they’re good at, and how they compare in real-world use cases.

Table of Contents

• What Is JavaScript?

• What Is C#?

• Performance

• Community and Support

• Ease of Learning

• Career Opportunities

• Final Thoughts

What Is JavaScript?

JavaScript is the language of the web. It runs in your browser and powers everything from simple websites to complex web apps. Want to make a button do something when clicked? That’s JavaScript. Want to load data without reloading the page? JavaScript again.

But here’s the cool part, JavaScript isn’t just for the front end anymore. With Node.js, it can run on servers too. This means JavaScript is now used to build entire applications, front to back. You can build APIs, handle databases, run background jobs, and more, all with the same language.

This makes JavaScript beginner-friendly. You can write your first script in a browser and still use the same language to build large systems. It’s fast to start, and you’ll find lots of resources and jobs if you go down this path.

What Is C#?

C# (pronounced “C sharp”) is a language built by Microsoft. It’s often used in combination with the .NET framework to create desktop apps, web services, and enterprise software. If a business runs a Windows-based software, chances are that it’s built with C#.

C# is also a top choice for game development, thanks to its strong connection with Unity, one of the most popular game engines in the world.

Unlike JavaScript, C# is statically typed. This means you need to define your variables clearly and follow more rules. That sounds like a lot at first, but it actually helps you catch errors early and keep big projects neat and tidy.

Now that you know what JavaScript and C# are, let’s compare them across five key areas.

Performance

C# usually performs better than JavaScript, especially when dealing with heavy workloads or tasks that demand a lot from the processor. This advantage comes from the fact that C# is a compiled language.

When you run a C# program, the code is first converted into an intermediate language and then compiled into machine code that your computer’s processor can execute directly. This compilation step removes much of the overhead that comes with interpreting code on the fly.

The result is faster execution and more efficient memory usage, which makes C# ideal for scenarios like handling large files, running complex mathematical calculations, or powering game engines where every millisecond counts.

JavaScript works differently. It’s an interpreted language, which means the code runs line by line in a runtime environment like a web browser or Node.js. Instead of being compiled to machine code ahead of time, the JavaScript engine reads and executes the code as it runs.

This approach offers flexibility and makes it easy to develop and deploy scripts quickly, which is a big reason why JavaScript dominates web development. But because it doesn’t have the same direct link to the machine as compiled languages do, it can struggle in situations that require extremely high performance, like processing massive datasets, rendering complex graphics, or building high-frequency trading systems.

However, modern JavaScript engines have come a long way. Engines like V8, which powers Chrome and Node.js, use Just-In-Time (JIT) compilation to optimize code while it’s running. They can identify patterns and compile parts of the code into machine instructions on the fly, dramatically improving speed compared to older interpreters.

This means that for most everyday web applications, the difference in performance between JavaScript and C# won’t be noticeable. Web dashboards, e-commerce sites, and typical server-side APIs run perfectly fine in JavaScript. But if your project involves raw computational power, game graphics, or real-time simulations, C# will still give you a clear edge.

Community and Support

JavaScript has one of the largest and most active developer communities in the world. Since it's used in almost every corner of the tech world, from websites and browsers to mobile apps and even desktop apps, it has become a go-to language for millions of developers.

That popularity means you’re never alone when learning or building with JavaScript. Whatever problem you run into, chances are that someone else has already faced it and posted about it online. Whether you prefer watching YouTube tutorials, reading blog posts, browsing GitHub repositories, or asking questions on Stack Overflow, you’ll find an overwhelming amount of help and resources to guide you.

C# also has a strong and dedicated community, though it’s more concentrated in the enterprise and professional development space. Backed by Microsoft, C# is consistently updated, well-documented, and deeply integrated with tools like Visual Studio and the .NET ecosystem.

You’ll find a wealth of high-quality documentation, official guides, and forums where professionals share their knowledge. While it might not generate the same buzz as JavaScript in startup circles or web-first communities, C# has a long-standing presence in large-scale systems, corporate software, game development (especially with Unity), and Windows applications. In those environments, it’s not uncommon to find teams of seasoned developers with years of experience in C#.

In short, both languages enjoy excellent support. JavaScript might feel more beginner-friendly due to its widespread usage and endless online tutorials, making it easy to pick up quickly. C#, on the other hand, shines in more structured, professional settings where stability, performance, and long-term maintainability are key. It may not have the same trendy reputation, but within its ecosystem, it’s trusted, respected, and well-loved.

Ease of Learning

JavaScript is one of the easiest languages to start with because it requires almost no setup. You can open your browser, type a few lines of code in the console, and immediately see the results.

This instant feedback loop makes learning feel exciting and interactive, especially for beginners who want to experiment and see quick outcomes. You can start small, tweak your code, and watch it come to life without installing tools, setting up projects, or even leaving the browser. This low barrier to entry is one of the biggest reasons JavaScript has become the go-to language for people taking their very first steps in programming.

But this simplicity comes with quirks. JavaScript is a very flexible language, which is both a blessing and a curse. It often lets you write code that “just works,” even if it isn’t the best approach. While this makes it forgiving in the early stages, it can also lead to confusing bugs down the line.

Small mistakes might not throw errors immediately, and behavior can change in unexpected ways. Anyone who has wrestled with tricky type conversions knows that JavaScript’s freedom can sometimes backfire. This flexibility demands that you develop discipline on your own if you want to avoid messy, hard-to-maintain code.

C#, on the other hand, feels more structured from the start. It enforces rules like defining types, organizing code into classes, and following a clear syntax. This can feel strict and even a little intimidating at first because you can’t just write anything and hope it works.

The compiler checks your code before it runs and points out mistakes early, which can be frustrating as a beginner. But this structure pays off as your projects grow. The rules and strong typing help you catch errors before they become serious problems, guide you toward better programming habits, and make large, complex codebases easier to manage. Once you get used to its style, C# actually makes programming feel more predictable and less error-prone.

In simple terms, JavaScript gives you the quickest path to start coding and see results, which is perfect for learning and experimenting. C# takes longer to warm up to, but it rewards that effort with cleaner, safer, and more maintainable projects as they grow in size and complexity.

Career Opportunities

Learning JavaScript unlocks a wide range of career opportunities because it powers so much of the modern digital world.

Starting with front-end development, you can build interactive websites and web applications that run in the browser using frameworks like React, Vue, or Angular. If you want to go deeper, Node.js allows you to use JavaScript for back-end development as well, which means you can handle server logic, databases, and APIs without ever switching languages.

This flexibility makes full-stack roles especially accessible to JavaScript developers, since you can manage both the front-end and back-end with a single skillset. On top of that, JavaScript extends into mobile app development with tools like React Native and Ionic, letting you build apps for both iOS and Android.

Because nearly every business needs a website or web application, the demand for JavaScript developers is both high and consistent, making it a strong career path for anyone entering the tech industry.

C# leads to a slightly different set of roles, often tied to enterprise or specialized development. It is heavily used in backend systems that run on the .NET ecosystem, where reliability and performance are critical. Many large companies rely on C# for internal tools, financial applications, and large-scale software products that power their core operations.

One of the most exciting areas for C# developers is game development, thanks to its deep integration with Unity, the most popular game engine in the world. From indie studios to major game companies, Unity uses C# as its primary scripting language, opening the door to careers in the gaming industry. Additionally, C# is a natural fit for building Windows desktop applications and enterprise-level cloud solutions, especially within companies already invested in Microsoft technologies.

Both languages can lead to solid salaries and stable careers, but the paths they open are different. JavaScript thrives in web and mobile-focused roles with a steady demand across startups, agencies, and tech companies of all sizes. C# excels in enterprise, backend, and game development, where the projects are often larger, more complex, and built for long-term reliability. Choosing between them really comes down to the type of work that excites you most and the industry you want to grow in.

Final Thoughts

JavaScript and C# are both modern, useful, and in demand. JavaScript is light, fast, and flexible, perfect for websites and small-to-large web apps. C# is powerful, structured, and ideal for Windows apps, enterprise systems, and games.

If you're new to coding and want to see results fast, start with JavaScript. If you're interested in game dev or building strong business apps, try C#.

You don’t have to choose forever. Many developers learn both. Start with one, explore the other later, and you’ll become a well-rounded coder ready for anything.

Join my newsletter for a summary of my articles every Friday. You can also connect with me on Linkedin.

C# provides a rich set of built-in data structures, such as lists, dictionaries, and more, making it easier to work with data differently. Each structure has strengths and is optimised for specific scenarios, so understanding their differences is key to writing clean, efficient, and maintainable code.

In this tutorial, we’ll explore:

• Lists: Your go-to for dynamic, ordered collections where elements can grow and shrink effortlessly.

• Arrays: The efficient choice for fixed-size collections with predictable memory usage and blazing-fast indexing.

• Dictionaries: Perfect for quick lookups and managing key-value pairs with unmatched speed and clarity.

• Stacks: Ideal for last-in-first-out (LIFO) operations, like tracking history or nested structures.

• Queues: Best for first-in-first-out (FIFO) tasks, like processing jobs or managing sequential workflows.

• HashSets: The choice for collections where uniqueness matters and fast lookups are key.

By the end of this guide, you'll understand the differences between these structures and be equipped to choose the right one for your next project.

Table of Contents

1. Arrays in C#

2. Lists in C#

3. Dictionaries in C#

4. HashSets in C#

5. Queues in C#

6. Stacks in C#

7. Common Problems

For some of the following examples, you’ll need the Animal record below:

public record Animal(int Age, string Name, int Legs, string Sound);

Arrays

An array in C# is a fixed-size collection of elements. Arrays are indexed, and their size is set when they are created unlike Lists and other collections. Once defined, the size of an array cannot be changed, making the memory efficient with a low overhead.

Single-Dimension Arrays

Arrays are zero-index based, meaning their index begins at 0, rather than 1. If you’re not familiar, an index is a pointer to help you find an item.

For example, if you have 5 names in an Array, the first name is index [0], and the last name would be at index [4].

Arrays are great in scenarios where low-level performance is critical, as they have very little overhead due to their lack of metadata (additional attached information).

int[] numbers = new int[] { 1, 2, 3, 4, 5 };

foreach(var number in numbers){
    Console.Write(number);
}
//Ouput
// 1 2 3 4 5

In the above example, we instantiate an array with its values (thus giving it a fixed length). But we can assign values after the array is created by using index assignment.

Note: You must still specify the size of the array at the time of creation, as the code needs to know the fixed size of the Array.

// create an empty array of 20 indexes
var numbers = new int[20];

// loop over available indexes and assign `i` 
for (int i = 0; i < numbers.Length; i++)
{
    numbers[i] = i + 1;
}

foreach (var number in numbers)
{
    Console.Write($" {number}");
}
// Output
//  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Multi-Dimensional Arrays

Arrays can also be multi-dimensional (for example, rows and columns), meaning they can hold two values. This makes them perfect for building grid-like structures.

Unlike a jagged array, where each element is an array that can have different lengths, a multidimensional array is a matrix-like structure where each dimension has a fixed size.

// Create a 2D multi-dimensional array to represent a chessboard

string[,] chessBoard = new string[8, 8];

// Creating the starting positions of a Chessboard
chessBoard[0, 0] = "Rook";
chessBoard[0, 1] = "Knight";
chessBoard[0, 2] = "Bishop";
chessBoard[0, 3] = "Queen";
chessBoard[0, 4] = "King";
chessBoard[0, 5] = "Bishop";
chessBoard[0, 6] = "Knight";
chessBoard[0, 7] = "Rook";

chessBoard[1, 0] = "Pawn";
chessBoard[1, 1] = "Pawn";
chessBoard[1, 2] = "Pawn";
chessBoard[1, 3] = "Pawn";
chessBoard[1, 4] = "Pawn";
chessBoard[1, 5] = "Pawn";
chessBoard[1, 6] = "Pawn";
chessBoard[1, 7] = "Pawn";

chessBoard[6, 0] = "Pawn";
chessBoard[6, 1] = "Pawn";
chessBoard[6, 2] = "Pawn";
chessBoard[6, 3] = "Pawn";
chessBoard[6, 4] = "Pawn";
chessBoard[6, 5] = "Pawn";
chessBoard[6, 6] = "Pawn";
chessBoard[6, 7] = "Pawn";

chessBoard[7, 0] = "Rook";
chessBoard[7, 1] = "Knight";
chessBoard[7, 2] = "Bishop";
chessBoard[7, 3] = "Queen";
chessBoard[7, 4] = "King";
chessBoard[7, 5] = "Bishop";
chessBoard[7, 6] = "Knight";
chessBoard[7, 7] = "Rook";


// Print the chessboard
for (int row = 0; row < 8; row++)
{
    for (int col = 0; col < 8; col++)
    {
        string piece = chessBoard[row, col] ?? "Empty";
        Console.Write($"{piece}\t");
    }
    Console.WriteLine();
}

Output:

Rook    Knight  Bishop  Queen   King    Bishop  Knight  Rook    
Pawn    Pawn    Pawn    Pawn    Pawn    Pawn    Pawn    Pawn    
Empty   Empty   Empty   Empty   Empty   Empty   Empty   Empty   
Empty   Empty   Empty   Empty   Empty   Empty   Empty   Empty   
Empty   Empty   Empty   Empty   Empty   Empty   Empty   Empty   
Empty   Empty   Empty   Empty   Empty   Empty   Empty   Empty   
Pawn    Pawn    Pawn    Pawn    Pawn    Pawn    Pawn    Pawn    
Rook    Knight  Bishop  Queen   King    Bishop  Knight  Rook

Jagged Array

Welcome to inception. A jagged array in C# is an array of arrays, where each "inner" array can have a different length.

Unlike multi-dimensional arrays, jagged arrays are not rectangular, meaning the rows can have varying sizes.

A usage example could be building a Calendar app. Below is a basic usage outputting the days of each month in the year:

int[][] daysInMonths = new int[12][];

// Initialize each month with its corresponding number of days
daysInMonths[0] = new int[31]; // January
daysInMonths[1] = new int[28]; // February (non-leap year)
daysInMonths[2] = new int[31]; // March
daysInMonths[3] = new int[30]; // April
daysInMonths[4] = new int[31]; // May
daysInMonths[5] = new int[30]; // June
daysInMonths[6] = new int[31]; // July
daysInMonths[7] = new int[31]; // August
daysInMonths[8] = new int[30]; // September
daysInMonths[9] = new int[31]; // October
daysInMonths[10] = new int[30]; // November
daysInMonths[11] = new int[31]; // December

// Print the number of days in each month
for (int month = 0; month < daysInMonths.Length; month++)
{
    Console.WriteLine($"Month {month + 1}: {daysInMonths[month].Length} days");
}

You should use an Array in:

• Performance-critical applications where memory overhead and speed matter.

• Fixed data sets where the size will not change.

• Multi-dimensional data, for example, graph coordinates (x, y)

Lists

A List<T> in C# is a resizable collection of items of the same type, signaled above by the letter T. It allows adding, removing, and accessing items by index. Unlike arrays, lists grow dynamically as needed.

Commonly used for sequential data, they support LINQ queries and various utility methods for data manipulation.

var animals = new List<Animal>()
{
    new Animal(10, "Dog", 4, "Woof"),
    new Animal(5, "Cat", 4, "Meow"),
    new Animal(2, "Lion", 4, "Roar"),
    new Animal(6, "Giraffe", 4, "Trumpet"),
    new Animal(15, "Red-Panda", 4, "Squeak")    
};

animals.Add(new Animal(2, "Hamster",4,"Squeak"));
animals.Remove(x=>x.Sound == "Meow"); // Remove all squeaking animals

Lists are a highly versatile data structure, where the order of items remains the same order in which they are added or removed (no manipulation. For example, whenever you call .Add() a method on a list, it will append the item to the list, and the order stays the same as before but with the additional animal.

You can modify the data (for example, filter, map, or sort) before sending lists to other areas of your application thanks to the extensive utility methods available in the List<T> class.

Dictionaries

Dictionaries work just like the term we know in the English language.

We have a key (a lookup term) and a value (the mapped object or data). Because of this, you might hear the term 'key-value pair' when referring to dictionaries.

Dictionaries are best used to efficiently retrieve data based on a unique identifier, such as an ID, name, or other uniquely identifying fields. They ensure their unique keys are ideal for scenarios requiring optimal performance without iterative searching.

I recommend using dictionaries when the order of elements is unimportant and you need to represent relationships, such as mapping countries to capitals, products to prices, or people to addresses.

var animalDictionary = new Dictionary<string, Animal>()
{
    { "Dog", new Animal(10, "Dog", 4, "Woof") },
    { "Cat", new Animal(5, "Cat", 4, "Meow") },
    { "Elephant", new Animal(8, "Elephant", 4, "Trumpet") },
    { "Lion", new Animal(2, "Lion", 4, "Roar") },
    { "Giraffe", new Animal(6, "Giraffe", 4, "Trumpet") },
};
// Add
animalDictionary.Add("Red panda", new Animal(2, "Red Panda", 4, "Squeaker"));

// Remove
animalDictionary.Remove("Cat");

//Get
var giraffe = animalDictionary["Giraffe"];

HashSets

A HashSet<T> is a collection in C# that stores unique elements. It uses a hash-based implementation to ensure very efficient lookups, additions, and deletions. This means it uses hash functions to quickly map keys to values, you can read more about that here.

Duplicate elements are automatically ignored.

How does this differ from a Dictionary? HashSets don't have keys like Dictionaries. Instead, they store values directly and are accessed by iterating over the elements using a foreach loop or LINQ queries.

var animalHashSet = new HashSet<Animal>()
{
    new Animal(3, "Lion", 4, "Roar"),
    new Animal(5, "Tiger", 4, "Roar"),
    new Animal(2, "Elephant", 4, "Trumpet"),
    new Animal(1, "Giraffe", 4, "Neigh")
};
// Add
animalHashSet.Add(new Animal(3, "Lion", 4, "Roar"));
// Remove
animalHashSet.Remove(x=>x.Sound == "Neigh");
// Get
animalHashSet.FirstOrDefault(x=>x.Name == "Elephant");

Above, we create a HashSet<Animal> and attempt to add a duplicate object. You may expect this to throw an error, as we know HashSets can only store unique values. But instead it handles it quite beautifully and simply doesn’t add the duplicate object, so the output is:

Animal { Age = 3, Name = Lion, Legs = 4, Sound = Roar }
Animal { Age = 5, Name = Tiger, Legs = 4, Sound = Roar }
Animal { Age = 2, Name = Elephant, Legs = 4, Sound = Trumpet }

Queues

Queues work in just the same way as a queue does in everyday life, with a first-in, first-out approach.

Queue<T> does not implement the ICollection interface like Dictionaries and Lists, meaning it doesn't have an Add() method. This means you cannot add elements to the Queue whilst instantiating. It also means you cannot use the Add() method to add items – instead, you use the Enqueue() method.

var arc = new Queue<string>();
arc.Enqueue("2 Lions");
arc.Enqueue("2 Tigers");
arc.Enqueue("2 Bears");

// Peek method allows to peek at the front of the queue
Console.WriteLine("Front of the Queue: " + arc.Peek());

// Output
// Front of the Queue: 2 Lions

Console.WriteLine($"Processing: {arc.Dequeue()}"); // Output: 2 Lions
Console.WriteLine($"Processing: {arc.Dequeue()}"); // Output: 2 Tigers

The Dequeue method not only returns the next item in the queue but also removes the item from the queue as expected. You can also clear the queue using the Clear() method.

Stacks

Stacks work oppositely to Queue<T>, in that instead of first-in-first-out, they work on a last-in-first-out mechanic.

var stack = new Stack<int>();
stack.Push(1);
stack.Push(2);
stack.Push(3);
stack.Push(4);
stack.Push(5);

// iterate over the Stack
foreach(var number in stack )
{
    Console.WriteLine(number);
}

// Output
// 5 4 3 2 1

You may think looping through the items in a Stack would work the same as a List or Queue and would still print them out in order of going in. But the system knows it’s a stack, and so it enumerates the items in reverse order of how they were added – that is, the most recently added element (5) is returned first.

You can also utilise the Pop() method which will return the last item in the collection, and remove it at the same time.

Common Problems

When using various collections, you will more than likely come across common problems, such as KeyNotFoundException when using dictionaries, IndexOutOfRangeException on lists/arrays, or InvalidOperationException when modifying a collection during iteration.

KeyNotFoundException

Scenario: You try to access a Dictionary key that doesn’t exist in the Dictionary. This will result in a KeyNotFoundException, and error.

var dictionary = new Dictionary<string, string>()
{
    { "Morning", "Good Morning" },
    { "Afternoon", "Good afternoon" },
    { "Evening", "Good evening" },
    { "Night", "Good night" },
};

var message = dictionary["Dusk"];

Console.WriteLine(message);

// Output
//Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key 'Dusk' was not present in the dictionary.

Solution: I recommend using the TryGetValue function, which will handle it gracefully and return the item as an out parameter (if it can be found, otherwise the default value).

TryGetValue returns a boolean to show whether it could or couldn’t find the provided key. This boolean value can then be utilised to determine functionality based on successful retrieval or not, rather than checking the output parameter, for example (if it is null/empty or not).

var dictionary = new Dictionary<string, string>()
{
    { "Morning", "Good Morning" },
    { "Afternoon", "Good afternoon" },
    { "Evening", "Good evening" },
    { "Night", "Good night" },
};

var input = Console.ReadLine(); // Dusk

dictionary.TryGetValue(input, out var message);
Console.WriteLine($"`Message:{message}`");
// Ouput = `Message ` (blank message as default string value is empty string

//or check if was able to retrieve and access output if it was
if(dictionary.TryGetValue(input,out var m)){
    Console.WriteLine(message);
}

IndexOutOfRangeException (Lists/Arrays):

Scenario: Trying to access an index that’s outside the valid range of a list or array.

As we know, Arrays are 0 index-based, so trying to access an index of [5] on an Array of 5 items will throw the IndexOutOfRangeException.

Solutions:

1. Ensure the index is within bounds using list.Count or array.Length before access.

2. Use the ElementAtOrDefault() method. If it can’t access an item at the given index, it will return the default value, which can then be handled accordingly.

var names = new string[]
{
    "Tony", "Clint", "Bob", "Alice", "Lisa"
};

var name = names.ElementAtOrDefault(6);
Console.WriteLine(name ?? "Name not found.");

InvalidOperationException (Iterating Collections):

Scenario: Modifying a collection (for example, adding or removing items) while iterating over it with a foreach loop will throw an InvalidOperationException because you're trying to remove an item from the list while iterating over it with a foreach loop.

var myList = new List<string> { "Apple", "Banana", "Cherry", "Banana" };

foreach (var item in myList)
{
    if (item == "Banana")
    {
        myList.Remove(item); // Throws InvalidOperationException
    }
}

Why This Happens:

• The foreach loop maintains an internal enumerator for the collection.

• Modifying the collection (for example, adding/removing items) invalidates the enumerator, which causes the runtime to throw an InvalidOperationException.

Solution 1: Use a for Loop

You can use a for loop with an index to safely modify the list during iteration:

var fruits = new List<string> { "Apple", "Banana", "Cherry", "Banana" };

for (int i = 0; i < fruits; i++)
{
    if (fruits[i] == "Banana")
    {
        fruits.RemoveAt(i);
        // Adjust the index to account for the removed item
        i--; 
    }
}
Console.WriteLine(string.Join(", ", fruits)); 
// Output: Apple, Cherry

Solution 2: Iterate Over a Copy

Another approach is to iterate over a copy of the list using ToList(). This way, you’re not directly iterating over the original collection, so modifications won’t affect the loop.

var originalList = new List<string> { "Apple", "Banana", "Cherry", "Banana" };

foreach (var item in originalList.ToList()) // Create a copy
{
    if (item == "Banana")
    {
        originalList.Remove(item); // Safe removal
    }
}
Console.WriteLine(string.Join(", ", originalList)); 
// Output: Apple, Cherry

Solution 3: Use LINQ to Filter

If you only want to remove items based on a condition, you can use LINQ to create a new filtered list:

var fruits = new List<string> { "Apple", "Banana", "Cherry", "Banana" };

fruits = fruits.Where(item => item != "Banana").ToList(); // Filter out "Banana"

Console.WriteLine(string.Join(", ", fruits)); 
// Output: Apple, Cherry

Closing Thoughts

In this article, you’ve learned about many of the common Data Structures for storing multiple objects and values.

Whether you're storing data in a fixed-size array, managing a dynamic list, working with first-in-first-out queues, last-in-first-out stacks, or key-value pair dictionaries, knowing when and how to use each collection is key to becoming a confident and proficient C# developer.

Mastering these concepts will not only improve your ability to handle data effectively but also lay the groundwork for more advanced topics in data structures and algorithms. Combining these data structures with LINQ can provide some performant and easy-to-use mechanics. To learn more about LINQ you can check out my article here.

As you continue your coding journey, keep experimenting with these collections, apply them in real-world scenarios, and deepen your understanding of their inner workings.

As always should you wish to discuss this article further, any other coding-related problems, or hear about other articles I’m writing, drop me a follow on X(Twitter)

Happy coding! 😊

In this article, you will learn how to create and use variables. You'll also learn about the different data types in JavaScript and how to use them.

Let's get started!

Table of Contents

• What is a Variable? Example #1

• What is a Variable? Example #2

• What is a Variable? Example #3

• How to Declare a Variable

• Variable Assignment and Initialization

• How to Call a Variable

• How to Name Variables

  • Reserved Words in JavaScript

  • Rules for Naming Variables in JavaScript

  • Popular Variable Naming Conventions

• Variable Data Types

  • Primitive Data Types

  • Reference types in JavaScript

• Summary

What is a Variable? Example #1

When a child is born, they are given a name and throughout their life, they'll be referred to by that name (unless the name gets changed at some point).

Have you seen anyone without a name? How were you able to call them?
In an ideal world, everyone should have a name or a unique way we can refer to them. In JavaScript, every variable has a name.

Everyone must have a name or a way by which we can refer to them.

What is a Variable? Example #2

In a math equation, when we say x = 1 it means, "anywhere you see x, you should replace it with 1". In this case, x is a variable, while 1 is the value of it. That is: x points to 1.

This means that without x, there will be no reference to 1. There could be other occurrences of 1 in the equation but those will be different than the 1 which x was referring to. For example:

/* The code below means x is 1 
 * So during execution, anywhere x appears after the line below, 
 * the complier replace x with 1.
 */
let x = 1;
let y = 1; // the value which y refers to is different from that of x
console.log(x); // This line will log 1 to the console.

In the code snippet above, x refers to the value 1, and y It also refers to another value 1, but note that both values are distinct, just like you can have two different brands of bottled water even though they both contain water.

So, when we mention the variable name x, we get the value assigned to that variable.

What is a Variable? Example #3

A variable can be conceptualized as a container. The variable's name serves as its identifier, its value represents the container's contents, and its type specifies the nature of those contents.

A popular water brand here in Nigeria is known as "Eva".

Let's say you bought Eva water, took it home, and placed it amongst other water brands. You can easily say to someone, "Please get me the Eva water over there" and because of the name, it becomes easy for the person to identify and get exactly what you need.

Just as you can easily distinguish your Eva water from other water brands by its name, a variable is uniquely identified by its name within a program. While there may be multiple variables storing data, the specific name of a variable allows you to reference its contents precisely.

In JavaScript, values are assigned a name and anytime we need that value, we simply mention the name to which it was assigned. When the code executes, the name of that variable is replaced by the value it refers to.

In the case of the analogy above, the content of the bottle is water and the type is a liquid. But assuming we have a variable x which refers to the value 1, the type of the variable is number.

// Add the line of code below to the previous code snippet to
// find out the data type of x;
console.log(typeof x)

In the code snippet above, number is printed to the console because variable x holds the value 1 which is a number.

Variables exist in our program to help us hold values and be able to refer to them whenever we need to. Anywhere a variable is mentioned, the value of that variable is what's being used for the computation at the time.

How to Declare a Variable

let score;

The program above declares/creates a variable called score.

In JavaScript, creating variables is that simple. The type of the variable is the type of the value stored in it. That is, if the variable score holds a value of 1, the type for the score variable is number. So we can say, score is a number variable.

To create a variable, we have to do the following;

1. Declare the variable using one of these keywords: let, const or var.

2. Determine a name to call the variable and write it on the same line as the keyword used in step 1.

let score; // creates variable 'score'

Notice that this time, we did not give it a value. We just simply created a container that will store something. For now, it is empty. Although it has no content at the moment, we'll surely provide content for it.

Variable Assignment and Initialization

We can assign a value to a variable by using the assignment (=) operator—the variable name to the left of it, and the value to the right.

score = 1;

The code snipped above assigns 1 as the value of score (this is called variable assignment).

When we combine variable declaration and assignment in one operation, it is called variable initialization.

let score = 1;

As seen above, we declare the variable score, and immediately on the same line, assign the value 1 to it.

This means that we provided an initial value for the variable when it was created.

How to Call a Variable

If you want to use a variable for an operation at any time in your program, you can simply just "call" it. To call a variable is the same as mentioning or using it.

console.log(score + 1) // 2

In the code snippet above, the variable score was used in the line of code. Therefore, It will be replaced with its actual value 1 during the code execution. This means we'd have 1 + 1 executed, resulting in 2.

In the next section, let's learn how to properly name our variables in other to ensure our codes are neat and readable.

How to Name Variables

Just like naming a human or pet or labeling an object, we always put in much thought to ensure that the name tells a story and gives an idea of how we feel about the role of that pet, human, or object.

JavaScript is somewhat liberal when it comes to how variable naming can be done and also how long it could be.

For example, pneumonoultramicroscopicsilicovolcanoconiosis is a valid variable name in JavaScript even though it is long.

It is generally a good practice to give meaningful names to variables and they should be of a reasonable length.

Let your variables be simple and contextual. For example: author, publishedDate, readTime, shouldCompress, and so on.

It should be self-explanatory. Just avoid cryptic names where possible.

Reserved Words in JavaScript

Even though we can create variables as we wish, some names are already being used within JavaScript to mean something specific. These names cannot be used by a developer to identify a variable. They are called reserved words.

For instance, the keyword catch is used to properly handle an error and prevent it from crashing an application. Hence, you cannot call a variable catch in your program.

Below are all the reserved words in JavaScript:

arguments await break case catch class const continue debugger default delete do else enum eval export extends false finally for function if implements import in Infinity instanceof interface let NaN new null package private protected public return static super switch this throw true try typeof undefined var void while with yield

NOTE: You do not need to memorize these keywords. If you try to use them, you'll get an error and you'll learn to recognize and know them with experience.

Also, JavaScript has some rules that you must follow when naming variables as well as generally accepted conventions (best practices) that you should know about. Let's talk about them in the next section.

Rules for Naming Variables in JavaScript

• Reserved words cannot be used as variable names.

• The first letter of your variable name should be an alphabet, underscore (_), or a dollar sign ($). You cannot use a number as the first character of your variable name. Even though other kinds of special characters are allowed to start a variable name, as a way of good practice and avoiding complexities at the start, you should just always start with a letter. Using an underscore or dollar sign is symbolic by convention and we'll learn what they mean in the future.

• The rest of the variable name may contain anything but symbols, punctuations, and reserved characters (+, -, *, and so on).

• Variable names are case-sensitive. This means Boy and boy will be treated as different variables in your program.

• A variable name can be as long as is necessary for it to make sense. There is no limit imposed by the language.

• Spaces are not allowed in variable names.

Popular Variable Naming Conventions

• Variable names with multiple words should use camel casing. That is, the first word has to be all lowercase while the first letter of subsequent words should be uppercase: studentRegistrationNumber

• Use uppercase letters for constant variables: const PI = 3.1432

• If a constant variable is composed of multiple words, use snake casing (separation of words with an underscore): const PROGRAM_NAME = "Vacation planner"

• If a variable is meant to be private, prefix its name with an underscore: let _memorySize = 2042.
  Note: This is just to let the team (others working on the project) know that the author intends to use it as private. It doesn't prevent the value of the variable from being accessed (there are other ways to ensure this).

• It is common practice to prefix Boolean variables with is or has: let isMarked = true.

In the next section, we'll learn about different data types and how to work with them.

Variable Data Types

Data type simply means "type of data" 😉.

The word "data" in this context means a piece of information. We'll use the word "value" sometimes to mean data and vice versa.

In JavaScript, we store values of different types in variables. These values have different attributes/properties and the type of data a variable holds will determine the operations you can perform with that variable.

For example, if you have water (value) stored in a container (variable), you can use the water (value) to wash or drink, but if what is stored in the container are candies, you can eat them but you won't be able to use them for washing.

If you have a variable that holds numbers, you can use them to perform arithmetic operations. If the variable holds a Boolean, you cannot use it for arithmetic operations but it can be used for logical operations.

The data types in JavaScript are categorized into two primary groups, namely;

• Primitive: Number, String, Boolean, Undefined, Null, BigInt, Symbol

• Reference: Object, Array, Function

In this article, we will not talk about Symbols and BigInt to avoid complexities. The goal is to do our best to explain fundamental concepts to beginners in the most simple way possible.

Let's consider primitive data types.

Primitive Data Types

Variables having these type of data are called primitives because they hold simple values. The word primitive can be translated to mean non-complex.

Primitive values are usually a single unit like 1, "cup", null, undefined, true, and so on. Let's briefly consider how these data types are used and what kind of operations you can perform with them.

• NUMBER: In JavaScript, all numbers are floating-point values. Whether they are numbers without decimal points like a whole number that can be negative, positive, zero, or values with a decimal point like 0.2, -0.5, 1, -2, 0. They are all of the number type.

This type of value can be used in arithmetic operations like multiplication, division, subtraction, addition, modulus, and so on.

let score1 = 2;
let score2 = 5;
let averageScore = (score1 + score2) // 2
console.log(averageScore) // 3.5

To check the data type of a variable's value, use the typeof operator like this: typeof variableName. That is: typeof score1

In the code snippet above, score1 is a variable which holds a value of 2, score2 holds a value of 5, while the averageScore variable stores the result from dividing the sum of score1 and score2 by 2, which evaluates to 3.5.

Using the typeof operator on the score1 variable will return number.

Exercise: Copy the code in the snippet above and run it in your code editor to see how it works for you. You can play around with the values and use the typeof operator to check the variables' data type.

When performing arithmetic operations, you may run into other Number types like Infinity, -Infinity and NaN.

Infinity means something without any limit. One way to reach infinity is to divide a number by 0.

let result = 12 / 0;

console.log(result) // Infinity

In the code above, we divided 12 by 0 and logged the result to the console which prints out Infinity.

Negative Infinity is used to denote a number that is less than any natural number. To arrive at negative infinity, copy the code in the snippet below and run it in your coding environment.

console.log(Number.NEGATIVE_INFINITY) // -Infinity

NaN means Not a Number. This will occur when you try to carry out an impossible mathematical operation as shown below:

const result = "Ella" / 2; // Trying to divide a string with a number
console.log(result) // NaN

The first line in the code above tries to divide a string by a number and the result is NaN.

You will not often reach infinity or -Infinity as a beginner doing basic/intermediate stuff, but it is something you should be aware of so that you do not get worked up when you see it occur in your code (this is something you do not want to cram in your head). NaN is will occur more often than the others. When you see it, just know something is wrong with the operation you are trying to perform.

• STRING: In JavaScript, a string is a collection of characters enclosed in quotes: "Cathy".

The snippet below shows how a string can be used in a JavaScript program:

let author = "Sleekcodes";
let publishedDate = "14 August 2023";

console.log("Written by: " + author); // Written by: Sleekcodes
console.log("Published on: " + publishedDate); // Published on: 14 August 2023"

I am sure you noticed the + operator used with strings. When this occurs, the result is that the string on the right and that on the left will be joined together to become one. This is called string concatenation.

The code above is simply saying, "Create a variable called author and store the text "sleekCodes" as its value, create another variable publishedDate and store the text "14 August 2023" in it."

Then in line 4, we tell the JavaScript engine to log (print) the string "Written by: Sleekcodes" to the console. Line 5 also says log "Published on: 14 August 2023" to the console.

Notice that in the code above, during execution, author gets replaced with the value "Sleekcodes" and publishedDate gets replaced with "14 August 2023" where used.

Strings are used to depict or convey data in text/alphabetic format. A string can be made up of zero or more characters. A string that has no character in it is called an empty string. For example: "".

• BOOLEAN: When we need to represent data in two possible states only like true/false, on/of, or yes/no, we use Boolean values. The value of a Boolean variable is either true or false.

let isQualified = true

if (isQualified) {
    console.log("Tola is qualified"); // Tola is qualified
}

The code above will print the statement "Tola is qualified", because the value of the variable isQualified is true. That operation is a type of conditional operation. This is where Boolean values shine.

Exercise: Change the value of isQualified to be false and observe what happens.

• UNDEFINED: This is both a value and a data type. undefined is used to indicate that a variable has no defined value. For instance, when a variable is declared (let age), and you try to access its value, the result would be undefined.

let age; // note that there is no value assigned to the variable here

console.log(age); // undefined

In the code snippet above, because age isn't given any explicit value, the compiler assigns the value undefined to the variable by default.

Exercise: Use the typeof operator on the variable age and see what you get. Also, assign the value undefined to age and use the typeof operator on it again to see the result.

• NULL: Null is a value we can assign to a variable to indicate that it has no value. It is used to represent "empty" or "unknown".

let age = null;

console.log(age); // null

As seen in the code snippet above, instead of letting the compiler assign undefined for us, we explicitly indicate that the variable has no value by assigning the value null to it.

This means age is empty or unknown.

People often get confused about the difference between undefined and null. One is the default value assigned to a variable without an explicit value, while the other (null) is a value assigned to a variable by the programmer deliberately to indicate that the variable is empty. As a rule of thumb, do not assign undefined to a variable, instead use null (the compiler auto-assigns undefined where needed).

Primitive data types have no complexity. They are plain and simple (a single value). This statement will make more sense when you read about how reference types work in the next section.

Consider the image below.

Part A above is the code you write, while Part B is what happens when the code runs. For primitive data types, the value is simply assigned to the variable (it is straightforward).

Primitive values are passed by value (they generate no reference). Do not worry about what this means yet because we'll explain in the next section.

Reference Types in JavaScript

Reference data types are data passed by "reference". A thorough understanding of this statement is crucial throughout your career as a JavaScript developer, so do well to pay close attention to the concept we are about to learn.

Consider the image below carefully.

In the image above, part A is the code you write, while part B is what happens "behind the scenes".

When you create a variable whose data type is in the reference category (objects, functions, arrays), instead of the value being directly assigned to the variable, a reference is generated for the value and that reference is what gets assigned to the variable.

The reference gets assigned to the variable, but it points to the actual value.

This means that when you try to use the variable anywhere, you are working with the reference to the actual value and anything done to the reference affects the actual value.

Think of it like a middleman between the actual value and the variable name.

Consider the example below:

let studentInfo = {
    name: "John Doe",
    age: 205
}

let staffInfo = studentInfo //6. This means; assign the ref of studentInfo to staffInfo

staffInfo.name = "Lorry Sante" //8. Change the value of name key in the reference which staffInfo holds.

console.log(studentInfo.name) //9. Lorry Sante

You should notice that, changing name in staffInfo object (line 8), causes the name in studentInfo object to change too (as seen in the output of line 9).

In fact, both variables are pointing to the same value technically (see image below);

When we say that a variable is passed by reference, it means that anywhere that variable is used, you are interacting with a reference (that points) to its actual value.

So in the code snippet above, when studentInfo was assigned to staffInfo, we just made staffInfo to store the reference of the studentInfo variable, effectively saying that both staffInfo and studentInfo variables are pointing to the same value.

Therefore, if the reference generated for studentInfo is 000xx2 and it is true that during execution, variables are replaced by whatever they hold, then staffInfo = studentInfo would become staffInfo = 000xx2 during execution, while staffInfo.name would become 000xx2.name.

If we had written studentInfo.name, then during execution, it becomes 000xx2.name, it should be clear now that both studentInfo and staffInfo holds references to one value. They are like different roads to one destination.

There are three main reference data types that you'll mostly come across in your journey as a JavaScript developer: Object, Array, and Function. Let's look into them one after the other.

• OBJECT: An object is a data structure used to store complex data in key/value pairs. The variable created in the previous session has an object type like this:

let studentInfo = {
    name: "John Doe",
    age: 205
}

You can see that it isn't primitive (simple). Unlike primitive types with just simple values, an object can be used to store different information which can be made up of even primitive and reference types.

Objects store data in key/value pairs like so: {key: value}

In the code snippet above, name is key, while "John Doe" is its value. Also, age is key, while 205 is its value.

If you notice, both name and age have primitive values (string and number).

To access the value of an object, we use the object name, dot (.) notation, and the key whose value we want to access. For example: objectName.key.

Objects can also contain nested objects like so:

let studentInfo = {
    name: "John Doe",
    age: 205,
    beneficiary: {
       name: "Tira Doe",
       age: 200,
       relationship: "Wife"
    }
}

In the above example, the studentInfo object has a nested object called beneficiary. beneficiary is a key whose value is an object (reference type). Objects can still hold arrays and functions too.

Accessing the value associated with a key in an object within another object (nested object) is natural. We simply use dot notation. Like so: parentObjectName.nestedObjectName.key

For example, to access the name of the beneficiary in studentInfo object above, we simply write studentInfo.beneficiary.name.

This is not all that you need to know about objects but it is a very sound way to start.

• ARRAY: An array is a kind of object but stores data using automatically assigned indexes instead of keys.

An array is created by writing a comma-separated list of values enclosed with square brackets: [0, 1, 2, 3, "Tosin", "Mike", {name: "Abel Joe", age: 250}]

If you pay close attention to the values used in the array above, you'll notice that they are of different data types. Yes, arrays also allow you to store values of different types in one place but this is strongly discouraged (you shouldn't do it at all). The values in an array should all be of the same type.

Example of a proper array: let scores = [1, 3, 5, 6, 9, 12]

To access a value in an array, we simply specify the array name, followed by a square bracket [] without any space between the name and the bracket. Then inside the square bracket, provide the index of the value you wish to access. That is: arrayName[index].

What's an index and how do we know what index refers to the value we want to access?

An index is simply a number automatically assigned to an array value. You can think of it as an address for values in the array. Arrays are 0 indexed (meaning they start counting from zero).

To determine the index of the value you wish to access, start counting from the start of the array and your count should start from 0.

Consider the image below;

To access the value 80 in the scores array depicted in the image above, we simply write scores[3]

There is a lot you can do with arrays as a JavaScript developer. For now, this is a simple introduction to the array data type.

• FUNCTION: A function is a different kind of variable and it's declared differently (using the function keyword instead of let, const or var). It is a construct used to carry out a specific task.

For example, if you need to add two numbers together multiple times within your code, it's best practice to create a dedicated function for this task. By reusing this function, you avoid redundant code and improve code maintainability compared to repeatedly writing the addition logic. Wait!!! You are not lost. The example below will confirm this 😊

Scenario 1 (without function):

let num1 = 2;
let num2 = 3;
let result = num1 + num2;

console.log(result) // 5

let num3 = 3;
let num4 = 8;
let result2 = num3 + num4;

console.log(result2) // 11

Scenario 2 (with function):

// function declaration.
function addNumbers (num1, num2) {
    return num1 + num2;
}

console.log(addNumbers(2, 3)); // 5
console.log(addNumbers(3, 8)); // 11

You will agree that, scenario 2 contains less code, looks neater, and even feels more natural.

Functions allow us write helpers that we can call to get a specific job done for us any time we want. We just have to tell it how to do the job once and call it anytime we need it to do that job (passing in any information required for the task as arguments) and it delivers.

Function Syntax:

To write a function, we use the function keyword, followed by the name of the function: functionName, a pair of brackets (), and a pair of curly braces {}.

function functionName () {}

There are a few things/conventions you should have in mind when writing functions:

• Function names should follow the same naming rules as variables.

• Function names should be verbs (to depict an action).

• The code logic for the actual task should be written between the opening { and closing } curly braces.

• If there are values required to carry out the task, they should be passed into the function as arguments. In this case, during the function declaration, parameters should be stated between the opening ( and closing ) brackets in a comma-separated fashion. That is: function addNumbers(num1, num2)....

• If no data is required to carry out the task, then the opening ( and closing ) brackets should be left empty: function sayHi()....

A parameter is a variable defined between the opening ( and closing ) of a function during its declaration: function doSomething (param1, param2) {...}.

An argument is the value passed into the function during its invocation/call: doSomething(1, 2)

As seen above, to call/invoke a function, write the function name, followed by an opening and a closing bracket (without any whitespace). Required arguments should be provided between the opening and closing brackets (if any).

To drive this concept home, let's create a function to multiply two digits:

//        functionName   param1 param2
function multiplyNumbers (num1, num2) {
    return num1 * num2; // task to carry out.
}

It's as simple as that.

Having done that, let's call/invoke the function.

multiplyNumbers(2, 3) // 6

Notice that while creating the function, we declared two parameters: num1 and num2. When calling the function, we assigned values to the two arguments: 1 and 2.

return keyword

Functions may return values or not.

If a function contains a return statement, like the multiplyNumbers function, then it will return a value if everything goes well. If there is no return statement for the function, it will return undefined.

function sayHi () {
    console.log("Hi");
}

If we invoke sayHi, it would log the text Hi to the console and it will also return undefined.

Remember that functions are like helpers, when you send a helper to carry out an assignment, you may require them to give you feedback (the result of the task they carried out) or you may not need feedback.

If you need feedback, add a return statement about what feedback you need. Otherwise, do not add a return statement to the function.

There is still a lot to learn about every data type we have highlighted in this article so take your time to practice these basics and when you feel comfortable enough using them, you'll see the need to dive deeper.

Summary

Variables are "pointers" to values. When you mention (use) a variable anywhere in your code, the variable identifier (name) is replaced with the value it points to. It's just like calling someone's name. The name doesn't respond, the person (value) behind the name is what you hope to get as a response.

By way of retention, do not try to cram all these rules and conventions. Feel free to refer back to this article when programming and in a short amount of time, you'll be used to all of them and you won't need to refer to any article ever again to name your variables properly.

If you should have anything in mind, remember to start variables with a lowercase letter if the variable is made of multiple words, subsequent words should start with uppercase letters. That is: age, dateOfBirth.

To create a variable, use the keyword let, const, or var, followed by the variable name. If you wish to initialize the variable, then on the same line before the semi-colon, input the assignment operator and variable value after it.

For example: let score; or let score = 3; (if you wish to initialize during declaration).

If you wish to use a variable, just mention its name and the value will be used during the execution of your code.

let a = 2;
let b = 3;
console.log(a + b) // 4

This article also showed you the different data types in JavaScript and how to use them.

Did this post help? Let’s keep the conversation going. Feel free to share your thoughts or questions on Twitter (x) or LinkedIn. You can find me on Twitter (x) @asoluka_tee and Tochukwu Austin Asoluka on LinkedIn.

In this article, we'll take a look at how variables and constant work in Go.

Table of contents:

• What are Variables?

• How to Create a Variable

• Explicit Declaration

• Shorthand Variable Declaration

• Multiple Variable Declarations

• Zero Values

• What is a Variable Scope

• Naming Conventions in Go

• What are Constants in Go

• How to Declare Constants in Go

• Type of Constants in Go

• That's a Wrap

What are Variables ?

A variable is a storage location identified by a name (or identifier) that holds a value. This value can change (or vary) during a program's execution. This is why it's called a variable.

For example:

myName := “temitope”

fmt.Println(myName)

myName:= ”oyedele”

fmt.Println(myName)

We created a variable with an identifier of myName which holds a string value.

If you noticed, we changed the value to another string, and we can do that multiple times because variables are allowed to do that.

Variables allow you to store data, which can be of different types, such as integers, floating-point numbers, strings, or objects.

How to Create a Variable in Go

There are two primary ways to create a variable in Go, explicit declaration and shorthand declaration.

Explicit Declaration

This is the traditional way to create a variable in Go. It works by using the var keyword and declaring the variable’s type, making your code more readable and clear.

package main

import "fmt"

func main() {

    var age int = 25

    var name string = "Temitope"

    var height float64 = 5.7

    fmt.Println(age, name, height)

}

You can see that, for each variable, we declared its datatype before assigning a value to it.

output:
25 Temitope 5.7

The var keyword and data type can also be used without having an initial value:

package main

import "fmt"

func main() {
    var age int
    var name string
    var height float64

    age = 25
    name = "Temitope"
    height = 5.7

    fmt.Println(age, name, height)
}

This way, the variables are declared first without an initial value. They are then assigned values later in the code. You'll still have the same output as the first.

Shorthand Variable Declaration

The shorthand variable declaration syntax (:=) is a more concise way to declare variables. This method allows you to declare and initialize a variable in a single line without explicitly stating its type, as the type is inferred from the value assigned.

package main

import "fmt"

func main() {

    age := 25

    name := "Temitope"

    height := 5.7

    fmt.Println(age, name, height)

}

Here, each variable was declared alongside its value, with Go inferring the datatype. For example, age was declared and initialized with a value of 25, and Go inferred its type as int. name was declared with the value "Temitope", and Go inferred its type as string. Lastly, height was declared with 5.7, and Go inferred its type as float64.

output:
25 Temitope 5.7

One of the drawbacks of the shorthand variable declaration is that you can only use it inside of a function.

Multiple Variable Declarations

You can declare and initialize multiple variables on the same line by separating each variable with a comma. This approach is simple and straightforward. it is commonly used when the variables are related or when you want to initialize them together. For example:

package main

import "fmt"

func main() {

    var age, height int = 25, 180

    var name, city string = "Temitope", "New York"

    fmt.Println(age, height)

    fmt.Println(name, city)
}

Here, the variable age and height are both declared as integers and initialized with the values 25 and 180, respectively. Variable name and city are also both declared as strings and initialized with "Temitope" and "New York":

Output:
25 180
Temitope New York

You can also declare multiple variables in a block like so:

package main

import "fmt"

func main() {

    var (
        age int = 25

        name string = "Temitope"

        height int = 180

        city string = "New York"
    )

    fmt.Println(age, name, height, city)

}

Here, the variables age, name, height, and city are declared within a single var block, with each variable getting its own line.

Output:
25 Temitope 180 New York

Zero Values

When variables are declared without being initialized, they are assigned zero values by default. These values differ depending on the type of variable. Below is an example of how you can declare default values:

package main

import "fmt"

func main() {

    var intValue int

    var floatValue float64

    var boolValue bool

    var stringValue string

    var ptrValue *int

    var sliceValue []int

    var mapValue map[string]int

    fmt.Println("Zero values:")

    fmt.Println("int:", intValue)

    fmt.Println("float64:", floatValue)

    fmt.Println("bool:", boolValue)

    fmt.Println("string:", stringValue)

    fmt.Println("pointer:", ptrValue)

    fmt.Println("slice:", sliceValue)

    fmt.Println("map:", mapValue)

}

In the code above, here’s what’s going to happen:

• The integer intValue will be given the zero value 0.

• The floating-point number floatValue will be given the zero value 0.

• The Boolean boolValue will be given the zero value false.

• The string stringValue will be given the zero value "" (empty string).

• The pointer ptrValue, slice sliceValue, and map mapValue will all be given the zero value nil.

Output:

Output:
Zero values:
int: 0
float64: 0
bool: false
string: 
pointer: <nil>
slice: []
map: map[]

What is a Variable Scope?

Variables can be declared either globally or locally. The scope of a variable determines where it can be accessed and modified within your code.

Global variables are declared outside of any function, typically at the top of a file, and they can be accessed by any function within the same package. Here’s an example:

package main

import "fmt"

var globalCounter int = 0

func incrementCounter() {

    globalCounter++

}

func main() {

    fmt.Println("Initial Counter:", globalCounter)

    incrementCounter()

    fmt.Println("After Increment:", globalCounter)

}

In the above example, globalCounter is the global variable, and it is accessible by both the incrementCounter function and the main function.

Also, the value of globalCounter persists across function calls. This means that whatever change is made to it in one function affects its value in other parts of the program.

Local variables, on the other hand, are declared within a function or a block and are only accessible within that specific function or block. They are created when the function or block is executed and destroyed once it is completed. For example:

package main

import "fmt"

func incrementCounter() {

    localCounter := 0

    localCounter++

    fmt.Println("Local Counter:", localCounter)

}

func main() {

    incrementCounter()

    incrementCounter()

}

In the code above, we have the localCounter as the local variable inside the incrementCounter function. Each time incrementCounter is called, a new localCounter is created, initialized to 0, and incremented.

The value of localCounter does not persist between function calls, so it cannot affect any part of a program when a change is made to the function.

Naming Conventions in Go

Proper naming of variables is crucial for writing clean, readable, and maintainable code. Go has some specific conventions and rules for naming variables. Below are some of them:

• Use descriptive names: Use names that clearly describe the purpose or content of the variable. For example, instead of using vague names like x or y, use names like age, totalPrice, or userName that clearly convey what the variable represents.

• Use CamelCase for Multi-Word Names: In Go, it's common practice to use camelCase for variable names that consist of multiple words. The first word is lowercase, and the first letter of each subsequent word is capitalized.

• Avoid Using Underscores: Unlike some other languages, Go prefers camelCase over using underscores to separate words in variable names. Stick to camelCase to adhere to Go’s idiomatic style.

• Use Short Names for Short-Lived Variables: For short-lived variables, such as loop counters or indices, it's acceptable to use short names like i, j, or k.

What are Constants in Go?

Constants are immutable values defined at compile time that cannot be modified throughout the program's execution. They are useful for defining values that are known ahead of time and will remain the same.

Imagine you're building an online store where the standard shipping fee is always $10. You can declare it as a constant, so you can use it throughout your program whenever shipping charges need to be applied. If the shipping rates change, you only need to update the value in one place.

How to Declare Constants in Go

You can declare constants using the const keyword, followed by the name, the type (optional if the value implies the type), and the value of the constant. Here’s how:

package main

import "fmt"

func main() {

    const pi float64 = 3.14159

    const greeting string = "Hello, World!"

    fmt.Println("Pi:", pi)

    fmt.Println("Greeting:", greeting)

}

If you try to change a constant after being declared, then you’ll get a compile-time error.

Types of Constants in Go

Constants can be categorized as either typed or untyped. Both types of constants serve the same purpose. They provide fixed, immutable values throughout the program. However, they differ in how Go handles their types and how flexible they are when used.

Untyped constants are not assigned a type unless they are used in a context that requires a type. When you declare an untyped constant, Go will infer the type at the point where the constant is used. This makes untyped constants more flexible because they can be used in a number of settings without requiring type conversion.

package main

import "fmt"

const gravity = 9.81

func main() {

    var height int = 10

    var acceleration float64 = gravity * float64(height)

    fmt.Println("Acceleration:", acceleration)

}

Here, gravity is the untyped constant. This means Go can infer its type based on how it is used. When gravity is used in a calculation with a float64, Go will automatically treat it as a float64.

Unlike untyped constants, the typed constants have an explicitly declared type. This means they can only be used in contexts that match that type or can be converted to a compatible type. Typed constants are stricter, ensuring that the value is always treated as the specific type it was declared with.

package main

import "fmt"

const speedOfLight int = 299792458

func main() {

    var distance int = speedOfLight * 2

    fmt.Println("Distance:", distance)

}

Here, speedOfLight is the typed constant with the type int.

It can only be used in operations with other int values or converted explicitly to a different type.

That’s a Wrap

In this article, we took a look at what variables and constants are and how to declare them in Go.

Variables and constants are critical tools in programming. They allow developers to efficiently manage and manipulate data. When you understand how to use them, you can improve the quality of your code.

Please share if you found this helpful.

But did you know that the journey of AI chatbots began way back in 1966 with ELIZA? ELIZA was not as sophisticated as today’s models like GPT, but it marked the beginning of the exciting path that led us to where we are now.

Language is the essence of human interaction, and in the digital age, teaching machines to understand and generate language has become a cornerstone of artificial intelligence.

The models we interact with today—such as GPT, Llama3, Gemini, and Claude—are known as Large Language Models (LLMs). This is because they are trained on vast datasets of text, enabling them to perform a wide range of language-related tasks.

But what exactly are LLMs, and why is there so much hype surrounding them?

In this article, you'll learn what LLMs are and what is the hype all about.

What Are LLMs?

Large Language Models are AI models trained on vast amounts of text data to understand, generate, and manipulate human language. They are based on deep learning architectures like transformers, which allow them to process and predict text in a way that mimics human understanding.

In simpler terms, an LLM is a computer program that has been trained on many examples to differentiate between an apple and a Boeing 787 – and to be able to describe each of them.

Before they're ready for use and can answer your questions, LLMs are trained on massive datasets. Realistically, a program cannot conclude anything from a single sentence. But after analyzing, say, trillions of sentences, it's able to build a logic to complete sentences or even generate its own.

How to Train an LLM

Here’s how the training process works:

1. Data Collection: The first step involves gathering millions (or even billions) of text documents from diverse sources, including books, websites, research papers, and social media. This extensive dataset serves as the foundation for the model’s learning process.

2. Learning Patterns: The model analyzes the collected data to identify and learn patterns in the text. These patterns include grammar rules, word associations, contextual relationships, and even some level of common sense. By processing this data, the model begins to understand how language works.

3. Fine-Tuning: After the initial training, the model is fine-tuned for specific tasks. This involves adjusting the model’s parameters to optimize its performance for tasks such as translation, summarization, sentiment analysis, or question-answering.

4. Evaluation and Testing: Once trained, the model is rigorously tested against a series of benchmarks to evaluate its accuracy, efficiency, and reliability. This step ensures that the model performs well in real-world applications.

After the training process is completed, the models are heavily tested on a series of benchmarks for accuracy, efficiency, security, and so on.

Applications of LLMs

LLMs have a wide range of applications, from content generation to prediction and a lot more.

Content Creation:

• Writing Assistance: Tools like Grammarly utilize LLMs to provide real-time suggestions for improving grammar, style, and clarity in writing. Whether you’re drafting an email or writing a novel, LLMs can help you polish your text.

• Automated Storytelling: AI models can now generate creative content, from short stories to full-length novels. These models can emulate the style of famous authors or even create entirely new literary styles.

Customer Service:

• Chatbots: Many companies deploy AI-powered chatbots that can understand and respond to customer inquiries in real time. These chatbots can handle a wide range of tasks, from answering frequently asked questions to processing orders.

• Personal Assistants: Virtual assistants like Siri and Alexa use LLMs to interpret and respond to voice commands, providing users with information, reminders, and entertainment on demand.

Healthcare:

• Medical Record Summarization: LLMs can assist healthcare professionals by summarizing patient records, making it easier to review critical information and make informed decisions.

• Diagnostic Assistance: AI models can analyze patient data and medical literature to assist doctors in diagnosing diseases and recommending treatments.

Research and Education:

• Literature Review: LLMs can sift through vast amounts of research papers to provide concise summaries, identify trends, and suggest new research directions.

• Educational Tools: AI-powered tutors can offer personalized learning experiences by adapting to a student’s progress and needs. These tools can provide instant feedback and tailored study plans.

Entertainment:

• Game Development: LLMs are used to create more dynamic and responsive characters in video games. These AI-driven characters can engage with players more realistically and interactively.

• Music and Art Generation: AI models are now capable of composing music, generating artwork, and even writing scripts for movies, pushing the boundaries of creative expression.

Challenges with LLMs

While LLMs are powerful, they are not without their challenges. ChatGPT has over 150 Million monthly users, this gives us an idea about how big the impact of AI is. But new technologies pose some challenges too.

Bias and Fairness:

• LLMs learn from the data they are trained on, which can include biases present in society. This can lead to biased or unfair outcomes in their predictions or responses. Addressing this requires careful dataset curation and algorithm adjustments to minimize bias.

Data Privacy:

• LLMs may inadvertently learn and retain sensitive information from the data they are trained on, raising privacy concerns. There’s ongoing research on how to make LLMs more privacy-preserving.

Resource Intensive:

• Training LLMs require immense computational power and large datasets, which can be costly and environmentally taxing. Efforts are being made to create more efficient models that require less energy and data.

Interpretability:

• LLMs are often seen as "black boxes," meaning it’s challenging to understand exactly how they arrive at certain conclusions. Developing methods to make AI more interpretable and explainable is an ongoing research area.

Coding with LLMs: A Replicate Example

For those of you who like to get your hands dirty with code, here’s a quick example of how to use an LLM with the Replicate library.

Replicate is a Python package that simplifies the process of running machine learning models in the cloud. It provides a user-friendly interface to access and utilize a vast collection of pre-trained models from the Replicate platform.

With Replicate, you can easily:

• Run models directly from your Python code or Jupyter notebooks.

• Access various model types, including image generation, text generation, and more.

• Leverage powerful cloud infrastructure for efficient model execution.

• Integrate AI capabilities into your applications without the complexities of model training and deployment.

Here’s a simple code snippet to generate text using Meta's llama3-70b-instruct model. Llama 3 is one of the latest open-source large language models developed by Meta. It's designed to be highly capable, versatile, and accessible, allowing users to experiment, innovate, and scale their AI applications.

import os
import replicate # pip install replicate

# Get your token from -> https://replicate.com/account/api-tokens
os.environ["REPLICATE_API_TOKEN"] = "TOKEN"
api = replicate.Client(api_token=os.environ["REPLICATE_API_TOKEN"])

# Running llama3 model using replicate
output = api.run(
    "meta/meta-llama-3-70b-instruct",
        input={"prompt": 'Hey how are you?'}
    )

# Printing llama3's response
for item in output:
    print(item, end="")

Explanation:

• We first save the replicate token using the os package as an environment variable.

• Then we use the Llama3 70b-instruct model to give a response based on our prompt. You can customize the output by changing the prompt.

And what is a prompt? A prompt is essentially a text-based instruction or query given to an AI model. It's like providing a starting point or direction for the AI to generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way.

For example:

• "Write a poem about a robot exploring the ocean."

• "Translate 'Hello, how are you?' into Spanish."

• "Explain quantum computing in simple terms."

These are all prompts that guide the AI to produce a specific output.

Using Meta's llama-3-70b-instruct, you can build various tools around applications that are mentioned in this article. Tweak the prompts based on your use case and you will be ready to go! ⚡️

Conclusion

In this article, we explored the world of Large Language Models, providing a high-level understanding of how they work and their training process. We delved into the core concepts of LLMs, including data collection, pattern learning, and fine-tuning, and discussed the extensive applications of LLMs across various industries.

While LLMs offer immense potential, they also come with challenges such as bias, privacy concerns, resource demands, and interpretability. Addressing these challenges is crucial as AI continues to evolve and integrate more deeply into our lives.

We also provided a glimpse into how you can start working with LLMs using the Replicate library, showing that even complex models like Llama3 70b-instruct can be accessible to developers with the right tools.

This handy little tool will let users select any color they like and instantly see its HEX and RGB values.

So, grab your favorite code editor, and let's get started!

Step 1: Set Up Your Project

1. Create a New Folder: Start by creating a new folder on your computer for this project. You can name it color-picker-tool.

2. Create Files: Inside the folder, create three files:

   • index.html

   • styles.css

   • script.js

Step 2: Build the HTML Structure

1. Open the index.html file in your code editor.

2. Add Basic HTML Structure: Add the following code into index.html: or press SHIFT+! then press Enter to set the Emmet structure, then change the document title to "Color Picker Tool".

3. Link your styles.css and script.js files too.

    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>Color Picker Tool</title>
        <link rel="stylesheet" href="styles.css">
    </head>
    <body>
   
        <script src="script.js"></script>
    </body>
    </html>
   

Explanation:

• <!DOCTYPE html>: This tells the browser that the document is an HTML5 document.

• <html lang="en">: The root element of the HTML document, with the language set to English.

• <head>: Contains meta-information about the document, like character set and title.

• <title>: Sets the title of the webpage, which appears in the browser tab.

• <link rel="stylesheet" href="styles.css">: Links to the CSS file that styles the page.

• <body>: Contains the content of the webpage.

• <script src="script.js"></script>: Links to the JavaScript file that adds interactivity to the page.

3. Add the body content:

    <div class="color-picker">
      <input type="color" id="colorInput" value="#ff0000">
        <div class="color-info">
           <p>HEX: <span id="hexValue">#ff0000</span></p>
           <p>RGB: <span id="rgbValue">rgb(255, 0, 0)</span></p>
        </div>
    </div>
   

   Explanation:

• <div class="color-picker">: A container for the color picker elements.

• <input type="color" id="colorInput" value="#ff0000">: An input element that lets users pick a color. The value attribute sets the default color.

• <div class="color-info">: A container for displaying color information.

• <p>HEX: <span id="hexValue">#ff0000</span></p>: Displays the HEX value of the selected color.

• <p>RGB: <span id="rgbValue">rgb(255, 0, 0)</span></p>: Displays the RGB value of the selected color.

Here is what we'll have:

Step 3: Style with CSS

1. Open the styles.css file in your code editor.

2. Add CSS Styles: Copy and paste the following code into styles.css:

    body {
        font-family: Arial, sans-serif;
        display: flex;
        justify-content: center;
        align-items: center;
        height: 100vh;
        margin: 0;
        background-color: #f0f0f0;
    }
    .color-picker {
        background-color: #fff;
        padding: 20px;
        border-radius: 8px;
        box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);
        text-align: center;
    }
    .color-info {
        margin-top: 20px;
    }
    p {
        margin: 5px 0;
        font-size: 16px;
    }
   

Explanation:

• body: Styles the body of the page. It centers the content both vertically and horizontally and sets a light gray background.

• font-family: Arial, sans-serif;: Sets the font for the text on the page.

• display: flex;: Uses Flexbox to layout the page.

• justify-content: center; and align-items: center;: Centers the content.

• height: 100vh;: Sets the height to 100% of the viewport height.

• margin: 0;: Removes default margin.

• background-color: #f0f0f0;: Sets the background color of the page.

• .color-picker: Styles the color picker container with a white background, padding, rounded corners, and a shadow for a card-like appearance.

• .color-info: Adds a margin at the top to separate it from the color input.

• p: Styles the paragraphs within the color info, setting margin and font size.

Here is what we will have:

At this point, we can pick a color but the color codes won't be displayed. To have the color codes displayed, we'll have to add some JavaScript.

Step 4: Add JavaScript Functionality

1. Open the script.js file in your code editor.

2. Add JavaScript Code: Add the following code into script.js:

    document.getElementById('colorInput').addEventListener('input', function() {
        const color = this.value;
        document.getElementById('hexValue').textContent = color;
        document.getElementById('rgbValue').textContent = hexToRgb(color);
    });
    function hexToRgb(hex) {
        const r = parseInt(hex.slice(1, 3), 16);
        const g = parseInt(hex.slice(3, 5), 16);
        const b = parseInt(hex.slice(5, 7), 16);
        return `rgb(${r}, ${g}, ${b})`;
    }
   

   Explanation:

   • document.getElementById('colorInput'): Selects the color input element by its ID.

   • .addEventListener('input', function() {...}): Adds an event listener that triggers whenever the user selects a new color.

   • const color = this.value;: Gets the current value of the color input, which is in HEX format.

   • document.getElementById('hexValue').textContent = color;: Updates the text content of the HEX value display with the selected color.

   • document.getElementById('rgbValue').textContent = hexToRgb(color);: Converts the HEX color to RGB and updates the RGB value display.

   • function hexToRgb(hex) {...}: A function that converts a HEX color string to an RGB string.

     • parseInt(hex.slice(1, 3), 16): Converts the first two characters of the HEX color (after the #) to a decimal number, representing the red component.

     • parseInt(hex.slice(3, 5), 16): Converts the next two characters to the green component.

     • parseInt(hex.slice(5, 7), 16): Converts the last two characters to the blue component.

     • return rgb(${r}, ${g}, ${b});: Returns the RGB color as a string.

Step 5: Test Your Color Picker Tool

1. Open the Project in a Browser: Open the index.html file in a web browser to view your color picker tool.

2. Interact with the Tool: Use the color input to select different colors. The HEX and RGB values should update automatically as you select new colors.

Final Thoughts

Congratulations! You've successfully created a color picker tool using HTML, CSS, and JavaScript.

This project is a great way to practice working with user input and manipulating the DOM. You can further enhance this tool by adding features like copying color values to the clipboard or saving favorite colors.

Enjoy experimenting and learning!

You need a good understanding of how functions work before learning recursion. I have used Python code for examples in this article because of its simple syntax, but the concept of recursion is the same for every programming language.

What is Recursion?

In most programming languages, a function can call another function. But a function can also call itself. Recursion is the technique where a function calls itself.

Here's an example:

def call_me():
    call_me()

Here, the function calls itself, which is called recursion.

But "calling itself" is just a programmatic definition of recursion. Recursion involves breaking down a problem into smaller pieces to the point that it cannot be further broken down. You solve the small pieces and put them together to solve the overall problem.

Real Life Analogy of Recursion

Lets understand how recursion really works with the help of an example.

Imagine you're in line for a Disney ride, and you don't know how many people are ahead of you.

To find out, you ask the person in front of you.

Trying to find out how many people are in front of you in line

That person also doesn't know, so they ask the person in front of them.

This process continues until the question reaches the person at the very front of the line, who sees that there is no one in front of them and replies that there are zero people ahead.

The replies then start to propagate back through the line. Each person adds one to the number they were told before passing the information back.

When the person at the front replies, "There are 0 people ahead" the next person adds one and replies, "There is 1 person ahead" and so on.

Everyone knows how many people are in front of them in line

By the time the response reaches the person directly in front of you, they add one more and tell you. This way, you can determine your position in the line by just adding 1 to the number the person in front of you gave.

This example illustrates how recursion breaks a problem into smaller subproblems and then combines their solutions to solve the original problem.

Each person in the line represents a smaller instance of the same problem: determining the number of people ahead. By solving these smaller instances and combining their results, the overall problem is resolved. This is exactly how recursion works.

Technical Details of Recursion

The most important things to consider while coding recursion is to find out:

• Recursive Case: Minimum work we can do. In the above example, asking the person in front of you how many people are ahead of them is the least amount of work we can do.

• Base Case: Condition where no work is required. In the above example, the person on the front of the line doesn’t have to ask anything so it is the condition where no work is required.

Simple Example of Recursion

Calculating a factorial is the simplest example of recursion that will really help you understand how it works.

There are many ways to calculate the factorial of a number. But here we will see the recursive way to find it.

Before thinking about how we do it, we need to know what the factorial of a number is.

The factorial of a number is the multiplication of all numbers from 1 up to that number.

For example, the factorial of 5 is 120 – that is 5×4×3×2×1.

We can also represent this mathematically like this:

5×(5−1)!

This means that if we know the value of (5−1)! we can easily get the factorial by just multiplying 5 by it.

Here’s how we find the factorials of 4, 3, 2, 1, and 0:

Factorial of 4 = 4×(4−1)!
Factorial of 3 = 3×(3−1)!
Factorial of 2 = 2×(2−1)!
Factorial of 1 = 1
Factorial of 0 = 1

By looking at this, it is clear that to find the factorial of 5, we must multiply 5 by 4!.

More general example

To find the factorial of n, we need to multiply n with (n−1)!. This is something you need to do recursively.

Now, there must be a stopping condition for recursion. The stopping condition is where we perform no further operation. When n is 1 or n is 0, we can simply stop the recursion as these values have known factorials. We can simply say the factorial of 1 is 1 and the same is true for 0.

So, breaking it down, the minimum amount of work we need to do to find the factorial of n is n×(n−1)!. And we can stop performing operations on it when we find the factorial of 1 or 0.

Let's see how it looks in code:

# calculate factorial of n
def fact(n):

   # no work required
    if n == 1 or n == 0:
        return 1

    # minimum amount of work
    return n * fact(n - 1)

n = 5

# calculate factorial
factorial = fact(n)
print(factorial)

Output:

120

Let's see how it works:

In the first function call, the factorial of 5 is evaluated. Then in the second call, the factorial of 4 is evaluated, and so on until the factorial of 2 is evaluated.

Recursively calculating Factorial of 5

While calling the factorial of 2, we have 2×fact(2−1), which is 2×fact(1).

This hits our base case. So, the recursion stops and 2×fact(1) returns 2×1 to the preceding function call, and the result gets popped up the stack.

Fourth function call returning 2 to preceding function call and getting popped up from the stack

Similarly, here’s how everything else evaluates:

3rd function call returning 6 to the preceding function call and getting popped up from the stack

2nd function call returning 24 to the preceding function call and getting popped up from the stack

1st function call returning 120 to the initial function call and getting popped up from the stack

So the function finally returns the value 120 to the initial function call.

Why Do We Need a Base Case?

In the above example we have used the stopping condition for the code. But what if we don’t add a stopping condition or what if the function we write never meets the stopping condition?

Will the code run forever?

No – even if you don’t terminate, your code won't run forever. Let’s understand why this is the case with the help of an example.

def print_five():
    print(5)

    # call itself
    print_five()

# function call
print_five()

Output:

5
5
5
...

RecursionError: maximum recursion depth exceeded

If you run the above code, you will see that the function doesn’t run forever and ends with a message RecursionError: maximum recursion depth exceeded.

When a function is invoked, it is stored in a call stack. Here's how the function print_five() is stored in the call stack when it is invoked for the first time.

Call stack on the first function call

The function calls itself again and again, and the function is stored in the call stack with each call.

Call stack after n function calls

But the call stack has a limited size and cannot store an unlimited number of functions.

No space on call stack resulting in stack overflow

When the stack is full, it cannot accommodate any more calls, causing a stack overflow error.

Therefore, the base case is essential to prevent such errors and ensure the recursion terminates properly.

Now let's see another example to understand recursion even more.

How to Check if a Word is a Palindrome

Before we dive into the code, you should know what a palindrome is. A palindrome is a word that reads the same forwards and backwards.

For example, racecar reads the same forwards and backwards.

To check if a word is a palindrome, we have to check if the first and last letters are the same. If they are, we then check if the second and second-to-last letters are the same.

Check if first and last characters of racecar are the same

In the context of racecar, the first and last letters are the same, so we check if the second and second-to-last letters are the same. They are, so now we check if the third and third-to-last letters are the same. Now there is only one letter left to check. A single letter is always a palindrome because it reads the same both ways.

How to check if racecar is a palindrome

So, now let's try thinking about it recursively, which involves the minimum amount of work and determining when no work is required.

Minimum amount of work

Check if the first and last letters are the same, and if they are, remove the first and last letters from the word.

No work required

When there is one letter or no letters left at all, we can simply say it is a palindrome.

Now, let's see how the code looks:

# check palindrome
def check_palindrome(text):

    # stopping condition
    # if the size of text is 1 or 0 return true
    if len(text) == 0 or len(text) == 1:
        return True

    # least amount of work
    # check if first and last char are the same
    # if same remove first and last char from string
    if(text[0]==text[-1]):
        return(check_palindrome(text[1:-1]))

    return False

# check if string is palindrome
text = "racecar"
is_palindrome = check_palindrome(text)
print(is_palindrome)

Here's how the above code works:

Check if racecar is a palindrome

Check if racecar is a palindrome

When to Use Recursion

Recursion can appear elegant and simple. But it often requires many steps to solve even simple problems due to the CPU overhead from repeatedly adding methods to the stack. So before using it, make sure you carefully consider whether it's the right solution for your problem.

When code requires multiple loops and looks confusing and messy, recursion can offer a cleaner solution. Its use, however, depends on the specific code and the type of data or data structure involved. For data structures like trees and graphs, recursion can be particularly useful.

Despite its simplicity in appearance, recursion can be hard to understand and may take multiple steps even for simple problems. So again, make sure you think about your particular use case.

Wrapping Up

This is just an introduction to recursion. There are many cases where recursion is used, and you might be confused about how everything works. I will cover more advanced examples on recursion in the next article.

By the way, here are the resources that I found simple and useful while learning recursion:

• freeCodeCamp video on recursion: I have to give a shout-out to freeCodeCamp for their excellent video on recursion, which inspired much of this article.

• Recursion by Programiz Pro: Another good resource is the Recursion course by Programiz. It's a premium course, so it's not free, but it's thoughtfully designed. Plus, you can actually practice directly on their platform, which makes it well worth it.

No matter where you learn from, don't spend too much time searching for the perfect resource. Just grasp the concepts and start practicing—that's the only way you'll truly learn.

While empowering you in your current role, learning Linux can also help you transition into other tech careers like DevOps, Cybersecurity, and Cloud Computing.

In this handbook, you'll learn the basics of the Linux command line, and then transition to more advanced topics like shell scripting and system administration. Whether you are new to Linux or have been using it for years, this book has something for you.

Important Note: All examples in this book are demonstrated in Ubuntu 22.04.2 LTS (Jammy Jellyfish). Most command line tools are more or less the same in other distributions. However, some GUI applications and commands may differ if you are working on another Linux distribution.

Table of Contents

• Part 1: Introduction to Linux

  • 1.1. Getting Started with Linux

• Part 2: Introduction to Bash Shell and System Commands

  • 2.1. Getting Started with the Bash shell

  • 2.2. Command Structure

  • 2.3. Bash Commands and Keyboard Shortcuts

  • 2.4. Identifying Yourself: The whoami Command

• Part 3: Understanding Your Linux System

  • 3.1. Discovering Your OS and Specs

• Part 4: Managing Files From the Command line

  • 4.1. The Linux File-system Hierarchy

  • 4.2. Navigating the Linux File-system

  • 4.3. Managing Files and Directories

  • 4.5. Basic Commands for Viewing Files

• Part 5: The Essentials of Text Editing in Linux

  • 5.1. Mastering Vim: The Complete Guide

  • 5.2. Mastering Nano

• Part 6: Bash Scripting

  • 6.1. Definition of Bash scripting

  • 6.2. Advantages of Bash Scripting

  • 6.3. Overview of Bash Shell and Command Line Interface

  • 6.4. How to Create and Execute Bash scripts

  • 6.5. Bash Scripting Basics

• Part 7: Managing Software Packages in Linux

  • 7.1. Packages and Package Management

  • 7.2. Installing a Package via Command Line

  • 7.3. Installing a Package via an Advanced Graphical Method – Synaptic

  • 7.4. Installing downloaded packages from a website

• Part 8: Advanced Linux Topics

  • 8.1. User Management

  • 8.2 Connecting to Remote Servers via SSH

  • 8.3. Advanced Log Parsing and Analysis

  • 8.4. Managing Linux Processes via Command Line

  • 8.5. Standard Input and Output Streams in Linux

  • 8.6 Automation in Linux – Automate Tasks with Cron Jobs

  • 8.7. Linux Networking Basics

  • 8.8. Linux Troubleshooting: Tools and Techniques

  • 8.9. General Troubleshooting Strategy for Servers

  • 8.10 Diagnosing Hardware Problems

• Conclusion

Part 1: Introduction to Linux

1.1. Getting Started with Linux

What is Linux?

Linux is an open-source operating system that is based on the Unix operating system. It was created by Linus Torvalds in 1991.

Open source means that the source code of the operating system is available to the public. This allows anyone to modify the original code, customise it, and distribute the new operating system to potential users.

Why should you learn about Linux?

In today's data center landscape, Linux and Microsoft Windows stand out as the primary contenders, with Linux having a major share.

Here are several compelling reasons to learn Linux:

• Given the prevalence of Linux hosting, there is a high chance that your application will be hosted on Linux. So learning Linux as a developer becomes increasingly valuable.

• With cloud computing becoming the norm, chances are high that your cloud instances will rely on Linux.

• Linux serves as the foundation for many operating systems for the Internet of Things (IoT) and mobile applications.

• In IT, there are many opportunities for those skilled in Linux.

What does it mean that Linux is an open-source operating system?

First, what is open source? Open source software is software whose source code is freely accessible, allowing anyone to utilize, modify, and distribute it.

Whenever source code is created, it is automatically considered copyrighted, and its distribution is governed by the copyright holder through software licenses.

In contrast to open source, proprietary or closed-source software restricts access to its source code. Only the creators can view, modify, or distribute it.

Linux is primarily open source, which means that its source code is freely available. Anyone can view, modify, and distribute it. Developers from anywhere in the world can contribute to its improvement. This lays the foundation of collaboration which is an important aspect of open source software.

This collaborative approach has led to the widespread adoption of Linux across servers, desktops, embedded systems, and mobile devices.

The most interesting aspect of Linux being open source is that anyone can tailor the operating system to their specific needs without being restricted by proprietary limitations.

Chrome OS used by Chromebooks is based on Linux. Android, that powers many smartphones globally, is also based on Linux.

What is a Linux Kernel?

The kernel is the central component of an operating system that manages the computer and its hardware operations. It handles memory operations and CPU time.

The kernel acts as a bridge between applications and the hardware-level data processing using inter-process communication and system calls.

The kernel loads into memory first when an operating system starts and remains there until the system shuts down. It is responsible for tasks like disk management, task management, and memory management.

If you are curious about what the Linux kernel looks like, here is the GitHub link.

What is a Linux distribution?

By this point, you know that you can re-use the Linux kernel code, modify it, and create a new kernel. You can further combine different utilities and software to create a completely new operating system.

A Linux distribution or distro is a version of the Linux operating system that includes the Linux kernel, system utilities, and other software. Being open source, a Linux distribution is a collaborative effort involving multiple independent open-source development communities.

What does it mean that a distribution is derived? When you say that a distribution is "derived" from another, the newer distro is built upon the base or foundation of the original distro. This derivation can include using the same package management system (more on this later), kernel version, and sometimes the same configuration tools.

Today, there are thousands of Linux distributions to choose from, offering differing goals and criteria for selecting and supporting the software provided by their distribution.

Distributions vary from one to the other, but they generally have several common characteristics:

• A distribution consists of a Linux kernel.

• It supports user space programs.

• A distribution may be small and single-purpose or include thousands of open-source programs.

• Some means of installing and updating the distribution and its components should be provided.

If you view the Linux Distributions Timeline, you'll see two major distros: Slackware and Debian. Several distributions are derived from them. For example, Ubuntu and Kali are derived from Debian.

What are the advantages of derivation? There are various advantages of derivation. Derived distributions can leverage the stability, security, and large software repositories of the parent distribution.

When building on an existing foundation, developers can drive their focus and effort entirely on the specialized features of the new distribution. Users of derived distributions can benefit from the documentation, community support, and resources already available for the parent distribution.

Some popular Linux distributions are:

1. Ubuntu: One of the most widely used and popular Linux distributions. It is user-friendly and recommended for beginners. Learn more about Ubuntu here.

2. Linux Mint: Based on Ubuntu, Linux Mint provides a user-friendly experience with a focus on multimedia support. Learn more about Linux Mint here.

3. Arch Linux: Popular among experienced users, Arch is a lightweight and flexible distribution aimed at users who prefer a DIY approach. Learn more about Arch Linux here.

4. Manjaro: Based on Arch Linux, Manjaro provides a user-friendly experience with pre-installed software and easy system management tools. Learn more about Manjaro here.

5. Kali Linux: Kali Linux provides a comprehensive suite of security tools and is mostly focused on cybersecurity and hacking. Learn more about Kali Linux here.

How to install and access Linux

The best way to learn is to apply the concepts as you go. In this section, we'll learn how to install Linux on your machine so you can follow along. You'll also learn how to access Linux on a Windows machine.

I recommend that you follow any one of the methods mentioned in this section to get access to Linux so you may follow along.

Install Linux as the primary OS

Installing Linux as the primary OS is the most efficient way to use Linux, as you can use the full power of your machine.

In this section, you will learn how to install Ubuntu, which is one of the most popular Linux distributions. I have left out other distributions for now, as I want to keep things simple. You can always explore other distributions once you are comfortable with Ubuntu.

• Step 1 – Download the Ubuntu iso: Go to the official website and download the iso file. Make sure to select a stable release that is labeled "LTS". LTS stands for Long Term Support which means you can get free security and maintenance updates for a long time (usually 5 years).

• Step 2 – Create a bootable pendrive: There are a number of softwares that can create a bootable pendrive. I recommend using Rufus, as it is quite easy to use. You can download it from here.

• Step 3 – Boot from the pendrive: Once your bootable pendrive is ready, insert it and boot from the pendrive. The boot menu depends on your laptop. You can google the boot menu for your laptop model.

• Step 4 – Follow the prompts. Once, the boot process starts, select try or install ubuntu.

  

  The process will take some time. Once the GUI appears, you can select the language, and keyboard layout and continue. Enter your login and name. Remember the credentials as you will need them to log in to your system and access full privileges. Wait for the installation to complete.

• Step 5 – Restart: Click on restart now and remove the pen drive.

• Step 6 – Login: Login with the credentials you entered earlier.

And there you go! Now you can install apps and customize your desktop.

For advanced installation, you can explore the following topics:

• Disk partitioning.

• Setting swap memory for enabling hibernation.

Accessing the terminal

An important part of this handbook is learning about the terminal where you'll run all the commands and see the magic happen. You can search for the terminal by pressing the "windows" key and typing "terminal". You can pin the Terminal in the dock where other apps are located for easy access.

💡 The shortcut for opening the terminal is ctrl+alt+t

You can also open the terminal from inside a folder. Right click where you are and click on "Open in Terminal". This will open the terminal in the same path.

How to use Linux on a Windows machine

Sometimes you might need to run both Linux and Windows side by side. Luckily, there are some ways you can get the best of both worlds without getting different computers for each operating system.

In this section, you'll explore a few ways to use Linux on a Windows machine. Some of them are browser-based or cloud-based and do not need any OS installation before using them.

Option 1: "Dual-boot" Linux + Windows With dual boot, you can install Linux alongside Windows on your computer, allowing you to choose which operating system to use at startup.

This requires partitioning your hard drive and installing Linux on a separate partition. With this approach, you can only use one operating system at a time.

Option 2: Use Windows Subsystem for Linux (WSL) Windows Subsystem for Linux provides a compatibility layer that lets you run Linux binary executables natively on Windows.

Using WSL has some advantages. The setup for WSL is simple and not time-consuming. It is lightweight compared to VMs where you have to allocate resources from the host machine. You don't need to install any ISO or virtual disc image for Linux machines which tend to be heavy files. You can use Windows and Linux side by side.

How to install WSL2

First, enable the Windows Subsystem for Linux option in settings.

• Go to Start. Search for "Turn Windows features on or off."

• Check the option "Windows Subsystem for Linux" if it isn't already.

  

• Next, open your command prompt and provide the installation commands.

• Open Command Prompt as an administrator:

  

• Run the command below:

wsl --install

This is the output:

Note: By default, Ubuntu will be installed.

• Once installation is complete, you'll need to reboot your Windows machine. So, restart your Windows machine.

After restarting, you might see a window like this:

Once installation of Ubuntu is complete, you'll be prompted to enter your username and password.

And, that's it! You are ready to use Ubuntu.

Launch Ubuntu by searching from the start menu.

And here we have your Ubuntu instance launched.

Option 3: Use a Virtual Machine (VM)

A virtual machine (VM) is a software emulation of a physical computer system. It allows you to run multiple operating systems and applications on a single physical machine simultaneously.

You can use virtualization software such as Oracle VirtualBox or VMware to create a virtual machine running Linux within your Windows environment. This allows you to run Linux as a guest operating system alongside Windows.

VM software provides options to allocate and manage hardware resources for each VM, including CPU cores, memory, disk space, and network bandwidth. You can adjust these allocations based on the requirements of the guest operating systems and applications.

Here are some of the common options available for virtualization:

• Oracle virtual box

• Multipass

• VMware workstation player

Option 4: Use a Browser-based Solution

Browser-based solutions are particularly useful for quick testing, learning, or accessing Linux environments from devices that don't have Linux installed.

You can either use online code editors or web-based terminals to access Linux. Note that you usually don't have full administration privileges in these cases.

Online code editors

Online code editors offer editors with built-in Linux terminals. While their primary purpose is coding, you can also utilize the Linux terminal to execute commands and perform tasks.

Replit is an example of an online code editor, where you can write your code and access the Linux shell at the same time.

Web-based Linux terminals:

Online Linux terminals allow you to access a Linux command-line interface directly from your browser. These terminals provide a web-based interface to a Linux shell, enabling you to execute commands and work with Linux utilities.

One such example is JSLinux. The screenshot below shows a ready-to-use Linux environment:

Option 5: Use a Cloud-based Solution

Instead of running Linux directly on your Windows machine, you can consider using cloud-based Linux environments or virtual private servers (VPS) to access and work with Linux remotely.

Services like Amazon EC2, Microsoft Azure, or DigitalOcean provide Linux instances that you can connect to from your Windows computer. Note that some of these services offer free tiers, but they are not usually free in the long run.

Part 2: Introduction to Bash Shell and System Commands

2.1. Getting Started with the Bash shell

Introduction to the bash shell

The Linux command line is provided by a program called the shell. Over the years, the shell program has evolved to cater to various options.

Different users can be configured to use different shells. But, most users prefer to stick with the current default shell. The default shell for many Linux distros is the GNU Bourne-Again Shell (bash). Bash is succeeded by the Bourne shell (sh).

To find out your current shell, open your terminal and enter the following command:

echo $SHELL

Command breakdown:

• The echo command is used to print on the terminal.

• The $SHELL is a special variable that holds the name of the current shell.

In my setup, the output is /bin/bash. This means that I am using the bash shell.

# output
echo $SHELL
/bin/bash

Bash is very powerful as it can simplify certain operations that are hard to accomplish efficiently with a GUI (or Graphical User Interface). Remember that most servers do not have a GUI, and it is best to learn to use the powers of a command line interface (CLI).

Terminal vs Shell

The terms "terminal" and "shell" are often used interchangeably, but they refer to different parts of the command-line interface.

The terminal is the interface you use to interact with the shell. The shell is the command interpreter that processes and executes your commands. You'll learn more about shells in Part 6 of the handbook.

What is a prompt?

When a shell is used interactively, it displays a $ when it is waiting for a command from the user. This is called the shell prompt.

[username@host ~]$

If the shell is running as root (you'll learn more about the root user later on), the prompt is changed to #.

[root@host ~]#

2.2. Command Structure

A command is a program that performs a specific operation. Once you have access to the shell, you can enter any command after the $ sign and see the output on the terminal.

Generally, Linux commands follow this syntax:

command [options] [arguments]

Here is the breakdown of the above syntax:

• command: This is the name of the command you want to execute. ls (list), cp (copy), and rm (remove) are common Linux commands.

• [options]: Options, or flags, often preceded by a hyphen (-) or double hyphen (--), modify the behavior of the command. They can change how the command operates. For example, ls -a uses the -a option to display hidden files in the current directory.

• [arguments]: Arguments are the inputs for the commands that require one. These could be filenames, user names, or other data that the command will act upon. For example, in the command cat access.log, cat is the command and access.log is the input. As a result, the cat command displays the contents of the access.log file.

Options and arguments are not required for all commands. Some commands can be run without any options or arguments, while others might require one or both to function correctly. You can always refer to the command's manual to check the options and arguments it supports.

💡Tip: You can view a command's manual using the man command.

You can access the manual page for ls with man ls, and it'll look like this:

Manual pages are a great and quick way to access the documentation. I highly recommend going through man pages for the commands that you use the most.

2.3. Bash Commands and Keyboard Shortcuts

When you are in the terminal, you can speed up your tasks by using shortcuts.

Here are some of the most common terminal shortcuts:

2.4. Identifying Yourself: The whoami Command

You can get the username you are logged in with by using the whoami command. This command is useful when you are switching between different users and want to confirm the current user.

Just after the $ sign, type whoami and press enter.

whoami

This is the output I got.

zaira@zaira-ThinkPad:~$ whoami
zaira

Part 3: Understanding Your Linux System

3.1. Discovering Your OS and Specs

Print system information using the uname Command

You can get detailed system information from the uname command.

When you provide the -a option, it prints all the system information.

uname -a
# output
Linux zaira 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb  9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

In the output above,

• Linux: Indicates the operating system.

• zaira: Represents the hostname of the machine.

• 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 9 13:32:52 UTC 2: Provides information about the kernel version, build date, and some additional details.

• x86_64 x86_64 x86_64: Indicates the architecture of the system.

• GNU/Linux: Represents the operating system type.

Find details of the CPU architecture using the lscpu Command

The lscpu command in Linux is used to display information about the CPU architecture. When you run lscpu in the terminal, it provides details such as:

• The architecture of the CPU (for example, x86_64)

• CPU op-mode(s) (for example, 32-bit, 64-bit)

• Byte Order (for example, Little Endian)

• CPU(s) (number of CPUs), and so on

  Let's try it out:

lscpu
# output
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 5 5500U with Radeon Graphics
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            1
    CPU max MHz:         4056.0000
    CPU min MHz:         400.0000

That was a whole lot of information, but useful too! Remember you can always skim the relevant information using specific flags. See the command manual with man lscpu.

Part 4: Managing Files From the Command line

4.1. The Linux File-system Hierarchy

All files in Linux are stored in a file-system. It follows an inverted-tree-like structure because the root is at the topmost part.

The / is the root directory and the starting point of the file system. The root directory contains all other directories and files on the system. The / character also serves as a directory separator between path names. For example, /home/alice forms a complete path.

The image below shows the complete file system hierarchy. Each directory servers a specific purpose.

Note that this is not an exhaustive list and different distributions may have different configurations.

Here is a table that shows the purpose of each directory:

💡 Tip: You can learn more about the file system using the man hier command.

You can check your file system using the tree -d -L 1 command. You can modify the -L flag to change the depth of the tree.

tree -d -L 1
# output
.
├── bin -> usr/bin
├── boot
├── cdrom
├── data
├── dev
├── etc
├── home
├── lib -> usr/lib
├── lib32 -> usr/lib32
├── lib64 -> usr/lib64
├── libx32 -> usr/libx32
├── lost+found
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin -> usr/sbin
├── snap
├── srv
├── sys
├── tmp
├── usr
└── var

25 directories

This list is not exhaustive and different distributions and systems may be configured differently.

4.2. Navigating the Linux File-system

Absolute path vs relative path

The absolute path is the full path from the root directory to the file or directory. It always starts with a /. For example, /home/john/documents.

The relative path, on the other hand, is the path from the current directory to the destination file or directory. It does not start with a /. For example, documents/work/project.

Locating your current directory using the pwd command

It is easy to lose your way in the Linux file system, especially if you are new to the command line. You can locate your current directory using the pwd command.

Here is an example:

pwd
# output
/home/zaira/scripts/python/free-mem.py

Changing directories using the cd command

The command to change directories is cd and it stands for "change directory". You can use the cd command to navigate to a different directory.

You can use a relative path or an absolute path.

For example, if you want to navigate the below file structure (following the red lines):

and you are standing at "home", the command would be like this:

cd home/bob/documents/work/project

Some other commonly used cd shortcuts are:

4.3. Managing Files and Directories

When working with files and directories, you might want to copy, move, remove, and create new files and directories. Here are some commands that can help you with that.

💡Tip: You can differentiate between a file and folder by looking at the first letter in the output of ls -l. A'-' represents a file and a 'd' represents a folder.

Creating new directories using the mkdir command

You can create an empty directory using the mkdir command.

# creates an empty directory named "foo" in the current folder
mkdir foo

You can also create directories recursively using the -p option.

mkdir -p tools/index/helper-scripts
# output of tree
.
└── tools
    └── index
        └── helper-scripts

3 directories, 0 files

Creating new files using the touch command

The touch command creates an empty file. You can use it like this:

# creates empty file "file.txt" in the current folder
touch file.txt

The file names can be chained together if you want to create multiple files in a single command.

# creates empty files "file1.txt", "file2.txt", and "file3.txt" in the current folder

touch file1.txt file2.txt file3.txt

Removing files and directories using the rm and rmdir command

You can use the rm command to remove both files and non-empty directories.

🛑 Note that you should use the -f flag with caution as you won't be asked before deleting a file. Also, be careful when running rm commands in the root folder as it might result in deleting important system files.

Copying files using the cp command

To copy files in Linux, use the cp command.

• Syntax to copy files:cp source_file destination_of_file

This command copies a file named file1.txt to a new file location /home/adam/logs.

cp file1.txt /home/adam/logs

The cp command also creates a copy of one file with the provided name.

This command copies a file named file1.txt to another file named file2.txt in the same folder.

cp file1.txt file2.txt

Moving and renaming files and folders using the mv command

The mv command is used to move files and folders from one directory to the other.

Syntax to move files:mv source_file destination_directory

Example: Move a file named file1.txt to a directory named backup:

mv file1.txt backup/

To move a directory and its contents:

mv dir1/ backup/

Renaming files and folders in Linux is also done with the mv command.

Syntax to rename files:mv old_name new_name

Example: Rename a file from file1.txt to file2.txt:

mv file1.txt file2.txt

Rename a directory from dir1 to dir2:

mv dir1 dir2

4.4. Locating Files and Folders Using the find Command

The find command lets you efficiently search for files, folders, and character and block devices.

Below is the basic syntax of the find command:

find /path/ -type f -name file-to-search

Where,

• /path is the path where the file is expected to be found. This is the starting point for searching files. The path can also be/or . which represents the root and current directory, respectively.

• -type represents the file descriptors. They can be any of the below:
  f – Regular file such as text files, images, and hidden files.
  d – Directory. These are the folders under consideration.
  l – Symbolic link. Symbolic links point to files and are similar to shortcuts.
  c – Character devices. Files that are used to access character devices are called character device files. Drivers communicate with character devices by sending and receiving single characters (bytes, octets). Examples include keyboards, sound cards, and the mouse.
  b – Block devices. Files that are used to access block devices are called block device files. Drivers communicate with block devices by sending and receiving entire blocks of data. Examples include USB and CD-ROM

• -name is the name of the file type that you want to search.

How to search files by name or extension

Suppose we need to find files that contain "style" in their name. We'll use this command:

find . -type f -name "style*"
#output
./style.css
./styles.css

Now let's say we want to find files with a particular extension like .html. We'll modify the command like this:

find . -type f -name "*.html"
# output
./services.html
./blob.html
./index.html

How to search hidden files

A dot at the beginning of the filename represents hidden files. They are normally hidden but can be viewed with ls -a in the current directory.

We can modify the find command as shown below to search for hidden files:

find . -type f -name ".*"

List and find hidden files

ls -la
# folder contents
total 5
drwxrwxr-x  2 zaira zaira 4096 Mar 26 14:17 .
drwxr-x--- 61 zaira zaira 4096 Mar 26 14:12 ..
-rw-rw-r--  1 zaira zaira    0 Mar 26 14:17 .bash_history
-rw-rw-r--  1 zaira zaira    0 Mar 26 14:17 .bash_logout
-rw-rw-r--  1 zaira zaira    0 Mar 26 14:17 .bashrc

find . -type f -name ".*"
# find output
./.bash_logout
./.bashrc
./.bash_history

Above you can see a list of hidden files in my home directory.

How to search log files and configuration files

Log files usually have the extension .log, and we can find them like this:

 find . -type f -name "*.log"

Similarly, we can search for configuration files like this:

 find . -type f -name "*.conf"

How to search other files by type

We can search for character block files by providing c to -type:

find / -type c

Similarly, we can find device block files by using b:

find / -type b

How to search directories

In the example below, we are finding the folders using the -type d flag.

ls -l
# list folder contents
drwxrwxr-x 2 zaira zaira 4096 Mar 26 14:22 hosts
-rw-rw-r-- 1 zaira zaira    0 Mar 26 14:23 hosts.txt
drwxrwxr-x 2 zaira zaira 4096 Mar 26 14:22 images
drwxrwxr-x 2 zaira zaira 4096 Mar 26 14:23 style
drwxrwxr-x 2 zaira zaira 4096 Mar 26 14:22 webp 

find . -type d 
# find directory output
.
./webp
./images
./style
./hosts

How to search files by size

An incredibly helpful use of the find command is to list files based on a particular size.

find / -size +250M

Here, we are listing files whose size exceeds 250MB.

Other units include:

• G: GigaBytes.

• M: MegaBytes.

• K: KiloBytes

• c : bytes.

Just replace with the relevant unit.

find <directory> -type f -size +N<Unit Type>

How to search files by modification time

By using the -mtime flag, you can filter files and folders based on the modification time.

find /path -name "*.txt" -mtime -10

For example,

• -mtime +10 means you are looking for a file modified 10 days ago.

• -mtime -10 means less than 10 days.

• -mtime 10 If you skip + or – it means exactly 10 days.

4.5. Basic Commands for Viewing Files

Concatenate and display files using the cat command

The cat command in Linux is used to display the contents of a file. It can also be used to concatenate files and create new files.

Here is the basic syntax of the cat command:

cat [options] [file]

The simplest way to use cat is without any options or arguments. This will display the contents of the file on the terminal.

For example, if you want to view the contents of a file named file.txt, you can use the following command:

cat file.txt

This will display all the contents of the file on the terminal at once.

Viewing text files interactively using less and more

While cat displays the entire file at once, less and more allow you to view the contents of a file interactively. This is useful when you want to scroll through a large file or search for specific content.

The syntax of the less command is:

less [options] [file]

The more command is similar to less but has fewer features. It is used to display the contents of a file one screen at a time.

The syntax of the more command is:

more [options] [file]

For both commands, you can use the spacebar to scroll one page down, the Enter key to scroll one line down, and the q key to exit the viewer.

To move backward you can use the b key, and to move forward you can use the f key.

Displaying the last part of files using tail

Sometimes you might need to view just the last few lines of a file instead of the entire file. The tail command in Linux is used to display the last part of a file.

For example, tail file.txt will display the last 10 lines of the file file.txt by default.

If you want to display a different number of lines, you can use the -n option followed by the number of lines you want to display.

# Display the last 50 lines of the file file.txt
tail -n 50 file.txt

💡Tip: Another usage of the tail is its follow-along (-f) option. This option enables you to view the contents of a file as they are being written. This is a useful utility for viewing and monitoring log files in real-time.

Displaying the beginning of files using head

Just like tail displays the last part of a file, you can use the head command in Linux to display the beginning of a file.

For example, head file.txt will display the first 10 lines of the file file.txt by default.

To change the number of lines displayed, you can use the -n option followed by the number of lines you want to display.

Counting words, lines, and characters using wc

You can count words, lines and characters in a file using the wc command.

For example, running wc syslog.log gave me the following output:

1669 9623 64367 syslog.log

In the output above,

• 1669 represents the number of lines in the file syslog.log.

• 9623 represents the number of words in the file syslog.log.

• 64367 represents the number of characters in the file syslog.log.

So, the command wc syslog.log counted 1669 lines, 9623 words, and 64367 characters in the file syslog.log.

Comparing files line by line using diff

Comparing and finding differences between two files is a common task in Linux. You can compare two files right within the command line using the diff command.

The basic syntax of the diff command is:

diff [options] file1 file2

Here are two files, hello.py and also-hello.py, that we will compare using the diff command:

# contents of hello.py

def greet(name):
    return f"Hello, {name}!"

user = input("Enter your name: ")
print(greet(user))

# contents of also-hello.py

more also-hello.py
def greet(name):
    return fHello, {name}!

user = input(Enter your name: )
print(greet(user))
print("Nice to meet you")

1. Check whether the files are the same or not

diff -q hello.py also-hello.py
# Output
Files hello.py and also-hello.py differ

2. See how the files differ. For that, you can use the -u flag to see a unified output:

diff -u hello.py also-hello.py
--- hello.py    2024-05-24 18:31:29.891690478 +0500
+++ also-hello.py    2024-05-24 18:32:17.207921795 +0500
@@ -3,4 +3,5 @@

 user = input(Enter your name: )
 print(greet(user))
+print("Nice to meet you")

In the above output:

• --- hello.py 2024-05-24 18:31:29.891690478 +0500 indicates the file being compared and its timestamp.

• +++ also-hello.py 2024-05-24 18:32:17.207921795 +0500 indicates the other file being compared and its timestamp.

• @@ -3,4 +3,5 @@ shows the line numbers where the changes occur. In this case, it indicates that lines 3 to 4 in the original file have changed to lines 3 to 5 in the modified file.

• user = input(Enter your name: ) is a line from the original file.

• print(greet(user)) is another line from the original file.

• +print("Nice to meet you") is the additional line in the modified file.

3. To see the diff in a side-by-side format, you can use the -y flag:

diff -y hello.py also-hello.py
# Output
def greet(name):                        def greet(name):
    return fHello, {name}!                        return fHello, {name}!

user = input(Enter your name: )                    user = input(Enter your name: )
print(greet(user))                        print(greet(user))
                                        >    print("Nice to meet you")

In the output:

• The lines that are the same in both files are displayed side by side.

• Lines that are different are shown with a > symbol indicating the line is only present in one of the files.

Part 5: The Essentials of Text Editing in Linux

Text editing skills using the command line are one of the most crucial skills in Linux. In this section, you will learn how to use two popular text editors in Linux: Vim and Nano.

I suggest that you master any one text editor of your choice and stick to it. It will save you time and make you more productive. Vim and nano are safe choices as they are present on most Linux distributions.

5.1. Mastering Vim: The Complete Guide

Introduction to Vim

Vim is a popular text editing tool for the command line. Vim comes with its advantages: it is powerful, customizable, and fast. Here are some reasons why you should consider learning Vim:

• Most servers are accessed via a CLI, so in system administration, you don't necessarily have the luxury of a GUI. But Vim has got your back – it'll always be there.

• Vim uses a keyboard-centric approach, as it is designed to be used without a mouse, which can significantly speed up editing tasks once you have learned the keyboard shortcuts. This also makes it faster than GUI tools.

• Some Linux utilities, for example editing cron jobs, work in the same editing format as Vim.

• Vim is suitable for all – beginners and advanced users. Vim supports complex string searches, highlighting searches, and much more. Through plugins, Vim provides extended capabilities to developers and system admins that includes code completion, syntax highlighting, file management, version control, and more.

Vim has two variations: Vim (vim) and Vim tiny (vi). Vim tiny is a smaller version of Vim that lacks some features of Vim.

How to start using vim

Start using Vim with this command:

vim your-file.txt

your-file.txt can either be a new file or an existing file that you want to edit.

Navigating Vim: Mastering movement and command modes

In the early days of the CLI, the keyboards didn't have arrow keys. Hence, navigation was done using the set of available keys, hjkl being one of them.

Being keyboard-centric, using hjkl keys can greatly speed up text editing tasks.

Note: Although arrow keys would work totally fine, you can still experiment with hjkl keys to navigate. Some people find this this way of navigation efficient.

💡Tip: To remember the hjkl sequence, use this: hang back, jump down, kick up, leap forward.

The three Vim modes

You need to know the 3 operating modes of Vim and how to switch between them. Keystrokes behave differently in each command mode. The three modes are as follows:

1. Command mode.

2. Edit mode.

3. Visual mode.

Command Mode. When you start Vim, you land in the command mode by default. This mode allows you to access other modes.

⚠ To switch to other modes, you need to be present in the command mode first

Edit Mode

This mode allows you to make changes to the file. To enter edit mode, press I while in command mode. Note the '-- INSERT' switch at the end of the screen.

Visual mode

This mode allows you to work on a single character, a block of text, or lines of text. Let's break it down into simple steps. Remember, use the below combinations when in command mode.

• Shift + V → Select multiple lines.

• Ctrl + V → Block mode

• V → Character mode

The visual mode comes in handy when you need to copy and paste or edit lines in bulk.

Extended command mode.

The extended command mode allows you to perform advanced operations like searching, setting line numbers, and highlighting text. We'll cover extended mode in the next section.

How to stay on track? If you forget your current mode, just press ESC twice and you will be back in Command Mode.

Editing Efficiently in Vim: Copy/pasting and searching

1. How to copy and paste in Vim

Copy-paste is known as 'yank' and 'put' in Linux terms. To copy-paste, follow these steps:

• Select text in visual mode.

• Press 'y' to copy/ yank.

• Move your cursor to the required position and press 'p'.

2. How to search for text in Vim

Any series of strings can be searched with Vim using the / in command mode. To search, use /string-to-match.

In the command mode, type :set hls and press enter. Search using /string-to-match. This will highlight the searches.

Let's search a few strings:

3. How to exit Vim

First, move to command mode (by pressing escape twice) and then use these flags:

• Exit without saving → :q!

• Exit and save → :wq!

Shortcuts in Vim: Making Editing Faster

Note: All these shortcuts work in the command mode only.

• Basic Navigation

  • h: Move left

  • j: Move down

  • k: Move up

  • l: Move right

  • 0: Move to the beginning of the line

  • $: Move to the end of the line

  • gg: Move to the beginning of the file

  • G: Move to the end of the file

  • Ctrl+d: Move half-page down

  • Ctrl+u: Move half-page up

• Editing

  • i: Enter insert mode before the cursor

  • I: Enter insert mode at the beginning of the line

  • a: Enter insert mode after the cursor

  • A: Enter insert mode at the end of the line

  • o: Open a new line below the current line and enter insert mode

  • O: Open a new line above the current line and enter insert mode

  • x: Delete the character under the cursor

  • dd: Delete the current line

  • yy: Yank (copy) the current line (use this in visual mode)

  • p: Paste below the cursor

  • P: Paste above the cursor

• Searching and Replacing

  • /: Search for a pattern which will take you to its next occurrence

  • ?: Search for a pattern that will take you to its previous occurrence

  • n: Repeat the last search in the same direction

  • N: Repeat the last search in the opposite direction

  • :%s/old/new/g: Replace all occurrences of old with new in the file

• Exiting

  • :w: Save the file but don't exit

  • :q: Quit Vim (fails if there are unsaved changes)

  • :wq or :x: Save and quit

  • :q!: Quit without saving

• Multiple Windows

  • :split or :sp: Split the window horizontally

  • :vsplit or :vsp: Split the window vertically

  • Ctrl+w followed by h/j/k/l: Navigate between split windows

5.2. Mastering Nano

Getting started with Nano: The user-friendly text editor

Nano is a user-friendly text editor that is easy to use and is perfect for beginners. It is pre-installed on most Linux distributions.

To create a new file using Nano, use the following command:

nano

To start editing an existing file with Nano, use the following command:

nano filename

List of key bindings in Nano

Let's study the most important key bindings in Nano. You'll use the key bindings to perform various operations like saving, exiting, copying, pasting, and more.

Write to a file and save

Once you open Nano using the nano command, you can start writing text. To save the file, press Ctrl+O. You'll be prompted to enter the file name. Press Enter to save the file.

Exit nano

You can exit Nano by pressing Ctrl+X. If you have unsaved changes, Nano will prompt you to save the changes before exiting.

Copying and pasting

To select a region, use ALT+A. A marker will show. Use arrows to select the text. Once selected, exit the marker with with ALT+^.

To copy the selected text, press Ctrl+K. To paste the copied text, press Ctrl+U.

Cutting and pasting

Select the region with ALT+A. Once selected, cut the text with Ctrl+K. To paste the cut text, press Ctrl+U.

Navigation

Use Alt \ to move to the beginning of the file.

Use Alt / to move to the end of the file.

Viewing line numbers

When you open a file with nano -l filename, you can view line numbers on the left side of the file.

Searching

You can search for a specific line number with ALt + G. Enter the line number to the prompt and press Enter.

You can also initiate search for a string with CTRL + W and press Enter. If you want to search backwards, you can press Alt+W after initiating the search with Ctrl+W.

Summary of keybindings in Nano

• General

  • Ctrl+X: Exit Nano (prompting to save if changes are made)

  • Ctrl+O: Save the file

  • Ctrl+R: Read a file into the current file

  • Ctrl+G: Display the help text

• Editing

  • Ctrl+K: Cut the current line and store it in the cutbuffer

  • Ctrl+U: Paste the contents of the cutbuffer into the current line

  • Alt+6: Copy the current line and store it in the cutbuffer

  • Ctrl+J: Justify the current paragraph

• Navigation

  • Ctrl+A: Move to the beginning of the line

  • Ctrl+E: Move to the end of the line

  • Ctrl+C: Display the current line number and file information

  • Ctrl+_ (Ctrl+Shift+-): Go to a specific line (and optionally, column) number

  • Ctrl+Y: Scroll up one page

  • Ctrl+V: Scroll down one page

• Search and Replace

  • Ctrl+W: Search for a string (then Enter to search again)

  • Alt+W: Repeat the last search but in the opposite direction

  • Ctrl+\: Search and replace

• Miscellaneous

  • Ctrl+T: Invoke the spell checker, if available

  • Ctrl+D: Delete the character under the cursor (does not cut it)

  • Ctrl+L: Refresh (redraw) the current screen

  • Alt+U: Undo the last operation

  • Alt+E: Redo the last undone operation

Part 6: Bash Scripting

6.1. Definition of Bash scripting

A bash script is a file containing a sequence of commands that are executed by the bash program line by line. It allows you to perform a series of actions, such as navigating to a specific directory, creating a folder, and launching a process using the command line.

By saving commands in a script, you can repeat the same sequence of steps multiple times and execute them by running the script.

6.2. Advantages of Bash Scripting

Bash scripting is a powerful and versatile tool for automating system administration tasks, managing system resources, and performing other routine tasks in Unix/Linux systems.

Some advantages of shell scripting are:

• Automation: Shell scripts allow you to automate repetitive tasks and processes, saving time and reducing the risk of errors that can occur with manual execution.

• Portability: Shell scripts can be run on various platforms and operating systems, including Unix, Linux, macOS, and even Windows through the use of emulators or virtual machines.

• Flexibility: Shell scripts are highly customizable and can be easily modified to suit specific requirements. They can also be combined with other programming languages or utilities to create more powerful scripts.

• Accessibility: Shell scripts are easy to write and don't require any special tools or software. They can be edited using any text editor, and most operating systems have a built-in shell interpreter.

• Integration: Shell scripts can be integrated with other tools and applications, such as databases, web servers, and cloud services, allowing for more complex automation and system management tasks.

• Debugging: Shell scripts are easy to debug, and most shells have built-in debugging and error-reporting tools that can help identify and fix issues quickly.

6.3. Overview of Bash Shell and Command Line Interface

The terms "shell" and "bash" are often used interchangeably. But there is a subtle difference between the two.

The term "shell" refers to a program that provides a command-line interface for interacting with an operating system. Bash (Bourne-Again SHell) is one of the most commonly used Unix/Linux shells and is the default shell in many Linux distributions.

Till now, the commands that you have been entering were basically being entered in a "shell".

Although Bash is a type of shell, there are other shells available as well, such as Korn shell (ksh), C shell (csh), and Z shell (zsh). Each shell has its own syntax and set of features, but they all share the common purpose of providing a command-line interface for interacting with the operating system.

You can determine your shell type using the ps command:

ps
# output:

    PID TTY          TIME CMD
  20506 pts/0    00:00:00 bash <--- the shell type
  20931 pts/0    00:00:00 ps

In summary, while "shell" is a broad term that refers to any program that provides a command-line interface, "Bash" is a specific type of shell that is widely used in Unix/Linux systems.

Note: In this section, we will be using the "bash" shell.

6.4. How to Create and Execute Bash scripts

Script naming conventions

By naming convention, bash scripts end with .sh. However, bash scripts can run perfectly fine without the sh extension.

Adding the Shebang

Bash scripts start with a shebang. Shebang is a combination of bash # and bang ! followed by the bash shell path. This is the first line of the script. Shebang tells the shell to execute it via bash shell. Shebang is simply an absolute path to the bash interpreter.

Below is an example of the shebang statement.

#!/bin/bash

You can find your bash shell path (which may vary from the above) using the command:

which bash

Creating your first bash script

Our first script prompts the user to enter a path. In return, its contents will be listed.

Create a file named run_all.sh using any editor of your choice.

vim run_all.sh

Add the following commands in your file and save it:

#!/bin/bash
echo "Today is " `date`

echo -e "\nenter the path to directory"
read the_path

echo -e "\n you path has the following files and folders: "
ls $the_path

Let's take a deeper look at the script line by line. I am displaying the same script again, but this time with line numbers.

  1 #!/bin/bash
  2 echo "Today is " `date`
  3
  4 echo -e "\nenter the path to directory"
  5 read the_path
  6
  7 echo -e "\n you path has the following files and folders: "
  8 ls $the_path

• Line #1: The shebang (#!/bin/bash) points toward the bash shell path.

• Line #2: The echo command displays the current date and time on the terminal. Note that the date is in backticks.

• Line #4: We want the user to enter a valid path.

• Line #5: The read command reads the input and stores it in the variable the_path.

• line #8: The ls command takes the variable with the stored path and displays the current files and folders.

Executing the bash script

To make the script executable, assign execution rights to your user using this command:

chmod u+x run_all.sh

Here,

• chmod modifies the ownership of a file for the current user :u.

• +x adds the execution rights to the current user. This means that the user who is the owner can now run the script.

• run_all.sh is the file we wish to run.

You can run the script using any of the mentioned methods:

• sh run_all.sh

• bash run_all.sh

• ./run_all.sh

Let's see it running in action 🚀

6.5. Bash Scripting Basics

Comments in bash scripting

Comments start with a # in bash scripting. This means that any line that begins with a # is a comment and will be ignored by the interpreter.

Comments are very helpful in documenting the code, and it is a good practice to add them to help others understand the code.

These are examples of comments:

# This is an example comment
# Both of these lines will be ignored by the interpreter

Variables and data types in Bash

Variables let you store data. You can use variables to read, access, and manipulate data throughout your script.

There are no data types in Bash. In Bash, a variable is capable of storing numeric values, individual characters, or strings of characters.

In Bash, you can use and set the variable values in the following ways:

1. Assign the value directly:

country=Netherlands

2. Assign the value based on the output obtained from a program or command, using command substitution. Note that $ is required to access an existing variable's value.

same_country=$country

This assigns the value of country to the new variable same_country.

To access the variable value, append $ to the variable name.

country=Netherlands
echo $country
# output
Netherlands
new_country=$country
echo $new_country
# output
Netherlands

Above, you can see an example of assigning and printing variable values.

Variable naming conventions

In Bash scripting, the following are the variable naming conventions:

1. Variable names should start with a letter or an underscore (_).

2. Variable names can contain letters, numbers, and underscores (_).

3. Variable names are case-sensitive.

4. Variable names should not contain spaces or special characters.

5. Use descriptive names that reflect the purpose of the variable.

6. Avoid using reserved keywords, such as if, then, else, fi, and so on as variable names.

Here are some examples of valid variable names in Bash:

name
count
_var
myVar
MY_VAR

And here are some examples of invalid variable names:

# invalid variable names

2ndvar (variable name starts with a number)
my var (variable name contains a space)
my-var (variable name contains a hyphen)

Following these naming conventions helps make Bash scripts more readable and easier to maintain.

Input and output in Bash scripts

Gathering input

In this section, we'll discuss some methods to provide input to our scripts.

1. Reading the user input and storing it in a variable

We can read the user input using the read command.

#!/bin/bash
echo "What's your name?"
read entered_name
echo -e "\nWelcome to bash tutorial" $entered_name

2. Reading from a file

This code reads each line from a file named input.txt and prints it to the terminal. We'll study while loops later in this section.

while read line
do
  echo $line
done < input.txt

3. Command line arguments

In a bash script or function, $1 denotes the initial argument passed, $2 denotes the second argument passed, and so forth.

This script takes a name as a command-line argument and prints a personalized greeting.

#!/bin/bash
echo "Hello, $1!"

We have supplied Zaira as our argument to the script.

Output:

Displaying output

Here we'll discuss some methods to receive output from the scripts.

1. Printing to the terminal:

echo "Hello, World!"

This prints the text "Hello, World!" to the terminal.

2. Writing to a file:

echo "This is some text." > output.txt

This writes the text "This is some text." to a file named output.txt. Note that the > operator overwrites a file if it already has some content.

3. Appending to a file:

echo "More text." >> output.txt

This appends the text "More text." to the end of the file output.txt.

4. Redirecting output:

ls > files.txt

This lists the files in the current directory and writes the output to a file named files.txt. You can redirect output of any command to a file this way.

You'll learn about output redirection in detail in section 8.5.

Conditional statements (if/else)

Expressions that produce a boolean result, either true or false, are called conditions. There are several ways to evaluate conditions, including if, if-else, if-elif-else, and nested conditionals.

Syntax:

if [[ condition ]];
then
    statement
elif [[ condition ]]; then
    statement 
else
    do this by default
fi

Syntax of bash conditional statements

We can use logical operators such as AND -a and OR -o to make comparisons that have more significance.

if [ $a -gt 60 -a $b -lt 100 ]

This statement checks if both conditions are true: a is greater than 60 AND b is less than 100.

Let's see an example of a Bash script that uses if, if-else, and if-elif-else statements to determine if a user-inputted number is positive, negative, or zero:

#!/bin/bash

# Script to determine if a number is positive, negative, or zero

echo "Please enter a number: "
read num

if [ $num -gt 0 ]; then
  echo "$num is positive"
elif [ $num -lt 0 ]; then
  echo "$num is negative"
else
  echo "$num is zero"
fi

The script first prompts the user to enter a number. Then, it uses an if statement to check if the number is greater than 0. If it is, the script outputs that the number is positive. If the number is not greater than 0, the script moves on to the next statement, which is an if-elif statement.

Here, the script checks if the number is less than 0. If it is, the script outputs that the number is negative.

Finally, if the number is neither greater than 0 nor less than 0, the script uses an else statement to output that the number is zero.

Seeing it in action 🚀

Looping and branching in Bash

While loop

While loops check for a condition and loop until the condition remains true. We need to provide a counter statement that increments the counter to control loop execution.

In the example below, (( i += 1 )) is the counter statement that increments the value of i. The loop will run exactly 10 times.

#!/bin/bash
i=1
while [[ $i -le 10 ]] ; do
   echo "$i"
  (( i += 1 ))
done

For loop

The for loop, just like the while loop, allows you to execute statements a specific number of times. Each loop differs in its syntax and usage.

In the example below, the loop will iterate 5 times.

#!/bin/bash

for i in {1..5}
do
    echo $i
done

Case statements

In Bash, case statements are used to compare a given value against a list of patterns and execute a block of code based on the first pattern that matches. The syntax for a case statement in Bash is as follows:

case expression in
    pattern1)
        # code to execute if expression matches pattern1
        ;;
    pattern2)
        # code to execute if expression matches pattern2
        ;;
    pattern3)
        # code to execute if expression matches pattern3
        ;;
    *)
        # code to execute if none of the above patterns match expression
        ;;
esac

Here, "expression" is the value that we want to compare, and "pattern1", "pattern2", "pattern3", and so on are the patterns that we want to compare it against.

The double semicolon ";;" separates each block of code to execute for each pattern. The asterisk "*" represents the default case, which executes if none of the specified patterns match the expression.

Let's see an example:

fruit="apple"

case $fruit in
    "apple")
        echo "This is a red fruit."
        ;;
    "banana")
        echo "This is a yellow fruit."
        ;;
    "orange")
        echo "This is an orange fruit."
        ;;
    *)
        echo "Unknown fruit."
        ;;
esac

In this example, since the value of fruit is apple, the first pattern matches, and the block of code that echoes This is a red fruit. is executed. If the value of fruit were instead banana, the second pattern would match and the block of code that echoes This is a yellow fruit. would execute, and so on.

If the value of fruit does not match any of the specified patterns, the default case is executed, which echoes Unknown fruit.

Part 7: Managing Software Packages in Linux

Linux comes with several built-in programs. But you might need to install new programs based on your needs. You might also need to upgrade the existing applications.

7.1. Packages and Package Management

What is a package?

A package is a collection of files that are bundled together. These files are essential for a particular program to run. These files contain the program's executable files, libraries, and other resources.

In addition to the files required for the program to run, packages also contain installation scripts, which copy the files to where they are needed. A program may contain many files and dependencies. With packages, it is easier to manage all the files and dependencies at once.

What is the difference between source and binary?

Programmers write source code in a programming language. This source code is then compiled into machine code that the computer can understand. The compiled code is called binary code.

When you download a package, you can either get the source code or the binary code. The source code is the human-readable code that can be compiled into binary code. The binary code is the compiled code that the computer can understand.

Source packages can be used with any type of machine if the source code is compiled properly. Binary, on the other hand, is compiled code that is specific to a particular type of machine or architecture.

You can find the architecture of your machine using the uname -m command.

uname -m
# output
x86_64

Package dependencies

Programs often share files. Instead of including these files in each package, a separate package can provide them for all programs.

To install a program that needs these files, you must also install the package containing them. This is called a package dependency. Specifying dependencies makes packages smaller and simpler by reducing duplicates.

When you install a program, its dependencies must also be installed. Most required dependencies are usually already installed, but a few extra ones might be needed. So, don't be surprised if several other packages are installed along with your chosen package. These are the necessary dependencies.

Package managers

Linux offers a comprehensive package management system for installing, upgrading, configuring, and removing software.

With package management, you can get access to an organized base of thousands of software packages along with having the ability to resolve dependencies and check for software updates.

Packages can be managed using either command-line utilities that can be easily automated by system administrators, or through a graphical interface.

Software channels/repositories

⚠️ Package management is different for different distros. Here, we are using Ubuntu.

Installing software is a bit different in Linux as compared to Windows and Mac.

Linux uses repositories to store software packages. A repository is a collection of software packages that are available for installation via a package manager.

A package manager also stores an index of all of the packages available from a repo. Sometimes the index is rebuilt to ensure that it is up to date and to know which packages have been upgraded or added to the channel since it last checked.

The generic process of downloading software from a repo looks something like this:

If we talk specifically about Ubuntu,

1. Index is fetched using apt update. (apt is explained in next section).

2. Required files/ dependencies requested according to index using apt install

3. Packages and dependencies installed locally.

4. Update dependencies and packages when required using apt update and apt upgrade

On Debian-based distros, you can file the list of repos (repositories) in /etc/apt/sources.list.

7.2. Installing a Package via Command Line

The apt command is a powerful command-line tool, which works with Ubuntu’s "Advanced Packaging Tool (APT)".

apt, along with the commands bundled with it, provides the means to install new software packages, upgrade existing software packages, update the package list index, and even upgrade the entire Ubuntu system.

To view the logs of the installation using apt, you can view the /var/log/dpkg.log file.

Following are the uses of the apt command:

Installing packages

For example, to install the htop package, you can use the following command:

sudo apt install htop

Updating the package list index

The package list index is a list of all the packages available in the repositories. To update the local package list index, you can use the following command:

sudo apt update

Upgrading the packages

Installed packages on your system can get updates containing bug fixes, security patches, and new features.

To upgrade the packages, you can use the following command:

sudo apt upgrade

Removing packages

To remove a package, like htop, you can use the following command:

sudo apt remove htop

7.3. Installing a Package via an Advanced Graphical Method – Synaptic

If you are not comfortable with the command line, you can use a GUI application to install packages. You can achieve the same results as the command line, but with a graphical interface.

Synaptic is a GUI package management application that helps in listing the installed packages, their status, pending updates, and so on. It offers custom filters to help you narrow down the search results.

You can also right-click on a package and view further details like the dependencies, maintainer, size, and the installed files.

7.4. Installing downloaded packages from a website

You may want to install a package you have downloaded from a website, rather than from a software repository. These packages are called .deb files.

Usingdpkgto install packages:dpkg is a command-line tool used to install packages. To install a package with dpkg, open the Terminal and type the following:

cd directory
sudo dpkg -i package_name.deb

Note: Replace "directory" with the directory where the package is stored and "package_name" with the filename of the package.

Alternatively, you can right-click, select "Open With Other Application," and choose a GUI app of your choice.

💡 Tip: In Ubuntu, you can see a list of installed packages with dpkg --list.

Part 8: Advanced Linux Topics

8.1. User Management

There can be multiple users with varying levels of access in a system. In Linux, the root user has the highest level of access and can perform any operation on the system. Regular users have limited access and can only perform operations they have been granted permission to do.

What is a user?

A user account provides separation between different people and programs that can run commands.

Humans identify users by a name, as names are easy to work with. But the system identifies users by a unique number called the user ID (UID).

When human users log in using the provided username, they have to use a password to authorize themselves.

User accounts form the foundations of system security. File ownership is also associated with user accounts and it enforces access control to the files. Every process has an associated user account that provides a layer of control for the admins.

There are three main types of user accounts:

1. Superuser: The superuser has complete access to the system. The name of the superuser is root. It has a UID of 0.

2. System user: The system user has user accounts that are used to run system services. These accounts are used to run system services and are not meant for human interaction.

3. Regular user: Regular users are human users who have access to the system.

The id command displays the user ID and group ID of the current user.

id
uid=1000(john) gid=1000(john) groups=1000(john),4(adm),24(cdrom),27(sudo),30(dip)... output truncated

To view the basic information of another user, pass the username as an argument to the id command.

id username

To view user-related information for processes, use the ps command with the -u flag.

ps -u
# Output
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  16968  3920 ?        Ss   18:45   0:00 /sbin/init splash
root         2  0.0  0.0      0     0 ?        S    18:45   0:00 [kthreadd]

By default, systems use the /etc/passwd file to store user information.

Here is a line from the /etc/passwd file:

root:x:0:0:root:/root:/bin/bash

The /etc/passwd file contains the following information about each user:

1. Username: root – The username of the user account.

2. Password: x – The password in encrypted format for the user account that is stored in the /etc/shadow file for security reasons.

3. User ID (UID): 0 – The unique numerical identifier for the user account.

4. Group ID (GID): 0 – The primary group identifier for the user account.

5. User Info: root – The real name for the user account.

6. Home directory: /root – The home directory for the user account.

7. Shell: /bin/bash – The default shell for the user account. A system user might use /sbin/nologin if interactive logins are not allowed for that user.

What is a group?

A group is a collection of user accounts that share access and resources. Groups have group names to identify them. The system identifies groups by a unique number called the group ID (GID).

By default, the information about groups is stored in the /etc/group file.

Here is an entry from the /etc/group file:

adm:x:4:syslog,john

Here is the breakdown of the fields in the given entry:

1. Group name: adm – The name of the group.

2. Password: x – The password for the group is stored in the /etc/gshadow file for security reasons. The password is optional and appears empty if not set.

3. Group ID (GID): 4 – The unique numerical identifier for the group.

4. Group members: syslog,john – The list of usernames that are members of the group. In this case, the group adm has two members: syslog and john.

In this specific entry, the group name is adm, the group ID is 4, and the group has two members: syslog and john. The password field is typically set to x to indicate that the group password is stored in the /etc/gshadow file.

The groups are further divided into 'primary' and 'supplementary' groups.

• Primary Group: Each user is assigned one primary group by default. This group usually has the same name as the user and is created when the user account is made. Files and directories created by the user are typically owned by this primary group.

• Supplementary Groups: These are extra groups a user can belong to in addition to their primary group. Users can be members of multiple supplementary groups. These groups let a user have permissions for resources shared among those groups. They help provide access to shared resources without affecting the system’s file permissions and keeping the security intact. While a user must belong to one primary group, belonging to supplementary groups is optional.

Access control: finding and understanding file permission

File ownership can be viewed using the ls -l command. The first column in the output of the ls -l command shows the permissions of the file. Other columns show the owner of the file and the group that the file belongs to.

Let's have a closer look into the mode column:

Mode defines two things:

• File type: File type defines the type of the file. For regular files that contain simple data it is blank -. For other special file types the symbol is different. For a directory which is a special file, it is d. Special files are treated differently by the OS.

• Permission classes: The next set of characters define the permissions for user, group, and others respectively.
  – User: This is the owner of a file and owner of the file belongs to this class.
  – Group: The members of the file’s group belong to this class
  – Other: Any users that are not part of the user or group classes belong to this class.

💡Tip: Directory ownership can be viewed using the ls -ld command.

How to Read Symbolic Permissions or the rwx permissions

The rwx representation is known as the Symbolic representation of permissions. In the set of permissions,

• r stands for read. It is indicated in the first character of the triad.

• w stands for write. It is indicated in the second character of the triad.

• x stands for execution. It is indicated in the third character of the triad.

Read:

For regular files, read permissions allow the file to be opened and read only. Users can't modify the file.

Similarly for directories, read permissions allow the listing of directory content without any modification in the directory.

Write:

When files have write permissions, the user can modify (edit, delete) the file and save it.

For folders, write permissions enable a user to modify its contents (create, delete, and rename the files inside it), and modify the contents of files that the user has write permissions to.

Examples of permissions in Linux

Now that we know how to read permissions, let's see some examples.

• -rwx------: A file that is only accessible and executable by its owner.

  -rw-rw-r--: A file that is open to modification by its owner and group but not by others.

• drwxrwx---: A directory that can be modified by its owner and group.

Execute:

For files, execute permissions allows the user to run an executable script. For directories, the user can access them, and access details about files in the directory.

How to Change File Permissions and Ownership in Linux using chmod and chown

Now that we know the basics of ownerships and permissions, let's see how we can modify permissions using the chmod command.

Syntax ofchmod:

chmod permissions filename

Where,

• permissions can be read, write, execute or a combination of them.

• filename is the name of the file for which the permissions need to change. This parameter can also be a list if files to change permissions in bulk.

We can change permissions using two modes:

1. Symbolic mode: this method uses symbols like u, g, o to represent users, groups, and others. Permissions are represented as r, w, x for read, write, and execute, respectively. You can modify permissions using +, - and =.

2. Absolute mode: this method represents permissions as 3-digit octal numbers ranging from 0-7.

Now, let's see them in detail.

How to Change Permissions using Symbolic Mode

The table below summarize the user representation:

We can use mathematical operators to add, remove, and assign permissions. The table below shows the summary:

Example:

Suppose I have a script and I want to make it executable for the owner of the file zaira.

Current file permissions are as follows:

Let's split the permissions like this:

To add execution rights (x) to owner (u) using symbolic mode, we can use the command below:

chmod u+x mymotd.sh

Output:

Now, we can see that the execution permissions have been added for owner zaira.

Additional examples for changing permissions via symbolic method:

• Removing read and write permission for group and others: chmod go-rw.

• Removing read permissions for others: chmod o-r.

• Assigning write permission to group and overriding existing permission: chmod g=w.

How to Change Permissions using Absolute Mode

Absolute mode uses numbers to represent permissions and mathematical operators to modify them.

The below table shows how we can assign relevant permissions:

Permissions can be revoked using subtraction. The below table shows how you can remove relevant permissions.

Example:

• Set read (add 4) for user, read (add 4) and execute (add 1) for group, and only execute (add 1) for others.

chmod 451 file-name

This is how we performed the calculation:

Note that this is the same as r--r-x--x.

• Remove execution rights from other and group.

To remove execution from other and group, subtract 1 from the execute part of last 2 octets.

• Assign read, write and execute to user, read and execute to group and only read to others.

This would be the same as rwxr-xr--.

How to Change Ownership using the chown Command

Next, we will learn how to change the ownership of a file. You can change the ownership of a file or folder using the chown command. In some cases, changing ownership requires sudo permissions.

Syntax of chown:

chown user filename

How to change user ownership with chown

Let's transfer the ownership from user zaira to user news.

chown news mymotd.sh

Command to change ownership: sudo chown news mymotd.sh.

Output:

How to change user and group ownership simultaneously

We can also use chown to change user and group simultaneously.

chown user:group filename

How to change directory ownership

You can change ownership recursively for contents in a directory. The example below changes the ownership of the /opt/script folder to allow user admin.

chown -R admin /opt/script

How to change group ownership

In case we only need to change the group owner, we can use chown by preceding the group name by a colon :

chown :admins /opt/script

How to switch between users

You can switch between users using the su command.

[user01@host ~]$ su user02
Password:
[user02@host ~]$

How to gain superuser access

The superuser or the root user has the highest level of access on a Linux system. The root user can perform any operation on the system. The root user can access all files and directories, install and remove software, and modify or override system configurations.

With great power comes great responsibility. If the root user is compromised, someone can gain complete control over the system. It is advised to use the root user account only when necessary.

If you omit the username, the su command switches to the root user account by default.

[user01@host ~]$ su
Password:
[root@host ~]#

Another variation of the su command is su -. The su command switches to the root user account but does not change the environment variables. The su - command switches to the root user account and changes the environment variables to those of the target user.

Running commands with sudo

To run commands as the root user without switching to the root user account, you can use the sudo command. The sudo command allows you to run commands with elevated privileges.

Running commands with sudo is a safer option rather than running the commands as the root user. This is because, only a specific set of users can be granted permission to run commands with sudo. This is defined in the /etc/sudoers file.

Also, sudo logs all commands that are run with it, providing an audit trail of who ran which commands and when.

In Ubuntu, you can find the audit logs here:

cat /var/log/auth.log | grep sudo

For a user that does not have access to sudo, it gets flagged in logs and prompts a message like this:

user01 is not in the sudoers file.  This incident will be reported.

Managing local user accounts

Creating users from the command line

The command used to add a new user is:

sudo useradd username

This command sets up a user's home directory and creates a private group designated by the user's username. Currently, the account lacks a valid password, preventing the user from logging in until a password is created.

Modifying existing users

The usermod command is used to modify existing users. Here are some of the common options used with the usermod command:

Here are some examples of the usermod command in Linux:

1. Change a user's login name:

    sudo usermod -l newusername oldusername
   

2. Change a user's home directory:

    sudo usermod -d /new/home/directory -m username
   

3. Add a user to a supplementary group:

    sudo usermod -aG groupname username
   

4. Change a user's shell:

    sudo usermod -s /bin/bash username
   

5. Lock a user's account:

    sudo usermod -L username
   

6. Unlock a user's account:

    sudo usermod -U username
   

7. Set an expiration date for a user account:

    sudo usermod -e YYYY-MM-DD username
   

8. Change a user's user ID (UID):

    sudo usermod -u newUID username
   

9. Change a user's primary group:

    sudo usermod -g newgroup username
   

10. Remove a user from a supplementary group:

    sudo gpasswd -d username groupname
    

Deleting users

The userdel command is used to delete a user account and related files from the system.

• sudo userdel username: removes the user's details from /etc/passwd but keeps the user's home directory.

• The sudo userdel -r username command removes the user's details from /etc/passwd and also deletes the user's home directory.

Changing user passwords

The passwd command is used to change a user's password.

• sudo passwd username: sets the initial password or changes the existing password of username. It is also used to change the password of the currently logged in user.

8.2 Connecting to Remote Servers via SSH

Accessing remote servers is one of the essential tasks for system administrators. You can connect to different servers or access databases through your local machine and execute commands, all using SSH.

What is the SSH protocol?

SSH stands for Secure Shell. It is a cryptographic network protocol that allows secure communication between two systems.

The default port for SSH is 22.

The two participants while communicating via SSH are:

• The server: the machine that you want access to.

• The client: The system that you are accessing the server from.

Connection to a server follows these steps:

1. Initiate Connection: The client sends a connection request to the server.

2. Exchange of Keys: The server sends its public key to the client. Both agree on the encryption methods to use.

3. Session Key Generation: The client and server use the Diffie-Hellman key exchange to create a shared session key.

4. Client Authentication: The client logs in to the server using a password, private key, or another method.

5. Secure Communication: After authentication, the client and server communicate securely with encryption.

How to connect to a remote server using SSH?

The ssh command is a built-in utility in Linux and also the default one. It makes accessing servers quite easy and secure.

Here, we are talking about how the client would make a connection to the server.

Prior to connecting to a server, you need to have the following information:

• The IP address or the domain name of the server.

• The username and password of the server.

• The port number that you have access to in the server.

The basic syntax of the ssh command is:

ssh username@server_ip

For example, if your username is john and the server IP is 192.168.1.10, the command would be:

ssh john@192.168.1.10

After that, you'll be prompted to enter the secret password. Your screen will look similar to this:

john@192.168.1.10's password: 
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-70-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Fri Jun  5 10:17:32 UTC 2024

  System load:  0.08               Processes:           122
  Usage of /:   12.3% of 19.56GB   Users logged in:     1
  Memory usage: 53%                IP address for eth0: 192.168.1.10
  Swap usage:   0%

Last login: Fri Jun  5 09:34:56 2024 from 192.168.1.2
john@hostname:~$ # start entering commands

Now you can execute the relevant commands on the server 192.168.1.10.

⚠️ The default port for ssh is 22 but it is also vulnerable, as hackers will likely attempt here first. Your server can expose another port and share the access with you. To connect to a different port, use the -p flag.

ssh -p port_number username@server_ip

8.3. Advanced Log Parsing and Analysis

Log files, when configured, are generated by your system for a variety of useful reasons. They can be used to track system events, monitor system performance, and troubleshoot issues. They are specifically useful for system administrators where they can track application errors, network events, and user activity.

Here is an example of a log file:

# sample log file
2024-04-25 09:00:00 INFO Startup: Application starting
2024-04-25 09:01:00 INFO Config: Configuration loaded successfully
2024-04-25 09:02:00 DEBUG Database: Database connection established
2024-04-25 09:03:00 INFO User: New user registered (UserID: 1001)
2024-04-25 09:04:00 WARN Security: Attempted login with incorrect credentials (UserID: 1001)
2024-04-25 09:05:00 ERROR Network: Network timeout on request (ReqID: 456)
2024-04-25 09:06:00 INFO Email: Notification email sent (UserID: 1001)
2024-04-25 09:07:00 DEBUG API: API call with response time over threshold (Duration: 350ms)
2024-04-25 09:08:00 INFO Session: User session ended (UserID: 1001)
2024-04-25 09:09:00 INFO Shutdown: Application shutdown initiated

A log file usually contains the following columns:

• Timestamp: The date and time when the event occurred.

• Log Level: The severity of the event (INFO, DEBUG, WARN, ERROR).

• Component: The component of the system that generated the event (Startup, Config, Database, User, Security, Network, Email, API, Session, Shutdown).

• Message: A description of the event that occurred.

• Additional Information: Additional information related to the event.

In real-time systems, log files tend to be thousands of lines long and are generated every second. They can be very wordy depending on the configuration. Every column in a log file is a piece of information that can be used to track down issues. This makes log files difficult to read and understand manually.

This is where log parsing comes in. Log parsing is the process of extracting useful information from log files. It involves breaking down the log files into smaller, more manageable pieces, and extracting the relevant information.

The filtered information can also be useful for creating alerts, reports, and dashboards.

In this section, you will explore some techniques for parsing log files in Linux.

Text extraction using grep

Grep is a built-in bash utility. It stands for "global regular expression print". Grep is used to match strings in files.

Here are some common uses of grep:

1. Search for a specific string in a file:

    grep "search_string" filename
   

   This command searches for "search_string" in the file named filename.

2. Search recursively in directories:

    grep -r "search_string" /path/to/directory
   

   This command searches for "search_string" in all files within the specified directory and its subdirectories.

3. Ignore case while searching:

    grep -i "search_string" filename
   

   This command performs a case-insensitive search for "search_string" in the file named filename.

4. Display line numbers with matching lines:

    grep -n "search_string" filename
   

   This command shows the line numbers along with the matching lines in the file named filename.

5. Count the number of matching lines:

    grep -c "search_string" filename
   

   This command counts the number of lines that contain "search_string" in the file named filename.

6. Invert match to display lines that do not match:

    grep -v "search_string" filename
   

   This command displays all lines that do not contain "search_string" in the file named filename.

7. Search for a whole word:

    grep -w "word" filename
   

   This command searches for the whole word "word" in the file named filename.

8. Use extended regular expressions:

    grep -E "pattern" filename
   

   This command allows the use of extended regular expressions for more complex pattern matching in the file named filename.

💡 Tip: If there are multiple files in a folder, you can use the below command to find the list of files containing the desired strings.

# find the list of files containing the desired strings
grep -l "String to Match" /path/to/directory

Text extraction using sed

sed stands for "stream editor". It processes data stream-wise, meaning it reads data one line at a time. sed allows you to search for patterns and perform actions on the lines that match those patterns.

Basic syntax ofsed:

The basic syntax of sed is as follows:

sed [options] 'command' file_name

Here, command is used to perform operations like substitution, deletion, insertion, and so on, on the text data. The filename is the name of the file you want to process.

sedusage:

1. Substitution:

The s flag is used to replace text. The old-text is replaced with new-text:

sed 's/old-text/new-text/' filename

For example, to change all instances of "error" to "warning" in the log file system.log:

sed 's/error/warning/' system.log

2. Printing lines containing a specific pattern:

Using sed to filter and display lines that match a specific pattern:

sed -n '/pattern/p' filename

For instance, to find all lines containing "ERROR":

sed -n '/ERROR/p' system.log

3. Deleting lines containing a specific pattern:

You can delete lines from the output that match a specific pattern:

sed '/pattern/d' filename

For example, to remove all lines containing "DEBUG":

sed '/DEBUG/d' system.log

4. Extracting specific fields from a log line:

You can use regular expressions to extract parts of lines. Suppose each log line starts with a date in the format "YYYY-MM-DD". You could extract just the date from each line:

sed -n 's/^\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\).*/\1/p' system.log

Text parsing with awk

awk has the ability to easily split each line into fields. It's well-suited for processing structured text like log files.

Basic syntax ofawk

The basic syntax of awk is:

awk 'pattern { action }' file_name

Here, pattern is a condition that must be met for the action to be performed. If the pattern is omitted, the action is performed on every line.

In the coming examples, you'll use this log file as an example:

2024-04-25 09:00:00 INFO Startup: Application starting
2024-04-25 09:01:00 INFO Config: Configuration loaded successfully
2024-04-25 09:02:00 INFO Database: Database connection established
2024-04-25 09:03:00 INFO User: New user registered (UserID: 1001)
2024-04-25 09:04:00 INFO Security: Attempted login with incorrect credentials (UserID: 1001)
2024-04-25 09:05:00 INFO Network: Network timeout on request (ReqID: 456)
2024-04-25 09:06:00 INFO Email: Notification email sent (UserID: 1001)
2024-04-25 09:07:00 INFO API: API call with response time over threshold (Duration: 350ms)
2024-04-25 09:08:00 INFO Session: User session ended (UserID: 1001)
2024-04-25 09:09:00 INFO Shutdown: Application shutdown initiated
  INFO

• Accessing columns usingawk

The fields in awk (separated by spaces by default) can be accessed using $1, $2, $3, and so on.

zaira@zaira-ThinkPad:~$ awk '{ print $1 }' sample.log
# output
2024-04-25
2024-04-25
2024-04-25
2024-04-25
2024-04-25
2024-04-25
2024-04-25
2024-04-25
2024-04-25
2024-04-25

zaira@zaira-ThinkPad:~$ awk '{ print $2 }' sample.log
# output
09:00:00
09:01:00
09:02:00
09:03:00
09:04:00
09:05:00
09:06:00
09:07:00
09:08:00
09:09:00

• Print lines containing a specific pattern (for example, ERROR)

awk '/ERROR/ { print $0 }' logfile.log

# output
2024-04-25 09:05:00 ERROR Network: Network timeout on request (ReqID: 456)

This prints all lines that contain "ERROR".

• Extract the first field (Date and Time)

awk '{ print $1, $2 }' logfile.log
# output
2024-04-25 09:00:00
2024-04-25 09:01:00
2024-04-25 09:02:00
2024-04-25 09:03:00
2024-04-25 09:04:00
2024-04-25 09:05:00
2024-04-25 09:06:00
2024-04-25 09:07:00
2024-04-25 09:08:007
2024-04-25 09:09:00

This will extract the first two fields from each line, which in this case would be the date and time.

• Summarize occurrences of each log level

awk '{ count[$3]++ } END { for (level in count) print level, count[level] }' logfile.log

# output
 1
WARN 1
ERROR 1
DEBUG 2
INFO 6

The output will be a summary of the number of occurrences of each log level.

• Filter out specific fields (for example, where the 3rd field is INFO)

awk '{ $3="INFO"; print }' sample.log

# output
2024-04-25 09:00:00 INFO Startup: Application starting
2024-04-25 09:01:00 INFO Config: Configuration loaded successfully
2024-04-25 09:02:00 INFO Database: Database connection established
2024-04-25 09:03:00 INFO User: New user registered (UserID: 1001)
2024-04-25 09:04:00 INFO Security: Attempted login with incorrect credentials (UserID: 1001)
2024-04-25 09:05:00 INFO Network: Network timeout on request (ReqID: 456)
2024-04-25 09:06:00 INFO Email: Notification email sent (UserID: 1001)
2024-04-25 09:07:00 INFO API: API call with response time over threshold (Duration: 350ms)
2024-04-25 09:08:00 INFO Session: User session ended (UserID: 1001)
2024-04-25 09:09:00 INFO Shutdown: Application shutdown initiated
  INFO

This command will extract all lines where the 3rd field is "INFO".

💡 Tip: The default separator in awk is a space. If your log file uses a different separator, you can specify it using the -F option. For example, if your log file uses a colon as a separator, you can use awk -F: '{ print $1 }' logfile.log to extract the first field.

Parsing log files with cut

The cut command is a simple yet powerful command used to extract sections of text from each line of input. As log files are structured and each field is delimited by a specific character, such as a space, tab, or a custom delimiter, cut does a very good job of extracting those specific fields.

The basic syntax of the cut command is:

cut [options] [file]

Some commonly used options for the cut command:

• -d : Specifies a delimiter used as the field separator.

• -f : Selects the fields to be displayed.

• -c : Specifies character positions.

For example, the command below would extract the first field (separated by a space) from each line of the log file:

cut -d ' ' -f 1 logfile.log

Examples of usingcutfor log parsing

Assume you have a log file structured as follows, where fields are space-separated:

2024-04-25 08:23:01 INFO 192.168.1.10 User logged in successfully.
2024-04-25 08:24:15 WARNING 192.168.1.10 Disk usage exceeds 90%.
2024-04-25 08:25:02 ERROR 10.0.0.5 Connection timed out.
...

cut can be used in the following ways:

1. Extracting the time from each log entry:

cut -d ' ' -f 2 system.log

# Output
08:23:01
08:24:15
08:25:02
...

This command uses a space as a delimiter and selects the second field, which is the time component of each log entry.

2. Extracting the IP addresses from the logs:

cut -d ' ' -f 4 system.log

# Output
192.168.1.10
192.168.1.10
10.0.0.5

This command extracts the fourth field, which is the IP address from each log entry.

3. Extracting log levels (INFO, WARNING, ERROR):

cut -d ' ' -f 3 system.log

# Output
INFO
WARNING
ERROR

This extracts the third field which contains the log level.

4. Combiningcutwith other commands:

The output of other commands can be piped to the cut command. Let's say you want to filter logs before cutting. You can use grep to extract lines containing "ERROR" and then use cut to get specific information from those lines:

grep "ERROR" system.log | cut -d ' ' -f 1,2 

# Output
2024-04-25 08:25:02

This command first filters lines that include "ERROR", then extracts the date and time from these lines.

5. Extracting multiple fields:

It is possible to extract multiple fields at once by specifying a range or a comma-separated list of fields:

cut -d ' ' -f 1,2,3 system.log` 

# Output
2024-04-25 08:23:01 INFO
2024-04-25 08:24:15 WARNING
2024-04-25 08:25:02 ERROR
...

The above command extracts the first three fields from each log entry that are date, time, and log level.

Parsing log files with sort and uniq

Sorting and removing duplicates are common operations when working with log files. The sort and uniq commands are powerful commands used to sort and remove duplicates from the input, respectively.

Basic syntax of sort

The sort command organizes lines of text alphabetically or numerically.

sort [options] [file]

Some key options for the sort command:

• -n: Sorts the file assuming the contents are numerical.

• -r: Reverses the order of sort.

• -k: Specifies a key or column number to sort on.

• -u: Sorts and removes duplicate lines.

The uniq command is used to filter or count and report repeated lines in a file.

The syntax of uniq is:

uniq [options] [input_file] [output_file]

Some key options for the uniq command are:

• -c: Prefixes lines by the number of occurrences.

• -d: Only prints duplicate lines.

• -u: Only prints unique lines.

Examples of using sort and uniq together for log parsing

Let's assume the following example log entries for these demonstrations:

2024-04-25 INFO User logged in successfully.
2024-04-25 WARNING Disk usage exceeds 90%.
2024-04-26 ERROR Connection timed out.
2024-04-25 INFO User logged in successfully.
2024-04-26 INFO Scheduled maintenance.
2024-04-26 ERROR Connection timed out.

1. Sorting log entries by date:

sort system.log

# Output
2024-04-25 INFO User logged in successfully.
2024-04-25 INFO User logged in successfully.
2024-04-25 WARNING Disk usage exceeds 90%.
2024-04-26 ERROR Connection timed out.
2024-04-26 ERROR Connection timed out.
2024-04-26 INFO Scheduled maintenance.

This sorts the log entries alphabetically, which effectively sorts them by date if the date is the first field.

1. Sorting and removing duplicates:

sort system.log | uniq

# Output
2024-04-25 INFO User logged in successfully.
2024-04-25 WARNING Disk usage exceeds 90%.
2024-04-26 ERROR Connection timed out.
2024-04-26 INFO Scheduled maintenance.

This command sorts the log file and pipes it to uniq, removing duplicate lines.

1. Counting occurrences of each line:

sort system.log | uniq -c

# Output
2 2024-04-25 INFO User logged in successfully.
1 2024-04-25 WARNING Disk usage exceeds 90%.
2 2024-04-26 ERROR Connection timed out.
1 2024-04-26 INFO Scheduled maintenance.

Sorts the log entries and then counts each unique line. According to the output, the line '2024-04-25 INFO User logged in successfully.' appeared 2 times in the file.

1. Identifying unique log entries:

sort system.log | uniq -u

# Output

2024-04-25 WARNING Disk usage exceeds 90%.
2024-04-26 INFO Scheduled maintenance.

This command shows lines that are unique.

2. Sorting by log level:

sort -k2 system.log

# Output
2024-04-26 ERROR Connection timed out.
2024-04-26 ERROR Connection timed out.
2024-04-25 INFO User logged in successfully.
2024-04-25 INFO User logged in successfully.
2024-04-26 INFO Scheduled maintenance.
2024-04-25 WARNING Disk usage exceeds 90%.

Sorts the entries based on the second field, which is the log level.

8.4. Managing Linux Processes via Command Line

A process is a running instance of a program. A process consists of:

• An address space of the allocated memory.

• Process states.

• Properties such as ownership, security attributes, and resource usage.

A process also has an environment that consists of:

• Local and global variables

• The current scheduling context

• Allocated system resources, such as network ports or file descriptors.

When you run the ls -l command, the operating system creates a new process to execute the command. The process has an ID, a state, and runs until the command completes.

Understanding process creation and lifecycle

In Ubuntu, all processes originate from the initial system process called systemd, which is the first process started by the kernel during boot.

The systemd process has a process ID (PID) of 1 and is responsible for initializing the system, starting and managing other processes, and handling system services. All other processes on the system are descendants of systemd.

A parent process duplicates its own address space (fork) to create a new (child) process structure. Each new process is assigned a unique process ID (PID) for tracking and security purposes. The PID and the parent's process ID (PPID) are part of the new process environment. Any process can create a child process.

Through the fork routine, a child process inherits security identities, previous and current file descriptors, port and resource privileges, environment variables, and program code. A child process may then execute its own program code.

Typically, a parent process sleeps while the child process runs, setting a request (wait) to be notified when the child completes.

Upon exiting, the child process has already closed or discarded its resources and environment. The only remaining resource, known as a zombie, is an entry in the process table. The parent, signaled awake when the child exits, cleans the process table of the child's entry, thus freeing the last resource of the child process. The parent process then continues executing its own program code.

Understanding process states

Processes in Linux assume different states during their lifecycle. The state of a process indicates what the process is currently doing and how it is interacting with the system. The processes transition between states based on their execution status and the system's scheduling algorithm.

The processes in a Linux system can be in one of the following states:

The processes transition between these states in the following ways:

How to view processes

You can use the ps command along with a combination of options to view processes on a Linux system. The ps command is used to display information about a selection of active processes. For example, ps aux displays all processes running on the system.

zaira@zaira:~$ ps aux
# Output
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0 168140 11352 ?        Ss   May21   0:18 /sbin/init splash
root           2  0.0  0.0      0     0 ?        S    May21   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   May21   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   May21   0:00 [rcu_par_gp]
root           5  0.0  0.0      0     0 ?        I<   May21   0:00 [slub_flushwq]
root           6  0.0  0.0      0     0 ?        I<   May21   0:00 [netns]
root          11  0.0  0.0      0     0 ?        I<   May21   0:00 [mm_percpu_wq]
root          12  0.0  0.0      0     0 ?        I    May21   0:00 [rcu_tasks_kthread]
root          13  0.0  0.0      0     0 ?        I    May21   0:00 [rcu_tasks_rude_kthread]
*... output truncated ....*

The output above shows a snapshot of the currently running processes on the system. Each row represents a process with the following columns:

1. USER: The user who owns the process.

2. PID: The process ID.

3. %CPU: The CPU usage of the process.

4. %MEM: The memory usage of the process.

5. VSZ: The virtual memory size of the process.

6. RSS: The resident set size, that is the non-swapped physical memory that a task has used.

7. TTY: The controlling terminal of the process. A ? indicates no controlling terminal.

8. STAT: The process state.

   • R: Running

   • I or S: Interruptible sleep (waiting for an event to complete)

   • D: Uninterruptible sleep (usually IO)

   • T: Stopped (either by a job control signal or because it is being traced)

   • Z: Zombie (terminated but not reaped by its parent)

   • Ss: Session leader. This is a process that has started a session, and it is a leader of a group of processes and can control terminal signals. The first S indicates the sleeping state, and the second s indicates it is a session leader.

9. START: The starting time or date of the process.

10. TIME: The cumulative CPU time.

11. COMMAND: The command that started the process.

Background and foreground processes

In this section, you'll learn how you can control jobs by running them in the background or foreground.

A job is a process that is started by a shell. When you run a command in the terminal, it is considered a job. A job can run in the foreground or the background.

To demonstrate control, you'll first create 3 processes and then run them in the background. After that, you'll list the processes and alternate them between the foreground and background. You'll see how to put them to sleep or exit completely.

1. Create Three Processes

Open a terminal and start three long-running processes. Use the sleep command, that keeps the process running for a specified number of seconds.

# run sleep command for 300, 400, and 500 seconds
sleep 300 &
sleep 400 &
sleep 500 &

The & at the end of each command moves the process to the background.

2. Display Background Jobs

Use the jobs command to display the list of background jobs.

jobs

The output should look something like this:

jobs
[1]   Running                 sleep 300 &
[2]-  Running                 sleep 400 &
[3]+  Running                 sleep 500 &

3. Bring a Background Job to the Foreground

To bring a background job to the foreground, use the fg command followed by the job number. For example, to bring the first job (sleep 300) to the foreground:

fg %1

This will bring job 1 to the foreground.

4. Move the Foreground Job Back to the Background

While the job is running in the foreground, you can suspend it and move it back to the background by pressing Ctrl+Z to suspend the job.

A suspended job will look like this:

zaira@zaira:~$ fg %1
sleep 300

^Z
[1]+  Stopped                 sleep 300

zaira@zaira:~$ jobs
# suspended job 
[1]+  Stopped                 sleep 300
[2]   Running                 sleep 400 &
[3]-  Running                 sleep 500 &

Now use the bg command to resume the job with ID 1 in the background.

# Press Ctrl+Z to suspend the foreground job
# Then, resume it in the background
bg %1

5. Display the jobs again

jobs
[1]   Running                 sleep 300 &
[2]-  Running                 sleep 400 &
[3]+  Running                 sleep 500 &

In this exercise, you:

• Started three background processes using sleep commands.

• Used jobs to display the list of background jobs.

• Brought a job to the foreground with fg %job_number.

• Suspended the job with Ctrl+Z and moved it back to the background with bg %job_number.

• Used jobs again to verify the status of the background jobs.

Now you know how to control jobs.

Killing processes

It is possible to terminate an unresponsive or unwanted process using the kill command. The kill command sends a signal to a process ID, asking it to terminate.

A number of options are available with the kill command.

# Options available with kill

kill -l
 1) SIGHUP     2) SIGINT     3) SIGQUIT     4) SIGILL     5) SIGTRAP
 6) SIGABRT     7) SIGBUS     8) SIGFPE     9) SIGKILL    10) SIGUSR1
11) SIGSEGV    12) SIGUSR2    13) SIGPIPE    14) SIGALRM    15) SIGTERM
16) SIGSTKFLT    17) SIGCHLD    18) SIGCONT    19) SIGSTOP    20) SIGTSTP
21) SIGTTIN    22) SIGTTOU    23) SIGURG    24) 
...terminated

Here are some examples of the kill command in Linux:

1. Kill a process by PID (Process ID):

    kill 1234
   

   This command sends the default SIGTERM signal to the process with PID 1234, requesting it to terminate.

2. Kill a process by name:

    pkill process_name
   

   This command sends the default SIGTERM signal to all processes with the specified name.

3. Forcefully kill a process:

    kill -9 1234
   

   This command sends the SIGKILL signal to the process with PID 1234, forcefully terminating it.

4. Send a specific signal to a process:

    kill -s SIGSTOP 1234
   

   This command sends the SIGSTOP signal to the process with PID 1234, stopping it.

5. Kill all processes owned by a specific user:

    pkill -u username
   

   This command sends the default SIGTERM signal to all processes owned by the specified user.

These examples demonstrate various ways to use the kill command to manage processes in a Linux environment.

Here is the information about the kill command options and signals in a tabular form: This table summarizes the most common kill command options and signals used in Linux for managing processes.

8.5. Standard Input and Output Streams in Linux

Reading an input and writing an output is an essential part of understanding the command line and shell scripting. In Linux, every process has three default streams:

1. Standard Input (stdin): This stream is used for input, typically from the keyboard. When a program reads from stdin, it receives data entered by the user or redirected from a file. A file descriptor is a unique identifier that the operating system assigns to an open file in order to keep track of open files.

   The file descriptor for stdin is 0.

2. Standard Output (stdout): This is the default output stream where a process writes its output. By default, the standard output is the terminal. The output can also be redirected to a file or another program. The file descriptor for stdout is 1.

3. Standard Error (stderr): This is the default error stream where a process writes its error messages. By default, the standard error is the terminal, allowing error messages to be seen even if stdout is redirected. The file descriptor for stderr is 2.

Redirection and Pipelines

Redirection: You can redirect the error and output streams to files or other commands. For example:

# Redirecting stdout to a file
ls > output.txt

# Redirecting stderr to a file
ls non_existent_directory 2> error.txt

# Redirecting both stdout and stderr to a file
ls non_existent_directory > all_output.txt 2>&1

In the last command,

• ls non_existent_directory: lists the contents of a directory named non_existent_directory. Since this directory does not exist, ls will generate an error message.

• > all_output.txt: The > operator redirects the standard output (stdout) of the ls command to the file all_output.txt. If the file does not exist, it will be created. If it does exist, its contents will be overwritten.

• 2>&1:: Here, 2 represents the file descriptor for standard error (stderr). &1 represents the file descriptor for standard output (stdout). The & character is used to specify that 1 is not the file name but a file descriptor.

So, 2>&1 means "redirect stderr (2) to wherever stdout (1) is currently going," which in this case is the file all_output.txt. Therefore, both the output (if there were any) and the error message from ls will be written to all_output.txt.

Pipelines:

You can use pipes (|) to pass the output of one command as the input to another:

ls | grep image
# Output
image-10.png
image-11.png
image-12.png
image-13.png
... Output truncated ...

8.6 Automation in Linux – Automate Tasks with Cron Jobs

Cron is a powerful utility for job scheduling that is available in Unix-like operating systems. By configuring cron, you can set up automated jobs to run on a daily, weekly, monthly, or other specific time basis. The automation capabilities provided by cron play a crucial role in Linux system administration.

The crond daemon (a type of computer program that runs in the background) enables cron functionality. The cron reads the crontab (cron tables) for running predefined scripts.

By using a specific syntax, you can configure a cron job to schedule scripts or other commands to run automatically.

What are cron jobs in Linux?

Any task that you schedule through crons is called a cron job.

Now, let's see how cron jobs work.

How to control access to crons

In order to use cron jobs, an admin needs to allow cron jobs to be added for users in the /etc/cron.allow file.

If you get a prompt like this, it means you don't have permission to use cron.

To allow John to use crons, include his name in /etc/cron.allow. Create the file if it doesn't exist. This will allow John to create and edit cron jobs.

Users can also be denied access to cron job access by entering their usernames in the file /etc/cron.d/cron.deny.

How to add cron jobs in Linux

First, to use cron jobs, you'll need to check the status of the cron service. If cron is not installed, you can easily download it through the package manager. Just use this to check:

# Check cron service on Linux system
sudo systemctl status cron.service

Cron job syntax

Crontabs use the following flags for adding and listing cron jobs:

• crontab -e: edits crontab entries to add, delete, or edit cron jobs.

• crontab -l: list all the cron jobs for the current user.

• crontab -u username -l: list another user's crons.

• crontab -u username -e: edit another user's crons.

When you list crons and they exist, you'll see something like this:

# Cron job example
* * * * * sh /path/to/script.sh

In the above example,

• * represents minute(s) hour(s) day(s) month(s) weekday(s), respectively. See details of these values below:

• sh represents that the script is a bash script and should be run from /bin/bash.

• /path/to/script.sh specifies the path to the script.

Below is a summary of the cron job syntax:

*   *   *   *   *  sh /path/to/script/script.sh
|   |   |   |   |              |
|   |   |   |   |      Command or Script to Execute        
|   |   |   |   |
|   |   |   |   |
|   |   |   |   |
|   |   |   | Day of the Week(0-6)
|   |   |   |
|   |   | Month of the Year(1-12)
|   |   |
|   | Day of the Month(1-31)  
|   |
| Hour(0-23)  
|
Min(0-59)

Cron job examples

Below are some examples of scheduling cron jobs.

It's okay if you are unable to grasp this all at once. You can practice and generate cron schedules with the crontab guru website.

How to set up a cron job

In this section, we will look at an example of how to schedule a simple script with a cron job.

1. Create a script called date-script.sh which prints the system date and time and appends it to a file. The script is shown below:

#!/bin/bash

echo `date` >> date-out.txt

2. Make the script executable by giving it execution rights.

chmod 775 date-script.sh

3. Add the script in the crontab using crontab -e.

Here, we have scheduled it to run per minute.

*/1 * * * * /bin/sh /root/date-script.sh

4. Check the output of the file date-out.txt. According to the script, the system date should be printed to this file every minute.

cat date-out.txt
# output
Wed 26 Jun 16:59:33 PKT 2024
Wed 26 Jun 17:00:01 PKT 2024
Wed 26 Jun 17:01:01 PKT 2024
Wed 26 Jun 17:02:01 PKT 2024
Wed 26 Jun 17:03:01 PKT 2024
Wed 26 Jun 17:04:01 PKT 2024
Wed 26 Jun 17:05:01 PKT 2024
Wed 26 Jun 17:06:01 PKT 2024
Wed 26 Jun 17:07:01 PKT 2024

How to troubleshoot crons

Crons are really helpful, but they might not always work as intended. Fortunately, there are some effective methods you can use to troubleshoot them.

1. Check the schedule.

First, you can try verifying the schedule that's set for the cron. You can do that with the syntax you saw in the above sections.

2. Check cron logs.

First, you need to check if the cron has run at the intended time or not. In Ubuntu, you can verify this from the cron logs located at /var/log/syslog.

If there is an entry in these logs at the correct time, it means the cron has run according to the schedule you set.

Below are the logs of our cron job example. Note the first column which shows the timestamp. The path of the script is also mentioned at the end of the line. Line #1, 3, and 5 show that the script ran as intended.

1 Jun 26 17:02:01 zaira-ThinkPad CRON[27834]: (zaira) CMD (/bin/sh /home/zaira/date-script.sh)
2 Jun 26 17:02:02 zaira-ThinkPad systemd[2094]: Started Tracker metadata extractor.
3 Jun 26 17:03:01 zaira-ThinkPad CRON[28255]: (zaira) CMD (/bin/sh /home/zaira/date-script.sh)
4 Jun 26 17:03:02 zaira-ThinkPad systemd[2094]: Started Tracker metadata extractor.
5 Jun 26 17:04:01 zaira-ThinkPad CRON[28538]: (zaira) CMD (/bin/sh /home/zaira/date-script.sh)

3. Redirect cron output to a file.

You can redirect a cron's output to a file and check the file for any possible errors.

# Redirect cron output to a file
* * * * * sh /path/to/script.sh &> log_file.log

8.7. Linux Networking Basics

Linux offers a number of commands to view network related information. In this section we will briefly discuss some of the commands.

View network interfaces with ifconfig

The ifconfig command gives information about network interfaces. Here is an example output:

ifconfig

# Output
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.100  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::a00:27ff:fe4e:66a1  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:4e:66:a1  txqueuelen 1000  (Ethernet)
        RX packets 1024  bytes 654321 (654.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 512  bytes 123456 (123.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 256  bytes 20480 (20.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 256  bytes 20480 (20.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The output of the ifconfig command shows the network interfaces configured on the system, along with details such as IP addresses, MAC addresses, packet statistics, and more.

These interfaces can be physical or virtual devices.

To extract IPv4 and IPv6 addresses, you can use ip -4 addr and ip -6 addr, respectively.

View network activity withnetstat

The netstat command shows network activity and stats by giving the following information:

Here are some examples of using the netstat command in the command line:

1. Display all listening and non-listening sockets:

    netstat -a
   

2. Show only listening ports:

    netstat -l
   

3. Display network statistics:

    netstat -s
   

4. Show routing table:

    netstat -r
   

5. Display TCP connections:

    netstat -t
   

6. Display UDP connections:

    netstat -u
   

7. Show network interfaces:

    netstat -i
   

8. Display PID and program names for connections:

    netstat -p
   

9. Show statistics for a specific protocol (for example, TCP):

    netstat -st
   

10. Display extended information:

    netstat -e
    

Check network connectivity between two devices using ping

ping is used to test network connectivity between two devices. It sends ICMP packets to the target device and waits for a response.

ping google.com

ping tests if you get a response back without getting a timeout.

ping google.com
PING google.com (142.250.181.46) 56(84) bytes of data.
64 bytes from fjr04s06-in-f14.1e100.net (142.250.181.46): icmp_seq=1 ttl=60 time=78.3 ms
64 bytes from fjr04s06-in-f14.1e100.net (142.250.181.46): icmp_seq=2 ttl=60 time=141 ms
64 bytes from fjr04s06-in-f14.1e100.net (142.250.181.46): icmp_seq=3 ttl=60 time=205 ms
64 bytes from fjr04s06-in-f14.1e100.net (142.250.181.46): icmp_seq=4 ttl=60 time=100 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 78.308/131.053/204.783/48.152 ms

You can stop the response with Ctrl + C.

Testing endpoints with the curl command

The curl command stands for "client URL". It is used to transfer data to or from a server. It can also be used to test API endpoints that helps in troubleshooting system and application errors.

As an example, you can use http://www.official-joke-api.appspot.com/ to experiment with the curl command.

• The curl command without any options uses the GET method by default.

curl http://www.official-joke-api.appspot.com/random_joke
{"type":"general",
"setup":"What did the fish say when it hit the wall?","punchline":"Dam.","id":1}

• curl -o saves the output to the mentioned file.

curl -o random_joke.json http://www.official-joke-api.appspot.com/random_joke
# saves the output to random_joke.json

• curl -I fetches only the headers.

curl -I http://www.official-joke-api.appspot.com/random_joke
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Vary: Accept-Encoding
X-Powered-By: Express
Access-Control-Allow-Origin: *
ETag: W/"71-NaOSpKuq8ChoxdHD24M0lrA+JXA"
X-Cloud-Trace-Context: 2653a86b36b8b131df37716f8b2dd44f
Content-Length: 113
Date: Thu, 06 Jun 2024 10:11:50 GMT
Server: Google Frontend

8.8. Linux Troubleshooting: Tools and Techniques

System activity report with sar

The sar command in Linux is a powerful tool for collecting, reporting, and saving system activity information. It's part of the sysstat package and is widely used for monitoring system performance over time.

To use sar you first need to install syssstat using sudo apt install sysstat.

Once installed, start the service with sudo systemctl start sysstat.

Verify the status with sudo systemctl status sysstat.

Once the status is active, the system will start collecting various stats that you can use to access and analyze historical data. We'll see that in detail soon.

The syntax of the sar command is as follows:

sar [options] [interval] [count]

For example, sar -u 1 3 will display CPU utilization statistics every second for three times.

sar -u 1 3
# Output
Linux 6.5.0-28-generic (zaira-ThinkPad)     04/06/24     _x86_64_    (12 CPU)

19:09:26        CPU     %user     %nice   %system   %iowait    %steal     %idle
19:09:27        all      3.78      0.00      2.18      0.08      0.00     93.96
19:09:28        all      4.02      0.00      2.01      0.08      0.00     93.89
19:09:29        all      6.89      0.00      2.10      0.00      0.00     91.01
Average:        all      4.89      0.00      2.10      0.06      0.00     92.95

Here are some common use cases and examples of how to use the sar command.

sar can be used for a variety of purposes:

1. Memory usage

To check memory usage (free and used), use:

sar -r 1 3

Linux 6.5.0-28-generic (zaira-ThinkPad)     04/06/24     _x86_64_    (12 CPU)

19:10:46    kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
19:10:47      4600104   8934352   5502124     36.32    375844   4158352  15532012     65.99   6830564   2481260       264
19:10:48      4644668   8978940   5450252     35.98    375852   4165648  15549184     66.06   6776388   2481284        36
19:10:49      4646548   8980860   5448328     35.97    375860   4165648  15549224     66.06   6774368   2481292       116
Average:      4630440   8964717   5466901     36.09    375852   4163216  15543473     66.04   6793773   2481279       139

This command displays memory statistics every second three times.

2. Swap space utilization

To view swap space utilization statistics, use:

sar -S 1 3

sar -S 1 3
Linux 6.5.0-28-generic (zaira-ThinkPad)     04/06/24     _x86_64_    (12 CPU)

19:11:20    kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
19:11:21      8388604         0      0.00         0      0.00
19:11:22      8388604         0      0.00         0      0.00
19:11:23      8388604         0      0.00         0      0.00
Average:      8388604         0      0.00         0      0.00

This command helps monitor the swap usage, which is crucial for systems running out of physical memory.

3. I/O devices load

To report activity for block devices and block device partitions:

sar -d 1 3

This command provides detailed stats about data transfers to and from block devices, and is useful for diagnosing I/O bottlenecks.

5. Network statistics

To view network statistics, like number of packets received (transmitted) by the network interface:

sar -n DEV 1 3
# -n DEV tells sar to report network device interfaces
sar -n DEV 1 3
Linux 6.5.0-28-generic (zaira-ThinkPad)     04/06/24     _x86_64_    (12 CPU)

19:12:47        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
19:12:48           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
19:12:48       enp2s0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
19:12:48       wlp3s0     10.00      3.00      1.83      0.37      0.00      0.00      0.00      0.00
19:12:48    br-5129d04f972f      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
.
.
.

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:       enp2s0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
...output truncated...

This displays network statistics every second for three seconds, helping in monitoring network traffic.

6. Historical data

Recall that previously we installed the sysstat package and ran the service. Follow the steps below to enable and access historical data.

1. Enable data collection: Edit the sysstat configuration file to enable data collection.

    sudo nano /etc/default/sysstat
   

   Change ENABLED="false" to ENABLED="true".

    vim /etc/default/sysstat
    #
    # Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat
    # and /etc/cron.daily/sysstat files
    #
   
    # Should sadc collect system activity informations? Valid values
    # are "true" and "false". Please do not put other values, they
    # will be overwritten by debconf!
    ENABLED="true"
   

2. Configure data collection interval: Edit the cron job configuration to set the data collection interval.

    sudo nano /etc/cron.d/sysstat
   

   By default, it collects data every 10 minutes. You can adjust the interval by modifying the cron job schedule. The relevant files will go to the /var/log/sysstat folder.

3. View historical data: Use the sar command to view historical data. For example, to view CPU usage for the current day:

    sar -u
   

   To view data from a specific date:

    sar -u -f /var/log/sysstat/sa<DD>
   

   Replace <DD> with the day of the month for which you want to view the data.

   In the below command, /var/log/sysstat/sa04 gives stats for the 4th day of the current month.

sar -u -f /var/log/sysstat/sa04
Linux 6.5.0-28-generic (zaira-ThinkPad)     04/06/24     _x86_64_    (12 CPU)

15:20:49     LINUX RESTART    (12 CPU)

16:13:30     LINUX RESTART    (12 CPU)

18:16:00        CPU     %user     %nice   %system   %iowait    %steal     %idle
18:16:01        all      0.25      0.00      0.67      0.08      0.00     99.00
Average:        all      0.25      0.00      0.67      0.08      0.00     99.00

7. Real-Time CPU Interruptions

To observe real-time interrupts per second served by the CPU, use this command:

sar -I SUM 1 3

# Output
Linux 6.5.0-28-generic (zaira-ThinkPad)     04/06/24     _x86_64_    (12 CPU)

19:14:22         INTR    intr/s
19:14:23          sum   5784.00
19:14:24          sum   5694.00
19:14:25          sum   5795.00
Average:          sum   5757.67

This command helps in monitoring how frequently the CPU is handling interrupts, which can be crucial for real-time performance tuning.

These examples illustrate how you can use sar to monitor various aspects of system performance. Regular use of sar can help in identifying system bottlenecks and ensuring that applications keep running efficiently.

8.9. General Troubleshooting Strategy for Servers

Why do we need to understand monitoring?

System monitoring is an important aspect of system administration. Critical applications demand a high level of proactiveness to prevent failure and reduce the outage impact.

Linux offers very powerful tools to gauge system health. In this section, you'll learn about the various methods available to check your system's health and identify the bottlenecks.

Find load average and system uptime

System reboots may occur which can sometimes mess up some configurations. To check how long the machine has been up, use the command: uptime. In addition to the uptime, the command also displays load average.

[user@host ~]$ uptime 19:15:00 up 1:04, 0 users, load average: 2.92, 4.48, 5.20

Load average is the system load over the last 1, 5, and 15 minutes. A quick glance indicates whether the system load appears to be increasing or decreasing over time.

Note: Ideal CPU queue is 0. This is only possible when there are no waiting queues for the CPU.

Per-CPU load can be calculated by dividing load average with the total number of CPUs available.

To find the number of CPUs, use the command lscpu.

lscpu
# output
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
.
.
.
output omitted

If the load average seems to increase and does not come down, the CPUs are overloaded. There is some process that is stuck or there is a memory leakage.

Calculating free memory

Sometimes, high memory utilization might be causing problems. To check the available memory and the memory in use, use the free command.

free -mh
# output
               total        used        free      shared  buff/cache   available
Mem:            14Gi       3.5Gi       7.7Gi       109Mi       3.2Gi        10Gi
Swap:          8.0Gi          0B       8.0Gi

Calculating disk space

To ensure the system is healthy, don't forget about the disk space. To list all the available mount points and their respective used percentage, use the below command. Ideally, utilized disk spaces should not exceed 80%.

The df command provides detailed disk spaces.

df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.5G  2.4M  1.5G   1% /run
/dev/nvme0n1p2  103G   34G   65G  35% /
tmpfs           7.3G   42M  7.2G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
efivarfs        246K   93K  149K  39% /sys/firmware/efi/efivars
/dev/nvme0n1p3  130G   47G   77G  39% /home
/dev/nvme0n1p1  511M  6.1M  505M   2% /boot/efi
tmpfs           1.5G  140K  1.5G   1% /run/user/1000

Determining process states

Process states can be monitored to see any stuck process with a high memory or CPU usage.

We saw previously that the ps command gives useful information about a process. Have a look at the CPU and MEM columns.

[user@host ~]$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 runner         1  0.1  0.0 1535464 15576 ?       S  19:18   0:00 /inject/init
 runner        14  0.0  0.0  21484  3836 pts/0    S   19:21   0:00 bash --norc
 runner        22  0.0  0.0  37380  3176 pts/0    R+   19:23   0:00 ps aux

Real-time system monitoring

Real time monitoring gives a window into the realtime system state.

One utility you can use to do this is the top command.

The top command displays a dynamic view of the system's processes, displaying a summary header followed by a process or thread list. Unlike its static counterpart ps, top continuously refreshes the system stats.

With top, you can see well-organised details in a compact window. There a number of flags, shortcuts, and highlighting methods that come along with top.

You can also kill processes using top. For that, press k and then enter the process id.

Interpreting logs

System and application logs carry tons of information about what the system is going through. They contain useful information and error codes that point towards errors. If you search for error codes in logs, issue identification and rectification time can be greatly reduced.

Network ports analysis

The network aspect should not be ignored as network glitches are common and may impact the system and traffic flows. Common network issues include port exhaustion, port choking, unreleased resources, and so on.

To identify such issues, we need to understand port states.

Some of the port states are explained briefly here:

Let's explore how we can analyze port-related information in Linux.

Port ranges: Port ranges are defined in the system, and range can be increased/decreased accordingly. In the below snippet, the range is from 15000 to 65000, which makes a total of 50000 (65000 - 15000) available ports. If utilized ports are reaching or exceeding this limit, then there is an issue.

[user@host ~]$ /sbin/sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 15000    65000

The error reported in logs in such cases can be Failed to bind to port or Too many connections.

Identifying packet loss

In system monitoring, we need to ensure that the outgoing and incoming communication is intact.

One helpful command is ping. ping hits the destination system and brings the response back. Note the last few lines of statistics that show packet loss percentage and time.

# ping destination IP
[user@host ~]$ ping 10.13.6.113
 PING 10.13.6.141 (10.13.6.141) 56(84) bytes of data.
 64 bytes from 10.13.6.113: icmp_seq=1 ttl=128 time=0.652 ms
 64 bytes from 10.13.6.113: icmp_seq=2 ttl=128 time=0.593 ms
 64 bytes from 10.13.6.113: icmp_seq=3 ttl=128 time=0.478 ms
 64 bytes from 10.13.6.113: icmp_seq=4 ttl=128 time=0.384 ms
 64 bytes from 10.13.6.113: icmp_seq=5 ttl=128 time=0.432 ms
 64 bytes from 10.13.6.113: icmp_seq=6 ttl=128 time=0.747 ms
 64 bytes from 10.13.6.113: icmp_seq=7 ttl=128 time=0.379 ms
 ^C
 --- 10.13.6.113 ping statistics ---
 7 packets transmitted, 7 received,0% packet loss, time 6001ms
 rtt min/avg/max/mdev = 0.379/0.523/0.747/0.134 ms

Packets can also be captured at runtime using tcpdump. We'll look into it later.

Gathering stats for issue post mortem

It is always a good practice to gather certain stats that would be useful for identifying the root cause later. Usually, after system reboot or services restart, we loose the earlier system snapshot and logs.

Below are some of the methods to capture system snapshot.

• Logs Backup

Before making any changes, copy log files to another location. This is crucial for understanding what condition the system was in during time of issue. Sometimes log files are the only window to look into past system states as other runtime stats are lost.

• TCP Dump

Tcpdump is a command-line utility that allows you to capture and analyze incoming and outgoing network traffic. It is mostly used to help troubleshoot network issues. If you feel that system traffic is being impacted, take tcpdump as follows:

sudo tcpdump -i any -w

# Where,
# -i any captures traffic from all interfaces
# -w specifies the output filename

# Stop the command after a few mins as the file size may increase
# use file extension as .pcap

Once tcpdump is captured, you can use tools like Wireshark to visually analyze the traffic.

8.10 Diagnosing Hardware Problems

Troubleshooting unexpected issues is a part of the learning process. Sometimes, you may notice frequent segmentation faults (SIGSEGV), overheating, or random crashes across unrelated applications. The issue could either be software or hardware related. While software-related issues depend on the specific application itself, hardware issues can be diagnosed with some standard steps.

In this section, we will discuss how to diagnose and rule out hardware issues related to memory, CPU, system sensors, power supply, and more.

8.10.1 Analyzing Memory Performance

Determine Available RAM

If you feel your system is getting slow and taking longer to finish tasks, check your system's available memory. This will ensure there is enough available memory including the swap memory.

The command to check available memory is free -mh, where -h is for human-readable output and -m is for displaying memory in MB.

free -mh
               total        used        free      shared  buff/cache   available
Mem:            14Gi       5.1Gi       2.4Gi        77Mi       7.3Gi       9.3Gi
Swap:          4.0Gi          0B       4.0Gi

In the above output, look at the "available" column in the "Mem" row. This shows how much RAM is free for use.

Another way to check the memory in real time is to use the top command. There are 2 ways to do this:

• When you are in top, press Shift + M to sort the processes by memory usage.

• Alternately, press m to see the memory usage in a progress bar like format:

If you see the memory consumed near to 100%, you might want to consider identifying the process that is consuming the memory and take necessary action. You might also want to consider adding more memory to your system.

Run a stress test on your hardware

The memtester command is a utility used for diagnosing memory-related issues by stressing the memory and checking for faults. It is often used in situations where you suspect faulty RAM might be causing system instability or crashes.

Here's how to use it effectively:

• First, install memtester.

    sudo apt install memtester
  

• Determine the amount of RAM to test and the number of passes you’d like your RAM to go through. In the command below, 1G is the amount of RAM to test (1 GB), and 5 is the number of test passes:

    sudo memtester 1G 5
  

If all tests pass, your RAM is likely error-free. If errors are reported, your RAM might be faulty and could require replacement or further inspection. You can always run the test again with a different amount of RAM or test passes.

Note that, you shouldn't test too much memory at once, as your system also needs memory for running processes. If you have more RAM than can be tested at once, test in smaller segments sequentially.

Below is a snippet of the memtester output if all tests pass. Notice the ”ok” status for each test.

memtester version 4.5.1 (64-bit)
Copyright (C) 2001-2020 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 1024MB (1073741824 bytes)
got  1024MB (1073741824 bytes), trying mlock ...locked.
Loop 1/5:
  Stuck Address       : ok
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok
  Block Sequential    : ok
  Checkerboard        : ok
  Bit Spread          : ok
  Bit Flip            : ok
  Walking Ones        : ok
  Walking Zeroes      : ok
  8-bit Writes        : ok
  16-bit Writes       : ok
.
.
.

Below is a snippet of the output if a test fails. Notice the FAILURE status for each test.

memtester version 4.5.1 (64-bit)
Copyright (C) 2001-2020 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 1024MB (1073741824 bytes)
got  1024MB (1073741824 bytes), trying mlock ...locked.
Loop 1/5:
  Stuck Address       : testing   1FAILURE: possible bad address line at offset 0x25378a58.
Skipping to next test...
  Random Value        : FAILURE: 0x4df704aaafdf8848 != 0x4df704aaafdfc848 at offset 0x05379a48.
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : testing   6FAILURE: 0x00000000 != 0x00004000 at offset 0x05379a48.
  Block Sequential    : testing   3FAILURE: 0x303030303030303 != 0x303030303034303 at offset 0x05379a48.
  Checkerboard        : testing   0FAILURE: 0xaaaaaaaaaaaaaaaa != 0xaaaaaaaaaaaaeaaa at offset 0x05379a48.
  Bit Spread          : testing  12FAILURE: 0xffffffffffffafff != 0xffffffffffffefff at offset 0x05379a48.
  Bit Flip            : testing   0FAILURE: 0x00000001 != 0x00004001 at offset 0x05379a48.
  Walking Ones        : ok
  Walking Zeroes      : testing   0FAILURE: 0x00000001 != 0x00001001 at offset 0x053af9f8.
  8-bit Writes        : -FAILURE: 0x57c7c8ba7d6f5b3b != 0x57c7c8ba7d6f1b3b at offset 0x0537da28.
  16-bit Writes       : -FAILURE: 0xd7768894fbf79099 != 0xd7768894fbf7d099 at offset 0x05379a48.
FAILURE: 0xfffc5633ffefca5d != 0xfffc5633ffefda5d at offset 0x053a5a38.
.
.
.

If errors persist across all test loops, it strongly suggests hardware issues, not transient software glitches.

8.10.2 Identifying Overheating Issues

Overheating can cause unexpected errors and crashes. To diagnose overheating issues, you can use a command line utility lm-sensors.

lm-sensors allow syou monitor hardware health by reading data from various sensors. It provides information about system temperatures, voltages, and fan speeds.

Here's how you can identify and monitor your system temperature using lm-sensors:

• First, install lm-sensors:

    sudo apt install lm-sensors
  

• Detect the available sensors on your system:

    sudo sensors-detect
  

  Follow the prompts and answer “YES” to detect the available sensors on your system.

• Once the available sensors are detected, you can view the temperature of your system using the sensors command:

    sensors
  

  In the output below, you can see the temperature reading at the edge of the GPU, which is 41.0 degrees Celsius. You can also see other pieces of information like voltage supplied, power consumption and voltage supplied.

    amdgpu-pci-0400
    Adapter: PCI adapter
    vddgfx:      731.00 mV 
    vddnb:       687.00 mV 
    edge:         +41.0°C  
    PPT:           7.00 W
  

  Using lm-sensors ensures that the system is operating within safe parameters. It helps to detect potential hardware problems early and take corrective actions to prevent hardware damage.

8.10.3 Evaluating Hard Drive Health

Disk errors can also cause application crashes. To identify disk issues, you can run disk check using smartmontools:

• First, install smartmontools:

    sudo apt install smartmontools
  

• Run a quick health check using the command below and replace /dev/sdX with your disk name (check with lsblk).

    sudo smartctl -H /dev/sdX
  

• Here is the result I got when I ran the command on my disk /dev/nvme0n1:

    sudo smartctl -H /dev/nvme0n1
    smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-52-generic] (local build)
    Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
  
    === START OF SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
  

• You can also run a detailed test:

    sudo smartctl -a /dev/nvme0n1
  

The detailed test provides a full report, including:

• Temperature

• Power-on hours

• Error counts

• Wear leveling (for SSDs), and more.

8.10.4 Conducting a CPU Stress Test

Faulty CPUs can also lead to a number of performance issues. To test your CPU, you can use the stress-ng utility:

• Install stress-ng:

    sudo apt install stress-ng
  

• Run a CPU stress test:

    stress-ng --cpu 4 --timeout 60
  

In the above command, 4 is the number of CPU cores you’d like to test and 60 is the duration in seconds. The command will stress all 4 CPU cores for 60 seconds. Notice the CPU is at 100% load during the test:

If the system crashes during this test, the CPU may be faulty.

8.10.5 Examining System Logs for Errors

systemd is a Linux system manager responsible for booting the system, managing system processes, and handling system services.

journalctl is a command to query the systemd journal logs. It provides detailed logging for system processes, kernel events, user applications, and more.

You can check system logs for hardware-related errors using the command: journalctl -k | grep -iE "error|fault|panic".

In the logs, look for messages about:

• Memory faults.

• I/O errors.

• Hardware timeouts.

Here is what errors in the log file can look like:

Feb 11 10:15:32 hostname kernel: [Hardware Error]: CPU 0: Machine Check: 0 Bank 4: b200000000070f0f
Feb 11 10:15:32 hostname kernel: [Hardware Error]: TSC 0 ADDR fef1c000 MISC 38a0000086 
Feb 11 10:15:32 hostname kernel: [Hardware Error]: PROCESSOR 0:306a9 TIME 1613045732 SOCKET 0 APIC 0 microcode 1f
Feb 11 10:16:45 hostname kernel: EXT4-fs error (device sda1): ext4_find_entry:1453: inode #2: comm ls: reading directory lblock 0
Feb 11 10:17:12 hostname kernel: [drm:drm_atomic_helper_commit_cleanup_done [drm_kms_helper]] *ERROR* [CRTC:36:pipe A] flip_done timed out
Feb 11 10:18:05 hostname kernel: Kernel panic - not syncing: Fatal exception

Conclusion

Thank you for reading the book until the end. If you found it helpful, consider sharing it with others.

This book doesn't end here, though. I will continue to improve it and add new materials in the future. If you found any issues or if you would like to suggest any improvements, feel free to open a PR/ Issue.

Stay Connected and Continue Your Learning Journey!

Your journey with Linux doesn't have to end here. Stay connected and take your skills to the next level:

1. Follow Me on Social Media:

   • X: I share useful short form content there. My DMs are always open.

   • LinkedIn: I share articles and posts on tech there. Leave a recommendation on LinkedIn and endorse me on relevant skills.

2. Get access to exclusive content: For one-on-one help and exclusive content go here.

My articles and books, like this one, are part of my mission to increase accessibility to quality content for everyone. This book will also be open to translation in other languages. Each piece takes a lot of time and effort to write. This book will be free, forever. If you've enjoyed my work and want to keep me motivated, consider buying me a coffee.

Thank you once again and happy learning!

Rust, known for being fast and safe, is no exception. Let's jump right in and create our very first Rust program together!

How to Write a Hello World Program in Rust

First, create a file named main.rs. Every Rust program contains .rs as its file extension.

Then write the following code in the file:

// main.rs
fn main() {
    println!("Hello World!");
}

In the code above, we are trying to print Hello World! in the console or terminal.

How to Compile Rust Code

In Rust, compiling and running the code are two separate processes.

First, you need to compile the Rust program. To compile it, write the following in the terminal (make sure in the terminal is at at the same directory where the Rust file lives):

rustc main.rs

For now, you won't see any output because the code was just compiled. But one thing you can see in the current directory is that a new executable file has been added with the same name as the Rust file.

How to Run the Executable File

Now you can run the executable file that gets generated after you successfully compiled the Rust code.

To run the executable file, write the following in your terminal:

./main

You should see an output like this:

Hello World!

Understanding the Hello World Code in Depth

fn main() {

}

The program starts with a function called main(). Every Rust executable code starts execution from the main function.

The main function can have some parameters inside the () parenthesis, but we have no need for them in the code so we left that empty.

Everything that is between the curly braces {} is the body of the function. It is necessary to have braces for the body otherwise it'll throw an error.

fn main() {
  println!("Hello World!");
}

This line does the work of printing the "Hello World!" text to the terminal or console.

Here, println! is not a function unlike other languages like C, Python, and so on. It is a macro – if there's an ! symbol at the end of a keyword, then it's a macro.

Finally, we passed the string as argument to the macro and it prints the string to the terminal.

To end the statement in Rust, you must use ;.If you don't provide the semi-colon at the end of every statement, then it'll throw an error.

Conclusion

And there we have it – your first Rust program! By printing out "Hello World!" you've dipped your toes into the world of Rust.

This simple program has given you a taste of Rust's syntax and how it compiles. As you keep going, you'll discover more about Rust's cool features and find out just how powerful it can be.

Ready to dive deeper? Let's keep exploring!

If you have any feedback, then feel free to DM me on Twitter and LinkedIn

Whether you're a newbie just starting out or an experienced developer looking to brush up on your skills, this guide offers a step-by-step approach to understanding and effectively using Git and GitHub.

By the end of this journey, you'll have a solid foundation in Git and GitHub. You'll be equipped with practical knowledge to streamline your coding workflow, collaborate seamlessly with teams, and contribute to open-source projects.

So, let's dive in and get started on your Git and GitHub adventure!

Table of Contents

• Who is this guide for?
• Technologies
• Terms
• What is GitHub?
• What is GitHub used for?
• Common tasks you'll perform with Git
• How to install Git
• How to configure Git
• How to set the default editor
• How to create a repository using the Github website
• How to create a repository using the Git command line
• How to connect a local repository to a remote repository on GitHub
• How to pull changes from a remote repository to a local repository
• How to work with Git commands
• How to make changes to a file
• How to check the status of the current branch
• How to stage changes
• How to commit changes
• How to push changes to a remote repository
• How to create a branch
• How to create a pull request
• How to merge a pull request
• Wrapping Up

Who is This Guide For?

This guide is for everyone who wants to level up their coding skills and become proficient in using Git and GitHub.

Whether you're:

• just starting your tech career and need to learn the basics of version control.
• an aspiring developer eager to integrate Git into your workflow.
• an experienced programmer looking to refresh your knowledge or discover new features.
• a team lead or manager interested in fostering a culture of collaboration and efficient code management.

Regardless of your background or experience, this guide is designed to empower you with the tools and knowledge you need to excel in your coding endeavors.

Technologies

Before you start, make sure:

• You have a GitHub account
• Git is installed on your machine
• You have a text editor, such as Visual Studio Code installed
• Node.js is installed on your machine

Terms

They are a lot of terms around Git and Github that you may meet when you're working with version control. Let me break it down for you before we start:

• Branch: A version of the codebase that diverges from the main branch to isolate changes for specific features, fixes, or experiments.
• Commit: A snapshot of your changes, saved to your local repository. Each commit is uniquely identified by a checksum.
• Stage: The area where Git tracks changes that are ready to be included in the next commit. Files in the staging area are prepared (staged) for the next commit.
• Merge: The process of integrating changes from one branch into another, typically the main branch.
• Pull Request: A proposal to merge changes from one branch into another, often used in collaborative environments to review and discuss changes before they are merged.
• Fork: A personal copy of someone else's project that lives on your GitHub account.
• Clone: The act of downloading a repository from a remote source to your local machine.
• Remote: A common repository that all team members use to exchange their changes.
• Origin: The default name Git gives to the server from which you cloned.
• Upstream: The original repository that was cloned.
• Master: The default branch name given to a repository when it is created. In modern practice, it is often replaced with main.
• Repository: A storage location where your project lives, containing all the files and revision history.
• Working Directory: The directory on your computer where you are making changes to your project.
• Staging Area: Also known as the "Index," it's an area where Git tracks changes that are ready to be committed.
• Index: Another name for the staging area, where Git tracks changes that are ready to be committed.
• HEAD: A reference to the last commit in the currently checked-out branch.
• Checkout: The action of switching from one branch to another or to a specific commit.
• Push: The action of sending your commits to a remote repository.
• Pull: The action of fetching changes from a remote repository and merging them into your current branch.
• Fetch: The action of retrieving updates from a remote repository without merging them into your current branch.

What is GitHub?

GitHub is a platform that hosts code, providing version control and collaboration features. It enables you and others to work together on projects from anywhere in the world.

This guide will introduce you to essential GitHub concepts such as repositories, branches, commits, and Pull Requests. You will learn how to create your own 'Hello World' repository and understand GitHub's Pull Request workflow, a widely-used method for creating and reviewing code.

By the end of this guide, you'll be equipped with the knowledge and skills to collaborate effectively on GitHub.

What is GitHub Used For?

GitHub is more than just a code hosting platform. It's a tool that allows for seamless collaboration and version control. Here are some of its uses:

• Hosting and sharing your code with others.
• Tracking and assigning issues to maintain an organized workflow.
• Managing pull requests to review and incorporate changes into your project.
• Creating your own website using GitHub Pages, a static site hosting service.
• Collaborating with others on projects, making it an excellent tool for open-source contributions.

What is Git?

Git is a free and open-source distributed version control system. It's designed to handle everything from small to very large projects with speed and efficiency

Git is easy to learn and has a tiny footprint with lightning-fast performance. It outclasses SCM tools like Subversion, CVS, Perforce, and ClearCase with features like cheap local branching, convenient staging areas, and multiple workflows

Git was initially designed and developed by Linus Torvalds for Linux kernel development.

Some features/benefits of Git:

• Allows you to track changes to your code over time.
• Enables you to collaborate with others on the same codebase.
• You can easily revert to a previous version of your code or experiment with new features without affecting the main codebase.
• Provides a record of all changes made to your code, including who made them and when which can be useful for auditing and debugging.

Common Tasks You'll Perform with Git

• Create a repository
• Create a branch
• Make changes to a file
• Stage changes
• Commit changes
• Push changes to a remote repository
• Merge changes
• Revert changes
• Delete a branch

How to Install Git

To install Git on your local machine, you need to follow these steps:

1. Download Git from the official website: Git Downloads

2. Install Git on your local machine by following the instructions provided on the official website: Installing Git

Congratulations! You have successfully installed Git on your local machine. You are now ready to start using Git for your projects.

How to Configure Git

Git offers a variety of configurations that can streamline your workflow. In this section, I will guide you through the process of setting up Git on your local machine. Let's get started.

Configuring your name and email address on your local machine is an essential step in setting up Git. These details are attached to each commit you make, providing context and ownership. Let's learn how to use the git config --global command to set your name and email globally on your local machine.

To set up your name, you need to type the following command in your terminal:

# Set a name that is identifiable for credit when reviewing version history

$ git config --global user.name "Your Name"

As you can see in the image above, I have entered my name.

After entering your name, press Enter to save it. You won't receive any response, but rest assured, your name has been stored successfully.

Just like we set up the user name, we also need to set up the user email. This email will be associated with each commit you make. Let's learn how to use the git config --global command to set your email globally on your local machine.

# Set an email address that will be associated with each history marker
$ git config --global user.email "your-email@example.com"

Make sure to replace this with your actual email used in your GitHub account.

Now that we have finished setting up your username and email for Git and GitHub, let's verify that everything is configured correctly.

To do this, use the following command:

git config --global --list

This command will list the username and email being used in the console for you to see.

You should see some information displayed in your terminal.

How to Set the Default Editor

In modern development, having a code editor can significantly simplify your workflow, especially when you're focused on coding.

Now, let's see how to configure Git to use a default editor by using this command:

# Set the default editor for Git
$ git config --global core.editor "code --wait"

Congratulations! You have successfully configured Git on your local machine. You are now ready to start using Git for your projects.

How to Create a Repository Using the Github Website

Creating a repository is the first step in using Git. A repository is a storage location where your projects live, containing all the files and revision history.

In this section, I will guide you through the process of creating a repository on GitHub.

There are two ways to create a repository: using the GitHub website or the command line. Let's get started. In this section, we'll focus on creating a repository using the GitHub website and the command line.

After logging into your GitHub account, you can create a new repository by following these steps:

1. Click on the + sign in the top right corner of the page and select New Repository from the dropdown menu.

Above is an image of the new repository button on GitHub.

2. You will be directed to a new page where you can fill in the details of your new repository. You will need to enter the following information:

3. Repository name: This is the name of your repository. It should be unique and descriptive.

4. Description: A brief description of your repository.
5. Public or Private: You can choose to make your repository public or private. Public repositories are visible to everyone, while private repositories are only visible to you and the people you share them with.
6. Initialize this repository with a README: You can choose to initialize your repository with a README file. This is useful if you want to provide information about your project or instructions on how to use it.

The image above shows the form where you'll fill in the details of your new repository.

3. Once you have filled in the details, click on the Create Repository button to create your new repository.

The image above shows the Create Repository button_.

Congratulations! You have successfully created a new repository on GitHub. You can now start adding files and making changes to your repository.

You should see a page like the one below:

Now let's create a repository using the command line.

How to Create a Repository Using the Git Command Line

To create a new repository using the command line, you need to follow these steps:

1. Open your terminal and navigate to the directory where you want to create your new repository.

2. Use the git init command to create a new repository. This command will create a new directory called .git in your current directory, which will contain all the necessary files for your repository.

# initialize a new repository called my-project

$ git init my-project

The image above shows the command to initialize a new repository called my-project.

3. Once you have created your new repository, you can start adding files and making changes to it. You can also connect your local repository to a remote repository on GitHub by following the instructions provided on the GitHub website.

Congratulations! You have successfully created a new repository using the command line.

Now we have successfully created a repository using the GitHub website and the command line – but how do we connect them? Now let's learn how to connect a local repository to a remote repository on GitHub.

How to Connect a Local Repository to a Remote Repository on GitHub

To connect your local repository to a remote repository on GitHub, you need to follow these steps:

1. On GitHub, navigate to the main page of the repository you created earlier.

2. Click on the Code button to copy the URL of your repository.

The image above shows the code button to copy the URL of your repository.

3. In your terminal, navigate to the directory of your local repository.

4. Use the git remote add origin command to connect your local repository to the remote repository on GitHub. Replace repository-URL with the URL of your repository.

$ git remote add origin repository-url

The image above shows the command to connect your local repository to the remote repository on GitHub.

5. Once you have connected your local repository to the remote repository on GitHub, you can start pushing your changes to the remote repository using the git push command.

Congratulations! You have successfully connected your local repository to the remote repository on GitHub.

How to Pull Changes from a Remote Repository to a Local Repository

To pull changes from the remote repository to the local repository, you need to follow these steps:

1. In your terminal, navigate to the directory of your local repository.

2. Use the git pull command to pull changes from the remote repository to the local repository.

$ git pull origin main

The image above shows the command to pull changes from the remote repository to the local repository.

After that, navigate the main branch by using the following command:

$ git checkout main

Congratulations! You have successfully pulled changes from a remote repository to a local repository. Your local repository is now up-to-date with the remote repository on GitHub*.

How to Work with Git Commands

In this section, we will cover some of the most commonly used Git commands and their functions. These commands will help you navigate your way through the Git workflow in your GitHub repository. Let's get started.

First, I will add some files so that we can start using the Git commands.

How to Make Changes to a File

To make changes to a file in Git, you need to follow these steps:

1. Open your terminal and navigate to the directory of your local repository.

2. Use a text editor to make changes to the file. For example, you can use the code command to open the file in Visual Studio Code.

$ code file-name  # For example, code index.html

3. Once you have made your changes, save the file and close the text editor.

Congratulations! You have successfully made changes to a file in your local repository. Next, let's proceed to the next step: staging changes.

The image above shows the new file I added which is a React and TypeScript app.

Visual Studio Code (VS Code) includes a source control feature that allows you to interact directly with your GitHub repository. This feature supports a variety of operations, including staging, committing, pushing, and pulling changes.

In addition to the source control feature, you can also use the integrated terminal in VS Code to interact with your GitHub repository.

Currently, if you look at the source control section in VS Code, you'll see the file we added listed under changes.

Next, let's explore how to use the terminal to interact with our GitHub repository.

Open your terminal and navigate to the directory of your local repository.

Now, let's use the git status command to check the status of the current branch.

How to Check the Status of the Current Branch

The git status command shows the status of the current branch, including any changes that have been made to the files in the repository. It provides information about which files have been modified, which files have been staged, and which files are untracked.

This command is useful for understanding the current state of your repository and determining which files need to be staged and committed.

#  Check the status of the current branch

$ git status  # On branch master

The image above shows the command to check the status of the current branch.

You may notice that parts of the file are highlighted in different colors. The red color indicates that the file has been modified, while the green color signifies that the file has been added to the staging area.

Currently, all files should be highlighted in red because we have not yet added any files to the staging area.

Let's add the file to the staging area using the git add command.

How to Stage Changes

The git add command adds files to the staging area, preparing them for the next commit. You can add all the files in the current directory to the staging area using the git add . command.

If you want to add a specific file, use the git add <file-name> command, replacing <file-name> with the name of your file. This process is known as staging, which prepares files for the next commit.

# Add files to the staging area

$ git add .  # Changes to be committed:

or 

$ git add file-name  # Changes to be committed:

Think of it like this: getting into the car is like adding files to the staging area, and driving the car is like making a commit.

Now, let's use the git commit command to commit the changes to the current branch.

How to Commit Changes

The git commit command commits the changes to the current branch. You can use the -m flag to add a message to your commit. This message should provide a brief summary of the changes you've made.

For instance, "Initial commit" could be your commit message. This command is used to save the changes to the local repository.

# Commit changes to the current branch

$ git commit -m "Commit message"   # For example, git commit -m "Initial commit"

We've successfully committed the changes to the current branch. Next, we'll push these changes to the remote repository on GitHub using the git push command.

How to Push Changes to a Remote Repository

The git push command pushes changes from your local repository to a remote repository on GitHub. You can use the git push command to push changes from your local repository to the remote repository on GitHub. This process is essential for updating the remote repository with the changes you've made locally.

# Push changes to a remote repository

$ git push origin main  # For example, git push origin master

Congratulations! You have successfully pushed your changes to the remote repository on GitHub. You can now view your changes on the GitHub website.

Now that we've successfully pushed our changes to the remote repository on GitHub, let's proceed to the next step: creating a branch.

Depending on your PC environment, your local repository may have a default branch named either main or master. In this guide, we'll use main as the default branch name, aligning with GitHub's recent change from master to main.

Before we start adding files, let's ensure our local repository is up-to-date with the remote repository by pulling any changes.

If the term branch seems unfamiliar, don't worry. In the next section, we'll cover how to create a branch and how to pull changes from the remote repository to the local repository.

How to Create a Branch

Branching is a fundamental concept in Git. It allows you to diverge from the main line of development and continue working without impacting the main codebase.

In this section, I'll guide you through the process of creating a new branch using the git branch command. This command creates a new branch but does not switch to it. In the following steps, we'll also cover how to switch to your newly created branch using the git checkout command. Let's dive in.

To create a new branch, you need to follow these steps:

1. Open your terminal and navigate to the directory of your local repository.

2. Use the git branch command to create a new branch. Replace <branch-name> with the name of your new branch.

# Create a new branch

$ git branch <branch-name>  # For example, git branch feature-branch

The git branch command creates a new branch but does not switch to it. To switch to your newly created branch, use the git checkout command.

# Switch to the newly created branch

$ git checkout <branch-name>  # For example, git checkout feature-branch

The git checkout command is used to switch from one branch to another. Replace <branch-name> with the name of your new branch. In this case, we're switching to the feature-branch branch. But we if want to delete the branch, we can use the following command:

# Delete a branch

$ git branch -d <branch-name>  # For example, git branch -d feature-branch

The git branch -d command is used to delete a branch. Replace <branch-name> with the name of the branch you want to delete. In this case, we're deleting the feature-branch branch.

Congratulations! You have successfully created a new branch and switched to it. You can now start adding files and making changes to your new branch.

Now you know how to create GitHub repository, connect a local repository to a remote repository on GitHub, pull changes from a remote repository to a local repository, work with Git commands, and create a branch.

Let's proceed to the next section, where we'll cover how to create a pull request. This is a crucial step in the collaborative workflow, as it allows you to propose changes and request a review from other collaborators.

How to Create a Pull Request

A pull request is a proposal to merge changes from one branch into another. It's a widely-used method for creating and reviewing code. In this section, I'll guide you through the process of creating a pull request using the GitHub website.

For instance, let's say you have a branch named feature-branch and you want to merge it into the main branch. We'll walk you through how to create a pull request for this scenario. Let's get started.

First, let's make a change to our feature branch by adding a file to it:

$ git checkout feature-branch

You should see something like this in your terminal:

git checkout feature-branch
Switched to a new branch 'feature-branch'
branch 'feature-branch' set up to track 'origin/feature-branch'.

Now, let's add a file to the feature branch.

$ touch feature-branch-file.txt

After running the command, you should see a new file called feature-branch-file.txt in your directory.

The touch command is used to create a new file. Replace feature-branch-file.txt with the name of your file. In this case, we're creating a new file called feature-branch-file.txt.

Now, let's add some content to the file.

$ echo "This is a file in the feature branch" >> feature-branch-file.txt

This command adds the text "This is a file in the feature branch" to the feature-branch-file.txt file.

The echo command is used to add content to a file. In this case, we're adding the text "This is a file in the feature branch" to the feature-branch-file.txt file.

Now that we have some text in the file, let's stage and commit the changes to the feature branch.

$ git add .

The git add . command stages all the changes in the current directory.

$ git commit -m "Add file to feature branch"

The git commit -m command commits the changes to the current branch. Replace Add file to feature branch with your own descriptive message. This message should provide a brief summary of the changes you've made. In this case, we're committing the changes to the feature branch.

Now, let's push the changes to the remote repository on GitHub.

$ git push origin feature-branch

The git push command is used to push changes from your local repository to the remote repository on GitHub. Replace feature-branch with the name of your branch. In this case, we're pushing the changes to the feature-branch branch.

Congratulations! You have successfully pushed your changes to the remote repository on GitHub. You can now view your changes on the GitHub website.

Now when you open your GitHub repository, you should see a message indicating that you recently pushed a new branch. You can click on the Compare & pull request button to create a pull request for the feature-branch branch.

The image above shows the Compare & pull request button on GitHub.

After clicking on the Compare & pull request button, you will be directed to a new page where you can fill in the details of your pull request.

You will need to enter the following information:

• Title: a brief summary of your pull request.
• Description: a detailed description of your pull request, including information about the changes you've made and why you've made them.
• Reviewers: you can choose to request a review from specific collaborators.
• Assignees: you can choose to assign your pull request to specific collaborators.
• Labels: you can choose to add labels to your pull request to categorize it.
• Projects: you can choose to add your pull request to a project board.
• Milestone: you can choose to add your pull request to a milestone.

The image above shows the form to fill in the details of your pull request.

You can decide to file the details of your pull request or create the pull request. After creating the pull request, you can view it on the GitHub website. You can also request a review from specific collaborators and make changes to your pull request if necessary.

Once your pull request has been reviewed and approved, you can merge it into the main branch. In our case we not going to file the form but we are going to create the pull request.

The image above shows the pull request created on GitHub.

Now that we have created a pull request, let's proceed to the next section, where we'll cover how to merge a pull request. This is the final step in the collaborative workflow, as it allows you to incorporate changes into the main codebase.

How to Merge a Pull Requset

Merging a pull request signifies the integration of changes from one branch into another, often the main branch. This step is pivotal in collaborative workflows, enabling the assimilation of modifications into the primary codebase.

In this section, we'll navigate the process of merging a pull request via the GitHub website.

After creating a pull request, you can merge it into the main branch by following these steps:

1. On GitHub, navigate to the main page of the repository where you created the pull request.

2. Click on the Pull requests tab to view the list of pull requests.

The image above shows the Pull requests tab on GitHub.

3. Click on the pull request you want to merge.

4. Click on the Merge pull request button to merge the pull request into the main branch.

5. Click on the Confirm merge button to confirm the merge.