That's exactly what LLM agents bring to the table. They're changing how we automate complex tasks, and they can help bring our AI ideas to life in a whole new way.

In this article, we'll explore what LLM agents are, how they work, and how you can build your very own using awesome open-source frameworks.

What we’ll cover:

1. The Current State of LLM Agents

   • From Chatbots to Autonomous Agents

   • What Can Agents Do Today?

   • What's Available to Build With?

   • Why Now Is the Best Time to Learn

2. What Are LLM Agents and Why Are They a Big Deal?

   • What Is an LLM?

   • So, What’s an LLM Agent?

   • Why Does This Matter?

3. The Rise of Open-Source Agent Frameworks

   • Popular Open-Source Agent Frameworks

   • What These Tools Enable

   • Why Use a Framework Instead of Building from Scratch?

4. Core Concepts Behind Agent Design

   • The Agent Loop

   • Key Components of an Agent

   • Multi-Agent Collaboration

5. Project: Automate Your Daily Schedule from Emails

   • What We’re Automating

   • Step 1: Install the Required Tools

   • Step 2: Define the Task

   • Step 3: Build the Workflow with LangGraph

6. Multi-Agent Collaboration with CrewAI

   • What Is CrewAI?

   • Sample Roles for the Email Summary Task

   • Sample CrewAI Code

7. What Actually Happens During Execution?

8. Are LLM Agents Safe? What to Know About Security and Privacy

9. Troubleshooting & Tips

10. Explore More Daily Automations

11. What’s Next in Agent Technology?

12. Final Summary

The Current State of LLM Agents

LLM agents are one of the most exciting developments in AI right now. They’re already helping automate real tasks but they’re also still evolving. So where are we today?

From Chatbots to Autonomous Agents

Large Language Models (LLMs) like GPT-4, Claude, Gemini, and LLaMA have evolved from simple chatbots into surprisingly capable reasoning engines. They've gone from answering trivia questions and generating essays to performing complex reasoning, following multi-step instructions, and interacting with tools like web search and code interpreters.

But here’s the catch: these models are reactive. They wait for input and give output. They don't retain memory between tasks, plan ahead, or pursue goals on their own. That’s where LLM agents come in – they bridge this gap by adding structure, memory, and autonomy.

What Can Agents Do Today?

Right now, LLM agents are already being used for:

• Summarizing emails or documents

• Planning daily schedules

• Running DevOps scripts

• Searching APIs or tools for answers

• Collaborating in small “teams” to complete complex tasks

But they’re not perfect yet. Agents can still:

• Get stuck in loops

• Misunderstand goals

• Require detailed prompts and guardrails

That’s because this technology is still early-stage. Frameworks are getting better fast, but reliability and memory are still works in progress. So just keep that in mind as you experiment.

Why Now Is the Best Time to Learn

The truth is: we’re still early. But not too early.

This is the perfect time to start experimenting with agents:

• The tooling is mature enough to build real projects

• The community is growing rapidly

• And you don’t need to be an AI expert just comfortable with Python

What Are LLM Agents and Why Are They a Big Deal?

Before we dive into the exciting world of agents, let's quickly chat a bit more about the basics.

What Is an LLM?

An LLM, or Large Language Model, is basically an AI that's learned from a massive amount of text from the internet – think books, articles, code, and tons more. You can picture it as a super-smart autocomplete engine. But it does way more than just finish your sentences. It can also:

• Answer tricky questions

• Summarize long articles or documents

• Write code, emails, or creative stories

• Translate languages instantly

• Even solve logic puzzles and have engaging conversations

Chances are you've heard of ChatGPT, which is powered by OpenAI's GPT models. Other popular LLMs you might come across include Claude (from Anthropic), LLaMA (by Meta), Mistral, and Gemini (from Google).

These models work by simply predicting the next word in a sentence based on the context. While that sounds straightforward, when trained on billions of words, LLMs become capable of surprisingly intelligent behavior, understanding your instructions, following step-by-step reasoning, and producing coherent responses across almost any topic you can imagine.

So, What’s an LLM Agent?

While LLMs are super powerful, they usually just react – they only respond when you ask them something. An LLM agent, on the other hand, is proactive.

LLM agents can:

• Break down big, complex tasks into smaller, manageable steps

• Make smart decisions and figure out what to do next

• Use "tools" like web search, calculators, or even other apps

• Work towards a goal, even if it takes multiple steps or tries

• Team up with other agents to accomplish shared objectives

In short, LLM agents can think, plan, act, and adapt.

Think of an LLM agent like your super-efficient new assistant: you give it a goal, and it figures out how to achieve it all on its own.

Why Does This Matter?

This shift from just responding to actively pursuing goals opens a ton of exciting possibilities:

• Automating boring IT or DevOps tasks

• Generating detailed reports from raw data

• Helping you with multi-step research projects

• Reading through your daily emails and highlighting key info

• Running your internal tools to take real-world actions

Unlike older, rule-based bots, LLM agents can reason, reflect, and learn from their attempts. This makes them a much better fit for real-world tasks that are messy, require flexibility, and depend on understanding context.

The Rise of Open-Source Agent Frameworks

Not too long ago, if you wanted to build an AI system that could act autonomously, it meant writing a ton of custom code, painstakingly managing memory, and trying to stitch together dozens of components. It was a complex, delicate, and highly specialized job.

But guess what? That's not the case anymore.

In 2024, a wave of fantastic open-source frameworks hit the scene. These tools have made it dramatically easier to build powerful LLM agents without you having to reinvent the wheel every time.

Popular Open-Source Agent Frameworks

What These Tools Enable

These frameworks give you ready-made building blocks to handle the trickier parts of creating agents:

• Planning – Letting agents decide their next move

• Tool Use – Easily connecting agents to things like file systems, web browsers, APIs, or databases

• Memory – Storing and retrieving past information or intermediate results for long-term context

• Multi-Agent Collaboration – Setting up teams of agents that work together on shared goals

Why Use a Framework Instead of Building from Scratch?

While you could build a custom agent from the ground up, using a framework will save you a huge amount of time and effort. Open-source agent libraries come packed with:

• Built-in support for orchestrating LLMs

• Proven patterns for task planning, keeping track of where you are, and getting feedback

• Easy integration with popular models like OpenAI, or even models you run locally

• The flexibility to grow from a single helpful agent to entire teams of agents

Basically, these frameworks let you focus on what your agent should do, rather than getting bogged down in how to build all the internal workings. Plus, choosing open source means you benefit from community contributions, transparency in how they work, and the freedom to tweak them to your exact needs, without getting locked into a single vendor.

Core Concepts Behind Agent Design

To really grasp how LLM agents operate, it helps to think of them as goal-driven systems that constantly cycle through observing, reasoning, and acting. This continuous loop allows them to tackle tasks that go beyond simple questions and answers, moving into true automation, tool usage, and adapting on the fly.

The Agent Loop

Most LLM agents function based on a mental model called the Agent Loop a step-by-step cycle that repeats until the job is done. Here’s how it typically works:

• Perceive: The agent starts by noticing something in its environment or receiving new information. This could be your prompt, a piece of data, or the current state of a system.

• Plan: Based on what it perceives and its overall goal, the agent decides what to do next. It might break the task into smaller sub-goals or figure out the best tool for the job.

• Act: The agent then acts. This could mean running a function, calling an API, searching the web, interacting with a database, or even asking another agent for help.

• Reflect: After acting, the agent looks at the outcome: Did it work? Was the result useful? Should it try a different approach? Based on this, it updates its plan and keeps going until the task is complete.

This loop is what makes agents so dynamic. It allows them to handle ever-changing tasks, learn from partial results, and correct their course qualities that are vital for building truly useful AI assistants.

Key Components of an Agent

To do their job effectively, agents are built around several crucial parts:

• Tools are how an agent interacts with the real (or digital) world. These can be anything from search engines, code execution environments, file readers, or API clients, to simple calculators or command-line scripts.

• Memory lets agents remember what they've done or seen across different steps. This might include previous things you've said, temporary results, or key decisions. Some frameworks offer short-term memory (just for one session), while others support long-term memory that can span multiple sessions or goals.

• Environment refers to the external data or system context the agent operates within think APIs, documents, databases, files, or sensor inputs. The more information and access an agent have to its environment, the more meaningful actions it can take.

• Goal is the agent's ultimate objective: what it's trying to achieve. Goals should be specific and clear for instance, “generate a daily schedule,” “summarize this document,” or “extract tasks from emails.”

Multi-Agent Collaboration

For more advanced systems, you can even have multiple agents working together to hit a shared target. Each agent can be given a specific role that highlights its specialty just like people working on a team.

For example:

• A researcher agent might be tasked with gathering information.

• A coder agent could write Python scripts or automation routines.

• A reviewer agent might check the results and ensure everything is up to snuff.

These agents can chat with each other, share information, and even debate or vote on decisions. This kind of teamwork allows AI systems to tackle bigger, more complex tasks while keeping things organized and modular.

Project: Automate Your Daily Schedule from Emails

What We’re Automating

Think about your typical morning routine:

• You open your inbox.

• You quickly scan through a bunch of emails.

• You try to spot meetings, tasks, and important reminders.

• Then, you manually write a to-do list or add things to your calendar.

Let's use an LLM agent to make that process effortless. Our agent will:

• Read a list of your email messages

• Pull out time-sensitive items like meetings or deadlines

• Summarize everything into a nice, clean daily schedule

Step 1: Install the Required Tools

To get started, you'll need three main tools: Python, VSCode, and an OpenAI API key.

1. Install Python 3.9 or Higher

Grab the latest version of Python 3.9+ from the official website: https://www.python.org/downloads/

Once it's installed, double-check it by running python --version in your terminal.

This command simply asks your system to report the Python version currently installed. You'll want to see Python 3.9.x or something higher to ensure compatibility with our project.

2. Install VSCode (Optional but Recommended)

VSCode is a fantastic, user-friendly code editor that works perfectly with Python. You can download it right here: https://code.visualstudio.com/.

3. Get Your OpenAI API Key

Head over to: https://platform.openai.com

Sign in or create a new account. Navigate to your API Keys page. Click “Create new secret key” and make sure to copy that key somewhere safe for later.

4. Install Python Libraries

Open your terminal or command prompt and install these essential packages:

pip install langgraph langchain openai

This command uses pip, Python's package manager, to download and install three crucial libraries for our agent:

• langgraph: The core framework we'll use to build our agent's workflow.

• langchain: A foundational library for working with large language models, upon which LangGraph is built.

• openai: The official Python library for connecting to OpenAI's powerful AI models.

If you're excited to try out multi-agent setups (which we'll cover in Step 5), also install CrewAI:

pip install crewai

This command installs CrewAI, a specialized framework that makes it easy to orchestrate multiple AI agents working together as a team.

5. Set Your OpenAI API Key

You need to make sure your Python code can find and use your OpenAI API key. This is typically done by setting it as an environment variable.

On macOS/Linux, run this in your terminal (replace "your-api-key" with your actual key):

export OPENAI_API_KEY="your-api-key"

This command sets an environment variable named OPENAI_API_KEY. Environment variables are a secure way for applications (like your Python script) to access sensitive information without hardcoding it directly into the code itself.

On Windows (using Command Prompt), do this:

set OPENAI_API_KEY="your-api-key"

This is the Windows equivalent command to set the OPENAI_API_KEY environment variable.

Now, your Python code will be all set to talk to the OpenAI model!

Step 2: Define the Task

We discussed this briefly in the beginning of this section. But to reiterate, this is what we’ll want our agent to do:

• Scan for meetings, events, and important tasks.

• Jot them down quickly in a notebook or an app.

• Create a rough mental plan for your day.

This routine takes time and mental energy. So having an agent do it for us will be super helpful.

Step 3: Build the Workflow with LangGraph

What Is LangGraph?

LangGraph is a cool framework that helps you build agents using a "graph-based" workflow, kind of like drawing a flowchart. It's powered by LangChain and gives you a lot more control over exactly how each step in your agent's process unfolds.

Each "node" in this graph represents a decision point or a function that:

• Takes some input (its current "state").

• Does some reasoning or takes an action (often involving the LLM and its tools).

• Returns an updated output (a new "state").

You draw the connections between these nodes, and LangGraph then executes it like a smart, automated state machine.

Why Use LangGraph?

• You get to control the precise order of execution.

• It's fantastic for building workflows that have multiple steps or even branch off into different paths.

• It plays nicely with both cloud-based models (like OpenAI) and models you run locally.

Alright – now let’s write the code.

1. Simulate Email Input

In a real application, your agent would probably connect to Gmail or Outlook to fetch your actual emails. For this example, though, we’ll just hardcode some sample messages to keep things simple:

Python

emails = """
1. Subject: Standup Call at 10 AM
2. Subject: Client Review due by 5 PM
3. Subject: Lunch with Sarah at noon
4. Subject: AWS Budget Warning – 80% usage
5. Subject: Dentist Appointment - 4 PM
"""

This multiline Python string, emails, acts as our stand-in for real email content. We're providing a simple, structured list of email subjects to demonstrate how the agent will process text.

2. Define the Agent Logic

Now, we'll tell OpenAI’s GPT model how to process this email text and turn it into a summary.

from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated, List
import operator

# Define the state for our graph
class AgentState(TypedDict):
    emails: str
    result: str

llm = ChatOpenAI(temperature=0, model="gpt-4o") # Using gpt-4o for better performance

def calendar_summary_agent(state: AgentState) -> AgentState:
    emails = state["emails"]
    prompt = f"Summarize today's schedule based on these emails, listing time-sensitive items first and then other important notes. Be concise and use bullet points:\n{emails}"
    summary = llm.invoke(prompt).content
    return {"result": summary, "emails": emails} # Ensure emails is also returned

Here’s what’s going on:

• Imports: We bring in necessary components:

  • ChatOpenAI to connect to the LLM,

  • StateGraph and END from langgraph.graph to build our agent workflow,

  • TypedDict, Annotated, and List from typing for type checking and structure,

  • operator (though not used in this snippet, it can help with comparisons or logic).

• AgentState: This TypedDict defines the shape of the data our agent will work with. It includes:

  • emails: the raw input messages.

  • result: the final output (the daily summary).

• llm = ChatOpenAI(...): Initializes the language model. We're using GPT-4o with temperature=0 to ensure consistent, predictable output perfect for structured summarization tasks.

• calendar_summary_agent(state: AgentState): This function is the "brain" of our agent. It:

  • Takes in the current state, which includes a list of emails.

  • Extracts the emails from that state.

  • Constructs a prompt that tells the model to generate a concise daily schedule summary using bullet points, prioritizing time-sensitive items.

  • Sends this prompt to the model with llm.invoke(prompt).content, which returns the LLM’s response as plain text.

  • Returns a new AgentState dictionary containing:

    • result: the generated summary,

    • emails: preserved in case we need it downstream.

3. Build and Run the Graph

Now, let's use LangGraph to map out the flow of our single-agent task and then run it.

builder = StateGraph(AgentState)
builder.add_node("calendar", calendar_summary_agent)
builder.set_entry_point("calendar")
builder.set_finish_point("calendar") # END is implicit if not set explicitly

graph = builder.compile()

# Run the graph using your simulated email data
result = graph.invoke({"emails": emails})
print(result["result"])

Here’s what’s going on:

• builder = StateGraph(AgentState): We're initiating a StateGraph object. By passing AgentState, we're telling LangGraph the expected data structure for its internal state.

• builder.add_node("calendar", calendar_summary_agent): This line adds a named "node" to our graph. We're calling it "calendar", and we're linking it to our calendar_summary_agent function, meaning that function will be executed when this node is active.

• builder.set_entry_point("calendar"): This sets "calendar" as the very first step in our workflow. When we start the graph, execution will begin here.

• builder.set_finish_point("calendar"): This tells LangGraph that once the "calendar" node finishes its job, the entire graph process is complete.

• graph = builder.compile(): This command takes our defined graph blueprint and "compiles" it into an executable workflow.

• result = graph.invoke({"emails": emails}): This is where the magic happens! We're telling our graph to start running. We pass it an initial state that contains our emails data. The graph will then process this data through its nodes until it reaches an end point, returning the final state.

• print(result["result"]): Finally, we grab the summarized schedule from the result (the final state of our graph) and print it to the console.

Example Output

Your Schedule:
- 10:00 AM – Standup Call
- 12:00 PM – Lunch with Sarah
- 4:00 PM – Dentist Appointment
- Submit client report by 5:00 PM
- AWS Budget Warning – check usage

Boom! You've just built an AI agent that can read your emails and whip up your daily schedule. Pretty cool, right? This is a simple yet powerful peek into what LLM agents can do with just a few lines of code.

Multi-Agent Collaboration with CrewAI

What Is CrewAI?

CrewAI is an exciting open-source framework that lets you build teams of agents that work together seamlessly just like a real-world project team! Each agent in a CrewAI setup:

• Has a specific, specialized role.

• Can communicate and share information with its teammates.

• Collaborates to achieve a shared goal.

This multi-agent approach is super useful when your task is too big or too complex for just one agent, or when breaking it down into specialized parts makes it clearer and more efficient.

Sample Roles for the Email Summary Task

Let's imagine our email summary task being handled by a small team of agents:

Sample CrewAI Code

from crewai import Agent, Crew, Task, Process
from langchain_openai import ChatOpenAI
import os

# Set your OpenAI API key from environment variables
# os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY" # Make sure this is set, or defined directly

# Initialize the LLM (using gpt-4o for better performance)
llm = ChatOpenAI(temperature=0, model="gpt-4o")

# Define the agents with specific roles and goals
extractor = Agent(
    role="Email Scanner",
    goal="Find all meetings, reminders, and tasks from the given emails, accurately extracting details like time, date, and subject.",
    backstory="You are an expert at scanning emails for key information. You meticulously extract every relevant detail.",
    verbose=True,
    allow_delegation=False,
    llm=llm
)

prioritizer = Agent(
    role="Schedule Optimizer",
    goal="Sort extracted items by urgency and time, preparing them for a daily agenda.",
    backstory="You are a master of time management, always knowing what needs to be done first. You organize tasks logically.",
    verbose=True,
    allow_delegation=False,
    llm=llm
)

formatter = Agent(
    role="Output Generator",
    goal="Generate a clean, polished, and concise daily agenda in bullet-point format, clearly listing all schedule items.",
    backstory="You are a professional secretary, ensuring all outputs are perfectly formatted and easy to read. You prioritize clarity.",
    verbose=True,
    allow_delegation=False,
    llm=llm
)

# Simulate email input
emails = """
1. Subject: Standup Call at 10 AM
2. Subject: Client Review due by 5 PM
3. Subject: Lunch with Sarah at noon
4. Subject: AWS Budget Warning – 80% usage
5. Subject: Dentist Appointment - 4 PM
"""

# Define the tasks for each agent
extract_task = Task(
    description=f"Extract all relevant events, meetings, and tasks from these emails: {emails}. Focus on precise details.",
    agent=extractor,
    expected_output="A list of extracted items with their details (e.g., '- Standup Call at 10 AM', '- Client Review due by 5 PM')."
)

prioritize_task = Task(
    description="Prioritize the extracted items by time and urgency. Meetings first, then deadlines, then other notes.",
    agent=prioritizer,
    context=[extract_task], # The output of extract_task is the input here
    expected_output="A prioritized list of schedule items."
)

format_task = Task(
    description="Format the prioritized schedule into a clean, easy-to-read daily agenda using bullet points. Ensure concise language.",
    agent=formatter,
    context=[prioritize_task], # The output of prioritize_task is the input here
    expected_output="A well-formatted daily agenda with bullet points."
)

# Instantiate the crew
crew = Crew(
    agents=[extractor, prioritizer, formatter],
    tasks=[extract_task, prioritize_task, format_task],
    process=Process.sequential, # Tasks are executed sequentially
    verbose=2 # Outputs more details during execution
)

# Run the crew
result = crew.kickoff()
print("\n########################")
print("## Final Daily Agenda ##")
print("########################\n")
print(result)

Here’s what’s going on:

• Imports: We bring in key classes from CrewAI: Agent, Crew, Task, and Process. We also import ChatOpenAI for our language model and os to handle environment variables.

• llm = ChatOpenAI(...): Just like in the LangGraph example, this sets up our OpenAI language model, making sure its responses are direct (temperature=0) and using the gpt-4o model.

• Agent Definitions (extractor, prioritizer, formatter):

  • Each of these variables creates an Agent instance. An agent is defined by its role (what it does), a specific goal it's trying to achieve, and a backstory (a sort of personality or expertise that helps the LLM understand its purpose better).

  • verbose=True is super helpful for debugging, as it makes the agents print out their "thoughts" as they work.

  • allow_delegation=False means these agents won't pass their assigned tasks to other agents (though this can be set to True for more complex delegation scenarios).

  • llm=llm connects each agent to our OpenAI language model.

• Simulated emails: We reuse the same sample email data for this example.

• Task Definitions (extract_task, prioritize_task, format_task):

  • Each Task defines a specific piece of work that an agent needs to perform.

  • description clearly tells the agent what the task involves.

  • agent assigns this task to one of our defined agents (e.g., extractor for extract_task).

  • context=[...] is a critical part of CrewAI's collaboration. It tells a task to use the output of a previous task as its input. For instance, prioritize_task takes the extract_task's output as its context.

  • expected_output gives the agent an idea of what its result should look like, helping guide the LLM.

• crew = Crew(...):

  • This is where we assemble our team! We create a Crew instance, giving it our list of agents and tasks.

  • process=Process.sequential tells the crew to execute tasks one after another in the order they're defined in the tasks list. CrewAI also supports more advanced processes like hierarchical ones.

  • verbose=2 will show you a very detailed log of the crew's internal workings and communication.

• result = crew.kickoff(): This command officially starts the entire multi-agent workflow. The agents will begin collaborating, passing information, and working through their assigned tasks in sequence.

• fprint(result): Finally, the consolidated output from the entire crew's collaborative effort is printed to your console.

CrewAI cleverly handles all the communication between agents, figures out who needs to work on what and when, and passes the output smoothly from one agent to the next it's like having a mini AI assembly line!

What Actually Happens During Execution?

So, whether you're using LangGraph or CrewAI, what's really going on behind the scenes when an agent runs? Let's break down the execution process:

• The system gets an input state (for example, your emails).

• The first agent or graph node reads this input and uses a Large Language Model (LLM) to make sense of it.

• Based on its understanding, the agent decides on an action like pulling out key events or calling a specific tool.

• If needed, the agent might invoke tools (like a web search or a file reader) to get more context or perform external operations.

• The result of that action is then passed to the next agent in the team (if it's a multi-agent setup) or returned directly to you.

Execution keeps going until:

• The task is fully completed.

• All agents have finished their assigned roles.

• A stopping condition or a designated "END" point in the workflow is reached.

Think of this as a super-smart workflow engine where every single step involves reasoning, making decisions, and remembering previous interactions.

Are LLM Agents Safe? What to Know About Security and Privacy

As cool as LLM agents are, they raise an important question: can you really trust an AI to run parts of your workflow or interact with your data? It depends. If you’re using services like OpenAI or Anthropic, your data is encrypted in transit and (as of now) isn’t used for training.

But some data might still be temporarily logged to prevent abuse. That’s usually fine for testing and personal projects, but if you’re working with sensitive business info, customer data, or anything private, you’ll want to be careful.

Use anonymized inputs, avoid exposing full datasets, and consider running agents locally using open-source models like LLaMA or Mistral if full control matters to you.

You can also set clear boundaries for your agents so they don’t overstep. Think of it like onboarding a new intern: you wouldn’t give them access to everything on day one.

Give agents only the tools and files they need, keep logs of what they do, and always review the results before letting them make real changes.

As this tech grows, more safety features are coming like better sandboxing, memory limits, and role-based access. But for now, it’s smart to treat your agents like powerful helpers that still need some human supervision.

Troubleshooting & Tips

Sometimes, agents can be a bit quirky! Here are some common issues you might run into and how to fix them:

A handy tip: You can also add print() statements or logging messages inside your agent functions to see what's happening at each stage and debug state transitions.

Explore More Daily Automations

Once you've built one agent-based task, you'll find it incredibly easy to adapt the pattern for other automations. Here are some cool ideas to get your creative juices flowing:

Each of these can be built using the very same principles and frameworks we discussed whether that's LangGraph or CrewAI.

What’s Next in Agent Technology?

LLM agents are evolving at lightning speed, and the next wave of innovation is already here:

• Smarter memory systems: Expect agents to have better long-term memory, allowing them to learn over extended periods and remember past conversations and actions.

• Multi-modal agents: Agents won't just handle text anymore! They'll be able to process and understand images, audio, and video, making them much more versatile.

• Advanced planning frameworks: Techniques like ReAct, Toolformer, and AutoGen are constantly improving agents' ability to reason, plan, and reduce those pesky "hallucinations."

• Edge deployment: Imagine agents running entirely offline on your local computer or device using lightweight models like LLaMA 3 or Mistral.

In the very near future, you'll see agents seamlessly integrated into:

• Your DevOps pipelines

• Big enterprise workflows

• Everyday productivity tools

• Mobile apps and smart devices

• Games, simulations, and educational platforms

Final Summary

Alright, let's quickly recap all the cool stuff you've just learned and accomplished:

• You've gotten a solid grasp of what LLM agents are and why they're so powerful.

• You've seen how open-source frameworks like LangGraph and CrewAI make building agents much easier.

• You've built a real LLM agent using LangGraph to automate a common daily task: summarizing your inbox!

• You've explored the world of multi-agent collaboration with CrewAI, understanding how teams of AIs can work together.

• You've learned how to take these principles and scale them to automate countless other tasks.

So, next time you find yourself stuck doing something repetitive, just ask yourself: "Hey, can I build an agent for that?" The answer is probably yes!

Resources Recap

Here are some helpful resources if you want to dive deeper into building LLM agents:

In this tutorial, you’ll learn about what scripting is, its many use cases, and some advantages and disadvantages of using scripts. We’ll also go through a few example scripts so you can see them in action.

What is a Script?

A script is a set of instructions written in any scripting language (like Bash, Python, JavaScript, and others) that helps you automate tasks or control processes. Unlike compiled programs, scripts are typically interpreted, meaning they are executed directly by a runtime environment without prior compilation.

Scripts are powerful tools for automating repetitive tasks, managing workflows, and solving small (and sometimes large) problems efficiently. Whether you’re a beginner or an experienced developer, understanding how to write scripts can enhance your productivity and broaden your technical capabilities.

Why Write Scripts?

I’ve already touched on what you can do with scripts. So let’s look at some of their advantages (and challenges, too) so you understand why they’re so powerful – and when to use them.

Advantages of scripts

1. Automation: Scripts can help you simplify repetitive tasks such as data processing or file management.

2. Efficiency: they can also save you time by automating tasks that you would otherwise have to do manually.

3. Error reduction: Scripts can help reduce human errors through consistent execution of instructions.

4. Flexibility: Scripts can adapt to a wide variety of tasks with minimal modification.

5. Integration: They can also seamlessly integrate with other systems, tools, or workflows.

Challenges with scripts

1. Performance: Scripts can be slower than compiled programs due to interpretation overhead.

2. Scalability: They’re not always suitable for large-scale or highly complex tasks.

3. Debugging: Debugging scripts can sometimes be challenging due to their dynamic nature.

4. Security risks: Poorly written scripts can expose vulnerabilities, especially if they execute system-level commands.

When to Use vs Not Use Scripts

Scripts are ideal for:

1. Tasks are simple, well-defined, or one-off

2. Prototyping or quickly automating a process

3. The scope is small enough to avoid complexity

Scripts are not ideal for:

1. Performance-critical tasks requiring high efficiency. Instead of a script, try using a dedicated ETL (Extract, Transform, Load) tool or a message broker, or similar alternative tools that fit your use-case.

2. Applications with extensive user interfaces. Instead, you can build a small application or a modular system with proper logging, testing, and documentation.

3. Scenarios needing long-term maintenance, where compiled programs might be more stable. Instead, use task schedulers or workflow managers like CRON, Airflow, AWS Lambda/GCP Functions.

How to Write Effective Scripts

This is the process I use for writing helpful scripts. After we go through this, we’ll see some examples of scripts in different languages so you can get some hands-on practice.

1. Define the problem: Before writing a script, identify the problem it will solve. Be clear about the tasks to automate and the expected outcomes.

2. Choose the right language:

   • Bash: Ideal for system-level tasks like file operations or server management.

   • Python: Great for data processing, web scraping, and more complex automation.

   • JavaScript: Suited for web development and browser-based automation.

3. Write the script: Use a text editor or an Integrated Development Environment (IDE), and make sure you follow best practices like using comments, meaningful variable names, and modular code. We’ll cover these below.

4. Test the script: Test the script in a controlled environment to ensure it performs as expected without causing errors.

5. Execute and deploy: Run the script in its intended environment. If necessary, schedule its execution using tools like Cron (for Bash) or task schedulers.

Example Scripts

Now that you know the basics, let's have some practice. Assume you have around 100 files with the names “book-part-1.pdf”, “book-part-2.pdf”, …, “book-part-100.pdf”. You want to replace all the hyphens (-) in the file names with underscores (_), because the website where you're trying to upload these documents doesn’t allow you to upload files with names containing hyphens.

Here are scripts written in three different languages that all perform the same operation. The process looks like this:

1. find all the files in a directory,

2. check if they contain hyphens (-) in their name, and

3. replace any hyphens with underscores (_).

Here are the file names to start (containing hyphens):

Bash Script

We’ll start with a bash script. Here it is:

#!/bin/bash
# Replace "-" with "_" in file names
DIRECTORY="/path/to/your/folder"
for FILE in "$DIRECTORY"/*; do
    if [[ "$FILE" == *-* ]]; then
        NEW_NAME=$(echo "$FILE" | sed 's/-/_/g')
        mv "$FILE" "$NEW_NAME"
        echo "Renamed: $FILE -> $NEW_NAME"
    fi
done

We define the directory (folder) at the top where our files reside. For each file in the directory, we check if the name contains a -. In such case, we create a new file name and store it in the variable NEW_NAME by copying the old file name using the echo command and replace the - with _ using the sed command. Finally we use the move command mv with the old and new file names as arguments.

Python Script

Next, let’s see what it would look like in Python:

import os
# Replace "-" with "_" in file names
directory = "/path/to/your/folder"
for filename in os.listdir(directory):
    if "-" in filename:
        old_path = os.path.join(directory, filename)
        new_filename = filename.replace("-", "_")
        new_path = os.path.join(directory, new_filename)
        os.rename(old_path, new_path)
        print(f"Renamed: {filename} -> {new_filename}")

The steps are pretty similar in Python. First, we define the directory and then iterate through each file in the directory. To find all files in the directory, we have to use the listdir method from the os package.

Then we check if the file name contains a - in the next line. In such a case, we find the current path (old_path) of the file by merging the directory and its file name. We can create the new file name by replacing the - with _ using the replace method.

We then generate the new file path (new_path) in the similar way we generate the old_path. Finally, we call the rename method in os package with old and new file paths as arguments.

JavaScript Script

And now let’s see how it would look in JavaScript:

const fs = require('fs');
const path = require('path');
const directory = '/path/to/your/folder';

fs.readdir(directory, (err, files) => {
    if (err) {
        console.error('Error reading directory:', err);
        return;
    }
    files.forEach(file => {
        if (file.includes('-')) {
            const oldPath = path.join(directory, file);
            const newFilename = file.replace(/-/g, '_');
            const newPath = path.join(directory, newFilename);
            fs.rename(oldPath, newPath, err => {
                if (err) {
                    console.error(`Error renaming ${file}:`, err);
                } else {
                    console.log(`Renamed: ${file} -> ${newFilename}`);
                }
            });
        }
    });
});

The JavaScript implementation is somewhat similar to the Python implementation – but you’ll need to write more code. Generally, devs don’t prefer JavaScript for these kind of scripts. Most of them rely on Bash/Python. JavaScript is better-suited for browser-based automation scripts.

Still, let’s see what we have here. In this JavaScript code, you have to use two different packages, fs and path. We define the directory at the top, read the files in the directory using the readdir method from the fs package, and pass the directory as the argument. Along with directory, we also pass a callback function that will be executed once the files are read.

Inside the callback function, we loop through each file and check if the file name includes a hyphen (-). If it does, we find the old path using the path package with the directory and file names as arguments. We then construct the new file name by replacing all hyphens with underscores using the replace method.

Similar to the old path, we find the new path using the new file name as an argument. Then we use the rename method from the fs package to rename the file by passing both the old and new file names. If there are errors during renaming or reading the files in a directory, we log the error message. Otherwise, we log the success message.

How to run these scripts

Ok, here’s how you can actually use these scripts:

1. Replace /path/to/your/folder with the actual directory containing the files.

2. Run the script in the corresponding environment:

   • Bash: Save as a .sh file, then execute with bash script.sh

   • Python: Save as a .py file, then execute with python script.py

   • JavaScript: Save as a .js file, then execute with node script.js

The screenshot below shows running the bash script to change the names of the files.

Recurring Scripts

Recurring scripts are designed to execute at regular intervals, like checking a system’s status weekly, cleaning up logs, or fetching data updates. These scripts typically use some form of task scheduler.

Common Approaches

1. CRON jobs: Most operating systems support CRON, which can trigger scripts based on a defined schedule.

2. Task queues: Tools like Celery (Python), Bull (Node.js), or Sidekiq (Ruby) can handle scheduled jobs with more flexibility.

3. Cloud schedulers: Services like AWS Lambda with EventBridge, Google Cloud Scheduler, or Azure Logic Apps allow you to set up recurring scripts in a serverless architecture.

One good example use case for recurring scripts would be sending a daily/weekly report of your system’s usage/performance. You could write a script that finds the number of users who have joined and subscribed to your product and send that report as an email every day/week.

Best Practices for Writing Scripts

Here are some things to keep in mind when you’re writing scripts:

1. Use comments: Explain complex parts of the script with comments.

In the below example, without the comment, someone might have to spend extra time figuring out why the tax rate is a decimal and not a percentage.

# Calculate the total price with tax
def calculate_price_with_tax(price, tax_rate):
    # Tax rate is expressed as a decimal (e.g., 0.07 for 7%)
    return price + (price * tax_rate)

2. Error handling: Account for possible errors and handle them gracefully.

In the below example, if the file is missing, the script won’t crash – instead, it will show a helpful error message.

try:
    with open('data.csv', 'r') as file:
        data = file.readlines()
except FileNotFoundError:
    print("Error: 'data.csv' file not found. Make sure the file exists before running the script.")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

3. Modular design: Break down the script into reusable functions or modules.

In the below example, by separating functionality into smaller, reusable functions, you can debug or reuse parts of the script independently.

def fetch_data_from_api(api_url):
    # Fetch data from the given API
    pass

def process_data(data):
    # Process the data into the desired format
    pass

def save_to_file(data, filename):
    # Save processed data to a file
    pass

# Main script
if __name__ == "__main__":
    data = fetch_data_from_api("https://example.com/api")
    processed_data = process_data(data)
    save_to_file(processed_data, "output.json")

4. Input validation: Validate user inputs to prevent unexpected errors or security risks.

Without validation, someone could input invalid or malicious data (for example, SQL injection strings in certain scenarios).

import re

# Validate that the input is a valid email address
def validate_email(email):
    pattern = r'^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$'
    if not re.match(pattern, email):
        raise ValueError("Invalid email address format")
    return email

# Example usage
try:
    user_email = validate_email(input("Enter your email: "))
    print(f"Valid email: {user_email}")
except ValueError as e:
    print(e)

5. Version control: Use Git or other version control tools to track changes.

If a change breaks the script, you can easily revert to a previous commit using git checkout. Plus, you can collaborate seamlessly with team members.

git init
git add script.py
git commit -m "Initial commit"

Conclusion

Writing scripts is a skill that can significantly boost your productivity and problem-solving abilities. By understanding the basics of scripting languages like Bash, Python, and JavaScript, you can automate tasks, streamline workflows, and save valuable time. Start small, build incrementally, and practice writing scripts for different use cases to master this invaluable skill.

I have an exercise for you. To run and verify this example script, you may think you have to manually create 100 files. That consumes a lot of time.

I wrote a script to generate those 100 files. I would also recommend that you try writing a script to generate 100 files with hyphens in their file names. Then try to run the example script to convert the hyphens to underscores.

This may sound difficult at the beginning, but believe me you just need to write 5 lines of bash code to generate 100 files. Not just 100 – you can even generate a million/billion/trillion files with just 5 lines of code.

If you wish to learn more about Scripts, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

Happy scripting!

This can be daunting in situations where multiple developers are actively working on a codebase, making changes to different branches, or managing multiple branch-specific configurations.

Like with every pull request or change pushed to a branch, you’ll need to review every line of code that’s been changed, added, or removed before deciding what gets merged or not. Configuration files in codebases are not exempt from this, making them prone to errors, as a simple change can affect your entire Continuous Integration setup.

When changes get made to the staging or production branch and a build is triggered, you’ll want to ensure that the correct resources attached to a branch are maintained. In some cases, you may need to define different redirect rules for each respective client, custom build commands, or other settings for each branch.

In this article, I’ll walk through how to manage branch-specific configurations including redirects for multiple branches automatically, using a simple bash script. I’ll also show you how to safely merge context-specific rules for your staging and production branches on Netlify.

What we’ll cover:

• Project Structure and Scenario

• What are Redirects/Rewrites?

• How Netlify Processes Redirects

  • Using the _redirects file syntax

  • Using the netlify.toml configuration file syntax

• The Problem: Managing Multiple netlify.toml Files for Different Branches

• How to Write the Script to Automatically Create Our Configuration File(s)

  • Sample Netlify.toml file

  • Step 1: Create the script folder and add the script file

  • Step 2: Modify package.json to include the script command

• How to Deploy Our Client to Netlify

  • First deployment of your project to Netlify

  • Subsequent Deployments / How to Set Up Branch Deployments

    • Step 1: Set Up Environment Variables in Netlify for each branch context — production, staging, and so on

    • Step 2: Trigger a new deploy

• Inspect Your Deployments

• Conclusion

Project Structure and Scenario

Consider a situation where you have two separate servers deployed for a project: one to serve requests to a staging environment (deployed to Render), and another to the production environment (deployed to Google Cloud Run).

You also have two separate client deployments on Netlify, each with their respective API_BASE_URLs, that are served by their respective servers as illustrated below:

The image below is a sample-project repository, with api and client folders/directories within it. This is an overview of the structure in each of the branches outlined above. Each directory contains its own package.json file, is treated as an independent component, and can be deployed to two separate services.

In your frontend deployment for each of the clients, all your requests made to endpoints that begin with /api/v1/ are routed to the server. Other routes remain within the frontend to direct you to pages within the client. So you’re required to write the correct rules to guide your client on how to process these requests. These are called redirect rules or rewrites.

What are Redirects/Rewrites?

Redirects, or rewrites, are rules you can create to have certain URLs automatically go to a new location anywhere on the internet (source: WPengine). These are also generally known as URL forwarding and you can use them anywhere – in entire websites, sections of a website, or an entire web application.

In web applications, redirects are often utilized to determine how to process requests. Web hosting platforms such as Netlify and Vercel use them as well, giving developers the option to determine how their web applications process requests.

How Netlify Processes Redirects

Netlify has two possible ways to specify redirect rules. You can do it using the _redirects file syntax or using the netlify.toml configuration file syntax. They achieve the same goal, but the netlify.toml syntax gives you more options and capabilities.

Using the _redirects file syntax

If you opt to use the redirect syntax, you should simply create a _redirects file in the public folder of your client app, and specify the redirect rules within it. That’s as easy as it gets. Below is an example of a redirect rule within the file:

The above rule can be interpreted as:

1. Send all my requests that match /api/v1 to the API URL specified, and return a 200 success status code. The asterisks (*) after /api/v1/ as seen in /api/v1/* instruct it to append any other extension of the original URL to the stated API URL. For example, if you have a /api/v1/users route in your frontend, that request will be redirected to https://your-api-base-url.com/api/v1/users. The :splat seen in the API URL is simply a placeholder.

2. Serve all other default routes through the index.html folder. This is required to ensure that you don’t encounter broken pages when you navigate to other pages and attempt to visit the previous page using the “back” button.

Using the netlify.toml configuration file syntax

The netlify.toml configuration file gives you a lot more flexibility when specifying redirect rules, including but not limited to matching the original request route, the required destination, the preferred status code response, header rules, signatures, country restrictions, roles and more.

This is a sample netlify.toml file sourced from Netlify’s documentation:

Quick Note: using the redirects file for redirecting certain requests to our API is perfectly fine. But it can be considered a security risk adding our API URL in plain text in the redirects file if the API_BASE_URL is supposed to be private. This is because any file in the public folder is what it sounds like – public – and anyone can access it.

If the direct locations you desire to have in your app are public URLs, then feel free to utilize the _redirects file syntax. But if you prefer to have a private URL(s), utilizing a netlify.toml configuration file in combination with the environment variables is generally a better idea.

The Problem: Managing Multiple netlify.toml Files for Different Branches

When you use the netlify.toml file to define your build commands and environment-specific settings, and you make changes that are pushed to your repository and open pull requests, you have to manually ignore or edit each netlify.tomlin each branch. This eventually becomes very stressful and susceptible to errors.

In addition to this, we want to avoid having our API URLs hardcoded in either our _redirects or netlify.tomlfile within our project codebase for security reasons. We will use environment variables as provided within our Netlify UI for production and staging contexts.

To avoid the above problems, we will use a small script in our codebase to dynamically generate the correct netlify.toml files for each branch. This approach eliminates conflicts and removes the need for manual intervention when switching between branches or handling pull requests.

How to Write the Script to Automatically Create Our Configuration File(s)

Sample Netlify.toml file

Below is a screenshot of a sample netlify.toml file we are trying to achieve for each build. You can see that all our requests that match api/v1/ in our codebase will be routed to our API.

You could have your API endpoint requests structured differently, for example /api/your-endpoint – just make sure to adjust the script accordingly. In this sample project, we use api/v1/your-endpoint as our structure.

Step 1: Create the script folder and add the script file

In the client directory, create a scripts/ directory and a configure-netlify.sh script file. You are required to do this for each branch in your repo. The content remains the same.

Open the configure-netlify.sh script file and paste the following content within it:

#!/bin/bash

# Ensure API_BASE_URL is set
if [ -z "$API_BASE_URL" ]; then
  echo "Error: API_BASE_URL environment variable is not set."
  exit 1  # Exit the script to stop the deployment
fi

echo "Using API endpoint: $API_BASE_URL"

# Define the desired Netlify configuration
NETLIFY_CONFIG="
[build]
  command = \"npm install && npm run build\"
  base = \"client\"
  publish = \"dist\"

[[redirects]]
  from = \"/api/v1/*\"
  to = \"$API_BASE_URL/:splat\"
  status = 200
  force = true

[[redirects]]
  from = \"/*\"
  to = \"/index.html\"
  status = 200
"

# Create or update the netlify.toml file
if [ ! -f "netlify.toml" ]; then
  echo "Creating netlify.toml file..."
else
  echo "Updating existing netlify.toml file..."
fi

echo "$NETLIFY_CONFIG" > netlify.toml

# Confirm successful configuration
echo "netlify.toml file has been configured successfully!"

The script does the following:

1. It checks the environment variables to ensure that the API_BASE_URL is set. If this isn’t set, it exits the build and causes it to fail. We did this to ensure that you don’t mistakenly create a successful deployment but with invalid URLs in production.

2. It then creates the content of the netlify.toml file as shown in the sample above. If your API endpoints use a different structure from api/v1/your-endpoint, you can adjust the script to fit your desired structure.

3. It checks if there already exists a netlify.toml file. If it doesn’t exist, it creates one and writes the content into it. If it exists, it updates it with the correct content during the build, using the API_BASE_URL set in the environment variables.

Step 2: Modify package.json to include the script command

To integrate this script with your build process, we will add a script command to the package.json file to call this script before running the actual build.

Add the configure-netlify command as follows: "configure-netlify": "bash scripts/configure-netlify.sh" within the scripts in your package.json file.

Update your build command to run the script before running the actual build: "build": "npm run configure-netlify && vite build".

Don’t forget to save these changes and push them to your remote repository.

How to Deploy Our Client To Netlify

When deploying our client to Netlify, we are given three options:

1. importing an existing project (that is, a project that exists on a git repository service such as GitHub and GitLab),

2. importing from a template, or

3. manually deploying a static site using the Netlify drop (drag and drop) interface.

For the configuration in our repository to work as expected during our build process, you’ll need to use the option that requires importing from an existing project such as GitHub. Using the drag-and-drop interface won’t work. If you must use this, then opt for the _redirects file syntax option to define your redirects.

First deployment of your project to Netlify

When deploying your project for the first time, you are given the option of deploying only one branch initially. You can only add and specify other options, such as other branches, in subsequent deployments.

To deploy your project, take the following steps:

1. Sign in to Netlify > netlify.com

2. Click "Add new site" > "Import an existing project" > "Deploy with GitHub"

3. Click "Configure Netlify on GitHub" > Search for your repository > Select it

4. Enter a unique site name for your project

5. Configure deploy settings. Here you are required to select the preferred branch to deploy. For the initial deployment, we will deploy the main branch which we use as the production branch.

   • Branch: main/master

   • Build command: npm run build

   • Publish directory: dist (Select the directory where your static file lives. In this sample project, it’s exported into the dist directory. Some tools export into build)

6. Enter the environment variables for your project. Don’t forget to enter your API_BASE_URL from your server. This is an essential requirement according to the bash script.

7. Click "Deploy site"

Your project should deploy correctly, and you’ll be able to see the netlify.toml configuration generated by the script by inspecting the deployment details at the bottom of the successful deployment page.

You can download this file to your local machine to see the configuration generated. It should match the sample netlify.toml file above. You can also test that it works using your generated site link.

Subsequent Deployments / How to Set Up Branch Deployments

Step 1: Set Up Environment Variables in Netlify for each branch context  — production, staging, and so on

When your project has been deployed successfully, you can set up deployments for your staging branch. To edit the configurations, you’ll need to:

1. Navigate to the list of your sites

2. Select your successfully deployed site

3. Click on “site configuration” on the left menu

4. Select “environment variables” > click the “Add a variable” button.

You will be given the option of adding variables individually or importing an entire .env file. You can choose either option. In the image below, I’ve selected “Import from a .env file”.

Seeing that our production site, deployed from the main branch (with the production environmental variables), has already been deployed, you’ll need to:

1. Uncheck the production branch (to prevent replacing the initially deployed main branch. Be careful not to mix up your environment variables for different branches)

2. Select “branch deploys”

3. Copy and paste the content of your .env file in the input section

4. Don’t forget to add the API_BASE_URL environment variable for your staging environment

Note that in selecting branch deploys, the environment variables imported here will affect all branch deploys, apart from the production branch. You can further customize your contexts by selecting a custom branch, but that’s an entirely different approach which may require you to further customize your netlify.toml configuration file or the bash script.

If you decide to import each environment variable individually, you are given a similar option as seen below. Ensure that you select the correct context for each branch.

DON’T USE THE SAME VALUES FOR ALL CONTEXTS. As seen in the image below, selecting “different value for each deploy context” will allow you to define the values for each one. In this case, we define the values for branch deploys. Your initially used production variable should already exist.

When all the variables have been imported, you can inspect them to confirm that they have been correctly imported by selecting the dropdown on the right beside each variable and inspecting their values.

Step 2: Trigger a new deploy

When all your environment variables have been imported for the different contexts – production and staging in this case – navigate to “deploys” on the left panel of your screen. Then hit the “Trigger deploy” button, clear the cache, and initiate a new deployment.

Inspect Your Deployments

You can confirm that your script works as expected by selecting any of the deployments and selecting the building dropdown in the “Deploy log”. You will see the command run, as well as your output and API URL for that deployment, as defined by your context.

Conclusion

By following the steps in this guide, and using your script and updated commands in each branch in your repo, when you push changes then Netlify will automatically generate or update the netlify.tomlfile in each branch. This ensures that the correct configurations and environment variables for each environment are used at build time.

Your script remains the same across all the branches. This lets you focus on other code changes while your script handles the correct configuration for you safely and easily.

Push changes to any branch to see this in action.

Feel free to connect with me on Twitter (@francisihej) or LinkedIn!

Two of these powerful features are piping and redirection. These features enable you to redirect the output and input of commands to and from other commands and files.

In this article, you'll learn what piping and redirection in Linux mean, followed by a deep dive into how to use these features yourself.

Prerequisites

To fully understand this guide, you'll need to at least have:

• Basic understanding of the Linux operating system.
• Basic experience working with the Linux command line.
• Access to a Linux command line to try out the commands.

Take a look at this Linux command line tutorial if you're new or need a refresher.

What is Piping in Linux?

Before we dive into the how, what does piping even mean? Piping is the act of directing the output of one Linux command as an input to another command. You can direct the standard output or standard error from one command to another using piping.

A simple example of piping is when you take the output of one command and use it as an input to another command. The pipe (|) metacharacter is used to achieve this.

If you're new to the concept of metacharacters, this is just a fancy name for characters with special meaning in the command line. There are other metacharacters in Linux aside from pipe (|). Common examples include less than (<), greater than (>), and ampersand (&), to name a few.

The Basics of Piping

The basic syntax of using the pipe command is as follows:

command1 | command2 | command3 | ... | commandN

In the syntax above, the terminal will execute the commands from left to right. It will start with command1, and then the output will be input to command2. Outputs of command2 will then be used as inputs of command3 and so on. The good thing about piping is that you can chain as many commands as you'd like.

Piping Examples

Below are several examples of using the pipe command to perform various tasks.

1. Counting the number of files and directories

ls -l | wc -l

In the example above, the first section lists all the files and directories in the current directory using the ls command. The additional -l option tells the ls command to list the contents in a long list format.

The output of the ls -l command is then piped to the second section. The wc -l command counts the number of lines from the output of the ls -l command in the first section and prints the result to the terminal.

2. Sorting a list of files and directories

ls | sort

In the command above, the ls command will output the lists of the files and directories in the current directory. The list is then piped to the sort command, which sorts them alphabetically and prints the result to the terminal.

3. Sort and display unique words from a file

cat words.txt | sort -r | uniq

The third example includes three separate commands connected by two pipes. The first command outputs the contents of the words.txt file, which contains a list of words.

The output is piped to the sort -r command, which sorts the words in reverse alphabetical order. Finally, the sorted words are piped to the uniq command, which removes duplicates and outputs the unique sorted words.

Why and When Should You Apply Command Piping Linux?

Piping takes more than just knowing the syntax. The syntax is pretty simple. To effectively use piping, you should understand its essence.

The purpose of piping is to help you chain commands, using the output of one as the input of another.

Piping is not meant to be a way to chain unrelated commands that you want to be executed sequentially. If you need to do that, write your commands in the terminal and separate them with semicolon (;) using the following syntax:

command1 ; command2 ; ... ; commandN

What Is Redirection in Linux?

Redirection is the act of dictating where the inputs or outputs of your commands go. By default, commands receive data from standard input and then output the results in standard output.

One of the main areas where redirection proves useful is when working with commands and files. You can, for example, redirect the output of a command to a file instead of printing the output in the terminal. Alternatively, you can declare a certain file as an input to a command.

Like piping, Linux provides special characters to perform redirection. Here are important file-redirection characters in Linux and what they do:

• > – directs the output of a command to a given file.
• < – directs the contents of a given file to a command.
• >> – directs the output of a command to a given file. Appends the output if the file exists and has content.
• 2> – directs error messages from a command to a given file.
• 2>> – directs an error message from a command to a given file. Appends the error message if the file exists and has content.
• &> – directs standard output and error to a given file.
• &>> – directs standard output and error to a given file. Appends to the file if it exists and has contents.

Let's look at each file-redirection character in-depth.

1. Output Redirection With >

The > symbol enables you to redirect the output of a command to a certain file. Using the symbol, you can redirect the output to any existing file. If it doesn't exist, the output redirection character will automatically create a new one.

However, you should be careful when writing to an existing file because its contents will be overwritten without a warning.

You can perform output redirection using the following syntax:

command > file

The output of running command will be written to the file instead of the standard output (or, in other words, printed to the terminal). Here's an example of how you can do output redirection:

ls -a > contents.txt

In the command above, the list of items in the current directory (including dotfiles, directories, and files) will be written to the contents.txt file. You won't see any output in the terminal due to the redirection.

2. Output Redirection With >>

>> lets you redirect the output of a command to a file. But, unlike using a single greater than character (>), >> will append the output if you try to write to an existing file (instead of overwriting its contents). If the file doesn't exist, it will create a new one.

The syntax is as follows:

command >> file

Here's an example of using output redirection with >> to perform the same action as before:

ls -a >> contents.txt

3. Input Redirection With <

The < character in the command line lets you redirect the input to a command from a file instead of the keyboard. The syntax of input redirection using < is as follows:

command < file

Here's an example of using input redirection:

wc -w < files.txt

In the example above, we're passing files.txt as the input to the wc -w command, which counts the number of words in the file. But you don't need to use the input redirection character in many scenarios because it's the default behavior. For instance, the command above is similar to the following:

wc -w files.txt

4. Error Redirection With 2> and 2>>

When working on the command line, you may run into errors. For instance, if you want to execute a file without appropriate permissions. Instead of letting the terminal throw an error by printing it, you can use the error redirection character to dictate where the error message should go.

A good place to redirect errors is a file dedicated to storing errors. Here's a simple example of a command trying to access the list of files in a directory that doesn't exist:

ls nonexistent 2> error.txt

In the example above, the terminal will throw an error since there's no file named nonexistent. But instead of printing it to the console, it will be stored in the error.txt file due to the error redirection character. However, the file will have nothing if there's no error.

If you must append an error to an existing file instead of overwriting its contents, use 2>> instead of 2>.

5. Output and Error Redirection With &> and &>>

Instead of choosing whether to redirect standard output or standard errors to a file, &> allows you to redirect both simultaneously. You can think of &> as a shorthand for combining the output redirection (>) and error redirection (2>) characters.

The syntax of using &> is as follows:

command &> output.txt

The error or output of the command is written to the output.txt file. Here's an example:

ls sample &> output.txt

In the command above, if the sample directory is not available or is available but you don't have read permissions, an appropriate error will be written in the output.txt file. But if it exists and you have read permissions, its contents will be output in the output.txt file.

Using &>> allows you to append the output to the file if it exists and has contents.

How to Combine Piping and Redirection to Unleash the Power of the Command Line

Using piping and redirection in combination can let you perform complex operations without heavy lifting. By learning how to combine the two, you can create complex commands to perform various actions with less typing.

Take the following command as an example:

ls | grep ".txt" > text_files.txt

The ls command lists the contents of the current directory. The output is piped to the grep command to filter text files. Finally, the output of the grep command is redirected to the text_files.txt file.

This simple yet powerful example shows that when it comes to piping and redirection, the only limit is your mind.

Conclusion

In this tutorial, you learned the basics of piping and redirection in Linux. We covered the basics together with examples of how you can use the two.

The two features in isolation can be powerful, but you can go further by combining them in your commands, as shown in the last section.

Regular expressions, also known as regex, work by defining patterns that you can use to search for certain characters or words inside strings.

Once you define the pattern you want to use, you can make edits, delete certain characters or words, substitute one thing for another, extract relevant information from a file or any string that contains that particular pattern, and so on.

Why Should You Learn Regex?

Regex let you to do text processing in a way that can save you a lot of time. It can also introduce some fun in the process.

Using regex can make locating information much easier. Once you find your target, you can batch edit/replate/delete or whatever processing you need to do.

Some practical examples of using regex are batch file renaming, parsing logs, validating forms, making mass edits in a codebase, and recursive search.

In this tutorial, we're going to cover regex basics with the help of this site. Later on, I will introduce some regex challenges that you'll solve using Python. I'll also show you how to use tools like sed and grep with regex.

Like many things in life, regular expressions are one of those things that you can only truly understand by doing. I encourage you to play around with regex as you are going through this article.

Table of Contents

• Regex Basics
  • Exact match
  • Character set
    • Match ranges in regex
    • Match any character not in the set
  • Character classes
  • Quantifiers
  • Capture groups
    • How to use logical OR in regex
    • How to reference capture groups
    • How to name capture groups
• How to use regex with command line tools
  • Recursive regex search with grep
  • Substitution with sed
• Advanced Regex: Lookarounds
  • Lookbehinds
  • Lookaheads
• Practical Examples of Regex
  • Logs parsing
  • Bulk File Renaming
  • Email validation
  • Password constraints
• Final words

Regex Basics

A regular expression is nothing but a sequence of characters that match a pattern. Besides using literal characters (like 'abc'), there are some meta characters (*,+,? and so on) which have special purposes. There are also features like character classes which can help you simplify your regular expressions.

Before writing any regex, you'll need to learn about all the basic cases and edge cases for the pattern you are looking for.

For instance, if you want to match 'Hello World', do you want the line to start with 'Hello' or can it start with anything? Do you want exactly one space between 'Hello' and 'World' or there can be more? Can other characters come after 'World' or should the line end there? Do you care about case sensitivity? And so on.

These are the kind of questions you must have the answer to before you sit down to write your regex.

Exact match

The most basic form of regex involves matching a sequence of characters in a similar way as you can do with Ctrl-F in a text editor.

On the top you can see the number of matches, and on the bottom an explanation is provided for what the regex matches character by character.

Character set

Regex character sets allow you to match any one character from a group of characters. The group is surrounded by square brackets [].

For example, t[ah]i matches "tai" and "thi". Here 't' and 'i' are fixed but between them can occur 'a' or 'h'.

Match ranges in regex

Sometimes you may want to match a group of characters which are sequential in nature, such as any uppercase English letter. But writing all 26 letters would be quite tedious.

Regex solves this issue with ranges. The "-" acts as a range operator. Some valid ranges are shown below:

You can also specify partial ranges, such as [b-e] to match any of the letters 'bcde' or [3-6] to match any of the numbers '3456'.

You are not limited to specifying only one range inside a character set. You can use multiple ranges and also combine them with any other additional character(s). Here, [3-6u-w;] will match any of '3456uvw' or semicolon ';'.

Match any character not in the set

If you prefix the set with a '^', the inverse operation will be performed. For example, [^A-Z0-9] will match anything except uppercase letters and digits.

Character classes

While writing regex, you'll need to match certain groups such as digits quite often and multiple times in the same expression as well.

So for example, how would you match a pattern like 'letter-digit-letter-digit'?

With what you've learned up until now, you can come up with [a-zA-Z]-[0-9]-[a-zA-z]-[0-9]. This works, but you can see how the expression can get quite messy as the pattern length gets bigger.

To make the expression simpler, classes have been assigned to well-defined character groups such as digits. The following table shows these classes and their equivalent expression with character sets:

Character classes are quite handy and make your expressions much cleaner. We will use them extensively throughout this tutorial, so you can use this table as a reference point and come back here if you forget any of the classes.

Most of the time, we won't care about all the positions in a pattern. The "." class saves us from writing all possible characters in a set.

For example, t.. matches anything that starts with t and any two characters afterwards. This may remind you of the SQL LIKE operator which would use t%% to accomplish the same thing.

Quantifiers

The word "pattern" and "repetition" go hand in hand. If you want to match a 3 digit number you can use \d\d\d. But what if you need to match 11 digits? You could write '\d' 11 times, but a general rule of thumb while writing regex or just doing any kind of programming is that if you find yourself repeating something more than twice, you are probably unaware of some feature.

In regex, you can use quantifiers for this purpose. To match 11 digits, you can simply write the expression \d{11}.

The table below lists the quantifiers you can use in regex:

In this example, the expression can\s+write matches can followed by 1 or more whitespaces followed by write. But you can see 'canwrite' is not matched as \s+ means at least one whitespace needs to be matched. This is useful when you are searching through text which is not trimmed.

Can you guess what can\s?write will match?

Capture groups

Capture groups are sub-expressions enclosed in parentheses (). You can have any number of capture groups, and even nested capture groups.

The expression (The ){2} matches 'The ' twice. But without a capture group, the expression The {2} would match 'The' followed by 2 spaces, as the quantifier will be applied on the space character and not on 'The ' as a group.

You can match any pattern inside capture groups as you would with any valid regex. Here (is\s+){2} matches if it finds 'is' followed by 1 or more spaces twice.

How to use logical OR in regex

You can use "|" to match multiple patterns. This is (good|bad|sweet) matches 'This is ' followed by any of 'good' or 'bad' or 'sweet'.

Again, you must understand the importance of capture groups here. Think about what the expression This is good|bad|sweet would match?

With a capture group, good|bad|sweet is isolated from This is. But if it's not inside a capture group, the entire regex is only one group. So the expression This is good|bad|sweet will match if the string contains 'This is good' or 'bad' or 'sweet'.

How to reference capture groups

Capture groups can be referenced in the same expression or while performing replacements as you can see on the Replacement tab.

Most tools and languages allow you to reference the nth captured group with '\n'. In this site '$n' is used while referencing on replacement. The syntax for replacement will vary depending on the tools or language you're using. For JavaScript, for example, its '$n', while for Python its '\n'.

In the expression (This) is \1 power, 'This' is captured and then referenced with '\1', effectively matching This is This power.

How to name capture groups

You can name your capture groups with the syntax (?<name>pattern) and backreference them in the same expression with \k<name>.

On replacement, referencing is done by $<name>. This is the syntax for JavaScript and can vary among languages. You can learn about the differences here. Also note that this feature might not be available in some languages.

In the expression (?<lang>[\w+]+) is the best but \k<lang> .*, the pattern [\w+]+ is captured with the name 'lang' and backreferenced with \k<lang>. This pattern will match any word character or '+' character 1 or more times. The .* at the end of the regex matches any character 0 or more times. And finally on replacement, the referencing is done by $<lang>.

How to Use Regex with Command Line Tools

There are good CLI tools available that let you perform regex from your terminal. These tools save you even more time as you can easily test different regex without writing code in some langauge and then compiling or interpreting it.

Some of the well-known tools are grep, sed, and awk. Let's look at a few examples to give you some ideas on how you can leverage these tools.

Recursive regex search with grep

You can execute the power of regex through grep. Grep can search patterns in a file or perform recursive search.

If you are on Windows, you can install grep using winget. Run this command in powershell:

winget install -e --id GnuWin32.Grep

I will show you the solution to a challenge I created for a CTF competition at my university.

The file attached to the challenge is a zip file that contains multiple levels of directories and a lot of files in it. The name of the competition was Coderush with flag format coderush{flag is here}. So you have to search for the pattern coderush{.*} which will match the flag format coderush{any character here}.

Unzip the file with unzip ripG.zip and cd into it with cd ripG.

There are 358 directories and 8731 files. Instead of searching the pattern in the files one by one, you can employ grep like this:

grep --color -R "coderush{.*}"

The "-R" flag enables recursive search.

You can learn more about grep and its command line options here

Substitution with sed

You can use sed to perform insertion, deletion, substitution on text files by specifying a regex. If you are on windows, you can get sed from here. Or if you use WSL, tools like grep and sed will already be available.

This is the most common usage of sed:

sed 's/pattern/replacement/g' filename
echo "${text}" | sed 's/pattern/replacement/g'

Here, the option "g" is specified to replace all occurrences.

Some other useful options are -n to suppress the default behaviour of printing all lines and using p instead of g to print only the lines which are affected by the regex.

Let's take a look at the content of texts.txt.

Hello rand chars World 56 rand chars
Henlo 52 rand chars W0rld rand chars
GREP rand chars Henlo 62 rand chars
Henlo 10 rand chars Henlo rand chars
GREP rand chars Henlo 45 rand chars

Our task is replacing Henlo number with Hello number only in the lines where "GREP" is present. So, we are searching for the pattern Henlo ([0-9]+) which will match 'Henlo ' followed by 1 or more digits and all the digits are captured. Then our replacement string will be Hello \1 – the '\1' is referencing the capture group containing the digits.

One way to accomplish that would be using grep to grep the lines which have "GREP" present then perform the replacement with sed.

grep "GREP" texts.txt | sed -En 's/Henlo ([0-9]+)/Hello \1/p'

The "-E" option enables extended regex without which you would need to escape the parentheses.

Or you could just use sed. Use /pattern/ to restrict substitution on only the lines where pattern is present.

sed -En '/GREP/ s/Henlo ([0-9]+)/Hello \1/p' texts.txt

Advanced Regex: Lookarounds

Lookaheads and Lookbehinds (together known as lookarounds) are features of regex that allow you to check the existence of a pattern without including it in the match.

You can think of them as zero width assertions – they assert the existence of a pattern but do not consume any characters in the match. These are very powerful features, but they're also computationally expensive. So make sure you keep an eye on performance if you are using them often.

Lookbehinds

Let's say you want to match the word 'linux', but you have 2 conditions.

1. The word 'GNU' must occur before 'linux' occurs. If a line contains 'linux' but doesn't have 'GNU' before it, we want to discard that line.
2. We want to match only linux and nothing else.

We already know how to satisfy the 1st condition. GNU.* will match 'GNU' followed by any number of characters. Then finally we match the word linux. This will match all of GNU-any-characters-linux.

But how do we prevent matching GNU.* while still maintaining the 1st condition?

That's where a positive lookbehind comes in. You can mark a capture group as a positive lookbehind by prefixing it with ?<=. In this example, the expression becomes (?<=GNU.*)linux.

Now only linux is matched and nothing else.

Note that the expressions (?<=GNU.*)linux and linux(?<=GNU.*) will behave exactly the same. In the 2nd expression, although linux is before the lookbehind, there is .* after 'GNU' which matches linux. This means it satisfies the lookbehind.

To make it simpler, think about the pattern without the lookbehind. The pattern GNU.* will match 'GNU' and anything after it, in our case matching linux.

Now we can derive a generalized statement that the expression (?<=C)X will match the pattern X – only if pattern C came before X (and C must not be included in the match).

You can also reverse the 1st condition. Match lines that contains the word linux only if GNU never came before it. This is called a negative lookbehind. The prefix in this case is ?<!. The inverse of the previous expression would be (?<!GNU.*)linux.

Lookaheads

Lookaheads are also assertions like lookbehinds, as you saw in the previous example. The only difference is that lookbehinds make an assertion before and lookaheads makes assertion after.

Let's say you have these two conditions:

1. Match Hello only if World comes somewhere after it.
2. Match only Hello and nothing else.

The prefix for a positive lookahead is ?=. The expression Hello(?=.*World) will meet both conditions. This is similar to Hello.*World except that only Hello will be matched whereas Hello.*World will match 'Hello', 'World' and anything in between.

Similar to the example in a positive lookbehind, the expressions Hello(?=.*World) and (?=.*World)Hello are equivalent. Because the .* before 'World' matches Hello, satisfying the 1st condition.

A negative lookahead is just the complement of a negative lookbehind. You can use it by prefixing it with ?!. (?!World)Hello will match Hello only if there is no World anywhere after it.

Here is a summary of the syntax for lookarounds when you want to match the pattern X with assertion C.

Practical Examples of Regex

Logs parsing

In this log file, these are the lines which we care about:

[1/10000] Train loss: 11.30368, Valid loss: 8.95446, Elapsed_time: 7.58941
[500/10000] Train loss: 0.96180, Valid loss: 0.20098, Elapsed_time: 82.48651
[1000/10000] Train loss: 0.04051, Valid loss: 0.11927, Elapsed_time: 156.86243

Our task is to extract the training loss and validation loss for purposes such as plotting loss over the epochs. We need to extract the training loss values like 11.30368, 0.96180, 0.04051 and put them in an array.

All the relevant values are prefixed with 'Train loss:', so we can use this in our regex as it is. To match the float numbers we have to match some digits followed by a "." and then followed by more digits. You can do this with \d+\.\d+. Because we want to keep track of these numbers, they should be inside a capture group.

As "." has special purpose in regex, when you want to match a "." character you have to escape it with a backslash. This is applicable for all characters with a special purpose. But you dont have to escape it inside a character set.

Putting it altogether, the expression for extracting training loss is Train loss: (\d+\.\d+). We can use the same logic to extract validation loss with Valid loss: (\d+\.\d+).

Here is one way to extract this information using Python:

import re

f = open("log_train.txt", "r").read()

train_loss = re.findall(r'Train loss: (\d+\.\d+)', f)
valid_loss = re.findall(r'Valid loss: (\d+\.\d+)', f)

train_loss = [float(i) for i in train_loss]
valid_loss = [float(i) for i in valid_loss]

print("train_loss =", train_loss)
print("")
print("valid_loss =", valid_loss)

When there is one capture group, re.findall searches all the lines and returns the values inside the capture group in a list.

Any regex function only return strings, so the values are converted to floats and printed out. Then you can directly use them in another Python script as a list of floats.

This is the result:

You could also use sed, save the output in train_losses.txt, and read from the file. First we use '/Train/' to target only the lines with 'Train' present then we are applying the same regex as before.

sed -En '/Train/ s/.*Train loss: ([0-9]+\.[0-9]+).*/\1/p' log_train.txt | tee train_losses.txt

".*" is added at the start and end so that sed matches the contents of all the relevant lines. Then the entire line is replaced by the value of the capture group. The tee command is used to redirect the output of sed into train_losses.txt while also printing the contents in the terminal.

Take a moment to think about what would you need to extract the epochs. You have to extract 500 from [500/10000] for all such lines. The array should look like [1, 500, 1000, 1500, ...]. You can follow the same approach as we used for the previous example.

Note that if you want to match "[" or "]", you have to escape it. The answer is given here.

Bulk File Renaming

You have these files with some random values as prefixes. You have to rename all files as 1.mp4, 2.mp4 and so on. This is how the files were generated.

This is a common scenario where you have a list of files which have their sequence number in the name but there are also some other characters that you don't want.

The pattern has to match anything up to Episode then an underscore and then the number and .mp4 at the end.

The relevant value is the number before '.mp4' which we will put inside a capture group. .*Episode_ will match everything up to the number. Then we can capture the number with ([0-9]+) and also match .mp4 with \.mp4.

So the final regex is .*Episode_([0-9]+)\.mp4. As we want to keep the .mp4 the replacement string will be \1.mp4.

This is one way to solve it using sed.

for i in *.mp4; do
    newname=$(echo $i | sed -En 's/.*Episode_([0-9]+)\.mp4/\1.mp4/p')
    mv $i $newname
done;ls

First the new name is saved in a variable and then the mv command is used to rename the file.

Could we have just used .* in place of .*Episode_ ? In this example, yes. But there might be filenames such as Steins_Gate0.mp4 where the 0 is part of the movie name and you didn't really want to rename this file so its always better to be as specific as possible.

What if some files were named as "Random_Episode6.mp4"? The difference being, there is no underscore after Episode. What change will you need to make?

The answer is that you'll need to add a "?" after the "_" to make it optional. The regex will be .*Episode_?([0-9]+)\.mp4.

Email validation

There are all sorts of complicated regex for validating email.

Here is a simple one: ^[^@ ]+@[^@.]+\.\w+$. It matches the format A@B.C

The table below breaks down this pattern into smaller pieces:

In the regexr site, you can enable the multline flag from the Flags tab in the upper right corner. The 'gm' at the end indicates that the multiline flag is enabled.

We can see that line 2,3,5,6 didn't match. Can you find out the reason and which part of the regex is responsible for disqualifying it?

The answer is given here

Password constraints

You can also use regex to impose constraints. Here we will uncover the power of positive lookaheads.

Lets say we want to accept a string only if there is a digit in it. You already know how to find a digit with the '\d' class. To accomplish that, we can use [^\d]*\d. This will match any non-digit character 0 or more times and then match a digit.

We can also use the expression .*\d to match one digit. So if there is no digit in the string then the lookahead will fail and the none of the characters of that string will be matched, returning an empty string "".

When we are using a programming language, we can check if the regex returned an empty string and determine that the constraints are not satisfied.

We will create a regex which imposes the following criteria:

1. Minimum 8 characters and maximum 16 characters.
2. At least one lower case letter.
3. At least one upper case letter.
4. At least one number.

To achieve this, you can use positive lookaheads. This is the regex:

^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{8,16}$

The table below explains which part of the regex imposes which constraint:

What modification you would need for imposing at least 5 upper case letters?

You may think (?=.*[A-Z]{5,}) will do the job. But this expression requires all the 5 letters to be together. A string like rand-ABCDE-rand will be matched but 0AxBCDxE0 will not be matched even though it has 5 upper case letters (as they are not adjacent).

Yet again, we have capture groups coming to the rescue. We want to match 5 uppercase letters anywhere in the string. We already know that we can match 1 uppercase letter with .*[A-Z]. Now we will put them inside a capture group and attach a quantifier of minimum 5. The expression will be (.*[A-Z]){5,}.

Here is the final answer:

In place of (?=.*[A-Z]) you will need (?=(.*[A-Z]){5,}). The expression becomes ^(?=.*[a-z])(?=(.*[A-Z]){5,})(?=.*\d).{8,16}$.

You could also require that the password not contain certain words to enforce stronger passwords.

For example, we want to reject the password if contains pass or 1234. Negative lookaheads is the tool for this job. The regex would be ^(?!.*(pass|1234)).*$.

In this regex, we put pass and 1234 inside a capture group and used the logical OR operator. This capture group is nested inside another capture group which is prefixed with ?!.*. This makes it a negative lookahead that matches if there are at least 8 characters by .{8,} with the condition that, pass or 1234 can't be present anywhere in the string.

Final Words

I hope you got a good amount of practice while going through this article. It's ok if you forget some syntax. What's important is understanding the core concepts and having a good idea of what's possible with regex. Then, if you forget a pattern, you can just google it or reference a cheatsheet.

The more you practice, the more you will get by without outside help. Eventually you will be able write super complex and effective regex completely offline.

There are already some good regex cheatsheets out there, so I wanted to create something more in-depth here that you can reference for the core concepts and common use cases.

If you're looking for a cheatsheet, the one from QuickRef is helpful. It's a good place to recall the syntax and they also provide some basic overview of regex related functions in various programming languages.

Most regex techniques are the same in all programming languages and tools – but certain tools might offer additional features. So do some research on the tool you are using to pick the best one for you.

My final suggestion would be not to force using regex just because you can. A lot of the times a regular string.find() is enough to get the job done. But if you live in the terminal, you really can do a lot just with regex for sure.

If you like this type of article, you may keep an eye on my blog or twitter.

We just published a bash scripting for beginners course on the freeCodeCamp.org YouTube channel. Herbert from Herbertech created this course.

In this course, you'll learn the basics of bash scripting, including how to use basic commands, write your first bash script, and use variables and positional arguments. You'll also learn about output and input redirection, test operators, if/elif/else statements, case statements, arrays, for loops, functions, and exit codes.

By the end of the course, you'll have a solid understanding of how to use bash scripting to automate tasks and streamline your workflow. You'll be able to write scripts that can perform complex operations, manipulate text files using AWK and SED, and handle errors using exit codes.

This course is perfect for beginners who are new to bash scripting or want to refresh their knowledge. The instructor has a wealth of experience in teaching programming and web development, and his teaching style is clear, concise, and easy to follow.

Whether you're a developer looking to automate tasks, a system administrator managing a server, or a power user looking to make your workflow more efficient, this course is an excellent way to get started with bash scripting.

Watch the course on the freeCodeCamp.org YouTube channel (1-hour watch).

In this article, we'll start with the basics of bash scripting which includes variables, commands, inputs/ outputs, and debugging. We'll also see examples of each along the way.

Let's get started. 🚀

Table of Contents

1. Pre-requisites
2. Introduction
3. Definition of Bash scripting
4. Advantages of Bash scripting
5. Overview of Bash shell and command line interface
6. How to Get Started with Bash Scripting
7. How to Run Bash Commands from the Command Line
8. How to Create and Execute Bash Scripts

9. Bash Scripting Basics

10. Comments in bash scripting

11. Variables and data types in Bash
12. Input and output in Bash scripts
13. Basic Bash commands (echo, read, etc.)

14. Conditional statements (if/else)

15. Looping and Branching in Bash

16. While loop
17. For loop
18. Case statements
19. How to Schedule Scripts using cron
20. How to Debug and Troubleshoot Bash Scripts
21. Conclusion
22. Resources for learning more about Bash scripting

Pre-requisites

To follow along with this tutorial, you should have the following accesses:

• A running version of Linux with access to the command line.

If you do not have Linux installed or you are just starting out, you can easily access the Linux command line through Replit. Replit is a browser-based IDE where you can access the bash shell in a few minutes.

You can also install Linux on top of your Windows system using WSL (Windows Subsystem for Linux). Here is a tutorial for that.

Introduction

Definition of Bash scripting

A bash script is a file containing a sequence of commands that are executed by the bash program line by line. It allows you to perform a series of actions, such as navigating to a specific directory, creating a folder, and launching a process using the command line.

By saving these commands in a script, you can repeat the same sequence of steps multiple times and execute them by running the script.

Advantages of Bash scripting

Bash scripting is a powerful and versatile tool for automating system administration tasks, managing system resources, and performing other routine tasks in Unix/Linux systems. Some advantages of shell scripting are:

• Automation: Shell scripts allow you to automate repetitive tasks and processes, saving time and reducing the risk of errors that can occur with manual execution.
• Portability: Shell scripts can be run on various platforms and operating systems, including Unix, Linux, macOS, and even Windows through the use of emulators or virtual machines.
• Flexibility: Shell scripts are highly customizable and can be easily modified to suit specific requirements. They can also be combined with other programming languages or utilities to create more powerful scripts.
• Accessibility: Shell scripts are easy to write and don't require any special tools or software. They can be edited using any text editor, and most operating systems have a built-in shell interpreter.
• Integration: Shell scripts can be integrated with other tools and applications, such as databases, web servers, and cloud services, allowing for more complex automation and system management tasks.
• Debugging: Shell scripts are easy to debug, and most shells have built-in debugging and error-reporting tools that can help identify and fix issues quickly.

Overview of Bash shell and command line interface

The terms "shell" and "bash" are used interchangeably. But there is a subtle difference between the two.

The term "shell" refers to a program that provides a command-line interface for interacting with an operating system. Bash (Bourne-Again SHell) is one of the most commonly used Unix/Linux shells and is the default shell in many Linux distributions.

A shell or command-line interface looks like this:

The shell accepts commands from the user and displays the output

In the above output, zaira@Zaira is the shell prompt. When a shell is used interactively, it displays a $ when it is waiting for a command from the user.

If the shell is running as root (a user with administrative rights), the prompt is changed to #. The superuser shell prompt looks like this:

[root@host ~]#

Although Bash is a type of shell, there are other shells available as well, such as Korn shell (ksh), C shell (csh), and Z shell (zsh). Each shell has its own syntax and set of features, but they all share the common purpose of providing a command-line interface for interacting with the operating system.

You can determine your shell type using the ps command:

ps

Here is the output for me:

Checking the shell type. I'm using bash shell

In summary, while "shell" is a broad term that refers to any program that provides a command-line interface, "Bash" is a specific type of shell that is widely used in Unix/Linux systems.

Note: In this tutorial, we will be using the "bash" shell.

How to Get Started with Bash Scripting

Running Bash commands from the command line

As mentioned earlier, the shell prompt looks something like this:

[username@host ~]$

You can enter any command after the $ sign and see the output on the terminal.

Generally, commands follow this syntax:

command [OPTIONS] arguments

Let's discuss a few basic bash commands and see their outputs. Make sure to follow along :)

• date: Displays the current date

zaira@Zaira:~/shell-tutorial$ date
Tue Mar 14 13:08:57 PKT 2023

• pwd: Displays the present working directory.

zaira@Zaira:~/shell-tutorial$ pwd
/home/zaira/shell-tutorial

• ls: Lists the contents of the current directory.

zaira@Zaira:~/shell-tutorial$ ls
check_plaindrome.sh  count_odd.sh  env  log  temp

• echo: Prints a string of text, or value of a variable to the terminal.

zaira@Zaira:~/shell-tutorial$ echo "Hello bash"
Hello bash

You can always refer to a commands manual with the man command.

For example, the manual for ls looks something like this:

You can see options for a command in detail using man

How to Create and Execute Bash scripts

Script naming conventions

By naming convention, bash scripts end with .sh. However, bash scripts can run perfectly fine without the sh extension.

Adding the Shebang

Bash scripts start with a shebang. Shebang is a combination of bash # and bang ! followed by the bash shell path. This is the first line of the script. Shebang tells the shell to execute it via bash shell. Shebang is simply an absolute path to the bash interpreter.

Below is an example of the shebang statement.

#!/bin/bash

You can find your bash shell path (which may vary from the above) using the command:

which bash

Creating our first bash script

Our first script prompts the user to enter a path. In return, its contents will be listed.

Create a file named run_all.sh using the vi command. You can use any editor of your choice.

vi run_all.sh

Add the following commands in your file and save it:

#!/bin/bash
echo "Today is " `date`

echo -e "\nenter the path to directory"
read the_path

echo -e "\n you path has the following files and folders: "
ls $the_path

Let's take a deeper look at the script line by line. I am displaying the same script again, but this time with line numbers.

  1 #!/bin/bash
  2 echo "Today is " `date`
  3
  4 echo -e "\nenter the path to directory"
  5 read the_path
  6
  7 echo -e "\n you path has the following files and folders: "
  8 ls $the_path

• Line #1: The shebang (#!/bin/bash) points toward the bash shell path.
• Line #2: The echo command is displaying the current date and time on the terminal. Note that the date is in backticks.
• Line #4: We want the user to enter a valid path.
• Line #5: The read command reads the input and stores it in the variable the_path.
• line #8: The ls command takes the variable with the stored path and displays the current files and folders.

Executing the bash script

To make the script executable, assign execution rights to your user using this command:

chmod u+x run_all.sh

Here,

• chmod modifies the ownership of a file for the current user :u.
• +x adds the execution rights to the current user. This means that the user who is the owner can now run the script.
• run_all.sh is the file we wish to run.

You can run the script using any of the mentioned methods:

• sh run_all.sh
• bash run_all.sh
• ./run_all.sh

Let's see it running in action 🚀

Bash Scripting Basics

Comments in bash scripting

Comments start with a # in bash scripting. This means that any line that begins with a # is a comment and will be ignored by the interpreter.

Comments are very helpful in documenting the code, and it is a good practice to add them to help others understand the code.

These are examples of comments:

# This is an example comment
# Both of these lines will be ignored by the interpreter

Variables and data types in Bash

Variables let you store data. You can use variables to read, access, and manipulate data throughout your script.

There are no data types in Bash. In Bash, a variable is capable of storing numeric values, individual characters, or strings of characters.

In Bash, you can use and set the variable values in the following ways:

1. Assign the value directly:

country=Pakistan

2. Assign the value based on the output obtained from a program or command, using command substitution. Note that $ is required to access an existing variable's value.

same_country=$country

To access the variable value, append $ to the variable name.

zaira@Zaira:~$ country=Pakistan
zaira@Zaira:~$ echo $country
Pakistan
zaira@Zaira:~$ new_country=$country
zaira@Zaira:~$ echo $new_country
Pakistan

Variable naming conventions

In Bash scripting, the following are the variable naming conventions:

1. Variable names should start with a letter or an underscore (_).
2. Variable names can contain letters, numbers, and underscores (_).
3. Variable names are case-sensitive.
4. Variable names should not contain spaces or special characters.
5. Use descriptive names that reflect the purpose of the variable.
6. Avoid using reserved keywords, such as if, then, else, fi, and so on as variable names.

Here are some examples of valid variable names in Bash:

name
count
_var
myVar
MY_VAR

And here are some examples of invalid variable names:

2ndvar (variable name starts with a number)
my var (variable name contains a space)
my-var (variable name contains a hyphen)

Following these naming conventions helps make Bash scripts more readable and easier to maintain.

Input and output in Bash scripts

Gathering input

In this section, we'll discuss some methods to provide input to our scripts.

1. Reading the user input and storing it in a variable

We can read the user input using the read command.

#!/bin/bash 

echo "What's your name?" 

read entered_name 

echo -e "\nWelcome to bash tutorial" $entered_name

2. Reading from a file

This code reads each line from a file named input.txt and prints it to the terminal. We'll study while loops later in this article.

while read line
do
  echo $line
done < input.txt

3. Command line arguments

In a bash script or function, $1 denotes the initial argument passed, $2 denotes the second argument passed, and so forth.

This script takes a name as a command-line argument and prints a personalized greeting.

echo "Hello, $1!"

We have supplied Zaira as our argument to the script.

#!/bin/bash
echo "Hello, $1!"

Output:

Displaying output

Here we'll discuss some methods to receive output from the scripts.

1. Printing to the terminal:

echo "Hello, World!"

This prints the text "Hello, World!" to the terminal.

2. Writing to a file:

echo "This is some text." > output.txt

This writes the text "This is some text." to a file named output.txt. Note that the >operator overwrites a file if it already has some content.

3. Appending to a file:

echo "More text." >> output.txt

This appends the text "More text." to the end of the file output.txt.

4. Redirecting output:

ls > files.txt

This lists the files in the current directory and writes the output to a file named files.txt. You can redirect output of any command to a file this way.

Basic Bash commands (echo, read, etc.)

Here is a list of some of the most commonly used bash commands:

1. cd: Change the directory to a different location.
2. ls: List the contents of the current directory.
3. mkdir: Create a new directory.
4. touch: Create a new file.
5. rm: Remove a file or directory.
6. cp: Copy a file or directory.
7. mv: Move or rename a file or directory.
8. echo: Print text to the terminal.
9. cat: Concatenate and print the contents of a file.
10. grep: Search for a pattern in a file.
11. chmod: Change the permissions of a file or directory.
12. sudo: Run a command with administrative privileges.
13. df: Display the amount of disk space available.
14. history: Show a list of previously executed commands.
15. ps: Display information about running processes.

Conditional statements (if/else)

Expressions that produce a boolean result, either true or false, are called conditions. There are several ways to evaluate conditions, including if, if-else, if-elif-else, and nested conditionals.

Syntax:

if [[ condition ]];
then
    statement
elif [[ condition ]]; then
    statement 
else
    do this by default
fi

We can use logical operators such as AND -a and OR -o to make comparisons that have more significance.

if [ $a -gt 60 -a $b -lt 100 ]

Let's see an example of a Bash script that uses if, if-else, and if-elif-else statements to determine if a user-inputted number is positive, negative, or zero:

#!/bin/bash

echo "Please enter a number: "
read num

if [ $num -gt 0 ]; then
  echo "$num is positive"
elif [ $num -lt 0 ]; then
  echo "$num is negative"
else
  echo "$num is zero"
fi

The script first prompts the user to enter a number. Then, it uses an if statement to check if the number is greater than 0. If it is, the script outputs that the number is positive. If the number is not greater than 0, the script moves on to the next statement, which is an if-elif statement. Here, the script checks if the number is less than 0. If it is, the script outputs that the number is negative. Finally, if the number is neither greater than 0 nor less than 0, the script uses an else statement to output that the number is zero.

Seeing it in action 🚀

Looping and Branching in Bash

While loop

While loops check for a condition and loop until the condition remains true. We need to provide a counter statement that increments the counter to control loop execution.

In the example below, (( i += 1 )) is the counter statement that increments the value of i. The loop will run exactly 10 times.

#!/bin/bash
i=1
while [[ $i -le 10 ]] ; do
   echo "$i"
  (( i += 1 ))
done

For loop

The for loop, just like the while loop, allows you to execute statements a specific number of times. Each loop differs in its syntax and usage.

In the example below, the loop will iterate 5 times.

#!/bin/bash

for i in {1..5}
do
    echo $i
done

Case statements

In Bash, case statements are used to compare a given value against a list of patterns and execute a block of code based on the first pattern that matches. The syntax for a case statement in Bash is as follows:

case expression in
    pattern1)
        # code to execute if expression matches pattern1
        ;;
    pattern2)
        # code to execute if expression matches pattern2
        ;;
    pattern3)
        # code to execute if expression matches pattern3
        ;;
    *)
        # code to execute if none of the above patterns match expression
        ;;
esac

Here, "expression" is the value that we want to compare, and "pattern1", "pattern2", "pattern3", and so on are the patterns that we want to compare it against.

The double semicolon ";;" separates each block of code to execute for each pattern. The asterisk "*" represents the default case, which executes if none of the specified patterns match the expression.

Let's see an example.

fruit="apple"

case $fruit in
    "apple")
        echo "This is a red fruit."
        ;;
    "banana")
        echo "This is a yellow fruit."
        ;;
    "orange")
        echo "This is an orange fruit."
        ;;
    *)
        echo "Unknown fruit."
        ;;
esac

In this example, since the value of "fruit" is "apple", the first pattern matches, and the block of code that echoes "This is a red fruit." is executed. If the value of "fruit" were instead "banana", the second pattern would match and the block of code that echoes "This is a yellow fruit." would execute, and so on. If the value of "fruit" does not match any of the specified patterns, the default case is executed, which echoes "Unknown fruit."

How to Schedule Scripts using cron

Cron is a powerful utility for job scheduling that is available in Unix-like operating systems. By configuring cron, you can set up automated jobs to run on a daily, weekly, monthly, or specific time basis. The automation capabilities provided by cron play a crucial role in Linux system administration.

Below is the syntax to schedule crons:

# Cron job example
* * * * * sh /path/to/script.sh

Here, the *s represent minute(s) hour(s) day(s) month(s) weekday(s), respectively.

Below are some examples of scheduling cron jobs.

Using crontab

The crontab utility is used to add and edit the cron jobs.

crontab -l lists the already scheduled scripts for a particular user.

You can add and edit the cron through crontab -e.

You can read more about corn jobs in my other article here.

How to Debug and Troubleshoot Bash Scripts

Debugging and troubleshooting are essential skills for any Bash scripter. While Bash scripts can be incredibly powerful, they can also be prone to errors and unexpected behavior. In this section, we will discuss some tips and techniques for debugging and troubleshooting Bash scripts.

Set the set -x option

One of the most useful techniques for debugging Bash scripts is to set the set -x option at the beginning of the script. This option enables debugging mode, which causes Bash to print each command that it executes to the terminal, preceded by a + sign. This can be incredibly helpful in identifying where errors are occurring in your script.

#!/bin/bash

set -x

# Your script goes here

Check the exit code

When Bash encounters an error, it sets an exit code that indicates the nature of the error. You can check the exit code of the most recent command using the $? variable. A value of 0 indicates success, while any other value indicates an error.

#!/bin/bash

# Your script goes here

if [ $? -ne 0 ]; then
    echo "Error occurred."
fi

Use echo statements

Another useful technique for debugging Bash scripts is to insert echo statements throughout your code. This can help you identify where errors are occurring and what values are being passed to variables.

#!/bin/bash

# Your script goes here

echo "Value of variable x is: $x"

# More code goes here

Use the set -e option

If you want your script to exit immediately when any command in the script fails, you can use the set -e option. This option will cause Bash to exit with an error if any command in the script fails, making it easier to identify and fix errors in your script.

#!/bin/bash

set -e

# Your script goes here

Troubleshooting crons by verifying logs

We can troubleshoot crons using the log files. Logs are maintained for all the scheduled jobs. You can check and verify in logs if a specific job ran as intended or not.

For Ubuntu/Debian, you can find cronlogs at:

/var/log/syslog

The location varies for other distributions.

A cron job log file can look like this:

2022-03-11 00:00:01 Task started
2022-03-11 00:00:02 Running script /path/to/script.sh
2022-03-11 00:00:03 Script completed successfully
2022-03-11 00:05:01 Task started
2022-03-11 00:05:02 Running script /path/to/script.sh
2022-03-11 00:05:03 Error: unable to connect to database
2022-03-11 00:05:03 Script exited with error code 1
2022-03-11 00:10:01 Task started
2022-03-11 00:10:02 Running script /path/to/script.sh
2022-03-11 00:10:03 Script completed successfully

Conclusion

In this article, we started with how to access the terminal and then ran some basic bash commands. We also studied what a bash shell is. We briefly looked at branching the code using loops and conditionals. Finally, we discussed automating the scripts using cron followed by some troubleshooting techniques.

Resources for learning more about Bash scripting

If you want to dig deeper into the world of bash scripting, I would suggest you have a look at this 6-hour course on Linux at freeCodeCamp.

What’s your favorite thing you learned from this tutorial? You can also connect with me on any of these platforms. 📧�

See you in the next tutorial, happy coding 😁

Banner image credits: Image by Freepik

The recent release of GPT 3.5, and specifically the very new ChatGPT tool, is definitely a huge leap forward.

You may have read about people asking the AI to draw up real estate contracts and wills, and even generate programming code. This article approaches the tool from a slightly different perspective.

You can watch a video version of this article here if you'd like to use it to supplement your learning:

Experimenting with ChatGPT and Bash Scripting

I decided to see how ChatGPT handles Linux Bash scripting. But (spoiler alert!) as I worked through the tool's suggestions - both good code and bad code - I began to realize that there was an even more powerful benefit staring me in the face: an opportunity to pair-program with an eminently helpful partner.

The AI ultimately failed to solve my problem, but the way it failed was absolutely fascinating. And watching the AI searching for new answers was absolutely crazy.

First Attempt – ChatGPT Tries to Solve the Problem

So let's see how it all happened. Here, more or less, is the initial prompt I provided to the AI:

Write me a Bash script that prompts for the URL of a YouTube video and then uses the yt-dlp program to download from that URL. The script should then change the file name to "myvideo.mp4"

After a few seconds, the ChatGPT told me that a script was coming. Here's how that code looked:

#!/bin/bash

# Prompt user for YouTube video URL
read -p "Enter YouTube video URL: " url

# Download video from YouTube using yt-dlp
yt-dlp $url

# Get the name of the downloaded video file
yt-dlp $url

# Change the downloaded video filename to "myvideo.mkv"
mv video-filename.mp4 myvideo.mp4

The AI obviously knows what Bash scripts are and how to present code. It even did a great job commenting each line of code. The AI knows how to properly prompt the user for input and then how to import the input string into a variable called url.

It also knows about the yt-dlp program and how it works, appropriately feeding it the URL that was provided. Then it at least attempted to rename the downloaded video file to myvideo.mp4. But everything kind of broke down when it assumed that the download filename would be video-filename.mp4. Whoops.

At any rate, the AI helpfully showed me how to create the script from its code and then run it.

Second Attempt – ChatGPT Corrects Itself

My first follow-up prompt pointed out the obvious mistake and, after a few seconds, ChatGPT admitted it was wrong and offered an improvement.

You're right, the script I provided would not work as expected because it assumes that the downloaded video will be saved to a file with the name video-filename.mp4. However, this is not guaranteed to be the case. To fix this...

It's just mind-blowing how the AI is completely engaged in the process here. It remembers its first code, listens to and understands my complaint, and thinks through a solution. Here's that second attempt:

#!/bin/bash

# Prompt user for YouTube video URL
read -p "Enter YouTube video URL: " url

# Download video from YouTube using yt-dlp
yt-dlp $url

# Get the name of the downloaded video file
filename=$(basename $url)

# Change the downloaded video filename to "myvideo.mkv"
mv $filename.mp4 myvideo.mp4

Ok, so I can see how the solution kind of looked right. basename is a Linux tool that can strip out file system directories and leave nothing but the actual filename.

The problem is that the variable url doesn't contain a local filename, but an internet address. This one definitely isn't going to work.

Third Attempt – ChatGPT Tries Again

So I tried again, explaining how that wasn't the right tool for the job. And ChatGPT gives it another try.

#!/bin/bash

# Prompt user for YouTube video URL
read -p "Enter YouTube video URL: " url

# Extract the filename from the URL
filename=$(basename $url)

# Download the video using the yt-dlp tool and specify the filename
yt-dlp -o $filename $url

# Change the downloaded video filename to "myvideo.mkv"
mv $filename.mp4 myvideo.mp4

This time, it shows that it had familiarized itself with the yt-dlp documentation, suggesting that we use the -o argument to give the downloading video a name, against which basename should now work. With each iteration, the AI gives me this updated code for my cutting and pasting pleasure.

Of course, that wasn't going to work either. With the next attempt, I get the feeling that the AI is a bit desperate.

If At First You Don't Succeed...At Least You'll Learn Something

I'm going to skip past the next attempt or two, as we ran into a side problem: Videos coming from YouTube aren't always MP4s, they're sometimes MKVs or even WEBMs. That uncertainty could mess with the filenames we're working with, so I asked ChatGPT to work that into the code.

#!/bin/bash

# Prompt user for YouTube video URL
read -p "Enter YouTube video URL: " url

# Extract the filename from the URL
filename=$(basename $url)

# Download the video using the yt-dlp tool and specify the filename
yt-dlp -F mp4 -O $filename $url

# Change the downloaded video filename to "myvideo.mkv"
mv $filename.mp4 myvideo.mp4

When I reported how that, too, wouldn't work, the AI started to get confused. First it told me that the -F option was expecting a valid video format, but that "mp4" wasn't valid. Then it gives me valid options, including "mp4".

When nothing the AI suggested worked, it gave up, advising me to make sure that yt-dlp was actually installed on my machine and showing me how to do that.

I figured that was a good time to give up. It was a surprisingly good effort and, even if it didn't solve the problem, it did show me some useful syntax and got me thinking.

In case you're interested, by the way, here's how an actual working script would look.

#!/bin/bash

# Prompt user for YouTube video URL
read -p "Enter YouTube video URL: " url

# Download video from YouTube using yt-dlp
yt-dlp $url

# Get the name of the downloaded video file
original_filename=$(ls | awk '/mp4/ || /webm/ || /mkv/')

# Change the downloaded video filename to "myvideo.mkv"
mv "$original_filename" /home/ubuntu/vids/myvideo.mp4

The first two lines of code are just the way ChatGPT suggested. But I then isolate the filename by listing all the files in the current directory and using awk to filter for only filenames containing either mp4, webm, or mkv. (This assumes that there will never be more than one video file in the directory at a time.)

The filename will then be written to the original_filename variable. I'll then use that variable as part of a move command to rename the file myvideo.mp4 As far as I can tell, that'll work no matter what format was actually sent.

Wrapping Up

Artificial intelligence might not be quite ready for real-world programming tasks today, but I wouldn't want to bet against it getting there tomorrow...or some time next year.

In the meantime though, I would absolutely recommend engaging with tools like ChatGPT whenever you're faced with a challenging programming task.

Two minds are (almost) always better than one.

Besides my my YouTube channel (to which you can subscribe), my website links to all kinds of technology goodness available as articles, books, and courses.

There are other environments such as the C Shell, Korn Shell, Z Shell, and even the Fish Shell. All of these different shell environments have their own pros and cons, and you should consider them before you choose one to use on your own system.

In this article, I'll go over a few popular shells along with their main features to help you pick one.

The Bash Shell

The Bash Shell (or the Bourne Again Shell) is a UNIX shell and command language. It was written by Brain Fox for the GNU Project as a free software replacement for the Bourne Shell (sh).

Bash was first released in 1989, and for most Linux distributions it's the default Shell environment. Other distros, like Kali Linux, use the Z Shell as their default shell.

Bash is one of the first programs that Linus Torvalds (the creator of Linux) ported to Linux.

Image Source

Something you should not get confused about is that Bash is also a programming language. So it's a "Shell", but you can also program behavior in Bash. For example:

#!/bin/bash
echo "Hello World"

Key points about Bash

• Most users use Bash, since it is the default shell environment on most systems
• Bash does not have an inline wildcard expression. A wildcard expression is when you would want to search for patterns in your Shell, similar to Regex. The three main wildcards are *, ?, and [].
• You can't automatically change the directory name
• # is treated as a comment in scripting
• It has shopt settings
• Prompt has backslash escapes
• User configuration settings are in .bashrc

The Z Shell

The Z Shell, or Zsh is also a UNIX shell that is very similar to Bash. You can also script and use the shell as a command interpreter.

Zsh is an extension of the Bourne shell with a lot of improvements. Zsh was released in 1990 by Paul Falstad, and it has some features that Bash, Korn Shell, and C Shell share.

macOS by default uses the Zsh Shell.

Image Source

Key points about Zsh

• Comes with autocompletion when using the terminal. So when you press Tab ↹ in order to autocomplete whatever command you want to run, not only does it autocomplete for you but will bring down a drop-down of all the other possible files and directories:

• Supports inline wildcard expressions
• Much more configurable than Bash
• Supports plugins and themes. Here's a list of plugins available for Zsh.

There are also frameworks built around the Z Shell. One of the most popular ones is Oh My Zsh, which is a community driven, open-source framework for managing Zsh configuration. (I use Oh My Zsh 😄)

Image Source

Zsh and Oh My Zsh are similar but not the same exact things. To reiterate, Oh My Zsh is a way of managing your Zsh configurations, it is not the Shell itself.

The Fish Shell

Fish is a UNIX shell environment with an emphasis on interactivity and usability. Unlike Zsh, Fish aims to give the user interactivity by default instead of trusting the user to implement their own configuration.

It was created by Axel Liljencrantz in 2005. Fish is considered to be an "exotic shell" due to the fact that it does not comply to the POSIX shell standards. [[Source](https://en.wikipedia.org/wiki/Fish_(Unix_shell)]

Image Source

Key points about Fish

• Fish has "search as you type" automatic suggestions based on your command history and the directory you are in. Similar to Bash's history search, Fish Shell's search history is always turned on. That way the user will be able to get interactive feedback when working in their terminal.

Image Source

• Fish also prefers features as commands rather than syntax. This makes features visible in terms of commands with options and help texts
• Since Fish by default comes with a lot of configurations already set, it is believed to be more beginner friendly than other sh options like Zsh.
• Fish's scripting language is different than Zsh and Bash. Zsh uses more aliases whereas Fish avoids using aliases in the scripting language.

If you were to just make scripts using basic commands such as, cd, cp, vim, ssh, and so on, you would not notice any difference in the way Fish and Bash's scripting languages work.

One of the biggest differences is when you try capturing output from a command. In Bash you may be used to this:

todays_date=$(date)
echo "Todays date is $todays_date"

Todays Date is Tue Dec 13 15:29:28 CST 2022

Whereas in Fish, capturing output works differently. The equivalent for Fish in scripting would look like this:

set date (date)
echo "Todays Date $date"

todays date is Tue Dec 13 21:35:03 UTC 2022

Conclusion

Bash, Z Shell, and Fish Shell all have their merits, along with some similarities. You can use each of them effectively in your work environment now that you know a bit more about them.

If you want something more configurable, you could use Zsh (or even install Oh My Zsh). If you want more of an interactive terminal experience without a lot of configuration, you could use Fish Shell. If you want the classic feel, you can just keep Bash.

It all really comes down to your preferences as a developer - so just choose the shell that works best for you.

Hope this helped you! Thank you for reading 🐚🐚🐚

Statements that help to execute different code branches based on certain conditions are known as conditional statements.

if...else is one of the most commonly used conditional statements. Like other programming languages, Bash scripting also supports if...else statements. And we will study that in detail in this blog post.

Syntax of if Statements

You can use if statements in a variety of ways. The generic structure of if statements is as follows:

• Using an if statement only: if...then...fi
• Using and if with an else statement: if...then...else...fi statements
• Using multiple else statements with if: if..elif..else..fi

##

How to Use the if Statement

When you are using a single if statement, the syntax is as follows:

if [ condition ]
then
    statement
fi

Note that the spaces are part of the syntax and should not be removed.

Let's go through an example where we are comparing two numbers to find if the first number is the smaller one.

#! /bin/sh

a=5
b=30

if [ $a -lt $b ]
then
        echo "a is less than b"
fi

If you run the above snippet, the condition if [ $a -lt $b ] evaluates to True , and the statement inside the if statement executes

Output:

a is less than b

How to Use the if .. else Statement

When you are using an if statement and you want to add another condition, the syntax is as follows:

if [ condition ]
then
    statement
else
    do this by default
fi

Let's see an example where we want to find if the first number is greater or smaller than the second one. Here, if [ $a -lt $b ] evaluates to false, which causes the else part of the code to run.

#! /bin/sh

a=99
b=45

if [ $a -lt $b ]
then
        echo "a is less than b"
else
        echo "a is greater than b"
fi

Output:

a is greater than b

How to Use if..elif..else Statements

Let's say you want to add further conditions and comparisons to make the code dynamic. In this case, the syntax would look like this:

if [ condition ]
then
    statement
elif [ condition ] 
then
    statement 
else
    do this by default
fi

To create meaningful comparisons, we can use AND -a and OR -o as well.

In this example, we will determine the type of triangle using these conditions:

• Scalene: A triangle where every side is different in length.
• Isosceles: A triangle where 2 sides are equal.
• Equilateral: A triangle where all sides are equal.

read a
read b
read c

if [ $a == $b -a $b == $c -a $a == $c ]
then
echo EQUILATERAL

elif [ $a == $b -o $b == $c -o $a == $c ]
then
echo ISOSCELES
else
echo SCALENE

fi

In the example above, the script would ask the user to enter the three sides of the triangle. Next, it would compare the sides and decide the triangle type.

3
4
5
SCALENE

Conclusion

You can easily branch your code based on conditions like if..else and make the code more dynamic. In this tutorial, you learned the syntax of if...else along with some examples.

I hope you found this tutorial helpful.

What’s your favorite thing you learned from this tutorial? Let me know on Twitter!

You can read my other posts here.

Compared to the GUI (or Graphical User Interface), the CLI is especially powerful. This is in part because you can rename files in bulk or even schedule the scripts to rename files at a certain point in time.

In this tutorial, you will see how you can rename files in the Linux command line using the built-in mv command.

How to Use the Linux mv Command

You can use the built-in Linux command mv to rename files.

The mv command follows this syntax:

mv [options] source_file destination_file

Here are some of the options that can come in handy with the mv command:

• -v , --verbose: Explains what is being done.
• -i, --interactive: Prompts before renaming the file.

Let's say you want to rename index.html to web_page.html. You use the mv command as follows:

zaira@Zaira:~/rename-files$ mv index.html web_page.html

Let's list the files and see if the file has been renamed:

zaira@Zaira:~/rename-files$ ls
web_page.html

How to Name Files in Bulk Using mv

Let's discuss a script where you can rename files in a bulk using a loop and the mv command.

Here we have a list of files with the extension .js.

zaira@Zaira:~/rename-files$ ls -lrt
total 0
-rw-r--r-- 1 zaira zaira 0 Sep 30 00:24 index.js
-rw-r--r-- 1 zaira zaira 0 Sep 30 00:24 config.js
-rw-r--r-- 1 zaira zaira 0 Sep 30 00:24 blog.js

Next, you want to convert them to .html.

You can use the command below to rename all the files in the folder:

for f in *.js; do mv -- "$f" "${f%.js}.html"; done

Let's break down this long string to see what's happening under the hood:

• The first part [for f in *.js] tells the for loop to process each “.js” file in the directory.
• The next part [do mv -- "$f" "${f%.js}.html] specifies what the processing will do. It is using mv to rename each file. The new file is going to be named with the original file’s name excluding the .js part. A new extension of .html will be appended instead.
• The last part [done] simply ends the loop once all the files have been processed.

zaira@Zaira:~/rename-files$ ls -lrt
total 0
-rw-r--r-- 1 zaira zaira 0 Sep 30 00:24 index.html
-rw-r--r-- 1 zaira zaira 0 Sep 30 00:24 config.html
-rw-r--r-- 1 zaira zaira 0 Sep 30 00:24 blog.html

Conclusion

As you can see, renaming files is quite easy using the CLI. It can be really powerful when deployed in a script.

What’s your favorite thing you learned here? Let me know on Twitter!

You can read my other posts here.

Image by storyset on Freepik

And to execute commands on the command line, you need a shell.

The Bash shell is popular in Unix-like operating systems like Mac and Linux. In fact in most Linux distros, Bash is the default shell.

You can also use Bash in Windows via WSL (Windows Subsystem for Linux).

After learning some basic Bash commands, it's time to get up to speed.

This tutorial is not for absolute beginners, but I hope both novices and advanced users might get something out of it.

Here are the 10 Bash commands that'll help you work faster with your terminal.

1. Use Control + L to clear the screen and Control + D to exit

To clear the terminal screen, type clear on the command line.

To exit, type exit.

Better still, pressing Ctrl + l ( ⌘ + l ) clears the screen and Ctrl + d (⌘ + d) closes the terminal.

2. Use the nohup command to spawn processes that don't end with the terminal session

Once programs are loaded into memory, they are called processes.

Sometimes, I open Firefox from the command line:
firefox https://freecodecamp.org.

But as soon as I close the terminal, Firefox also crashes.

To prevent this, use the nohup (no hang up) command: nohup firefox https://freecodecamp.org.

Now when I close the terminal, Firefox doesn't crash but my tab crashes.

The fix: make Firefox a background process by appending the & symbol.

nohup firefox https://freecodecamp.org &

Now even if I quit the terminal, my tabs are all intact.

3. Use pkill to kill processes by typing in only a part of the name

By using the killall command, we can kill a process by its name:

killall firefox

Better still, use pkill to terminate by typing only a part of the name.

pkill fire*

4. Prepend the time command to know how fast something executes

Do you want to know how long it takes something to execute in the shell?

Just prepend time to the command: time gcc -g *.c.

5. On Linux, use cat /etc/*rel* to view the distro name

Typing uname -a shows the system information.

Wanna double check what distro you're running?

Just type cat /etc/*rel* on the shell and press enter.

6. Use the sed command in text files to find and replace

Do you want to replace multiple occurrences of a word in a text file?

Use the sed command.

sed s'/apples/oranges/g' myfile.txt

Here, all the occurrences of the word 'apples' are changed into 'oranges'.

If you only need to replace the first occurrence in every line, just take out the 'g' suffix at the end: sed s'/apples/oranges/' myfile.txt.

The 'g' is for global.

The forward slash / is the delimiter. In fact, you can use any delimiter.

Let's use a single underscore _ as the delimiter: sed s'_apples_oranges_'g myfile.txt`.

Simply using sed only replaces on the standard output. The original file is unchanged.

To change the file 'in place', use the -i flag: sed -i s'_apples_oranges_g' myfile.txt.

7. Know the public IP address of your computer using curl

There are two types of IP addresses: private and public.

A system assigns internal IP addresses which can be checked using the ifconfig command.

But do you want to know the public IP of your computer – the IP address that the ISP assigns to your interface?

While online, just use curl ifconfig.me ; echo or curl ifconfig.co ; echo on the command line.

8. Use Ctrl + R (⌘ + R) for reverse search

Pressing the 'up' arrow key shows the last command you've typed.

Typing history shows all the commands you have typed that are stored in bash history.

Better still, type Ctrl + r (⌘ + r) on the shell and start typing in the command.

As you type, the shell autocompletes from history. Press 'enter' as soon as you find the match.

If you remember only one thing from this tutorial, remember this key combination: Ctrl + r (⌘ + r).

It will save you a lot of time, guaranteed!

9. Use the shell for doing math

For simple calculations that don't input or output fractions, you can simply use:

:~$ echo $((19*34))
:~$ 646

For calculations that involve fractions, just echo the expression and pipe it into the bc command.

:~$ echo "scale=2; 9*3/((2*2)+1)" | bc
:~$ 5.40

Here, 'scale' is the precision we need. I have specified it to be two decimal points.

10. Use brace expansion to create files in bulk

How do we create 100 files inside a folder?

file1.txt, file2.txt, file3.txt ... file100.txt

By using brace expansion: touch file{1..100}.txt.

We need to create three files for our project: app.html, app.css, and app.js

Instead of creating one by one, we can simply use brace expansion to create all of them in one go.

:~$ touch app.{html,css,js}
:~$ ls
app.html app.css app.js
:~$

Or inside the project folder, we need to create five directories: images, css, src, templates, and scripts.

We can use:

:~$ mkdir {images,css,src,templates,scripts}
:~$ ls
images css src templates scripts
:~$

Only one caveat here: just ensure that there are no spaces between the words inside the braces.

Wrapping up

I have listed 10 Bash command line tips by which you can work up to speed on the terminal.

Learn these Bash commands and it will hold you in good stead in your programming journey.

Happy Coding!

They're common commands for Linux admins and people doing DevOps, but are handy to know even if you don't use the command line often.

In this article, I'll go into what both of these commands do, how to use them, and some frequently asked questions.

What Are the Differences Between sudo apt-get update and sudo apt-get upgrade?

The main difference is that sudo apt-get update fetches the latest version of the package list from your distro's software repository, and any third-party repositories you may have configured. In other words, it'll figure out what the latest version of each package and dependency is, but will not actually download or install any of those updates.

The sudo apt-get upgrade command downloads and installs the updates for each outdated package and dependency on your system. But just running sudo apt-get upgrade will not automatically upgrade the outdated packages – you'll still have a chance to review the changes and confirm that you want to perform the upgrades.

How to Use the sudo apt-get update Command

In your Debian-based Linux distro (Debian, Ubuntu, Linux Mint, Kali Linux, Raspberry Pi OS, and so on), open a terminal window.

Depending on your distro, the terminal might go by different names depending on how you open it. For example, in Ubuntu and Linux Mint, the default terminal is Gnome Terminal, but may be listed under Terminal in the application menu.

In the terminal, enter sudo apt-get update in the command line, enter in your admin password, and press the Enter key.

If there are updates, you'll see an output similar to this:

kris@pihole:~ $ sudo apt-get update
Hit:1 https://ftp.harukasan.org/raspbian/raspbian bullseye InRelease
Get:2 https://download.docker.com/linux/raspbian bullseye InRelease [26.7 kB]
Get:3 http://archive.raspberrypi.org/debian bullseye InRelease [23.7 kB]       
Get:4 http://packages.azlux.fr/debian buster InRelease [3,989 B]               
Get:5 http://archive.raspberrypi.org/debian bullseye/main armhf Packages [282 kB]
Get:6 http://packages.azlux.fr/debian buster/main armhf Packages [3,418 B]
Fetched 340 kB in 4s (94.8 kB/s)     
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.

If you want to see which packages can be upgraded, run apt list --upgradable:

kris@pihole:~ $ apt list --upgradable
Listing... Done
libcamera0/stable 0~git20220426+18e68a9b-1 armhf [upgradable from: 0~git20220303+e68e0f1e-1]
raspi-config/stable 20220425 all [upgradable from: 20220419]
rpi-eeprom/stable 13.13-1 armhf [upgradable from: 13.12-1]

But if there are no newer versions of packages or dependencies in your distro's software repository, you'll see an output like this:

kris@pihole:~ $ sudo apt-get update
Get:1 https://download.docker.com/linux/raspbian bullseye InRelease [26.7 kB]
Hit:2 https://ftp.harukasan.org/raspbian/raspbian bullseye InRelease           
Hit:3 http://packages.azlux.fr/debian buster InRelease                         
Hit:4 http://archive.raspberrypi.org/debian bullseye InRelease
Fetched 26.7 kB in 3s (8,789 B/s)
Reading package lists... Done

Notice that there is no mention of packages that can be upgraded, and no note about running apt list --upgradable.

But this does not necessarily mean there's no outdated software on your system, just that you already got the latest version of the package list. You may have run sudo apt-get update multiple times.

You can always run apt list --upgradable again to see if anything can be upgraded.

Or you can use the more modern sudo apt update command instead. This command will always show you the number of packages that can be upgraded, or a note saying everything is up to date.

For more information about the differences between apt and apt-get, check out this section below.

How to Use the sudo apt-get upgrade Command

After running the sudo apt-get update command, in the same terminal window, type in sudo apt-get upgrade, enter your password if necessary, and hit enter.

Then, you'll see an output similar to this:

kris@pihole:~ $ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libcamera0 raspi-config rpi-eeprom
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,616 kB of archives.
After this operation, 1,596 kB of additional disk space will be used.
Do you want to continue? [Y/n]

Towards the bottom of the output, you'll see the packages that will be upgraded:

The following packages will be upgraded:
  libcamera0 raspi-config rpi-eeprom
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

The amount of data that needs to be fetched, and the amount of storage space the upgraded packages will use once installed:

Need to get 2,616 kB of archives.
After this operation, 1,596 kB of additional disk space will be used.

And finally, you'll see a prompt asking if you want to continue with the upgrade:

Do you want to continue? [Y/n]

You can enter y, Y, or yes to continue with the upgrade, or n, N, or no to exit out of the upgrade command.

If you choose to exit out, you'll see an output like this:

kris@pihole:~ $ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libcamera0 raspi-config rpi-eeprom
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,616 kB of archives.
After this operation, 1,596 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.

If you choose to continue with the upgrade, you'll see a long output like this:

kris@pihole:~ $ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libcamera0 raspi-config rpi-eeprom
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,616 kB of archives.
After this operation, 1,596 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.raspberrypi.org/debian bullseye/main armhf libcamera0 armhf 0~git20220426+18e68a9b-1 [548 kB]
Get:2 http://archive.raspberrypi.org/debian bullseye/main armhf raspi-config all 20220425 [30.3 kB]
Get:3 http://archive.raspberrypi.org/debian bullseye/main armhf rpi-eeprom armhf 13.13-1 [2,037 kB]
Fetched 2,616 kB in 3s (1,019 kB/s)   
Reading changelogs... Done
(Reading database ... 43496 files and directories currently installed.)
Preparing to unpack .../libcamera0_0~git20220426+18e68a9b-1_armhf.deb ...
Unpacking libcamera0:armhf (0~git20220426+18e68a9b-1) over (0~git20220303+e68e0f1e-1) ...
Preparing to unpack .../raspi-config_20220425_all.deb ...
Unpacking raspi-config (20220425) over (20220419) ...
Preparing to unpack .../rpi-eeprom_13.13-1_armhf.deb ...
Unpacking rpi-eeprom (13.13-1) over (13.12-1) ...
Setting up rpi-eeprom (13.13-1) ...
Setting up libcamera0:armhf (0~git20220426+18e68a9b-1) ...
Setting up raspi-config (20220425) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u2) ...

And once that's complete, all of the outdated packages and dependencies will be updated.

One important thing to remember about the sudo apt-get upgrade command is that it only upgrades what it can without removing anything.

For example, if an upgrade requires a new dependency, the upgrade command will download and install it, but it will not remove the old dependency. Removing old dependencies requires a different command. You'll see this a lot when you upgrade to a new kernel version.

If you see a message similar to this after upgrading:

The following packages were automatically installed and are no longer required:
  g++-8 gir1.2-mutter-4 libapache2-mod-php7.2 libcrystalhd3
Use 'sudo apt autoremove' to remove them.

You can follow the suggestion and use sudo apt autoremove to remove those unnecessary packages.

How to Use Special Options With the sudo apt-get upgrade Command

There are a number of special options or parameters that you can use with the sudo apt-get upgrade command, but two stand out: --dry-run and --yes.

How to Use the --dry-run Option:

The --dry-run (alternatively, -s or --simulate) option simulates what would happen during the upgrade process, but doesn't actually change anything on your system:

kris@pihole:~ $ sudo apt-get upgrade --dry-run
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libcamera0 raspi-config rpi-eeprom
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Inst libcamera0 [0~git20220303+e68e0f1e-1] (0~git20220426+18e68a9b-1 Raspberry Pi Foundation:stable [armhf])
Inst raspi-config [20220331] (20220425 Raspberry Pi Foundation:stable [all])
Inst rpi-eeprom [13.12-1] (13.13-1 Raspberry Pi Foundation:stable [armhf])
Conf libcamera0 (0~git20220426+18e68a9b-1 Raspberry Pi Foundation:stable [armhf])
Conf raspi-config (20220425 Raspberry Pi Foundation:stable [all])
Conf rpi-eeprom (13.13-1 Raspberry Pi Foundation:stable [armhf])

Though again, while Debian and Debian-based distros are very stable, this option is useful if you want to make sure there are no conflicts during an upgrade.

How to Use the --yes Option:

The --yes (alternatively, -y or --assume-yes) option automatically answers yes to any prompts if it's safe to do so:

kris@pihole:~ $ sudo apt-get upgrade --yes
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libcamera0 raspi-config rpi-eeprom
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,616 kB of archives.
After this operation, 1,596 kB of additional disk space will be used.
Get:1 http://archive.raspberrypi.org/debian bullseye/main armhf libcamera0 armhf 0~git20220426+18e68a9b-1 [548 kB]
Get:2 http://archive.raspberrypi.org/debian bullseye/main armhf raspi-config all 20220425 [30.3 kB]
Get:3 http://archive.raspberrypi.org/debian bullseye/main armhf rpi-eeprom armhf 13.13-1 [2,037 kB]
...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u2) ...

Note that the Do you want to continue? [Y/n] is skipped above, and all packages are upgraded.

FAQs

What Are sudo and apt-get?

An important thing to note about sudo apt-get update and sudo apt-get upgrade is that both commands are made up of three parts: sudo, apt-get, and update or upgrade.

sudo stands for "superuser do", and allows you to run programs with root or admin privileges.

For example, rebooting a system requires superuser/root-level privileges, so running reboot in the terminal might return errors similar to this:

Failed to set wall message, ignoring: Interactive authentication required.
Failed to reboot system via logind: Interactive authentication required.
Failed to open initctl fifo: Permission denied
Failed to talk to init daemon.

But if you run sudo reboot, then enter your admin password, you will run the reboot command as a superuser, and your system will restart immediately.

apt-get is a command line tool in Debian and Debian-based Linux distros that you use to install and manage packages.

What's the Difference Between apt-get and apt?

apt is a more modern tool for installing and managing applications on Debian and Debian-based distros.

For the most part, apt and apt-get can be used interchangeably – sudo apt update and sudo apt-get update both update the package list on your system.

The main differences you'll notice is that apt is easier to type, its output is generally more useful, and it includes some user-friendly features like a progress bar when installing packages.

While most of the examples in this article use apt-get, I strongly encourage you to use apt instead.

Are sudo apt-get update and sudo apt-get upgrade Safe to Use?

Yes, Debian and Debian-based distros are generally very stable, and the update and upgrade commands are safe to use. This is because major updates for packages / dependencies, and the distros themselves, are only released once or twice a year.

The downside is that, unlike with bleeding edge distros like Arch Linux, if you want to use the most recent version of a package, you might need to put in some extra work. You may need to configure a third-party repository via a PPA, use an alternative packaging system like Snap of Flatpak, or compile the package yourself.

But the stability that comes with slightly older software is worth it, at least in my opinion.

Can You Chain the sudo apt-get update and sudo apt-get upgrade Commands?

You might be thinking, isn't it tedious to run sudo apt-get update, wait for that to complete, then run sudo apt-get upgrade?

While both sudo apt-get update and sudo apt-get upgrade run pretty quickly, sometimes it's easier to execute a string of commands and check back on them a few minutes later.

With the && operator, you can chain multiple commands together like this:

sudo apt-get update && sudo apt-get upgrade

The important thing to remember with the && operator is that the command after the operator only runs if the command before it is successful.

Using the example above, sudo apt-get upgrade only runs if sudo apt-get update succeeds. If there's some sort of error, like a network problem while updating the package list, then sudo apt-get update is skipped.

What Are sudo apt-get dist-upgrade and sudo apt full-upgrade, and Are They Safe to Use?

According to this Stack Overflow thread, these commands do the same thing under the hood – they upgrade outdated packages, and also intelligently remove some packages whenever necessary.

Essentially they're like the combination of the sudo apt-get upgrade and sudo apt autoremove commands.

Running these commands should be safe in most cases.

But a lot of people, myself included, recommend using sudo apt-get update and sudo apt-get upgrade instead. You have more of a chance to review upcoming changes, and since upgrade never removes packages, it's less destructive.

./thanks_for_reading.sh

If you found this breakdown on sudo apt-get update and sudo apt-get upgrade useful, please share it with your friends so more people can benefit from it.

Also, feel free to reach out on Twitter and let me know what you think.

The command line is one of the most useful and efficient tools we have as developers and as computer users in general. But using it can feel a bit overwhelming and complex when you're starting out.

In this article I'll try my best to simply explain the parts that make up the command line interface, and the basics of how it works, so you can start using it for your daily tasks.

Let's go! =D

Table of Contents

• Difference between console, terminal, command line (CLI) and Shell

  • Console

  • Terminal

  • Shell

  • Command line (CLI)

• Why should I even care about using the terminal?

• Different kinds of shells

  • A bit of history - Posix

  • How do I know what shell I'm running?

  • What shell is better?

    • A comment about customization

• Most common and useful commands to use

  • Git commands

• Our first script

• Round up

Difference between console, command line (CLI), terminal and Shell

I think a good place to start is to know exactly what the command line is.

When referring to this, you may have heard the terms Terminal, console, command line, CLI, and shell. People often use these words interchangeably but the truth is they're actually different things.

Differentiating each isn't necesarilly crucial knwoledge to have, but it will help clarify things. So lets briefly explain each one.

Console:

The console is the physical device that allows you to interact with the computer.

In plain English, it's your computer screen, keyboard, and mouse. As a user, you interact with your computer through your console.

Terminal:

A terminal is a text input and output environment. It is a program that acts as a wrapper and allows us to enter commands that the computer processes.

In plain English again, it's the "window" in which you enter the actual commands your computer will process.

Keep in mind the terminal is a program, just like any other. And like any program, you can install it and uninstall it as you please. It's also possible to have many terminals installed in your computer and run whichever you want whenever you want.

All operating systems come with a default terminal installed, but there are many options out there to choose from, each with its own functionalities and features.

Shell:

A shell is a program that acts as command-line interpreter. It processes commands and outputs the results. It interprets and processes the commands entered by the user.

Same as the terminal, the shell is a program that comes by default in all operating systems, but can also be installed and uninstalled by the user.

Different shells come with different syntax and characteristics as well. It's also possible to have many shells installed at your computer and run each one whenever you want.

In most Linux and Mac operating systems the default shell is Bash. While on Windows it's Powershell. Some other common examples of shells are Zsh and Fish.

Shells work also as programming languages, in the sense that with them we can build scripts to make our computer execute a certain task. Scripts are nothing more than a series of instructions (commands) that we can save on a file and later on execute whenever we want.

We'll take a look at scripts later on in this article. For now just keep in mind that the shell is the program your computer uses to "understand" and execute your commands, and that you can also use it to program tasks.

Also keep in mind that the terminal is the program in which the shell will run. But both programs are independent. That means, I can have any shell run on any terminal. There's no dependance between both programs in that sense.

Command line or CLI (command line interface):

The CLI is the interface in which we enter commands for the computer to process. In plain English once again, it's the space in which you enter the commands the computer will process.

This is practically the same as the terminal and in my opinion these terms can be used interchangeably.

One interesting thing to mention here is that most operating systems have two different types of interfaces:

• The CLI, which takes commands as inputs in order for the computer to execute tasks.

• The other is the GUI (graphical user interface), in which the user can see things on the screen and click on them and the computer will respond to those events by executing the corresponding task.

Why should I even care about using the terminal?

We just mentioned that most operating systems come with a GUI. So if we can see things on the screen and click around to do whatever we want, you might wonder why you should learn this complicated terminal/cli/shell thing?

The first reason is that for many tasks, it's just more efficient. We'll see some examples in a second, but there are many tasks where a GUI would require many clicks around different windows. But on the CLI these tasks can be executed with a single command.

In this sense, being comfortable with the command line will help you save time and be able to execute your tasks quicker.

The second reason is that by using commands you can easily automate tasks. As previously mentioned, we can build scripts with our shell and later on execute those scripts whenever we want. This is incredibly useful when dealing with repetitive tasks that we don't want to do over and over again.

Just to give some examples, we could build a script that creates a new online repo for us, or that creates a certain infrastructure on a cloud provider for us, or that executes a simpler task like changing our screen wallpaper every hour.

Scripting is a great way to save up time with repetitive tasks.

The third reason is that sometimes the CLI will be the only way in which we'll be able to interact with a computer. Take, for example, the case when you would need to interact with a cloud platform server. In most of these cases, you won't have a GUI available, just a CLI to run commands in.

So being comfortable with the CLI will allow you to interact with computers on all ocassions.

The last reason is it looks cool and it's fun. You don't see movie hackers clicking around their computers, right? ;)

Different kinds of shells

Before diving into the actual commands you can run in your terminal, I think it's important to recognize the different types of shells out there and how to identify which shell you're currently running.

Different shells come with different syntax and different features, so to know exactly what command to enter, you first need to know what shell you're running.

A bit of history – Posix

For shells, there's a common standard called Posix.

Posix works for shells in a very similar way that ECMAScript works for JavaScript. It's a standard that dictates certain characteristics and features that all shells should comply with.

This standard was stablished in the 1980's and most current shells were developed according to that standard. That's why most shells share similar syntax and similar features.

How do I know what shell I'm running?

To know what shell you're currently running, just open your terminal and enter echo $0. This will print the current running program name, which in this case is the actual shell.

What shell is better?

There's not A LOT of difference between most shells. Since most of them comply with the same standard, you'll find that most of them work similarly.

There are some slight differences you might want to know, though:

• As mentioned, Bash is the most widely used and comes installed by default on Mac and Linux.

• Zsh is very similar to Bash, but it was created after it and comes with some nice improvements over it. If you'd like to have more detail about its differences, here's a cool article about it.

• Fish is another commonly used shell that comes with some nice built-in features and configurations such as autocompletion and syntax highlighting. The thing about Fish is that it's not Posix complaint, while Bash and Zsh are. This means that some of the commands you'll be able to run on Bash and Zsh won't run on Fish and viceversa. This makes Fish scripting less compatible with most computers compared to Bash and Zsh.

• There are also other shells like Ash or Dash (the naming just makes everything more confusing, I know...) that are stripped-down versions of Posix shells. This means they only offer the features required in Posix, and nothing else. While Bash and Zsh add more features than what Posix requires.

The fact that shells add more features makes them easier and friendlier to interact with, but slower to execute scripts and commands.

So a common practice is to use this "enhanced" shells like Bash or Zsh for general interaction, and a "stripped" shell like Ash or Dash to execute scripts.

When we get to scripting later on, we'll see how we can define what shell will execute a given script.

If you're interested in a more detailed comparison between these shells, here's a video that explains it really well:

If had to recommend a shell, I would recommend bash as it's the most standard and commonly-used one. This means you'll be able to translate your knowledge into most environments.

But again, truth is there's not A LOT of difference between most shells. So in any case you can try a few and see which one you like best. ;)

A comment about customization

I just mentioned that Fish comes with built-in configuration such as autocompletion and syntax highlighting. This come built-in in Fish, but in Bash or Zsh you can configure these features, too.

The point is that shells are customizable. You can edit how the program works, what commands you have available, what information your prompt shows, and more.

We won't see customization options in detail here, but know that when you install a shell in your computer, certain files will be created on your system. Later on you can edit those files to customize your program.

Also, there are many plugins available online that allow you to customize your shell in an easier way. You just install them and get the features that plugin offers. Some examples are OhMyZsh and Starship.

These customization options are also true for Terminals.

So not only do you have many shell and terminal options to choose from – you also have many configuration options for each shell and terminal.

If you're starting out, all this information can feel a bit overwhelming. But just know that there are many options available, and each option can be customized too. That's it.

Most common and useful commands to use

Now that we have a foundation of how the CLI works, let's dive into the most useful commands you can start to use for your daily tasks.

Keep in mind that these examples will be based on my current configuration (Bash on a Linux OS). But most commands should apply to most configurations anyway.

• Echo prints in the terminal whatever parameter we pass it.

echo Hello freeCodeCamp! // Output: Hello freeCodeCamp!

• pwd stands for print working directory and it prints the "place" or directory we are currently at in the computer.

pwd // Output: /home/German

• ls presents you the contents of the directory you're currently in. It will present you with both the files and other directories your current directory contains.

For example, here I'm on a React project directory I've been working on lately:

ls // Output:
node_modules  package.json  package-lock.json  public  README.md  src

If you pass this command the flag or paremter -a It will also show you hidden files or directories. Like .git or .gitignore files

ls -a // Output:
.   .env  .gitignore    package.json       public     src
..  .git  node_modules  package-lock.json  README.md

• cd is short for Change directory and it will take you from your current directory to another.

While on my home directory, I can enter cd Desktop and it will take me to the Desktop Directory.

If I want to go up one directory, meaning go to the directory that contains the current directory, I can enter cd ..

If you enter cd alone, it will take you straight to your home directory.

• mkdir stands for make directory and it will create a new directory for you. You have to pass the command the directory name parameter.

If I wanted to create a new directory called "Test" I would enter mkdir test.

• rmdir stands for Remove directory and it does just that. It needs the directory name parameter just as mkdir: rmdir test.

• touch allows you to create an empty file in your current directory. As parameters it takes the file name, like touch test.txt.

• rm allows you to delete files, in the same way rmdir allows you to remove directories. rm test.txt

• cp allows you to copy files or directories. This command takes two parameters: the first one is the file or directory you want to copy, and the second one is the destination of your copy (where do you want to copy your file/directory to).

If I want to make a copy of my txt file in the same directory, I can enter the following:

cp test.txt testCopy.txt

See that the directory doesn't change, as for "destination" I enter the new name of the file.

If I wanted to copy the file into a diferent directory, but keep the same file name, I can enter this:

cp test.txt ./testFolder/

And if I wanted to copy to a different folder changing the field name, of course I can enter this:

cp test.txt ./testFolder/testCopy.txt

• mv is short for move, and lets us move a file or directory from one place to another. That is, create it in a new directory and delete it in the previous one (same as you could do by cutting and pasting).

Again, this command takes two paremers, the file or directory we want to move and the destination.

mv test.txt ./testFolder/

We can change the name of the file too in the same command if we want to:

mv test.txt ./testFolder/testCopy.txt

• head allows you to view the beginning of a file or piped data directly from the terminal.

head test.txt // Output:
this is the beginning of my test file

• tail works the same but it will show you the end of the file.

tail test.txt // Output:

this is the end of my test file

• The --help flag can be used on most commands and it will return info on how to use that given command.

cd --help // output:
cd: cd [-L|[-P [-e]] [-@]] [dir]
Change the shell working directory.

Change the current directory to DIR. The default DIR is the value of the HOME shell variable.

The variable CDPATH defines the search path for the directory containing DIR. Alternative directory names in CDPATH are separated by a colon :.

A null directory name is the same as the current directory if DIR begins with ....

• In a similar way, the man command will return info about any particular command.

    man cp // output:

    CP(1)                            User Commands                           CP(1)

    NAME
           cp - copy files and directories

    SYNOPSIS
           cp [OPTION]... [-T] SOURCE DEST
           cp [OPTION]... SOURCE... DIRECTORY
           cp [OPTION]... -t DIRECTORY SOURCE...

    DESCRIPTION
           Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.

           Mandatory  arguments  to  long  options are mandatory for short options
           too.

           -a, --archive
                  same as -dR --preserve=all

           --attributes-only
                  don't copy the file data, just the attributes
    ...

You can even enter man bash and that will return a huge manual about everything there's to know about this shell. ;)

• code will open your default code editor. If you enter the command alone, it just opens the editor with the latest file/directory you opened.

You can also open a given file by passing it as parameter: code test.txt.

Or open a new file by passing the new file name: code thisIsAJsFile.js.

• edit will open text files on your default command line text editor (which if you're on Mac or Linux will likely be either Nano or Vim).

If you open your file and then can't exit your editor, first look at this meme:

And then type :q! and hit enter.

The meme is funny because everyone struggles with CLI text editors at first, as most actions (like exiting the editor) are done with keyboard shortcuts. Using these editors is a whole other topic, so go look for tutorials if you're interested in learning more. ;)

• ctrl+c allows you to exit the current process the terminal is running. For example, if you're creating a react app with npx create-react-app and want to cancel the build at some point, just hit ctrl+c and it will stop.

• Copying text from the terminal can be done with ctrl+shift+c and pasting can be done with ctrl+shift+v

• clear will clear your terminal from all previous content.

• exit will close your terminal and (this is not a command but it's cool too) ctrl+alt+t will open a new terminal for you.

• By pressing up and down keys you can navigate through the previous commands you entered.

• By hitting tab you will get autocompletion based on the text you've written so far. By hitting tab twice you'll get suggestions based on the text you've written so far.

For example if I write edit test and tab twice, I get testFolder/ test.txt. If I write edit test. and hit tab my text autocompletes to edit test.txt

Git commands

Besides working around the file system and installing/uninstalling things, interacting with Git and online repos is probably the most common things you're going to use the terminal for as a developer.

It's a whole lot more efficient to do it from the terminal than by clicking around, so let's take a look at the most useful git commands out there.

• git init will create a new local repository for you.

git init // output:
Initialized empty Git repository in /home/German/Desktop/testFolder/.git/

• git add adds one or more files to staging. You can either detail a specific file to add to staging or add all changed files by typing git add .

• git commit commits your changes to the repository. Commits must always be must be accompanied by the -m flag and commit message.

git commit -m 'This is a test commit' // output:
[master (root-commit) 6101dfe] This is a test commit
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 test.js

• git status tells you what branch are you currently on and whether you have changes to commit or not.

git status  // output:
On branch master
nothing to commit, working tree clean

• git clone allows you to clone (copy) a repository into the directory you're currently in. Keep in mind you can clone both remote repositories (in GitHub, GitLab, and so on) and local repositories (those that are stored in your computer).

git clone https://github.com/coccagerman/MazeGenerator.git // output:
Cloning into 'MazeGenerator'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 15 (delta 1), reused 11 (delta 0), pack-reused 0
Unpacking objects: 100% (15/15), done.

• git remote add origin is used to detail the URL of the remote repository you're going to use for your project. In case you'd like to change it at some point, you can do it by using the command git remote set-url origin.

git remote add origin https://github.com/coccagerman/testRepo.git

Keep in mind you need to create your remote repo first in order to get its URL. We'll see how you can do this from the command line with a little script later on. ;)

• git remote -v lets you list the current remote repository you're using.

git remote -v // output:
origin    https://github.com/coccagerman/testRepo.git (fetch)
origin    https://github.com/coccagerman/testRepo.git (push)

• git push uploads your commited changes to your remote repo.

git push // output:
Counting objects: 2, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 266 bytes | 266.00 KiB/s, done.
Total 2 (delta 0), reused 0 (delta 0)

• git branch lists all the available branches on your repo and tells you what branch you're currently on. If you want to create a new branch, you just have to add the new branch name as parameter like git branch <branch name>.

git branch // output:
* main

• git checkout moves you from one branch to another. It takes your destination branch as paremeter.

git checkout newBranch // output:
Switched to branch 'newBranch'

• git pull pulls (downloads) the code from your remote repository and combines it with your local repo. This is particularly useful when working in teams, when many developers are working on the same code base. In this case each developer periodically pulls from the remote repo in order to work in a code base that includes the changes done by all the other devs.

If there's new code in your remote repo, the command will return the actual files that were modified in the pull. If not, we get Already up to date.

git pull // output:
Already up to date.

• git diff allows you to view the differences between the branch you're currently in and another.

git diff newBranch // output:
diff --git a/newFileInNewBranch.js b/newFileInNewBranch.js
deleted file mode 100644
index e69de29..0000000

As a side comment, when comparing differences between branches or repos, ussually visual tools like Meld are used. It's not that you can't visualize it directly in the terminal, but this tools are greate for a clearer visualization.

• git merge merges (combines) the branch you're currently in with another. Keep in mind the changes will be incorporated only to the branch you're currently in, not to the other one.

git merge newBranch // output:
Updating f15cf51..3a3d62f
Fast-forward
 newFileInNewBranch.js | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 newFileInNewBranch.js

• git log lists all previous commits you've done in the repo.

git log // output:
commit 3a3d62fe7cea7c09403c048e971a5172459d0948 (HEAD -> main, tag: TestTag, origin/main, newBranch)
Author: German Cocca <german.cocca@avature.net>
Date:   Fri Apr 1 18:48:20 2022 -0300

    Added new file

commit f15cf515dd3ec398210108dce092debf26ff9e12
Author: German Cocca <german.cocca@avature.net>
    ...

• The --help flag will show you information about a given command, exactly the same way it works with bash.

git diff --help // output:
GIT-DIFF(1)                       Git Manual                       GIT-DIFF(1)

NAME
       git-diff - Show changes between commits, commit and working tree, etc

SYNOPSIS
       git diff [options] [<commit>] [--] [<path>...]
       git diff [options] --cached [<commit>] [--] [<path>...]
       ...

Our first script

Now we're ready to get to the truly fun and awesome part of the command line, scripting!

As I mentioned previously, a script is nothing more than a series of commands or instructions that we can execute at any given time. To explain how we can code one, we'll use a simple example that will allow us to create a github repo by running a single command. ;)

• First thing to do is create a .sh file. You can put it wherever want. I called mine newGhRepo.sh.

• Then open it on your text/code editor of choice.

• On our first line, we'll write the following: #! /bin/sh

This is called a shebang, and its function is to declare what shell is going to run this script.

Remember previously when we mentioned that we can use a given shell for general interaction and another given shell for executing a script? Well, the shebang is the instruction that dictates what shell runs the script.

As mentioned too, we're using a "stripped down" shell (also known as sh shells) to run the scripts as they're more efficient (though the difference might be unnoticeable to be honest, It's just a personal preference). In my computer I have dash as my sh shell.

If we wanted this script to run with bash the shebang would be #! /bin/bash

• Our next line will be repoName=$1

Here we're declaring a variable called repoName, and assigning it to the value of the first parameter the script receives.

A parameter is a set of characters that is entered after the script/comand. Like with the cd command, we need to specify a directory parameter in order to change directory (ie: cd testFolder).

A way we can identify parameters within a script is by using dollar sign and the order in which that parameter is expected.

If I'm expecting more than one parameter I could write:

paramOne=$1
paramTwo=$2
paramThree=$3
...

• So we're expecting the repository name as parameter of our script. But what happens if the user forgets to enter it? We need to plan for that so next we're going to code a conditional that keeps asking the user to enter the repo name until that parameter is received.

We can do that like this:

while [ -z "$repoName" ]
do
   echo 'Provide a repository name'
   read -r -p $'Repository name:' repoName
done

What we're doing here is:

1. While the repoName variable is not assigned (while [ -z "$repoName" ])

2. Write to the console this message (echo 'Provide a repository name')

3. Then read whatever input the user provides and assign the input to the repoName variable (read -r -p $'Repository name:' repoName)

• Now that we have our repo name in place, we can create our local Git repo like this:

echo "# $repoName" >> README.md
git init
git add .
git commit -m "First commit"

This is creating a readme file and writting a single line with the repo name (echo "# $repoName" >> README.md) and then initializing the git repo and making a first commit.

• Then it's time to upload our repo to github. To do that we're going to take advantage of the github API in the following command:

curl -u coccagerman https://api.github.com/user/repos -d '{"name": "'"$repoName"'", "private":false}'

curl is a command to transfer data from or to a server, using one of the many supported protocols.

Next we're using the -u flag to declare the user we're creating the repo for (-u coccagerman).

Next comes the endpoint provided by the GitHub API (https://api.github.com/user/repos)

And last we're using the -d flag to pass parameters to this command. In this case we're indicating the repository name (for which we're using our repoName variable) and setting private option to false, since we want our repo to be puiblic.

Lots of other config options are available in the API, so check the docs for more info.

• After running this command, GitHub will prompt us to enter our private token for authentication.

If you don't have a private token yet, you can generate it in GitHub in Settings > Developer settings > Personal access tokens

• Cool, we're almost done now! What we need now is the remote URL of our newly created GitHub repo.

To get that we're going to use curl and the GitHub API again, like this:

GIT_URL=$(curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/coccagerman/"$repoName" | jq -r '.clone_url')

Here we're declaring a variable called GIT_URL and assigning it to whatever the following command returns.

The -H flag sets the header of our request.

Then we pass the GitHub API endpoint, which should contain our user name and repo name (https://api.github.com/repos/coccagerman/"$repoName").

Then we're piping the return value of our request. Piping just means passing the return value of a process as the input value of another process. We can do it with the | symbol like <process1> | <process2>.

And finally we run the jq command, which is a tool for processing JSON inputs. Here we tell it to get the value of .clone_url which is where our remote git URL will be according to the data format provided by the GitHub API.

• And as last step, we rename our master branch to main, add the remote origin we just obtained, and push our code to GitHub! =D

git branch -M main
git remote add origin $GIT_URL
git push -u origin main

Our full script should look something like this:

#! /bin/sh
repoName=$1

while [ -z "$repoName" ]
do
    echo 'Provide a repository name'
    read -r -p $'Repository name:' repoName
done

echo "# $repoName" >> README.md
git init
git add .
git commit -m "First commit"

curl -u <yourUserName> https://api.github.com/user/repos -d '{"name": "'"$repoName"'", "private":false}'

GIT_URL=$(curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/<yourUserName>/"$repoName" | jq -r '.clone_url')

git branch -M main
git remote add origin $GIT_URL
git push -u origin main

• Now it's time to test our script! To execute it there're two things we can do.

One option is to enter the shell name and pass the file as parameter, like: dash ../ger/code/projects/scripts/newGhRepo.sh.

And the other is to make the file executable by running chmod u+x ../ger/code/projects/scripts/newGhRepo.sh.

Then you can just execute the file directly by running ../ger/code/projects/scripts/newGhRepo.sh.

And that's it! We have our script up and running. Everytime we need a new repo we can just execute this script from whatever directory we're in.

But there's something a bit annoying about this. We need to remember the exact route of the script directory. Wouldn't it be cool to execute the script with a single command that it's always the same independently of what directory we're at?

In come bash aliases to solve our problem.

Aliases are a way bash provides for making names for exact commands we want to run.

To create a new alias, we need to edit the bash configuration files in our system. This files are normally located in the home directory. Aliases can be defined in different files (mainly .bashrc or .bash_aliases).

I have a .bash_aliases file on my system, so let's edit that.

• In our CLI we enter cd to go over home directory.

• Then we can enter ls -a to list all files (includen hidden ones) and check if we have either a .bashrc or .bash_aliases file in our system.

• We open the file with our text/code editor of choice.

• And we write our new alias like this: alias newghrepo="dash /home/German/Desktop/ger/code/projects/scripts/newGhRepo.sh"

Here I'm declaring the alias name, the actual command I'm going to enter to run the script (newghrepo).

And between quotes, define what that alias is going to do ("dash /home/German/Desktop/ger/code/projects/scripts/newGhRepo.sh")

See that I'm passing the absolute path of the script, so that this command works the same no matter what my current directory is.

If you don't know what the absolute path of your script is, go to the script directory on your terminal and enter readlink -f newGhRepo.sh. That should return the full path for you. ;)

• After we're done editing, we save our file, restart our terminal, and voilà! Now we can run our script by just entering newghrepo, no matter in what directory we currently are. Much quicker than opening the browser and clicking around to create our repo! =D

I hope this gives you a little taste of the kind of optimizations that are possible with scripting. It certainly requires a bit more work the first time you write, test, and set up the script. But after that, you'll never have to perform that task manually again. ;)

Round up

The terminal can feel like an intimidating and intricate place when you're starting out. But it's certainly worth it to put time and effort into learning the ins and outs of it. The efficiency benefits are too good to pass up!

If you're interested in learning more about the terminal and Bash, Zach Gollwitzer has an awesome crash course series on youtube. He has also great tutorials on other topics such as Node and Javascript, so I recommend that you follow him. ;)

As always, I hope you enjoyed the article and learned something new. If you want, you can also follow me on linkedin or twitter.

Cheers and see you in the next one! =D

This saves you time because you don't have to write certain commands again and again. You can perform daily tasks efficiently and even schedule them for automatic execution.

You can also set certain scripts to execute on startup such as showing a particular message on launching a new session or setting certain environment variables.

The applications and uses of scripting are numerous, so let's dive in.

In this article, you will learn:

1. What is a bash shell?

2. What is a bash script and how do you identify it?

3. How to create your first bash script and execute it.

4. The basic syntax of shell scripting.

5. How to see a system's scheduled scripts.

6. How to automate scripts by scheduling via cron jobs.

The best way to learn is by practicing. I highly encourage you to follow along using Replit. You can access a running Linux shell within minutes.

Introduction to the Bash Shell

The Linux command line is provided by a program called the shell. Over the years, the shell program has evolved to cater to various options.

Different users can be configured to use different shells. But most users prefer to stick with the current default shell. The default shell for many Linux distros is the GNU Bourne-Again Shell (bash). Bash succeeded the Bourne shell (sh).

When you first launch the shell, it uses a startup script located in the .bashrc or .bash_profile file which allows you to customize the behavior of the shell.

When a shell is used interactively, it displays a $ when it is waiting for a command from the user. This is called the shell prompt.

[username@host ~]$

If shell is running as root, the prompt is changed to #. The superuser shell prompt looks like this:

[root@host ~]#

Bash is very powerful as it can simplify certain operations that are hard to accomplish efficiently with a GUI. Remember that most servers do not have a GUI, and it is best to learn to use the powers of a command line interface (CLI).

What is a Bash Script?

A bash script is a series of commands written in a file. These are read and executed by the bash program. The program executes line by line.

For example, you can navigate to a certain path, create a folder and spawn a process inside it using the command line.

You can do the same sequence of steps by saving the commands in a bash script and running it. You can run the script any number of times.

How Do You Identify a Bash Script?

File extension of .sh.

By naming conventions, bash scripts end with a .sh. However, bash scripts can run perfectly fine without the sh extension.

Scripts start with a bash bang.

Scripts are also identified with a shebang. Shebang is a combination of bash # and bang ! followed the the bash shell path. This is the first line of the script. Shebang tells the shell to execute it via bash shell. Shebang is simply an absolute path to the bash interpreter.

Below is an example of the shebang statement.

#! /bin/bash

The path of the bash program can vary. We will see later how to identify it.

Execution rights

Scripts have execution rights for the user executing them.

An execution right is represented by x. In the example below, my user has the rwx (read, write, execute) rights for the file test_script.sh

File colour

Executable scripts appear in a different colour from rest of the files and folders.

In my case, the scripts with execution rights appear as green.

How to Create Your First Bash Script

Let's create a simple script in bash that outputs Hello World.

Create a file named hello_world.sh

touch hello_world.sh

Find the path to your bash shell.

which bash

In my case, the path is /usr/bin/bash and I will include this in the shebang.

Write the command.

We will echo "hello world" to the console.

Our script will look something like this:

#! /usr/bin/bash
echo "Hello World"

Edit the file hello_world.sh using a text editor of your choice and add the above lines in it.

Provide execution rights to your user.

Modify the file permissions and allow execution of the script by using the command below:

chmod u+x hello_world.sh

chmod modifies the existing rights of a file for a particular user. We are adding +x to user u.

Run the script.

You can run the script in the following ways:

./hello_world.sh

bash hello_world.sh.

Here's the output:

Two ways to run scripts

The Basic Syntax of Bash Scripting

Just like any other programming language, bash scripting follows a set of rules to create programs understandable by the computer. In this section, we will study the syntax of bash scripting.

How to define variables

We can define a variable by using the syntax variable_name=value. To get the value of the variable, add $ before the variable.

#!/bin/bash
# A simple variable example
greeting=Hello
name=Tux
echo $greeting $name

Tux is also the name of the Linux mascot, the penguin.

Hi, I am Tux.

Arithmetic Expressions

Below are the operators supported by bash for mathematical calculations:

Let's run a few examples.

Note the spaces, these are part of the syntax

Numerical expressions can also be calculated and stored in a variable using the syntax below:

var=$((expression))

Let's try an example.

#!/bin/bash

var=$((3+9))
echo $var

Fractions are not correctly calculated using the above methods and truncated.

For decimal calculations, we can use bc command to get the output to a particular number of decimal places. bc (Bash Calculator) is a command line calculator that supports calculation up to a certain number of decimal points.

echo "scale=2;22/7" | bc

Where scale defines the number of decimal places required in the output.

Getting output to 2 decimal places

How to read user input

Sometimes you'll need to gather user input and perform relevant operations.

In bash, we can take user input using the read command.

read variable_name

To prompt the user with a custom message, use the -p flag.

read -p "Enter your age" variable_name

Example:

#!/bin/bash

echo "Enter a numner"
read a

echo "Enter a numner"
read b

var=$((a+b))
echo $var

Numeric Comparison logical operators

Comparison is used to check if statements evaluate to true or false. We can use the below shown operators to compare two statements:

Syntax:

if [ conditions ]
    then
         commands
fi

Example:

Let's compare two numbers and find their relationship:

read x
read y

if [ $x -gt $y ]
then
echo X is greater than Y
elif [ $x -lt $y ]
then
echo X is less than Y
elif [ $x -eq $y ]
then
echo X is equal to Y
fi

Output:

Conditional Statements (Decision Making)

Conditions are expressions that evaluate to a boolean expression (true or false). To check conditions, we can use if, if-else, if-elif-else and nested conditionals.

The structure of conditional statements is as follows:

• if...then...fi statements

• if...then...else...fi statements

• if..elif..else..fi

• if..then..else..if..then..fi..fi.. (Nested Conditionals)

Syntax:

if [[ condition ]]
then
    statement
elif [[ condition ]]; then
    statement 
else
    do this by default
fi

To create meaningful comparisons, we can use AND -a and OR -o as well.

The below statement translates to: If a is greater than 40 and b is less than 6.

if [ $a -gt 40 -a $b -lt 6 ]

Example: Let's find the triangle type by reading the lengths of its sides.

read a
read b
read c

if [ $a == $b -a $b == $c -a $a == $c ]
then
echo EQUILATERAL

elif [ $a == $b -o $b == $c -o $a == $c ]
then 
echo ISOSCELES
else
echo SCALENE

fi

Output:

Test case #1

Test case #2

Test case #3

Looping and skipping

For loops allow you to execute statements a specific number of times.

Looping with numbers:

In the example below, the loop will iterate 5 times.

#!/bin/bash

for i in {1..5}
do
    echo $i
done

Looping with strings:

We can loop through strings as well.

#!/bin/bash

for X in cyan magenta yellow  
do
    echo $X
done

While loop

While loops check for a condition and loop until the condition remains true. We need to provide a counter statement that increments the counter to control loop execution.

In the example below, (( i += 1 )) is the counter statement that increments the value of i.

Example:

#!/bin/bash
i=1
while [[ $i -le 10 ]] ; do
   echo "$i"
  (( i += 1 ))
done

Reading files

Suppose we have a file sample_file.txt as shown below:

We can read the file line by line and print the output on the screen.

#!/bin/bash

LINE=1

while read -r CURRENT_LINE
    do
        echo "$LINE: $CURRENT_LINE"
    ((LINE++))
done < "sample_file.txt"

Output:

Lines with line number printed

How to execute commands with back ticks

If you need to include the output of a complex command in your script, you can write the statement inside back ticks.

Syntax:

var= commands

Example: Suppose we want to get the output of a list of mountpoints with tmpfs in their name. We can craft a statement like this: df -h | grep tmpfs.

To include it in the bash script, we can enclose it in back ticks.

#!/bin/bash

var=`df -h | grep tmpfs`
echo $var

Output:

How to get arguments for scripts from the command line

It is possible to give arguments to the script on execution.

$@ represents the position of the parameters, starting from one.

#!/bin/bash

for x in $@
do
    echo "Entered arg is $x"
done

Run it like this:

./script arg1 arg2

How to Automate Scripts by Scheduling via cron Jobs

Cron is a job scheduling utility present in Unix like systems. You can schedule jobs to execute daily, weekly, monthly or in a specific time of the day. Automation in Linux heavily relies on cron jobs.

Below is the syntax to schedule crons:

# Cron job example
* * * * * sh /path/to/script.sh

Here, * represents minute(s) hour(s) day(s) month(s) weekday(s), respectively.

Below are some examples of scheduling cron jobs.

You can learn about cron in detail in this blog post.

How to Check Existing Scripts in a System

Using crontab

crontab -l lists the already scheduled scripts for a particular user.

My scheduled scripts

Using the find command

The find command helps to locate files based on certain patterns. As most of the scripts end with .sh, we can use the find script like this:

find . -type f -name "*.sh"

Where,

• . represents the current directory. You can change the path accordingly.

• -type f indicates that the file type we are looking for is a text based file.

• *.sh tells to match all files ending with .sh.

If you are interested to read about the find command in detail, check my other post.

Wrapping up

In this tutorial we learned the basics of shell scripting. We looked into examples and syntax which can help us write meaningful programs.

What’s your favorite thing you learned from this tutorial? Let me know on Twitter!

You can read my other posts here.

Work vector created by macrovector - www.freepik.com