Get-ChildItem (also known as gci, ls, dir ) is a very powerful command. And one of its most iconic uses is to find/search for a file. It's more precise and more reliable than Windows Explorer. It even has better filtering options that show the results that are more relevant to you.

In this tutorial, you'll learn how to use gci and how to combine it with other commands so that it becomes an even more powerful tool. Remember to enable copy-pasting in Windows PowerShell, so it's easier for you to follow along. You can see how to enable it here.

What we'll cover:

1. Basic explanation of the Get-ChildItem command

   • Most used examples of searching by gci command

2. Setup for other more complex examples

3. When is the -Path option not needed?

4. Advanced Searching – Combining Get-ChildItem with the Where-Object Command

   • 4.1. How to search through only a particular directory

   • 4.2. How to search while excluding a particular directory

   • 4.3 Searching only 1 directory from many with exactly the same name

   • 4.4 Filter how deep (how many folders in) you want to search for the file

5. How to Search Through Hidden Files

6. How can you know all the properties that you can use as a filter?

   • How to retrieve only 1 desired property

7. I don't know the file’s name, but I know what's inside it. How do I find the file by its content?

8. I can't see the full path - how do I fix this?

9. Hard to read? Open the results in the text editor of your choice

10. Summary - the ultimate commands for searching and finding whatever you need

1. Basic Explanation of the Get-ChildItem Command

Let's take a look at the example searching script to understand how it works:

Get-ChildItem -Recurse -Path "C:\path to\your directory\" -Filter "*whatImLookingFor*"

Get-ChildItem (aliases: dir, ls, gci) lists the content of a folder or directory just like the Linux ls command does.

This command works by searching every single file and directory in the path specified. It shows you everything it found that matches the filter. It doesn't mean that this command doesn't look everywhere else – because it does.

So you specify the path that is the parent (folder), which means that every folder and file under it is its child. If you know some CSS and JavaScript, treat it the same way that these languages do.

If you don't use -Recurse or -Depth, then the command works only in your current directory (parent Depth level 0) and searches for its children inside that directory (children Depth level 0).

If you use -Recurse, then the gci will search for what you want on ALL LEVELS. But by using-Depth, you can specify how deep you want it to look for a file/folder.

To recurse means "to repeat an operation". So, -Recurse means that gci will repeat the search for your file or folder in every child element of the "Documents" directory, and every directory inside it, all levels deep.

All of these files and folders are children of your "Documents" folder. If you delete the folder, you delete everything inside it too.

-Filter filters the output of the command to only show what matches the filter (examples of how to use filter are further in the article).

-Path tells where the command should be looking for files (by using "C:\", for example, you're telling it to look at the very basis of your computer). If you want to search in certain directory it would look like this:

Get-ChildItem -Path "C:\path to\your directory\"

OR

Get-ChildItem -Path "~\Documents\path to\your directory\"

~\ here is a shorthand for "inside current user's folder" or "C:\Users\YourUsername".

Next, we can specify whether we'd like to look for a file or a folder, so we have fewer results to look at:

Get-ChildItem -Path C:\ -Recurse -Filter "*whatImLookingFor*" -File

Get-ChildItem -Path C:\ -Recurse -Filter "*whatImLookingFor*" -Directory

You might be wondering how you can stop the search if it takes too long. When you're using -Recurse, the output that you'll get might become quite overwhelming, especially if you didn't specify your command enough (more about that in step 3 and step 4). Luckily, you can stop any command in PowerShell after starting it with Ctrl + C OR Ctrl + Z OR Ctrl + X. All of them should work.

Most Used Examples of Searching by gci Command

Here are some handy examples of searching scripts that you can use:

Example #1: search for all executive files on your PC (remember that you can stop this command with one of shortcuts, like Ctrl + C):

Get-ChildItem -Path C:\ -Recurse -Filter "*.exe" -File

REMEMBER:
In order to paste commands into the PowerShell, you have to first enable it. Here's how.

This command will show you a very long list of executable files and their folders (as shown in the image below).

These lists might be so long that it's impossible to find anything in them. That's why you'll learn how to use more advanced techniques of filtering in step 4 to see fewer unnecessary results that don't fit your criteria.

Example #2: search for an executable file that has "notepad" in its name (or search for any program you need, basically):

Get-ChildItem -Path C:\ -Recurse -Filter "notepad*.exe" -File

One of the results will show you the location of the file you want:

In our case it's the C:\Windows\System32 folder.

You can mix it however you want! Thanks to that command, you don't have to remember much about your file and it will still work.

Get-ChildItem -Path C:\ -Recurse -Filter "n*pad*.*xe"

So what if you see some errors while scanning the whole system. Should you worry?

It's ok! Sometimes you might get lots of errors. They will most likely occur when a script scours the system folders/files. If you want to get rid of them, add -ErrorAction SilentlyContinue, like you see here:

Get-ChildItem -Path C:\ -Recurse -Filter "notepad*.exe" -File -ErrorAction SilentlyContinue

You can try it now ;)

2. Setup for Other More Complex Examples

Now, let's look at even more use cases for this command. But first, we'll create a space where I can show you examples.

First, create new folder inside your "Documents" folder. Let's call it "Items".

Inside it, create two text documents. Name one of them "Item 1- Green Bracelet" and the other "Item 2- Blue Bracelet" (Yes, make sure you write the first letter of each word in UPPER CASE).

Copy these files now.

Go one folder back (you can use the Ctrl + UpArrow shortcut ) and create another folder next to "Items" called "More items":

Paste the copied files inside the "More items" folder and change their names, so they have only lower case letters ("item 1- green bracelet" and "item 2- blue bracelet" ).

PRO TIP:
You can click once on a file with your mouse and then type the F2 key on your keyboard in order to change their names.

3. When is the -Path option not needed?

You don't have to specify the path every time. You can always just move to the desired directory with the cd (change directory) command.

This command will move you to your Documents folder:

cd ~\Documents\

Now, you should be able to see PowerShell pointing to your Documents folder on the left of the screen:

If you don't see this, then you can use double quotes " ", like in this command:

cd "~\Documents\"

Make sure that PowerShell is pointing to our desired folder. Now, the searching command looks like this without the -Path option:

Get-ChildItem -Recurse -Filter "*item*" -File

Pretty simple, right?

As you can see in the image above, we first moved to our desired directory, so later we could perform the search inside it without specifying the -Path option/parameter.

But the -Path option is very useful, either when you're creating a script or you want to search for something without moving away from the current directory:

Get-ChildItem -Path ~\Documents\ -Recurse -Filter "*item*" -File

Get-ChildItem -Path ~\Documents\ -Recurse -Filter "*item*" -Directory

Here's an example. I'm inside the System32 folder and I want to know whether the thing I'm looking for is inside the Documents folder without moving in there:

And it really is there!

From now on, because you already know what the -Path option is being used for, I won't be using it unless it's necessary.

4. Advanced Searching – Combining Get-ChildItem with the Where-Object Command

Sometimes you might have several folders named exactly the same, but they're in different places. You might want to exclude them based on their content, which folder they are in, or based on their-Depth level (see the graphic with the explanation about -Depth level in step 1). That's what we're going to cover in the next few points.

For this part of the tutorial, make sure you've gone through step 2 (but you can skip step 3 if you want).

4.1. Searching through only a particular directory

Let's say that we're now looking for the bracelets that we created in step 2. But, we want to see the results from only one folder. For that, we'll use case-sensitive search (-clike) to get only our preferred results. But -clike doesn't work with gci alone. We need to apply another filter with the Where-Object { } command:

Get-ChildItem -Path ~\Documents\ -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*Item*" }

OR (clearer version, without the -Path option):

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*Item*" }

Let's review what's going on here:

• Get-ChildItem -Recurse -Filter "*item*" searches for all files and folders with "item" in their name

• | – the "pipe" symbol is used to get the output of the previous command (the list of all files and folders filtered by gci) and send it to the next command (Where-Object is applying another filter to what is already filtered by gci).

• Where-Object { } is the command used for filtering the lists of objects. The filter is being specified inside the { } curly brackets.

• \(_ refers to all the separate objects. Treat it as "ForEachObjectFromList". And treat the whole sequence after the | as "FindObjectsFromList that have a name with 'Item' ".
  \)_ is very often used with Where-Object, but also with some other commands.

• .Name – we choose a Name property to get from every object.

• -clike finds a match that is 100% correct. All letters must be the exact same case as the phrase we specified. c stands for "case sensitive" and it checks every letter to see if it's upper case or lower case.

So, Where-Object { $_.Name -clike "*Item*" } is a filter that takes the Name parameter of every object from the list (created by gci) and checks with -clike if any Name has the word "Item" in it.

As you can see in the image below, now we'll get only the files with upper case names in our result:

IMPORTANT:
-like alone means that we're looking for a certain pattern, no matter what case the letters are. The c in -clike means that we look for the thing with exactly the same capitalization of the letters (both upper and lower case, hence the "c").

If you want to see the files without the upper case first letter, you can do that by changing "*Item*" from our current command to "*item*":

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*item*" }

Let's try it out!

4.2. How to search while excluding a particular directory

In step 4.1 we learned how to search only for files/folders with specific case-sensitive names in them. After applying only two changes to our previous code, we can exclude certain directories from our search.

Here's our starting command once again:

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*Item*" }

Change #1

In the example above, -clike shows only files/folders including specific phrase in their names. If we change it to -cnotlike, we'll exclude from the search all files/folders with that specific phrase in their name.

Now our code looks like this:

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -cnotlike "*Item*" }

Change #2

After the first change, Where-Object { \(_.Name -cnotlike "*Item*" } only excludes the names, not full paths. In order to avoid that, we need to exclude an actual path to these files. We can do that by changing \)_.Name to $_.FullName, which checks for a certain phrase in the whole path to the file and in the file's name.

Now, your command should look like this:

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.FullName -cnotlike "*Item*" }

We excluded the "Items" folder from our search. You should now be able to see the files only from the "More items" directory. Try it out yourself!

What if you want to exclude the "More items" directory instead? Just change the phrase inside the filter to something like this:

Get-ChildItem -Recurse -Filter "*green*" -File |   
Where-Object { $_.FullName -cnotlike "*More*" }

We also changed the name of the file from "*item*" to "*green*" in our gci search (first line of code). That's why now we'll see only one bracelet in our result list:

The gci command has two filters applied. First, it searches for files with phrase "green" in their names. The second filter is the "Where-Object" command, which excludes anything that has the word "More" in its path. In our case, the "More items" folder got excluded.

We don't even need the case-sensitive filter in our case. The command will work the same when we exclude just a lowercase word "more". So let's change -cnotlike "*More*" to -notlike "*more*" and see if it's true:

Get-ChildItem -Recurse -Filter "*green*" -File |   
Where-Object { $_.FullName -notlike "*more*" }

As you can see, the result is the same! Despite different cases of the letters, we still got the right keyword. So, case-sensitive search isn't always needed – only when you want to be very specific.

Sometimes, being too specific might be bad and make your code not work as intended. To see what I mean, let's look at the example below. Let's apply case-sensitive search once again, but to our unchanged, lowercase keyword "more" and see if it still works:

Get-ChildItem -Recurse -Filter "*green*" -File |   
Where-Object { $_.FullName -cnotlike "*more*" }

Case-sensitive search doesn't filter out anything now, because it's too specific. Both the "Items" and "More items" folders omit the filter now.

FAQ:

If the Where-Object command is what actually filters the output for us, shouldn't we drop (delete) the -Filter option from gci?

No, we should still use the -Filter option, because it already separates around 99% of the possible files, so the Where-Object command has to work roughly only on 1% of the objects. It makes this part of the command AT LEAST 100 times faster (more often 100,000 times or even faster).

You can try using this command in -Path C:/ with and without the -Filter option. In my case, using the -Filter shortened the time needed for the whole sequence of commands to finish from 16 seconds to 8 seconds (first 7.99 seconds is used by gci, so that's why the time got shortened only by a half). That's what we call ✨optimization✨ :D

4.3 Searching only 1 directory from many with exactly the same name

We've learned how to search for a phrase anywhere inside the path of a file. But what if we want to search inside exactly the "More items" folder? For that, we'll use the -match filter (which works similarly to the -like filter).

Our phrase will also use "\", instead of "\". This is because "\" is the symbol for a folder, but alone in programming it also has some other features, which we don't want.

This command will look for a match for the "More items" folder in the path of every file from the list. Then, it will show you this file if it matches.

What if we want to check for two folders, one next to the other, simultaneously? Very easy! Just connect them with the sign for a folder "\". Here, the command will search inside the "More items" folder only if it's inside the "Documents" folder:

As you can see, we didn't use "More items", only "More". You can shorten that filter how you want. It will still be applied to the whole path. See the example below:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.FullName -match "s\\Mo*" }

Earlier, we used the not statement in -like filter to exclude certain files and directories. The same can be done with -notmatch:

Get-ChildItem -Recurse -Filter "*green*" -File | 
Where-Object { $_.FullName -notmatch "ents\\Ite*" }

Be aware that we're now excluding the "Items" folder from the search, not "More items".

And, with -cmatch we can apply the same case-sensitive filter as with -clike:

Get-ChildItem -Recurse -Filter "*green*" -File | 
Where-Object { $_.FullName -cmatch "green*" }

I hope you get the gist of it now.

4.4 Filter how deep (how many folders in) you want to search for the file

Sometimes you might have a very long path to some of your files. If you don't want to waste time searching every folder on your computer recursively, you can use -Depth option. It specifies how many folders to search inside your folder tree. I already showed you the picture of a folder tree in the beginning of this article, but you should take a look at it here once again.

So, how does the -Depth parameter work?

-Depth 0 means that our command will search only the current folder. It will show results of all children of Depth level 0. Those results are:
1 "child file" and 2 "child folders".

-Depth 1 searches the current folder and its child-folders. It will show the results of all children of Depth level 1. Those results are:
1 "child file", 2 "child folders", 2 "grandchild files" and 1 "grandchild folder".

-Depth 2 searches the current folder and its child and grandchild folders. It will show results of all children of Depth level 2. Those results are:
1 "child file", 2 "child folders", 2 "grandchild files", 1 "grandchild folder" and 1 "great grandchild file".

Let's see the difference between these two commands:

Get-ChildItem -Recurse -Filter "*item*" -Depth 0

Get-ChildItem -Recurse -Filter "*item*" -Depth 1

The first command will show you only the files and folders inside our current directory.
The second command will also search for them inside every folder found inside the current folder.

For the sake of practice, let's combine it with Where-Object to find the green bracelet:

Get-ChildItem -Recurse -Filter "*item*" -Depth 1 | Where-Object { $_.name -clike"*green*" }

I hope that this example showed you how easy it is to use multiple options ( -Depth, -Recurse) and filters (-Filter, Where-Object).

5. How to Search Through Hidden Files

Some files are not that easily accessible to the user. You can see some of the hidden files and folders in Windows Explorer (here's how). But sometimes it's easier to find what you need if you see only those hidden files. That's possible with PowerShell.

The options we're going to use for that are:

• -Force: show files otherwise not accessible by the user, such as hidden files.

• -Hidden: show only those hidden files and directories.

This example will search for hidden files in our user's folder:

gci -Path ~\ -Force -Hidden

Everything here is usually invisible to the typical user. But not for you now :D

The interesting thing is that there are more files not available to the user than the available ones. If you're brave enough, you can see them yourself (Remember! Ctrl + C stops the command!):

gci -Path ~\ -Force -Hidden -Recurse

6. How can you know all the properties that you can use as a filter?

Up until now, we'vce used some common properties, like Name and Fullname. But there are many others that you might want to access, like CreationTime (date of creating the file) or LastWriteTime (date of last edit of the file).

In this section, I'll first show you how to see all the possible properties. After that, you'll learn how to retrieve only the property you want for scripting purposes.

Go through step 2 above if you haven't already, because we're going to use the same files that we created before.

Move to the Documents folder in PowerShell.

I hope that this script looks familiar to you now. It searches for files with "item" in their names and checks if these names contain the word "green" (all lowercase letters):

Get-ChildItem -Recurse -Filter "*item*" | 
Where-Object { $_.Name -clike "*green*" }

We know that only one file should appear (if you don't trust me, just see for yourself). So, we're going to see every possible property we can use by appending (adding at the end) this fragment of code:
| Select-Object -Property *

Select-Object (alias: select) is used for selecting different types of properties. By using an option -Property we tell it to show both values and names of all the properties.

For example:

Name of property: FullName
Value of property: ~\Documents\More items\item 1- green bracelet.txt

The asterisk * at the end tells this command to show these names and values for every property possible.

The final version of this command looks like this:

Get-ChildItem -Recurse -Filter "*item*" | 
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -Property *

Try finding the FullName property in there :D

This command showed us all possible properties that we can use for that 1 file that it found. If there were more files fitting the filter, then every single one of them would have a similar list of properties. But for different types of files you will get different results.

How to retrieve only 1 desired property

You've already learned how to check for all possible properties. So, how do we use any of them? Just put one of them instead an asterisk * at the end of the command, like we put CreationTime in here:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -Property CreationTime

You can use any other property for the sake of this exercise, like LastWriteTime:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -Property LastWriteTime

What if you want to retrieve only the value of the property without its name (because you already know its name and it also messes up your script)? You can retrieve just the value, by changing the -Property to -ExpandProperty:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -ExpandProperty LastWriteTime

See the result:

7. I don't know the file’s name, but I know what's inside it. How do I find the file by its content?

Sometimes it's easier to find a file by searching it by its content. Or perhaps you have lots of similar files and you'd like to check them quickly without opening and closing them. I'll show you some techniques that will let you achieve that in no time.

This command will search every file on your system for the specified word or phrase (in our case, the phrase is "match"):

Get-ChildItem -Path C:\ -Recurse -File | 
Select-String -Pattern 'match' -List

Here's what's happening:

• Get-ChildItem -Path C:\ -Recurse -File: as you already know, this part searches for every file on your computer.

• | – passes the list of files to the next command. So, the next command will search for a certain phrase only in the files listed by gci.

• Select-String – "String" is a common word in programming used to describe a word/phrase/some text. So, we select the phrase that we want to search for. That phrase is specified by the -Pattern parameter (in our case it's "match").

• -List tells the command to show only the first found match in every file (great if you want to just see the list of all found files).

Here's an example output of our command:

Of course, you have quite a lot of files, and some images may also appear in your search (like .svg files that are basically text files that tell the system how to draw an icon). So, it's always best to specify what type of file you're searching for. Let's look for the phrase "red" inside .svg files:

Get-ChildItem -Filter "*.svg" -Recurse | 
Select-String -Pattern 'red' -List

On the other hand, some text documents will never appear in your search (for example .doc and .docx documents are encoded in such a way that they're impossible to decode without Word).

But in regular text files, you can search for phrases with an emphasis on big and small letters with the -CaseSensitive option. Here, we're going to search for the phrase "github" with only lowercase letters:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List -CaseSensitive

Other options that you'll often use with the Select-String command are:

• Select-String -AllMatch will show you all matches found in every searched file (instead of only 1 match found per file, like with -List).
  Select-String -Context 3 shows the three lines of text before and after the line in which the match is found.
  Select-String -Raw won't show you the paths, just the content of the files. This is great for automation and scripts. It's often combined with the -Context option.

Let's see some of these options in action:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -AllMatch -Context 3

Thanks to the -Context parameter, you can see a total of seven lines (three lines before and three lines after the match) in this file, one after another. This makes it easier to differentiate it from all the other matches found by -AllMatch that might be put in a very similar context.

If you ever feel like there's too much clutter on your screen, you can combine Select-String with Select-Object to get only the paths of the files with matched phrases.

The command below will search every .txt file on your computer for the phrase specified:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List

Let's add the Select-Object -Property Path filter at the end. Now, the command will only show the paths, so there's less clutter on your screen:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List | 
Select-Object -Property Path

Some of the paths are not fully visible. We'll fix that in the next step.

8. I can't see the full path - how do I fix this?

Let's format the results with the Format-Table -Wrap -AutoSize command. -Autosize allows the result to take the whole available space. -Wrap allows wrapping (continuing the text in the next line when it doesn't fit in the space available), which creates more space if it's needed.

Here's an example:

Get-ChildItem -Path C:\ -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List | 
Select -Property Path | 
Format-Table -Wrap -AutoSize

Now, you can see the whole paths (or any other results you need) even in PowerShell!

9. Hard to read? Open the results in the text editor of your choice

You can send the results of any script/command in two ways:

> ~\Documents\command_output.txt
AND
| Out-File ~\Documents\command_output.txt

Both of these will create a file inside your Documents folder, which you can later open in any program of your choice and edit.

Just add whichever solution you prefer to the end of your command, like here:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'match' -List | 
Select -Property Path | 
Out-File ~\Documents\command_output.txt

In the image below, first you'll see the same command, but without exporting the results to another file. The second command, at the bottom of the image, will export the results to the other file without showing them in PowerShell:

You'll see the results from second command after opening the file in any text editor:

But, what if you can't see the full path even in your text editor?

To address this, you can add | Format-Table -Wrap -AutoSize right before sending the results to the file:

Get-ChildItem -Path C:\ -Filter "*.txt" -Recurse | 
Select-String -Pattern 'match' -List | 
Select -Property Path | 
Format-Table -Wrap -AutoSize |
Out-File ~\Documents\command_output.txt

And open the file to see the whole path!

Just remember that you have to copy each line one by one. Where you see the arrows in the screenshot above is a "newline" character, which you have to delete. Only after doing that can you copy the whole path and paste it into Windows Explorer or into some script.

10. Summary: the Ultimate Commands for Searching and Finding Whatever You Need

Here you can download a free cheat sheet with explanations of the commands and examples in one place.

Most used commands:

• Case-sensitive search:

Get-ChildItem -Path C:\ -Recurse -Filter "*whatYouNeed*" |   
Where-Object { $_.Name -clike "*whatYouNeed*" } |   
Select-Object { $_.FullName } |
Format-Table -Wrap -AutoSize

• Alternatively, send the result to a file:

Get-ChildItem -Path C:\ -Recurse -Filter "*whatYouNeed*" |   
Where-Object { $_.Name -clike "*whatYouNeed*" } |   
Select-Object { $_.FullName } |
Format-Table -Wrap -AutoSize |
Out-File ~\Documents\command_output.txt

• Search by file's content:

Get-ChildItem -Path C:\ -Recurse | 
Select-String -Pattern 'what you remember' -AllMatch -Context 2 |
Format-Table -Wrap -AutoSize

• Alternatively, send the result to the file:

Get-ChildItem -Path C:\ -Recurse | 
Select-String -Pattern 'what you remember' -CaseSensitive -AllMatch -Context 2 |
Format-Table -Wrap -AutoSize |
Out-File ~\Documents\command_output.txt

These commands should work for anything you want to find. I hope you understand now how they function after reading through this tutorial ;)

Wrapping Up

If you want to learn more about these commands, I show you how to work with them in depth in my tutorial “Learn PowerShell commands like a Linux user”.

If what you found here helped you in any way, consider following me on my social media in order to help me reach further audience: Mastodon, LinkedIn.

You can also rate me on Github and support me on Ko-fi!

Thank you for any support you're able to give. Have a great day!

Nmap allows network admins to identify devices running on their network, discover open ports and services, and detect vulnerabilities.

Here is the basic syntax to use nmap:

nmap <ip/url>

Let’s do a quick scan and see what we can find. We can use the URL scanme.nmap.org to try out a scan. Nmap allows us to use this server to practice scans.

As you can see, we have found some open ports and services. These act as entry points for further analysis or exploitation.

Nmap is usually the first tool that ethical hackers learn. Here is a full tutorial if you want to learn more about Nmap.

Nmap Scripting Engine

A key feature is the Nmap Scripting Engine (NSE). It lets users run scripts to do detailed network scans and gather specific information.

Scripts help you perform a list of actions automatically instead of performing them step by step.

These scripts cover a range of functionalities, from service detection to vulnerability scanning. In this article, we’ll look at a few useful Nmap scripts.

I’ll walk you through each script, explain what it does, and show you how to use it. By the end, you’ll have a solid understanding of how to use these scripts as an ethical hacker.

Note: This tutorial is to help you understand network security. Hacking or even scanning another server without permission is illegal.

HTTP-Enum

Imagine you’re tasked with checking a website’s security and want to see if there are any hidden pages or directories. You suspect there might be admin panels, login pages, or test files that aren’t linked on the main site.

Finding these hidden areas could reveal critical security weaknesses, such as unprotected admin pages or old files that might still hold sensitive information.

The http-enum script is used to scan a web server and find common directories and files that might be hidden from the main site navigation.

Think of it like opening doors in a building to see what’s behind each one. It searches for paths like login pages, admin panels, config files, and other directories that aren’t typically linked on the main website.

For example, a login page or an admin section may exist at specific paths but aren’t visible to regular users. This information is useful because knowing these locations can help you identify security weak points.

Here is the command to run the http-enum script:

nmap - script http-enum -p 80 <target-ip>

As you can see, the above sample result shows /login.php, /docs and other exposed URL paths. These can be entry points to find restricted information in a web server.

SMB-OS-Discovery

Suppose you’re exploring a company’s network to understand what kind of systems they have in place, specifically in a Windows environment.

Knowing the exact operating system and version of each server helps you assess vulnerabilities. For example, an older version of Windows might have unpatched flaws that need attention.

The smb-os-discovery script targets servers that use the SMB protocol, mainly found in Windows environments, to gather information about the server’s operating system. It can reveal details like the Windows version, the server name, and its domain.

This script helps you understand what type of system you’re dealing with, which is key for checking security flaws specific to that OS.

Here is the syntax to run the smb-os-discovery script.

nmap - script smb-os-discovery -p 445 <target-ip>

As you can see in the above sample result, the script connects to the SMB service on the target and retrieves OS information. This can help you quickly identify the Windows version and other details about the server.

HTTP-Headers

Imagine you’re evaluating a website’s configuration and security settings. You want to know what kind of server it’s running, what methods are allowed, and if it’s enforcing HTTPS connections.

These details give you insights into whether the server’s configuration aligns with best practices, helping you spot any missing security settings.

The http-headers script checks the headers sent by a web server when a user connects to it it. Headers tell you the server type (like Apache or NGINX), security settings (like HTTPS requirements), allowed methods, and caching rules.

These details are like the server’s blueprint for communication, often revealing if the server has certain protections enabled.

Here is the syntax to run the http-headers script:

nmap - script http-headers -p 80 <target-ip>

You can see that the sample response shows headers like like X-Powered-By, Set-Cookie, and so on. These headers can help to find security issues such as cross-site scripting (XSS) and clickjacking.

SSH-Brute

Let’s say you’re testing a server’s defenses against unauthorized access through SSH. You know that weak passwords are a common risk, so you need a way to check if any accounts have easily guessable credentials.

This test will help you identify weak SSH logins that need stronger passwords to protect the server.

The ssh-brute script tries to log into an SSH server by guessing usernames and passwords. SSH, or Secure Shell, is often used for remote logins.

If the usernames and passwords are easy to guess, this script might find a way in. It’s a useful test to see if login credentials are strong enough to prevent unauthorized access.

Here is the syntax to run the ssh-brute script:

nmap - script ssh-brute -p 22 <target-ip>

As you can see, this script tries different username-password combinations on the SSH server. If successful, it will display the correct credentials.

DNS-Brute

Imagine you’re mapping out a company’s network and want to see if they have any subdomains that aren’t publicly listed. Each subdomain might serve a different purpose, such as hosting email servers or internal testing sites.

Discovering these subdomains helps you check if any of them are exposing sensitive services.

The dns-brute script helps you find subdomains associated with a given domain by trying out common names, like “www,” “mail,” or “ftp.” Subdomains can host separate services and applications, each with its own set of vulnerabilities.

Here is the syntax to run the dns-brute script:

nmap - script dns-brute <target-domain>

As you can see, the script attempts to resolve a list of common subdomains, and finds one internal hostname. Using this script can reveal subdomains that aren’t listed in public records, helping you to gain a fuller picture of an organization’s network layout.

Conclusion

These Nmap scripts provide a powerful way to audit, troubleshoot, and secure networks. By understanding what each script does and how to use it, you’ll be able to uncover hidden issues and safeguard your infrastructure.

To learn how to build a career in Cybersecurity, check out The Hacker’s Handbook. To practice hacking real systems and get help from other hackers, join The Hacker’s Hub.

Tampermonkey is a browser extension that lets you add custom scripts to websites, making them work or look the way you want. It's like giving websites a makeover or adding new features.

These scripts are called userscripts and you can make tampermonkey run those scripts when you visit a particular site.

How to Use Tampermonkey to Customize the User Experience of a Website

There are a lot of use cases for Tampermonkey. The most obvious one is adding your own custom styling. You can add custom css for a specific site and change up the appearance as you want. But as you can run a script, you can also manipulate the DOM elements.

I will list out of the things I have done to give you some ideas to what is possible.

1. Increase readability by changing font properties.
2. Remove ads on sites that don't allow an ad blocker.
3. Declutter a site so that you can focus on the portion that you're interested in.
4. Add keyboard shortcuts for repetitive tasks.
5. Add buttons for custom actions.
6. Automatically fill out form data.

Today you'll get a glimpse of what you can do with Tampermonkey by writing scripts that'll work on the freeCodeCamp /news site.

First we will see how to declutter for a more focused reading experience. Then we will attach copy buttons on the code snippets. And lastly, we'll automatically generate table of contents that you can access with a toggle button.

The source code is available on GitHub.

It's worth pointing out that any changes your script makes will only be available in your browser. So as long as you don't deal with any sensetive data on that site, you can go as wild as you want.

But be aware that some sites may have some policy in regards to using third party JavaScript and take disciplinary action if you violate that policy.

Prerequisites

As we will be modifying a website, basic knowledge of HTML, CSS, and JavaScript is required to go through this tutorial. Some experience with DOM manipulation would be great as well.

If you can effectively manipulate exisiting DOM elements, that will allow you to write Tampermonkey scripts to make yourself more productive on any website.

Table of Contents

• How to Install Tampermonkey
• How to Create a New Userscript
• How to Change a Button's Behaviour
• How to Implement Reader View for freeCodeCamp /news Articles
• How to Add Copy Buttons to the Code Snippets
• How to Auto-generate a Table of Contents
• Key Takeways

How to Install Tampermonkey

For Chrome, the extension is available on the chrome web store.

It's also available for Firefox, which you can install from Firefox Add-Ons.

For other browsers, you can visit the Tampermonkey Home Page. Currently, Chrome, Firefox, Edge, Safari and Opera are officially supported. But the one from Chrome web store works fine on Chromium-based browsers like Brave.

How to Create a New Userscript

The easiest way to get started is using the create new script option from the toolbar.

You will be presented with something like this:

// ==UserScript==
// @name         New Userscript
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://www.freecodecamp.org/news/*
// @icon         https://www.google.com/s2/favicons?sz=64&domain=freecodecamp.org
// @grant        none
// ==/UserScript==

(function() {
    'use strict';

    // Your code here...
})();

The userscript header contains information about the script and other important parameters as well.

For now the relevant part is @match. This tells Tampermonkey for which sites to run the script. I have changed to match any article on the freeCodeCamp news site by using the "*" wildcard.

Test it out by putting something simple as alert("HI") on the function. Then navigate to any article on freecodecamp.org/news.

Lets do something interesting next. This will be a brief introduction to DOM manipulation.

Before writing code in the userscript, it's better to write out your code in the dev console first. Then, when you have working code, you can just paste it in the Tampermonkey script. This is how we will write all the scripts in this article as well.

How to Change a Button's Behaviour

At the end of each freeCodeCamp article there is a button that says "Tweet a thanks". You can see the onclick function of the button by using the element picker from the dev tools and clicking on that button.

Clicking on that button will open a new window in Twitter with some default text for a tweet like the following:

Thank you @twitter-username-of-author for writing this helpful article.

Title of Article
https://www.freecodecamp.org/news/slug-of-article

Let's say that you want to change up the default text to something like this:

This article is quite fascinating.

Title of Article by @twitter-username-of-author
https://www.freecodecamp.org/news/slug-of-article

The new text is static as you will chose it, but there are some variables here. You need to extract these 3 things:

1. The link of the article.
2. The Twitter username of the author.
3. Title of the article.

Extracting the link is as location.href.

Now how to extract the Twitter username?

At first glance, you may think the name after the author image is the answer. But that's actually the author's name (which is not necessarily unique) – it's not their Twitter username. So, where would you start looking for it?

Since you can just click the button or inspect it, you already know the values of all these variables. So a good way to find the Twitter username would be just to search for it in the entire HTML document. As it is unique, there won't be many occurrences.

Open the dev tools and search for the Twitter username. You will find this tag:

<meta name="twitter:creator" content="@username" />

Which is not surprising, as most publishing sites put information such article title, tags, description, and author information in the meta tags for SEO purposes. But most of the time you can just find what you are looking for by using the inspection tool.

So how do we extract the "content" property of this meta tag? Considering that an article is written by only one author, there will always be only one meta tag with name "twitter:creator". So you can just use the querySelector.

const username = document.querySelector('meta[name="twitter:creator"]').content

Now on to extracting the title. If you have wandered around looking for the Twitter username, you will find that there is also a meta tag with name "twitter:title". But this time, the title is something that you can visually see and inspect. In most cases that's the easier way to go about it.

If you inspect on the article title above the cover image, you will see it's inside an h1 with the class "post-full-title". You can select this with the following:

const title = document.querySelector("h1.post-full-title").textContent

Now we have all the pieces we need to make the change. Inspect the target button and you will see it comes with an id of "tweet-btn". Now let's define our text.

const text = `This article is quite fascinating.

${title} by ${username}
${location.href}
`

Putting it all together by changing the button's onclick, here's what you should have:

function change_tweet_text() {
    document.getElementById("tweet-btn").onclick = () => {
        const username = document.querySelector(
            'meta[name="twitter:creator"]'
        ).content
        const title = document.querySelector("h1.post-full-title").textContent

        const text = `This article is quite fascinating.

${title} by ${username}
${location.href}
`

        const share_link = encodeURI(
            `https://twitter.com/intent/tweet?text=${text}`
        )

        window.open(share_link, "share-twitter", "width=550, height=235")

        return false
    }
}

Because the text contains newline characters, we have to convert it url-encoded format.

Paste the function inside the userscript and call it.

;(function () {
    change_tweet_text()
})()

Now reload the article page and click on the button to see if it's working as expected.

It is possible that it won't work as expected. Especially if the page takes a lot of time to load.

With this script, we are modifying a button which is a DOM element. But what if the element is not yet loaded? Then you will face unexpected results.

Tampermonkey provides the "// @run-at" header which you can specify to "document-end", but I've found that it still produces unexpected results sometimes.

The right way to mitigate it is to use the "load" event which is emitted when the whole page is loaded. So, we can refactor our driver function in the following way:

addEventListener("load", main)

function main() {
    change_tweet_text()
}

You can be extra careful and throw a delay as well:

addEventListener("load", setTimeout(main, 2000))

From now on, we will call all the functions in this main function which will run after the load event is fired.

Note that you could have achieved the same result using a regular expression. In that case, you could extract the variables from the button's onclick which contains the URL with the full text.

But extracting information from the tags is better because if one day the default tweet text changes, you would need the change the regex as well.

How to Implement Reader View for freeCodeCamp /news Articles

If you make your browser fullscreen, the top navigation bar still stays at the top – which is not the best reading experience. So, we want to hide everything except the main article body.

Let's start inspecting. You will see that, close to the "main" tag, there is an "article" tag with class "post". It contains all the text of the article including the heading and cover image.

This time, we will live-edit the CSS of the page. In Firefox, you can use the "Style Editor" tab in the dev tools. Click on the "+" icon and start testing out your CSS.

As of now, there is no built-in way to hide all elements except one (in our case article.post) using CSS. You might be tempted to use :not(article.post) but it won't work because if any ancestor of the element is hidden, then all the descendents will be hidden as well.

We can accomplish our goal by using the ":has" selector. :has(article.post) will select all the ancestors of article.post so we can select the inverse of it with :not(:has(article.post)).

But there's still a problem: the descendent of article.post is also ignored by this selector. We can bring them back by chaining another not –:not(article.post, article.post *). This will select everything that is not an article.post or a descendent of it.

This is the final CSS:

:not(:has(article.post)):not(article.post, article.post *) {
    display: none;
}

At the time of writing this, the ":has" selector is experimental in Firefox. But you can go to about:config and enable it by changing the value of layout.css.has-selector.enabled to true.

What if you want to use the navigation bar? It would be a shame to manually write the CSS and remove it when you don't want it, right? Especially now that you can do some scripting.

We will add a keyboard shortcut to the site that will make the browser fullscreen. We'll also apply that CSS and, upon exiting fullscreen, the CSS will be removed.

One of the exposed APIs of tampermonkey is GM_addStyle(css) for adding CSS. For this API to work, you need to include // @grant GM_addStyle to the userscript header and also remove // @grant none if you have that.

To create the shortcut, you will use a "keydown" event listener.

function add_declutter_toggle(key) {
    const css = `
        .declutter :not(:has(article.post)):not(article.post, article.post *) {
            display: none;
        }`

    GM_addStyle(css)

    document.addEventListener("keydown", (event) => {
        if (event.key === key) {
            if (document.fullscreen) {
                document.exitFullscreen()
                document.body.classList.remove("declutter")
            } else {
                document.body.classList.add("declutter")
                document.documentElement.requestFullscreen()
            }
        }
    })
}

We are applying the CSS to the elements with the class "declutter". By toggling the "declutter" on the body we are essentially toggling the CSS on the entire page.

Now, we just have to add the function call in our main function.

function main() {
    change_tweet_text()
    add_declutter_toggle("F")
}

Reload the page and you will see that pressing "F" will toggle fullscreen and also remove everything except article body when in the fullscreen state.

How to Add Copy Buttons to the Code Snippets

The first step is always identifying the correct selector. If you inspect any code snippet you will see that they're contained in <code> blocks and have a <pre> as a parent. So, to select all code snippets, the selector can be pre > code.

Now the question is where to place the button? For easy access, we want the button to be on top of the code snippet and aligned to the right. You may think, placing it between the <pre> tag and <code> tag could work – but that will interfere with the content of the code snippet which is not ideal.

The ideal solution is wrapping the <pre> tag with a div and creating the structure as div > button > pre > code.

The copying is done by the clipboard API using the navigator.clipboard.writeText method. We will also apply some styling and let the reader know their copying was successful by changing the button text from "copy" to "copied" for a small duration of time.

function attach_code_copy_btn() {
    const css = `
        .copy-btn {
            float: right;
            margin-bottom: 5px;
            border-radius: 1rem;
            font-size: 0.8em;
            width: 7rem;
        }

        .pre-wrapper {
            width: 100%;
        }
    `
    GM_addStyle(css)

    const codes = document.querySelectorAll("pre > code")
    codes.forEach((code) => {
        const pre = code.parentElement
        const btn = document.createElement("button")
        btn.textContent = "copy"
        btn.classList.add("copy-btn")
        btn.addEventListener("click", () =>
            navigator.clipboard.writeText(code.textContent).then(() => {
                btn.textContent = "copied"
                setTimeout(() => (btn.textContent = "copy"), 2500)
            })
        )
        const div = document.createElement("div")
        div.classList.add("pre-wrapper")
        div.appendChild(btn)
        wrap(pre, div)
    })
}

function wrap(elem, wrapper) {
    elem.parentNode.insertBefore(wrapper, elem)
    wrapper.appendChild(elem)
}

The attach_code_copy_btn function finds all code snippets and wraps them around a div, adds classes to them, and attaches a button which will copy the code content on click.

Call the function just as before:

function main() {
    change_tweet_text()
    add_declutter_toggle("F")
    attach_code_copy_btn()
}

Heres some practice, try adding click event listeners to all <code> tags that are not inside a <pre> tag. On a mouse click on the <code> block the contents of the block will be copied.

How to Auto-generate a Table of Contents

freeCodeCamp is a great place for reading in-depth articles. The articles are well organized as well. Most authors use proper headings and provide a table of contents. But the ToC is not accessible on every part of the page.

If you want to jump somewhere you will have to go back to the ToC then click on your section of interest. And then there are shorter articles that don't provide a ToC at all.

That's where we can introduce a userscript that will automatically generate a table of contents which is accessible anywhere on the page. This improves the user experience when reading longer articles.

The first thing you should know is how a ToC works. A section in the ToC is just an anchor tag with an href value of #some-section. Here "some-section" can be the name property of an anchor tag of the target location (for example, <a name="some-section">) or if its used as an id on any element (for example, <h2 id="some-section">).

If you inspect on any heading tag, you will see that the id attribute is present. This is automatically inserted when the article page is built. So we can use that for our ToC.

This is a ToC generator function modified from a StackOverflow answer.

function generate_toc() {
    let toc = ""
    let level = 1

    let container = document.querySelector(".post-content")
    const regex = /<h([2-5])\s+id="([^"]+)">([^<]+)<\/h([2-5])>/gi
    const matches = [...container.innerHTML.matchAll(regex)]
    matches.forEach((match) => {
        if (match.length != 5) return
        const [_, openLevel, id, titleText, closeLevel] = match

        if (openLevel !== closeLevel) {
            return
        }

        if (openLevel > level) {
            toc += new Array(openLevel - level + 1).join("<ol>")
        } else if (openLevel < level) {
            toc += new Array(level - openLevel + 1).join("</li></ol>")
        } else {
            toc += new Array(level + 1).join("</li>")
        }

        level = parseInt(openLevel)

        if (!id) {
            id = titleText.replace(/ /g, "_")
        }
        toc += '<li><a href="#' + id + '">' + titleText + "</a></li>"
    })

    if (level) {
        toc += new Array(level + 1).join("</ol>")
    }

    return toc
}

We are using a regex to match h2, h3, h4, and h5 tags and creating an ordered list to display to ToC contents and returning the HTML of the whole ToC.

Now we will add this HTML to a div that will contain the ToC. That div will be wrapped inside a dialog component so that we can use a modal to easily show/hide the ToC. We will also add a button that will be used as the toggle.

function show_toc() {
    document.documentElement.style.overflow = "hidden"
    document.getElementById("toc-dialog").showModal()
}

function add_toc_toggle() {
    const dialog = document.createElement("dialog")
    dialog.setAttribute("id", "toc-dialog")
    const toc_div = document.createElement("div")
    toc_div.classList.add("toc-content")
    toc_div.innerHTML = "<h2>Table of Contents</h2>"
    dialog.appendChild(toc_div)

    const show_toc_btn = document.createElement("button")
    show_toc_btn.setAttribute("id", "toc-toggle")
    show_toc_btn.innerHTML =
        '<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M.361 256C.361 397 114 511 255 511C397 511 511 397 511 256C511 116 397 2.05 255 2.05C114 2.05 .361 116 .361 256zM192 150V363H149V150H192zM234 150H362V193H234V150zM362 235V278H234V235H362zM234 320H362V363H234V320z"></path></svg>'

    show_toc_btn.addEventListener("click", () => {
        show_toc()
    })

    // Click anywhere to close
    dialog.addEventListener("click", (event) => {
        document.documentElement.style.overflow = "auto"
        event.currentTarget.close()
    })

    dialog.addEventListener("close", (event) => {
        document.documentElement.style.overflow = "auto"
        event.currentTarget.close()
    })

    document.querySelector(".post-content").appendChild(dialog)
    document.querySelector(".post-content").appendChild(show_toc_btn)
}

Finally we will provide a driver function that will perform all the necessary tasks related to ToC generation. We will also add a keyboard shortcut to toggle the ToC.

function add_toc(toc_toggle_key) {
    const css = `
        .toc-content {
            margin: 0;
            padding: 0;
            max-width: 65vw;
        }

        @media (width <= 800px) {
            .toc-content {
                max-width: 100vw;
            }
        }

        .toc-content h2 {
            padding: 20px 30px;
        }

        .toc-content ol {
            counter-reset: item;
        }

        .toc-content li {
            display: flex;
            white-space: nowrap;
            gap: 5px;
        }

        .toc-content li:before {
            content: counters(item, ".") " ";
            counter-increment: item;
        }

        .toc-content a {
            text-decoration: none;
            white-space: normal;
            flex-grow: 1;
        }

        .toc-content li:hover {
            background: rgba(82, 142, 227, 0.3);
            cursor: pointer;
        }

#toc-toggle {
            background: transparent;
            position: fixed;
            bottom: 5%;
            right: 5%;
            width: 5rem;
        }

#toc-toggle svg {
            fill: rgb(82, 142, 227);
            scale: 1;
        }

#toc-toggle svg:hover {
            scale: 1.15;
        }

#toc-dialog {
            padding: 0;
            border: 0;
        }
`
    GM_addStyle(css)

    add_toc_toggle()
    document.querySelector(".toc-content").innerHTML += generate_toc()

    if (toc_toggle_key) {
        document.addEventListener("keydown", (event) => {
            if (event.key === toc_toggle_key) {
                show_toc()
            }
        })
    }
}

Now, just call the driver function in main.

function main() {
    change_tweet_text()
    add_declutter_toggle("F")
    attach_code_copy_btn()
    add_toc(";")
}

Now whenever, you read an article on freeCodeCamp, you can use the toggle button or a shortcut to view the ToC.

For practice, try making the toc sticky in the side (left/right). If you are on a wider screen, having the toc on the side will make the reading experience even better.

Key Takeways

You should never run arbitrary JavaScript on any occasion. That goes for Tampermonkey scripts as well.

I recommend going through every line of code before using a userscript. As you can already imagine, a userscript can drastically alter a website's appearance. If you are not careful enough, you might click a button that you expect to do one thing but it does something completely different, so always err on the side of caution.

When writing userscripts, keep these things in mind:

1. Always test your code in the dev console first and take care of all edge cases.
2. Prefer creating/manipulating DOM elements using built-in functions instead of modifying the innerHTML.
3. If you find any unexpected behaviour, try throwing a long delay and checking if that resolves the issue.

You can read my other articles on my blog. Find me on Twitter LinkedIn.

Regular expressions, also known as regex, work by defining patterns that you can use to search for certain characters or words inside strings.

Once you define the pattern you want to use, you can make edits, delete certain characters or words, substitute one thing for another, extract relevant information from a file or any string that contains that particular pattern, and so on.

Why Should You Learn Regex?

Regex let you to do text processing in a way that can save you a lot of time. It can also introduce some fun in the process.

Using regex can make locating information much easier. Once you find your target, you can batch edit/replate/delete or whatever processing you need to do.

Some practical examples of using regex are batch file renaming, parsing logs, validating forms, making mass edits in a codebase, and recursive search.

In this tutorial, we're going to cover regex basics with the help of this site. Later on, I will introduce some regex challenges that you'll solve using Python. I'll also show you how to use tools like sed and grep with regex.

Like many things in life, regular expressions are one of those things that you can only truly understand by doing. I encourage you to play around with regex as you are going through this article.

Table of Contents

• Regex Basics
  • Exact match
  • Character set
    • Match ranges in regex
    • Match any character not in the set
  • Character classes
  • Quantifiers
  • Capture groups
    • How to use logical OR in regex
    • How to reference capture groups
    • How to name capture groups
• How to use regex with command line tools
  • Recursive regex search with grep
  • Substitution with sed
• Advanced Regex: Lookarounds
  • Lookbehinds
  • Lookaheads
• Practical Examples of Regex
  • Logs parsing
  • Bulk File Renaming
  • Email validation
  • Password constraints
• Final words

Regex Basics

A regular expression is nothing but a sequence of characters that match a pattern. Besides using literal characters (like 'abc'), there are some meta characters (*,+,? and so on) which have special purposes. There are also features like character classes which can help you simplify your regular expressions.

Before writing any regex, you'll need to learn about all the basic cases and edge cases for the pattern you are looking for.

For instance, if you want to match 'Hello World', do you want the line to start with 'Hello' or can it start with anything? Do you want exactly one space between 'Hello' and 'World' or there can be more? Can other characters come after 'World' or should the line end there? Do you care about case sensitivity? And so on.

These are the kind of questions you must have the answer to before you sit down to write your regex.

Exact match

The most basic form of regex involves matching a sequence of characters in a similar way as you can do with Ctrl-F in a text editor.

On the top you can see the number of matches, and on the bottom an explanation is provided for what the regex matches character by character.

Character set

Regex character sets allow you to match any one character from a group of characters. The group is surrounded by square brackets [].

For example, t[ah]i matches "tai" and "thi". Here 't' and 'i' are fixed but between them can occur 'a' or 'h'.

Match ranges in regex

Sometimes you may want to match a group of characters which are sequential in nature, such as any uppercase English letter. But writing all 26 letters would be quite tedious.

Regex solves this issue with ranges. The "-" acts as a range operator. Some valid ranges are shown below:

You can also specify partial ranges, such as [b-e] to match any of the letters 'bcde' or [3-6] to match any of the numbers '3456'.

You are not limited to specifying only one range inside a character set. You can use multiple ranges and also combine them with any other additional character(s). Here, [3-6u-w;] will match any of '3456uvw' or semicolon ';'.

Match any character not in the set

If you prefix the set with a '^', the inverse operation will be performed. For example, [^A-Z0-9] will match anything except uppercase letters and digits.

Character classes

While writing regex, you'll need to match certain groups such as digits quite often and multiple times in the same expression as well.

So for example, how would you match a pattern like 'letter-digit-letter-digit'?

With what you've learned up until now, you can come up with [a-zA-Z]-[0-9]-[a-zA-z]-[0-9]. This works, but you can see how the expression can get quite messy as the pattern length gets bigger.

To make the expression simpler, classes have been assigned to well-defined character groups such as digits. The following table shows these classes and their equivalent expression with character sets:

Character classes are quite handy and make your expressions much cleaner. We will use them extensively throughout this tutorial, so you can use this table as a reference point and come back here if you forget any of the classes.

Most of the time, we won't care about all the positions in a pattern. The "." class saves us from writing all possible characters in a set.

For example, t.. matches anything that starts with t and any two characters afterwards. This may remind you of the SQL LIKE operator which would use t%% to accomplish the same thing.

Quantifiers

The word "pattern" and "repetition" go hand in hand. If you want to match a 3 digit number you can use \d\d\d. But what if you need to match 11 digits? You could write '\d' 11 times, but a general rule of thumb while writing regex or just doing any kind of programming is that if you find yourself repeating something more than twice, you are probably unaware of some feature.

In regex, you can use quantifiers for this purpose. To match 11 digits, you can simply write the expression \d{11}.

The table below lists the quantifiers you can use in regex:

In this example, the expression can\s+write matches can followed by 1 or more whitespaces followed by write. But you can see 'canwrite' is not matched as \s+ means at least one whitespace needs to be matched. This is useful when you are searching through text which is not trimmed.

Can you guess what can\s?write will match?

Capture groups

Capture groups are sub-expressions enclosed in parentheses (). You can have any number of capture groups, and even nested capture groups.

The expression (The ){2} matches 'The ' twice. But without a capture group, the expression The {2} would match 'The' followed by 2 spaces, as the quantifier will be applied on the space character and not on 'The ' as a group.

You can match any pattern inside capture groups as you would with any valid regex. Here (is\s+){2} matches if it finds 'is' followed by 1 or more spaces twice.

How to use logical OR in regex

You can use "|" to match multiple patterns. This is (good|bad|sweet) matches 'This is ' followed by any of 'good' or 'bad' or 'sweet'.

Again, you must understand the importance of capture groups here. Think about what the expression This is good|bad|sweet would match?

With a capture group, good|bad|sweet is isolated from This is. But if it's not inside a capture group, the entire regex is only one group. So the expression This is good|bad|sweet will match if the string contains 'This is good' or 'bad' or 'sweet'.

How to reference capture groups

Capture groups can be referenced in the same expression or while performing replacements as you can see on the Replacement tab.

Most tools and languages allow you to reference the nth captured group with '\n'. In this site '$n' is used while referencing on replacement. The syntax for replacement will vary depending on the tools or language you're using. For JavaScript, for example, its '$n', while for Python its '\n'.

In the expression (This) is \1 power, 'This' is captured and then referenced with '\1', effectively matching This is This power.

How to name capture groups

You can name your capture groups with the syntax (?<name>pattern) and backreference them in the same expression with \k<name>.

On replacement, referencing is done by $<name>. This is the syntax for JavaScript and can vary among languages. You can learn about the differences here. Also note that this feature might not be available in some languages.

In the expression (?<lang>[\w+]+) is the best but \k<lang> .*, the pattern [\w+]+ is captured with the name 'lang' and backreferenced with \k<lang>. This pattern will match any word character or '+' character 1 or more times. The .* at the end of the regex matches any character 0 or more times. And finally on replacement, the referencing is done by $<lang>.

How to Use Regex with Command Line Tools

There are good CLI tools available that let you perform regex from your terminal. These tools save you even more time as you can easily test different regex without writing code in some langauge and then compiling or interpreting it.

Some of the well-known tools are grep, sed, and awk. Let's look at a few examples to give you some ideas on how you can leverage these tools.

Recursive regex search with grep

You can execute the power of regex through grep. Grep can search patterns in a file or perform recursive search.

If you are on Windows, you can install grep using winget. Run this command in powershell:

winget install -e --id GnuWin32.Grep

I will show you the solution to a challenge I created for a CTF competition at my university.

The file attached to the challenge is a zip file that contains multiple levels of directories and a lot of files in it. The name of the competition was Coderush with flag format coderush{flag is here}. So you have to search for the pattern coderush{.*} which will match the flag format coderush{any character here}.

Unzip the file with unzip ripG.zip and cd into it with cd ripG.

There are 358 directories and 8731 files. Instead of searching the pattern in the files one by one, you can employ grep like this:

grep --color -R "coderush{.*}"

The "-R" flag enables recursive search.

You can learn more about grep and its command line options here

Substitution with sed

You can use sed to perform insertion, deletion, substitution on text files by specifying a regex. If you are on windows, you can get sed from here. Or if you use WSL, tools like grep and sed will already be available.

This is the most common usage of sed:

sed 's/pattern/replacement/g' filename
echo "${text}" | sed 's/pattern/replacement/g'

Here, the option "g" is specified to replace all occurrences.

Some other useful options are -n to suppress the default behaviour of printing all lines and using p instead of g to print only the lines which are affected by the regex.

Let's take a look at the content of texts.txt.

Hello rand chars World 56 rand chars
Henlo 52 rand chars W0rld rand chars
GREP rand chars Henlo 62 rand chars
Henlo 10 rand chars Henlo rand chars
GREP rand chars Henlo 45 rand chars

Our task is replacing Henlo number with Hello number only in the lines where "GREP" is present. So, we are searching for the pattern Henlo ([0-9]+) which will match 'Henlo ' followed by 1 or more digits and all the digits are captured. Then our replacement string will be Hello \1 – the '\1' is referencing the capture group containing the digits.

One way to accomplish that would be using grep to grep the lines which have "GREP" present then perform the replacement with sed.

grep "GREP" texts.txt | sed -En 's/Henlo ([0-9]+)/Hello \1/p'

The "-E" option enables extended regex without which you would need to escape the parentheses.

Or you could just use sed. Use /pattern/ to restrict substitution on only the lines where pattern is present.

sed -En '/GREP/ s/Henlo ([0-9]+)/Hello \1/p' texts.txt

Advanced Regex: Lookarounds

Lookaheads and Lookbehinds (together known as lookarounds) are features of regex that allow you to check the existence of a pattern without including it in the match.

You can think of them as zero width assertions – they assert the existence of a pattern but do not consume any characters in the match. These are very powerful features, but they're also computationally expensive. So make sure you keep an eye on performance if you are using them often.

Lookbehinds

Let's say you want to match the word 'linux', but you have 2 conditions.

1. The word 'GNU' must occur before 'linux' occurs. If a line contains 'linux' but doesn't have 'GNU' before it, we want to discard that line.
2. We want to match only linux and nothing else.

We already know how to satisfy the 1st condition. GNU.* will match 'GNU' followed by any number of characters. Then finally we match the word linux. This will match all of GNU-any-characters-linux.

But how do we prevent matching GNU.* while still maintaining the 1st condition?

That's where a positive lookbehind comes in. You can mark a capture group as a positive lookbehind by prefixing it with ?<=. In this example, the expression becomes (?<=GNU.*)linux.

Now only linux is matched and nothing else.

Note that the expressions (?<=GNU.*)linux and linux(?<=GNU.*) will behave exactly the same. In the 2nd expression, although linux is before the lookbehind, there is .* after 'GNU' which matches linux. This means it satisfies the lookbehind.

To make it simpler, think about the pattern without the lookbehind. The pattern GNU.* will match 'GNU' and anything after it, in our case matching linux.

Now we can derive a generalized statement that the expression (?<=C)X will match the pattern X – only if pattern C came before X (and C must not be included in the match).

You can also reverse the 1st condition. Match lines that contains the word linux only if GNU never came before it. This is called a negative lookbehind. The prefix in this case is ?<!. The inverse of the previous expression would be (?<!GNU.*)linux.

Lookaheads

Lookaheads are also assertions like lookbehinds, as you saw in the previous example. The only difference is that lookbehinds make an assertion before and lookaheads makes assertion after.

Let's say you have these two conditions:

1. Match Hello only if World comes somewhere after it.
2. Match only Hello and nothing else.

The prefix for a positive lookahead is ?=. The expression Hello(?=.*World) will meet both conditions. This is similar to Hello.*World except that only Hello will be matched whereas Hello.*World will match 'Hello', 'World' and anything in between.

Similar to the example in a positive lookbehind, the expressions Hello(?=.*World) and (?=.*World)Hello are equivalent. Because the .* before 'World' matches Hello, satisfying the 1st condition.

A negative lookahead is just the complement of a negative lookbehind. You can use it by prefixing it with ?!. (?!World)Hello will match Hello only if there is no World anywhere after it.

Here is a summary of the syntax for lookarounds when you want to match the pattern X with assertion C.

Practical Examples of Regex

Logs parsing

In this log file, these are the lines which we care about:

[1/10000] Train loss: 11.30368, Valid loss: 8.95446, Elapsed_time: 7.58941
[500/10000] Train loss: 0.96180, Valid loss: 0.20098, Elapsed_time: 82.48651
[1000/10000] Train loss: 0.04051, Valid loss: 0.11927, Elapsed_time: 156.86243

Our task is to extract the training loss and validation loss for purposes such as plotting loss over the epochs. We need to extract the training loss values like 11.30368, 0.96180, 0.04051 and put them in an array.

All the relevant values are prefixed with 'Train loss:', so we can use this in our regex as it is. To match the float numbers we have to match some digits followed by a "." and then followed by more digits. You can do this with \d+\.\d+. Because we want to keep track of these numbers, they should be inside a capture group.

As "." has special purpose in regex, when you want to match a "." character you have to escape it with a backslash. This is applicable for all characters with a special purpose. But you dont have to escape it inside a character set.

Putting it altogether, the expression for extracting training loss is Train loss: (\d+\.\d+). We can use the same logic to extract validation loss with Valid loss: (\d+\.\d+).

Here is one way to extract this information using Python:

import re

f = open("log_train.txt", "r").read()

train_loss = re.findall(r'Train loss: (\d+\.\d+)', f)
valid_loss = re.findall(r'Valid loss: (\d+\.\d+)', f)

train_loss = [float(i) for i in train_loss]
valid_loss = [float(i) for i in valid_loss]

print("train_loss =", train_loss)
print("")
print("valid_loss =", valid_loss)

When there is one capture group, re.findall searches all the lines and returns the values inside the capture group in a list.

Any regex function only return strings, so the values are converted to floats and printed out. Then you can directly use them in another Python script as a list of floats.

This is the result:

You could also use sed, save the output in train_losses.txt, and read from the file. First we use '/Train/' to target only the lines with 'Train' present then we are applying the same regex as before.

sed -En '/Train/ s/.*Train loss: ([0-9]+\.[0-9]+).*/\1/p' log_train.txt | tee train_losses.txt

".*" is added at the start and end so that sed matches the contents of all the relevant lines. Then the entire line is replaced by the value of the capture group. The tee command is used to redirect the output of sed into train_losses.txt while also printing the contents in the terminal.

Take a moment to think about what would you need to extract the epochs. You have to extract 500 from [500/10000] for all such lines. The array should look like [1, 500, 1000, 1500, ...]. You can follow the same approach as we used for the previous example.

Note that if you want to match "[" or "]", you have to escape it. The answer is given here.

Bulk File Renaming

You have these files with some random values as prefixes. You have to rename all files as 1.mp4, 2.mp4 and so on. This is how the files were generated.

This is a common scenario where you have a list of files which have their sequence number in the name but there are also some other characters that you don't want.

The pattern has to match anything up to Episode then an underscore and then the number and .mp4 at the end.

The relevant value is the number before '.mp4' which we will put inside a capture group. .*Episode_ will match everything up to the number. Then we can capture the number with ([0-9]+) and also match .mp4 with \.mp4.

So the final regex is .*Episode_([0-9]+)\.mp4. As we want to keep the .mp4 the replacement string will be \1.mp4.

This is one way to solve it using sed.

for i in *.mp4; do
    newname=$(echo $i | sed -En 's/.*Episode_([0-9]+)\.mp4/\1.mp4/p')
    mv $i $newname
done;ls

First the new name is saved in a variable and then the mv command is used to rename the file.

Could we have just used .* in place of .*Episode_ ? In this example, yes. But there might be filenames such as Steins_Gate0.mp4 where the 0 is part of the movie name and you didn't really want to rename this file so its always better to be as specific as possible.

What if some files were named as "Random_Episode6.mp4"? The difference being, there is no underscore after Episode. What change will you need to make?

The answer is that you'll need to add a "?" after the "_" to make it optional. The regex will be .*Episode_?([0-9]+)\.mp4.

Email validation

There are all sorts of complicated regex for validating email.

Here is a simple one: ^[^@ ]+@[^@.]+\.\w+$. It matches the format A@B.C

The table below breaks down this pattern into smaller pieces:

In the regexr site, you can enable the multline flag from the Flags tab in the upper right corner. The 'gm' at the end indicates that the multiline flag is enabled.

We can see that line 2,3,5,6 didn't match. Can you find out the reason and which part of the regex is responsible for disqualifying it?

The answer is given here

Password constraints

You can also use regex to impose constraints. Here we will uncover the power of positive lookaheads.

Lets say we want to accept a string only if there is a digit in it. You already know how to find a digit with the '\d' class. To accomplish that, we can use [^\d]*\d. This will match any non-digit character 0 or more times and then match a digit.

We can also use the expression .*\d to match one digit. So if there is no digit in the string then the lookahead will fail and the none of the characters of that string will be matched, returning an empty string "".

When we are using a programming language, we can check if the regex returned an empty string and determine that the constraints are not satisfied.

We will create a regex which imposes the following criteria:

1. Minimum 8 characters and maximum 16 characters.
2. At least one lower case letter.
3. At least one upper case letter.
4. At least one number.

To achieve this, you can use positive lookaheads. This is the regex:

^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{8,16}$

The table below explains which part of the regex imposes which constraint:

What modification you would need for imposing at least 5 upper case letters?

You may think (?=.*[A-Z]{5,}) will do the job. But this expression requires all the 5 letters to be together. A string like rand-ABCDE-rand will be matched but 0AxBCDxE0 will not be matched even though it has 5 upper case letters (as they are not adjacent).

Yet again, we have capture groups coming to the rescue. We want to match 5 uppercase letters anywhere in the string. We already know that we can match 1 uppercase letter with .*[A-Z]. Now we will put them inside a capture group and attach a quantifier of minimum 5. The expression will be (.*[A-Z]){5,}.

Here is the final answer:

In place of (?=.*[A-Z]) you will need (?=(.*[A-Z]){5,}). The expression becomes ^(?=.*[a-z])(?=(.*[A-Z]){5,})(?=.*\d).{8,16}$.

You could also require that the password not contain certain words to enforce stronger passwords.

For example, we want to reject the password if contains pass or 1234. Negative lookaheads is the tool for this job. The regex would be ^(?!.*(pass|1234)).*$.

In this regex, we put pass and 1234 inside a capture group and used the logical OR operator. This capture group is nested inside another capture group which is prefixed with ?!.*. This makes it a negative lookahead that matches if there are at least 8 characters by .{8,} with the condition that, pass or 1234 can't be present anywhere in the string.

Final Words

I hope you got a good amount of practice while going through this article. It's ok if you forget some syntax. What's important is understanding the core concepts and having a good idea of what's possible with regex. Then, if you forget a pattern, you can just google it or reference a cheatsheet.

The more you practice, the more you will get by without outside help. Eventually you will be able write super complex and effective regex completely offline.

There are already some good regex cheatsheets out there, so I wanted to create something more in-depth here that you can reference for the core concepts and common use cases.

If you're looking for a cheatsheet, the one from QuickRef is helpful. It's a good place to recall the syntax and they also provide some basic overview of regex related functions in various programming languages.

Most regex techniques are the same in all programming languages and tools – but certain tools might offer additional features. So do some research on the tool you are using to pick the best one for you.

My final suggestion would be not to force using regex just because you can. A lot of the times a regular string.find() is enough to get the job done. But if you live in the terminal, you really can do a lot just with regex for sure.

If you like this type of article, you may keep an eye on my blog or twitter.

Table of contents

1. Automating Workflows with Blueprints, C++, and Python
2. Why the Unreal Engine is interesting
   • Why use Blueprints for workflow and even game programming
   • Why use C++ for workflow and game programming
   • Why use Python for workflow optimisation
3. Writing our own automated project clean-up script using Python
   • But does it pay off?
4. Learn how to automate and optimise workflows with scripting on Udemy

Automating Workflows with Blueprints, C++, and Python

Every job has repetitive tasks and processes that can be automated. Those tasks can take up a significant amount of your time.

Someone with only basic knowledge about scripting can build a script that might cut down the time to execute those tasks to a bare minimum. In the long run, this time saving accumulates into additional hours that can be used for more productive work.

Learning how to automate things, therefore, is an invaluable skill. A skill that can be acquired in different, more specific sectors and then applied to other, more general ones.

Especially considering the current difficult times, learning a new skill that will have a significant impact on your employability is vital.

In this article, we want to describe the importance of automation when working with real-time software like the Unreal Engine.

Why the Unreal Engine is interesting

Epic’s Unreal Engine is one of the most popular game and real-time 3D applications out there. It's used for the creation of entertainment content like games and interactive setups.

Disciplines like virtual and augmented reality require the usage of sophisticated processes that, just like any other pipeline, can be optimised through automation.

In addition to that, the gaming and 3D real-time market is continually growing, which increases the demand for talent.

According to a study conducted by Burning Glass Technologies, a labor market analytics firm, the average salaries for Unreal developers increased by 22% last year, the wages for Artists, by 51%.

Source: [https://www.unrealengine.com/en-US/blog/demand-for-unreal-engine-and-real-time-3d-skills-at-all-time-high](https://www.unrealengine.com/en-US/blog/demand-for-unreal-engine-and-real-time-3d-skills-at-all-time-high" rel="nofollow noopener)

Why use Blueprints for workflow and even game programming

In addition to the general C++ programming interface, the Unreal Engine provides a graphical programming system called Blueprints. They expose the full functionality as C++ code, which means that everything, including in-game features, can be programmed without huge performance drawbacks.

The graphical interface allows us to quickly compose entities into a flow to create functionality. Looking at an example that sets the material for a static mesh actor, we can see that the Blueprint is easily readable.

Blueprints script to set a material for a static mesh actor

Why use C++ for workflow and game programming

The “native” approach to coding for the unreal engine is C++. It is used to create in-game logic, simplify level creation and workflows, and improve the development pipeline. It’s more complex and difficult to learn then Blueprints but can add an additional boost in run-time and performance.

Its performance advantage makes it the language of choice for essential, lower-level operations such as rendering and physics in game development. For developers that are already proficient with C++, it is a convenient addition to be able to automate processes without learning an additional language directly.

Why use Python for workflow optimisation

Compared to Blueprints or C++, Python is one of the de-facto standard languages used for automating tasks. It’s simple to learn, understandable, and extremely versatile since it can run on nearly any platform without additional effort.

Epic lists Python as one of the required skills, for example, for Data Pipeliners in their Creator’s Field Guide to Emerging Careers in Interactive 3D.

https://epicgames.ent.box.com/s/n12ixy53l8cknz73npimsr54frkvm72c

The Unreal Engine has full Python scripting support. Unfortunately, it is not suitable for real-time and in-game scripting, but can only be used for Unreal Editor scripting. Python’s simplicity, however, makes it an incredible option for fast prototyping of pipeline automation.

Epic themselves try to promote the usage of Python using the documentation, API docs, and even a recorded webinar about Python scripting for the Unreal Engine.

Writing our own automated project clean-up script using Python

Larger projects can get messy fast. Having a script at hand that can do a clean-up by looking at all of our selected assets and automatically moving them into appropriate folders helps us drastically improve our workflow.

The below schematic drawing explains the idea behind the script.

As of now, the by default version of Python set in Unreal is Python 2.7. If you want to use Python 3, you can follow the process described here to switch to Python 3.

To begin with, we first need to import the unreal library into our scope. Once we have done this, we can create class instances of the elementary classes. For now, we only need the [EditorUtilityLibrary](https://docs.unrealengine.com/en-US/PythonAPI/class/EditorUtilityLibrary.html?highlight=editorutilitylibrary) which enables us to get a list of all the selected assets.

We can get the number of selected assets by using Python's len() method and use Unreal's logging method to get output on the Debug Log in the Unreal Engine.

import unreal

# create unreal class instances
editor_util = unreal.EditorUtilityLibrary()

# get the selected assets
selected_assets = editor_util.get_selected_assets()
num_assets = len(selected_assets)

unreal.log(num_assets)

The code snippet above will print a single number to the Debug Log. For each of the selected assets, we now want to get some information like the asset name and class.

The Unreal [ObjectBase](https://docs.unrealengine.com/en-US/PythonAPI/class/_ObjectBase.html?highlight=objectbase#unreal._ObjectBase) object has several helper methods to get the name, class, and other attributes. We will make use of the get_fname() and get_class() methods.

for assets in selected_assets:
    # get the class instance and the clear text name
    asset_name = asset.get_fname()
    asset_class = asset.get_class()

    unreal.log("{} - {}".format(asset_name, asset_class))

However, this will only give us the class definition and not the clear text name of the class itself, which we want to use for folder creation.

To get the display name instead of the class definition, we need to create an instance of the [SystemLibrary](https://docs.unrealengine.com/en-US/PythonAPI/class/SystemLibrary.html?highlight=systemlibrary). It's get_class_display_name() method takes a class definition and returns the class name as String.

# create unreal class instances
editor_util = unreal.EditorUtilityLibrary()
system_lib = unreal.SystemLibrary()

...

for assets in selected_assets:
    # get the class instance and the clear text name
    asset_name = asset.get_fname()
    asset_class = asset.get_class()
    class_name = system_lib.get_class_display_name(asset_class)

    unreal.log("Name: {} - Class: {}".format(asset_name, class_name))

Now we can see something like this "Name: NewMaterial - Class: Material" printed to our log. This is precisely the kind of information we needed.

The last step is to "rename" our assets to a given location. For example, every Material will be renamed to "/Material/<Name of Material Asset>" which will move it into the according to folders.

To "rename" assets, we need an additional class. The rename_loaded_asset() method is part of the EditorAssetLibrary, so we need to create an instance of this class first. In addition to that, we have to create a new location to which the asset will be relocated.

To keep this more platform-independent, we will use the os module and its path.join() method.

Once we have created the new_path variable, we can use it in the method call to rename_loaded_asset() to relocate our current asset.

import os
import unreal

# create unreal class instances
editor_util = unreal.EditorUtilityLibrary()
system_lib = unreal.SystemLibrary()
editor_asset_lib = unreal.EditorAssetLibrary()

...

for assets in selected_assets:
    # get the class instance and the clear text name
    asset_name = asset.get_fname()
    asset_class = asset.get_class()
    class_name = system_lib.get_class_display_name(asset_class)

    # assemble new path and relocate asset
    new_path = os.path.join("/Game", class_name, asset_name)
    editor_asset_lib.rename_loaded_asset(asset_name, new_path)

    unreal.log("Moved {} to {}".format(asset_name, new_path))

Executing this script in the Unreal Engine, the log will provide you with such a message: "Moved NewMaterial to /Game/Material/NewMaterial".
Observing our project, we can now see that all the selected assets have been cleaned into folders named according to their classes.

As you can see, creating a basic script is quite simple. Of course, we need to take care of error handling, suitable logging, and a friendly user interface for more sophisticated tools, but even simple scripts can save a lot of time.

But does it pay off?

To show you how big the demand for automation in this sector is, here is a screenshot of the monthly sales of a tool with functionality containing the script we created in this article.

Of course, it is essential to understand the needs of users and artists working in the Unreal Engine to know which tasks are suitable for automation.

Learn how to automate and optimise workflows with scripting on Udemy

To help close the gap of Unreal scripting resources, we created an extensive Udemy on-demand course to learn Unreal Engine Editor scripting from scratch.

You can use the promo code, AUTOMATEUNREAL20. You can get the course for as little as $10 now. That’s less than three Frappuccinos from Starbucks!

If you have any questions or feedback, feel free to reach out to us on Twitter or directly in the discussion section of the course :)

Redis is a popular in-memory grid used for interprocess communication and data storage. You might’ve heard that it lets you run Lua scripts, but you are still not sure why. If this sounds like you, read on.

Prerequisites

You should have Redis installed on your system to follow this guide. It might be helpful to check Redis commands reference while reading.

Why do I need Lua Scripts?

In short: performance gain. Most tasks you do in Redis involve many steps. Instead of doing these steps in the language of your application, you can do it inside Redis with Lua.

• This may result in better performance.
• Also, all steps within a script are executed in an atomic way. No other Redis command can run while a script is executing.

For example, I use Lua scripts to change JSON strings stored in Redis. I describe this in detail closer to the end of this article.

But I don’t know any Lua

A person who doesn't know any lua

Don’t worry, Lua is not very hard to understand. If you know any language of the C family, you should be okay with Lua. Also, I am providing working examples in this article.

Show me an example

Let’s start by running scripts via redis-cli. Start it with:

redis-cli

Now run the following command:

eval “redis.call(‘set’, KEYS[1], ARGV[1])” 1 key:name value

The EVAL command is what tells Redis to run the script which follows. The ”redis.call(‘set’, KEYS[1], ARGV[1])”string is our script which is functionally identical to the Redis’s set command. Three parameters follow the script text:

1. The number of provided keys
2. Key name
3. First argument

Script arguments fall into two groups: KEYS and ARGV.

We specify how many keys the script requires with the number immediately following it. In our example, it is 1. Immediately after this number, we need to provide these keys, one after another. They are accessible as KEYS table within the script. In our case, it contains a single value key:name at index 1.

Note, that Lua indexed tables start with index 1, not 0.

We can provide any number of arguments after the keys, which will be available in Lua as the ARGV table. In this example, we provide a single ARGV-argument: string value. As you already guessed, the above command sets the key key:name to value value.

It is considered a good practice to provide keys which the script uses as KEYS, and all other arguments as ARGV. So you shouldn’t specify KEYS as 0 and then provide all keys within the ARGV table.

Let’s now check if the script completed successfully. We are going to do this by running another script which gets the key from Redis:

eval “return redis.call(‘get’, KEYS[1])” 1 key:name

The output should be ”value”, which means that the previous script successfully set the key “key:name”.

Can you explain the script?

Doge after seeing the script above

Our first script consists of a single statement: the redis.call function:

redis.call(‘set’, KEYS[1], ARGV[1])

Withredis.call you can execute any Redis command. The first argument is the name of this command followed by its parameters. In the case of the set command, these arguments are key and value. All Redis commands are supported. According to the documentation:

Redis uses the same Lua interpreter to run all the commands

Our second script does a little more than just running a single command — it also returns a value:

eval “return redis.call(‘get’, KEYS[1])” 1 key:name

Everything returned by the script is sent to the calling process. In our case, this process is redis-cli and you will see the result in your terminal window.

Something more complex?

A person planning to build a complex Redis script

I once used Lua scripts to return elements from a hash map in a particular order. The order itself was specified by hash keys stored in a sorted set.

Let’s first set up our data by running these commands in redis-cli:

hmset hkeys key:1 value:1 key:2 value:2 key:3 value:3 key:4 value:4 key:5 value:5 key:6 value:6
zadd order 1 key:3 2 key:1 3 key:2

These commands create a hash map at key hkeys and a sorted set at key order which contains selected keys from hkeys in a specific order.

You might want to check hmset and zadd commands reference for details.

Let’s run the following script:

eval “local order = redis.call(‘zrange’, KEYS[1], 0, -1); return redis.call(‘hmget’,KEYS[2],unpack(order));” 2 order hkeys

You should see the following output:

“value:3”
“value:1”
“value:2”

Which means that we got values of the keys we wanted and in the correct order.

Do I have to specify full script text to run it?

No! Redis allows you to preload a script into memory with the SCRIPT LOAD command:

script load “return redis.call(‘get’, KEYS[1])”

You should see an output like this:

“4e6d8fc8bb01276962cce5371fa795a7763657ae”

This is the unique hash of the script which you need to provide to the EVALSHA command to run the script:

evalsha 4e6d8fc8bb01276962cce5371fa795a7763657ae 1 key:name

Note: you should use actual SHA1 hash returned by the SCRIPT LOAD command, the hash above is only an example.

What did you mention about changing JSON?

Sometimes people store JSON objects in Redis. Whether it is a good idea or not is another story, but in practice, this happens a lot.

If you have to change a key in this JSON object, you need to get it from Redis, parse it, change the key, then serialize and set it back to Redis. There are a couple of problems with this approach:

1. Concurrency. Another process can change this JSON between our get and set operations. In this case, the change will be lost.
2. Performance. If you do these changes often enough and if the object is rather big, this might become the bottleneck of your app. You can win some performance by implementing this logic in Lua.

Let’s add a test JSON string to Redis under key obj:

set obj ‘{“a”:”foo”,”b”:”bar”}’

Now let’s run our script:

EVAL ‘local obj = redis.call(“get”,KEYS[1]); local obj2 = string.gsub(obj,”(“ .. ARGV[1] .. “\”:)([^,}]+)”, “%1” .. ARGV[2]); return redis.call(“set”,KEYS[1],obj2);’ 1 obj b bar2

Now we will have the following object under key obj:

{“a”:”foo”,”b”:”bar2"}

You can instead load this script with the SCRIPT LOAD command:

SCRIPT LOAD ‘local obj = redis.call(“get”,KEYS[1]); local obj2 = string.gsub(obj,”(“ .. ARGV[1] .. “\”:)([^,}]+)”, “%1” .. ARGV[2]); return redis.call(“set”,KEYS[1],obj2);’

and then run it like this:

EVALSHA <your_script_sha> 1 obj b bar2

Some notes:

• The .. is the string concatenation operator in Lua.
• We use a RegEx pattern to match key and replace its value. If you don’t understand this Regular Expression, you can check my recent guide.
• One difference of the Lua RegEx flavor from most other flavors is that we use % as both backreference mark and escape character for RegEx special symbols.
• We still escape ” with \ and not % because we escape Lua string delimiter, not RegEx special symbol.

Should I always use Lua scripts?

No. I recommend only using them when you can prove that it results in better performance. Always run benchmarks first.

If all you want is atomicity, then you should check Redis transactions instead.

Also, your script shouldn’t be too long. Remember that while a script is running, everything else is waiting for it to finish. If your script takes quite some time, it can cause bottlenecks instead of improving performance. The script stops after reaching a timeout (5 seconds by default).

Redis scripts should not take too much time

Last Word

For more information on Lua check lua.org.

You can check my node.js library on GitHub for some examples of Lua scripts (see src/lua folder). You can also use this library in node.js to change JSON objects without writing any Lua scripts yourself.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Thank you for reading this article. Questions and comments are much appreciated. You are also welcome to follow me on Twitter.

I recently came across a few such things, particularly in Tom Hudson’s dotfiles. Tom seems to like to script things, and some of those things include automatically setting up symlinks, and installing Ubuntu repository applications and other programs. This got me thinking. Could I automate the set up of a new machine to replicate my current one?

Being someone generally inclined to take things apart in order to see how they work, I know I’ve messed up my laptop on occasion. (Usually when I’m away from home, and my backup harddrive isn’t.) On those rare but really inconvenient situations when my computer becomes a shell of its former self, (ba-dum-ching) it’d be quite nice to have a fast, simple way of putting Humpty Dumpty back together again, just the way I like.

In contrast to creating a disk image and restoring it later, a collection of bash scripts is easier to create, maintain, and move around. They require no special utilities, only an external transportation method. It’s like passing along the recipe, instead of the whole bundt cake. (Mmm, cake.)

Additionally, functionality like this would be super useful when setting up a virtual machine, or VM, or even just a virtual private server, or VPS. (Both of which, now that I write this, would probably make more forgiving targets for my more destructive experimentations… live and learn!)

Well, after some grepping and Googling and digging around, I now have a suite of scripts that can do this:

See a video of the setup here.

This is the tail end of a test run of the set up scripts on a fresh Ubuntu desktop, loaded off a bootable USB. It had all my programs and settings restored in under three minutes!

This post will cover how to achieve the automatic set up of a computer running Ubuntu Desktop (in my case, Ubuntu LTS 18.04) using bash scripts. The majority of the information covered is applicable to all the Linux desktop flavours, though some syntax may differ. The bash scripts cover three main areas: linking dotfiles, installing software from Ubuntu and elsewhere, and setting up the desktop environment. We’ll cover each of these areas and go over the important bits so that you can begin to craft your own scripts.

Dotfiles

Dotfiles are what most Linux enthusiasts call configuration files. They typically live in the user’s home directory (denoted in bash scripts with the builtin variable $HOME) and control the appearance and behaviour of all kinds of programs. The file names begin with ., which denotes hidden files in Linux (hence “dot” files). Here are some common dotfiles and ways in which they’re useful.

.bashrc

The .bashrc file is a list of commands executed at startup by interactive, non-login shells. Interactive vs non-interactive shells can be a little confusing, but aren’t necessary for us to worry about here. For our purposes, any time you open a new terminal, see a prompt, and can type commands into it, your .bashrc was executed.

Lines in this file can help improve your workflow by creating aliases that reduce keystrokes, or by displaying a helpful prompt with useful information. It can even run user-created programs, like Eddie. For more ideas, you can have a look at my .bashrc file on GitHub.

.vimrc

The .vimrc dotfile configures the champion of all text editors, Vim. (If you haven’t yet wielded the powers of the keyboard shortcuts, I highly recommend a fun game to learn Vim with.)

In .vimrc, we can set editor preferences such as display settings, colours, and custom keyboard shortcuts. You can take a look at my .vimrc on GitHub.

Other dotfiles may be useful depending on the programs you use, such as .gitconfig or .tmux.conf. Exploring dotfiles on GitHub is a great way to get a sense of what’s available and useful to you!

Linking dotfiles

We can use a script to create symbolic links, or symlinks for all our dotfiles. This allows us to keep all the files in a central repository, where they can easily be managed, while also providing a sort of placeholder in the spot that our programs expect the configuration file to be found. This is typically, but not always, the user home directory. For example, since I store my dotfiles on GitHub, I keep them in a directory with a path like ~/github/dotfiles/ while the files themselves are symlinked, resulting in a path like ~/.vimrc.

To programmatically check for and handle any existing files and symlinks, then create new ones, we can use this elegant shell script. I compliment it only because I blatantly stole the core of it from Tom’s setup script, so I can’t take the credit for how lovely it is.

The symlink.sh script works by attempting to create symlinks for each dotfile in our $HOME. It first checks to see if a symlink already exists, or if a regular file or directory with the same name exists. In the former case, the symlink is removed and remade; in the latter, the file or directory is renamed, then the symlink is made.

Installing software

One of the beautiful things about exploring shell scripts is discovering how much can be achieved using only the command line. As someone whose first exposure to computers was through a graphical operating system, I find working in the terminal to be refreshingly fast.

With Ubuntu, most programs we likely require are available through the default Ubuntu software repositories. We typically search for these with the command apt search <program> and install them with sudo apt install <program>. Some software we’d like may not be in the default repositories, or may not be offered there in the most current version. In these cases, we can still install these programs in Ubuntu using a PPA, or Personal Package Archive. We’ll just have to be careful that the PPAs we choose are from the official sources.

If a program we’d like doesn’t appear in the default repositories or doesn’t seem to have a PPA, we may still be able to install it via command line. A quick search for “installation command line” should get some answers.

Since bash scripts are just a collection of commands that we could run individually in the terminal, creating a script to install all our desired programs is as straightforward as putting all the commands into a script file. I chose to organize my installation scripts between the default repositories, which are installed by my aptinstall.sh script, and programs that involve external sources, handled with my programs.sh script.

Setting up the desktop environment

On the recent occasions when I’ve gotten a fresh desktop (intentionally or otherwise) I always seem to forget how long it takes to remember, find, and then change all the desktop environment settings. Keyboard shortcuts, workspaces, sound settings, night mode… it adds up!

Thankfully, all these settings have to be stored somewhere in a non-graphical format, which means that if we can discover how that’s done, we can likely find a way to easily manipulate the settings with a bash script. Lo and behold the terminal command, gsettings list-recursively.

There are a heck of a lot of settings for GNOME desktop environment. We can make the list easier to scroll through (if, like me, you’re sometimes the type of person to say “Just let me look at everything and figure out what I want!”) by piping to less: gsettings list-recursively | less. Alternatively, if we have an inkling as to what we might be looking for, we can use grep: gsettings list-recursively | grep 'keyboard'.

We can manipulate our settings with the gsettings set command. It can sometimes be difficult to find the syntax for the setting we want, so when we’re first building our script, I recommend using the GUI to make the changes, then finding the gsettings line we changed and recording its value.

For some inspiration, you can view my desktop.sh settings script on GitHub.

Putting it all together

Having modular scripts (one for symlinks, two for installing programs, another for desktop settings) is useful for both keeping things organized and for being able to run some but not all of the automated set up. For instance, if I were to set up a VPS in which I only use the command line, I wouldn’t need to bother with installing graphical programs or desktop settings.

In cases where I do want to run all the scripts, however, doing so one-by-one is a little tedious. Thankfully, since bash scripts can themselves be run by terminal commands, we can simply write another master script to run them all!

Here’s my master script to handle the set up of a new Ubuntu desktop machine:

#!/bin/bash

./symlink.sh
./aptinstall.sh
./programs.sh
./desktop.sh

# Get all upgrades
sudo apt upgrade -y

# See our bash changes
source ~/.bashrc

# Fun hello
figlet "... and we're back!" | lolcat

I threw in the upgrade line for good measure. It will make sure that the programs installed on our fresh desktop have the latest updates. Now a simple, single bash command will take care of everything!

You may have noticed that, while our desktop now looks and runs familiarly, these scripts don’t cover one very important area: our files. Hopefully, you have a back up method for those that involves some form of reliable external hardware. If not, and if you tend to put your work in external repository hosts like GitHub or GitLab, I do have a way to automatically clone and back up your GitHub repositories with bash one-liners.

Relying on external repository hosts doesn’t offer 100% coverage, however. Files that you wouldn’t put in an externally hosted repository (private or otherwise) consequently can’t be pulled. Git ignored objects that can’t be generated from included files, like private keys and secrets, will not be recreated. Those files, however, are likely small enough that you could fit a whole bunch on a couple encrypted USB flash drives (and if you don’t have private key backups, maybe you ought to do that first?).

That said, I hope this post has given you at least some inspiration as to how dotfiles and bash scripts can help to automate setting up a fresh desktop. If you come up with some settings you find useful, please help others discover them by sharing your dotfiles, too!

I learned by doing projects (often freeCodeCamp ones!). But I needed to make sure I didn't lose my hard work, and that others could see the hard work I was putting in, so every project had to go on Github. The more complete projects I had on Github, the easier it would be for recruiters. But the steps required to set up a project, initialise a repo, and sync with Github were really annoying and repetitive, so I decided to solve the problem.

Bad news: this isn't going to be a big, fancy, detailed and technically sexy post. It's going to be very un-sexy.

Good news: you don't need to be a shell scripting god(dess) to do it.

So my typical project setup work flow usually goes like this:

1. Go to my ../projects folder and run mkdir project-of-some-name to create a folder with the name project-of-some-name.

2. cd into that project folder and do git init to initialise a local git repo in there.

3. run touch README.MD to create the README file, open it and add some basic descriptions, including links to the resources / tutorials I was implementing in that project. Save the file.

4. run git add . and then git commit -m ' ...some initial commit message...

5. open a browser, go to Github, login, create a new (remote) repository, copy the url, return to my terminal, make sure I was in the correct project folder project-of-some-name...then run the git scripts needed to set up the remote repo as the 'upstream' repo and connect my local repo to it. Then, finally, I can run a git push and my local commit would get pushed up

6. lie down and take a nap, exhausted from this repetitive process.

Admittedly, this was my process, but I liked to stay organised and always be able to access my projects so I can refer to them.

Since automation is a great way to practice your coding skills, I decided to write a small shell script that automates these horrible and repetitive steps. The script is at the bottom of this post, and be warned - it's not sophisticated or fancy. But it sure gets the job done, and I don't need to log in to Github and fool around will all those steps!

Before you copy the script, you need to know how to run it on your Mac. So below are the steps you need to implement to be able to use the script to automate your setup workflow.

1. I keep my scripts in my root/home folder, in a sub-folder called scripts. I suggest you do the same or similar. To get to the root/home folder, in your terminal type cd ~ because the tilda ( ~ )is the symbol for the home folder. In your Mac Finder app it shows up as the one with a house icon. So all my scripts are stored in ~/scripts

2. This matters because to run a shell script from any directory in the terminal, you have to type out the full path. In my case I have to type out ~/scripts/git-script.sh to run the script. But we're getting ahead of ourselves.

3. copy the code chunk at the bottom of this post and then open a text editor, paste it in and then save it as [filename].sh. The .sh is the extension for shell scripts. Save the file in the directory you want to save it at - again I recommend ~/scripts as the folder to save your scripts in.

4. Navigate to that folder in your terminal. To be safe run ls in the terminal to check that you can see the script is there. If it's not you're in the wrong folder or step 3 didn't successfully complete.

5. Make the shell script executable. To do that you type the following in the terminal: chmod +x <<the-correct-filename.sh>>. This is the unix way to make a shell script "executable". I'm not confident I fully understand what that means, other than it's needed to make any shell scripts you write executable, so don't ask me and I won't lie to you.

6. navigate to your projects folder and make a new folder that you intend to house your project. Effectively, you've got to do this: mkdir - create a project-of-some-name inside the folder where you keep all your projects. So your project will eventually be placed inside my-computer/my-projects/project-of-some-name. cd into this folder and then type pwd to get the full path. Copy that - you will need to paste it shortly. It should look like my-computer/my-projects/project-of-some-name

7. open your terminal again, and then type ~/scripts/<<the-correct-filename.sh>>``. The script runs! You will be guided through some input... The main steps are:

   what do you want to call your Github repo (don't use spaces- 'my-awesome-project' is good. Don't use 'my awesome project' as the repo name.

   Enter a description that shows up in the Github repo's description. For this it's safe to use spaces.

Enter the project path you got in step 6, the one that you get after typing pwd in the terminal and getting something like my-computer/my-projects/project-of-some-name

enter your Github username (not email address) and then your Github password. Be careful as you type as these values don't show up on the screen.

....that's it. The script will set up your git repo locally inside my-computer/my-projects/project-of-some-name and then create a README.MD (blank) and then commit it locally, then set up a remote repo in Github (log you in via API) etc and then push everything up!

finally, you will see that the terminal you were interacting with has changed the currently active directory to your project folder. It will now be at my-computer/my-projects/project-of-some-name and you can type in ls and see the README.MD file. If you then type git status you will see your local repo's status (the state of your local project) and if you type in git remote it will show you the Github url for your project!

Done! Happy Coding!

Annnd.....finally......here is the script! I've commented each step so you can reason your way through it.

# Make executable with chmod +x <<filename.sh>>

CURRENTDIR=${pwd}

# step 1: name of the remote repo. Enter a SINGLE WORD ..or...separate with hyphens
echo "What name do you want to give your remote repo?"
read REPO_NAME

echo "Enter a repo description: "
read DESCRIPTION


# step 2:  the local project folder path
echo "what is the absolute path to your local project directory?"
read PROJECT_PATH

echo "What is your github username?"
read USERNAME

# step 3 : go to path 
cd "$PROJECT_PATH"


# step 4: initialise the repo locally, create blank README, add and commit
git init
touch README.MD
git add README.MD
git commit -m 'initial commit -setup with .sh script'


# step 5 use github API to log the user in
curl -u ${USERNAME} https://api.github.com/user/repos -d "{\"name\": \"${REPO_NAME}\", \"description\": \"${DESCRIPTION}\"}"

#  step 6 add the remote github repo to local repo and push
git remote add origin https://github.com/${USERNAME}/${REPO_NAME}.git
git push --set-upstream origin master

# step 7 change to your project's root directory.
cd "$PROJECT_PATH"

echo "Done. Go to https://github.com/$USERNAME/$REPO_NAME to see." 
echo " *** You're now in your project root. ***"

Thanks for reading!

If you would like to learn more about my journey into code, check out episode 53 of the freeCodeCamp podcast, where Quincy (founder of freeCodeCamp) and I share our experiences as career changers that may help you on your journey. You can also access the podcast on iTunes, Stitcher, and Spotify.

I will also hold a few AMAs and webinars in the coming months. If this is of interest to you please let me know by going here. And of course, you can also Tweet me at @ZubinPratap.

Shell scripts vs python or Perl

It's 2020 now, who writes shell scripts anymore? Am I right? Well, apparently I do. ¯_(ツ)_/¯

There are some good arguments for that here and here which mainly revolve around 2 things:

1. Shell exists in all Unix systems and makes use of system default features.
2. Shell is an “interactive command function” designed to get user inputs during the process of running them.

Also, here’s an additional relevant reading about the differences between sh and bash.

Arguments

In some occasions, you will need to pass an argument (or expect one) into the script like how you might pass a param into a function. In that case, you will use something like $1 for the first argument, $2 for the second. Here's an example of how it would look like:

In script run_this.sh:

echo "The input message was $1."

Running the command:

./run_this.sh userInputThe input message was userInput.

_Note: The params are separated by spaces so if you want to input a string as a param that contains a space, it might need to be something like ./run_this.sh "user input" just so "user input" would be counted as $1 entirely._

In the occasion where you are not sure how long the user input might be and you want to capture it all, you would use $@ instead. In the following example, I took in the entire string and print it out word by word after breaking them into a string array according to the spaces in between.

In script run_this.sh:

userInputs=($@)for i in "${userInputs[@]}";; do  echo "$i"done

Running the command:

./run_this.sh who knows how long this can gowhoknowshowlongthiscango

Functions

If you have done any sort of programming, you should be familiar with the concept of functions. It's basically a set of commands / operations that you will be repeating over and over again. Instead of repeating it multiple times in your code, you can put them into a function. Then just call the function which effectively reduces the lines of code that need to be written.

Side note: If you don’t know already, LOC is a horrible metric for any sort of measurement in terms of programming. Don’t take this from me, take this from Bill Gates:

“Measuring programming progress by lines of code is like measuring aircraft building progress by weight.”

Here’s how a normal function looks like:

# Declaring the functiondoSomething() {

}

# Calling the functiondoSomething

Pretty straightforward and easy to understand. Now, here are a few differences between functions in shell scripts and a normal programming language.

Parameters

If you were to pass a parameter / use a parameter into a function in Java, you have to declare them in the function declaration. They look something like this.

public static void main(String[] args) {    doSomething("random String");}

private static void doSomething (String words) {    System.out.println(words);}

In the shell, however, they do not require a declaration of types or names at all. Each of them is like a separate script that lives in the script itself. If you were to use a param, just pass it in and call it like how you would do it if you were taking in input for this script at the top level. Something like this:

doSomething() {    echo $1}

doSomething "random String"

1. Similar to above, if you want to take in everything, you will use $@ instead of $1 since $1 would only use the first input (and $2 for the second etc.).
2. Functions need to be declared ahead of where they are being called. (Usually beginning of the file before any main operations.)

Return

Let’s say we create a script like below named run_this.sh:

doSomething() {    echo "magic"    return 0}

output=`doSomething`echo $output

Now let’s run it and see what is being assigned to the output variable.

$ ./run_this.shmagic

Note that instead of 0, it shows magic instead. This is because when you do output=doSomething`, it assigns the output message tooutput` instead of the return value since the output message is how you communicate almost anything in the shell script.

So when does it make sense to use the return call? When you are using it as part of an if statement. Something like this:

In script run_this.sh:

doSomething() {    echo "magic"    return 0}

if doSomething; then    echo "Its true!"fi

Running the command:

./run_this.shIts true!

In this case, return 0 means true while return 1 meant false in a traditional boolean sense.

Multi-line echo

There are times when you need to print a multi-line message. There are a few ways to go around this. The easiest way is to use echo multiple times like this:

echo "line1"echo "line2"echo "line3"

It works but probably not the most elegant way to get around this. Instead, you can use cat <&lt; EOF instead. Something like this:

cat << EOFline1line2line3EOF

Note that there should not be anything (including spaces or tabs) before EOF. If you want to do it in an ifstatement, it should look something like this.

if [ "a" == "a" ]; then  cat << EOFline1line2line3EOFfi

Realize that even the messages themselves are aligned to the left. This is because if you leave them tabbed, the output message shown in the command line will also be tabbed. Also, if EOF is tabbed, the shell will complain about it and usually ends the script there.

Flags / Options

You’ve probably seen some of the scripts or commands that comes with an ability to add flags (and sometimes arguments for the specific flag). Something like git commit -a -m "Some commit message".

Here’s a quick example of how it looks like (I’ve tried to be as comprehensive as possible with the example.)

In script run_this.sh:

while getopts ac: opt; do    case $opt in        a)            echo "\"a\" was executed."            ;;        c)            echo "\"c\" was executed with parameter \"$OPTARG\"."            ;;        \?)            echo "Invalid option: -$opt"            exit 1            ;;        :)            echo "option -$opt requires an argument."            exit 1            ;;    esacdone

Running the command:

./run_this.sh

./run_this.sh -a"a" was executed.

./run_this.sh -coption -c requires an argument.

./run_this.sh -c abcd"c" was executed with parameter "abcd".

./run_this.sh -a -c abc"a" was executed."c" was executed with parameter "abc".

./run_this.sh -xInvalid option: -x

In the above example, the differences between option -a and -c is that in the getopts line, c has a colon (:) following it after therefore telling the program to expect a parameter for the option. Another thing to keep in mind is that the options need to be declared in an alphabetical way. If you declare something like acb, the bdeclaration would be ignored, and using the -b flag would lead to the error message instead of the b case in the switch condition.

Thanks for reading!

About me

I currently work at Facebook as a Software Engineer. I spend some of my free time experimenting and building new things with technologies I find fun and interesting. Follow my exploration journey here or on GitHub.

References

• Small getopts tutorial
• How to output a multiline string in Bash

ABS is a programming language that puts together the productivity of bash scripts with the elegance of high-level languages such as Python or Ruby. It lets you issue system commands by simply wrapping them in backticks (very similar to how you would do it in Bash) and lets you use their output with clear and concise syntax.

This is, for example, a script that would try to issue a curl command, and exit if the server at example.org would not reply within 10 seconds:

A few weeks ago the ABS team managed to pull together a new minor release of the language, 1.2.0, which includes loads of interesting features — let’s get to them!

~/.absrc

ABS will now look for a default ~/.absrc file to preload every time you run a script. This is especially useful if you’d like to dump “base” functions you’re likely to re-use across scripts in a common place. Your .absrc could look like:

tenth = f(x) {   return x / 10 }

so that in any other abs script you can tenth(x).

~/.abs_history

We also introduced an history file in order to be able to repeat commands easily when using the ABS repl. This is, by default, located at ~/.abs_history and gets synced every time you close a repl session:

$ absHello alex, welcome to the ABS (1.2.0) programming language!Type 'quit' when you're done, 'help' if you get lost!⧐  `sleep 1`

⧐  quitAdios!$ tail ~/.abs_history`sleep 1`

require(file)

A big one here: you can now require external files through require(path/to/file.abs ).

This is a stepping stone that allows creating base libraries that can be re-used across ABS scripts, and organizes ABS code a tad better.

Background commands

Another big feature here: you can now issue “background” commands that won’t block your ABS script (these commands are executed within a Goroutine).

A background command differs from a regular one simply because it employs an & at the end of the command itself — let’s see them in action:

`sleep 10`echo("Hello world!") # This will be printed after 10s

`sleep 10 &`echo("Hello world!") # This will be printed immediately

You can check whether a background command is done with the .done property:

cmd = `sleep 10 &`cmd.done # falsewait(10000)cmd.done # true

and we’ve added the wait() function if you need to block until the command is done:

cmd = `sleep 10 &`cmd.wait() # The script will be blocked for 10secho("Hello world!")

Misc

A few more features that made it into this release:

• number functions such as floor, round and ceil
• cd(), which switches the cwd of a script
• you can play around with your prompt by setting the environment variables ABS_PROMPT_LIVE_PREFIX=true and ABS_PROMPT_PREFIX=templated_string. The templated string can use {dir}, {user}, {host} that will be replaced on-the-fly. For further info, have a look at the sample .absrc file

Bugfixes

As usual, we managed to fix some minor bugs along the way:

• fixed a few random panics when calling built-in functions without enough arguments (#193)
• windows commands are now using cmd.exe rather than bash, as bash might not be available on the system (#180)
• better error messages when parsing “invalid” numbers (#182)
• the ABS installer was not working with wget 1.20.1 (#178)
• the ABS parser now supports numbers in scientific notation (eg. 8.366100560806463e-7, #174)
• errors on built-in functions would not report the correct error line / column numbers (#168)

Now what?

Install ABS with a simple one-liner:

bash <(curl https://www.abs-lang.org/installer.sh)

…and start scripting like it’s 2019!

PS: Again, many thanks to Erich, who’s been taking a larger role as the weeks went by. Without him, many of the stuff included in 1.2 wouldn’t be possible!

PPS: 1.3.0 is already well underway — expect it at some point in April. We’ll be introducing extremely interesting features such as the ability to kill background commands, so it’s going to be an exciting release!

Originally published at odino.org (21st March 2019).
_You can follow me on Twitter — rants are welcome!_ ?