Tampermonkey is a browser extension that lets you add custom scripts to websites, making them work or look the way you want. It's like giving websites a makeover or adding new features.

These scripts are called userscripts and you can make tampermonkey run those scripts when you visit a particular site.

How to Use Tampermonkey to Customize the User Experience of a Website

There are a lot of use cases for Tampermonkey. The most obvious one is adding your own custom styling. You can add custom css for a specific site and change up the appearance as you want. But as you can run a script, you can also manipulate the DOM elements.

I will list out of the things I have done to give you some ideas to what is possible.

1. Increase readability by changing font properties.
2. Remove ads on sites that don't allow an ad blocker.
3. Declutter a site so that you can focus on the portion that you're interested in.
4. Add keyboard shortcuts for repetitive tasks.
5. Add buttons for custom actions.
6. Automatically fill out form data.

Today you'll get a glimpse of what you can do with Tampermonkey by writing scripts that'll work on the freeCodeCamp /news site.

First we will see how to declutter for a more focused reading experience. Then we will attach copy buttons on the code snippets. And lastly, we'll automatically generate table of contents that you can access with a toggle button.

The source code is available on GitHub.

It's worth pointing out that any changes your script makes will only be available in your browser. So as long as you don't deal with any sensetive data on that site, you can go as wild as you want.

But be aware that some sites may have some policy in regards to using third party JavaScript and take disciplinary action if you violate that policy.

Prerequisites

As we will be modifying a website, basic knowledge of HTML, CSS, and JavaScript is required to go through this tutorial. Some experience with DOM manipulation would be great as well.

If you can effectively manipulate exisiting DOM elements, that will allow you to write Tampermonkey scripts to make yourself more productive on any website.

Table of Contents

• How to Install Tampermonkey
• How to Create a New Userscript
• How to Change a Button's Behaviour
• How to Implement Reader View for freeCodeCamp /news Articles
• How to Add Copy Buttons to the Code Snippets
• How to Auto-generate a Table of Contents
• Key Takeways

How to Install Tampermonkey

For Chrome, the extension is available on the chrome web store.

It's also available for Firefox, which you can install from Firefox Add-Ons.

For other browsers, you can visit the Tampermonkey Home Page. Currently, Chrome, Firefox, Edge, Safari and Opera are officially supported. But the one from Chrome web store works fine on Chromium-based browsers like Brave.

How to Create a New Userscript

The easiest way to get started is using the create new script option from the toolbar.

You will be presented with something like this:

// ==UserScript==
// @name         New Userscript
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://www.freecodecamp.org/news/*
// @icon         https://www.google.com/s2/favicons?sz=64&domain=freecodecamp.org
// @grant        none
// ==/UserScript==

(function() {
    'use strict';

    // Your code here...
})();

The userscript header contains information about the script and other important parameters as well.

For now the relevant part is @match. This tells Tampermonkey for which sites to run the script. I have changed to match any article on the freeCodeCamp news site by using the "*" wildcard.

Test it out by putting something simple as alert("HI") on the function. Then navigate to any article on freecodecamp.org/news.

Lets do something interesting next. This will be a brief introduction to DOM manipulation.

Before writing code in the userscript, it's better to write out your code in the dev console first. Then, when you have working code, you can just paste it in the Tampermonkey script. This is how we will write all the scripts in this article as well.

How to Change a Button's Behaviour

At the end of each freeCodeCamp article there is a button that says "Tweet a thanks". You can see the onclick function of the button by using the element picker from the dev tools and clicking on that button.

Clicking on that button will open a new window in Twitter with some default text for a tweet like the following:

Thank you @twitter-username-of-author for writing this helpful article.

Title of Article
https://www.freecodecamp.org/news/slug-of-article

Let's say that you want to change up the default text to something like this:

This article is quite fascinating.

Title of Article by @twitter-username-of-author
https://www.freecodecamp.org/news/slug-of-article

The new text is static as you will chose it, but there are some variables here. You need to extract these 3 things:

1. The link of the article.
2. The Twitter username of the author.
3. Title of the article.

Extracting the link is as location.href.

Now how to extract the Twitter username?

At first glance, you may think the name after the author image is the answer. But that's actually the author's name (which is not necessarily unique) – it's not their Twitter username. So, where would you start looking for it?

Since you can just click the button or inspect it, you already know the values of all these variables. So a good way to find the Twitter username would be just to search for it in the entire HTML document. As it is unique, there won't be many occurrences.

Open the dev tools and search for the Twitter username. You will find this tag:

<meta name="twitter:creator" content="@username" />

Which is not surprising, as most publishing sites put information such article title, tags, description, and author information in the meta tags for SEO purposes. But most of the time you can just find what you are looking for by using the inspection tool.

So how do we extract the "content" property of this meta tag? Considering that an article is written by only one author, there will always be only one meta tag with name "twitter:creator". So you can just use the querySelector.

const username = document.querySelector('meta[name="twitter:creator"]').content

Now on to extracting the title. If you have wandered around looking for the Twitter username, you will find that there is also a meta tag with name "twitter:title". But this time, the title is something that you can visually see and inspect. In most cases that's the easier way to go about it.

If you inspect on the article title above the cover image, you will see it's inside an h1 with the class "post-full-title". You can select this with the following:

const title = document.querySelector("h1.post-full-title").textContent

Now we have all the pieces we need to make the change. Inspect the target button and you will see it comes with an id of "tweet-btn". Now let's define our text.

const text = `This article is quite fascinating.

${title} by ${username}
${location.href}
`

Putting it all together by changing the button's onclick, here's what you should have:

function change_tweet_text() {
    document.getElementById("tweet-btn").onclick = () => {
        const username = document.querySelector(
            'meta[name="twitter:creator"]'
        ).content
        const title = document.querySelector("h1.post-full-title").textContent

        const text = `This article is quite fascinating.

${title} by ${username}
${location.href}
`

        const share_link = encodeURI(
            `https://twitter.com/intent/tweet?text=${text}`
        )

        window.open(share_link, "share-twitter", "width=550, height=235")

        return false
    }
}

Because the text contains newline characters, we have to convert it url-encoded format.

Paste the function inside the userscript and call it.

;(function () {
    change_tweet_text()
})()

Now reload the article page and click on the button to see if it's working as expected.

It is possible that it won't work as expected. Especially if the page takes a lot of time to load.

With this script, we are modifying a button which is a DOM element. But what if the element is not yet loaded? Then you will face unexpected results.

Tampermonkey provides the "// @run-at" header which you can specify to "document-end", but I've found that it still produces unexpected results sometimes.

The right way to mitigate it is to use the "load" event which is emitted when the whole page is loaded. So, we can refactor our driver function in the following way:

addEventListener("load", main)

function main() {
    change_tweet_text()
}

You can be extra careful and throw a delay as well:

addEventListener("load", setTimeout(main, 2000))

From now on, we will call all the functions in this main function which will run after the load event is fired.

Note that you could have achieved the same result using a regular expression. In that case, you could extract the variables from the button's onclick which contains the URL with the full text.

But extracting information from the tags is better because if one day the default tweet text changes, you would need the change the regex as well.

How to Implement Reader View for freeCodeCamp /news Articles

If you make your browser fullscreen, the top navigation bar still stays at the top – which is not the best reading experience. So, we want to hide everything except the main article body.

Let's start inspecting. You will see that, close to the "main" tag, there is an "article" tag with class "post". It contains all the text of the article including the heading and cover image.

This time, we will live-edit the CSS of the page. In Firefox, you can use the "Style Editor" tab in the dev tools. Click on the "+" icon and start testing out your CSS.

As of now, there is no built-in way to hide all elements except one (in our case article.post) using CSS. You might be tempted to use :not(article.post) but it won't work because if any ancestor of the element is hidden, then all the descendents will be hidden as well.

We can accomplish our goal by using the ":has" selector. :has(article.post) will select all the ancestors of article.post so we can select the inverse of it with :not(:has(article.post)).

But there's still a problem: the descendent of article.post is also ignored by this selector. We can bring them back by chaining another not –:not(article.post, article.post *). This will select everything that is not an article.post or a descendent of it.

This is the final CSS:

:not(:has(article.post)):not(article.post, article.post *) {
    display: none;
}

At the time of writing this, the ":has" selector is experimental in Firefox. But you can go to about:config and enable it by changing the value of layout.css.has-selector.enabled to true.

What if you want to use the navigation bar? It would be a shame to manually write the CSS and remove it when you don't want it, right? Especially now that you can do some scripting.

We will add a keyboard shortcut to the site that will make the browser fullscreen. We'll also apply that CSS and, upon exiting fullscreen, the CSS will be removed.

One of the exposed APIs of tampermonkey is GM_addStyle(css) for adding CSS. For this API to work, you need to include // @grant GM_addStyle to the userscript header and also remove // @grant none if you have that.

To create the shortcut, you will use a "keydown" event listener.

function add_declutter_toggle(key) {
    const css = `
        .declutter :not(:has(article.post)):not(article.post, article.post *) {
            display: none;
        }`

    GM_addStyle(css)

    document.addEventListener("keydown", (event) => {
        if (event.key === key) {
            if (document.fullscreen) {
                document.exitFullscreen()
                document.body.classList.remove("declutter")
            } else {
                document.body.classList.add("declutter")
                document.documentElement.requestFullscreen()
            }
        }
    })
}

We are applying the CSS to the elements with the class "declutter". By toggling the "declutter" on the body we are essentially toggling the CSS on the entire page.

Now, we just have to add the function call in our main function.

function main() {
    change_tweet_text()
    add_declutter_toggle("F")
}

Reload the page and you will see that pressing "F" will toggle fullscreen and also remove everything except article body when in the fullscreen state.

How to Add Copy Buttons to the Code Snippets

The first step is always identifying the correct selector. If you inspect any code snippet you will see that they're contained in <code> blocks and have a <pre> as a parent. So, to select all code snippets, the selector can be pre > code.

Now the question is where to place the button? For easy access, we want the button to be on top of the code snippet and aligned to the right. You may think, placing it between the <pre> tag and <code> tag could work – but that will interfere with the content of the code snippet which is not ideal.

The ideal solution is wrapping the <pre> tag with a div and creating the structure as div > button > pre > code.

The copying is done by the clipboard API using the navigator.clipboard.writeText method. We will also apply some styling and let the reader know their copying was successful by changing the button text from "copy" to "copied" for a small duration of time.

function attach_code_copy_btn() {
    const css = `
        .copy-btn {
            float: right;
            margin-bottom: 5px;
            border-radius: 1rem;
            font-size: 0.8em;
            width: 7rem;
        }

        .pre-wrapper {
            width: 100%;
        }
    `
    GM_addStyle(css)

    const codes = document.querySelectorAll("pre > code")
    codes.forEach((code) => {
        const pre = code.parentElement
        const btn = document.createElement("button")
        btn.textContent = "copy"
        btn.classList.add("copy-btn")
        btn.addEventListener("click", () =>
            navigator.clipboard.writeText(code.textContent).then(() => {
                btn.textContent = "copied"
                setTimeout(() => (btn.textContent = "copy"), 2500)
            })
        )
        const div = document.createElement("div")
        div.classList.add("pre-wrapper")
        div.appendChild(btn)
        wrap(pre, div)
    })
}

function wrap(elem, wrapper) {
    elem.parentNode.insertBefore(wrapper, elem)
    wrapper.appendChild(elem)
}

The attach_code_copy_btn function finds all code snippets and wraps them around a div, adds classes to them, and attaches a button which will copy the code content on click.

Call the function just as before:

function main() {
    change_tweet_text()
    add_declutter_toggle("F")
    attach_code_copy_btn()
}

Heres some practice, try adding click event listeners to all <code> tags that are not inside a <pre> tag. On a mouse click on the <code> block the contents of the block will be copied.

How to Auto-generate a Table of Contents

freeCodeCamp is a great place for reading in-depth articles. The articles are well organized as well. Most authors use proper headings and provide a table of contents. But the ToC is not accessible on every part of the page.

If you want to jump somewhere you will have to go back to the ToC then click on your section of interest. And then there are shorter articles that don't provide a ToC at all.

That's where we can introduce a userscript that will automatically generate a table of contents which is accessible anywhere on the page. This improves the user experience when reading longer articles.

The first thing you should know is how a ToC works. A section in the ToC is just an anchor tag with an href value of #some-section. Here "some-section" can be the name property of an anchor tag of the target location (for example, <a name="some-section">) or if its used as an id on any element (for example, <h2 id="some-section">).

If you inspect on any heading tag, you will see that the id attribute is present. This is automatically inserted when the article page is built. So we can use that for our ToC.

This is a ToC generator function modified from a StackOverflow answer.

function generate_toc() {
    let toc = ""
    let level = 1

    let container = document.querySelector(".post-content")
    const regex = /<h([2-5])\s+id="([^"]+)">([^<]+)<\/h([2-5])>/gi
    const matches = [...container.innerHTML.matchAll(regex)]
    matches.forEach((match) => {
        if (match.length != 5) return
        const [_, openLevel, id, titleText, closeLevel] = match

        if (openLevel !== closeLevel) {
            return
        }

        if (openLevel > level) {
            toc += new Array(openLevel - level + 1).join("<ol>")
        } else if (openLevel < level) {
            toc += new Array(level - openLevel + 1).join("</li></ol>")
        } else {
            toc += new Array(level + 1).join("</li>")
        }

        level = parseInt(openLevel)

        if (!id) {
            id = titleText.replace(/ /g, "_")
        }
        toc += '<li><a href="#' + id + '">' + titleText + "</a></li>"
    })

    if (level) {
        toc += new Array(level + 1).join("</ol>")
    }

    return toc
}

We are using a regex to match h2, h3, h4, and h5 tags and creating an ordered list to display to ToC contents and returning the HTML of the whole ToC.

Now we will add this HTML to a div that will contain the ToC. That div will be wrapped inside a dialog component so that we can use a modal to easily show/hide the ToC. We will also add a button that will be used as the toggle.

function show_toc() {
    document.documentElement.style.overflow = "hidden"
    document.getElementById("toc-dialog").showModal()
}

function add_toc_toggle() {
    const dialog = document.createElement("dialog")
    dialog.setAttribute("id", "toc-dialog")
    const toc_div = document.createElement("div")
    toc_div.classList.add("toc-content")
    toc_div.innerHTML = "<h2>Table of Contents</h2>"
    dialog.appendChild(toc_div)

    const show_toc_btn = document.createElement("button")
    show_toc_btn.setAttribute("id", "toc-toggle")
    show_toc_btn.innerHTML =
        '<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M.361 256C.361 397 114 511 255 511C397 511 511 397 511 256C511 116 397 2.05 255 2.05C114 2.05 .361 116 .361 256zM192 150V363H149V150H192zM234 150H362V193H234V150zM362 235V278H234V235H362zM234 320H362V363H234V320z"></path></svg>'

    show_toc_btn.addEventListener("click", () => {
        show_toc()
    })

    // Click anywhere to close
    dialog.addEventListener("click", (event) => {
        document.documentElement.style.overflow = "auto"
        event.currentTarget.close()
    })

    dialog.addEventListener("close", (event) => {
        document.documentElement.style.overflow = "auto"
        event.currentTarget.close()
    })

    document.querySelector(".post-content").appendChild(dialog)
    document.querySelector(".post-content").appendChild(show_toc_btn)
}

Finally we will provide a driver function that will perform all the necessary tasks related to ToC generation. We will also add a keyboard shortcut to toggle the ToC.

function add_toc(toc_toggle_key) {
    const css = `
        .toc-content {
            margin: 0;
            padding: 0;
            max-width: 65vw;
        }

        @media (width <= 800px) {
            .toc-content {
                max-width: 100vw;
            }
        }

        .toc-content h2 {
            padding: 20px 30px;
        }

        .toc-content ol {
            counter-reset: item;
        }

        .toc-content li {
            display: flex;
            white-space: nowrap;
            gap: 5px;
        }

        .toc-content li:before {
            content: counters(item, ".") " ";
            counter-increment: item;
        }

        .toc-content a {
            text-decoration: none;
            white-space: normal;
            flex-grow: 1;
        }

        .toc-content li:hover {
            background: rgba(82, 142, 227, 0.3);
            cursor: pointer;
        }

#toc-toggle {
            background: transparent;
            position: fixed;
            bottom: 5%;
            right: 5%;
            width: 5rem;
        }

#toc-toggle svg {
            fill: rgb(82, 142, 227);
            scale: 1;
        }

#toc-toggle svg:hover {
            scale: 1.15;
        }

#toc-dialog {
            padding: 0;
            border: 0;
        }
`
    GM_addStyle(css)

    add_toc_toggle()
    document.querySelector(".toc-content").innerHTML += generate_toc()

    if (toc_toggle_key) {
        document.addEventListener("keydown", (event) => {
            if (event.key === toc_toggle_key) {
                show_toc()
            }
        })
    }
}

Now, just call the driver function in main.

function main() {
    change_tweet_text()
    add_declutter_toggle("F")
    attach_code_copy_btn()
    add_toc(";")
}

Now whenever, you read an article on freeCodeCamp, you can use the toggle button or a shortcut to view the ToC.

For practice, try making the toc sticky in the side (left/right). If you are on a wider screen, having the toc on the side will make the reading experience even better.

Key Takeways

You should never run arbitrary JavaScript on any occasion. That goes for Tampermonkey scripts as well.

I recommend going through every line of code before using a userscript. As you can already imagine, a userscript can drastically alter a website's appearance. If you are not careful enough, you might click a button that you expect to do one thing but it does something completely different, so always err on the side of caution.

When writing userscripts, keep these things in mind:

1. Always test your code in the dev console first and take care of all edge cases.
2. Prefer creating/manipulating DOM elements using built-in functions instead of modifying the innerHTML.
3. If you find any unexpected behaviour, try throwing a long delay and checking if that resolves the issue.

You can read my other articles on my blog. Find me on Twitter LinkedIn.

Regular expressions, also known as regex, work by defining patterns that you can use to search for certain characters or words inside strings.

Once you define the pattern you want to use, you can make edits, delete certain characters or words, substitute one thing for another, extract relevant information from a file or any string that contains that particular pattern, and so on.

Why Should You Learn Regex?

Regex let you to do text processing in a way that can save you a lot of time. It can also introduce some fun in the process.

Using regex can make locating information much easier. Once you find your target, you can batch edit/replate/delete or whatever processing you need to do.

Some practical examples of using regex are batch file renaming, parsing logs, validating forms, making mass edits in a codebase, and recursive search.

In this tutorial, we're going to cover regex basics with the help of this site. Later on, I will introduce some regex challenges that you'll solve using Python. I'll also show you how to use tools like sed and grep with regex.

Like many things in life, regular expressions are one of those things that you can only truly understand by doing. I encourage you to play around with regex as you are going through this article.

Table of Contents

• Regex Basics
  • Exact match
  • Character set
    • Match ranges in regex
    • Match any character not in the set
  • Character classes
  • Quantifiers
  • Capture groups
    • How to use logical OR in regex
    • How to reference capture groups
    • How to name capture groups
• How to use regex with command line tools
  • Recursive regex search with grep
  • Substitution with sed
• Advanced Regex: Lookarounds
  • Lookbehinds
  • Lookaheads
• Practical Examples of Regex
  • Logs parsing
  • Bulk File Renaming
  • Email validation
  • Password constraints
• Final words

Regex Basics

A regular expression is nothing but a sequence of characters that match a pattern. Besides using literal characters (like 'abc'), there are some meta characters (*,+,? and so on) which have special purposes. There are also features like character classes which can help you simplify your regular expressions.

Before writing any regex, you'll need to learn about all the basic cases and edge cases for the pattern you are looking for.

For instance, if you want to match 'Hello World', do you want the line to start with 'Hello' or can it start with anything? Do you want exactly one space between 'Hello' and 'World' or there can be more? Can other characters come after 'World' or should the line end there? Do you care about case sensitivity? And so on.

These are the kind of questions you must have the answer to before you sit down to write your regex.

Exact match

The most basic form of regex involves matching a sequence of characters in a similar way as you can do with Ctrl-F in a text editor.

On the top you can see the number of matches, and on the bottom an explanation is provided for what the regex matches character by character.

Character set

Regex character sets allow you to match any one character from a group of characters. The group is surrounded by square brackets [].

For example, t[ah]i matches "tai" and "thi". Here 't' and 'i' are fixed but between them can occur 'a' or 'h'.

Match ranges in regex

Sometimes you may want to match a group of characters which are sequential in nature, such as any uppercase English letter. But writing all 26 letters would be quite tedious.

Regex solves this issue with ranges. The "-" acts as a range operator. Some valid ranges are shown below:

You can also specify partial ranges, such as [b-e] to match any of the letters 'bcde' or [3-6] to match any of the numbers '3456'.

You are not limited to specifying only one range inside a character set. You can use multiple ranges and also combine them with any other additional character(s). Here, [3-6u-w;] will match any of '3456uvw' or semicolon ';'.

Match any character not in the set

If you prefix the set with a '^', the inverse operation will be performed. For example, [^A-Z0-9] will match anything except uppercase letters and digits.

Character classes

While writing regex, you'll need to match certain groups such as digits quite often and multiple times in the same expression as well.

So for example, how would you match a pattern like 'letter-digit-letter-digit'?

With what you've learned up until now, you can come up with [a-zA-Z]-[0-9]-[a-zA-z]-[0-9]. This works, but you can see how the expression can get quite messy as the pattern length gets bigger.

To make the expression simpler, classes have been assigned to well-defined character groups such as digits. The following table shows these classes and their equivalent expression with character sets:

Character classes are quite handy and make your expressions much cleaner. We will use them extensively throughout this tutorial, so you can use this table as a reference point and come back here if you forget any of the classes.

Most of the time, we won't care about all the positions in a pattern. The "." class saves us from writing all possible characters in a set.

For example, t.. matches anything that starts with t and any two characters afterwards. This may remind you of the SQL LIKE operator which would use t%% to accomplish the same thing.

Quantifiers

The word "pattern" and "repetition" go hand in hand. If you want to match a 3 digit number you can use \d\d\d. But what if you need to match 11 digits? You could write '\d' 11 times, but a general rule of thumb while writing regex or just doing any kind of programming is that if you find yourself repeating something more than twice, you are probably unaware of some feature.

In regex, you can use quantifiers for this purpose. To match 11 digits, you can simply write the expression \d{11}.

The table below lists the quantifiers you can use in regex:

In this example, the expression can\s+write matches can followed by 1 or more whitespaces followed by write. But you can see 'canwrite' is not matched as \s+ means at least one whitespace needs to be matched. This is useful when you are searching through text which is not trimmed.

Can you guess what can\s?write will match?

Capture groups

Capture groups are sub-expressions enclosed in parentheses (). You can have any number of capture groups, and even nested capture groups.

The expression (The ){2} matches 'The ' twice. But without a capture group, the expression The {2} would match 'The' followed by 2 spaces, as the quantifier will be applied on the space character and not on 'The ' as a group.

You can match any pattern inside capture groups as you would with any valid regex. Here (is\s+){2} matches if it finds 'is' followed by 1 or more spaces twice.

How to use logical OR in regex

You can use "|" to match multiple patterns. This is (good|bad|sweet) matches 'This is ' followed by any of 'good' or 'bad' or 'sweet'.

Again, you must understand the importance of capture groups here. Think about what the expression This is good|bad|sweet would match?

With a capture group, good|bad|sweet is isolated from This is. But if it's not inside a capture group, the entire regex is only one group. So the expression This is good|bad|sweet will match if the string contains 'This is good' or 'bad' or 'sweet'.

How to reference capture groups

Capture groups can be referenced in the same expression or while performing replacements as you can see on the Replacement tab.

Most tools and languages allow you to reference the nth captured group with '\n'. In this site '$n' is used while referencing on replacement. The syntax for replacement will vary depending on the tools or language you're using. For JavaScript, for example, its '$n', while for Python its '\n'.

In the expression (This) is \1 power, 'This' is captured and then referenced with '\1', effectively matching This is This power.

How to name capture groups

You can name your capture groups with the syntax (?<name>pattern) and backreference them in the same expression with \k<name>.

On replacement, referencing is done by $<name>. This is the syntax for JavaScript and can vary among languages. You can learn about the differences here. Also note that this feature might not be available in some languages.

In the expression (?<lang>[\w+]+) is the best but \k<lang> .*, the pattern [\w+]+ is captured with the name 'lang' and backreferenced with \k<lang>. This pattern will match any word character or '+' character 1 or more times. The .* at the end of the regex matches any character 0 or more times. And finally on replacement, the referencing is done by $<lang>.

How to Use Regex with Command Line Tools

There are good CLI tools available that let you perform regex from your terminal. These tools save you even more time as you can easily test different regex without writing code in some langauge and then compiling or interpreting it.

Some of the well-known tools are grep, sed, and awk. Let's look at a few examples to give you some ideas on how you can leverage these tools.

Recursive regex search with grep

You can execute the power of regex through grep. Grep can search patterns in a file or perform recursive search.

If you are on Windows, you can install grep using winget. Run this command in powershell:

winget install -e --id GnuWin32.Grep

I will show you the solution to a challenge I created for a CTF competition at my university.

The file attached to the challenge is a zip file that contains multiple levels of directories and a lot of files in it. The name of the competition was Coderush with flag format coderush{flag is here}. So you have to search for the pattern coderush{.*} which will match the flag format coderush{any character here}.

Unzip the file with unzip ripG.zip and cd into it with cd ripG.

There are 358 directories and 8731 files. Instead of searching the pattern in the files one by one, you can employ grep like this:

grep --color -R "coderush{.*}"

The "-R" flag enables recursive search.

You can learn more about grep and its command line options here

Substitution with sed

You can use sed to perform insertion, deletion, substitution on text files by specifying a regex. If you are on windows, you can get sed from here. Or if you use WSL, tools like grep and sed will already be available.

This is the most common usage of sed:

sed 's/pattern/replacement/g' filename
echo "${text}" | sed 's/pattern/replacement/g'

Here, the option "g" is specified to replace all occurrences.

Some other useful options are -n to suppress the default behaviour of printing all lines and using p instead of g to print only the lines which are affected by the regex.

Let's take a look at the content of texts.txt.

Hello rand chars World 56 rand chars
Henlo 52 rand chars W0rld rand chars
GREP rand chars Henlo 62 rand chars
Henlo 10 rand chars Henlo rand chars
GREP rand chars Henlo 45 rand chars

Our task is replacing Henlo number with Hello number only in the lines where "GREP" is present. So, we are searching for the pattern Henlo ([0-9]+) which will match 'Henlo ' followed by 1 or more digits and all the digits are captured. Then our replacement string will be Hello \1 – the '\1' is referencing the capture group containing the digits.

One way to accomplish that would be using grep to grep the lines which have "GREP" present then perform the replacement with sed.

grep "GREP" texts.txt | sed -En 's/Henlo ([0-9]+)/Hello \1/p'

The "-E" option enables extended regex without which you would need to escape the parentheses.

Or you could just use sed. Use /pattern/ to restrict substitution on only the lines where pattern is present.

sed -En '/GREP/ s/Henlo ([0-9]+)/Hello \1/p' texts.txt

Advanced Regex: Lookarounds

Lookaheads and Lookbehinds (together known as lookarounds) are features of regex that allow you to check the existence of a pattern without including it in the match.

You can think of them as zero width assertions – they assert the existence of a pattern but do not consume any characters in the match. These are very powerful features, but they're also computationally expensive. So make sure you keep an eye on performance if you are using them often.

Lookbehinds

Let's say you want to match the word 'linux', but you have 2 conditions.

1. The word 'GNU' must occur before 'linux' occurs. If a line contains 'linux' but doesn't have 'GNU' before it, we want to discard that line.
2. We want to match only linux and nothing else.

We already know how to satisfy the 1st condition. GNU.* will match 'GNU' followed by any number of characters. Then finally we match the word linux. This will match all of GNU-any-characters-linux.

But how do we prevent matching GNU.* while still maintaining the 1st condition?

That's where a positive lookbehind comes in. You can mark a capture group as a positive lookbehind by prefixing it with ?<=. In this example, the expression becomes (?<=GNU.*)linux.

Now only linux is matched and nothing else.

Note that the expressions (?<=GNU.*)linux and linux(?<=GNU.*) will behave exactly the same. In the 2nd expression, although linux is before the lookbehind, there is .* after 'GNU' which matches linux. This means it satisfies the lookbehind.

To make it simpler, think about the pattern without the lookbehind. The pattern GNU.* will match 'GNU' and anything after it, in our case matching linux.

Now we can derive a generalized statement that the expression (?<=C)X will match the pattern X – only if pattern C came before X (and C must not be included in the match).

You can also reverse the 1st condition. Match lines that contains the word linux only if GNU never came before it. This is called a negative lookbehind. The prefix in this case is ?<!. The inverse of the previous expression would be (?<!GNU.*)linux.

Lookaheads

Lookaheads are also assertions like lookbehinds, as you saw in the previous example. The only difference is that lookbehinds make an assertion before and lookaheads makes assertion after.

Let's say you have these two conditions:

1. Match Hello only if World comes somewhere after it.
2. Match only Hello and nothing else.

The prefix for a positive lookahead is ?=. The expression Hello(?=.*World) will meet both conditions. This is similar to Hello.*World except that only Hello will be matched whereas Hello.*World will match 'Hello', 'World' and anything in between.

Similar to the example in a positive lookbehind, the expressions Hello(?=.*World) and (?=.*World)Hello are equivalent. Because the .* before 'World' matches Hello, satisfying the 1st condition.

A negative lookahead is just the complement of a negative lookbehind. You can use it by prefixing it with ?!. (?!World)Hello will match Hello only if there is no World anywhere after it.

Here is a summary of the syntax for lookarounds when you want to match the pattern X with assertion C.

Practical Examples of Regex

Logs parsing

In this log file, these are the lines which we care about:

[1/10000] Train loss: 11.30368, Valid loss: 8.95446, Elapsed_time: 7.58941
[500/10000] Train loss: 0.96180, Valid loss: 0.20098, Elapsed_time: 82.48651
[1000/10000] Train loss: 0.04051, Valid loss: 0.11927, Elapsed_time: 156.86243

Our task is to extract the training loss and validation loss for purposes such as plotting loss over the epochs. We need to extract the training loss values like 11.30368, 0.96180, 0.04051 and put them in an array.

All the relevant values are prefixed with 'Train loss:', so we can use this in our regex as it is. To match the float numbers we have to match some digits followed by a "." and then followed by more digits. You can do this with \d+\.\d+. Because we want to keep track of these numbers, they should be inside a capture group.

As "." has special purpose in regex, when you want to match a "." character you have to escape it with a backslash. This is applicable for all characters with a special purpose. But you dont have to escape it inside a character set.

Putting it altogether, the expression for extracting training loss is Train loss: (\d+\.\d+). We can use the same logic to extract validation loss with Valid loss: (\d+\.\d+).

Here is one way to extract this information using Python:

import re

f = open("log_train.txt", "r").read()

train_loss = re.findall(r'Train loss: (\d+\.\d+)', f)
valid_loss = re.findall(r'Valid loss: (\d+\.\d+)', f)

train_loss = [float(i) for i in train_loss]
valid_loss = [float(i) for i in valid_loss]

print("train_loss =", train_loss)
print("")
print("valid_loss =", valid_loss)

When there is one capture group, re.findall searches all the lines and returns the values inside the capture group in a list.

Any regex function only return strings, so the values are converted to floats and printed out. Then you can directly use them in another Python script as a list of floats.

This is the result:

You could also use sed, save the output in train_losses.txt, and read from the file. First we use '/Train/' to target only the lines with 'Train' present then we are applying the same regex as before.

sed -En '/Train/ s/.*Train loss: ([0-9]+\.[0-9]+).*/\1/p' log_train.txt | tee train_losses.txt

".*" is added at the start and end so that sed matches the contents of all the relevant lines. Then the entire line is replaced by the value of the capture group. The tee command is used to redirect the output of sed into train_losses.txt while also printing the contents in the terminal.

Take a moment to think about what would you need to extract the epochs. You have to extract 500 from [500/10000] for all such lines. The array should look like [1, 500, 1000, 1500, ...]. You can follow the same approach as we used for the previous example.

Note that if you want to match "[" or "]", you have to escape it. The answer is given here.

Bulk File Renaming

You have these files with some random values as prefixes. You have to rename all files as 1.mp4, 2.mp4 and so on. This is how the files were generated.

This is a common scenario where you have a list of files which have their sequence number in the name but there are also some other characters that you don't want.

The pattern has to match anything up to Episode then an underscore and then the number and .mp4 at the end.

The relevant value is the number before '.mp4' which we will put inside a capture group. .*Episode_ will match everything up to the number. Then we can capture the number with ([0-9]+) and also match .mp4 with \.mp4.

So the final regex is .*Episode_([0-9]+)\.mp4. As we want to keep the .mp4 the replacement string will be \1.mp4.

This is one way to solve it using sed.

for i in *.mp4; do
    newname=$(echo $i | sed -En 's/.*Episode_([0-9]+)\.mp4/\1.mp4/p')
    mv $i $newname
done;ls

First the new name is saved in a variable and then the mv command is used to rename the file.

Could we have just used .* in place of .*Episode_ ? In this example, yes. But there might be filenames such as Steins_Gate0.mp4 where the 0 is part of the movie name and you didn't really want to rename this file so its always better to be as specific as possible.

What if some files were named as "Random_Episode6.mp4"? The difference being, there is no underscore after Episode. What change will you need to make?

The answer is that you'll need to add a "?" after the "_" to make it optional. The regex will be .*Episode_?([0-9]+)\.mp4.

Email validation

There are all sorts of complicated regex for validating email.

Here is a simple one: ^[^@ ]+@[^@.]+\.\w+$. It matches the format A@B.C

The table below breaks down this pattern into smaller pieces:

In the regexr site, you can enable the multline flag from the Flags tab in the upper right corner. The 'gm' at the end indicates that the multiline flag is enabled.

We can see that line 2,3,5,6 didn't match. Can you find out the reason and which part of the regex is responsible for disqualifying it?

The answer is given here

Password constraints

You can also use regex to impose constraints. Here we will uncover the power of positive lookaheads.

Lets say we want to accept a string only if there is a digit in it. You already know how to find a digit with the '\d' class. To accomplish that, we can use [^\d]*\d. This will match any non-digit character 0 or more times and then match a digit.

We can also use the expression .*\d to match one digit. So if there is no digit in the string then the lookahead will fail and the none of the characters of that string will be matched, returning an empty string "".

When we are using a programming language, we can check if the regex returned an empty string and determine that the constraints are not satisfied.

We will create a regex which imposes the following criteria:

1. Minimum 8 characters and maximum 16 characters.
2. At least one lower case letter.
3. At least one upper case letter.
4. At least one number.

To achieve this, you can use positive lookaheads. This is the regex:

^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{8,16}$

The table below explains which part of the regex imposes which constraint:

What modification you would need for imposing at least 5 upper case letters?

You may think (?=.*[A-Z]{5,}) will do the job. But this expression requires all the 5 letters to be together. A string like rand-ABCDE-rand will be matched but 0AxBCDxE0 will not be matched even though it has 5 upper case letters (as they are not adjacent).

Yet again, we have capture groups coming to the rescue. We want to match 5 uppercase letters anywhere in the string. We already know that we can match 1 uppercase letter with .*[A-Z]. Now we will put them inside a capture group and attach a quantifier of minimum 5. The expression will be (.*[A-Z]){5,}.

Here is the final answer:

In place of (?=.*[A-Z]) you will need (?=(.*[A-Z]){5,}). The expression becomes ^(?=.*[a-z])(?=(.*[A-Z]){5,})(?=.*\d).{8,16}$.

You could also require that the password not contain certain words to enforce stronger passwords.

For example, we want to reject the password if contains pass or 1234. Negative lookaheads is the tool for this job. The regex would be ^(?!.*(pass|1234)).*$.

In this regex, we put pass and 1234 inside a capture group and used the logical OR operator. This capture group is nested inside another capture group which is prefixed with ?!.*. This makes it a negative lookahead that matches if there are at least 8 characters by .{8,} with the condition that, pass or 1234 can't be present anywhere in the string.

Final Words

I hope you got a good amount of practice while going through this article. It's ok if you forget some syntax. What's important is understanding the core concepts and having a good idea of what's possible with regex. Then, if you forget a pattern, you can just google it or reference a cheatsheet.

The more you practice, the more you will get by without outside help. Eventually you will be able write super complex and effective regex completely offline.

There are already some good regex cheatsheets out there, so I wanted to create something more in-depth here that you can reference for the core concepts and common use cases.

If you're looking for a cheatsheet, the one from QuickRef is helpful. It's a good place to recall the syntax and they also provide some basic overview of regex related functions in various programming languages.

Most regex techniques are the same in all programming languages and tools – but certain tools might offer additional features. So do some research on the tool you are using to pick the best one for you.

My final suggestion would be not to force using regex just because you can. A lot of the times a regular string.find() is enough to get the job done. But if you live in the terminal, you really can do a lot just with regex for sure.

If you like this type of article, you may keep an eye on my blog or twitter.