by Ty Irvine

How to List Out All URLs Associated With a Website Fast-ish

So you need a list containing all the URLs for a website? Are you doing some redirects perhaps? Hit the limit on XML Sitemaps? Cool, me too. I’ve got just the tool for you that’ll get it done at about the same speed as XML Sitemaps, but you’ll look way cooler doing it.

Where the tutorial actually starts

To get your list of URLs, we’re going to use Wget!

What the Frigg is Wget?

“Wget is a free software package for retrieving files using HTTP, HTTPS, and FTP, the most widely-used Internet protocols.” — Brew ‍Formulas

And you can also use it to request a big list of URLs associated with a domain.‍

‍1. Installing Wget

To install Wget if you haven’t already, you’re going to need first to install HomeBrew; aka Brew. ? Brew is a package manager, meaning it installs software for you and manages it. You can check out the instructions on their website or just follow the ones below.

Install Brew

Paste this into a Terminal Prompt and hit enter twice ⮐ (It may ask you for a password.)

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"‍

Install Wget

Now that you have Brew installed it’s time to install Wget. Paste this into a Terminal Prompt and hit enter ⮐

brew install wget

2. Time To Get ‘Dem URLs‍

Now with Wget installed we simply download the website and then display all of its URLs. Start by downloading the website you’d like with

Wget -r www.shutterandcode.com

Then once the download is complete we’ll list out the URLs with

Find www.shutterandcode.com

(Make sure to use the same website domain as what was downloaded).

Conclusion

After a series of casual tests pitting Wget against XML Sitemaps using smaller websites, I found that they are both pretty much on par with each other. Occasionally one would be faster than the other but overall they both had similar speeds.‍

If you’d like to know more about Wget commands simply type this into your prompt

wget --help

I hope you enjoyed reading this! Don’t forget to like, comment, and subscribe! ?

p.s. don’t actually feel obligated to like, comment, and or subscribe because it is simply a joke for YouTubers :)

UPDATE: if you don’t want the site actually to download to your computer add in ‘ — spider’ after ‘wget’ like

wget -r --spider www.example.com

Check out the original post and the rest of the Snippets! series at

Shutter&Code — Le Blog