Sometimes you might want to extract data from a website into a different format. But what if the website has no easy way to export that data? This is where web scraping comes in.
We've released a crash course on the freeCodeCamp.org YouTube channel that will teach you how to perform web scraping using the Python library Beautiful Soup.
This course was developed by Jim Ergin from JimShapedCoding. Jim has been using and teaching Python for many years.
Beautiful Soup will allow you to gather any information you want from any website you want. It could be a bank website, social media, Wikipedia, or any other website.
Beautiful Soup provides methods for navigating, searching, and modifying a parse tree. It makes it easier to dissect an HTML document and extract the data you need. And it doesn't take much code to write an application.
In this course, you will first learn how to scrape a basic HTML page just to learn the concepts. Then, you will move on to scraping a real website. At the end, you will learn how to store the information you scrape from the website.
Here are the topics covered in this course:
- Basic HTML Structure, HTML Tags Explanation
- Packages installation
- Scraping local files
- Beautiful Soup find & find_all() methods
- Web browser inspect tool
- Grabbing prices in a basic web scraping project
- Using the requests library to see a website's HTML
- Scraping a production website
- Looping through similar soup.find_all() objects
- Filtering scraped jobs
- Setting up a project to scrape every 10 minutes
- Storing the jobs paragraph in text files
Watch the full crash course on the freeCodeCamp.org YouTube channel (1-hour watch).