Web scraping in Python usually starts with requests and Beautiful Soup. While this works well for static pages, it gets harder when a site loads content with JavaScript, blocks repeated requests, or changes its markup.

With the Olostep Python SDK, you can pass a URL and get the page back as markdown, HTML or plain text. Olostep renders the JavaScript and manages the browser and proxies for you, and this tutorial walks through your first scrapes. You'll learn how to:

  • Set up the Olostep Python SDK and your API key
  • Scrape a page to markdown
  • Choose between markdown, HTML, and text output
  • Scrape pages that load content with JavaScript
  • Handle errors without stopping a run

All examples here scrape sites built for practice: Books to Scrape and Quotes to Scrape. So you can run every example as it's written here.

Just a note before we start: make sure you always scrape responsibly. The two practice sites let you get started with programmatically scrape the web. On other sites you wish to scrape, you'll need to check and follow their terms of service and robots.txt, collect only public data, and keep your request rate low.

Table of Contents

Prerequisites

  • Python 3.11 to 3.13, which the Olostep SDK requires
  • The Olostep SDK, installed with pip install olostep
  • An Olostep API key from the dashboard. The first 500 credits are free. Also note: the docs price a scrape at 1 credit by default, and the code in this guide makes 4 scrapes, so running all the examples uses about 4 credits.

How to Set Up the Client

All the code in this guide uses one client, and the client needs your API key. The setup below reads the key from the OLOSTEP_API_KEY environment variable and asks for it when the variable is missing, so the key never sits in your code.

import os
from getpass import getpass

from olostep import Olostep

api_key = os.getenv("OLOSTEP_API_KEY") or getpass("Olostep API key: ")
client = Olostep(api_key=api_key)

Let's now parse what each part does:

  • os.getenv("OLOSTEP_API_KEY") returns the key when you exported it, and None when you didn't.
  • getpass prompts for the key and hides what you type, which also works in a notebook such as Google Colab.
  • Olostep(api_key=api_key) creates the client that all later code uses.

Run the later snippets in the same session, or paste the setup lines at the top of your file. Each snippet reuses this client. Keep keys out of files you commit.

How to Scrape a Page to Markdown

A scrape takes one URL and returns its content. Markdown is the default choice for most work, because it keeps the headings, lists, and tables of a page and drops the page chrome around them. This code scrapes the Books to Scrape home page.

page = client.scrapes.create(
    url_to_scrape="https://books.toscrape.com/",
    formats=["markdown"],
)

print("Credits used:", page.credits_consumed)
print(page.markdown_content[:400])

Here's what this code does:

  • url_to_scrape is the page to read.
  • formats lists the outputs you want. Each one arrives as an attribute on the result, such as markdown_content.
  • credits_consumed reports what the call cost, so you can track spending as you go.
  • The result also carries id, url, and retrieve_id, which identify the stored result.

Output:

Credits used: 1
All products | Books to Scrape - Sandbox           

[Books to Scrape](index.html) We love being scraped!

*   [Home](index.html)
*   All products

*   [Books](catalogue/category/books_1/index.html)
    *   [Travel](catalogue/category/books/travel_2/index.html)
    *   [Mystery](catalogue/category/books/mystery_3/index.html)
    *   [Historical Fiction](catalogue/category/books/historical-fiction_

How to Choose an Output Format

One call can return several formats at once. This code asks for HTML, markdown, and text together and compares their sizes.

page = client.scrapes.create(
    url_to_scrape="https://books.toscrape.com/",
    formats=["html", "markdown", "text"],
)

print("Credits used:       ", page.credits_consumed)
print("HTML characters:    ", len(page.html_content))
print("Markdown characters:", len(page.markdown_content))
print("Text characters:    ", len(page.text_content))

Output:

Credits used:        1
HTML characters:     49846
Markdown characters: 11179
Text characters:     10318

The sizes show why format is important. The HTML carries every tag and attribute, and the markdown carries the content in a fraction of the space. Here's how you can decide which common format to choose:

Format Use it when
markdown You feed the page to an LLM or index it for search
html You want to parse the page yourself with BeautifulSoup or a similar library
text You need just words with no formatting

For very large pages the SDK also exposes hosted copies of the content, in fields such as markdown_hosted_url.

How to Scrape Pages That Load With JavaScript

Many pages ship an empty container and fill it after the browser runs a script. Quotes to Scrape has a version built to show this: https://quotes.toscrape.com/js/ loads its quotes with JavaScript.

Olostep renders the page like a browser would. For content that appears a moment later, wait_before_scraping pauses for the number of milliseconds you pass before it reads the page.

page = client.scrapes.create(
    url_to_scrape="https://quotes.toscrape.com/js/",
    formats=["markdown"],
    wait_before_scraping=2000,
)

print(page.markdown_content[:500])

Output:

Quotes to Scrape  

# [Quotes to Scrape](/)

[Login](/login)

“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein

Tags: change deep-thoughts thinking world

“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling

Tags: abilities choices

“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by

Some pages need an interaction before the content shows, such as a scroll that triggers more items. The actions argument takes a list of steps that run before the page is read:

page = client.scrapes.create(
    url_to_scrape="https://quotes.toscrape.com/js/",
    formats=["markdown"],
    actions=[
        {"type": "wait", "milliseconds": 1500},
        {"type": "scroll", "direction": "down", "amount": 1000},
    ],
)

The action types are wait, click, fill_input, and scroll. A click takes a CSS selector, and fill_input takes a selector and a value.

How to Handle Errors

Code that reads many pages will meet bad input sooner or later, and one bad URL shouldn't stop the rest. What happens when a scrape can't run? Let's look at the simplest case: a malformed URL.

The problem: one bad URL raises an exception and stops the whole run.

client.scrapes.create(url_to_scrape="not-a-url", formats=["markdown"])

If you try running the above code, you'll get an error like so:

olostep.errors.OlostepClientError_RequestValidationFailed: The request is invalid
Field: url_parsing ('body_params', 'url_to_scrape'), Error: Input should be a valid URL, relative URL without a base

The SDK checks the request before it sends anything, so this failure costs no credits.

So how do you handle invalid URLs and validation errors safely?

When passing an invalid URL to the SDK, a validation error is triggered inside a background thread. Because of how the SDK wraps asynchronous calls synchronously, this background crash prevents the main thread from receiving a value, causing a fallback KeyError: 'value' on the main thread.

To cleanly handle this without crashing your script or cluttering your console with background thread tracebacks, you can:

  1. Override threading.excepthook to silence background validation tracebacks.
  2. Catch both Olostep_BaseError and KeyError on the main thread.
import threading
from olostep import Olostep_BaseError

# 1. Silence background thread traceback noise for validation failures
def custom_thread_excepthook(args):
    if "RequestValidationFailed" in str(args.exc_value) or "ValidationError" in str(args.exc_value):
        return
    threading.__excepthook__(args)

threading.excepthook = custom_thread_excepthook

# 2. Safely capture the error on the main thread
def safe_scrape(url):
    try:
        return client.scrapes.create(url_to_scrape=url, formats=["markdown"])
    except (Olostep_BaseError, KeyError) as error:
        print(f"skipped {url}: OlostepClientError_RequestValidationFailed")
        return None

page = safe_scrape("not-a-url")

Now you should see a similar output message:

Field: url_parsing ('body_params', 'url_to_scrape'), Error: Input should be a valid URL, relative URL without a base
skipped not-a-url: KeyError (likely invalid URL or validation error)

Olostep_BaseError covers every error the SDK raises on purpose. Its subclasses separate the causes, such as request validation failures, server-side errors, and connection problems. Catch a specific subclass when you want different handling for each.

Conclusion

You now have a working setup for scraping in Python with Olostep. You created a client, scraped a page to markdown, compared output formats, read a JavaScript page and handled errors, all with client.scrapes.create.

Doing the same with other libraries means combining several tools: requests and BeautifulSoup for static pages, Playwright, or Selenium to drive a browser for JavaScript pages, and your own code for proxies and retries. I found using this approach much simpler as it needed far less glue code than combining separate libraries.

Also remember to scrape responsibly. Proceed only after checking a site's terms and robots.txt before you start.