Crawl a website with limits and path filters

A crawl starts at one URL, follows links inside the bounds you set, and scrapes each page. You get a job id at once and read the pages while the job runs. This guide shows how to set the bounds, poll the job, read failures and estimate the cost.

Start a crawl

POST /api/v1/crawl returns 201 with the job and credits: 0. Starting a crawl is free.

curl -X POST https://api.wayfern.com/api/v1/crawl \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 200,
    "maxDepth": 3,
    "includePaths": ["^/docs/"],
    "excludePaths": ["^/docs/changelog/"],
    "scrapeOptions": { "formats": ["markdown"], "onlyMainContent": true }
  }'

The job starts as queued, then moves to scraping. It ends as completed, failed or cancelled.

Set the bounds

Field	Default	What it does
limit	50	Most pages to scrape, from 1 to 5,000.
maxDepth	3	Most link hops from the start URL, from 0 to 10.
includePaths	none	Regular expressions. A URL must match one to be crawled.
excludePaths	none	Regular expressions. A URL that matches one is skipped.
allowSubdomains	false	Follow links onto subdomains of the start host.
allowExternalLinks	false	Follow links to other sites.
useSitemap	true	Add the URLs from the site’s sitemaps to the queue.
respectRobots	true	Read robots.txt and obey its crawl-delay.
delay	0	Milliseconds between batches of requests, from 0 to 60,000.
concurrency	3	Pages fetched at the same time, from 1 to 5.
scrapeOptions	scrape defaults	The scrape body without url. Each page uses it.

Some details change what you get:

• Path patterns test the path and the query string. They are not anchored, so start a pattern with ^ to match from the start of the path. Each list takes up to 50 patterns. A pattern that does not compile is a 400.
• maxDepth also limits the path depth. A URL can have at most maxDepth path segments below the path of the start URL. Sitemap URLs enter the queue at depth 1.
• respectRobots uses robots.txt only for its crawl-delay. When the crawl-delay is longer than delay, the crawl-delay wins. Disallowed paths are not filtered, so set the scope with includePaths and excludePaths.
• Each page uses one of the 6 Web Data slots of your account. When other calls hold the slots, a page fails with rate_limited.

Poll the job and read pages

GET /api/v1/crawl/:id returns the job and a window of pages, oldest first. skip defaults to 0. limit defaults to 25, with a maximum of 100. Polling is free.

import os
import time

import requests

API = "https://api.wayfern.com/api/v1/crawl"
auth = {"Authorization": f"Bearer {os.environ['WAYFERN_API_TOKEN']}"}

start = requests.post(
    API,
    headers=auth,
    json={"url": "https://docs.example.com", "limit": 200, "includePaths": ["^/docs/"]},
    timeout=60,
)
start.raise_for_status()
job = start.json()["data"]

pages = []
while True:
    res = requests.get(
        f"{API}/{job['id']}",
        headers=auth,
        params={"skip": len(pages), "limit": 100},
        timeout=60,
    )
    res.raise_for_status()
    body = res.json()["data"]
    pages.extend(body["data"])
    job = body["job"]
    if job["status"] not in ("queued", "scraping") and not body["data"]:
        break
    time.sleep(3)

print(job["status"], job["completed"], job["failed"], job["creditsUsed"])

Read failures and cancel

GET /api/v1/crawl/:id/errors lists failed pages. Each entry has url, error, message and at. error is a sentinel such as not_found or blocked. The job keeps the first 50 errors. The failed counter counts all of them.

DELETE /api/v1/crawl/:id cancels a running job. The pages it already crawled stay readable and stay billed.

A job can also end without your action:

• When your balance or your monthly spend limit cannot pay for a page, the job ends as failed. error says which limit stopped it.
• A job that runs longer than 6 hours ends as failed.
• When the service restarts during a job, the job ends as cancelled.

Results and errors stay readable until expiresAt, 7 days after the start.

What a crawl costs

Each successful page costs 1 credit. The scrape add-ons apply per page: summary adds 2, and screenshot, pdf or json adds 4. A failed page costs nothing. A page served from the cache is billed like a live page. Residential bandwidth costs 320 credits per GB.

Before the job starts, Wayfern checks that you can pay for one page. It does not reserve the full limit. creditsUsed on the job is the running total.

Crawl starts allow 10 requests a minute, and polling allows 240. To get pages pushed to you, add a webhook: see Crawl webhooks. To count the URLs first, see Map a site. The full reference is at Crawl.