Rate limits and concurrency

Wayfern has two kinds of limits. Concurrency limits count the work that runs at the same time for your account. Rate limits count requests per minute to each route. This guide gives the numbers and shows how to read a 429.

Open browser pages per plan

The plan caps how many browser pages your account can keep open at the same time, across all connections.

Plan	Pages open at once	Seats
Developer	20	1
Business	100	10
Scale	400	50
Enterprise	up to 1,000	up to 1,000
Credit packs, no plan	20	1
No plan and no packs	0	1

When you open a page past the cap, the CDP command fails with error -32000 and the message Concurrent session limit reached (max N open pages). The pages you already have keep running. A higher cap does not cost more. You pay for the time pages are open.

Each browser link also caps its connections, from 1 to 100, with a default of 10. A connection past that cap closes with code 4003.

Work in progress per account

These caps are the same on every plan.

What	At once	Past the cap
Web Data requests	6	The next request fails at once with 429 rate_limited.
Open captcha tasks	5	The next solve fails at once with 429 rate_limited. A task is open until it is ready or fails.
Operation calls to one site	2 a second per project, 20 a second for all projects	A call waits up to 10 seconds for its turn, then fails with reason rate.

Each crawled page uses one of the 6 Web Data slots. A crawl with concurrency: 5 leaves one slot for other calls. When the shared browser pool is full, a Web Data request waits up to 15 seconds for a slot, then fails with 429.

Rate limits per route

Rate limits are counted per client IP address and per route, over one minute. They are the same on every plan.

Route	Limit
POST /api/v1/scrape	60 a minute
POST /api/v1/screenshot, /brand, /styleguide, /fonts, /map	30 a minute each
POST /api/v1/extract	20 a minute
POST /api/v1/crawl	10 a minute
GET /api/v1/crawl/:id	240 a minute
GET /api/v1/crawl/:id/errors	120 a minute
DELETE /api/v1/crawl/:id	60 a minute
POST and GET /api/search/v1	30 a minute
POST /api/captcha/solve	20 a minute
GET /api/captcha/solve/:id	120 a minute
/mcp	120 a minute
Operations learn, create, call, check	10, 20, 600 and 30 a minute
Operations check now	also 10 a clock hour per operation
POST /api/auth/export	3 an hour
Every other route	120 a minute

The browser WebSocket at /ws has no rate limit. The link cap applies to it. The full table is at Rate limits.

Read the headers

A response from a rate-limited route carries three headers:

• X-RateLimit-Limit, the requests allowed in the window
• X-RateLimit-Remaining, the requests left in the window
• X-RateLimit-Reset, the seconds until the window resets

When you pass the limit, the answer is 429 with a Retry-After header in seconds. On the Web Data, Search and Captcha APIs, the body is the usual envelope with error: "rate_limited".

curl -s -D - -o /dev/null -X POST https://api.wayfern.com/api/v1/scrape \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com" }' | grep -i -E "^(x-ratelimit|retry-after)"

A 429 from a concurrency cap has no Retry-After. Wait for one of your running requests to finish, then retry.

Stay inside the limits

Keep your own queue at the size of the cap, and pace starts to the route limit. This Python worker pool runs 6 scrapes at once and backs off on 429.

import os
import time
from concurrent.futures import ThreadPoolExecutor

import requests

auth = {"Authorization": f"Bearer {os.environ['WAYFERN_API_TOKEN']}"}


def scrape(url: str) -> dict:
    for attempt in range(5):
        res = requests.post(
            "https://api.wayfern.com/api/v1/scrape",
            headers=auth,
            json={"url": url, "formats": ["markdown"]},
            timeout=180,
        )
        if res.status_code != 429:
            res.raise_for_status()
            return res.json()
        wait = float(res.headers.get("Retry-After") or 2 ** attempt)
        time.sleep(wait)
    raise RuntimeError(f"still rate limited: {url}")


urls = [f"https://example.com/page/{n}" for n in range(1, 31)]
with ThreadPoolExecutor(max_workers=6) as pool:
    for result in pool.map(scrape, urls):
        print(result["credits"], result["data"]["metadata"]["sourceUrl"])

Six workers can send more than 60 scrapes a minute when pages load fast. The loop above waits for Retry-After when that happens. See Concurrency and seats and Handle API errors.