Make Web Data requests faster

A Web Data request spends its time in four places: getting a browser, loading the page, waiting for content and building the output. This guide shows which options change each part and what their defaults are.

How a page load runs

1. Wayfern gets a browser from its pool. When no warm browser is free, a new one takes about 45 seconds to start. This time is not part of your timeout. Browser start-up has its own limit of 90 seconds.
2. The browser opens the URL and waits for DOMContentLoaded. It does not wait for every image or script.
3. When you set waitForSelector, it waits for that selector, up to timeout. When the selector never appears, the request continues with the page as it is. It does not fail.
4. When you set waitFor, it then waits that many milliseconds, from 0 to 30,000.
5. Wayfern reads the HTML and builds the formats you asked for.

Timeouts

timeout is in milliseconds, from 1,000 to 120,000. It limits the page work, not browser start-up.

Endpoint	Default timeout
Scrape, extract, map and crawled pages	30,000
Brand, styleguide and fonts	45,000
Screenshot	60,000

A page that does not load in time fails with 503 timeout, and the request costs nothing. Set the timeout of your HTTP client above the request timeout plus browser start-up. The examples in these docs use 120 to 180 seconds.

Options that change the speed

Option	Default	Effect
blockAssets	true	Blocks images, media and fonts. Pages load with fewer requests and fewer bytes.
waitFor	0	Adds a fixed wait. Use it only for pages that render content late.
waitForSelector	none	Waits until the content you need exists, and stops waiting as soon as it does.
maxAge	0 on scrape	A cache hit skips the browser. See Caching and freshness.
proxy	auto	See the next section.
onlyMainContent	true	Does not change load time. It makes markdown and html smaller.

Formats also change the work:

• summary and json add a language model call after the page loads. When you ask for both, the two calls run at the same time.
• screenshot and pdf turn blockAssets off, so images load. These requests also skip the cache.
• Brand, styleguide and fonts always load images, fonts and stylesheets, and wait up to 8 seconds for the network to go quiet.

HTML larger than 10 MB fails with 413 too_large.

Proxy choice

proxy	What happens	Time budget
datacenter	Uses the datacenter exit only.	timeout
auto	Starts on the datacenter exit. When the site answers blocked, it tries once more through a residential exit. Other failures are not retried.	timeout, then 2 × timeout for the residential try
residential	Uses a residential exit from the start.	2 × timeout

The datacenter exit is free. Residential bandwidth costs 320 credits per GB. With auto, a blocked page can take up to three times timeout before it fails. When you know a site blocks datacenter exits, use residential and skip the first try. When you know it does not, use datacenter. See Reduce residential proxy bandwidth.

A fast scrape

This request blocks assets, waits for one selector, and accepts a cached copy up to 10 minutes old.

curl -X POST https://api.wayfern.com/api/v1/scrape \
  --max-time 150 \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://shop.example/products/desk-lamp",
    "formats": ["markdown"],
    "blockAssets": true,
    "waitForSelector": "[data-testid=price]",
    "timeout": 20000,
    "proxy": "datacenter",
    "maxAge": 600000
  }'

Run requests in parallel

Your account can run 6 Web Data requests at once. A seventh request fails at once with 429 rate_limited. When the shared browser pool is full, a request waits up to 15 seconds for a slot, then fails with 429. Each crawled page uses one of the 6 slots. Keep your own worker pool at 6 or less. See Rate limits and concurrency.

const urls = ["https://example.com/a", "https://example.com/b", "https://example.com/c"];
const limit = 6;
const results = [];

async function worker(queue) {
  for (let url = queue.shift(); url; url = queue.shift()) {
    const res = await fetch("https://api.wayfern.com/api/v1/scrape", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.WAYFERN_API_TOKEN}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ url, formats: ["markdown"] }),
      signal: AbortSignal.timeout(150_000),
    });
    results.push({ url, status: res.status, body: await res.json() });
  }
}

const queue = [...urls];
await Promise.all(Array.from({ length: Math.min(limit, urls.length) }, () => worker(queue)));
console.log(results.map((r) => `${r.status} ${r.url}`).join("\n"));