Caching, maxAge and data freshness

Wayfern keeps recent page captures in a cache. You decide per request how old a cached copy may be. This guide shows how maxAge works, what the cache keys on, which repeated calls are free and how long each kind of result is kept.

How maxAge works

maxAge is in milliseconds. A cached copy that is older than maxAge is not used.

Endpoint	maxAge default	Cache hit billed
Scrape	0, always live	Yes, 1 credit per page
Crawl pages, through scrapeOptions.maxAge	0, always live	Yes, 1 credit per page
Extract	3,600,000 (1 hour)	No

A live fetch also writes to the cache. A scrape with maxAge: 0 still fills the cache for later calls.

The cache keeps an entry for 1 hour. A maxAge above 3,600,000 has no further effect. A page whose entry is larger than 2 MB is not cached. When the cache cannot be read, the request fetches the page live.

curl -X POST https://api.wayfern.com/api/v1/scrape \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/pricing", "maxAge": 600000 }'

data.cached is true when the page came from the cache. On extract, each entry in data.sources has status: "cached" and billed: false.

A cache hit returns without opening a browser. On scrape the price is the same as a live fetch, because you get the same page.

What the cache keys on

The key is the normalized URL plus the options that change the captured page.

The URL is normalized first:

• The host is lowercased and the default port is removed.
• The fragment after # is removed.
• Tracking parameters such as utm_*, gclid, fbclid and msclkid are removed.
• The other query parameters are sorted.

For scrape and crawl pages, these options are part of the key: mobile, blockAssets, waitFor, waitForSelector, proxy when it is not auto, country and headers.

These options are not part of the key: formats, onlyMainContent, includeTags, excludeTags and removeBase64Images. Wayfern applies them to the cached page after it reads it. You can change them and still get a cache hit.

Extract keys its entries on the URL, proxy and country only. With default options, a scrape and an extract of the same URL use different entries. A scrape does not fill the extract cache.

A scrape that asks for screenshot or pdf does not read or write the cache. It always loads the page.

What is free on a re-run

Call	Repeated call
Extract over cached pages	Free for the cached pages
Polling a crawl, reading its errors	Free
Scrape cache hit	Billed
Crawl page from the cache	Billed
Search result page from the cache	Billed, 2 credits per provider page
Operation call from its result cache	Billed, 1 credit
Brand, styleguide, fonts, map, screenshot	No cache. Each call loads the page and is billed.

To change an extract prompt or schema without paying for the pages again, re-run within the hour. See Extract structured data.

import os

import requests

auth = {"Authorization": f"Bearer {os.environ['WAYFERN_API_TOKEN']}"}
url = "https://example.com/pricing"

for max_age in (0, 600_000):
    res = requests.post(
        "https://api.wayfern.com/api/v1/scrape",
        headers=auth,
        json={"url": url, "formats": ["markdown"], "maxAge": max_age},
        timeout=120,
    )
    res.raise_for_status()
    body = res.json()
    print(f"maxAge={max_age}: cached={body['data']['cached']} credits={body['credits']}")

The first call fetches live and fills the cache. The second call reads the cache and costs the same 1 credit.

Other caches

• Search results are cached for 5 minutes.
• robots.txt files are cached for 1 hour.
• Operations have their own result cache. It is off by default. Set cacheTtlSeconds on the operation or on one call, up to 86,400 seconds. Only answers are cached, and a hit is billed like a call.

How long results are kept

Result	Kept
Page cache entry	1 hour
Crawl pages and errors	7 days, until the job’s expiresAt
Screenshots and PDFs	7 days. Download them to keep them longer.
Operation learn drafts	30 minutes
Operation events	90 days

See Retention and caches for the full list.