Wayfern
Start building Sign in

Extract structured data with a JSON Schema

Pull typed data from one or many pages with a prompt or a JSON Schema, expand a path with /*, cap the pages fetched and re-run free on pages still in the cache.

Updated

POST /api/v1/extract reads one or more pages and returns one JSON result that follows your prompt, your JSON Schema or both. This guide shows the request, how pages are combined, the /* expansion, the page cap and why a second run over the same pages can cost nothing.

Make a request

Send urls and at least one of prompt or schema. Without both, the call is a 400.

curl -X POST https://api.wayfern.com/api/v1/extract \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing", "https://example.com/plans/enterprise"],
    "prompt": "every pricing tier with its monthly price in US dollars and included seats",
    "schema": {
      "type": "object",
      "properties": {
        "tiers": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "monthlyUsd": { "type": "number" },
              "seats": { "type": "integer" }
            },
            "required": ["name", "monthlyUsd"]
          }
        }
      },
      "required": ["tiers"]
    }
  }'
FieldDefaultWhat it does
urlsrequiredThe pages to read. Duplicates are removed.
promptnoneWhat to extract, in plain words, up to 4,000 characters.
schemanoneThe JSON Schema the result must follow.
maxPages10Most source pages to fetch, from 1 to 100. It also caps the bill.
maxAge3,600,000Use a cached copy of a page up to this many milliseconds old.

proxy, country and timeout also apply. The default timeout is 30 seconds per page.

Read the result

{
  "success": true,
  "data": {
    "data": {
      "tiers": [
        { "name": "Starter", "monthlyUsd": 0, "seats": 1 },
        { "name": "Growth", "monthlyUsd": 49, "seats": 5 }
      ]
    },
    "sources": [
      { "url": "https://example.com/pricing", "status": "fetched", "billed": true },
      { "url": "https://example.com/plans/enterprise", "status": "failed", "billed": false, "error": "not_found" }
    ]
  },
  "credits": 1
}

data.data is the result. data.sources lists each page with its status: fetched, cached or failed. billed says whether the page cost a credit.

How pages are combined

Wayfern fetches up to 4 pages at a time. It turns each page into main-content Markdown, then sends all pages to the language model in one call. The model merges facts across the pages. When pages disagree, it prefers the most specific one. A field that no page states is null.

  • Each page adds up to 10,000 tokens of Markdown. With more than 10 pages, each page gets a smaller share.
  • The answer is capped at 800 tokens. Ask for compact output, and split a large job into several calls.
  • When the language model is not available, the call fails with 503 llm_unavailable and costs nothing.
  • When no page loads, the call fails with the first page error and costs nothing.

Expand a path with /*

A URL that ends in /* expands to the URLs that map finds for it. For example, https://example.com/docs/* maps https://example.com/docs.

  • The expansion stops at maxPages, counted with the plain URLs in the list.
  • The map keeps every URL on the same host. It does not filter by the path, so URLs outside /docs/ can appear. Short paths come first.
  • To stay inside one path, call map yourself with search, then send the URLs you keep.
  • The expansion is billed as a map call, at 2 credits per started 10 URLs. That charge is not in the credits of the extract response.

Re-run on cached pages

Extract bills 1 credit per uncached source page. A page served from the cache costs nothing. Failed pages cost nothing.

Each fetched page goes into the cache for 1 hour, unless it is larger than 2 MB. With the default maxAge of one hour, a second run over the same URLs within that hour reads the cached pages. You can change the prompt or the schema and pay nothing for the pages.

import os

import requests

auth = {"Authorization": f"Bearer {os.environ['WAYFERN_API_TOKEN']}"}
urls = ["https://example.com/pricing"]

for prompt in ["plan names only", "plan names with monthly price and seats"]:
    res = requests.post(
        "https://api.wayfern.com/api/v1/extract",
        headers=auth,
        json={"urls": urls, "prompt": prompt},
        timeout=180,
    )
    res.raise_for_status()
    body = res.json()
    billed = sum(1 for s in body["data"]["sources"] if s["billed"])
    print(body["credits"], "credits,", billed, "pages billed:", body["data"]["data"])

The first run costs 1 credit. The second run costs 0 when the page is still cached.

  • Set maxAge to 0 to force a live fetch. A live fetch is billed.
  • The cache keeps a page for at most 1 hour, so a maxAge above one hour has no further effect.
  • The cache entry is keyed on the URL, proxy and country. A scrape with default options uses a different entry, so it does not fill the extract cache.

Before the call starts, Wayfern checks that your balance covers every page as uncached. The route allows 20 requests a minute. For one page, the scrape json format is another option at 1 + 4 credits. See Caching and freshness and the Extract reference.

Related guides