Scrape a page to Markdown

POST /api/v1/scrape loads one page in a Wayfern browser and returns it in the formats you ask for. This guide shows the request, each format, the content filters and what a scrape costs.

Make a request

Only url is required. formats defaults to ["markdown"]. Use an API token with the web scope.

curl -X POST https://api.wayfern.com/api/v1/scrape \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown", "links"]
  }'

The response has three keys: success, data and credits. credits is what the call debited.

{
  "success": true,
  "data": {
    "markdown": "# Pricing\n\nStart free, then pay for what you use.",
    "links": [
      { "url": "https://example.com/docs", "text": "Documentation", "external": false, "rel": [] }
    ],
    "metadata": {
      "title": "Pricing - Example",
      "language": "en",
      "canonical": "https://example.com/pricing",
      "statusCode": 200,
      "sourceUrl": "https://example.com/pricing"
    },
    "cached": false
  },
  "credits": 1
}

The API rejects unknown body fields with 400, so a typo in a key fails at once. The full field list is in the Scrape reference.

Choose formats

Format	What data holds
markdown	The page as Markdown, after the content filters below.
html	The HTML that the content filters kept.
rawHtml	The full page HTML, with no filters.
links	Every link on the page, with url, text, external and rel.
summary	A short summary of the page from a language model. Add an instruction with summaryPrompt, up to 2,000 characters.
json	Structured data shaped by jsonOptions.schema, jsonOptions.prompt or both.

metadata comes with every format. It has the title, description, language, canonical URL, Open Graph and Twitter tags, JSON-LD blocks, statusCode and sourceUrl.

Some rules apply:

• An empty formats array is a 400.
• json without jsonOptions.schema or jsonOptions.prompt is a 400. The prompt can have up to 2,000 characters.
• summary and json need the language model. When it is not available, the call fails with 503 llm_unavailable and costs nothing.
• screenshot and pdf are also formats. See Full-page screenshots and PDF generation.

Keep only the content you need

These options change markdown and html. rawHtml and links always come from the full page.

• onlyMainContent is true by default. It removes navigation, headers, footers, sidebars and cookie notices, then keeps the main article. Set it to false to convert the whole body.
• includeTags is a list of CSS selectors to keep. Everything else is dropped. When you set it, onlyMainContent has no effect, because your selectors define the main content.
• excludeTags is a list of CSS selectors to remove. When a selector is in both lists, the element stays and its excluded children are removed.
• Each list takes up to 50 selectors. A selector that does not compile is skipped, and the others still apply.
• removeBase64Images is true by default. It replaces inline base64 images with their alt text.

const res = await fetch("https://api.wayfern.com/api/v1/scrape", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.WAYFERN_API_TOKEN}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://docs.example.com/quickstart",
    formats: ["markdown"],
    includeTags: ["main article"],
    excludeTags: [".feedback-widget", "pre.terminal-output"],
  }),
});
const body = await res.json();
if (!res.ok) throw new Error(`${body.error}: ${body.message}`);
console.log(body.credits, body.data.markdown);

What a scrape costs

Item	Credits
One page	1
summary format	+2
screenshot, pdf or json format	+4, once per page, however many of the three you ask for
Residential bandwidth	320 per GB, only when the request uses a residential exit

A scrape with markdown, summary and json costs 1 + 2 + 4 = 7 credits.

A failed scrape costs nothing. Wayfern checks your balance before it opens a browser and debits only when it has output. A page served from the cache is billed like a live fetch. See Caching and freshness for maxAge.

The scrape route allows 60 requests a minute, and your account can run 6 Web Data requests at once. See Rate limits and Pricing.