Map a website's URLs before you crawl

POST /api/v1/map returns the URLs of a site without scraping them. Use it to size a crawl before you pay for one, or to build a URL list for extract. This guide shows how discovery works, how to filter the result and what it costs.

Make a request

curl -X POST https://api.wayfern.com/api/v1/map \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "search": "pricing",
    "limit": 100
  }'

Field	Default	What it does
url	required	The site to map, up to 2,048 characters.
search	none	Keep only URLs whose address or title contains this term, up to 256 characters.
limit	100	Most URLs to return, from 1 to 5,000.
includeSubdomains	false	Also keep URLs on subdomains of the site.
ignoreSitemap	false	Skip sitemaps. Use the page links only.
sitemapOnly	false	Use sitemaps only. No browser opens.

proxy, country and timeout also apply. The default timeout is 30 seconds. Sending both ignoreSitemap and sitemapOnly is a 400.

Each URL in data.links has url, title, description, source and lastModified. source is sitemap or links.

{
  "success": true,
  "data": {
    "links": [
      {
        "url": "https://docs.example.com/pricing",
        "title": null,
        "description": null,
        "source": "sitemap",
        "lastModified": "2026-10-02T00:00:00.000Z"
      }
    ]
  },
  "credits": 2
}

How discovery works

1. Wayfern reads the Sitemap lines in robots.txt. It also tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap/sitemap.xml, /wp-sitemap.xml and /sitemap.xml.gz.
2. It follows sitemap indexes, up to 25 sitemaps per call. Each sitemap gets 12 seconds and up to 25 MB. Sitemaps load over plain HTTP, without a browser.
3. When the sitemaps give fewer URLs than limit, and sitemapOnly is off, a browser loads url and adds the links on that page. Only that one page is read. Map does not follow links further.
4. Wayfern keeps the URLs on the same host, or on the same site with includeSubdomains. It applies search, puts short paths first, sorts equal depths by URL and cuts the list at limit.

Sitemap entries have no title, so search matches only their URL. Link entries use the anchor text as the title. The match does not care about case.

When no URL is found, the call fails with 404 not_found and costs nothing. When the page fails to load but the sitemaps gave URLs, you get the sitemap URLs.

What a map costs

Map costs 2 credits per started block of 10 returned URLs.

URLs returned	Credits
10	2
12	4
100	20
5,000	1,000

Before it starts, Wayfern checks that your balance covers the full limit. With limit: 5000, you need 1,000 credits available, even if the site has fewer URLs. You pay only for the URLs it returns. Residential bandwidth, when used, costs 320 credits per GB.

Map has no cache. Each call reads the sitemaps again. The route allows 30 requests a minute.

Feed the result into a crawl or an extract

Map a site, keep the URLs you want, then send them to extract.

import os

import requests

auth = {"Authorization": f"Bearer {os.environ['WAYFERN_API_TOKEN']}"}
base = "https://api.wayfern.com/api/v1"

mapped = requests.post(
    f"{base}/map",
    headers=auth,
    json={"url": "https://docs.example.com", "search": "/docs/api/", "limit": 50},
    timeout=120,
)
mapped.raise_for_status()
urls = [link["url"] for link in mapped.json()["data"]["links"]]
print(len(urls), "URLs for", mapped.json()["credits"], "credits")

extract = requests.post(
    f"{base}/extract",
    headers=auth,
    json={"urls": urls[:10], "prompt": "every endpoint path with its HTTP method"},
    timeout=300,
)
extract.raise_for_status()
print(extract.json()["data"]["data"])

To crawl the same scope, set includePaths on the crawl to match what you mapped. A crawl reads the sitemaps itself and does not bill for that. See Crawl a site, Extract structured data and the Map reference.