Receive crawl events with webhooks

A crawl webhook sends each crawled page and each status change to your endpoint, so you do not have to poll. This guide shows the request fields, the events, the payload, how to authenticate deliveries and what happens when your endpoint fails.

Add a webhook to a crawl

Put a webhook object in the body of POST /api/v1/crawl.

Field	Required	What it does
url	yes	Your endpoint. It must use https://. An http:// URL is a 400.
headers	no	Headers sent with every delivery, for example your own secret.
events	no	Any of started, page, completed, failed. Omit it to get all four.
metadata	no	An object that comes back unchanged in every delivery.

curl -X POST https://api.wayfern.com/api/v1/crawl \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 100,
    "webhook": {
      "url": "https://api.example.com/hooks/wayfern",
      "headers": { "X-Hook-Token": "'"$HOOK_TOKEN"'" },
      "events": ["page", "completed", "failed"],
      "metadata": { "runId": "nightly-42" }
    }
  }'

Events

Event	When	Extra fields
started	The job begins.	none
page	A page was crawled and billed.	data, the page document
completed	The job finished its queue or reached limit.	none
failed	The job stopped early.	error, when there is a reason

A cancelled job also ends with a failed event. That event has no error field. A job that ran out of credits sends failed with an error that says so.

Payload and headers

Each delivery is a POST with a JSON body.

{
  "type": "page",
  "crawlId": "4f1c9b2a-6d3e-4a17-9c88-0b5e2f7a1d34",
  "at": "2026-10-29T09:14:22.481Z",
  "completed": 12,
  "failed": 1,
  "data": {
    "markdown": "# Quickstart\n\nInstall the SDK and make your first call.",
    "metadata": { "title": "Quickstart", "statusCode": 200, "sourceUrl": "https://docs.example.com/quickstart" },
    "cached": false
  },
  "metadata": { "runId": "nightly-42" }
}

completed and failed are the job counters at the time of the event. data has the formats you set in scrapeOptions.

The headers are:

• Content-Type: application/json
• User-Agent: WayfernWebhook/1.0
• X-Wayfern-Timestamp, the send time in milliseconds since 1970
• X-Wayfern-Signature, only when the deployment has a signing secret
• The headers you set on the webhook

Check that a delivery is real

X-Wayfern-Signature has the form sha256= followed by a hex HMAC-SHA256. The HMAC covers the timestamp, a dot and the raw body, and its key is the signing secret of the Wayfern deployment. There is no secret per account, and no API or dashboard page shows the deployment secret.

To authenticate deliveries, send your own secret in webhook.headers and compare it on your endpoint. Use a constant-time compare.

import { timingSafeEqual } from "node:crypto";
import express from "express";

const app = express();
const expected = Buffer.from(process.env.HOOK_TOKEN);

app.post("/hooks/wayfern", express.json({ limit: "5mb" }), (req, res) => {
  const given = Buffer.from(req.get("X-Hook-Token") ?? "");
  if (given.length !== expected.length || !timingSafeEqual(given, expected)) {
    return res.status(401).end();
  }
  // Answer first, then work: the crawl waits for this response.
  res.status(200).end();
  queueForProcessing(req.body);
});

function queueForProcessing(event) {
  console.log(event.type, event.crawlId, event.completed);
}

app.listen(3000);

Retries and timeouts

• Each event gets up to 3 attempts. Each attempt has a 10-second timeout.
• Wayfern waits 0.5 seconds before the second attempt and 1 second before the third.
• A network error, a timeout, a 5xx, a 408 or a 429 causes a retry.
• Any other 4xx stops the retries for that event.
• A retry sends the same body and the same timestamp.

The crawl waits for each delivery before it continues. A slow endpoint makes the crawl slow, so return 2xx at once and do the work later.

A delivery that fails all attempts does not fail the crawl. The pages stay readable through GET /api/v1/crawl/:id until the job expires, 7 days after the start. Use polling to fill gaps.

A retry after a timeout can deliver an event that your endpoint already received. For page events, the pair crawlId and completed is unique per page, so use it to drop duplicates.

See Crawl a site for the crawl options and Crawl for the reference.