Scrape pages behind a login

Wayfern has no login feature and no credential store. You log in with your own code, keep the session state yourself, and send it back on the next run. This guide shows how, for browser sessions and for Web Data calls.

What Wayfern does not store

• Wayfern keeps no cookies, local storage or browser profile between sessions. When the last connection on a browser closes, Wayfern stops that browser.
• Wayfern has no store for site passwords or cookies. Each Web Data call uses only the headers in that request.

Wayfern does keep some output that can hold logged-in content:

• A scrape capture stays in the page cache for 1 hour, unless it is over 2 MB. The cache key includes your headers, so only a request with the same headers can read it.
• Crawl results, screenshots and PDFs are kept for 7 days.
• If the link records sessions, a replay holds screenshots of every page. Connect with record=0 to skip recording.

Log in over CDP and save the state

Log in with Playwright, then save the cookies and local storage to a file that you control.

import { chromium } from "playwright";

// session=acct1 holds one residential exit IP for the whole login flow.
const url =
  `wss://api.wayfern.com/ws?token=${process.env.WAYFERN_LINK_TOKEN}` +
  "&country=US&session=acct1";

const browser = await chromium.connectOverCDP(url, { timeout: 120_000 });
const context = browser.contexts()[0];
const page = await context.newPage();

await page.goto("https://example.com/login");
await page.fill("input[name=email]", process.env.SITE_EMAIL);
await page.fill("input[name=password]", process.env.SITE_PASSWORD);
await page.click("button[type=submit]");
await page.waitForURL("**/account");

await context.storageState({ path: "state.json" });
await browser.close();

Keep state.json as safe as the password. It is a working login.

To keep one IP from sign-in to the last request, use a residential exit with session=<id>. Wayfern asks for that exit IP for 24 hours. Without session, the residential exit can change between the connections that the browser opens.

Restore the state on the next run

connectOverCDP gives you the browser’s own context, which exists before your code runs. Restore cookies with addCookies and local storage with a script per origin:

import { readFile } from "node:fs/promises";
import { chromium } from "playwright";

const state = JSON.parse(await readFile("state.json", "utf8"));
const browser = await chromium.connectOverCDP(
  `wss://api.wayfern.com/ws?token=${process.env.WAYFERN_LINK_TOKEN}&country=US&session=acct1`,
  { timeout: 120_000 },
);
const context = browser.contexts()[0];
await context.addCookies(state.cookies);

const page = await context.newPage();
for (const { origin, localStorage } of state.origins) {
  await page.goto(origin);
  await page.evaluate((items) => {
    for (const { name, value } of items) window.localStorage.setItem(name, value);
  }, localStorage);
}

await page.goto("https://example.com/account");
console.log(await page.title());
await browser.close();

Two connections of the same link that are open at the same time can run in one browser and share its default context, cookies included. Use one link per account to keep logins apart.

Send cookies on Web Data calls

Pass a session cookie or a token in headers:

curl -X POST https://api.wayfern.com/api/v1/scrape \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/account/orders",
    "headers": { "cookie": "session=abc123" },
    "formats": ["markdown"]
  }'

• Wayfern sends the headers only to requests on the origin of url. A redirect to another origin, and subresources from other origins, do not get them.
• You can send up to 20 headers. Each value can have up to 4,096 characters.
• host, content-length, connection and transfer-encoding are refused with 400.
• scrape, screenshot, a crawl’s scrapeOptions, brand, styleguide and fonts take headers. map and extract do not.
• The MCP tools take no headers. Call the REST API for a logged-in page.

If the site answers 401 or 403, the call fails with blocked and is not billed. In auto mode, Wayfern first retries through a residential exit. Read the error reference in Handle API errors and the field list in shared navigation options.