// Use case · Authenticated scraping

Reach the pages behind a login, and stay signed in

For jobs where the data sits behind a sign-in, a consent overlay or a captcha: drive the login over CDP, hold one exit IP for the run, and solve challenges when they appear.

  • captcha task types

    11

  • credit per solved captcha

    1

  • credits per open-page hour

    8

POST /api/captcha/solve 200 OK
{  "id": "cap_6f1c2d8e…",  "status": "ready",  "type": "turnstile",  "solution": {    "token": "0.Rk9P…"  }} // Write solution.token into the form field// cf-turnstile-response, then submit.
credit, debited only on success
The challenge step of a scripted login. The answer is always solution.token. A solve costs 1 credit, debited only when a solution comes back.

// Why

The pages a scraping API cannot reach

The problem

A plain fetch stops at the first wall

  • The data sits behind a sign-in form, a redirect and a cookie set three hops later.
  • A consent overlay swallows clicks, or a captcha guards the form.
  • A cookie issued to one IP and replayed from another is the cheapest signal a site has.

What Wayfern does

A browser you drive, an exit that holds still

  • Log in with ordinary Playwright over CDP. The browser context keeps its cookies for the rest of the run.
  • Send a challenge's sitekey and page.url() to POST /api/captcha/solve. The solver uses the same exit as the session. Write solution.token into the form: 1 credit, only on success.
  • Pin one residential exit with session=<id>, so the login and the last page come from the same address.

// How it works

Connect, log in, solve, read

A session is created when the connection opens. There is no endpoint to call first and nothing to install beyond the driver you already use.

  1. Step 1 of 4

    Connect with a sticky exit

    Open the CDP socket with country and a session id: one residential IP for the whole run.

    ?token=…&country=us&session=run4711

  2. Step 2 of 4

    Log in

    Fill, click, wait for the URL. Credentials and second factors come from your own secret store.

    page.fill() · page.click()

  3. Step 3 of 4

    Solve the challenge

    If a Turnstile, reCAPTCHA or hCaptcha widget appears, send its sitekey and page.url() to the solve endpoint. The solver uses the exit of the session. Write back solution.token.

    POST /api/captcha/solve

  4. Step 4 of 4

    Read the pages

    Every later navigation reuses the same cookies and the same exit until you close the socket.

    page.goto("/reports/q1")

// Code

It is just Playwright, pointed somewhere else

Both endpoints in this script are real: the CDP gateway and POST /api/captcha/solve. The second tab skips the browser when you already hold a session cookie.

Log in through a Turnstile, or reuse a cookie
import { chromium } from "playwright";

// One CDP connection is one browser session. Exit selectors ride on the query
// string: "country" turns on a residential exit, "session" pins it for the run.
const browser = await chromium.connectOverCDP(
  `wss://api.wayfern.com/ws?token=${LINK_TOKEN}&country=us&session=run4711`,
);
const context = browser.contexts()[0];
const page = await context.newPage();

await page.goto("https://app.example.com/login");
await page.fill("#email", process.env.TARGET_EMAIL);
await page.fill("#password", process.env.TARGET_PASSWORD);

// A Turnstile widget guarding the form: solve it, then write the token into
// the field the form actually posts.
const widget = page.locator(".cf-turnstile[data-sitekey]").first();
if ((await widget.count()) > 0) {
  const websiteKey = await widget.getAttribute("data-sitekey");
  const headers = {
    // An API token (wf_...) carrying the "captcha" scope.
    Authorization: `Bearer ${WAYFERN_API_TOKEN}`,
    "Content-Type": "application/json",
  };

  // websiteURL is page.url(): Wayfern finds this session by the open page and
  // solves through its sticky exit.
  let res = await fetch("https://api.wayfern.com/api/captcha/solve", {
    method: "POST",
    headers,
    body: JSON.stringify({
      task: { type: "turnstile", websiteURL: page.url(), websiteKey },
    }),
  });
  let data = await res.json();
  // 202: not ready after about 90 seconds. Read the task again.
  while (res.status === 202) {
    res = await fetch(`https://api.wayfern.com/api/captcha/solve/${data.id}`, { headers });
    data = await res.json();
  }
  if (!res.ok) throw new Error(`${data.error}: ${data.message}`);

  // The answer is always solution.token.
  await page.evaluate((token) => {
    const field = document.querySelector('input[name="cf-turnstile-response"]');
    if (field instanceof HTMLInputElement) field.value = token;
  }, data.solution.token);
}

await page.click("button[type=submit]");
await page.waitForURL("**/dashboard");

// Logged in. Every later navigation reuses this context (same cookies, same
// sticky exit) until you close the connection.
await page.goto("https://app.example.com/reports/q1");
console.log(await page.title());
await browser.close();

// Walls

What stops a plain request, and the answer to each

A logged-in page fails for one of four reasons. None of them is a flag you flip and hope: each is a surface you call.

  • Wall 01

    It wants a session

    A form, a redirect, a cookie set three hops later. Over CDP the login is ordinary Playwright, and the context keeps its cookies for the rest of the run.

    wss://api.wayfern.com/ws

  • Wall 02

    It wants a consent click

    On screenshot and PDF captures, Wayfern clicks the accept control it recognises, hides what is left with CSS and unlocks scrolling. On by default.

    capture.blockCookieBanners: true

  • Wall 03

    It wants a challenge solved

    reCAPTCHA, hCaptcha, Turnstile, FunCaptcha, GeeTest, Amazon WAF, Friendly Captcha, Prosopo, ALTCHA or a plain image. Post the task, read solution.token, write it into the form.

    POST /api/captcha/solve

  • Wall 04

    It has judged your IP

    In auto mode a 401, 402, 403 or 407, or a redirect onto a known challenge path, is retried once through a residential exit. Set residential to start there.

    proxy: 'auto' | 'residential'

// Identity

One exit, from login to last page

If the run is logged in, the exit has to hold still.

  • On a browser session: sticky

    Add country and a session id (letters and digits) to the gateway URL. The id is encoded into the upstream credentials with a 24-hour TTL, so a multi-step flow keeps one exit IP. Narrow it with region, city or isp.

    ?token=…&country=us&session=run4711

  • On the Web Data API: rotating

    A one-shot call takes a residential exit without a session id, so the IP rotates per connection. Right for independent fetches; for a logged-in flow, drive a session or send your credential on every call.

    proxy: 'residential', country: 'us'

// Captcha

11 task types, one endpoint

POST /api/captcha/solve takes a task and returns the answer in solution.token. Send page.url() as websiteURL, and the solver uses the proxy of the browser session that has the page open. Use a token that has the captcha scope.

  • recaptcha_v2

    What it solves
    reCAPTCHA v2: checkbox, invisible or Enterprise
    Common fields
    websiteURL, websiteKey, invisible, enterprise
  • recaptcha_v3

    What it solves
    reCAPTCHA v3, score-based
    Common fields
    websiteURL, websiteKey, minScore, action
  • hcaptcha

    What it solves
    hCaptcha, also invisible and Enterprise
    Common fields
    websiteURL, websiteKey, enterprisePayload
  • turnstile

    What it solves
    Cloudflare Turnstile
    Common fields
    websiteURL, websiteKey, action
  • funcaptcha

    What it solves
    FunCaptcha / Arkose Labs
    Common fields
    websiteURL, websiteKey, data
  • geetest

    What it solves
    GeeTest v3 and v4
    Common fields
    websiteURL, gt, challenge, version
  • amazon_waf

    What it solves
    Amazon WAF captcha
    Common fields
    websiteURL, websiteKey, iv, context
  • friendly_captcha

    What it solves
    Friendly Captcha
    Common fields
    websiteURL, websiteKey
  • prosopo

    What it solves
    Prosopo Procaptcha
    Common fields
    websiteURL, websiteKey
  • altcha

    What it solves
    ALTCHA
    Common fields
    websiteURL, challengeURL or challengeJSON
  • image_to_text

    What it solves
    Plain image or text captcha
    Common fields
    body (base64), phrase, numeric, math
A solve is debited on success only. If no solution comes back, nothing is charged.

// What it costs

Three meters, one balance

Session time, captcha solves and residential bytes, all drawn from the same credits. Datacenter egress is free.

A 30-minute logged-in export

One page open for half an hour on a sticky residential exit, with two challenges on the way.

Credits for: A 30-minute logged-in export
ItemCredits
1 page open for 30 minutes4
2 captcha solves2
0.05 GB residential traffic16

Total

22credits

about $0.28
at $0.0125 / credit

About 69 runs a month on Developer ($19 for 1,520 credits).

40 report pages with a cookie you hold

No session: /scrape carries your cookie on each call. Residential bytes are metered on top when used.

Credits for: 40 report pages with a cookie you hold
ItemCredits
40 scraped pages40

Total

40credits

about $0.50
at $0.0125 / credit

About 38 runs a month on Developer ($19 for 1,520 credits).

Each /api/v1 response returns the exact charge in its credits field. Every rate is on the pricing page.

// FAQ

Questions about logged-in scraping

Do I give Wayfern my passwords?

No. The login runs in your own Playwright or Puppeteer code over CDP. Credentials stay in your environment and your script types them into the page.

Does Wayfern keep my login between runs?

No. There are no persistent customer profiles: cookies and storage live in the browser for as long as your socket is open. Log in again on the next run, or send the session cookie you hold with /scrape.

What if a captcha solve fails?

Nothing is debited. A task costs 1 credit only when it is solved. A task that fails, or has no answer after 5 minutes, costs nothing. websiteURL must be an absolute http(s) URL, and a field that the task type does not take fails with 400.

Can I skip the browser if I already have a session cookie?

Yes. The Web Data API carries up to 20 custom headers per call, a cookie among them, at 1 credit per scraped page. It rejects the four headers it must own (Host, Content-Length, Connection, Transfer-Encoding) instead of dropping them silently.

When does a request switch to a residential exit?

With proxy: "auto" (the default on the Web Data API) a navigation that reads as blocked is retried once through a residential exit. Other failures surface at once, because retrying a timeout on a slower path only spends the budget twice. On a session, you choose the exit in the URL.

What does an authenticated run cost?

Three meters on one balance: session time pro-rated from 8 credits per open-page hour (60-second minimum per page), 1 credit per successful solve, and 320 credits per GB of residential traffic. Datacenter egress is free.

// Next step

Log in once, hold the session, keep reading

Hand challenges to an endpoint and keep one exit for the whole run. The same balance covers the browser, the solves and the bytes.