// Use case · Authenticated scraping
For jobs where the data sits behind a sign-in, a consent overlay or a captcha: drive the login over CDP, hold one exit IP for the run, and solve challenges when they appear.
captcha task types
11
credit per solved captcha
1
credits per open-page hour
8
{ "id": "cap_6f1c2d8e…", "status": "ready", "type": "turnstile", "solution": { "token": "0.Rk9P…" }} // Write solution.token into the form field// cf-turnstile-response, then submit. // Why
The problem
What Wayfern does
page.url() to POST /api/captcha/solve. The solver uses the same exit as the session. Write solution.token into the form: 1 credit, only on success.session=<id>, so the login and the last page come from the same address.// How it works
A session is created when the connection opens. There is no endpoint to call first and nothing to install beyond the driver you already use.
Step 1 of 4
Open the CDP socket with country and a session id: one residential IP for the whole run.
?token=…&country=us&session=run4711
Step 2 of 4
Fill, click, wait for the URL. Credentials and second factors come from your own secret store.
page.fill() · page.click()
Step 3 of 4
If a Turnstile, reCAPTCHA or hCaptcha widget appears, send its sitekey and page.url() to the solve endpoint. The solver uses the exit of the session. Write back solution.token.
POST /api/captcha/solve
Step 4 of 4
Every later navigation reuses the same cookies and the same exit until you close the socket.
page.goto("/reports/q1")
// Code
Both endpoints in this script are real: the CDP gateway and POST /api/captcha/solve. The second tab skips the browser when you already hold a session cookie.
import { chromium } from "playwright";
// One CDP connection is one browser session. Exit selectors ride on the query
// string: "country" turns on a residential exit, "session" pins it for the run.
const browser = await chromium.connectOverCDP(
`wss://api.wayfern.com/ws?token=${LINK_TOKEN}&country=us&session=run4711`,
);
const context = browser.contexts()[0];
const page = await context.newPage();
await page.goto("https://app.example.com/login");
await page.fill("#email", process.env.TARGET_EMAIL);
await page.fill("#password", process.env.TARGET_PASSWORD);
// A Turnstile widget guarding the form: solve it, then write the token into
// the field the form actually posts.
const widget = page.locator(".cf-turnstile[data-sitekey]").first();
if ((await widget.count()) > 0) {
const websiteKey = await widget.getAttribute("data-sitekey");
const headers = {
// An API token (wf_...) carrying the "captcha" scope.
Authorization: `Bearer ${WAYFERN_API_TOKEN}`,
"Content-Type": "application/json",
};
// websiteURL is page.url(): Wayfern finds this session by the open page and
// solves through its sticky exit.
let res = await fetch("https://api.wayfern.com/api/captcha/solve", {
method: "POST",
headers,
body: JSON.stringify({
task: { type: "turnstile", websiteURL: page.url(), websiteKey },
}),
});
let data = await res.json();
// 202: not ready after about 90 seconds. Read the task again.
while (res.status === 202) {
res = await fetch(`https://api.wayfern.com/api/captcha/solve/${data.id}`, { headers });
data = await res.json();
}
if (!res.ok) throw new Error(`${data.error}: ${data.message}`);
// The answer is always solution.token.
await page.evaluate((token) => {
const field = document.querySelector('input[name="cf-turnstile-response"]');
if (field instanceof HTMLInputElement) field.value = token;
}, data.solution.token);
}
await page.click("button[type=submit]");
await page.waitForURL("**/dashboard");
// Logged in. Every later navigation reuses this context (same cookies, same
// sticky exit) until you close the connection.
await page.goto("https://app.example.com/reports/q1");
console.log(await page.title());
await browser.close();// Walls
A logged-in page fails for one of four reasons. None of them is a flag you flip and hope: each is a surface you call.
Wall 01
A form, a redirect, a cookie set three hops later. Over CDP the login is ordinary Playwright, and the context keeps its cookies for the rest of the run.
wss://api.wayfern.com/ws
Wall 02
On screenshot and PDF captures, Wayfern clicks the accept control it recognises, hides what is left with CSS and unlocks scrolling. On by default.
capture.blockCookieBanners: true
Wall 03
reCAPTCHA, hCaptcha, Turnstile, FunCaptcha, GeeTest, Amazon WAF, Friendly Captcha, Prosopo, ALTCHA or a plain image. Post the task, read solution.token, write it into the form.
POST /api/captcha/solve
Wall 04
In auto mode a 401, 402, 403 or 407, or a redirect onto a known challenge path, is retried once through a residential exit. Set residential to start there.
proxy: 'auto' | 'residential'
// Identity
If the run is logged in, the exit has to hold still.
Add country and a session id (letters and digits) to the gateway URL. The id is encoded into the upstream credentials with a 24-hour TTL, so a multi-step flow keeps one exit IP. Narrow it with region, city or isp.
?token=…&country=us&session=run4711
A one-shot call takes a residential exit without a session id, so the IP rotates per connection. Right for independent fetches; for a logged-in flow, drive a session or send your credential on every call.
proxy: 'residential', country: 'us'
// Captcha
POST /api/captcha/solve takes a task and returns the answer in solution.token. Send page.url() as websiteURL, and the solver uses the proxy of the browser session that has the page open. Use a token that has the captcha scope.
| task.type | What it solves | Common fields |
|---|---|---|
| recaptcha_v2 | reCAPTCHA v2: checkbox, invisible or Enterprise | websiteURL, websiteKey, invisible, enterprise |
| recaptcha_v3 | reCAPTCHA v3, score-based | websiteURL, websiteKey, minScore, action |
| hcaptcha | hCaptcha, also invisible and Enterprise | websiteURL, websiteKey, enterprisePayload |
| turnstile | Cloudflare Turnstile | websiteURL, websiteKey, action |
| funcaptcha | FunCaptcha / Arkose Labs | websiteURL, websiteKey, data |
| geetest | GeeTest v3 and v4 | websiteURL, gt, challenge, version |
| amazon_waf | Amazon WAF captcha | websiteURL, websiteKey, iv, context |
| friendly_captcha | Friendly Captcha | websiteURL, websiteKey |
| prosopo | Prosopo Procaptcha | websiteURL, websiteKey |
| altcha | ALTCHA | websiteURL, challengeURL or challengeJSON |
| image_to_text | Plain image or text captcha | body (base64), phrase, numeric, math |
recaptcha_v2
recaptcha_v3
hcaptcha
turnstile
funcaptcha
geetest
amazon_waf
friendly_captcha
prosopo
altcha
image_to_text
// What it costs
Session time, captcha solves and residential bytes, all drawn from the same credits. Datacenter egress is free.
One page open for half an hour on a sticky residential exit, with two challenges on the way.
| Item | Credits |
|---|---|
| 1 page open for 30 minutes | 4 |
| 2 captcha solves | 2 |
| 0.05 GB residential traffic | 16 |
Total
22credits
about $0.28
at $0.0125 / credit
About 69 runs a month on Developer ($19 for 1,520 credits).
No session: /scrape carries your cookie on each call. Residential bytes are metered on top when used.
| Item | Credits |
|---|---|
| 40 scraped pages | 40 |
Total
40credits
about $0.50
at $0.0125 / credit
About 38 runs a month on Developer ($19 for 1,520 credits).
Each /api/v1 response returns the
exact charge in its credits field.
Every rate is on the pricing page.
// FAQ
No. The login runs in your own Playwright or Puppeteer code over CDP. Credentials stay in your environment and your script types them into the page.
No. There are no persistent customer profiles: cookies and storage live in the browser for as long as your socket is open. Log in again on the next run, or send the session cookie you hold with /scrape.
Nothing is debited. A task costs 1 credit only when it is solved. A task that fails, or has no answer after 5 minutes, costs nothing. websiteURL must be an absolute http(s) URL, and a field that the task type does not take fails with 400.
Yes. The Web Data API carries up to 20 custom headers per call, a cookie among them, at 1 credit per scraped page. It rejects the four headers it must own (Host, Content-Length, Connection, Transfer-Encoding) instead of dropping them silently.
With proxy: "auto" (the default on the Web Data API) a navigation that reads as blocked is retried once through a residential exit. Other failures surface at once, because retrying a timeout on a slower path only spends the budget twice. On a session, you choose the exit in the URL.
Three meters on one balance: session time pro-rated from 8 credits per open-page hour (60-second minimum per page), 1 credit per successful solve, and 320 credits per GB of residential traffic. Datacenter egress is free.
// Keep reading
// Next step
Hand challenges to an endpoint and keep one exit for the whole run. The same balance covers the browser, the solves and the bytes.