Get past bot walls and challenge pages
How Wayfern handles bot walls and challenge pages: the anti-detect browser, the residential retry in auto mode, captcha solving, and what it cannot promise.
Updated
Wayfern gives you tools for bot walls. It does not promise that a page will load. This guide says what each tool does, where it applies, and what happens when it fails.
The anti-detect browser
Every browser session and every Web Data call runs in a Wayfern anti-detect browser.
- When you set a viewport, Wayfern resizes the real window. It emulates nothing and runs no script in your pages, so the window and screen values that a page reads agree.
- Live view and replays capture over a separate connection. They run no script in your pages.
Wayfern publishes no detection rate and no success rate.
Web Data: the residential retry in auto mode
Wayfern classifies the first navigation of a Web Data call:
| Response | Result |
|---|---|
| 401, 402, 403 or 407 | blocked |
A page that ends on a /cdn-cgi/challenge, /sorry, /captcha, /blocked or /access-denied path | blocked |
| 429 | rate_limited |
| 404 or 410 | not_found |
| 5xx | upstream_error |
In auto mode, the default, a blocked result on the datacenter exit gets one more try through a residential exit. Wayfern does not retry other results. If the residential try is also refused, the call returns 403 with "error": "blocked". Failed calls are not billed.
Wayfern reads only the status code and the final URL. A challenge page that answers 200 at the same URL comes back as the page content. The Web Data API does not solve captchas.
curl -X POST https://api.wayfern.com/api/v1/scrape \
-H "Authorization: Bearer $WAYFERN_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{ "url": "https://example.com/pricing", "proxy": "auto", "waitFor": 3000 }' {
"error": "blocked",
"code": 403,
"request_id": "3b7c1d0e-9f2a-4c58-8d31-6e0a5b4c7d92",
"message": "blocked: https://example.com/pricing",
"reason": "https://example.com/pricing"
} Branch on error, not on message. Use waitFor or waitForSelector when a page builds its content after load.
Browser sessions
Sessions do not retry and do not solve captchas by themselves. Your code decides what to do:
- Reconnect with a
countryon the connection URL to use a residential exit. Addsession=<id>to keep that IP for the rest of the flow. - Solve a captcha with the Captcha API, and put the token in the page. Send
page.url()aswebsiteURL, and the solver uses the proxy of the session. See Solve captchas. - Watch the session live, or record it, to see what the site showed. See Live view and replays.
Search API
The Search API sends Google queries through a residential exit, and changes the exit when Google blocks it. When an engine shows a captcha page, Wayfern tries to solve it in the same browser and read the results again. A successful solve debits 1 credit, the same as a Captcha API solve.
If Wayfern cannot get results, the call returns 429 captcha_detected, 429 rate_limited or 403 blocked, with the engine in reason. When you ask for several engines, a failed engine is listed in meta.engines_failed and the results of the other engines still come back.
Captcha API
The Captcha API solves reCAPTCHA v2 and v3, hCaptcha, Cloudflare Turnstile, FunCaptcha, GeeTest, Amazon WAF, Friendly Captcha, Prosopo, ALTCHA and image captchas. The answer is always in solution.token.
When one of your browser sessions has the page open, the solver uses the proxy of that session. The token then comes from the IP that the page submits from. Send page.url() as websiteURL. A solved task costs 1 credit. Failed tasks cost nothing.
What Wayfern cannot promise
- A page can stay blocked on every exit. Wayfern then returns
blockedand does not bill the call. - A site can change its wall at any time, so a request that passes today can fail later.
- A site can refuse a captcha token after a successful solve. The solve is still billed, because the solver returned a token.
For status codes and retries, read Handle API errors and the Web Data errors.