Get past bot walls and challenge pages

How Wayfern handles bot walls and challenge pages: the anti-detect browser, the residential retry in auto mode, captcha solving, and what it cannot promise.

Updated

Wayfern gives you tools for bot walls. It does not promise that a page will load. This guide says what each tool does, where it applies, and what happens when it fails.

The anti-detect browser

Every browser session and every Web Data call runs in a Wayfern anti-detect browser.

  • When you set a viewport, Wayfern resizes the real window. It emulates nothing and runs no script in your pages, so the window and screen values that a page reads agree.
  • Live view and replays capture over a separate connection. They run no script in your pages.

Wayfern publishes no detection rate and no success rate.

Web Data: the residential retry in auto mode

Wayfern classifies the first navigation of a Web Data call:

ResponseResult
401, 402, 403 or 407blocked
A page that ends on a /cdn-cgi/challenge, /sorry, /captcha, /blocked or /access-denied pathblocked
429rate_limited
404 or 410not_found
5xxupstream_error

In auto mode, the default, a blocked result on the datacenter exit gets one more try through a residential exit. Wayfern does not retry other results. If the residential try is also refused, the call returns 403 with "error": "blocked". Failed calls are not billed.

Wayfern reads only the status code and the final URL. A challenge page that answers 200 at the same URL comes back as the page content. The Web Data API does not solve captchas.

curl -X POST https://api.wayfern.com/api/v1/scrape \
  -H "Authorization: Bearer $WAYFERN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/pricing", "proxy": "auto", "waitFor": 3000 }'
{
  "error": "blocked",
  "code": 403,
  "request_id": "3b7c1d0e-9f2a-4c58-8d31-6e0a5b4c7d92",
  "message": "blocked: https://example.com/pricing",
  "reason": "https://example.com/pricing"
}

Branch on error, not on message. Use waitFor or waitForSelector when a page builds its content after load.

Browser sessions

Sessions do not retry and do not solve captchas by themselves. Your code decides what to do:

  • Reconnect with a country on the connection URL to use a residential exit. Add session=<id> to keep that IP for the rest of the flow.
  • Solve a captcha with the Captcha API, and put the token in the page. Send page.url() as websiteURL, and the solver uses the proxy of the session. See Solve captchas.
  • Watch the session live, or record it, to see what the site showed. See Live view and replays.

Search API

The Search API sends Google queries through a residential exit, and changes the exit when Google blocks it. When an engine shows a captcha page, Wayfern tries to solve it in the same browser and read the results again. A successful solve debits 1 credit, the same as a Captcha API solve.

If Wayfern cannot get results, the call returns 429 captcha_detected, 429 rate_limited or 403 blocked, with the engine in reason. When you ask for several engines, a failed engine is listed in meta.engines_failed and the results of the other engines still come back.

Captcha API

The Captcha API solves reCAPTCHA v2 and v3, hCaptcha, Cloudflare Turnstile, FunCaptcha, GeeTest, Amazon WAF, Friendly Captcha, Prosopo, ALTCHA and image captchas. The answer is always in solution.token.

When one of your browser sessions has the page open, the solver uses the proxy of that session. The token then comes from the IP that the page submits from. Send page.url() as websiteURL. A solved task costs 1 credit. Failed tasks cost nothing.

What Wayfern cannot promise

  • A page can stay blocked on every exit. Wayfern then returns blocked and does not bill the call.
  • A site can change its wall at any time, so a request that passes today can fail later.
  • A site can refuse a captcha token after a successful solve. The solve is still billed, because the solver returned a token.

For status codes and retries, read Handle API errors and the Web Data errors.