LLM web scraping: build a fetch-to-JSON pipeline

LLM web scraping is a four-stage pipeline: fetch rendered content, convert it to markdown or trimmed HTML, extract fields with a JSON schema, then validate and retry. The model replaces selector writing only. Fetching, validation and monitoring stay your job, and as Browserless puts it, a bad fetch gives the model nothing real to read.

Your selector-based scraper worked until a site redesign changed the markup, and now it returns nothing. Most LLM web scraping tutorials start from HTML you already hold, as if the page always arrives intact. Sometimes it arrives as an empty shell or a challenge page.

We build and run an anti-detect Chromium, so we see what the fetch step actually returns. After reading, you can build a fetch, extract and validate pipeline, and choose per site between LLM extraction, selectors and a hybrid.

Key Takeaways

• An LLM changes none of the request’s IP, cookie-to-IP consistency or browser identity, so a model never makes a fetch undetectable.
• ScrapeGraphAI’s rule: pick selectors for stable, high-volume fields and an LLM when meaning matters more than markup; combine both when you need predictable cost.
• Every Wayfern Web Data API response reports the credits it debited, so you can track cost per record, and failed requests are not billed.

What is LLM web scraping and how does it work?

LLM web scraping uses a language model to turn page content into structured data, in four stages where only the extraction step is new.

1. Fetch and render. Browserless notes that most modern websites are JavaScript-rendered, so a real browser has to run before the model sees anything worth extracting.
2. Prepare the input. Strip scripts, styles and navigation, then convert the page to markdown or trimmed HTML. Fewer tokens means lower cost and less noise.
3. Call the model with a schema. Send the cleaned text plus a JSON schema that names each field and its type, such as name, price and currency.
4. Validate and store. Parse the output against the schema, retry on failure, then write the record to your database.

Only stage 3 differs from traditional scraping. ScrapeGraphAI describes the change as mapping page content to a requested schema using semantic context instead of relying only on CSS or XPath selectors.

Stages 1, 2 and 4 are still your code.

Why does the fetch layer decide what the LLM sees?

The fetch layer decides what the LLM sees because the model reads only what your request returns, so a bad fetch becomes bad input before extraction starts. Three failures look fine on the surface:

• JavaScript page: the server returns an empty shell, and the content never loads.
• Blocked request: you get a challenge page or an empty body, and the status code alone may not tell you.
• Rate-limited request: you get an error page, and the model will still try to fill your schema from it.

Each one hands the model plausible input, and you get a confident, wrong object back.

Open-source projects and hosted vendors both treat the model as one stage. The ScrapeGraphAI README describes “a web scraping python library that uses LLM and direct graph logic to create scraping pipelines” and points to a hosted version as “an even faster and simpler way to scrape at scale”.

Browserless says production pipelines need “browser infrastructure, session management, and error handling”. Someone still owns the fetch.

A hosted browser gives you a rendered DOM, an exit IP and a browser identity. Wayfern applies that identity inside its own Chromium build, and per its API reference, “no script runs in your pages”.

The default exit is a datacenter exit. Residential exits take country, region, city and isp, and a sticky session id holds one residential IP across a flow. You can also attach your own proxy.

That consistency matters because, as our page on anti-detect browser sessions over CDP puts it, “A cookie issued to one IP and replayed from another is an easy signal for a site to read.”

The model changes none of this. No fetch is undetectable, and detection is an arms race.

Add a guard before the model call. Reject empty bodies, tiny bodies and anything that looks like a challenge page, so you don’t spend tokens on them.

How do you build an LLM web scraping pipeline in code?

You build an LLM web scraping pipeline by wrapping one model call in checks that catch a bad fetch, a malformed answer and an invented value. This version uses the Wayfern scrape endpoint and a call_llm wrapper you point at any vendor’s JSON-schema mode.

import os, logging, requests
from jsonschema import validate, ValidationError

SCHEMA = {"type": "object", "required": ["name", "price"],
          "properties": {"name": {"type": "string"}, "price": {"type": "string"}},
          "additionalProperties": False}
MARKDOWN_KEY = "markdown"  # confirm this field name in /docs/api
BLOCKED = ("just a moment", "verify you are human", "access denied")

def fetch(url):
    r = requests.post("https://api.wayfern.com/api/v1/scrape",
        headers={"Authorization": f"Bearer {os.environ['WF_TOKEN']}"},
        json={"url": url, "formats": ["markdown"], "proxy": "auto"}, timeout=120)
    r.raise_for_status()
    data = r.json()
    logging.info("%s credits=%s", url, data.get("credits"))
    return data[MARKDOWN_KEY]

def usable(md):
    return len(md.strip()) > 200 and not any(b in md.lower() for b in BLOCKED)

def trim(md, limit=20000):
    lines = [l for l in md.splitlines() if len(l.split()) > 3]
    return "\n".join(lines)[:limit]

def grounded(data, source):
    return all(str(v) in source for v in data.values())

def extract(url, call_llm, attempts=3):
    md = fetch(url)
    if not usable(md):
        raise RuntimeError(f"empty or challenge page: {url}")
    page, error = trim(md), ""
    for _ in range(attempts):
        data = call_llm(f"Extract the fields.\n{error}\n---\n{page}", SCHEMA)
        try:
            validate(data, SCHEMA)
            if grounded(data, page):
                return data
            error = "A value was not found in the page text. Copy values exactly."
        except ValidationError as e:
            error = f"Previous output failed validation: {e.message}"
    logging.error("extraction failed: %s", url)
    raise RuntimeError(f"extraction failed after {attempts} attempts: {url}")

1. Fetch markdown. The proxy: "auto" setting starts on a datacenter exit and retries on residential when blocked. Field names and options are in the Web Data API reference.
2. Reject bad pages. A challenge page or empty shell sent to a model gets a confident, wrong answer. usable() checks before you pay for tokens.
3. Trim. Dropping navigation and link-only lines before the call is your main lever on token cost for long pages.
4. Call the model. Keep call_llm(prompt, schema) vendor-neutral so you can swap models without touching the pipeline.
5. Validate. jsonschema catches missing fields and wrong types. additionalProperties: False blocks invented keys.
6. Check grounding, then retry. A schema-valid price can still be made up. Evidence quotes checked with ungrounded(), shown in the hallucination section, are stricter than this substring test.

Cap retries at 3, append the error each time, and fail loudly with the URL logged. Silent fallbacks put bad rows into your dataset.

Playwright over CDP, Web Data API or MCP: which should feed your LLM?

Pick Playwright over CDP when the page needs a login or clicks, the Web Data API when the page is public, and the MCP server when an agent must find URLs before it reads them.

Path	Use when	What you handle	Cost signal
Playwright over CDP	Login, clicks, scrolling, multi-step flows	Selectors, waits, cookies	Open-page hours
Web Data API (scrape, extract)	Public pages, markdown or JSON in one call	Schema, validation	Per page
MCP server plus Search API	Agent finds URLs, then reads them	Tool loop, credit cap	Per search, per page

Playwright’s documentation says connectOverCDP “attaches Playwright to an existing browser instance using the Chrome DevTools Protocol”. A minimal Python session looks like this:

browser = await p.chromium.connect_over_cdp("wss://api.wayfern.com/ws?token=LINK_TOKEN&country=US&session=run42")
page = await browser.contexts[0].new_page()
await page.goto("https://example.com/login")
html = await page.content()  # into the pipeline above

Opening the socket starts the session. Wayfern’s docs say to expect a cold start of about 45 to 50 seconds, longer with a residential exit.

Per Wayfern’s pricing, browser time costs 8 credits per open-page hour. For the sign-in flow, see scraping behind logins.

For public pages, scrape and extract cost 1 credit per page, per Wayfern’s pricing. Pick this path unless a click is required, because browser time is the pricier path.

An agent that has no URL yet needs discovery. The MCP server at https://api.wayfern.com/mcp takes a Bearer wf_ token with the mcp scope and exposes search_web, scrape_page and extract_data.

Search costs 2 credits per provider results page. Try it with your own token using the MCP server docs, and see live web data for AI agents for agent setups.

The honest catch: Wayfern has no persistent profiles, so save and restore cookies in your own code. It has no per-vector fingerprint control, no mobile emulation, only Chromium, and no free tier.

If privacy or local models matter more than a hosted fetch layer, run the open-source ScrapeGraphAI library with Ollama.

How do you stop an LLM from hallucinating scraped values?

You stop an LLM from hallucinating scraped values by checking every answer in code, because no model setting makes an answer true.

Schema validation comes first. OpenAI’s documentation says Structured Outputs means the model “will always generate responses” that match your JSON Schema. Anthropic’s documentation says structured outputs “constrain Claude’s responses to follow a specific schema”.

Both promise shape, not truth: a valid price field can still hold the wrong number. Add four more checks:

1. Allow null. Mark every field nullable and add a not_found option, so the model never has to guess a required value.
2. Ask for evidence. Request a short verbatim quote per field, then run the check below.
3. Check types and ranges. Require price greater than 0, a date inside a plausible window, and an enum for currency.
4. Retry once with the error. Send the failing field and the reason back to the model. If it fails again, route the record to review or fall back to a CSS selector.

import re

def norm(s):
    s = re.sub(r"\s+", " ", str(s)).lower()
    return re.sub(r"[$€£,]", "", s).strip()

def ungrounded(fields, source_md):
    src = norm(source_md)
    return [k for k, f in fields.items()
            if f["value"] is not None
            and not (norm(f["evidence"]) in src and norm(f["value"]) in src)]

A non-empty return list means a retry. Success is an empty list on every record.

The realistic error is a misread, not an invented value. The model picks the list price instead of the sale price, or reads 03/04/2026 as March when the site means April.

Both strings exist in the source, so the grounding check passes. Catch them with evidence quotes that include the label next to the number.

These checks reduce wrong values. They don’t remove them.

LLM extraction, CSS selectors or a hybrid: which should you use?

Use selectors for stable, static, high-volume pages, LLM extraction for irregular layouts, many different sites and one-off jobs, and a hybrid when you want predictable cost with a fallback.

Site type	Best approach	Caveat
Stable, static, high-volume pages	CSS selectors	Break on a redesign, and you rewrite them
Irregular layouts or many different sites	LLM extraction	Token cost, latency, misread values
One-off job	LLM extraction	Still needs fetching and validation
Known site that changes now and then	Hybrid	Two code paths to test

ScrapeGraphAI’s guide says LLM extraction “adds model cost, latency, and output-validation requirements.”

Cost grows with page length, because you pay for every token you send, so trim to markdown first. Latency is one model call per page.

Schema changes ripple. Change a field and the schema, prompt and validator all change, so old records stop matching new ones.

The arXiv paper “Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users” benchmarks LLM-powered scraping for non-specialists. Read its per-tier results against your own targets before you pick.

The hybrid is a small amount of code. Run the selector on the known path. If it returns nothing or fails validation, send the trimmed page to the LLM and log the fallback.

Watch that fallback count. A sudden jump probably means a layout changed.

For fetch tools that pair with either approach, see web scraping tools compared by pages and sessions.

Is LLM web scraping allowed? robots.txt, terms and data rights

LLM web scraping is lawful or not under the same three layers as any crawler: robots.txt, the site’s terms, and data rights. The model call changes none of them.

robots.txt. RFC 9309 defines the Robots Exclusion Protocol as a way for service owners “to control how content served by their services may be accessed” by crawlers. The RFC also states: “These rules are not a form of access authorization.”

That makes robots.txt a request about crawler behavior. It isn’t a lock, and it isn’t a legal ruling either way.

Site terms. A site’s terms can limit automated access or reuse of its content, whatever its robots.txt says. Read them for every host you collect from.

Data rights. Personal data and copyrighted content carry their own rules. A page being publicly visible doesn’t remove them.

Sending page text to a third-party model API is also a data-handling decision. You pass that text, and any personal data inside it, to another processor, so check that vendor’s retention terms first.

Run these checks before each job:

• Fetch and read robots.txt before you crawl a host.
• Keep request rates polite, so you don’t strain the site.
• Skip login-gated pages and personal data unless you have a lawful basis.
• Record what you collected, from where, and why.

Wayfern’s terms of service require lawful use: we build for public-data collection, monitoring, research, testing and agents.

This is general information, not legal advice. Ask counsel about your own case.

Frequently asked questions

What is the best way to do LLM web scraping?

Fetch the rendered page with a browser or scraping API, convert it to markdown, ask the model for JSON that matches a schema, validate it in code, and retry on failure. Use selectors instead when the site is static and stable, because they add no model cost.

Can an LLM scrape a JavaScript-heavy site by itself?

It can’t, because the model reads only the text you send it. Browserless notes that most modern sites render with JavaScript, so a real browser has to run before the model sees anything worth extracting. Use a CDP session or the Web Data API to get the rendered page.

Does using an LLM make scraping undetectable?

The model doesn’t change the fetch, so a site still sees your IP, cookies and browser identity. Wayfern doesn’t claim undetectability either. A challenge page just reaches the model as content, so check for one before you extract.

How do I check that the model didn’t invent a value?

Validate the JSON against your schema, then confirm each extracted string appears in the source markdown. Normalize whitespace and number formats before you compare. Reject any record that fails either check and retry it.

How much does LLM web scraping cost per page?

There are two costs. The fetch is 1 credit per page for scrape or extract, and Wayfern’s pricing page has current plans. The model cost is input tokens for the markdown, so check your vendor’s current pricing page, and expect long pages to cost more.

What should you do next with LLM web scraping?

Today: Pick one target page and write its JSON schema with nullable fields, so a missing value comes back as null instead of a guess.

This week: Run the fetch, guard, extract, validate and retry pipeline on 20 pages you choose. Log each failure by cause: empty fetch, schema failure or grounding failure.

This month: Decide per site between an LLM, selectors or a hybrid using the site-type table, then add monitoring so drift shows up in a log before it reaches your data.

Start your LLM web scraping with Wayfern’s Web Data API for public pages, or a hosted browser over CDP when a page needs a login. There’s no free tier. The cheapest self-serve options are a credit pack or the Developer plan, detailed on the plans and credit packs page.

Sources

LLM Web Scraping: Extract Data with AI - browserless.io · LLM Web Scraping: How AI Models Replace Scrapers · ScrapeGraphAI/Scrapegraph-ai: Python scraper based on AI… · API Reference - CDP, Web Data & Search | Wayfern · Anti-Detect Browser Sessions over CDP | Wayfern · Pricing - Credits for Browsers & the Web Data API | Wayfern · RFC 9309: Robots Exclusion Protocol · Wayfern terms of service. Every other source is linked where the article cites it.