MCP server for web data: Setup, costs, limits

An MCP server for web data exposes search, scrape and crawl as tools an agent calls on demand. Wayfern’s hosted server is at https://api.wayfern.com/mcp: send a Bearer wf_ token with the mcp scope, and every tool result reports credits_charged against one shared credit balance.

If you’re wiring an MCP server for web data into an agent, tool counts are easy to compare. The cost per call, the behavior at zero balance and the limits take longer to find.

After reading, you can connect one server to your agent client, price an agent loop in credits, and choose between scrape-style servers and browser-control servers for a given job.

Key Takeaways

• Failed Web Data work isn’t billed, and per Wayfern pricing a zero balance refuses calls, and auto top-up is off by default.
• Send the token in an Authorization header as a scoped, revocable credential, never in a URL query string.
• No MCP web-data server can guarantee a request avoids a block, so decide first between fetching pages and driving a browser.

What is an MCP server for web data?

An MCP server for web data is a service that exposes web tasks such as search, scraping and crawling as tools an AI agent can call over the Model Context Protocol. The client lists the tools, the agent picks one, and the server returns page content or search results.

Discovery comes first. The MCP specification on tools says: “To discover available tools, clients send a tools/list request.” Each tool comes back with a name, a description and an inputSchema, a JSON Schema for its parameters.

The agent then calls one tool with arguments that match that schema.

Every listed definition is text the model carries in context. Load several servers and those definitions add up, whether or not the agent uses them.

Older descriptions of remote MCP say HTTP plus Server-Sent Events. The specification’s transports page now lists two standard transports, stdio and Streamable HTTP, and says Streamable HTTP replaces the HTTP+SSE transport from protocol version 2024-11-05.

With stdio, the client launches the server as a subprocess on your machine.

How do you connect the Wayfern MCP server to an agent?

1. Copy the endpoint.
2. Mint a wf_ token with the mcp scope.
3. Send it as the header Authorization: Bearer wf_.... The account needs an active plan or a positive credit balance, or calls are refused.

A generic remote-server entry has this shape:

{
  "mcpServers": {
    "wayfern": {
      "url": "https://api.wayfern.com/mcp",
      "headers": {
        "Authorization": "Bearer $WF_TOKEN"
      }
    }
  }
}

Clients name the URL and header keys differently, and not every client expands environment variables. Check the MCP docs for your client’s exact syntax before you paste the entry.

If a call is refused, check the token’s scope and the account balance.

Why the token goes in a header

The MCP authorization specification states: “Access tokens MUST NOT be included in the URI query string.” RFC 6750 gives the reason: the URL containing a token is likely to be logged, so the query-string method should be avoided when a header is possible.

Proxies, server logs and browser history record URLs. They don’t record your Authorization header by default.

What a leaked token can do

It can still spend credits on MCP tool calls, so set an expiry when you mint it.

Mint one token per client or environment. If one leaks, revoke it and the others keep working.

Which tools does this MCP server for web data expose, and what do they cost?

Here’s what each tool in this MCP server for web data does and what it costs per call.

Job	Tool	What it does	Credits per call	What it returns
Search	search_web	Runs a live query on Google, Bing or DuckDuckGo	2 per provider results page	Rows with rank, absolute_rank, type, ad flag, url and engine
Search	list_search_engines	Lists the engines you can pass to search	Free	Engine names
Read a page	scrape_page	Fetches one page	1 per page	Markdown, HTML, links or JSON
Read a page	extract_data	Turns a page into structured data	1 per page	JSON
Read a site	map_site	Discovers URLs on a site	2 per 10 URLs	A list of URLs
Read a site	crawl_site	Starts a multi-page crawl	1 per page	A crawl job to poll
Read a site	crawl_status	Checks a running crawl	Free	Crawl progress and results
Capture	capture_screenshot	Renders a page	5	Screenshot or PDF
Brand and design	brand_profile	Pulls brand data for a site	10	Brand profile
Brand and design	site_styleguide	Pulls a site’s styleguide	10	Styleguide
Brand and design	site_fonts	Lists the fonts a site uses	5	Font data

Every result reports credits_charged, so your agent can total its own spend as it goes.

One wf_ token with the mcp scope and one credit balance cover all of these jobs. Your client config holds one endpoint and one Authorization header instead of separate keys for search, scraping and screenshots.

The free tools cover the calls an agent repeats. A crawl runs asynchronously, so the agent can poll crawl_status as often as it needs. It can also call list_search_engines before a search to check which engines it can name.

All 11 definitions load whether the agent uses one tool or all of them, so the context overhead stays fixed. The full rate table, including plan credits and packs, is on the Wayfern pricing page.

What happens when your credits run out mid-session?

When your Wayfern balance reaches zero mid-session, the MCP server refuses further calls and closes open sessions, so the agent’s next tool call fails instead of running on credit you didn’t buy.

The pricing page states the rule in three lines: “Hard stop at zero. Nothing runs on credit you did not buy. Auto top-up is off until you turn it on.”

Credit packs never expire and are spent after plan credits.

Three habits keep an agent run from ending in a surprise:

• Have the agent add up credits_charged for the run and stop at a budget you set.
• Treat a refused call as an empty balance and end the run instead of retrying in a loop.
• Decide on auto top-up deliberately. With it on, an unattended loop can keep spending past your plan; with it off, the worst case is a stopped run.

How do you run search-then-read and crawl workflows through MCP?

You run both workflows by mapping each job to specific tools: search_web then scrape_page for reading, and map_site, crawl_site and crawl_status for building a RAG corpus. The AI agents use case shows the wider pattern.

Search then read

This script connects with the official MCP TypeScript SDK, lists tools, searches, scrapes the first organic URL and prints what each call cost. Confirm the argument names against the input schemas that listTools() returns and the tool reference in the MCP docs.

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";

const transport = new StreamableHTTPClientTransport(
  new URL("https://api.wayfern.com/mcp"),
  { requestInit: { headers: { Authorization: `Bearer ${process.env.WF_TOKEN}` } } },
);
const client = new Client({ name: "search-then-read", version: "1.0.0" });
await client.connect(transport);

console.log((await client.listTools()).tools.map((t) => t.name));

const parse = (r: any) => JSON.parse(r.content[0].text);

const search = parse(await client.callTool({
  name: "search_web",
  arguments: { text: "streamable http mcp transport", engines: ["google"], limit: 10 },
}));
console.log("engines_failed:", search.meta?.engines_failed);
console.log("search credits:", search.credits_charged);

const organic = search.results.find((r: any) => r.rank != null && !r.ad);
const page = parse(await client.callTool({
  name: "scrape_page",
  arguments: { url: organic.url },
}));
console.log("scrape credits:", page.credits_charged);

Pick rows by rank. On Wayfern’s SERP API page, rank counts organic results only. absolute_rank counts every row including ads, so selecting by it can send the agent to a sponsored link.

One search results page costs 2 credits and each scrape costs 1, per the published credit rates. A loop with one search and three scrapes is 2 + 3 = 5 credits. The credits_charged fields report what each call actually cost.

Crawl for RAG

1. Call map_site to list URLs and pick the paths you want.
2. Call crawl_site once. It returns before the crawl finishes.
3. Poll crawl_status until it reports completion. crawl_status costs nothing, so polling is cheap. Calling crawl_site again starts, and bills for, a second crawl.
4. Store the output.

Two failures that look like success

A blocked engine often looks like an empty result list. When search_web returns nothing, check meta.engines_failed, then retry or add another engine from Google, Bing and DuckDuckGo.

A partial or empty crawl result means the job hasn’t finished. Keep polling crawl_status.

How do you choose an MCP server for web data?

Choose an MCP server for web data by settling one fork first: whether the agent needs a page’s content in one call or control of a browser one step at a time.

Check	Scrape, extract and search tools	Step-by-step browser control
Calls per task	One or two	Many (navigate, snapshot, click, type)
Fits	Search then read, crawl for RAG	Logins, forms, multi-step flows
Context cost	Few tools, small schemas	More tools, more round trips
Examples	Wayfern, Firecrawl, Bright Data	Playwright MCP

According to fastcrw’s list of scraping MCP servers, Firecrawl MCP offers screenshots and structured extraction through MCP. Playwright MCP gives full browser control for complex single-page apps and interactions.

Bright Data’s README describes one server that covers web search, page scraping, structured data extraction and browser automation, so some servers sit on both sides of the fork.

Then work down this checklist:

• Hosting: a hosted server needs a URL and a token. A local server needs a process on your machine, and you own its upgrades.
• Tool count: every tool you load adds its description and input schema to the model’s context, so fewer, well-scoped tools or selectable tool groups cost less.
• Geo-targeting: check whether you can set a country per call, and what happens when the location isn’t available.
• Output formats: confirm the server returns what your pipeline reads, whether markdown, HTML, links, JSON, screenshots or PDF.
• Concurrency and rate limits: find the cap that applies to your plan before you size an agent loop.
• Free allowance: the Bright Data README states a free tier of 5,000 requests per month, with no credit card required.

Read the Wayfern MCP docs for per-tool inputs and costs, and Wayfern plan pricing for credit rates and concurrency by plan. Our buyer’s guide to tools for AI agents covers cloud browsers and proxy networks too.

Which server fits which job:

• Wayfern MCP: agents that search, read and crawl on one token and one balance.
• Playwright MCP: local, step-by-step control when the agent must click and type through a flow.
• Firecrawl MCP: page extraction with screenshots and structured output.
• Bright Data MCP: teams that want to try a free tier first.

What does an MCP server for web data not do?

An MCP server for web data returns pages to an agent, and none of them, Wayfern’s included, can guarantee a request avoids a block. Each site decides per request, so an unblocking claim that names no mechanism is marketing.

The auto proxy mode starts on a datacenter exit and retries on residential when blocked. It’s a retry strategy, and a page can still fail.

The Wayfern MCP server for web data handles scraping and search; it isn’t a profile manager. It doesn’t offer:

• Named profiles, saved cookie jars or cookie import and export.
• Per-vector fingerprint switches.
• Mobile device emulation.
• Firefox or WebKit (Chromium only).
• Downloadable binaries for self-hosting.

If a person manages many long-lived identities, a desktop profile manager such as Multilogin fits better. Our comparison of Multilogin alternatives for automated scraping covers where each one wins. For step-by-step control of a browser on your own machine, run a local Playwright MCP server.

Use the server for lawful work only: public-data collection, testing, research and monitoring.

Frequently asked questions

Which MCP server should I use for web data?

Pick by the job. If the agent needs search results and page content as markdown or JSON, use a scrape-and-search server. If it must click through a login or form, use a step-by-step browser control server such as Playwright MCP. Check tool count, hosting, geo-targeting and rate limits first.

What transport do remote MCP servers use?

Per the MCP transports specification, remote servers use Streamable HTTP, which replaced the older HTTP plus SSE transport. Local servers use stdio. The Wayfern server at https://api.wayfern.com/mcp is a Streamable HTTP endpoint.

Can I put my API token in the MCP server URL?

No. Send it in the Authorization: Bearer header. The MCP authorization spec forbids access tokens in the URI query string. A header-carried wf_ token with only the mcp scope can also be revoked if it leaks.

What happens when my credits run out?

Calls are refused at a zero balance and open sessions close. Auto top-up is off by default, so nothing bills beyond the credits you bought. Wayfern’s pricing page has the details.

Where do I find setup steps for the Wayfern MCP server?

Setup steps are in the MCP docs.

Connect, budget one loop and test on your targets

Start today. Mint a wf_ token with the mcp scope and add your MCP server for web data to one client, with the token in the Authorization header.

This week, run the search-then-scrape loop on your real queries and sum credits_charged across each run to set your per-run budget. Current rates are on our pricing page.

Before production, decide whether to turn on auto top-up, and whether any logged-in step needs a browser-control server instead. Then test your MCP server for web data against your own target sites.

Create a Wayfern account and start with the MCP docs. One client and one small loop keep the first test’s cost to what you measure.

Sources

MCP Server for AI Agents | Wayfern Docs · API Reference - CDP, Web Data & Search | Wayfern · Pricing - Credits for Browsers & the Web Data API | Wayfern · Tools - Model Context Protocol Specification · Transports - Model Context Protocol Specification · Authorization - Model Context Protocol · SERP API | Wayfern · Transports - Model Context Protocol. Every other source is linked where the article cites it.