Website to markdown: one page or a full site

Website to markdown conversion runs in three steps: fetch the page (plain HTTP or a rendered browser DOM), extract the main content, then map the HTML to markdown. The tool is your one decision: a free converter or a local library for a single public page, and a scrape API that renders for many pages, JavaScript-heavy pages or sites that block bots.

Your pipeline sends a URL into a website to markdown step and gets back an empty shell, a cookie banner or a login page instead of the article. Because you run jobs, one bad conversion repeats on every page in the queue.

You’ll be able to choose among a hosted converter page, a library you run yourself and a rendered-browser API, then call it, handle blocked pages and convert a whole site. Wayfern’s Web Data API is one option, and the other routes win where they fit better.

Key Takeaways

• A common cause of empty output is raw HTML fetched from a page that builds its content in the browser.
• Start with a rendered scrape and proxy: "auto"; move to your own CDP session only when the page needs a login.
• Filter discovered URLs before scraping, because every fetched page spends credits.
• Tables, code languages, images and media can degrade, so test a sample first.

How does website to markdown conversion work?

Website to markdown conversion fetches a page’s HTML (raw, or after a browser has rendered it), strips navigation, banners and ads to isolate the main content, then maps headings, links, lists and tables to markdown syntax. The result is plain text that language models and documentation tools can read.

Step 1: fetch. A plain HTTP GET returns the HTML the server sent. A rendered browser returns the DOM after scripts have run.

Playwright’s page.content() “Gets the full HTML contents of the page, including the doctype,” and chromium.connectOverCDP() attaches to an “existing browser instance using the Chrome DevTools Protocol.”

Three things break at this step. Single-page apps return a near-empty shell to a plain GET. Lazy-loaded content appears only after a scroll or click. Cookie banners belong to the rendered DOM, so they can reach the output.

A plain GET of a single-page app route returns a script tag and an empty container. Run page.content() after navigation and the same route returns the finished text.

Step 2: main-content extraction. Mozilla’s Readability is the published library for this job, and its README describes it as “used for Firefox Reader View.” It needs a DOM: you “create a new Readability object from a DOM document object.”

So Readability runs after the fetch, and a shell fetch gives it nothing to extract.

Step 3: mapping. Turndown’s README says it will “Convert HTML into Markdown with JavaScript.” The GitHub Flavored Markdown spec, which extends CommonMark, defines a table as “a single header row, a delimiter row separating the header from the data, and zero or more data rows.”

That structure cannot express a merged cell, so spans get flattened or dropped.

MarkItDown’s URL converter describes a simpler flow: the page “is fetched for you” and the conversion “runs locally in your browser.” It names no rendering step, so a script-built page reaches it as the server sent it.

How do you convert a single web page to markdown with an API?

You convert a single web page to markdown by sending its URL, with your wf_ token, to the Wayfern Web Data API scrape endpoint and reading the markdown from the response.

1. Set the token. Run export WF_TOKEN=your_wf_token. Keep it in the environment, never in the script.

2. Send the request. This curl call uses POST https://api.wayfern.com/api/v1/scrape with formats set to markdown, as the Web Data API reference shows.

curl -X POST https://api.wayfern.com/api/v1/scrape \
  -H "Authorization: Bearer $WF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/pricing", "formats": ["markdown"], "proxy": "auto"}'

3. Do the same in Python. Print the whole body first, so you see the real field names before you write parsing code.

import os, requests

r = requests.post(
    "https://api.wayfern.com/api/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['WF_TOKEN']}"},
    json={"url": "https://example.com/pricing", "formats": ["markdown"], "proxy": "auto"},
    timeout=120,
)
r.raise_for_status()
print(r.json())

4. Check the response. You get the markdown body, page metadata and the credits charged. The API reference says failed requests are not billed.

A successful call shows a nonzero charge. If the markdown body is empty, confirm proxy: "auto" is set and re-run.

The markdown has this shape. The sample is illustrative, not real output:

## Plans

See the [full comparison](https://example.com/compare).

| Plan | Seats |
|------|-------|
| Basic | 1 |
| Team | 10 |

Scrape costs 1 credit per page, and the output add-on is extra, according to Wayfern’s published rates.

Page captures are cached for 1 hour. A repeat call inside that hour can return the earlier capture, which saves a fetch but means a price that changed 10 minutes ago may not appear.

The honest catch: for one public, static page, an API is more setup than the job needs. Firecrawl’s free converter, for example, says “Paste any URL and get clean, well-structured markdown in seconds.” Many pages in one job call for the whole-site workflow instead.

Try the scrape endpoint with the /docs/api reference.

What does a website to markdown conversion lose?

A website to markdown conversion loses whatever markdown has no syntax for: cell spans, pixels, code language labels, media players and page chrome. Each loss has a step that limits it.

1. Tables lose merged cells. The GitHub Flavored Markdown (GFM) spec says “The header row must match the delimiter row in the number of cells.” Spans have no syntax, so split the merged cell or repeat its value. In JavaScript, Turndown handles tables through turndown-plugin-gfm, “A Turndown plugin which adds GitHub Flavored Markdown extensions.”

2. Images become references. Markdown keeps the image URL and alt text, not the pixels. Lazy-loaded images can carry a placeholder source instead of the real file. If the output shows placeholder URLs, fetch from a rendered browser or read the attribute that holds the real source.

3. Code blocks lose their language. A converter can label a fenced block only if the source marks the language, usually with a class such as language-python on the code element. Without that class you get a bare fence. Add the label in post-processing or match your converter to the site’s class naming.

4. Embedded media is dropped. Iframes and video players have no markdown form, so converters drop them or leave a link. If the player URL matters, read the src attribute from the HTML yourself.

5. Navigation, footers and cookie banners leak in. Main-content extraction with Mozilla’s Readability removes most of them. A selector does the same job when you know the content element: Microlink’s converter page says it handles “blogs, docs, SPAs, or JavaScript-rendered pages” and lets you block ads or target HTML selectors. Check /docs/api for a selector option on Wayfern scrape before you depend on one.

6. Less text, fewer tokens. Stripping markup and boilerplate shortens what a model reads. Context.dev puts it this way: “Markdown strips the noise and preserves structure: headings, lists, links, and code blocks.” No dated source publishes a percentage, so measure the saving on your own pages.

Which fetch do you need for blocked or login-walled pages?

The fetch you need for a blocked or login-walled page is the cheapest one that returns the content; escalate only when the response shows it isn’t enough.

Situation	Fetch	Credit consequence
Content is in the server HTML, no block	Plain HTTP fetch plus a library such as Turndown	None from Wayfern
Content appears after scripts run	Web Data API scrape	1 credit per page
Site blocks datacenter ranges or varies by country	Residential exit with a country	Page credit plus residential bandwidth
Page needs a login	Your own session over CDP	8 credits per open-page hour

With proxy: "auto", the Wayfern Web Data API starts on a datacenter exit and retries on residential when blocked. Residential bandwidth has its own rate on Wayfern’s pricing page.

A login-walled page takes three steps:

1. Connect Playwright to a link token with chromium.connectOverCDP(). Add a session id so one residential IP holds for the whole flow.
2. Restore your saved cookies before page.goto().
3. Read page.content() and convert it with Turndown, or pass it on.

const browser = await chromium.connectOverCDP(
  "wss://api.wayfern.com/ws?token=LINK_TOKEN&session=run42",
);
const context = browser.contexts()[0];
await context.addCookies(JSON.parse(await fs.readFile("cookies.json", "utf8")));
const page = await context.newPage();
await page.goto("https://example.com/account");
const html = await page.content();

Success means the account page’s HTML, not the sign-in form. Our guide to scraping behind logins covers the sign-in step itself.

The honest catch: Wayfern has no persistent profiles and no cookie import or export, so your code saves and restores cookies. A cookie issued to one IP and replayed from another is an easy signal for a site to read, which is why the sticky session id matters.

For session-bound fetches without a browser, check the headers and cookies options in Firecrawl’s scrape documentation.

How do you convert a whole website to markdown?

You convert a whole website to markdown by discovering its URLs, filtering them, then fetching each page at 1 credit per page, as Wayfern’s pricing lists it.

1. Discover URLs. Read the site’s sitemap.xml if it publishes one. Otherwise call the Wayfern map endpoint, which costs 2 credits per 10 URLs. To find pages across the web, the Search API can restrict results to one site.
2. Filter the list. Keep the URL patterns you want and drop login, cart and duplicate pages before you spend a credit on them.
3. Fetch each page. Loop over the scrape endpoint with bounded concurrency. The scrape limit is 60 requests a minute, listed on /docs/limits, so four workers that each wait 4.5 seconds stay under it.

import os, time, requests
from concurrent.futures import ThreadPoolExecutor

H = {"Authorization": f"Bearer {os.environ['WF_TOKEN']}"}
API = "https://api.wayfern.com/api/v1"

m = requests.post(f"{API}/map", headers=H, json={"url": "https://example.com"})
keep = [u for u in m.json()["links"] if "/docs/" in u and "/login" not in u]

def fetch(u):
    time.sleep(4.5)  # 4 workers stay near 53 requests a minute
    r = requests.post(f"{API}/scrape", headers=H,
                      json={"url": u, "formats": ["markdown"]})
    return u, r.json().get("markdown")

with ThreadPoolExecutor(max_workers=4) as pool:
    pages = dict(pool.map(fetch, keep))

Check the exact field names against /docs/api. The crawl endpoint is the managed alternative, and Wayfern’s API reference says it keeps results for 7 days. The RAG knowledge base use case shows that route.

For agents, Wayfern’s MCP server at https://api.wayfern.com/mcp exposes search_web, map_site and scrape_page. Authenticate with a Bearer wf_ token that carries the mcp scope. Rates and the credit balance match the REST API, and every result reports credits_charged.

A worked example: mapping 100 URLs costs 20 credits and scraping them costs 100, so 120 credits, or $1.50 at the $0.0125 list rate per credit. At 60 requests a minute, the scrape loop takes at least 2 minutes.

You carry the responsibilities. RFC 9309 says robots.txt rules are “not a form of access authorization”, so they’re a request convention, not access control. Read the site’s terms, collect only public data you may lawfully use, and throttle your own crawl.

When should you skip the API and use a free tool or library?

Skip the API when you’re converting one public page, because a free paste tool or a local library does that job with no per-page cost.

Approach	Best for	Limit
Free paste tools (Firecrawl, Microlink, MarkItDown)	One public page, right now	Vendor-set caps, one URL at a time
Local library (Turndown with GFM plugin, Mozilla Readability)	Pages whose content is in the server HTML	No rendering without a browser; you maintain it
Browser extension (web2md)	Pages behind your own login	Manual, one page per click
Hosted rendered-browser API (Firecrawl, Microlink API, Wayfern Web Data API)	Pipelines and JavaScript-heavy pages	Per-page cost; Wayfern has no free tier

Limits on the free converter pages are set by each vendor and change. Read Microlink’s and Firecrawl’s own docs and pricing pages for current caps.

Check the data path too. You submit a URL, and the tool fetches the page. Each tool’s privacy policy governs what happens next.

Turndown with its GFM plugin maps HTML to markdown, and Mozilla Readability extracts the main content. Both run on your machine at no per-page cost. They don’t render JavaScript unless you add a browser, and you maintain the code.

For your own session, web2md says its extension reaches “login-walled pages that server-side converters can’t reach.” It converts the page you’re viewing, so it’s manual per page.

The honest catch for Wayfern: there’s no free tier, and the cheapest route is the Developer plan or a credit pack. If Firecrawl or a local library fits your job, use it. Our web scraping buyer’s guide compares the wider field.

• Free paste tool: a single public page.
• Local library: static HTML you convert in bulk for free.
• Extension: a few pages behind your login.
• Rendered-browser API: many URLs, scripted pages, a pipeline.

Frequently asked questions

What’s the easiest way to convert a website to markdown?

For one public page, paste the URL into a free converter or run Turndown with Mozilla Readability locally. For many pages, JavaScript-heavy pages or blocked sites, call a scrape API that returns markdown and, when needed, renders the page in a browser.

Why does my converter return an empty page for a JavaScript site?

It fetched raw HTML, which is a shell until scripts run. Convert the rendered DOM instead. In Playwright, page.content() returns the full HTML of the rendered page.

Am I charged when a scrape fails?

Not on the Wayfern Web Data API: the API reference says failed requests are not billed. Log the credits each response reports. Rates are in Wayfern’s pricing docs.

Can I convert a page behind a login to markdown?

Yes, with a rendered browser that uses your own cookies. Log in over CDP, save the cookies and storage in your code, and restore them next run. Wayfern has no persistent profiles, so it won’t keep them.

Is it legal to convert someone else’s website to markdown?

This isn’t legal advice, and the answer depends on the site and your jurisdiction. robots.txt isn’t access control: RFC 9309 says it is not a substitute for valid content security measures. Read the site’s terms and collect only public data lawfully.

Next steps for your markdown pipeline

Today: Run one representative URL through a free converter, then through Turndown plus Readability. Compare both outputs to the page itself: the headings, tables, code blocks, and whether the body text is there at all.

This week: If the output is empty or blocked, call the scrape endpoint with proxy: "auto" and read the credits charged in the response. That number is your per-page cost for that site.

This month: Map one site, filter the URLs down to the sections you need, and scrape them in a loop that stays under the rate limit. Then decide whether an agent should call search_web, map_site and scrape_page through the MCP tools.

Create a Wayfern account, mint a wf_ token and run the scrape call from the docs quickstart. There is no free tier, but credit packs work without a plan, so you can buy a small pack and test your website to markdown pipeline on real targets. Rates are on the Wayfern pricing page.

Sources

Website to Markdown — Free Converter, Extension & API · Playwright Page class documentation · Playwright BrowserType class documentation · mozilla/readability README · GitHub Flavored Markdown Spec · Wayfern pricing · Wayfern API reference · RFC 9309: Robots Exclusion Protocol. Every other source is linked where the article cites it.