Book a CallCreate AccountLogin
Use case · AI training data & RAG

Turn the web into a dataset your model can read.

For ML and platform teams building RAG pipelines or training corpora who need current, clean, deduplicated text from thousands of sites without writing a scraper per site.

Buy NowSign up with Google
Illustration pending
The problem

Why this is hard to do well

RAG and training pipelines need clean, current text at scale, not raw HTML. The Map API finds the pages worth reading, the Crawl API returns them as deduplicated Markdown in one asynchronous job, and the Unlocker API handles the protected ones.

Raw HTML is 90% navigation and scripts; embedding it wastes tokens and degrades retrieval quality.

Sites block bulk crawlers, and a corpus with gaps where the useful sites were is a biased corpus.

Freshness matters for retrieval: the index has to be rebuilt on a schedule, not once.

The approach

Which products, in which order

1. Map APIfrom $0.042 / 1KMap each domain first and filter by keyword so you crawl the 180 pages that matter, not 5,000.
2. Crawl APIfrom $0.042 / 1KThe Crawl API walks them asynchronously and returns clean Markdown, deduplicated, with a webhook when done.
3. Unlocker APIfrom $0.042 / 1KProtected or JavaScript-heavy pages fall back to the Unlocker API inside the same job.
Architecture

How the pieces fit together

InputDomain listSeeds, depth and pattern limits
DatafuelDatafuelMap · Crawl · Unlocker fallbackCrawl APIMap APIUnlocker API
OutputCorpusMarkdown + metadata → vector store

Failed requests inside the Datafuel layer are retried on a fresh identity and never billed; only what reaches the output column costs anything.

Step by step

From zero to working

01Map each seed domain and filter URLs by keyword or path pattern.
02Start a Crawl job with depth and page limits so the cost is capped up front.
03Receive the webhook and pull Markdown plus metadata per page.
04Chunk, embed and index; keep the URL and last-updated fields.
05Re-run on a schedule and diff to update only changed pages.
Code example

Real and runnable

import requests
r = requests.post(
    "https://scraping-api.datafuel.ai/api/v1/task",
    headers={"Authorization": "Bearer $DATAFUEL_KEY"},
    json={
        "type": "crawl",
        "attributes": {
            "url": "https://docs.example.com",
            "max_depth": 3,
            "limit": 500,
            "include_paths": ["/guides/*", "/reference/*"],
            "output": "markdown",
            "webhook_url": "https://your.app/hooks/crawl"
        }
    },
)
print(r.json())
Replace $DATAFUEL_KEY with a free key and this runs as is.Run this with a free key
Products used

What you need, and what it costs

Crawl APIEvery page beneath a start URL as LLM-ready Markdown, in one asynchronous job.from $0.042 / 1KSee product
Map APIEvery URL of a site from its sitemaps and page links, deduplicated and filtered, for one flat credit.from $0.042 / 1KSee product
Unlocker APIAny page past any protection, as Markdown or structured JSON, in one request.from $0.042 / 1KSee product
Start freeAll prices on one page
What to watch for

Honest caveats

  • Respect robots.txt and licensing: a page you can fetch is not automatically a page you may train on.
  • Depth limits are your cost control; start shallow and widen once you have seen the output.
  • Boilerplate removal is good, not perfect; keep a sample review step in the pipeline.
Results

In numbers

1 creditper page crawled
512+tasks per bulk job [placeholder]
—pages / day across customers [placeholder]

Figures marked placeholder await real customer numbers; a case study replaces this block when one is available.

FAQ

Questions teams ask about aI training data & RAG

Questions teams ask about aI training data & RAG
Why Markdown instead of HTML?
Markdown keeps headings, lists and tables and drops navigation and scripts, which is what an embedding model needs.
How do I avoid paying for pages I do not need?
Map first and filter by keyword or path, then set depth and page limits on the crawl.
Can I bring my own AI key?
Yes. AI In-Flight processing runs on your provider key; tokens are billed by them, pages by us.
What happens when a site blocks the crawler?
Protected pages route through the Unlocker API automatically; failed pages are not billed.
Is there an MCP server?
Yes; agents can call Map, Crawl and Unlocker as tools.
How large can a job be?
Bulk jobs run asynchronously with concurrency limits per plan; enterprise plans raise them.
Related use casesAll use cases
GEO / AI visibilitySee how AI engines cite you, and your competitors, by country.LLM Scraper APIResidentialLearn moreMarket researchPublic web data from every market, collected the same way everywhere.ResidentialUnlocker APICrawl APIGuide coming soonSERP trackingUnblocked results from any city, at any volume.ResidentialMobileSearch APIGuide coming soon
Get started

Ready to build?

Start with the free tier and scale as your project grows. No credit card, no sales call.

Get API keyTalk to us
Talk to an engineer, not a chatbot.