Crawl API
One call walks a whole site and reads every page beneath the start URL, handing each one back as LLM-ready Markdown instead of raw HTML. The crawl follows links breadth-first, stays on your site and under your path, and fetches every page with the Unlocker engine, so protected sites are not a special case.
- One call, hundreds of pages: breadth-first crawl under your start URL, same site only
- max_pages, max_depth and RE2 path filters draw the perimeter before it starts
- Markdown or JSON for every page, clean enough to embed as is
- Asynchronous: 202 with a job_id, then a webhook or a status call when it is done
- Deduplicated: one entry per page, whatever the number of links to it
- 1 credit per page on Basic, 5 with js_rendering; failed pages refunded
Sign up with GoogleMCP readyConnect your agent in one command:
npx -y @datafuel/mcp initDatafuelCrawl API
Processing
100 pages
POST/api/v1/crawlurlhttps://react.dev/learnmax_pages · max_depth100 · 3result_formatmarkdown
Each page is a regular Unlocker task: 1 credit when queued on Basic, 5 with js_rendering, refunded if it fails. Demo data; the pace is not measured latency.202 Accepted · job_id returned before any page is fetched
Following links breadth-first · depth 0 → 3
Completed · 100 pages queued, stop_reason: max_pages
react.dev/learn
pending
processing
completed
discovered
0
1
14
41
88
131
131
enqueued
0/ 100
1/ 100
12/ 100
37/ 100
74/ 100
100/ 100
100/ 100
done
0
0
1
12
41
79
100
skipped
0
0
2
4
14
31
31
total_cost
0cr
1cr
12cr
37cr
74cr
100cr
100cr
done / max_pages 100
stop_reason: null
stop_reason: max_pages
depth 0
/learn
depth 1
/learn/installation/learn/describing-the-ui/learn/adding-interactivity/learn/managing-state/learn/escape-hatches/learn/thinking-in-react
depth 2
/learn/your-first-component/learn/writing-markup-with-jsx/learn/passing-props-to-a-component/learn/responding-to-events/learn/state-a-components-memory/learn/render-and-commit+31 more
depth 3
/learn/conditional-rendering/learn/rendering-lists/learn/keeping-components-pure/learn/updating-objects-in-state+54 more
skipped
/blog/…outside /learn/reference/reactoutside /learn
GET /crawl/{job_id}/results
{
"pages": [
{ "url": "https://react.dev/learn", "depth": 0, "status": "completed",
"result": "# Quick Start\nAn introduction to the React concepts you will use every day…" },
{ "url": "https://react.dev/learn/installation", "depth": 1, "status": "completed",
"result": "# Installation\nReact is designed for gradual adoption, so start with as little as you need…" },
{ "url": "https://react.dev/learn/your-first-component", "depth": 2, "status": "completed",
"result": "# Your First Component\nComponents are one of the core concepts of React…" },
{ "url": "https://react.dev/learn/rendering-lists", "depth": 3, "status": "completed",
"result": "# Rendering Lists\nYou will often want to display several similar components from a collection of data…" },
… 96 more pages
]
"next_cursor": "eyJvZmZzZXQiOjEwMH0"
}
One POST with the start URL. A crawl is a job: the API answers with a job_id at once and you poll GET /crawl/{job_id} for progress.
The start page is fetched, its links are queued at depth 1, theirs at depth 2, and so on. Only same-site links under /learn are followed; the rest are skipped and never billed.
100 pages queued and done. max_pages stopped this crawl; max_depth or running out of credits would too, and stop_reason tells you which.
GET /crawl/{job_id}/results pages through every crawled page as Markdown, with its depth. Pass next_cursor back as cursor to continue.
Everything in one job
Six things the endpoint does, at a glance.Multi-page traversalOne start URL; the crawl follows links breadth-first and reads every page beneath it.
Depth and pattern controlmax_depth, max_pages and RE2 include/exclude paths set the perimeter before it starts.
Markdown or JSON outputresult_format returns clean Markdown for embedding, or structured JSON per page.
Asynchronous with webhook202 and a job_id at once; a webhook fires when the job completes. Status on demand.
DeduplicatedOne entry per page, however many links point to it. Same-site URLs only.
Safety limitsNever more pages than max_pages, never off-site, failed pages refunded.
Datafuel / CrawlCrawling184 pages ready
POST/api/v1/crawl
"url": "https://docs.example.com/""max_pages": 200
.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md.md
184 pages crawled · 184 Markdown files1 call · 184 credits
01 · Corpus
From one URL to a complete corpus
Give the crawl a start URL and it walks every page beneath it, breadth-first, and hands each one back as clean Markdown. One call, one dataset: no link lists to maintain, no per-page requests to orchestrate.
Demo data. Page counts are illustrative; real crawls follow the site's links up to your limits.
Datafuel / CrawlEstimatingBounded
max_depth53
max_pages1,000200
include_paths["/docs/*"]
pages reachabledepth 11depth 224depth 3310depth 41,900depth 56,400
Maximum this job can cost1,000 credits200 credits
02 · Controls
Controls that avoid surprises
max_depth and max_pages draw the perimeter before the crawl starts, and include/exclude patterns narrow it further. The job never queues more pages than you allowed, so the cost is bounded and predictable, not an open-ended bill.
Demo data. Estimates use Basic pricing, 1 credit per page; js_rendering pages cost 5.
Datafuel / CrawlConvertingEmbedded
The Markdown is what you embed. No cleanup step in between.
03 · RAG ready
Ready for RAG and AI pipelines
Each page comes back as structured Markdown: headings, lists and tables preserved, navigation, scripts and boilerplate removed. No post-processing step: chunk it, embed it, retrieve it.
Demo data. Chunking and embeddings happen in your pipeline; the API returns the Markdown.
Datafuel / CrawlJob runningCompleted
POST/api/v1/crawl"webhook": "https://app.example.com/hooks/crawl"you
202 Accepted"job_id": "crw_8f3a91c2"< 1 s
queuedrunning · fetching pagescompleted0 / 500 pages236 / 500 pages500 / 500 pages
Datafuel side. Your app is idle: no polling, no open connection.POSThttps://app.example.com/hooks/crawl"status": "completed"waitingdelivered · 200
500 pages · one job · one notificationGET /crawl/crw_8f3a91c2/results
04 · Async
Asynchronous at scale
A crawl is a job, not a request: you get a 202 and a job_id immediately, the crawl runs on our side, and a webhook tells you when it is done. You block nothing and poll nothing; status and paged results are there whenever you ask.
Demo data. job_id, timings and page counts are illustrative.
Pricing
Pay only for successful requests
Pick a monthly plan, buy credits once, or run unlimited threads. Failed requests are never billed.
Every plan includes every endpoint and every interface (API, MCP). One shared credit balance: you only pay for volume, and only for successful requests.
How many requests a month?995,860
≈ 995,860 credits · Growth1K300K1M3.5M12MCustom
Free
Test every endpoint with real credits. No card, no expiry pressure.
$0/ mo, billed monthly
1,000 credits / month1,000 pages crawled2 concurrent threads
- 1,000 trial credits
- Every endpoint, API + MCP
- Community support
GrowthBest fit
For production crawlers and agents that need headroom.
$79/ mo, billed monthly
1,000,000 credits / month1,000,000 pages crawled25 concurrent threads$0.079 per 1K pages$0.395 per 1K with JS
- Everything in Starter
- Auto top-up
- Priority chat support
Business
For teams shipping data products on a schedule.
$199/ mo, billed monthly
3,500,000 credits / month3,500,000 pages crawled50 concurrent threads$0.057 per 1K pages$0.284 per 1K with JS
- Everything in Growth
- Usage alerts per API key
- Account manager
Scale
High volume at the lowest per-page rate. Checks out instantly.
$499/ mo, billed monthly
12,000,000 credits / month12,000,000 pages crawled150 concurrent threads$0.042 per 1K pages$0.208 per 1K with JS
- Everything in Business
- Dedicated IP pool
- 99.9% SLA
Enterprise
Committed volume, invoicing and governance.
Custom
Custom credits / month250+ concurrent threads
- Custom volume pricing
- Custom concurrency (250+)
- SSO · audit log · DPA
Compare
| Free | Starter | Growth | Business | Scale | Enterprise | |
|---|---|---|---|---|---|---|
| Credits / month | 1K trial | 300K | 1M | 3.5M | 12M | Custom |
| Concurrent threads | 2 | 10 | 25 | 50 | 150 | 250+ |
| API keys | 1 | 3 | 10 | 25 | Unlimited | Unlimited |
| JS rendering | ||||||
| Residential proxies | ||||||
| AI in-flight (your key) | ||||||
| Geo-targeting · 195+ countries | ||||||
| Batch multi-URL | ||||||
| Unlocker · Crawl · Map · LLM Scraper | ||||||
| Auto top-up | — | — | ||||
| Support | Community | Priority chat | Account manager | Slack + phone | Dedicated + SLA |
What does a request actually cost?1 credit = 1 basic page · costs don't stack: JS + Residential is 20 credits, not 5 + 10
1×Basic HTTPSimple HTML, datacenter proxy
5×JS RenderingHeadless browser for dynamic sites
10×Residential proxyReal residential IPs
20×JS + ResidentialFull power for protected sites
FreeAI In-FlightLLM extraction on any Unlocker request, zero extra credits
FAQ
Frequently asked questions
Scope, results and billing, answered straight.
Crawl APIScope, depth and results.
What counts as a successful request?
It starts at your URL and follows links breadth-first, staying on the same site and under the start path. max_pages is a hard stop on pages queued, max_depth limits how many links away from the start it goes, and include_paths / exclude_paths (RE2) narrow it further; exclude always wins.
What counts as a successful request?
A job. POST /api/v1/crawl answers 202 with a job_id at once. Poll GET /crawl/{job_id} for status and page counts, then page through GET /crawl/{job_id}/results with next_cursor when it completes.
What counts as a successful request?
The page as Markdown by default (set result_format for other formats), with its URL and depth. Every page is fetched with the full Unlocker options, so protected or JavaScript-heavy sites are not a special case.
What counts as a successful request?
When you only need the URLs. Map returns every same-site URL from sitemaps and on-page links in one call for a flat credit, which is the cheapest way to size a site or pick exactly which pages to fetch.
BillingCredits, failures and limits.
What counts as a successful request?
Per page queued: 1 credit on Basic proxies, 5 with js_rendering, and the Premium rates from the table above when you pick that tier. The crawl never bills beyond your max_pages.
What counts as a successful request?
They are refunded like any task. Blocks, unsolved CAPTCHAs, timeouts and empty bodies cost nothing; you pay for pages that came back with content.
What counts as a successful request?
No. Credits stay on your account until you use them, on any scraping product: Crawl today, Unlocker or LLM Scraper tomorrow, from the same balance.
What counts as a successful request?
Yes, per account. GET /users/@me shows current_concurrency and concurrency_limit; raise it from the dashboard or by talking to us.
Get started
Ready to build?
Start with the free tier and scale as your project grows. No credit card, no sales call.
Talk to an engineer, not a chatbot.