Skip to content

Crawl API

One call walks a whole site and reads every page beneath the start URL, handing each one back as LLM-ready Markdown instead of raw HTML. The crawl follows links breadth-first, stays on your site and under your path, and fetches every page with the Unlocker engine, so protected sites are not a special case.

  • One call, hundreds of pages: breadth-first crawl under your start URL, same site only
  • max_pages, max_depth and RE2 path filters draw the perimeter before it starts
  • Markdown or JSON for every page, clean enough to embed as is
  • Asynchronous: 202 with a job_id, then poll the status and page through the results
  • Deduplicated: one entry per page, whatever the number of links to it
  • 1 credit per page on Basic, 5 with js_rendering; failed pages refunded
From 1 credit per page. Failed pages are refunded. No card to start.
MCP readyConnect your agent in one command:npx -y @datafuel/mcp initClaudeOpenAI / ChatGPTGeminiCursorMistral

Everything in one job

Six things the endpoint does, at a glance.
Multi-page traversalOne start URL; the crawl follows links breadth-first and reads every page beneath it.
Depth and pattern controlmax_depth, max_pages and RE2 include/exclude paths set the perimeter before it starts.
Markdown or JSON outputresult_format returns clean Markdown for embedding, or structured JSON per page.
Asynchronous job202 and a job_id at once; poll GET /crawl/{job_id} and page through results as they land.
DeduplicatedOne entry per page, however many links point to it. Same-site URLs only.
Safety limitsNever more pages than max_pages, never off-site, failed pages refunded.
01 · Corpus

From one URL to a complete corpus

Give the crawl a start URL and it walks every page beneath it, breadth-first, and hands each one back as clean Markdown. One call, one dataset: no link lists to maintain, no per-page requests to orchestrate.

Demo data. Page counts are illustrative; real crawls follow the site's links up to your limits.

02 · Controls

Controls that avoid surprises

max_depth and max_pages draw the perimeter before the crawl starts, and include/exclude patterns narrow it further. The job never queues more pages than you allowed, so the cost is bounded and predictable, not an open-ended bill.

Demo data. Estimates use Basic pricing, 1 credit per page; js_rendering pages cost 5.

03 · RAG ready

Ready for RAG and AI pipelines

Each page comes back as structured Markdown: headings, lists and tables preserved, navigation, scripts and boilerplate removed. No post-processing step: chunk it, embed it, retrieve it.

Demo data. Chunking and embeddings happen in your pipeline; the API returns the Markdown.

04 · Async

Asynchronous at scale

A crawl is a job, not a request: you get a 202 and a job_id immediately and the crawl runs on our side. Your code blocks on nothing: poll GET /crawl/{job_id} for progress and page through the results while pages are still arriving.

Demo data. job_id, timings and page counts are illustrative.

Pricing

Pay only for successful requests

Pick a monthly plan, buy credits once, or run unlimited threads. Failed requests are never billed.

Every plan includes every endpoint and every interface (API, MCP). One shared credit balance: you only pay for volume, and only for successful requests.

How many requests a month?1,000,000 pages crawled / month
≈ 1,000,000 credits · Growth
1K300K1M3.5M12MCustom
Free

Test every endpoint with real credits. No card required.

$0one-time trial
1,000 credits, one-time1,000 pages crawled2 concurrent threads
  • Trial credits valid for 1 year
  • Every endpoint, API + MCP
  • Community support
GrowthBest fit

For production crawlers and agents that need headroom.

$79/ mo, billed monthly
1,000,000 credits / month1,000,000 pages crawled25 concurrent threads$0.079 per 1K pages$0.395 per 1K with JS
  • Everything in Starter
  • Auto top-up
  • Priority chat support
Business

For teams shipping data products on a schedule.

$199/ mo, billed monthly
3,500,000 credits / month3,500,000 pages crawled50 concurrent threads$0.057 per 1K pages$0.284 per 1K with JS
  • Everything in Growth
  • Usage alerts per API key
  • Account manager
Scale

High volume at the lowest per-page rate. Checks out instantly.

$499/ mo, billed monthly
12,000,000 credits / month12,000,000 pages crawled150 concurrent threads$0.042 per 1K pages$0.208 per 1K with JS
  • Everything in Business
  • Dedicated IP pool
  • 99.9% SLA
Enterprise

Committed volume, invoicing and governance.

Custom
Custom credits / month250+ concurrent threads
  • Custom volume pricing
  • Custom concurrency (250+)
  • SSO · audit log · DPA
Compare
FreeStarterGrowthBusinessScaleEnterprise
Credits / month1K one-time300K1M3.5M12MCustom
Concurrent threads2102550150250+
API keys131025UnlimitedUnlimited
JS rendering
Residential proxies
AI in-flight (your key)
Geo-targeting · 195+ countries
Batch multi-URL
Unlocker · Crawl · Map · Search · LLM Scraper
Auto top-up——
SupportCommunityEmailPriority chatAccount managerSlack + phoneDedicated + SLA
What does a request actually cost?1 credit = 1 basic page · costs don't stack: JS + Residential is 20 credits, not 5 + 10
1×Basic HTTPSimple HTML, datacenter proxy
5×JS RenderingHeadless browser for dynamic sites
10×Residential proxyReal residential IPs
20×JS + ResidentialFull power for demanding sites
FreeAI In-FlightLLM extraction on any Unlocker request with your own AI key, zero extra credits
FAQ

Frequently asked questions

Scope, results and billing, answered straight.

Crawl APIScope, depth and results.
How does a crawl decide which pages to visit?
It starts at your URL and follows links breadth-first, staying on the same site and under the start path. max_pages is a hard stop on pages queued, max_depth limits how many links away from the start it goes, and include_paths / exclude_paths (RE2) narrow it further; exclude always wins.
Is a crawl one call or a job?
A job. POST /api/v1/crawl answers 202 with a job_id at once. Poll GET /crawl/{job_id} for status and page counts, then page through GET /crawl/{job_id}/results with next_cursor when it completes.
What do I get back for each page?
The page as Markdown by default (set result_format for other formats), with its URL and depth. Every page is fetched with the full Unlocker options, so protected or JavaScript-heavy sites are not a special case.
When should I use Map instead?
When you only need the URLs. Map returns every same-site URL from sitemaps and on-page links in one call, for 1 credit on Basic (10 on Premium), which is the cheapest way to size a site or pick exactly which pages to fetch.
BillingCredits, failures and limits.
How is a crawl billed?
Per page queued: 1 credit on Basic proxies, 5 with js_rendering, and the Premium rates from the table above when you pick that tier. The crawl never bills beyond your max_pages.
What happens to pages that fail?
They are refunded like any task. Blocks, unsolved CAPTCHAs, timeouts and empty bodies cost nothing; you pay for pages that came back with content.
Do credits expire?
Pay-as-you-go credits never expire. Unused monthly plan credits roll over for one month, and trial credits are valid for 1 year. The same balance works on every scraping product.
Is there a concurrency limit?
Yes, per account. GET /users/@me shows current_concurrency and concurrency_limit; raise it from the dashboard or by talking to us.
Get started

Ready to build?

Start with the free tier and scale as your project grows. No credit card, no sales call.

Talk to an engineer, not a chatbot.