Map API
One call returns every URL of a site, from its sitemaps and the links on the page, deduplicated and filtered by keyword. One flat credit per call, however many links come back: the fastest way to decide what to crawl.
- Sitemap-first: robots.txt sitemaps and indexes, then on-page links
- search, limit, sitemap_only shape the list
- Deduplicated, same-site URLs, each with its source
- truncated flag and sitemaps array in every response
- Synchronous: one call, one response, nothing scraped
- One flat credit per call, however many links come back
Sign up with GoogleMCP readyConnect your agent in one command:
npx -y @datafuel/mcp initDatafuel / MapMapping ikea.comMap ready
POST/api/v1/mapone call · one credit
"url": "https://www.ikea.com/us/en/""search": "sofa"
1 Sitemaps2 Page links3 FilterResult
STEP 1 · SITEMAPSReads robots.txt, then every sitemap it lists
3 sitemaps
984 URLs
244 URLs
12 URLs
1,240URLs collected from sitemaps
STEP 2 · PAGE LINKSAdds the links found on the page itself
ikea.com/us/en/
/us/en/cat/sofas-fu003//us/en/rooms/living-room/sofas//us/en/ideas/small-space-sofas//us/en/cat/sofa-beds-45702//us/en/new/… 91 more links on the page
+96links not in any sitemap1,336 candidates so far
STEP 3 · DEDUPE & FILTERKeeps one copy of each URL, then only those matching “sofa”
1,336
1,187
482
482URLs kept · 854 dropped as duplicates or off-topic
482 URLs, ready to crawlsource
sitemap
sitemap
sitemap
page
sitemap
page
sitemap
sitemap
Nothing is scraped · URLs only1 credit
Everything in one call
What the endpoint does, at a glance.Sitemap-first discoverySitemaps listed in robots.txt and sitemap indexes are followed first.
On-page links fill the gapsLinks on the page itself are added for URLs the sitemaps miss.
Deduplicated, same-siteOne list, no duplicates, only URLs of the site you asked for.
searchKeep URLs whose address or title contains a term.
limitCap the list; truncated tells you when the cap was hit.
sitemap_onlySkip the page fetch and read sitemaps only, for the fastest answer.
Structured responselinks with their source, total, truncated, sitemaps and credits.
SynchronousOne call, one response. No job to poll.
Feeds jobs and crawlsHand the list to POST /job, or scope a crawl to what you found.
Child sitemapsMap one child sitemap at a time from the sitemaps array.
One flat credit per callHowever many links come back.
Failed calls are freeYou pay only when the list comes back.
Datafuel / MapMapping
412 URLs mapped0 pages fetched · 1 credit
01 · Site structure
See the whole structure of a site
One call draws the tree: the home page, its sections, every branch beneath them. You see the map before you spend a single crawl or unlock credit, so the next request is a decision, not a guess.
Demo data. Depth shown is illustrative; real trees follow the site's sitemaps and links.
Datafuel / MapFiltering
URLs found5,0003,2401,120180
search: "sofa" · limit: 500180 URLs kept1 credit · nothing fetched yet
02 · Filter first
Filter before you download
search, limit and sitemap_only shrink 5,000 URLs to the 180 you actually need, before any page is fetched. You pay only for the pages that matter: the map is one flat credit, the crawl or unlock that follows covers just your shortlist.
Demo data. Counts are illustrative.
Pricing
Pay only for successful requests
Pick a monthly plan, buy credits once, or run unlimited threads. Failed requests are never billed.
Every plan includes every endpoint and every interface (API, MCP). One shared credit balance: you only pay for volume, and only for successful requests.
How many requests a month?995,860
≈ 995,860 credits · Growth1K300K1M3.5M12MCustom
Free
Test every endpoint with real credits. No card, no expiry pressure.
$0/ mo, billed monthly
1,000 credits / month1,000 Map calls2 concurrent threads
- 1,000 trial credits
- Every endpoint, API + MCP
- Community support
GrowthBest fit
For production crawlers and agents that need headroom.
$79/ mo, billed monthly
1,000,000 credits / month1,000,000 Map calls25 concurrent threads$0.079 per 1K Map calls
- Everything in Starter
- Auto top-up
- Priority chat support
Business
For teams shipping data products on a schedule.
$199/ mo, billed monthly
3,500,000 credits / month3,500,000 Map calls50 concurrent threads$0.057 per 1K Map calls
- Everything in Growth
- Usage alerts per API key
- Account manager
Scale
High volume at the lowest per-page rate. Checks out instantly.
$499/ mo, billed monthly
12,000,000 credits / month12,000,000 Map calls150 concurrent threads$0.042 per 1K Map calls
- Everything in Business
- Dedicated IP pool
- 99.9% SLA
Enterprise
Committed volume, invoicing and governance.
Custom
Custom credits / month250+ concurrent threads
- Custom volume pricing
- Custom concurrency (250+)
- SSO · audit log · DPA
Compare
| Free | Starter | Growth | Business | Scale | Enterprise | |
|---|---|---|---|---|---|---|
| Credits / month | 1K trial | 300K | 1M | 3.5M | 12M | Custom |
| Concurrent threads | 2 | 10 | 25 | 50 | 150 | 250+ |
| API keys | 1 | 3 | 10 | 25 | Unlimited | Unlimited |
| JS rendering | ||||||
| Residential proxies | ||||||
| AI in-flight (your key) | ||||||
| Geo-targeting · 195+ countries | ||||||
| Batch multi-URL | ||||||
| Unlocker · Crawl · Map · LLM Scraper | ||||||
| Auto top-up | — | — | ||||
| Support | Community | Priority chat | Account manager | Slack + phone | Dedicated + SLA |
What does a request actually cost?1 credit = 1 basic page · costs don't stack: JS + Residential is 20 credits, not 5 + 10
1×Basic HTTPSimple HTML, datacenter proxy
5×JS RenderingHeadless browser for dynamic sites
10×Residential proxyReal residential IPs
20×JS + ResidentialFull power for protected sites
FreeAI In-FlightLLM extraction on any Unlocker request, zero extra credits
FAQ
Frequently asked questions
Discovery, output and billing, answered straight.
Map APIDiscovery, filters and output.
What counts as a successful request?
Sitemap-first: the sitemaps listed in robots.txt and any sitemap indexes are followed, then the links on the page itself fill the gaps. Everything is deduplicated into one same-site list.
What counts as a successful request?
search keeps URLs whose address or title contains a term, limit caps the list, and sitemap_only skips the page fetch and reads sitemaps only. truncated tells you when a cap was hit.
What counts as a successful request?
The list of links with their source, the total, the truncated flag, the sitemaps array (so you can map one child sitemap at a time) and the credits used. Nothing is scraped.
What counts as a successful request?
When you need the content of the pages, not their addresses. Crawl follows links under your start URL and returns every page as Markdown; Map is the cheap first step to decide what that crawl should cover.
BillingCredits, failures and limits.
What counts as a successful request?
One flat credit per call, however many links come back. It is the cheapest way to size a site before you spend crawl or unlock credits on it.
What counts as a successful request?
A failed call costs nothing, like any task. You only pay when the URL list comes back.
What counts as a successful request?
No. Credits stay on your account until you use them, on any scraping product, from the same balance.
What counts as a successful request?
Yes, per account. GET /users/@me shows current_concurrency and concurrency_limit; raise it from the dashboard or by talking to us.
Get started
Ready to build?
Start with the free tier and scale as your project grows. No credit card, no sales call.
Talk to an engineer, not a chatbot.