Crawl and map a website
Capture a visual site tree with HTML, Markdown, and JSON for every page
Crawl and map a website
Website crawls turn a starting URL into a hierarchy of rendered pages. Each page can include a screenshot plus HTML, Markdown, and a normalized JSON document, all captured from the same browser state.
The crawler is available on Starter and every higher plan. Each captured page uses one screenshot credit.

The Website crawler page shows a real crawl of these docs, 71 pages in 11 minutes, with an interactive replay where you can change depth, limit and sitemap and test include and exclude patterns.
Crawl a documentation site
Start with a narrow scope
Use a depth of two and an include pattern to keep discovery inside the documentation section:
curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
-H 'X-API-Key: YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/docs/",
"depth": 2,
"limit": 50,
"includePatterns": ["https://example.com/docs/*"],
"excludePatterns": ["*/docs/archive/*"],
"outputs": ["screenshot", "markdown", "json"],
"fullPage": true
}'The root page is depth 0; its links are depth 1, and links discovered from those pages are depth 2.
Poll until the crawl is terminal
const terminal = new Set([
'COMPLETED', 'PARTIALLY_COMPLETED', 'FAILED', 'CANCELLED',
]);
async function waitForCrawl(id) {
while (true) {
const response = await fetch(
`https://api.allscreenshots.com/v1/crawls/${id}`,
{ headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
);
const crawl = await response.json();
console.log(`${crawl.progress.completed}/${crawl.progress.discovered} pages`);
if (terminal.has(crawl.status)) return crawl;
await new Promise((resolve) => setTimeout(resolve, 1500));
}
}Rebuild the hierarchy
Fetch the page records and connect each page to its parentId:
const response = await fetch(
`https://api.allscreenshots.com/v1/crawls/${crawlId}/pages?pageSize=100`,
{ headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
);
const { pages } = await response.json();
const children = Map.groupBy(
pages.filter((page) => page.parentId),
(page) => page.parentId,
);
const root = pages.find((page) => page.parentId === null);A page is stored only once. When several pages link to the same URL, its first discoverer becomes its parent. A crawl can hold up to 200 pages (captured plus skipped and failed), so request page=1 as well when totalPages is 2.
Download the page bundle
async function downloadOutput(crawlId, page, type) {
const output = page.outputs.find((item) => item.type === type);
if (!output) return null;
const response = await fetch(
output.resultUrl,
{ headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
);
return type === 'screenshot' ? response.arrayBuffer() : response.text();
}Use Markdown for search or LLM ingestion, JSON for page metadata and link analysis, HTML for archival, and screenshots for visual review.
Or download everything at once
When the crawl has finished, one request returns every page:
# All pages as one Markdown file, ready to give to a model
curl 'https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=markdown' \
-H 'X-API-Key: YOUR_API_KEY' --output docs.md
# Every stored file plus manifest.json
curl 'https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=zip' \
-H 'X-API-Key: YOUR_API_KEY' --output docs.zipformat=jsonl returns one JSON document per page.
Dashboard site atlas
Open Crawler in the dashboard to configure the same options without writing code. The live site atlas shows first-discovery relationships, page status, and screenshot thumbnails. Select a node to preview and download any captured output, or see why a page was skipped or failed. Download all exports a finished crawl as a ZIP, one Markdown file, or JSON Lines.
Skip polling with a webhook
Add webhookUrl (and optionally webhookSecret) to the request and you receive one CRAWL_COMPLETED or CRAWL_FAILED call when the crawl ends. See crawl webhooks.
Recommended settings
- Begin with depth
1or2and a limit of25pages. - Start at the section you want, with a trailing slash, such as
https://example.com/docs/: the crawl stays under that folder. SetentireSiteto follow links anywhere on the site. - Add include patterns before enabling subdomains.
- Keep the screenshot output on when you want to inspect the tree visually.
- Use Markdown rather than HTML for semantic search and model context.
- Treat
PARTIALLY_COMPLETEDas usable: successful pages and their outputs remain available. - Keep
sitemapatincludefor sites whose pages are not all linked from the home page; useonlyto capture exactly what the sitemap lists. - To allow or block the crawler in
robots.txt, use the user agentAllScreenshotsBot.
See the crawl API reference for every request field, status, and response shape.
For complete workflows, see Turn a website into an LLM knowledge base and Audit a site visually before a redesign.
Extract everything from a URL in one call
Capture a screenshot, the page markdown, and structured JSON from a single request — then hand the bundle to an LLM
Turn a website into an LLM knowledge base
Crawl a site, export every page as Markdown or JSON Lines, and split it into chunks for search or retrieval