Website crawler
Capture up to 100 pages of a site as screenshots, HTML, Markdown and JSON with one request, then export them in one file
Website crawler
Send one URL to POST /v1/crawls and the crawler follows the site's links and sitemap, rendering each page in a real browser. Every page comes back as a screenshot, HTML, Markdown and a JSON page document, or any mix of the four. When the crawl ends you can download the whole site as a ZIP, one Markdown file or JSON Lines.
The crawler is available on Starter and every higher plan. Each captured page uses 1 screenshot credit, however many outputs you choose. Pages that fail, are skipped, or show a bot check or an error page aren't charged.
curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
-H 'X-API-Key: YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://docs.allscreenshots.com/",
"depth": 3,
"limit": 100,
"entireSite": true,
"outputs": ["screenshot", "markdown", "json"]
}'A real crawl
We ran the request above against this documentation site. The crawler read the 71 URLs in the site's sitemap and the 19 internal links on the home page, found 73 pages in all, and captured 71 of them. The other two were links to pages that don't exist: they returned 404, were reported as HTTP_ERROR and weren't charged. Because of those two, the crawl ended as PARTIALLY_COMPLETED. It used 71 credits.
| Discovered | 73 pages: 1 start page, 70 at depth 1, 2 at depth 2 |
| Captured and charged | 71 |
| Failed | 2 (HTTP_ERROR, 404) |
| Render time per page | 9.3 seconds median, full-page screenshots |
| Whole crawl | 11 minutes, one page at a time with 500 ms between requests |

The two 404s were broken links in our own docs, on the social images page and a use-case page. We fixed them; the crawl is a quick way to find yours.
Try the settings
The explorer replays that crawl. Change depth, limit or sitemap and watch which pages the crawler would capture and in what order. The pages, their links, screenshots and outcomes are real; the order and limits follow the crawler's rules.
- discovered
- 1
- completed
- 0
- failed
- 0
- skipped
- 0
- charged
- 0
- queued
- 1
Depth 0 (start URL) · 1 page
Depth 1 · 0 pages
Depth 2 · 0 pages
Simulation over a real crawl of docs.allscreenshots.com (71 pages captured, depth 3, limit 100, finished in 11 min). Pages, links, screenshots and outcomes are from that crawl; the order and limits follow the crawler's rules. Pages the real crawl didn't reach show their path instead of a screenshot.
Some things to try:
limit: 25withdepth: 2captures the first 25 pages and queues 25 more, twice the limit, in case some fail. The 25 left in the queue end asSKIPPEDwithLIMIT_REACHEDand aren't charged.sitemap: "skip"follows links only. On this site the home page has 19 internal links, so the crawl spreads out over three levels instead of one: 18 pages at depth 1, 40 at depth 2 and 2 at depth 3.sitemap: "only"captures exactly the 71 URLs the sitemap lists and follows no links, so the two broken links are never visited.depth: 0captures only the start URL.
How discovery works
Scope
A crawl stays on the site it starts on and, unless you set entireSite, inside the folder of the start URL.
Default
"url": "https://example.com/docs/intro"
- example.com/docs/introfollowed
- example.com/docs/api/authfollowed
- www.example.com/docs/faqfollowed
- example.com/blog/launchnot followed
- help.example.com/docs/startnot followed
entireSite
"entireSite": true
- example.com/docs/introfollowed
- example.com/docs/api/authfollowed
- www.example.com/docs/faqfollowed
- example.com/blog/launchfollowed
- help.example.com/docs/startnot followed
entireSite + includeSubdomains
"entireSite": true, "includeSubdomains": true
- example.com/docs/introfollowed
- example.com/docs/api/authfollowed
- www.example.com/docs/faqfollowed
- example.com/blog/launchfollowed
- help.example.com/docs/startfollowed
https://example.com/docs/intro. By default the crawl stays under /docs/, the folder of the start URL; www. and the bare domain count as the same site. entireSite lifts the folder limit and includeSubdomains adds other subdomains.- The folder is the start URL itself when it ends with
/(/docs/), and otherwise its parent folder:/wiki/for/wiki/Web_crawler, the whole site for/pricing. www.example.comandexample.comare the same site.includeSubdomainsalso follows links to subdomains, such asdocs.example.comfromwww.example.com.- If the start URL redirects, pages under where it landed are in scope too.
- Links to files that aren't web pages (PDFs, images, video, archives, office documents) are never queued.
Which links come first
A page's links are queued in this order:
- Links in the page's content, interleaved with sitemap URLs on the start page
- Links in the header, menus, sidebars and footer
- Links that aren't visible
Pages are captured lowest depth first, in the order they were found. With a small limit, that means you get the pages an article or a product links to rather than every item in the site menu.
Sitemaps
sitemap | What the crawler does |
|---|---|
include (default) | Adds the sitemap's URLs to the links found on the start page |
skip | Ignores sitemaps and follows links only |
only | Captures the start page and the sitemap's URLs, and follows no links |
The crawler reads the sitemaps named in robots.txt and /sitemap.xml, follows a sitemap index one level, and reads gzip files. It uses at most 5 sitemap files and 5,000 URLs per crawl. Sitemap URLs are found at depth 1, so they need depth of 1 or more.
Include and exclude patterns
includePatterns and excludePatterns are globs that must match the whole URL:
*matches any run of characters,/included.?matches one character.- An exclude wins over an include.
- With no includes, every URL that no exclude matches is followed.
Try them against the URLs of the real crawl:
16 of 73 URLs from the real crawl would be followed.
- /not followed:matches no include
- /api-reference/async-jobsnot followed:matches no include
- /api-reference/batchesnot followed:matches no include
- /api-reference/bulknot followed:matches no include
- /api-reference/composenot followed:matches no include
- /api-reference/crawlsnot followed:matches no include
- /api-reference/outputsnot followed:matches no include
- /api-reference/schedulesnot followed:matches no include
- /api-reference/screenshotsnot followed:matches no include
- /api-reference/webhooksnot followed:matches no include
- /dashboard/domainsnot followed:matches no include
- /features/ai-cleanupnot followed:matches no include
- /features/ai-extractionnot followed:matches no include
- /features/blockingnot followed:matches no include
- /features/full-page-capturenot followed:matches no include
- /features/interactive-actionsnot followed:matches no include
- /features/sessionsnot followed:matches no include
- /features/social-imagesnot followed:matches no include
- /features/stealth-modenot followed:matches no include
- /features/stylingnot followed:matches no include
- /features/viewport-and-devicesnot followed:matches no include
- /getting-started/authenticationnot followed:matches no include
- /getting-started/quickstartnot followed:matches no include
- /getting-started/rate-limitsnot followed:matches no include
- /guides/canvas-exportsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/canvas-image-processingfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/canvas-webhooksfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/canvas-workflowsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/ci-cd-integrationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/cicd-integrationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/crawl-a-websitefollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/full-page-screenshotsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/multi-output-extractionfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/openclaw-integrationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/pdf-generationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/remove-popups-and-overlaysfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/scheduled-screenshotsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/social-media-previewsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/visual-regression-testingfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/website-monitoringfollowed:matches https://docs.allscreenshots.com/guides/*
- /integrations/n8nnot followed:matches no include
- /llm-txtnot followed:matches no include
- /sdksnot followed:matches no include
- /sdks/csharpnot followed:matches no include
- /sdks/gonot followed:matches no include
- /sdks/javanot followed:matches no include
- /sdks/javascriptnot followed:matches no include
- /sdks/kotlinnot followed:matches no include
- /sdks/phpnot followed:matches no include
- /sdks/pythonnot followed:matches no include
- /sdks/rubynot followed:matches no include
- /sdks/rustnot followed:matches no include
- /sdks/swiftnot followed:matches no include
- /sdks/typescriptnot followed:matches no include
- /supportnot followed:matches no include
- /tools/agent-pluginsnot followed:matches no include
- /tools/canvanot followed:matches no include
- /tools/clinot followed:matches no include
- /tools/figmanot followed:matches no include
- /tools/mcpnot followed:matches no include
- /tools/mkdocsnot followed:matches no include
- /use-casesnot followed:matches no include
- /use-cases/ci-cd-workflowsnot followed:matches no include
- /use-cases/compliance-auditingnot followed:matches no include
- /use-cases/content-archivingnot followed:matches no include
- /use-cases/customer-onboardingnot followed:matches no include
- /use-cases/ecommerce-monitoringnot followed:matches no include
- /use-cases/legal-evidencenot followed:matches no include
- /use-cases/report-generationnot followed:matches no include
- /use-cases/social-media-og-imagesnot followed:matches no include
- /use-cases/structured-extractionnot followed:matches no include
- /use-cases/visual-regression-testingnot followed:matches no include
- /use-cases/website-monitoringnot followed:matches no include
Patterns match the whole URL, including https:// and the host, so a pattern that starts with a path matches nothing. The start URL is always captured; patterns decide which discovered links are followed.
Patterns decide which discovered links are followed; the start URL is always captured. You can send up to 25 of each, up to 2,048 characters long.
Progress and statuses
Poll GET /v1/crawls/{id}, or set webhookUrl to get one call when the crawl ends. The progress object counts pages:
{
"id": "zBFfJpP5Ev",
"status": "PARTIALLY_COMPLETED",
"progress": { "discovered": 73, "completed": 71, "failed": 2, "skipped": 0, "charged": 71, "queued": 0 }
}QUEUED
RUNNING
COMPLETEDEvery page it tried was captured or deliberately skippedPARTIALLY_COMPLETEDSome pages captured, and some failed, or credits or the API key limit ran outFAILEDNo page was capturedCANCELLEDYou cancelled it while queued or running
COMPLETED: every page the crawl tried was captured or skipped on purpose.PARTIALLY_COMPLETED: at least one page was captured and at least one failed, or the crawl stopped because credits or the API key's limit ran out.FAILED: no page was captured.CANCELLED: you cancelled it. Pages already captured stay available.
Every skipped or failed page has an errorCode, such as ROBOTS_DISALLOWED, LIMIT_REACHED, DUPLICATE_PAGE or HTTP_ERROR. The crawl reference lists them all.
What each page stores
| Output | What you get |
|---|---|
screenshot | The page as an image, full page by default, in format (WebP by default) |
html | The page's HTML as the browser loaded it |
markdown | The whole page body as Markdown |
json | A page document: URL, requested URL, title, description, language, canonical URL, HTTP status, depth, parent page, headings, links (each marked internal or not), visible text and OpenGraph tags |
These differ from the outputs of a screenshot request, which return cleaned HTML, the main content as Markdown, and JSON by selector, page data or AI. Use /v1/screenshots with outputs for a few pages you need in that form.
Outputs are kept for 7 days. Crawl and page records stay until you delete the crawl.
Exports
When a crawl has ended, GET /v1/crawls/{id}/export returns every page in one download. For the crawl above:
format | What came back |
|---|---|
zip | 18.2 MB: a folder per page with its files, and manifest.json (214 files) |
markdown | 535 KB: one file with all 71 pages, each headed by its source URL and title |
jsonl | 1.7 MB: 73 lines, one per page, failed pages included, with the Markdown, the JSON document and a screenshotUrl |
The ZIP looks like this:
The Markdown export starts like this:
# Crawl of https://docs.allscreenshots.com/
---
Source: https://docs.allscreenshots.com/
Title: Introduction | Allscreenshots Docs
# Introduction
Capture high-quality screenshots of any website with a simple APIAdd outputs=markdown,json to leave screenshots out of a zip or jsonl export.
Politeness and robots.txt
- The crawler identifies itself as
AllScreenshotsBot. Rules forAllScreenshotsBotapply whenrobots.txthas them, otherwise the rules for*. - A disallowed URL is skipped with
ROBOTS_DISALLOWED. - A missing
robots.txtallows everything. One that returns a server error or can't be reached after three tries stops the crawl of that site (ROBOTS_UNAVAILABLE). - Requests to one host are at least
crawlDelayMsapart: 500 ms by default, up to 30 seconds. A longerCrawl-delayinrobots.txtis respected up to 30 seconds.
Billing
| What | Counts as |
|---|---|
| A captured page | 1 screenshot, however many outputs |
| A failed or skipped page, a bot check or an HTTP error page | Nothing |
| A page that redirects to one the crawl already captured | Nothing (DUPLICATE_PAGE) |
| A crawl that runs out of credits | It stops; captured pages stay, the rest are skipped with QUOTA_EXCEEDED |
Limits
| Limit | Value |
|---|---|
Pages captured per crawl (limit) | 100 |
| Depth | 5 |
| Crawls running at once per organization | 2, the rest wait in the queue |
| Unfinished crawls per organization | 10 |
| Patterns | 25 includes and 25 excludes, 2,048 characters each |
| Sitemaps | 5 files, 5,000 URLs |
| Output retention | 7 days |
Crawls don't support PDF output, selector or AI JSON, actions, device presets, sessions, stealth mode or AI cleanup. Send the URLs you need that way to /v1/screenshots or bulk.
Next steps
- Crawl and map a website: poll, rebuild the page tree and download outputs.
- Turn a website into an LLM knowledge base: crawl, export Markdown and index it.
- Audit a site visually before a redesign: screenshots of every page in one ZIP.
- Crawl API reference: every field, status, error code and the webhook payload.