Website crawls
Render and extract a hierarchy of pages from one website
Website crawls
The crawl API starts at one URL, follows the links on each rendered page and the site's sitemap, and captures every page as a screenshot, HTML, Markdown, or a JSON page document. Crawls run asynchronously and are available on Starter and every higher plan.
Each captured page uses one screenshot credit, however many outputs you select. Failed and skipped pages are not charged, and neither are pages that show a bot check or an HTTP error instead of content.
Start a crawl
POST /v1/crawlscurl -X POST 'https://api.allscreenshots.com/v1/crawls' \
-H 'X-API-Key: YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: docs-crawl-2026-11-16' \
-d '{
"url": "https://example.com",
"depth": 2,
"limit": 25,
"outputs": ["screenshot", "html", "markdown", "json"],
"fullPage": true,
"viewport": { "width": 1440, "height": 900 },
"format": "webp",
"webhookUrl": "https://your-app.com/webhooks/allscreenshots"
}'The response is 202 Accepted:
{
"id": "a1b2c3d4e5",
"status": "QUEUED",
"statusUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5",
"createdAt": "2026-09-27T04:00:00Z"
}Send an Idempotency-Key header (1–100 letters, digits, _ or -) to make retries safe. Repeating the request with the same key and body returns the original crawl with an Idempotency-Replayed: true header instead of starting a second one.
Request fields
| Field | Type | Default | Description |
|---|---|---|---|
url | string | required | HTTP or HTTPS starting URL. |
depth | integer | 2 | Link distance from the root, from 0 to 5. The root is depth 0. |
limit | integer | 25 | Maximum number of pages captured and charged, from 1 to 100. |
outputs | string[] | all | Any non-empty combination of screenshot, html, markdown, and json. |
sitemap | string | include | include adds the URLs in the site's sitemap to the links found on pages. only crawls the sitemap URLs without following links. skip ignores the sitemap. |
includePatterns | string[] | [] | Full-URL glob patterns that a discovered URL must match. |
excludePatterns | string[] | [] | Full-URL glob patterns to skip. Exclusions win over inclusions. |
entireSite | boolean | false | Follow links anywhere on the site. By default the crawl stays under the starting URL's folder: /docs/ for https://example.com/docs/ or https://example.com/docs/intro, /wiki/ for https://example.com/wiki/Some_article. A start at the home page covers the whole site either way. |
includeSubdomains | boolean | false | Also follow links to subdomains of the site, for example docs.example.com from www.example.com. |
crawlDelayMs | integer | 500 | Minimum wait between two requests to the same hostname, from 500 ms to 30 seconds. A longer Crawl-delay in robots.txt is respected up to 30 seconds. |
viewport | object | 1440 × 900 | Browser viewport dimensions and optional device scale factor. |
fullPage | boolean | true | Capture the entire scrollable page. |
format | string | webp | Screenshot format: png, jpeg, or webp. |
quality | integer | 80 | JPEG/WebP quality from 1 to 100. |
renderDelay | integer | 0 | Wait after rendering before extracting outputs, up to 30 seconds. |
waitUntil | string | domcontentloaded | load, domcontentloaded, or networkidle. |
timeout | integer | 30000 | Per-page timeout in milliseconds, from 1 to 60 seconds. |
darkMode | boolean | false | Request the browser's dark color scheme. |
blockAds | boolean | true | Block common ads and trackers. |
blockCookieBanners | boolean | true | Hide common cookie consent overlays. |
blockPopups | boolean | true | Remove page-blocking marketing, newsletter, location, survey, app-promotion, notification, and age-gate overlays from every rendered page. |
webhookUrl | string | none | Called once when the crawl finishes, fails, or is cancelled. See Webhooks. |
webhookSecret | string | none | Signs the webhook body. It is stored with the crawl and never returned. |
How the page limit works
limit counts captured pages. Pages that are skipped (blocked by robots.txt, redirected off the site) or fail do not use it up: the crawler keeps up to twice the limit in its queue and moves on to the next URL. When the limit is reached, the remaining queued pages are marked SKIPPED with LIMIT_REACHED.
Links to files that are not web pages, such as PDFs, images, videos, archives and office documents, are not queued.
Links in a page's content are queued before links in its header, menus, sidebars and footer, and links that are not visible come last. A crawl with a small limit therefore follows the page's own links (articles, products, documentation sections) rather than the site menu.
Poll crawl status
curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5' \
-H 'X-API-Key: YOUR_API_KEY'{
"id": "a1b2c3d4e5",
"url": "https://example.com/",
"status": "RUNNING",
"depth": 2,
"limit": 25,
"outputs": ["screenshot", "markdown", "json"],
"sitemap": "include",
"webhookUrl": "https://your-app.com/webhooks/allscreenshots",
"progress": {
"discovered": 9,
"completed": 4,
"failed": 0,
"skipped": 1,
"charged": 4,
"queued": 4
},
"createdAt": "2026-09-27T04:00:00Z",
"startedAt": "2026-09-27T04:00:01Z"
}Response fields
| Field | Description |
|---|---|
status | QUEUED, RUNNING, COMPLETED, PARTIALLY_COMPLETED, FAILED, or CANCELLED. |
progress.discovered | URLs found and queued so far, including the root. |
progress.completed | Pages captured. |
progress.failed | Pages that could not be captured. |
progress.skipped | Pages not captured on purpose: robots.txt, an off-site redirect, the page limit, or an early stop. |
progress.charged | Credits used by this crawl. |
progress.queued | Pages still waiting. 0 once the crawl has ended. |
errorCode, errorMessage | Why the crawl stopped early or captured nothing. |
A crawl is:
COMPLETEDwhen every page it tried was captured or deliberately skipped.PARTIALLY_COMPLETEDwhen at least one page was captured and at least one failed, or when it stopped early because credits or the API key's limit ran out.FAILEDwhen no page was captured.
List crawls
curl 'https://api.allscreenshots.com/v1/crawls?page=0&pageSize=20' \
-H 'X-API-Key: YOUR_API_KEY'Returns { "crawls": [...], "page", "pageSize", "total", "totalPages" }, newest first. An API key sees the crawls it started; the dashboard shows every crawl in the organization.
List crawled pages
curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages?page=0&pageSize=100' \
-H 'X-API-Key: YOUR_API_KEY'Each page includes an id, parentId, depth, status, HTTP metadata, and its available outputs. parentId is the first page that discovered the URL and can be used to render a hierarchy. Pages from the sitemap have the root page as their parent. A crawl can hold up to 200 pages, so read totalPages and request the next page when there is one.
{
"pages": [{
"id": "p9q8r7s6t5",
"parentId": "r1o2o3t4p5",
"url": "https://example.com/about",
"depth": 1,
"status": "COMPLETED",
"title": "About",
"httpStatus": 200,
"outputs": [{
"type": "screenshot",
"contentType": "image/webp",
"size": 48219,
"resultUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages/p9q8r7s6t5/outputs/screenshot",
"expiresAt": "2026-10-04T04:00:05Z"
}]
}],
"page": 0,
"pageSize": 100,
"total": 9,
"totalPages": 1
}Page statuses are QUEUED, RUNNING, COMPLETED, FAILED, SKIPPED, and CANCELLED. A failed or skipped page has an errorCode and errorMessage:
| Page code | Status | Meaning |
|---|---|---|
ROBOTS_DISALLOWED | SKIPPED | The site's robots.txt does not allow this URL. |
ROBOTS_UNAVAILABLE | SKIPPED | The site's robots.txt returned a server error or could not be reached, so the site was not crawled. |
REDIRECT_OUT_OF_SCOPE | SKIPPED | The page redirected to another site. |
DUPLICATE_PAGE | SKIPPED | The page landed on a URL this crawl already captured, for example /docs redirecting to /docs/. Not charged. |
LIMIT_REACHED | SKIPPED | The crawl had already captured limit pages. |
QUOTA_EXCEEDED | SKIPPED | The crawl stopped because no screenshot credits remained. |
API_KEY_LIMIT_EXCEEDED | SKIPPED | The crawl stopped because the API key reached its screenshot limit. |
PAGE_BLOCKED | FAILED | The site showed a bot check instead of the page. Not charged. |
HTTP_ERROR | FAILED | The site answered with an error status, kept in httpStatus. Not charged. |
PAGE_CRASHED | FAILED | The page could not be rendered after three attempts. |
| other codes | FAILED | The page could not be rendered, for example RENDER_FAILED or a timeout. |
Download an output
curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages/p9q8r7s6t5/outputs/markdown' \
-H 'X-API-Key: YOUR_API_KEY' \
--output about.mdWhat each output holds
Crawl outputs share names with the outputs of a screenshot request but are made for whole sites, so three of them differ:
| Output | In a crawl | With /v1/screenshots outputs (default) |
|---|---|---|
screenshot | The page in format, full page by default | The viewport by default |
html | The page's HTML as loaded, scripts included | Cleaned: no scripts, absolute URLs |
markdown | The whole page body | The main content only |
json | A page document (below) | The page's own data (auto), selectors, or AI |
The page document has id, parentId, url (where the page landed), requestedUrl, title, description, language, canonicalUrl, statusCode, contentType, depth, headings, links (each with url, text and internal), the visible text and openGraph.
Stored HTML is returned as a download-safe document: it cannot run scripts when opened from the API.
Export a whole crawl
GET /v1/crawls/{id}/export?format=zip|markdown|jsonlDownloads every page of a finished crawl in one file. A crawl that is still running returns 409 CRAWL_NOT_FINISHED.
| Format | Contents |
|---|---|
zip (default) | One folder per page, named after its URL path, with each stored output, plus manifest.json listing every page with its URL, title, status, depth, parent and files. |
markdown | One Markdown file with every captured page in crawl order, each preceded by its source URL and title. Needs the markdown output. |
jsonl | One JSON document per line and per page: the page fields plus markdown, html, data (the JSON output) and screenshotUrl for the outputs you stored. |
Add outputs=markdown,json to include only some outputs in zip and jsonl exports.
A ZIP export holds one folder per page, numbered in crawl order, and a manifest:
crawl-a1b2c3d4e5.zip
├── manifest.json
├── 001-index/
│ ├── screenshot.webp
│ ├── page.md
│ └── page.json
├── 002-getting-started-quickstart/
│ └── ...manifest.json lists every page, including skipped and failed ones, with id, url, title, status, depth, parentId, httpStatus, errorCode, folder and files. Outputs that expired before the export are listed in expiredOutputs.
curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/export?format=markdown' \
-H 'X-API-Key: YOUR_API_KEY' \
--output site.mdWebhooks
When a crawl has a webhookUrl, one POST is sent when it ends:
CRAWL_COMPLETEDforCOMPLETED,PARTIALLY_COMPLETED, andCANCELLEDcrawls (readstatus).CRAWL_FAILEDwhen no page was captured.
{
"event": "CRAWL_COMPLETED",
"timestamp": "2026-09-27T04:03:10Z",
"data": {
"crawlId": "a1b2c3d4e5",
"url": "https://example.com/",
"status": "COMPLETED",
"progress": { "discovered": 31, "completed": 25, "failed": 0, "skipped": 6, "charged": 25 },
"errorCode": null,
"errorMessage": null,
"statusUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5",
"pagesUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages",
"completedAt": "2026-09-27T04:03:09Z"
}
}With a webhookSecret, the body is signed in X-Webhook-Signature exactly as for other webhooks. If your endpoint answers with 429, a 5xx status, or cannot be reached, delivery is retried with growing intervals for up to 24 hours. Every attempt carries the same X-Webhook-Delivery-Id, so you can ignore repeats.
Cancel or delete
# Stop the crawl. The page being rendered is discarded and not charged.
curl -X POST 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/cancel' \
-H 'X-API-Key: YOUR_API_KEY'
# Delete metadata and every stored output. Cancel a running crawl first.
curl -X DELETE 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5' \
-H 'X-API-Key: YOUR_API_KEY'Once cancel returns, no further pages are stored or charged. Deleting a crawl that is still running returns 409 CRAWL_NOT_FINISHED.
Limits
| Limit | Value |
|---|---|
| Pages captured per crawl | 100 |
| Depth | 5 |
| Crawls running at once per organization | 2 (others wait in the queue, oldest first) |
| Unfinished crawls per organization | 10 |
| Wait between requests to one hostname | 500 ms to 30 seconds |
| Sitemap files read per crawl | 5, including those listed in a sitemap index |
| Sitemap URLs used per crawl | 5,000 |
includePatterns, excludePatterns | 25 each, up to 2,048 characters per pattern |
| Output retention | 7 days |
Starting a crawl counts toward your plan's capture rate limit, not the general API rate limit.
Errors
| Status | Code | When |
|---|---|---|
400 | VALIDATION_ERROR, INVALID_REQUEST | A field is out of range or malformed, or the export format is unknown. |
400 | EXPORT_UNAVAILABLE | A Markdown export was requested for a crawl without the markdown output. |
400 | INVALID_IDEMPOTENCY_KEY | The Idempotency-Key header has invalid characters or length. |
402 | QUOTA_EXCEEDED | No screenshot credits remain. |
403 | CAPABILITY_NOT_AVAILABLE | Crawling needs Starter or a higher plan. |
403 | BLOCKED_URL | The starting URL or webhook URL points to a private or reserved address. |
404 | CRAWL_NOT_FOUND, CRAWL_PAGE_NOT_FOUND, CRAWL_OUTPUT_NOT_FOUND | The crawl, page or output does not exist for this API key. |
409 | CRAWL_NOT_FINISHED | Delete or export of a crawl that is still running. |
409 | CRAWL_CANNOT_BE_CANCELLED | The crawl has already ended. |
409 | IDEMPOTENCY_CONFLICT | The Idempotency-Key was used with a different request body. |
429 | RATE_LIMITED | Too many capture requests. |
429 | TOO_MANY_CRAWLS | The organization already has 10 unfinished crawls. |
503 | CRAWL_UNAVAILABLE | Crawling is temporarily unavailable. Retry later. |
What crawls don't do
A crawl captures pages with the options in Request fields only. It doesn't support PDF output, selector or AI JSON, actions, device presets, sessions (cookies, headers, storage), stealth mode or AI cleanup. To capture specific pages with those, send their URLs to /v1/screenshots or bulk instead.
Scope, safety, and retention
- Crawls follow links found in the rendered page and URLs from the site's sitemap. They stay on the starting site and, unless
entireSiteis set, under the starting URL's folder. If the starting URL redirects, pages under the path it redirected to are included too. www.example.comandexample.comare treated as the same site.- Internationalized domain names such as
müller.deare accepted and crawled in their ASCII form (xn--mller-kva.de). - URL fragments are removed, cycles are deduplicated, and non-HTTP links are ignored.
- The crawler identifies itself to
robots.txtasAllScreenshotsBot. Rules forAllScreenshotsBotapply when present; otherwise the rules for*apply. A missingrobots.txtallows everything; one that returns a server error or cannot be reached stops the crawl of that site. Crawl-delayis respected up to 30 seconds.- Every URL, redirect, and browser request is checked against private and reserved network ranges.
- Output files expire after seven days. Crawl metadata remains in history until you delete it.