Allscreenshots
API reference

Website crawls

Render and extract a hierarchy of pages from one website

Website crawls

The crawl API starts at one URL, follows the links on each rendered page and the site's sitemap, and captures every page as a screenshot, HTML, Markdown, or a JSON page document. Crawls run asynchronously and are available on Starter and every higher plan.

Each captured page uses one screenshot credit, however many outputs you select. Failed and skipped pages are not charged, and neither are pages that show a bot check or an HTTP error instead of content.

Start a crawl

POST /v1/crawls
curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: docs-crawl-2026-11-16' \
  -d '{
    "url": "https://example.com",
    "depth": 2,
    "limit": 25,
    "outputs": ["screenshot", "html", "markdown", "json"],
    "fullPage": true,
    "viewport": { "width": 1440, "height": 900 },
    "format": "webp",
    "webhookUrl": "https://your-app.com/webhooks/allscreenshots"
  }'

The response is 202 Accepted:

{
  "id": "a1b2c3d4e5",
  "status": "QUEUED",
  "statusUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5",
  "createdAt": "2026-09-27T04:00:00Z"
}

Send an Idempotency-Key header (1–100 letters, digits, _ or -) to make retries safe. Repeating the request with the same key and body returns the original crawl with an Idempotency-Replayed: true header instead of starting a second one.

Request fields

FieldTypeDefaultDescription
urlstringrequiredHTTP or HTTPS starting URL.
depthinteger2Link distance from the root, from 0 to 5. The root is depth 0.
limitinteger25Maximum number of pages captured and charged, from 1 to 100.
outputsstring[]allAny non-empty combination of screenshot, html, markdown, and json.
sitemapstringincludeinclude adds the URLs in the site's sitemap to the links found on pages. only crawls the sitemap URLs without following links. skip ignores the sitemap.
includePatternsstring[][]Full-URL glob patterns that a discovered URL must match.
excludePatternsstring[][]Full-URL glob patterns to skip. Exclusions win over inclusions.
entireSitebooleanfalseFollow links anywhere on the site. By default the crawl stays under the starting URL's folder: /docs/ for https://example.com/docs/ or https://example.com/docs/intro, /wiki/ for https://example.com/wiki/Some_article. A start at the home page covers the whole site either way.
includeSubdomainsbooleanfalseAlso follow links to subdomains of the site, for example docs.example.com from www.example.com.
crawlDelayMsinteger500Minimum wait between two requests to the same hostname, from 500 ms to 30 seconds. A longer Crawl-delay in robots.txt is respected up to 30 seconds.
viewportobject1440 × 900Browser viewport dimensions and optional device scale factor.
fullPagebooleantrueCapture the entire scrollable page.
formatstringwebpScreenshot format: png, jpeg, or webp.
qualityinteger80JPEG/WebP quality from 1 to 100.
renderDelayinteger0Wait after rendering before extracting outputs, up to 30 seconds.
waitUntilstringdomcontentloadedload, domcontentloaded, or networkidle.
timeoutinteger30000Per-page timeout in milliseconds, from 1 to 60 seconds.
darkModebooleanfalseRequest the browser's dark color scheme.
blockAdsbooleantrueBlock common ads and trackers.
blockCookieBannersbooleantrueHide common cookie consent overlays.
blockPopupsbooleantrueRemove page-blocking marketing, newsletter, location, survey, app-promotion, notification, and age-gate overlays from every rendered page.
webhookUrlstringnoneCalled once when the crawl finishes, fails, or is cancelled. See Webhooks.
webhookSecretstringnoneSigns the webhook body. It is stored with the crawl and never returned.

How the page limit works

limit counts captured pages. Pages that are skipped (blocked by robots.txt, redirected off the site) or fail do not use it up: the crawler keeps up to twice the limit in its queue and moves on to the next URL. When the limit is reached, the remaining queued pages are marked SKIPPED with LIMIT_REACHED.

Links to files that are not web pages, such as PDFs, images, videos, archives and office documents, are not queued.

Links in a page's content are queued before links in its header, menus, sidebars and footer, and links that are not visible come last. A crawl with a small limit therefore follows the page's own links (articles, products, documentation sections) rather than the site menu.

Poll crawl status

curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5' \
  -H 'X-API-Key: YOUR_API_KEY'
{
  "id": "a1b2c3d4e5",
  "url": "https://example.com/",
  "status": "RUNNING",
  "depth": 2,
  "limit": 25,
  "outputs": ["screenshot", "markdown", "json"],
  "sitemap": "include",
  "webhookUrl": "https://your-app.com/webhooks/allscreenshots",
  "progress": {
    "discovered": 9,
    "completed": 4,
    "failed": 0,
    "skipped": 1,
    "charged": 4,
    "queued": 4
  },
  "createdAt": "2026-09-27T04:00:00Z",
  "startedAt": "2026-09-27T04:00:01Z"
}

Response fields

FieldDescription
statusQUEUED, RUNNING, COMPLETED, PARTIALLY_COMPLETED, FAILED, or CANCELLED.
progress.discoveredURLs found and queued so far, including the root.
progress.completedPages captured.
progress.failedPages that could not be captured.
progress.skippedPages not captured on purpose: robots.txt, an off-site redirect, the page limit, or an early stop.
progress.chargedCredits used by this crawl.
progress.queuedPages still waiting. 0 once the crawl has ended.
errorCode, errorMessageWhy the crawl stopped early or captured nothing.

A crawl is:

  • COMPLETED when every page it tried was captured or deliberately skipped.
  • PARTIALLY_COMPLETED when at least one page was captured and at least one failed, or when it stopped early because credits or the API key's limit ran out.
  • FAILED when no page was captured.

List crawls

curl 'https://api.allscreenshots.com/v1/crawls?page=0&pageSize=20' \
  -H 'X-API-Key: YOUR_API_KEY'

Returns { "crawls": [...], "page", "pageSize", "total", "totalPages" }, newest first. An API key sees the crawls it started; the dashboard shows every crawl in the organization.

List crawled pages

curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages?page=0&pageSize=100' \
  -H 'X-API-Key: YOUR_API_KEY'

Each page includes an id, parentId, depth, status, HTTP metadata, and its available outputs. parentId is the first page that discovered the URL and can be used to render a hierarchy. Pages from the sitemap have the root page as their parent. A crawl can hold up to 200 pages, so read totalPages and request the next page when there is one.

{
  "pages": [{
    "id": "p9q8r7s6t5",
    "parentId": "r1o2o3t4p5",
    "url": "https://example.com/about",
    "depth": 1,
    "status": "COMPLETED",
    "title": "About",
    "httpStatus": 200,
    "outputs": [{
      "type": "screenshot",
      "contentType": "image/webp",
      "size": 48219,
      "resultUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages/p9q8r7s6t5/outputs/screenshot",
      "expiresAt": "2026-10-04T04:00:05Z"
    }]
  }],
  "page": 0,
  "pageSize": 100,
  "total": 9,
  "totalPages": 1
}

Page statuses are QUEUED, RUNNING, COMPLETED, FAILED, SKIPPED, and CANCELLED. A failed or skipped page has an errorCode and errorMessage:

Page codeStatusMeaning
ROBOTS_DISALLOWEDSKIPPEDThe site's robots.txt does not allow this URL.
ROBOTS_UNAVAILABLESKIPPEDThe site's robots.txt returned a server error or could not be reached, so the site was not crawled.
REDIRECT_OUT_OF_SCOPESKIPPEDThe page redirected to another site.
DUPLICATE_PAGESKIPPEDThe page landed on a URL this crawl already captured, for example /docs redirecting to /docs/. Not charged.
LIMIT_REACHEDSKIPPEDThe crawl had already captured limit pages.
QUOTA_EXCEEDEDSKIPPEDThe crawl stopped because no screenshot credits remained.
API_KEY_LIMIT_EXCEEDEDSKIPPEDThe crawl stopped because the API key reached its screenshot limit.
PAGE_BLOCKEDFAILEDThe site showed a bot check instead of the page. Not charged.
HTTP_ERRORFAILEDThe site answered with an error status, kept in httpStatus. Not charged.
PAGE_CRASHEDFAILEDThe page could not be rendered after three attempts.
other codesFAILEDThe page could not be rendered, for example RENDER_FAILED or a timeout.

Download an output

curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages/p9q8r7s6t5/outputs/markdown' \
  -H 'X-API-Key: YOUR_API_KEY' \
  --output about.md

What each output holds

Crawl outputs share names with the outputs of a screenshot request but are made for whole sites, so three of them differ:

OutputIn a crawlWith /v1/screenshots outputs (default)
screenshotThe page in format, full page by defaultThe viewport by default
htmlThe page's HTML as loaded, scripts includedCleaned: no scripts, absolute URLs
markdownThe whole page bodyThe main content only
jsonA page document (below)The page's own data (auto), selectors, or AI

The page document has id, parentId, url (where the page landed), requestedUrl, title, description, language, canonicalUrl, statusCode, contentType, depth, headings, links (each with url, text and internal), the visible text and openGraph.

Stored HTML is returned as a download-safe document: it cannot run scripts when opened from the API.

Export a whole crawl

GET /v1/crawls/{id}/export?format=zip|markdown|jsonl

Downloads every page of a finished crawl in one file. A crawl that is still running returns 409 CRAWL_NOT_FINISHED.

FormatContents
zip (default)One folder per page, named after its URL path, with each stored output, plus manifest.json listing every page with its URL, title, status, depth, parent and files.
markdownOne Markdown file with every captured page in crawl order, each preceded by its source URL and title. Needs the markdown output.
jsonlOne JSON document per line and per page: the page fields plus markdown, html, data (the JSON output) and screenshotUrl for the outputs you stored.

Add outputs=markdown,json to include only some outputs in zip and jsonl exports.

A ZIP export holds one folder per page, numbered in crawl order, and a manifest:

crawl-a1b2c3d4e5.zip
├── manifest.json
├── 001-index/
│   ├── screenshot.webp
│   ├── page.md
│   └── page.json
├── 002-getting-started-quickstart/
│   └── ...

manifest.json lists every page, including skipped and failed ones, with id, url, title, status, depth, parentId, httpStatus, errorCode, folder and files. Outputs that expired before the export are listed in expiredOutputs.

curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/export?format=markdown' \
  -H 'X-API-Key: YOUR_API_KEY' \
  --output site.md

Webhooks

When a crawl has a webhookUrl, one POST is sent when it ends:

  • CRAWL_COMPLETED for COMPLETED, PARTIALLY_COMPLETED, and CANCELLED crawls (read status).
  • CRAWL_FAILED when no page was captured.
{
  "event": "CRAWL_COMPLETED",
  "timestamp": "2026-09-27T04:03:10Z",
  "data": {
    "crawlId": "a1b2c3d4e5",
    "url": "https://example.com/",
    "status": "COMPLETED",
    "progress": { "discovered": 31, "completed": 25, "failed": 0, "skipped": 6, "charged": 25 },
    "errorCode": null,
    "errorMessage": null,
    "statusUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5",
    "pagesUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages",
    "completedAt": "2026-09-27T04:03:09Z"
  }
}

With a webhookSecret, the body is signed in X-Webhook-Signature exactly as for other webhooks. If your endpoint answers with 429, a 5xx status, or cannot be reached, delivery is retried with growing intervals for up to 24 hours. Every attempt carries the same X-Webhook-Delivery-Id, so you can ignore repeats.

Cancel or delete

# Stop the crawl. The page being rendered is discarded and not charged.
curl -X POST 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/cancel' \
  -H 'X-API-Key: YOUR_API_KEY'

# Delete metadata and every stored output. Cancel a running crawl first.
curl -X DELETE 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5' \
  -H 'X-API-Key: YOUR_API_KEY'

Once cancel returns, no further pages are stored or charged. Deleting a crawl that is still running returns 409 CRAWL_NOT_FINISHED.

Limits

LimitValue
Pages captured per crawl100
Depth5
Crawls running at once per organization2 (others wait in the queue, oldest first)
Unfinished crawls per organization10
Wait between requests to one hostname500 ms to 30 seconds
Sitemap files read per crawl5, including those listed in a sitemap index
Sitemap URLs used per crawl5,000
includePatterns, excludePatterns25 each, up to 2,048 characters per pattern
Output retention7 days

Starting a crawl counts toward your plan's capture rate limit, not the general API rate limit.

Errors

StatusCodeWhen
400VALIDATION_ERROR, INVALID_REQUESTA field is out of range or malformed, or the export format is unknown.
400EXPORT_UNAVAILABLEA Markdown export was requested for a crawl without the markdown output.
400INVALID_IDEMPOTENCY_KEYThe Idempotency-Key header has invalid characters or length.
402QUOTA_EXCEEDEDNo screenshot credits remain.
403CAPABILITY_NOT_AVAILABLECrawling needs Starter or a higher plan.
403BLOCKED_URLThe starting URL or webhook URL points to a private or reserved address.
404CRAWL_NOT_FOUND, CRAWL_PAGE_NOT_FOUND, CRAWL_OUTPUT_NOT_FOUNDThe crawl, page or output does not exist for this API key.
409CRAWL_NOT_FINISHEDDelete or export of a crawl that is still running.
409CRAWL_CANNOT_BE_CANCELLEDThe crawl has already ended.
409IDEMPOTENCY_CONFLICTThe Idempotency-Key was used with a different request body.
429RATE_LIMITEDToo many capture requests.
429TOO_MANY_CRAWLSThe organization already has 10 unfinished crawls.
503CRAWL_UNAVAILABLECrawling is temporarily unavailable. Retry later.

What crawls don't do

A crawl captures pages with the options in Request fields only. It doesn't support PDF output, selector or AI JSON, actions, device presets, sessions (cookies, headers, storage), stealth mode or AI cleanup. To capture specific pages with those, send their URLs to /v1/screenshots or bulk instead.

Scope, safety, and retention

  • Crawls follow links found in the rendered page and URLs from the site's sitemap. They stay on the starting site and, unless entireSite is set, under the starting URL's folder. If the starting URL redirects, pages under the path it redirected to are included too.
  • www.example.com and example.com are treated as the same site.
  • Internationalized domain names such as müller.de are accepted and crawled in their ASCII form (xn--mller-kva.de).
  • URL fragments are removed, cycles are deduplicated, and non-HTTP links are ignored.
  • The crawler identifies itself to robots.txt as AllScreenshotsBot. Rules for AllScreenshotsBot apply when present; otherwise the rules for * apply. A missing robots.txt allows everything; one that returns a server error or cannot be reached stops the crawl of that site.
  • Crawl-delay is respected up to 30 seconds.
  • Every URL, redirect, and browser request is checked against private and reserved network ranges.
  • Output files expire after seven days. Crawl metadata remains in history until you delete it.

On this page