Allscreenshots
Features

Website crawler

Capture up to 100 pages of a site as screenshots, HTML, Markdown and JSON with one request, then export them in one file

Website crawler

Send one URL to POST /v1/crawls and the crawler follows the site's links and sitemap, rendering each page in a real browser. Every page comes back as a screenshot, HTML, Markdown and a JSON page document, or any mix of the four. When the crawl ends you can download the whole site as a ZIP, one Markdown file or JSON Lines.

The crawler is available on Starter and every higher plan. Each captured page uses 1 screenshot credit, however many outputs you choose. Pages that fail, are skipped, or show a bot check or an error page aren't charged.

curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://docs.allscreenshots.com/",
    "depth": 3,
    "limit": 100,
    "entireSite": true,
    "outputs": ["screenshot", "markdown", "json"]
  }'

A real crawl

We ran the request above against this documentation site. The crawler read the 71 URLs in the site's sitemap and the 19 internal links on the home page, found 73 pages in all, and captured 71 of them. The other two were links to pages that don't exist: they returned 404, were reported as HTTP_ERROR and weren't charged. Because of those two, the crawl ended as PARTIALLY_COMPLETED. It used 71 credits.

Discovered73 pages: 1 start page, 70 at depth 1, 2 at depth 2
Captured and charged71
Failed2 (HTTP_ERROR, 404)
Render time per page9.3 seconds median, full-page screenshots
Whole crawl11 minutes, one page at a time with 500 ms between requests
A grid of 24 page screenshots from the crawl of the Allscreenshots documentation: the introduction, guides and API reference pages.
The first 24 of the 71 pages the crawl captured

The two 404s were broken links in our own docs, on the social images page and a use-case page. We fixed them; the crawl is a quick way to find yours.

Try the settings

The explorer replays that crawl. Change depth, limit or sitemap and watch which pages the crawler would capture and in what order. The pages, their links, screenshots and outcomes are real; the order and limits follow the crawler's rules.

sitemap

Links on pages plus the sitemap

discovered
1
completed
0
failed
0
skipped
0
charged
0
queued
1

Depth 0 (start URL) · 1 page

Depth 1 · 0 pages

    Depth 2 · 0 pages

      CapturedQueuedSkipped (robots.txt, duplicate)FailedSkipped: LIMIT_REACHED

      Simulation over a real crawl of docs.allscreenshots.com (71 pages captured, depth 3, limit 100, finished in 11 min). Pages, links, screenshots and outcomes are from that crawl; the order and limits follow the crawler's rules. Pages the real crawl didn't reach show their path instead of a screenshot.

      Some things to try:

      • limit: 25 with depth: 2 captures the first 25 pages and queues 25 more, twice the limit, in case some fail. The 25 left in the queue end as SKIPPED with LIMIT_REACHED and aren't charged.
      • sitemap: "skip" follows links only. On this site the home page has 19 internal links, so the crawl spreads out over three levels instead of one: 18 pages at depth 1, 40 at depth 2 and 2 at depth 3.
      • sitemap: "only" captures exactly the 71 URLs the sitemap lists and follows no links, so the two broken links are never visited.
      • depth: 0 captures only the start URL.

      How discovery works

      Scope

      A crawl stays on the site it starts on and, unless you set entireSite, inside the folder of the start URL.

      Default

      "url": "https://example.com/docs/intro"

      • example.com/docs/introfollowed
      • example.com/docs/api/authfollowed
      • www.example.com/docs/faqfollowed
      • example.com/blog/launchnot followed
      • help.example.com/docs/startnot followed

      entireSite

      "entireSite": true

      • example.com/docs/introfollowed
      • example.com/docs/api/authfollowed
      • www.example.com/docs/faqfollowed
      • example.com/blog/launchfollowed
      • help.example.com/docs/startnot followed

      entireSite + includeSubdomains

      "entireSite": true, "includeSubdomains": true

      • example.com/docs/introfollowed
      • example.com/docs/api/authfollowed
      • www.example.com/docs/faqfollowed
      • example.com/blog/launchfollowed
      • help.example.com/docs/startfollowed
      Starting at https://example.com/docs/intro. By default the crawl stays under /docs/, the folder of the start URL; www. and the bare domain count as the same site. entireSite lifts the folder limit and includeSubdomains adds other subdomains.
      • The folder is the start URL itself when it ends with / (/docs/), and otherwise its parent folder: /wiki/ for /wiki/Web_crawler, the whole site for /pricing.
      • www.example.com and example.com are the same site.
      • includeSubdomains also follows links to subdomains, such as docs.example.com from www.example.com.
      • If the start URL redirects, pages under where it landed are in scope too.
      • Links to files that aren't web pages (PDFs, images, video, archives, office documents) are never queued.

      A page's links are queued in this order:

      1. Links in the page's content, interleaved with sitemap URLs on the start page
      2. Links in the header, menus, sidebars and footer
      3. Links that aren't visible

      Pages are captured lowest depth first, in the order they were found. With a small limit, that means you get the pages an article or a product links to rather than every item in the site menu.

      Sitemaps

      sitemapWhat the crawler does
      include (default)Adds the sitemap's URLs to the links found on the start page
      skipIgnores sitemaps and follows links only
      onlyCaptures the start page and the sitemap's URLs, and follows no links

      The crawler reads the sitemaps named in robots.txt and /sitemap.xml, follows a sitemap index one level, and reads gzip files. It uses at most 5 sitemap files and 5,000 URLs per crawl. Sitemap URLs are found at depth 1, so they need depth of 1 or more.

      Include and exclude patterns

      includePatterns and excludePatterns are globs that must match the whole URL:

      • * matches any run of characters, / included.
      • ? matches one character.
      • An exclude wins over an include.
      • With no includes, every URL that no exclude matches is followed.

      Try them against the URLs of the real crawl:

      Try

      16 of 73 URLs from the real crawl would be followed.

      • /not followed:
      • /api-reference/async-jobsnot followed:
      • /api-reference/batchesnot followed:
      • /api-reference/bulknot followed:
      • /api-reference/composenot followed:
      • /api-reference/crawlsnot followed:
      • /api-reference/outputsnot followed:
      • /api-reference/schedulesnot followed:
      • /api-reference/screenshotsnot followed:
      • /api-reference/webhooksnot followed:
      • /dashboard/domainsnot followed:
      • /features/ai-cleanupnot followed:
      • /features/ai-extractionnot followed:
      • /features/blockingnot followed:
      • /features/full-page-capturenot followed:
      • /features/interactive-actionsnot followed:
      • /features/sessionsnot followed:
      • /features/social-imagesnot followed:
      • /features/stealth-modenot followed:
      • /features/stylingnot followed:
      • /features/viewport-and-devicesnot followed:
      • /getting-started/authenticationnot followed:
      • /getting-started/quickstartnot followed:
      • /getting-started/rate-limitsnot followed:
      • /guides/canvas-exportsfollowed:
      • /guides/canvas-image-processingfollowed:
      • /guides/canvas-webhooksfollowed:
      • /guides/canvas-workflowsfollowed:
      • /guides/ci-cd-integrationfollowed:
      • /guides/cicd-integrationfollowed:
      • /guides/crawl-a-websitefollowed:
      • /guides/full-page-screenshotsfollowed:
      • /guides/multi-output-extractionfollowed:
      • /guides/openclaw-integrationfollowed:
      • /guides/pdf-generationfollowed:
      • /guides/remove-popups-and-overlaysfollowed:
      • /guides/scheduled-screenshotsfollowed:
      • /guides/social-media-previewsfollowed:
      • /guides/visual-regression-testingfollowed:
      • /guides/website-monitoringfollowed:
      • /integrations/n8nnot followed:
      • /llm-txtnot followed:
      • /sdksnot followed:
      • /sdks/csharpnot followed:
      • /sdks/gonot followed:
      • /sdks/javanot followed:
      • /sdks/javascriptnot followed:
      • /sdks/kotlinnot followed:
      • /sdks/phpnot followed:
      • /sdks/pythonnot followed:
      • /sdks/rubynot followed:
      • /sdks/rustnot followed:
      • /sdks/swiftnot followed:
      • /sdks/typescriptnot followed:
      • /supportnot followed:
      • /tools/agent-pluginsnot followed:
      • /tools/canvanot followed:
      • /tools/clinot followed:
      • /tools/figmanot followed:
      • /tools/mcpnot followed:
      • /tools/mkdocsnot followed:
      • /use-casesnot followed:
      • /use-cases/ci-cd-workflowsnot followed:
      • /use-cases/compliance-auditingnot followed:
      • /use-cases/content-archivingnot followed:
      • /use-cases/customer-onboardingnot followed:
      • /use-cases/ecommerce-monitoringnot followed:
      • /use-cases/legal-evidencenot followed:
      • /use-cases/report-generationnot followed:
      • /use-cases/social-media-og-imagesnot followed:
      • /use-cases/structured-extractionnot followed:
      • /use-cases/visual-regression-testingnot followed:
      • /use-cases/website-monitoringnot followed:

      Patterns match the whole URL, including https:// and the host, so a pattern that starts with a path matches nothing. The start URL is always captured; patterns decide which discovered links are followed.

      Patterns decide which discovered links are followed; the start URL is always captured. You can send up to 25 of each, up to 2,048 characters long.

      Progress and statuses

      Poll GET /v1/crawls/{id}, or set webhookUrl to get one call when the crawl ends. The progress object counts pages:

      {
        "id": "zBFfJpP5Ev",
        "status": "PARTIALLY_COMPLETED",
        "progress": { "discovered": 73, "completed": 71, "failed": 2, "skipped": 0, "charged": 71, "queued": 0 }
      }

      QUEUED

      Waits for a slot: 2 crawls run at once per organization

      RUNNING

      Pages are captured one at a time, lowest depth first
      • COMPLETEDEvery page it tried was captured or deliberately skipped
      • PARTIALLY_COMPLETEDSome pages captured, and some failed, or credits or the API key limit ran out
      • FAILEDNo page was captured
      • CANCELLEDYou cancelled it while queued or running
      A crawl ends in one of four statuses. Captured pages and their outputs stay available for 7 days whichever way it ended.
      • COMPLETED: every page the crawl tried was captured or skipped on purpose.
      • PARTIALLY_COMPLETED: at least one page was captured and at least one failed, or the crawl stopped because credits or the API key's limit ran out.
      • FAILED: no page was captured.
      • CANCELLED: you cancelled it. Pages already captured stay available.

      Every skipped or failed page has an errorCode, such as ROBOTS_DISALLOWED, LIMIT_REACHED, DUPLICATE_PAGE or HTTP_ERROR. The crawl reference lists them all.

      What each page stores

      OutputWhat you get
      screenshotThe page as an image, full page by default, in format (WebP by default)
      htmlThe page's HTML as the browser loaded it
      markdownThe whole page body as Markdown
      jsonA page document: URL, requested URL, title, description, language, canonical URL, HTTP status, depth, parent page, headings, links (each marked internal or not), visible text and OpenGraph tags

      These differ from the outputs of a screenshot request, which return cleaned HTML, the main content as Markdown, and JSON by selector, page data or AI. Use /v1/screenshots with outputs for a few pages you need in that form.

      Outputs are kept for 7 days. Crawl and page records stay until you delete the crawl.

      Exports

      When a crawl has ended, GET /v1/crawls/{id}/export returns every page in one download. For the crawl above:

      formatWhat came back
      zip18.2 MB: a folder per page with its files, and manifest.json (214 files)
      markdown535 KB: one file with all 71 pages, each headed by its source URL and title
      jsonl1.7 MB: 73 lines, one per page, failed pages included, with the Markdown, the JSON document and a screenshotUrl

      The ZIP looks like this:

      manifest.json
      screenshot.webp
      page.md
      page.json

      The Markdown export starts like this:

      # Crawl of https://docs.allscreenshots.com/
      
      ---
      
      Source: https://docs.allscreenshots.com/
      Title: Introduction | Allscreenshots Docs
      
      # Introduction
      
      Capture high-quality screenshots of any website with a simple API

      Add outputs=markdown,json to leave screenshots out of a zip or jsonl export.

      Politeness and robots.txt

      • The crawler identifies itself as AllScreenshotsBot. Rules for AllScreenshotsBot apply when robots.txt has them, otherwise the rules for *.
      • A disallowed URL is skipped with ROBOTS_DISALLOWED.
      • A missing robots.txt allows everything. One that returns a server error or can't be reached after three tries stops the crawl of that site (ROBOTS_UNAVAILABLE).
      • Requests to one host are at least crawlDelayMs apart: 500 ms by default, up to 30 seconds. A longer Crawl-delay in robots.txt is respected up to 30 seconds.

      Billing

      WhatCounts as
      A captured page1 screenshot, however many outputs
      A failed or skipped page, a bot check or an HTTP error pageNothing
      A page that redirects to one the crawl already capturedNothing (DUPLICATE_PAGE)
      A crawl that runs out of creditsIt stops; captured pages stay, the rest are skipped with QUOTA_EXCEEDED

      Limits

      LimitValue
      Pages captured per crawl (limit)100
      Depth5
      Crawls running at once per organization2, the rest wait in the queue
      Unfinished crawls per organization10
      Patterns25 includes and 25 excludes, 2,048 characters each
      Sitemaps5 files, 5,000 URLs
      Output retention7 days

      Crawls don't support PDF output, selector or AI JSON, actions, device presets, sessions, stealth mode or AI cleanup. Send the URLs you need that way to /v1/screenshots or bulk.

      Next steps

      On this page