Website crawls

The crawl API discovers links from rendered pages and captures each page as a screenshot, HTML, Markdown, or a normalized JSON page document. Crawls run asynchronously and are available on every paid plan.

Each successfully rendered page uses one screenshot credit, regardless of how many output formats you select. Failed and skipped pages are not charged.

Start a crawl

POST /v1/crawls

curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "depth": 2,
    "limit": 25,
    "outputs": ["screenshot", "html", "markdown", "json"],
    "fullPage": true,
    "viewport": { "width": 1440, "height": 900 },
    "format": "webp"
  }'

The response is 202 Accepted:

{
  "id": "a1b2c3d4e5",
  "status": "QUEUED",
  "statusUrl": "https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5",
  "createdAt": "2026-09-27T04:00:00Z"
}

Request fields

Field	Type	Default	Description
`url`	string	required	HTTP or HTTPS starting URL.
`depth`	integer	`2`	Link distance from the root, from `0` to `5`. The root is depth `0`.
`limit`	integer	`25`	Maximum number of unique pages, from `1` to `100`.
`outputs`	string[]	all	Any non-empty combination of `screenshot`, `html`, `markdown`, and `json`.
`includePatterns`	string[]	`[]`	Full-URL glob patterns that a discovered URL must match.
`excludePatterns`	string[]	`[]`	Full-URL glob patterns to skip. Exclusions win over inclusions.
`includeSubdomains`	boolean	`false`	Follow links onto subdomains of the effective root host.
`crawlDelayMs`	integer	`500`	Minimum delay per hostname. A longer `robots.txt` delay wins.
`viewport`	object	`1440 × 900`	Browser viewport dimensions and optional device scale factor.
`fullPage`	boolean	`true`	Capture the entire scrollable page.
`format`	string	`webp`	Screenshot format: `png`, `jpeg`, or `webp`.
`quality`	integer	`80`	JPEG/WebP quality from `1` to `100`.
`renderDelay`	integer	`0`	Wait after rendering before extracting outputs, up to 30 seconds.
`waitUntil`	string	`domcontentloaded`	`load`, `domcontentloaded`, or `networkidle`.
`timeout`	integer	`30000`	Per-page timeout in milliseconds, from 1 to 60 seconds.
`darkMode`	boolean	`false`	Request the browser's dark color scheme.
`blockAds`	boolean	`true`	Block common ads and trackers.
`blockCookieBanners`	boolean	`true`	Hide common cookie consent overlays.

Poll crawl status

curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5' \
  -H 'X-API-Key: YOUR_API_KEY'

{
  "id": "a1b2c3d4e5",
  "url": "https://example.com/",
  "status": "RUNNING",
  "depth": 2,
  "limit": 25,
  "outputs": ["screenshot", "markdown", "json"],
  "progress": {
    "discovered": 9,
    "completed": 4,
    "failed": 0,
    "skipped": 1,
    "charged": 4,
    "queued": 4
  },
  "createdAt": "2026-09-27T04:00:00Z",
  "startedAt": "2026-09-27T04:00:01Z"
}

Crawl statuses are QUEUED, RUNNING, COMPLETED, PARTIALLY_COMPLETED, FAILED, and CANCELLED. A crawl is partially completed when at least one page succeeds and another fails, is skipped, or cannot run because credits are exhausted.

List crawled pages

curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages?page=0&pageSize=100' \
  -H 'X-API-Key: YOUR_API_KEY'

Each page includes an id, parentId, depth, status, HTTP metadata, and its available outputs. parentId is the first page that discovered the URL and can be used to render a hierarchy.

{
  "pages": [{
    "id": "p9q8r7s6t5",
    "parentId": "r1o2o3t4p5",
    "url": "https://example.com/about",
    "depth": 1,
    "status": "COMPLETED",
    "title": "About",
    "httpStatus": 200,
    "outputs": [{
      "type": "screenshot",
      "contentType": "image/webp",
      "size": 48219,
      "resultUrl": "/v1/crawls/a1b2c3d4e5/pages/p9q8r7s6t5/outputs/screenshot",
      "expiresAt": "2026-10-04T04:00:05Z"
    }]
  }],
  "page": 0,
  "pageSize": 100,
  "total": 9,
  "totalPages": 1
}

Page statuses are QUEUED, RUNNING, COMPLETED, FAILED, SKIPPED, and CANCELLED.

Download an output

curl 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/pages/p9q8r7s6t5/outputs/markdown' \
  -H 'X-API-Key: YOUR_API_KEY' \
  --output about.md

The JSON output has a built-in page-document schema with the requested and effective URL, title, description, language, canonical URL, HTTP metadata, depth, parent ID, headings, rendered links, visible text, and Open Graph metadata.

Cancel or delete

# Stop discovery after the page currently rendering
curl -X POST 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5/cancel' \
  -H 'X-API-Key: YOUR_API_KEY'

# Delete metadata and every stored output
curl -X DELETE 'https://api.allscreenshots.com/v1/crawls/a1b2c3d4e5' \
  -H 'X-API-Key: YOUR_API_KEY'

Scope, safety, and retention

Crawls follow links found in the rendered DOM and stay on the effective root hostname by default.
URL fragments are removed, cycles are deduplicated, and non-HTTP links are ignored.
robots.txt rules and crawl delays are always respected.
Every URL, redirect, and browser request is checked against private and reserved network ranges.
Output files expire after seven days. Crawl metadata remains in history until you delete it.

Website crawls

On this page