Allscreenshots
Guides

Crawl and map a website

Capture a visual site tree with HTML, Markdown, and JSON for every page

Crawl and map a website

Website crawls turn a starting URL into a hierarchy of rendered pages. Each page can include a screenshot plus HTML, Markdown, and a normalized JSON document, all captured from the same browser state.

The crawler is available on Starter and every higher plan. Each captured page uses one screenshot credit.

A grid of page screenshots from a crawl of the Allscreenshots documentation.

The Website crawler page shows a real crawl of these docs, 71 pages in 11 minutes, with an interactive replay where you can change depth, limit and sitemap and test include and exclude patterns.

Crawl a documentation site

Start with a narrow scope

Use a depth of two and an include pattern to keep discovery inside the documentation section:

curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/docs/",
    "depth": 2,
    "limit": 50,
    "includePatterns": ["https://example.com/docs/*"],
    "excludePatterns": ["*/docs/archive/*"],
    "outputs": ["screenshot", "markdown", "json"],
    "fullPage": true
  }'

The root page is depth 0; its links are depth 1, and links discovered from those pages are depth 2.

Poll until the crawl is terminal

const terminal = new Set([
  'COMPLETED', 'PARTIALLY_COMPLETED', 'FAILED', 'CANCELLED',
]);

async function waitForCrawl(id) {
  while (true) {
    const response = await fetch(
      `https://api.allscreenshots.com/v1/crawls/${id}`,
      { headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
    );
    const crawl = await response.json();
    console.log(`${crawl.progress.completed}/${crawl.progress.discovered} pages`);
    if (terminal.has(crawl.status)) return crawl;
    await new Promise((resolve) => setTimeout(resolve, 1500));
  }
}

Rebuild the hierarchy

Fetch the page records and connect each page to its parentId:

const response = await fetch(
  `https://api.allscreenshots.com/v1/crawls/${crawlId}/pages?pageSize=100`,
  { headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
);
const { pages } = await response.json();

const children = Map.groupBy(
  pages.filter((page) => page.parentId),
  (page) => page.parentId,
);
const root = pages.find((page) => page.parentId === null);

A page is stored only once. When several pages link to the same URL, its first discoverer becomes its parent. A crawl can hold up to 200 pages (captured plus skipped and failed), so request page=1 as well when totalPages is 2.

Download the page bundle

async function downloadOutput(crawlId, page, type) {
  const output = page.outputs.find((item) => item.type === type);
  if (!output) return null;
  const response = await fetch(
    output.resultUrl,
    { headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
  );
  return type === 'screenshot' ? response.arrayBuffer() : response.text();
}

Use Markdown for search or LLM ingestion, JSON for page metadata and link analysis, HTML for archival, and screenshots for visual review.

Or download everything at once

When the crawl has finished, one request returns every page:

# All pages as one Markdown file, ready to give to a model
curl 'https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=markdown' \
  -H 'X-API-Key: YOUR_API_KEY' --output docs.md

# Every stored file plus manifest.json
curl 'https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=zip' \
  -H 'X-API-Key: YOUR_API_KEY' --output docs.zip

format=jsonl returns one JSON document per page.

Dashboard site atlas

Open Crawler in the dashboard to configure the same options without writing code. The live site atlas shows first-discovery relationships, page status, and screenshot thumbnails. Select a node to preview and download any captured output, or see why a page was skipped or failed. Download all exports a finished crawl as a ZIP, one Markdown file, or JSON Lines.

Skip polling with a webhook

Add webhookUrl (and optionally webhookSecret) to the request and you receive one CRAWL_COMPLETED or CRAWL_FAILED call when the crawl ends. See crawl webhooks.

  • Begin with depth 1 or 2 and a limit of 25 pages.
  • Start at the section you want, with a trailing slash, such as https://example.com/docs/: the crawl stays under that folder. Set entireSite to follow links anywhere on the site.
  • Add include patterns before enabling subdomains.
  • Keep the screenshot output on when you want to inspect the tree visually.
  • Use Markdown rather than HTML for semantic search and model context.
  • Treat PARTIALLY_COMPLETED as usable: successful pages and their outputs remain available.
  • Keep sitemap at include for sites whose pages are not all linked from the home page; use only to capture exactly what the sitemap lists.
  • To allow or block the crawler in robots.txt, use the user agent AllScreenshotsBot.

See the crawl API reference for every request field, status, and response shape.

For complete workflows, see Turn a website into an LLM knowledge base and Audit a site visually before a redesign.

On this page