Crawl and map a website

Website crawls turn a starting URL into a hierarchy of rendered pages. Each page can include a screenshot plus HTML, Markdown, and a normalized JSON document, all captured from the same browser state.

The crawler is a beta feature available on Starter and every higher paid plan. Each successful page consumes one screenshot credit.

Crawl a documentation site

Start with a narrow scope

Use a depth of two and an include pattern to keep discovery inside the documentation section:

curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/docs/",
    "depth": 2,
    "limit": 50,
    "includePatterns": ["https://example.com/docs/*"],
    "excludePatterns": ["*/docs/archive/*"],
    "outputs": ["screenshot", "markdown", "json"],
    "fullPage": true
  }'

The root page is depth 0; its links are depth 1, and links discovered from those pages are depth 2.

Poll until the crawl is terminal

const terminal = new Set([
  'COMPLETED', 'PARTIALLY_COMPLETED', 'FAILED', 'CANCELLED',
]);

async function waitForCrawl(id) {
  while (true) {
    const response = await fetch(
      `https://api.allscreenshots.com/v1/crawls/${id}`,
      { headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
    );
    const crawl = await response.json();
    console.log(`${crawl.progress.completed}/${crawl.progress.discovered} pages`);
    if (terminal.has(crawl.status)) return crawl;
    await new Promise((resolve) => setTimeout(resolve, 1500));
  }
}

Rebuild the hierarchy

Fetch the page records and connect each page to its parentId:

const response = await fetch(
  `https://api.allscreenshots.com/v1/crawls/${crawlId}/pages?pageSize=100`,
  { headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
);
const { pages } = await response.json();

const children = Map.groupBy(
  pages.filter((page) => page.parentId),
  (page) => page.parentId,
);
const root = pages.find((page) => page.parentId === null);

A page is stored only once. When several pages link to the same URL, its first discoverer becomes its parent.

Download the page bundle

async function downloadOutput(crawlId, page, type) {
  const output = page.outputs.find((item) => item.type === type);
  if (!output) return null;
  const response = await fetch(
    `https://api.allscreenshots.com${output.resultUrl}`,
    { headers: { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY } },
  );
  return type === 'screenshot' ? response.arrayBuffer() : response.text();
}

Use Markdown for search or LLM ingestion, JSON for page metadata and link analysis, HTML for archival, and screenshots for visual review.

Dashboard site atlas

Open Crawler in the dashboard to configure the same options without writing code. The live site atlas shows first-discovery relationships, page status, and screenshot thumbnails. Select a node to preview and download any captured output.

Recommended beta settings

Begin with depth 1 or 2 and a limit of 25 pages.
Add include patterns before enabling subdomains.
Keep the screenshot output on when you want to inspect the tree visually.
Use Markdown rather than HTML for semantic search and model context.
Treat PARTIALLY_COMPLETED as usable: successful pages and their outputs remain available.

See the crawl API reference for every request field, status, and response shape.

Crawl and map a website

On this page