Allscreenshots
Guides

Turn a website into an LLM knowledge base

Crawl a site, export every page as Markdown or JSON Lines, and split it into chunks for search or retrieval

Turn a website into an LLM knowledge base

A chatbot or agent that answers questions about a product needs the product's docs, help center or website as text. The website crawler renders every page in a real browser, so pages built with JavaScript come back with their content, and returns each one as Markdown. One export call then gives you the whole site in one file.

This guide crawls a docs site, exports it, and splits it into chunks ready for an embedding model or a search index. We ran it on this documentation: 71 pages became one 535 KB Markdown file.

Crawling needs Starter or a higher plan. Each captured page uses one screenshot credit, whichever outputs you choose.

Start the crawl

Ask for Markdown and the JSON page document. Leave out screenshots if you don't need them: they don't change the price but they make exports much larger.

curl -X POST 'https://api.allscreenshots.com/v1/crawls' \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://docs.example.com/",
    "depth": 3,
    "limit": 100,
    "outputs": ["markdown", "json"],
    "excludePatterns": ["*/changelog/*", "*?*"]
  }'
  • Start at the section you want with a trailing slash, such as https://example.com/docs/: the crawl stays under that folder.
  • Keep sitemap at its default, include, so pages nothing links to are found too.
  • excludePatterns keeps out sections you don't want answers from. *?* skips every URL with a query string, which often means filtered or sorted copies of the same page. Try patterns in the pattern tester.

Wait for it to finish

Poll the crawl until its status is final, or add a webhookUrl to the request and wait for the call.

const API = 'https://api.allscreenshots.com/v1';
const headers = { 'X-API-Key': process.env.ALLSCREENSHOTS_API_KEY };

async function waitForCrawl(id) {
  for (;;) {
    const crawl = await fetch(`${API}/crawls/${id}`, { headers }).then((r) => r.json());
    if (['COMPLETED', 'PARTIALLY_COMPLETED', 'FAILED', 'CANCELLED'].includes(crawl.status)) return crawl;
    await new Promise((resolve) => setTimeout(resolve, 5000));
  }
}

PARTIALLY_COMPLETED is usable: it means some pages failed, for example broken links that return 404. The pages that were captured are all there.

Export the site

For a model with a large context window, the Markdown export may be all you need: one file with every page, each headed by its source URL and title.

curl 'https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=markdown' \
  -H 'X-API-Key: YOUR_API_KEY' --output site.md
# Crawl of https://docs.allscreenshots.com/

---

Source: https://docs.allscreenshots.com/
Title: Introduction | Allscreenshots Docs

# Introduction

Capture high-quality screenshots of any website with a simple API

For retrieval, use JSON Lines instead. Each line is one page with its URL, title, status, depth and Markdown, so every chunk can keep a link back to its source.

curl 'https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=jsonl' \
  -H 'X-API-Key: YOUR_API_KEY' --output site.jsonl
{"crawlId":"zBFfJpP5Ev","id":"iguxHeoKT9","parentId":null,"url":"https://docs.allscreenshots.com/","depth":0,"status":"COMPLETED","title":"Introduction | Allscreenshots Docs","description":"Capture high-quality screenshots of any website with a simple API","language":"en","canonicalUrl":"https://docs.allscreenshots.com/","httpStatus":200,"errorCode":null,"errorMessage":null,"markdown":"# Introduction\n\nCapture high-quality screenshots of any website with a…","data":{"url":"https://docs.allscreenshots.com/","title":"Introduction | Allscreenshots Docs","headings":"…","links":"…","text":"…"}}

The export of our docs crawl had 73 lines: 71 captured pages and the 2 broken links, which have "status": "FAILED" and no Markdown.

Split pages into chunks

Split each page at its headings so a chunk covers one topic, and keep the page URL and the heading with it. This script reads the JSON Lines export and writes chunks.jsonl:

import fs from 'node:fs';
import readline from 'node:readline';

const MAX_CHARS = 4000;
const out = fs.createWriteStream('chunks.jsonl');
const lines = readline.createInterface({ input: fs.createReadStream('site.jsonl') });

for await (const line of lines) {
  const page = JSON.parse(line);
  if (page.status !== 'COMPLETED' || !page.markdown) continue;

  // One section per heading (#, ## or ###), with its heading.
  const sections = page.markdown.split(/\n(?=#{1,3} )/);
  sections.forEach((section, i) => {
    // Headings often link to themselves: keep the text of [Text](#anchor).
    const heading = section.match(/^#{1,3} (.+)/)?.[1].replace(/\[([^\]]*)\]\([^)]*\)/g, '$1') ?? page.title;
    // Long sections are cut at paragraph breaks.
    let chunk = '';
    let part = 0;
    const flush = () => {
      if (!chunk.trim()) return;
      out.write(JSON.stringify({ id: `${page.id}-${i}-${part++}`, url: page.url, title: page.title, heading, text: chunk.trim() }) + '\n');
      chunk = '';
    };
    for (const paragraph of section.split('\n\n')) {
      if (chunk.length + paragraph.length > MAX_CHARS) flush();
      chunk += paragraph + '\n\n';
    }
    flush();
  });
}
out.end();

On our docs crawl it wrote 1,306 chunks from 71 pages. Each looks like this:

{"id":"iguxHeoKT9-2-0","url":"https://docs.allscreenshots.com/","title":"Introduction | Allscreenshots Docs","heading":"Quick example","text":"## [Quick example](#quick-example)\n\nCapture a screenshot with a single API call:…"}

Index the chunks

Send each chunk's text to your embedding model and store the vector with url, title and heading, in whichever vector database or search index you use. At answer time, retrieve the closest chunks and pass them to the model with their URLs, so answers can cite the page they came from.

Keep it fresh

Docs change. Run the crawl again on a schedule (a cron job that calls POST /v1/crawls, or a webhook that starts your indexing when the crawl ends) and replace the chunks of each URL with the new ones. Compare the page's Markdown with the stored one first to skip pages that didn't change.

Tips

  • Whole body, not just the article. A crawl's Markdown is the whole page body, menus and footer included, so the same navigation text appears in every page. Drop repeated lines across pages, or use the JSON document's headings and text if you only need headings and visible text.
  • Main content only for a few pages. For a handful of important pages, a screenshot request with a Markdown output returns only the main content, without menus.
  • Large sites. One crawl captures up to 100 pages. Split a larger site into sections, one crawl per folder, such as /docs/ and /blog/.
  • Respectful crawling. The crawler obeys robots.txt as AllScreenshotsBot and waits at least 500 ms between requests to a host. Only crawl sites you're allowed to use this way.

Next steps

On this page