docrawl

Docs‑focused crawler that converts documentation sites to clean Markdown, mirrors the site’s path structure, and adds useful metadata — while staying polite and secure.

Features

• Markdown output with YAML frontmatter (title, source_url, fetched_at, optional security_flags, quarantined)
• Path‑mirroring folder structure with index.md per directory
• Robots‑aware crawling (official matcher) and rate limiting
• Optional sitemap seeding from /sitemap.xml for broader coverage on --all
• Content extraction tuned for common docs frameworks with configurable selectors
• Same‑origin image download and Markdown relinking to local assets
• HTML/URL sanitization and prompt‑injection detection with quarantine
• Run manifest (manifest.json) and persistent visited‑URL cache

Install

Prerequisites: Rust (stable toolchain).

cargo build --release

Binary: target/release/docrawl

Library Usage (Embed in Your Tool)

You can use docrawl as a library and ship a single binary for your own CLI or service.

Add to Cargo.toml (using the Git URL until it’s published on crates.io):

[dependencies]
docrawl = { git = "https://github.com/neur0map/docrawl" }
url = "2"
tokio = { version = "1", features = ["full"] }

Minimal programmatic crawl:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cfg = docrawl::CrawlConfig {
        base_url: url::Url::parse("https://example.com/docs")?,
        output_dir: std::path::PathBuf::from("./out"),
        user_agent: format!("mytool/{}", env!("CARGO_PKG_VERSION")),
        max_depth: Some(2),
        rate_limit_per_sec: 8,
        follow_sitemaps: false,
        concurrency: 8,
        timeout: None,
        resume: false,
        config: docrawl::Config { host_only: true, skip_assets: true, ..Default::default() },
    };
    let stats = docrawl::crawl(cfg).await?;
    eprintln!("pages={} assets={}", stats.pages, stats.assets);
    Ok(())
}

Notes:

• docrawl::crawl uses the default on-disk layout (same as the CLI) and returns simple Stats.
• You own the UX: wire your flags to CrawlConfig and Config to keep one binary.

Quick Start

docrawl "https://example.com/docs"          # default depth=10
docrawl "https://example.com" --all         # full same‑origin site
docrawl "https://example.com" --depth 2     # start + links + their links
docrawl "https://example.com" -o ./export   # choose output root
docrawl "https://example.com/docs" --fast   # quick smoke test (no assets, higher rate)

CLI

• url (positional): Absolute starting URL. Only same‑origin links are followed by default.
• --all: Unlimited depth. Crawls the whole same‑origin site (still honors robots).
• --depth <n>: Max link depth from the start page. Default is 10 when --all isn’t set.
• -o, --output <path>: Output root; a host‑named folder is created inside.
• --concurrency <n>: Number of parallel fetch workers (bounded). Default: 8.
• --rate <n>: Global requests per second. Default: 2.
• --timeout-minutes <n>: Graceful shutdown after N minutes. The crawler stops scheduling new work, drains active tasks, writes the manifest, and exits.
• --resume: Resume from the previous run using the persisted frontier (see Cache). Useful after a timeout or manual stop.
• --host-only: Restrict scope to the exact origin (scheme+host+port).
• --external-assets: Allow downloading images from other domains.
• --allow-svg: Permit saving SVG images.
• --no-assets: Skip image downloads (fastest).
• --max-pages <n>: Stop after writing this many pages.
• --selector <CSS>: Preferred CSS selector for content (repeatable).
• --exclude <REGEX>: Exclude URLs matching the regex (repeatable).
• --fast: Preset for quick crawls: raises --rate/--concurrency to at least 16 and implies --no-assets.

Depth is link‑hop depth (not URL path depth). Hub‑style homepages can expose many pages even at small depths.

Configuration (optional)

Place docrawl.config.json in the working directory or in the output root. All fields are optional.

{
  "host_only": true,
  "external_assets": false,
  "allow_svg": false,
  "max_pages": 500,
  "selectors": [".theme-doc-markdown", ".article-content"],
  "exclude_patterns": ["\\.pdf$", "/private/"]
}

• host_only: Limit scope to exact origin (scheme+host+port). Default: same‑domain allowed.
• external_assets: Allow images from outside the site’s origin/domain. Default: false.
• allow_svg: Permit saving SVG images. Default: false.
• max_pages: Stop after writing this many pages.
• selectors: Preferred CSS selectors for the main content; tried before built‑ins.
• exclude_patterns: Regex patterns; if a canonical URL matches any, it’s skipped.

Behavior

• Output root defaults to CWD; inside it a folder named after the host is created (e.g., ./out/example.com/).
• Depth 0: only the start page. --all overrides --depth.
• Scope: same‑origin by default; exact origin if host_only is true.
• Robots: evaluated with the robotstxt matcher (agent‑specific allow/deny).
• Sitemaps: when --all is set, seeds from /sitemap.xml to front‑load coverage (non‑recursive for nested indexes).
• Extraction: tries common docs containers (Docusaurus, Sphinx, MkDocs) and falls back to <main>, <article>, <body>; you can supply selectors to override.
• Images: same‑origin images are downloaded into the site folder and Markdown links are rewritten to local relative paths. External or unsupported types are skipped unless enabled in config.

Security

• Sanitization (pre‑conversion): removes active/embedded elements (script, iframe, object, form, link, meta, etc.), normalizes and neutralizes risky URLs.
• Markdown scan (post‑conversion): detects likely prompt‑injection phrases and risky patterns; flags are recorded in frontmatter under security_flags.
• Quarantine: if any flags are detected, the page file is still created but the body is replaced with a short notice explaining why it was skipped; frontmatter includes quarantined: true.
• Asset safety: only safe image types (png, jpeg/jpg, gif, webp, bmp) are downloaded by default. SVG and data: images are disabled unless allowed via config.

Example frontmatter with flags:

---
title: Example
source_url: https://example.com/page
fetched_at: 2025-01-01T00:00:00Z
quarantined: true
security_flags:
  - llm_ignore_previous
  - javascript_link
---

Output

<output_root>/example.com/
├── index.md
├── guide/
│   └── index.md
├── assets/            # images saved under site paths
└── manifest.json

• Markdown files include YAML frontmatter. Images are referenced using relative paths.
• manifest.json records id, url, relative path, title, quarantined, security_flags, and timestamps for each saved page.

Cache

• Location: <output_root>/.docrawl_cache (sled key–value store).
• Purpose: persist the visited set across runs so re‑runs don’t re‑fetch unchanged URLs.
• If the cache cannot be opened (e.g., read‑only filesystem), crawling proceeds with in‑memory dedupe only; Markdown output still works.
• Frontier persistence for resume: the crawler stores pending URLs in the cache. Use --resume to continue from where you left off (e.g., after --timeout-minutes or Ctrl‑C). If you don’t use --resume, the persisted frontier is cleared at the start and new seeds are used.

Proxy & macOS Notes

• The HTTP client honors HTTP_PROXY/HTTPS_PROXY if present. If neither is set, system proxy detection is disabled to avoid rare macOS System Configuration panics.
• If you need a proxy, export it before running:
  • export HTTPS_PROXY=http://user:pass@host:port
  • export HTTP_PROXY=http://user:pass@host:port

Development and Testing

• Build: cargo build or cargo build --release
• Try a small depth first: docrawl "https://example.com/docs" --depth 2
• Verify robots compliance: disallowed paths should be skipped.
• Tune extraction with selectors if needed for your site.
• Confirm quarantined pages render a notice and include security_flags.

Roadmap

• Conditional GETs (ETag/Last‑Modified)
• Recursive sitemap index traversal
• CLI flags mirroring config fields
• Site‑specific selector presets