argiope

A high-performance web crawler written for broken-link detection and image downloading.

Overview: argiope is a fast, lightweight web crawler designed for broken-link detection, website archiving, and batch image downloading. It features intelligent HTML scanning, URL normalization, and sophisticated report generation. Built with zero external dependencies and compiled to a single static binary.

Key Features:

• Crawl websites and detect broken links (4xx/5xx/timeout errors)
• Generate comprehensive reports in text, Markdown, or self-contained HTML format
• Download images from web pages to organized directory structures
• Generate portable HTML browsing pages for downloaded image libraries with library.html landing page, nested index.html navigation, and per-folder reader.html viewers
• Specialized support for downloading manga chapters from MangaFox (fanfox.net) by title with optional chapter range filtering
• BFS (breadth-first search) traversal with configurable crawl depth, request timeouts, and rate limiting
• Domain-restricted crawling with automatic same-origin security checks
• Lightweight HTML scanner for precise link and image extraction from complex pages
• Intelligent URL normalization and relative-to-absolute URL resolution
• Parallel crawling support for improved performance on large sites
• Zero external dependencies — uses only Zig's standard library
• Single static binary — no runtime, no installation hassles

Commands:

• check <url>: Crawl a website and generate a detailed broken-link report with timing statistics
• images <url>: Download all images from a website or manga chapters from MangaFox to an organized structure
• library <dir>: Generate or regenerate HTML browser for existing image directories

Usage Examples:

• argiope check https://example.com --depth 5 — Check site for broken links up to 5 levels deep
• argiope images https://example.com/gallery -o ./images — Archive gallery images with HTML browser
• argiope images https://fanfox.net/manga/title --chapters 1-50 — Download manga chapters 1-50
• argiope check https://example.com --report report.html --report-format html — Generate HTML report for CI pipelines
• argiope images https://example.com -o ./archive --parallel --depth 3 — Fast parallel crawling and download

Report Features: Reports include detailed statistics: total URLs checked, OK count, broken count, error count, internal vs external split, and timing information (total time, average/min/max response times). HTML reports are self-contained with inline CSS and styled with status badges.

HTML Browser: The generated portable HTML browser features light/dark/system theme modes with localStorage-backed preferences, relative links for offline browsing, percent-encoded filenames for local file access, and thumbnail galleries with ordered prev/next navigation. Works with generic image collections and deep MangaFox chapter trees alike.

MangaFox Features:

• Automatic chapter detection via RSS feeds with fallback to HTML parsing
• Numeric chapter ordering (1, 2, 10, 11, 100) not alphabetic
• Full support for decimal chapter numbers (5.5, 100.1) with correct sorting
• Chapter range filtering via --chapters N-M flag
• Organized output: [manga-title]/[chapter]/[page].jpg
• Verbose mode shows detailed chapter discovery information

Performance & Reliability: Configurable request timeouts, delay between requests, and concurrent crawling options ensure efficient resource usage and reliable operation on rate-limited sites. Automatic redirect following, response size limiting, and detailed error reporting make it production-ready for CI pipelines and large-scale archiving. Perfect for website archiving, CI/CD integration, and offline browsing.