CRAWLER ENGINE

Omnivore's crawler engine is a high-performance, parallel web crawler built with Rust and Tokio. It features intelligent URL frontier management, politeness controls, and automatic rate limiting.

ARCHITECTURE OVERVIEW

Core Components

1. URL Frontier

The frontier manages the queue of URLs to be crawled. It implements priority queuing, deduplication, and depth tracking:

• Priority Queue: URLs are prioritized based on domain importance and depth
• Deduplication: Bloom filters prevent crawling the same URL multiple times
• Depth Control: Configurable maximum depth to prevent infinite crawling
• Domain Buckets: URLs grouped by domain for politeness enforcement

2. Worker Pool

Parallel workers powered by Tokio's async runtime handle concurrent page fetching:

RUST
COPY
001// Worker configuration
002pub struct WorkerConfig {
003    pub num_workers: usize,      // Number of parallel workers
004    pub timeout: Duration,        // Request timeout per page
005    pub retry_limit: u32,        // Max retries for failed requests
006    pub user_agent: String,      // Custom user agent string
007}

3. Politeness Engine

Ensures respectful crawling behavior across domains:

• Robots.txt Parser: Automatic fetching and caching of robots.txt files
• Rate Limiting: Per-domain request delays (default: 1 second)
• Crawl-Delay: Respects crawl-delay directives from robots.txt
• Concurrent Limits: Maximum concurrent requests per domain

4. Content Processor

Extracts and processes content from fetched pages:

• HTML Parsing: Fast HTML parsing with scraper crate
• Link Extraction: Automatic discovery of internal and external links
• Content Extraction: Text, metadata, and structured data extraction
• JavaScript Support: Optional headless browser for JS-heavy sites

CONFIGURATION

Basic Configuration

TOML
COPY
001[crawler]
002# Number of parallel workers
003workers = 10
004
005# Maximum crawl depth
006max_depth = 3
007
008# Request timeout in seconds
009timeout = 30
010
011# User agent string
012user_agent = "Omnivore/0.1.0"
013
014# Respect robots.txt
015respect_robots = true
016
017# Delay between requests to same domain (ms)
018politeness_delay = 1000

Advanced Configuration

TOML
COPY
001[crawler.advanced]
002# URL frontier settings
003frontier_size = 100000
004priority_queue = true
005bloom_filter_size = 10000000
006
007# Network settings
008max_redirects = 5
009connect_timeout = 10
010dns_cache_size = 1000
011
012# Content settings
013max_page_size = 10485760  # 10MB
014accepted_content_types = ["text/html", "application/xhtml+xml"]
015extract_links = true
016extract_metadata = true
017
018# Storage settings
019checkpoint_interval = 1000
020compression = "gzip"

USAGE EXAMPLES

Basic Crawl

BASH
COPY
001# Simple crawl with default settings
002omnivore crawl https://example.com
003
004# Crawl with custom workers and depth
005omnivore crawl https://example.com --workers 20 --depth 5
006
007# Crawl with output to file
008omnivore crawl https://example.com --output results.json

Advanced Crawl

BASH
COPY
001# Crawl with custom configuration
002omnivore crawl https://example.com --config crawler.toml
003
004# Crawl with filtering
005omnivore crawl https://example.com \
006  --include-pattern "*/blog/*" \
007  --exclude-pattern "*/admin/*"
008
009# Crawl with custom headers
010omnivore crawl https://example.com \
011  --header "Authorization: Bearer token" \
012  --header "Accept-Language: en-US"

Programmatic Usage

RUST
COPY
001use omnivore::{Crawler, CrawlerConfig};
002
003#[tokio::main]
004async fn main() -> Result<(), Box<dyn std::error::Error>> {
005    // Create crawler configuration
006    let config = CrawlerConfig::builder()
007        .workers(10)
008        .max_depth(3)
009        .politeness_delay(1000)
010        .build();
011
012    // Initialize crawler
013    let mut crawler = Crawler::new(config);
014
015    // Set callbacks
016    crawler.on_page_crawled(|page| {
017        println!("Crawled: {}", page.url);
018    });
019
020    // Start crawling
021    let results = crawler
022        .crawl("https://example.com")
023        .await?;
024
025    println!("Crawled {} pages", results.len());
026    Ok(())
027}

PERFORMANCE TUNING

Worker Optimization

The number of workers should be tuned based on your system resources and target sites:

• CPU-bound: Set workers to 2x CPU cores for optimal performance
• Network-bound: Can increase workers up to 100+ for slow sites
• Memory: Each worker consumes ~10-50MB depending on page size

BASH
COPY
001# For CPU-bound processing (heavy parsing)
002omnivore crawl https://example.com --workers 16
003
004# For network-bound crawling (slow sites)
005omnivore crawl https://slow-site.com --workers 100
006
007# Memory-constrained environment
008omnivore crawl https://example.com --workers 5 --max-page-size 1048576

BEST PRACTICES

TROUBLESHOOTING

Common Issues

High Memory Usage

BASH
COPY
001# Reduce workers and page size
002omnivore crawl https://example.com \
003  --workers 5 \
004  --max-page-size 1048576 \
005  --frontier-size 10000

Slow Crawling

BASH
COPY
001# Increase workers and reduce delays
002omnivore crawl https://example.com \
003  --workers 50 \
004  --politeness-delay 500 \
005  --timeout 10

Blocked by Site

BASH
COPY
001# Use custom user agent and headers
002omnivore crawl https://example.com \
003  --user-agent "Mozilla/5.0..." \
004  --header "Accept: text/html" \
005  --crawl-delay 3000