CONFIGURATION

Tune Omnivore for your workload using a simple TOML file.

Default Locations

• ~/.config/omnivore/crawler.toml
• /etc/omnivore/crawler.toml
• ./crawler.toml

Example

TOML
COPY
001[crawler]
002max_workers = 10
003max_depth = 5
004user_agent = "Omnivore/0.1 (+https://omnivore.pranavkarra.me)"
005respect_robots_txt = true
006
007[crawler.politeness]
008default_delay_ms = 100
009max_requests_per_second = 10.0
010backoff_multiplier = 2.0
011
012[storage]
013data_dir = "~/.local/share/omnivore"
014cache_size_mb = 512

Environment Variables

BASH
COPY
001# Override config path
002OMNIVORE_CONFIG=./crawler.toml omnivore crawl https://example.com