OMNIVORE DOCUMENTATION

Welcome to the comprehensive documentation for Omnivore, the Universal Rust Web Crawler & Knowledge Graph Builder. Get started quickly or dive deep into advanced features.

Installation

Get Omnivore installed on your system with Homebrew, Docker, or from source.

Quick Start

Start crawling websites in minutes with simple commands and examples.

[]

Configuration

Configure Omnivore for your specific crawling needs and use cases.

Crawler Engine

Understand how Omnivore's parallel crawler engine works under the hood.

Knowledge Graphs

Build and query knowledge graphs from your crawled data.

CLI Reference

Complete reference for all CLI commands and options.

Git Extraction

Extract and analyze source code from Git repositories with intelligent filtering.

WHAT'S INSIDE

This documentation covers everything you need to know about Omnivore:

Core Features

Parallel Crawling - Async/await with Tokio runtime for maximum performance
Git Repository Analysis - Extract and analyze source code with intelligent filtering
Smart Processing - AI-powered entity recognition and content classification
Knowledge Graphs - Build entity-relationship graphs automatically
Respectful Crawling - Built-in robots.txt compliance and rate limiting
Multiple Storage - Support for RocksDB, PostgreSQL, and graph databases

Architecture

omnivore-core - Core crawler and processing engine
omnivore-cli - Command-line interface with rich features
Plugin System - Extensible architecture for custom processors

Deployment Options

Homebrew - Easy installation on macOS and Linux
Docker - Containerized deployment with orchestration
From Source - Build and customize for your needs

GETTING HELP

Need Help?

Community Resources

Documentation

QUICK EXAMPLE

Here's a simple example to get you started:

BASH
001# Install Omnivore
002brew install omnivore
003
004# Crawl a website
005omnivore crawl https://example.com --workers 5 --depth 3
006
007# Extract code from a Git repository
008omnivore git https://github.com/user/repo --output code-analysis.txt
009
010# Build knowledge graph
011omnivore graph crawl-results.json --output knowledge-graph.db

WHAT'S NEXT?

New to Omnivore? Start with the Installation Guide
Ready to crawl? Jump to Quick Start
Want to customize? Check out Configuration
Building integrations? Explore the CLI and data formats in the docs

More documentation references are available on readthedocs.org.

Built with Rust. Developed by Pranav Karra