QUICK START GUIDE

Get up and running with Omnivore in just a few minutes. This guide will walk you through your first crawl, data extraction, and knowledge graph creation.

Step 1: Your First Crawl

Let's start with a simple website crawl.

BASH
COPY
001# Basic crawl with default settings
002omnivore crawl https://httpbin.org/html

Add More Options

BASH
COPY
001# Crawl with more workers and depth
002omnivore crawl https://example.com \
003  --workers 5 \
004  --depth 3 \
005  --delay 200 \
006  --output results.json
007
008# This will:
009# - Use 5 parallel workers
010# - Crawl up to 3 levels deep
011# - Wait 200ms between requests
012# - Save results to results.json

Understanding the Output

Step 2: Parse Specific Content

Extract specific data from HTML using custom parsing rules.

Create a Parser Configuration

BASH
COPY
001# Create a parser config file
002cat > parser-rules.yaml << EOF
003rules:
004  - name: "title"
005    selector: "title"
006    required: true
007  - name: "headings"
008    selector: "h1, h2, h3"
009    multiple: true
010  - name: "links"
011    selector: "a[href]"
012    attribute: "href"
013    multiple: true
014EOF

Parse with Custom Rules

BASH
COPY
001# Parse a specific HTML file
002omnivore parse index.html --rules parser-rules.yaml
003
004# Parse from URL and extract data
005curl -s https://example.com | omnivore parse --rules parser-rules.yaml

Step 3: Build Knowledge Graphs

Transform your crawled data into a knowledge graph for deeper analysis.

BASH
COPY
001# Build a knowledge graph from crawl results
002omnivore graph results.json --output knowledge-graph.db
003
004# Query the knowledge graph
005omnivore graph-query knowledge-graph.db \
006  --query "MATCH (n:Website)-[r:LINKS_TO]->(m:Website) RETURN n, r, m LIMIT 10"

Step 4: Analyze Git Repositories

Extract and analyze source code from Git repositories with intelligent filtering.

BASH
COPY
001# Analyze a GitHub repository
002omnivore git https://github.com/rust-lang/cargo --output cargo-analysis.txt
003
004# Analyze your local project
005omnivore git . --output my-project.txt
006
007# Extract specific file types
008omnivore git ./my-project --only "*.py,*.md" --output python-docs.txt
009
010# Exclude test files
011omnivore git ./my-project --exclude "**/tests/**" --output src-only.txt

Common Use Cases

1. News Site Monitoring

BASH
COPY
001# Monitor a news website for new articles
002omnivore crawl https://news.ycombinator.com \
003  --workers 3 \
004  --depth 2 \
005  --respect-robots \
006  --output hn-$(date +%Y%m%d).json
007
008# Extract article data with custom rules
009omnivore parse hn-*.json --rules news-extractor.yaml

2. E-commerce Product Scraping

BASH
COPY
001# Scrape product information (respect rate limits!)
002omnivore crawl https://example-store.com/products \
003  --workers 2 \
004  --delay 1000 \
005  --depth 3 \
006  --output products.json
007
008# Parse product data
009omnivore parse products.json --rules product-extractor.yaml

3. Research Paper Collection

BASH
COPY
001# Collect academic papers
002omnivore crawl https://arxiv.org/list/cs.AI/recent \
003  --workers 5 \
004  --depth 2 \
005  --output papers.json
006
007# Build citation network
008omnivore graph papers.json \
009  --schema academic-papers \
010  --output citation-network.db

4. Code Repository Analysis

BASH
COPY
001# Analyze open source project structure
002omnivore git https://github.com/facebook/react --output react-codebase.txt
003
004# Extract documentation from a project
005omnivore git ./my-project --only "*.md,*.rst,*.txt" --output docs.txt
006
007# Generate code report with JSON output
008omnivore git ./my-project --json --output codebase.json
009
010# Analyze multiple repositories for comparison
011for repo in "rust-lang/cargo" "golang/go" "nodejs/node"; do
012  omnivore git https://github.com/$repo --output "${repo//\//-}.txt"
013done

Configuration Examples

Basic Configuration File

Create ~/.config/omnivore/crawler.toml:

TOML
COPY
001[crawler]
002max_workers = 10
003max_depth = 5
004user_agent = "Omnivore/1.0 (+https://yoursite.com/bot)"
005respect_robots_txt = true
006timeout_ms = 30000
007
008[crawler.politeness]
009default_delay_ms = 100
010max_requests_per_second = 10.0
011backoff_multiplier = 2.0
012
013[parser]
014clean_text = true
015extract_metadata = true
016
017[storage]
018data_dir = "~/.local/share/omnivore"
019cache_size_mb = 1024
020compression = "zstd"

Domain-Specific Configuration

TOML
COPY
001# Site-specific settings
002[[crawler.site_configs]]
003domain = "example.com"
004delay_ms = 500
005max_requests_per_second = 2.0
006custom_headers = { "X-API-Key" = "your-key" }
007
008[[crawler.site_configs]]
009domain = "news.ycombinator.com"
010delay_ms = 1000
011max_depth = 3

Monitoring and Statistics

View Crawl Statistics

BASH
COPY
001# Current session stats
002omnivore stats
003
004# Historical stats
005omnivore stats --session crawl-20240809
006
007# Export stats to CSV
008omnivore stats --export stats.csv

Real-time Monitoring

BASH
COPY
001# Watch crawl progress
002omnivore crawl https://example.com --watch

Tips for Success

Ready to dive deeper? Check out the Configuration Guide to customize Omnivore for your specific needs.