Git Repository Code Extraction

Intelligently extract meaningful code from Git repositories. Automatically filters out noise like build artifacts, dependencies, and generated files while preserving the important source code you need for analysis or LLM processing.

Smart Filtering

Automatically excludes build artifacts, node_modules, and generated files

Gitignore Respect

Honors .gitignore files by default for accurate extraction

Flexible Output

Export as JSON, plain text, or preserve directory structure

Workflow Overview

The Git extraction tool operates in four distinct phases to intelligently extract code:

1. Source Acquisition

Determines if working with a remote URL or local directory. For remote repos, performs a shallow clone.

2. File Filtering

Applies intelligent filtering rules to identify important files, respecting .gitignore and user patterns.

3. Content Aggregation

Reads filtered files into memory, detecting and skipping binary files automatically.

4. Output & Cleanup

Formats content as requested (JSON, text, or directory) and cleans up temporary files.

Command Structure

Basic Syntax

BASH
001omnivore git <source> [flags]

The <source> parameter can be:

A remote repository URL: https://github.com/user/repo.git
A local directory path: . or ../my-project

Filtering Options

Include Patterns

Use --include to switch to whitelist mode, where only files matching specified patterns are kept:

BASH
001# Include only Rust source files and Cargo.toml
002omnivore git . --include "src/**/*.rs,Cargo.toml" --json
003
004# Include only JavaScript and TypeScript files
005omnivore git https://github.com/user/repo.git \
006  --include "**/*.js,**/*.ts,**/*.jsx,**/*.tsx" \
007  --output ./js-code

Exclude Patterns

Use --exclude to blacklist specific files or patterns:

BASH
001# Exclude test files and logs
002omnivore git . --exclude "*.log,**/*test*,**/*spec*" --txt
003
004# Exclude specific directories
005omnivore git . --exclude "target/,dist/,build/" --output ./clean-code

Gitignore Handling

BASH
001# Respect .gitignore (default behavior)
002omnivore git . --json
003
004# Ignore .gitignore files
005omnivore git . --no-gitignore --output ./all-files

Output Formats

JSON Output

Perfect for programmatic processing and integration with other tools:

BASH
001# Output as JSON to stdout
002omnivore git . --json
003
004# Save JSON to file
005omnivore git . --json > codebase.json
006
007# Pipe to jq for processing
008omnivore git . --json | jq '.[] | select(.path | contains("src"))'

JSON structure:

JSON
001[
002  {
003    "path": "src/main.rs",
004    "content": "fn main() {\n    println!(\"Hello, world!\");\n}"
005  },
006  {
007    "path": "Cargo.toml",
008    "content": "[package]\nname = \"omnivore\"\nversion = \"0.1.0\""
009  }
010]

Text Output

Ideal for feeding into LLMs or text processing pipelines:

BASH
001# Output as plain text with separators
002omnivore git . --txt
003
004# Pipe to an LLM
005omnivore git . --txt | llm "Analyze this codebase"
006
007# Save to file
008omnivore git . --txt > codebase.txt

Text format example:

TEXT
001---
002File: src/main.rs
003---
004fn main() {
005    println!("Hello, world!");
006}
007---
008File: Cargo.toml
009---
010[package]
011name = "omnivore"
012version = "0.1.0"

Directory Output

Preserves the original file structure in a new directory:

BASH
001# Extract to directory (default if no format specified)
002omnivore git https://github.com/user/repo.git --output ./extracted
003
004# Local repository extraction
005omnivore git . --output ../clean-copy
006
007# With filtering
008omnivore git . \
009  --include "src/**/*" \
010  --output ./src-only

Advanced Features

Binary File Handling

BASH
001# Skip binary files (default)
002omnivore git . --json
003
004# Include binary files (only works with --output)
005omnivore git . --allow-binary --output ./with-binaries
006
007# Binary files are automatically detected and skipped in JSON/TXT modes

Clone Depth Control

BASH
001# Shallow clone (default: depth 1)
002omnivore git https://github.com/user/repo.git --output ./code
003
004# Specific clone depth
005omnivore git https://github.com/user/repo.git --depth 10 --output ./code
006
007# Full clone (all history)
008omnivore git https://github.com/user/repo.git --depth 0 --output ./code

File Size Limits

BASH
001# Set maximum file size (10MB)
002omnivore git . --max-file-size 10485760 --json
003
004# Skip large generated files (1MB limit)
005omnivore git . --max-file-size 1048576 --txt

Temporary Clone Management

BASH
001# Keep temporary clone for debugging
002omnivore git https://github.com/user/repo.git \
003  --keep \
004  --output ./code
005
006# Temporary clones are deleted by default

Filtering Logic

Multi-Layer Filtering Funnel

Files pass through multiple filtering layers to determine inclusion:

1. System Excludes

Always ignores: .git/, .svn/, .hg/

2. .gitignore Processing

Respects all .gitignore files (unless --no-gitignore)

3. User Excludes

Applies --exclude patterns

4. Binary Detection

Skips binary files (unless --allow-binary)

5. User Includes

If specified, only keeps files matching --include patterns

Common Use Cases

Prepare Codebase for LLM Analysis

BASH
001# Extract only source code as text
002omnivore git . \
003  --exclude "**/*test*,**/*spec*,docs/" \
004  --txt | llm "Review this code for security issues"
005
006# Create JSON for structured analysis
007omnivore git . \
008  --include "src/**/*.py,*.py" \
009  --json | python analyze_code.py

Repository Migration

BASH
001# Clone and clean a repository
002omnivore git https://github.com/old/repo.git \
003  --exclude "node_modules/,dist/,*.log" \
004  --output ./clean-repo
005
006# Extract only source files
007omnivore git . \
008  --include "src/**/*,package.json,README.md" \
009  --output ../migration/

Code Documentation

BASH
001# Extract code for documentation
002omnivore git . \
003  --include "**/*.rs" \
004  --exclude "target/,tests/" \
005  --json | jq -r '.[].content' > all_code.txt
006
007# Generate file listing
008omnivore git . --json | jq -r '.[].path' > file_list.txt

Multi-Repository Analysis

BASH
001#!/bin/bash
002# analyze_repos.sh
003
004repos=(
005  "https://github.com/user/repo1.git"
006  "https://github.com/user/repo2.git"
007  "https://github.com/user/repo3.git"
008)
009
010for repo in "${repos[@]}"; do
011  name=$(basename "$repo" .git)
012  omnivore git "$repo" \
013    --include "**/*.go" \
014    --json > "${name}.json"
015done
016
017# Combine all JSON files
018jq -s 'add' *.json > combined.json

Error Handling

Authentication

The tool relies on system-wide Git credentials. If authentication fails:

BASH
001# For SSH URLs, ensure SSH keys are configured
002ssh-add ~/.ssh/id_rsa
003
004# For HTTPS URLs, configure credential helper
005git config --global credential.helper cache
006
007# Error message example:
008# "Authentication failed. Ensure your Git credentials are configured correctly."

Invalid Repository

BASH
001# Local directory must be a Git repository
002omnivore git /path/to/non-git-dir --json
003# Error: '/path/to/non-git-dir' is not a Git repository
004
005# Verify repository
006ls -la .git/  # Should exist for local repos

Conflicting Flags

BASH
001# Cannot use multiple output formats
002omnivore git . --json --txt
003# Error: Cannot use --json and --txt together
004
005# Cannot use format flags with --output
006omnivore git . --json --output ./dir
007# Error: Cannot use --json with --output

Empty Results

BASH
001# If filtering results in no files
002omnivore git . --include "*.xyz" --json
003# Output: []
004
005omnivore git . --include "*.xyz" --txt
006# Output: (empty)
007
008omnivore git . --include "*.xyz" --output ./empty
009# Creates empty directory

Performance Considerations

Large Repositories

Use shallow clones (--depth 1) for faster operations
Set file size limits to skip large generated files
Use specific include patterns to reduce processing

BASH
001# Optimize for large repos
002omnivore git https://github.com/large/repo.git \
003  --depth 1 \
004  --include "src/**/*.java" \
005  --max-file-size 1048576 \
006  --output ./extracted

Memory Management

For very large codebases, consider processing in chunks:

BASH
001# Process different parts separately
002omnivore git . --include "frontend/**/*" --output ./frontend-code
003omnivore git . --include "backend/**/*" --output ./backend-code
004omnivore git . --include "shared/**/*" --output ./shared-code

Security Considerations

Security Notes

Symbolic links are not followed to prevent security risks
Binary files are detected and excluded by default
File paths are sanitized to prevent directory traversal
Temporary clones are created in secure system temp directories
Git credentials are never stored or logged by the tool

Best Practices

Recommendations

Always use shallow clones for remote repositories unless full history is needed
Set appropriate file size limits to avoid memory issues
Use include patterns for specific file types when possible
Test patterns on small repositories first
Respect .gitignore files unless you have a specific reason not to
Use JSON output for programmatic processing
Use text output for LLM consumption
Keep binary detection enabled for text/JSON outputs

Troubleshooting

Common Issues and Solutions

Network Timeouts

BASH
001# Increase Git timeout
002git config --global http.postBuffer 524288000
003git config --global http.timeout 600
004
005# Use SSH instead of HTTPS
006omnivore git [email protected]:user/repo.git --output ./code

Character Encoding Issues

BASH
001# Files with non-UTF8 encoding are skipped
002# Warning: Skipped file.txt (invalid UTF-8 encoding)
003
004# Convert files to UTF-8 first if needed
005iconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt

Large File Warnings

BASH
001# Set appropriate size limits
002omnivore git . --max-file-size 5242880 --json  # 5MB limit
003
004# Or exclude large file patterns
005omnivore git . --exclude "*.sqlite,*.db,*.zip" --json

Integration Examples

GitHub Actions Workflow

YAML
001name: Code Analysis
002on: [push, pull_request]
003
004jobs:
005  analyze:
006    runs-on: ubuntu-latest
007    steps:
008      - uses: actions/checkout@v2
009
010      - name: Install Omnivore
011        run: |
012          curl -sSfL https://raw.githubusercontent.com/Pranav-Karra-3301/omnivore/master/install.sh | sh
013
014      - name: Extract Code
015        run: |
016          omnivore git . \
017            --include "src/**/*.rs" \
018            --json > codebase.json
019
020      - name: Analyze
021        run: |
022          python analyze.py codebase.json

Pre-commit Hook

BASH
001#!/bin/sh
002# .git/hooks/pre-commit
003
004# Extract and check code size
005size=$(omnivore git . --json | wc -c)
006max_size=10485760  # 10MB
007
008if [ $size -gt $max_size ]; then
009  echo "Error: Codebase too large ($size bytes)"
010  exit 1
011fi
012
013echo "Codebase size: $size bytes"

Built with Rust. Developed by Pranav Karra