Knowledge Graphs

Transform crawled web data into structured knowledge graphs. Extract entities, discover relationships, and build queryable graph databases for advanced insights and analysis.

Core Concepts

What is a Knowledge Graph?

A knowledge graph is a structured representation of information where:

• Nodes (Entities): Represent distinct objects, concepts, or data points
• Edges (Relationships): Define connections and associations between entities
• Properties: Attributes that describe entities and relationships

Graph Structure

JSON
COPY
001{
002  "nodes": [
003    {
004      "id": "person_1",
005      "type": "Person",
006      "properties": {
007        "name": "John Doe",
008        "title": "CEO",
009        "email": "[email protected]"
010      }
011    },
012    {
013      "id": "company_1",
014      "type": "Company",
015      "properties": {
016        "name": "Example Corp",
017        "industry": "Technology",
018        "founded": "2010"
019      }
020    }
021  ],
022  "edges": [
023    {
024      "source": "person_1",
025      "target": "company_1",
026      "type": "WORKS_FOR",
027      "properties": {
028        "since": "2015",
029        "role": "Chief Executive Officer"
030      }
031    }
032  ]
033}

Entity Extraction

Supported Entity Types

Extraction Configuration

TOML
COPY
001[knowledge_graph.extraction]
002# Entity extraction settings
003extract_entities = true
004entity_types = ["person", "organization", "location", "product"]
005
006# Confidence threshold (0-1)
007min_confidence = 0.7
008
009# Custom patterns
010[knowledge_graph.extraction.patterns]
011email = "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"
012phone = "\+?[1-9]\d{1,14}"
013social_handle = "@[A-Za-z0-9_]+"
014
015# NER model settings
016[knowledge_graph.extraction.ner]
017model = "en_core_web_lg"  # spaCy model
018batch_size = 100
019max_length = 10000

Relationship Discovery

Automatic Relationship Detection

Omnivore automatically discovers relationships based on:

• Co-occurrence: Entities appearing in the same context
• Syntactic patterns: Grammar-based relationship extraction
• Semantic analysis: Understanding meaning and context
• Link analysis: Hyperlink and reference relationships

Relationship Types

RUST
COPY
001// Common relationship types
002enum RelationType {
003    // Organizational
004    WorksFor,
005    ManagesEmployeeOf,
006    ParentCompany,
007    Subsidiary,
008
009    // Personal
010    Knows,
011    RelatedTo,
012    ContactOf,
013
014    // Locational
015    LocatedIn,
016    HeadquarteredIn,
017    OperatesIn,
018
019    // Transactional
020    Customer,
021    Supplier,
022    Partner,
023    Competitor,
024
025    // Content
026    Mentions,
027    References,
028    LinksTo,
029    CitedBy,
030}

Building Knowledge Graphs

Basic Graph Construction

BASH
COPY
001# Build graph from crawled data
002omnivore graph build crawl-results.json --output knowledge-graph.json
003
004# Specify entity types to extract
005omnivore graph build crawl-results.json \
006  --entities person,organization,location \
007  --output knowledge-graph.json
008
009# Set confidence threshold
010omnivore graph build crawl-results.json \
011  --min-confidence 0.8 \
012  --output knowledge-graph.json

Advanced Configuration

BASH
COPY
001# Use custom extraction config
002omnivore graph build crawl-results.json \
003  --config graph-config.toml \
004  --output knowledge-graph.json
005
006# Parallel processing
007omnivore graph build crawl-results.json \
008  --workers 10 \
009  --batch-size 1000 \
010  --output knowledge-graph.json
011
012# Include metadata
013omnivore graph build crawl-results.json \
014  --include-metadata \
015  --include-timestamps \
016  --output knowledge-graph.json

Programmatic API

RUST
COPY
001use omnivore::{KnowledgeGraph, GraphBuilder, EntityExtractor};
002
003#[tokio::main]
004async fn main() -> Result<(), Box<dyn std::error::Error>> {
005    // Load crawled data
006    let crawl_data = load_crawl_results("results.json")?;
007
008    // Configure entity extractor
009    let extractor = EntityExtractor::builder()
010        .add_entity_type(EntityType::Person)
011        .add_entity_type(EntityType::Organization)
012        .min_confidence(0.75)
013        .build();
014
015    // Build knowledge graph
016    let graph_builder = GraphBuilder::new(extractor);
017    let knowledge_graph = graph_builder
018        .process(crawl_data)
019        .extract_entities()
020        .discover_relationships()
021        .build()
022        .await?;
023
024    // Query the graph
025    let ceo_nodes = knowledge_graph
026        .query()
027        .nodes()
028        .filter_type(EntityType::Person)
029        .filter_property("title", "CEO")
030        .execute()?;
031
032    println!("Found {} CEOs", ceo_nodes.len());
033
034    // Export to different formats
035    knowledge_graph.export_json("graph.json")?;
036    knowledge_graph.export_graphml("graph.graphml")?;
037    knowledge_graph.export_neo4j("bolt://localhost:7687")?;
038
039    Ok(())
040}

Graph Storage & Export

Storage Backends

1. Neo4j Export

BASH
COPY
001# Export to Neo4j database
002omnivore graph export knowledge-graph.json \
003  --format neo4j \
004  --uri bolt://localhost:7687 \
005  --username neo4j \
006  --password password
007
008# Batch import for large graphs
009omnivore graph export knowledge-graph.json \
010  --format neo4j-import \
011  --output-dir ./neo4j-import/

2. ArangoDB Export

BASH
COPY
001# Export to ArangoDB
002omnivore graph export knowledge-graph.json \
003  --format arangodb \
004  --url http://localhost:8529 \
005  --database knowledge \
006  --collection entities

3. GraphML Format

BASH
COPY
001# Export to GraphML (compatible with most graph tools)
002omnivore graph export knowledge-graph.json \
003  --format graphml \
004  --output knowledge-graph.graphml

4. JSON-LD Format

BASH
COPY
001# Export as JSON-LD for semantic web
002omnivore graph export knowledge-graph.json \
003  --format jsonld \
004  --context schema.org \
005  --output knowledge-graph.jsonld

Querying Knowledge Graphs

Built-in Query Language

BASH
COPY
001# Find all entities of a specific type
002omnivore graph query knowledge-graph.json \
003  --type Person \
004  --limit 10
005
006# Search by property
007omnivore graph query knowledge-graph.json \
008  --property "title=CEO" \
009  --type Person
010
011# Find relationships
012omnivore graph query knowledge-graph.json \
013  --relationship "WORKS_FOR" \
014  --source-type Person \
015  --target-type Company

Graph Traversal

BASH
COPY
001# Find connected entities (1 hop)
002omnivore graph traverse knowledge-graph.json \
003  --start "person_123" \
004  --depth 1
005
006# Find paths between entities
007omnivore graph path knowledge-graph.json \
008  --from "person_123" \
009  --to "company_456" \
010  --max-depth 5
011
012# Find clusters/communities
013omnivore graph analyze knowledge-graph.json \
014  --algorithm community-detection \
015  --output communities.json

Analysis & Insights

Graph Metrics

BASH
COPY
001# Calculate graph statistics
002omnivore graph stats knowledge-graph.json
003
004# Output:
005# Nodes: 10,543
006# Edges: 45,231
007# Average degree: 8.58
008# Clustering coefficient: 0.234
009# Connected components: 12

Centrality Analysis

BASH
COPY
001# Find most important nodes
002omnivore graph analyze knowledge-graph.json \
003  --algorithm pagerank \
004  --output pagerank-scores.json
005
006# Betweenness centrality
007omnivore graph analyze knowledge-graph.json \
008  --algorithm betweenness \
009  --output betweenness-scores.json
010
011# Degree centrality
012omnivore graph analyze knowledge-graph.json \
013  --algorithm degree \
014  --output degree-scores.json

Pattern Detection

BASH
COPY
001# Find patterns and motifs
002omnivore graph patterns knowledge-graph.json \
003  --min-support 0.1 \
004  --max-size 5 \
005  --output patterns.json
006
007# Detect anomalies
008omnivore graph anomalies knowledge-graph.json \
009  --algorithm isolation-forest \
010  --threshold 0.9 \
011  --output anomalies.json

Visualization

Export for Visualization Tools

BASH
COPY
001# Gephi format
002omnivore graph export knowledge-graph.json \
003  --format gexf \
004  --output graph.gexf
005
006# Cytoscape format
007omnivore graph export knowledge-graph.json \
008  --format cyjs \
009  --output graph.cyjs
010
011# D3.js format
012omnivore graph export knowledge-graph.json \
013  --format d3 \
014  --output graph-d3.json

Built-in Visualization Server

BASH
COPY
001# Start visualization server
002omnivore graph serve knowledge-graph.json \
003  --port 8080 \
004  --host 0.0.0.0
005
006# With custom layout
007omnivore graph serve knowledge-graph.json \
008  --layout force-directed \
009  --port 8080

Use Cases

Performance Considerations

Scaling Large Graphs

• Streaming Processing: Process large datasets in chunks
• Distributed Computing: Use multiple workers for parallel processing
• Index Optimization: Create indexes on frequently queried properties
• Graph Partitioning: Split large graphs into manageable subgraphs

TOML
COPY
001[knowledge_graph.performance]
002# Streaming settings
003stream_processing = true
004chunk_size = 10000
005max_memory = "4GB"
006
007# Parallel processing
008workers = 8
009batch_size = 1000
010
011# Caching
012enable_cache = true
013cache_size = "1GB"
014
015# Optimization
016create_indexes = true
017optimize_queries = true

Best Practices