stygian

High-performance web scraping toolkit for Rust — graph execution, anti-detection browser automation, and diagnostics-driven acquisition planning.

---

What is stygian?

Stygian is a monorepo with six complementary Rust crates for building robust, scalable web scraping systems:

• stygian-graph — Graph-based scraping engine (DAGs, AI extraction, distributed execution)
• stygian-browser — Anti-detection browser automation (stealth features, human behavior)
• stygian-proxy — Proxy pool management (multi-protocol, health checking, sticky sessions)
• stygian-charon — Diagnostics & policy planning (HAR forensics, provider classification, SLOs)
• stygian-mcp — MCP aggregator (LLM tool integration via JSON-RPC 2.0)
• stygian-plugin — Visual data extraction with MCP tools and browser extension

---

Quick Start

Install

# From crates.io
cargo install stygian-mcp --features extract

# Or add to Cargo.toml
stygian-graph = { version = "*", features = ["browser"] }
stygian-browser = "*"

Browser Automation

use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pool = BrowserPool::new(BrowserConfig::default()).await?;
    let handle = pool.acquire().await?;
    let mut page = handle.browser()?.new_page().await?;

    page.navigate(
        "https://example.com",
        WaitUntil::Selector("body".to_string()),
        Duration::from_secs(30),
    ).await?;

    println!("Page loaded: {} bytes", page.content().await?.len());
    handle.release().await;
    Ok(())
}

MCP Integration

Wire into Claude or VS Code via JSON-RPC 2.0 for LLM-powered tool access. See the book for setup.

---

Documentation

→ Complete docs and guides

Key sections:

• stygian-browser API
• stygian-graph architecture
• MCP tool reference
• Example pipelines

---

Architecture

Hexagonal design with zero I/O in domain layer:

Domain (business logic)
  ↓
Ports (traits)
  ↓
Adapters (HTTP, browser, AI, storage)

All crates follow this pattern for maximum testability and modularity.

---

Development

# Build & test
cargo test --workspace --all-features

# Strict linting
cargo clippy --workspace -- -D warnings

# Format check
cargo fmt --all -- --check

---

License

Dual-licensed under:

• AGPL-3.0 — open-source use
• Commercial License — proprietary use

See LICENSE-COMMERCIAL.md for details.

---

Acknowledgments

Built with chromiumoxide, petgraph, tokio, and reqwest.

---

Status: Active development | Rust 2024 edition | Linux + macOS

For feature requests or issues, see GitHub Issues.