⇄ 01 | Multi-Vendor LLM Gateway & Routing Engine (routerio)

"Every agentic enterprise starts here — infrastructure before intelligence."

📋 The Problem

As enterprises deploy hundreds of agents, they face severe operational challenges: * Vendor Lock-in: Hardcoding client libraries for a single provider (e.g. OpenAI) leaves the enterprise vulnerable to provider outages, price increases, or model deprecations. * Unpredictable Spend: Directing simple classification tasks to premium models (e.g., GPT-4o, Claude 3.5 Sonnet) instead of lightweight models (Gemini 2.5 Flash, GPT-4o-Mini) wastes up to 90% of LLM budgets. * Zero Visibility: No central audit log of prompt/response tokens, latencies, semantic hits, or provider health.

📐 The Solution: `routerio`

routerio is an intelligent, high-performance, and resilient LLM routing gateway that sits between all enterprise applications/agents and upstream model providers (Anthropic, OpenAI, Google Gemini, and local Ollama).

Exposing a unified, OpenAI-compatible /v1/chat/completions API, routerio automatically optimizes model selection in real-time based on prompt complexity, user policies, and active upstream health monitoring.

graph TD
    Client[Agent / Client Application] -->|v1/chat/completions| GW[router.io Gateway]

    subgraph router_core ["router.io Core"]
        GW --> CacheCheck{Semantic Cache Lookup}
        CacheCheck -->|Hit > 85% Similarity| ReturnCache[Instant Cache Response]
        CacheCheck -->|Miss| Classifier[Complexity Classifier]

        Classifier -->|Low/Med/High| PolicyEngine[Policy Optimization Engine]
        PolicyEngine -->|Cost / Perf / Latency| FilterCompliance[Compliance Filter]
        FilterCompliance -->|Global / US-Only| FilterBreakers[Circuit Breakers Status]

        FilterBreakers -->|Active Provider Selected| Exec[Execution Layer]
        FilterBreakers -->|If Tripped / Fails| Fallback[Failover Fallback Chain]

        Exec -.-> Log[SQLite Audit Logger]
        Fallback -.-> Log
    end

    ReturnCache --> Client
    Exec --> Upstream((Upstream LLM Providers))
    Fallback --> Upstream

    Upstream -->|Response| Client

    style router_core fill:#f9f9f9,stroke:#333,stroke-width:1px

⚡ Core Agentic Patterns & Capabilities

1. Dynamic Complexity Classification

The gateway inspects incoming prompts for lexical indicators, token densities, and code syntax block structures to classify task complexity (Low, Medium, High). * Low: Routed automatically to cost-effective models (e.g., Gemini 2.5 Flash). * High: Escalated immediately to premium reasoning engines (e.g., Claude 3.5 Sonnet).

2. Multi-Policy Engine

Clients can request specific optimization parameters in their request payloads: * cost: Aggressively prioritizes lightweight tiers. * performance: Directs queries strictly to premier frontier models. * latency: Dynamically calculates the fastest responding upstream model.

3. Dependency-Free Semantic Cache

An elegant local Cosine Similarity lookup matches incoming queries against historic logs. If a matching query is found above an 85% similarity threshold, it is returned instantly with 0ms latency and $0.00 spend.

4. Circuit Breaker Pattern

Continuously monitors upstream status. If a provider encounters 3 consecutive timeouts or error spikes, its circuit trips to OPEN for 60 seconds, preventing user latency spikes by skipping the provider instantly.

5. Governance Console

A premium glassmorphic custom dark theme web interface (HTML5, Vanilla CSS3, Chart.js) showing throughput, active circuit breaker statuses, cost savings, and historical traces.

🛠 Tech Stack

Backend: Python 3.11+, FastAPI, Uvicorn
Database: SQLite (audit trails, circuit breaker state, semantic cache)
Integrations: OpenAI, Anthropic, Google Generative AI, Ollama
Visuals: Chart.js, Tailwind-free Premium custom glassmorphism stylesheet

🔗 Repository

Explore the complete codebase and deployment instructions: * GitHub Repository: https://github.com/mailtotanvir/routerio