Open-source, high-performance LLM gateway written in Rust. Connect to any LLM provider with a single API. Observability Included.

Traceloop Hub is a next-gen high-performance LLM gateway written in Rust that centralizes control and tracing of all LLM calls. It provides a unified OpenAI-compatible API for connecting to multiple LLM providers with observability built-in.

• Multi-Provider Support: OpenAI, Anthropic, Azure OpenAI, Google VertexAI, AWS Bedrock
• OpenAI Compatible API: Drop-in replacement for OpenAI API calls
• Two Deployment Modes:
  • YAML Mode: Simple static configuration with config files
  • Database Mode: Dynamic configuration with PostgreSQL and Management API
• Built-in Observability: OpenTelemetry tracing and Prometheus metrics
• High Performance: Written in Rust with async/await support
• Hot Reload: Dynamic configuration updates (Database mode)
• Pipeline System: Extensible request/response processing
• Unified Architecture: Single crate structure with integrated Management API

# YAML Mode (simple deployment) docker run -p 3000:3000 -v $(pwd)/config.yaml:/app/config.yaml traceloop/hub # Database Mode (with management API) docker run -p 3000:3000 -p 8080:8080 \ -e HUB_MODE=database \ -e DATABASE_URL=postgresql://user:pass@host:5432/db \ traceloop/hub

# Clone and build git clone https://github.com/traceloop/hub.git cd hub cargo build --release # YAML Mode ./target/release/hub # Database Mode  HUB_MODE=database DATABASE_URL=postgresql://user:pass@host:5432/db ./target/release/hub

The project uses a unified single-crate architecture:

hub/ ├── src/ # Main application code │ ├── main.rs # Application entry point │ ├── lib.rs # Library exports │ ├── config/ # Configuration management │ ├── providers/ # LLM provider implementations │ ├── models/ # Data models │ ├── pipelines/ # Request processing pipelines │ ├── routes.rs # HTTP routing │ ├── state.rs # Application state management │ ├── management/ # Management API (Database mode) │ │ ├── api/ # REST API endpoints │ │ ├── db/ # Database models and repositories │ │ ├── services/ # Business logic │ │ └── dto.rs # Data transfer objects │ └── types/ # Shared type definitions ├── migrations/ # Database migrations ├── helm/ # Kubernetes deployment ├── tests/ # Integration tests └── docs/ # Documentation 

Perfect for simple deployments and development environments.

Features:

• Static configuration via config.yaml
• No external dependencies
• Simple provider and model setup
• No management API
• Single port (3000)

Example config.yaml:

providers: - key: openai type: openai api_key: sk-... models: - key: gpt-4 type: gpt-4 provider: openai pipelines: - name: chat type: Chat plugins: - ModelRouter: models: [gpt-4]

Ideal for production environments requiring dynamic configuration.

Features:

• PostgreSQL-backed configuration
• REST Management API (/api/v1/management/*)
• Hot reload without restarts
• Configuration polling and synchronization
• SecretObject system for credential management
• Dual ports (3000 for Gateway, 8080 for Management)

Setup:

1. Set up PostgreSQL database
2. Run migrations: sqlx migrate run
3. Set environment variables:

   HUB_MODE=database DATABASE_URL=postgresql://user:pass@host:5432/db

Port 3000:

• POST /api/v1/chat/completions - Chat completions
• POST /api/v1/completions - Text completions
• POST /api/v1/embeddings - Text embeddings
• GET /health - Health check
• GET /metrics - Prometheus metrics
• GET /swagger-ui - OpenAPI documentation

Port 8080:

• GET /health - Management API health check
• GET|POST|PUT|DELETE /api/v1/management/providers - Provider management
• GET|POST|PUT|DELETE /api/v1/management/model-definitions - Model management
• GET|POST|PUT|DELETE /api/v1/management/pipelines - Pipeline management

providers: - key: openai type: openai api_key: sk-... # Optional organization_id: org-... base_url: https://api.openai.com/v1

providers: - key: anthropic type: anthropic api_key: sk-ant-...

providers: - key: azure type: azure api_key: your-key resource_name: your-resource api_version: "2023-05-15"

providers: - key: bedrock type: bedrock region: us-east-1 # Uses IAM roles or AWS credentials

providers: - key: vertexai type: vertexai project_id: your-project location: us-central1 # Uses service account JSON or API key

# YAML Mode helm install hub ./helm # Database Mode helm install hub ./helm \ --set management.enabled=true \ --set management.database.host=postgres \ --set management.database.existingSecret=postgres-secret

version: '3.8' services: # YAML Mode hub-yaml: image: traceloop/hub ports: - "3000:3000" volumes: - ./config.yaml:/app/config.yaml # Database Mode hub-database: image: traceloop/hub ports: - "3000:3000" - "8080:8080" environment: - HUB_MODE=database - DATABASE_URL=postgresql://hub:password@postgres:5432/hub depends_on: - postgres postgres: image: postgres:15 environment: - POSTGRES_DB=hub - POSTGRES_USER=hub - POSTGRES_PASSWORD=password

• Rust 1.87+
• PostgreSQL 12+ (for database mode)
• sqlx-cli (for migrations)

# Build OSS version cargo build # Test cargo test # Format cargo fmt # Lint cargo clippy # Run YAML mode cargo run # Run database mode HUB_MODE=database DATABASE_URL=postgresql://... cargo run

# Install sqlx-cli cargo install sqlx-cli --no-default-features --features postgres # Run migrations sqlx migrate run # Use setup script for complete setup ./scripts/setup-db.sh

The project follows a unified single-crate architecture:

• src/main.rs: Application entry point with mode detection
• src/lib.rs: Library exports for all modules
• src/config/: Configuration management and validation
• src/providers/: LLM provider implementations
• src/models/: Request/response data models
• src/pipelines/: Request processing pipelines
• src/management/: Management API (Database mode)
• src/types/: Shared type definitions
• src/state.rs: Thread-safe application state
• src/routes.rs: Dynamic HTTP routing

• Hot Reload: Configuration changes without restarts (Database mode)
• Atomic Updates: Thread-safe configuration updates
• Dynamic Routing: Pipeline-based request steering
• Comprehensive Testing: Integration tests with testcontainers
• OpenAPI Documentation: Auto-generated API specs

Configure in your pipeline:

pipelines: - name: traced-chat type: Chat plugins: - Tracing: endpoint: http://jaeger:14268/api/traces api_key: your-key - ModelRouter: models: [gpt-4]

Available at /metrics:

• Request counts and latencies
• Provider-specific metrics
• Error rates
• Active connections

┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Client App │───▶│ Traceloop Hub │───▶│ LLM Provider │ └─────────────────┘ │ │ │ (OpenAI, etc.) │ │ ┌─────────────┐ │ └─────────────────┘ │ │ Config Mode │ │ │ │ YAML | DB │ │ ┌─────────────────┐ │ └─────────────┘ │───▶│ Observability │ │ │ │ (OTel, Metrics) │ │ ┌─────────────┐ │ └─────────────────┘ │ │ Management │ │ │ │ API (DB) │ │ │ └─────────────┘ │ └──────────────────┘ 

Licensed under the Apache License, Version 2.0. See LICENSE for details.

We welcome contributions! Please see our Contributing Guide for details.

• 📖 Documentation
• 💬 Slack Community
• 🐛 Issue Tracker
• 📧 Email Support