Haystack
Get Enterprise Support
Get Started
📢 Unified Haystack Ecosystem: One Name, One Product Family, One Look
Maintained by deepset

Integration: OpenRouter

Use the OpenRouter API for text generation models.

Authors
deepset
deepset-ai
GitHub Repo PyPI Package

Table of Contents

Overview

OpenRouterChatGenerator lets you call any LLMs available on OpenRouter, including:

  • OpenAI variants such as openai/gpt-4o
  • Anthropic’s claude-3.5-sonnet
  • Community-hosted open-source models (Llama 2, Mixtral, etc.)

For more information on models available via the OpenRouter API, see the OpenRouter docs.

In addition to basic chat completion, the component exposes OpenRouter-specific features:

  • Provider / model routing – choose fallback models or provider ordering with the generation_kwargs parameter.
  • Extra HTTP headers – add attribution or tracing headers via extra_headers.

In order to follow along with this guide, you’ll need a OpenRouter API key. Add it as an environment variable, OPENROUTER_API_KEY.

Installation

pip install openrouter-haystack

Usage

You can use OpenRouterChatGenerator as standalone, within a pipeline or with the Agent component.

Here’s an example of using it as a standalone component:

import os from haystack.dataclasses import ChatMessage from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator  os.environ["OPENROUTER_API_KEY"] = "YOUR_OPENROUTER_API_KEY"  client = OpenRouterChatGenerator() # defaults to openai/gpt-4o-mini response = client.run(  [ChatMessage.from_user("What are Agentic Pipelines? Be brief.")] ) print(response["replies"]) 
{'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text='The capital of Vietnam is Hanoi.')], _name=None, _meta={'model': 'openai/gpt-4o-mini', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 8, 'prompt_tokens': 13, 'total_tokens': 21, 'completion_tokens_details': CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None), 'prompt_tokens_details': PromptTokensDetails(audio_tokens=None, cached_tokens=0)}})]}

OpenRouterChatGenerator also support streaming responses if you pass a streaming callback:

import os from haystack.dataclasses import ChatMessage from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator  os.environ["OPENROUTER_API_KEY"] = "YOUR_OPENROUTER_API_KEY"  def show(chunk): # simple streaming callback  print(chunk.content, end="", flush=True)  client = OpenRouterChatGenerator(  model="openrouter/auto", # let OpenRouter pick a model  streaming_callback=show,  generation_kwargs={  "provider": {"sort": "throughput"}, # pick the fastest provider  } )  response = client.run([ChatMessage.from_user("Summarize RAG in two lines.")])  print (response)

License

openrouter-haystack is distributed under the terms of the Apache-2.0 license.