DEV Community

TOMOAKI ishihara
TOMOAKI ishihara

Posted on

AI-Powered Code Reviews with OpenRouter - Complete PR-Agent Setup Guide

#github #ai #codereview #openrouter

👋 Introduction

Code reviews take time, and large teams often struggle with PR . PR-Agent (now Qodo Merge) uses AI to automatically review pull requests, generate descriptions, and suggest improvements.

This guide shows you how to set up PR-Agent with OpenRouter, giving you cost-effective access to multiple AI models (GPT-4o, Claude, Gemini, etc.) through a unified API.

📌 Why OpenRouter?

Challenges with Direct OpenAI API

  • Single provider dependency (OpenAI models only)
  • Complex management of multiple provider APIs
  • Difficult to compare pricing across different model

OpenRouter Benefits

  • Unified Multi-Model API - Access GPT-4o, Claude, Gemini, Llama, etc. through one interface
  • Flexible Model Selection - Choose the right model for each task, optimizing cost-effectiveness
  • Early Access to Latest Models - Quick access to new models like GPT-5
  • Transparent Pricing - Clear pricing for each model (5.5% fee on credit purchases)

Prerequisites

  • GitHub repository (private or public)
  • OpenRouter account (free to create)
  • GitHub Actions permissions

Step 1: OpenRouter Setup

1.1 Account Creation and API Key

  1. Visit OpenRouter and create an account
  2. Purchase Credits (Important)
    • Go to the "Credits" page in your dashboard
    • Minimum purchase is $5 (5.5% fee applies to credit purchases)
    • Auto-recharge option available for convenience
    • ⚠️ Free Tier: New accounts receive a small free credit allowance, but additional purchases are needed for production use(FAQ: What free tier options exist?)
  3. In the dashboard, go to "API Keys" and generate a new key
  4. Copy and securely store the key

1.2 Available Models

OpenRouter provides access to models like:

openai/gpt-5 # Latest model (2025 release) openai/gpt-4o # Balanced performance anthropic/claude-3-opus-20240229 # High-quality analysis google/gemini-pro # Google's model meta-llama/llama-2-70b-chat # Open source
Enter fullscreen mode Exit fullscreen mode

Check pricing at OpenRouter Models.

Step 2: GitHub Actions Workflow Setup

2.1 Configure Secrets

In your GitHub repository:

  1. Go to Settings → Secrets and variables → Actions
  2. Click "New repository secret"
  3. Add the following secret: 
Name: OPENROUTER_API_KEY Value: or-xxxxxxxx (your OpenRouter API key)
Enter fullscreen mode Exit fullscreen mode

2.2 Create Workflow File

Create .github/workflows/pr_agent.yml

name: Code Review(OpenRouter) on: pull_request: pull_request_review_comment: types: [created] # Concurrency control concurrency: group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}-${{ github.event_name == 'pull_request_review_comment' && 'pr_comment' || 'pr' }} cancel-in-progress: ${{ github.event_name != 'pull_request_review_comment' }} jobs: pr_agent_job: if: | github.event.sender.type != 'Bot' && github.event_name == 'pull_request' && github.event.pull_request.changed_files < 50 runs-on: ubuntu-latest permissions: issues: write pull-requests: write contents: write id-token: write name: Run pr agent on every pull request steps: - name: PR Agent action step with OpenRouter id: pragent uses: qodo-ai/pr-agent@main env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENROUTER.KEY: ${{ secrets.OPENROUTER_API_KEY }} # OpenRouter API認証 github_action_config.auto_review: "true" # enable\disable auto review github_action_config.auto_describe: "true" # enable\disable auto describe github_action_config.auto_improve: "true" # enable\disable auto improve
Enter fullscreen mode Exit fullscreen mode

Step 3: Configuration File

Create .pr_agent.toml in your repository root.

[config] model = "openai/gpt-5" # I specified gpt-5, but an error occurred. The correct setting is "openai/gpt-5". fallback_models = ["openai/gpt-4o", "anthropic/claude-opus-4.1"] ai_timeout = 300 custom_model_max_tokens = 200000 # Critical! Errors without this response_language = "en-US" ignore_pr_title = ["^\\[Auto\\]", "^Auto"] ignore_pr_labels = ['invalid'] [ignore] glob = ['dist/**'] [github_action_config] # set as env var in .github/workflows/pr-agent.yaml #auto_review = true #auto_improve = true #auto_describe = true [pr_reviewer] extra_instructions = """\ (1) Act as a highly experienced software engineer (2) Provide thorough review of code, documents, and articles (3) Suggest concrete code snippets for improvement (4) **Never** comment on indentation, whitespace, blank lines, or other purely stylistic issues unless they change program semantics. (5) **Priority Review Areas - Check systematically:** - **Security**: Plaintext passwords, SQL injection, input validation, exception leaks - **Error Handling**: Bare except clauses, missing try-catch, silent error suppression - **Resource Management**: Missing context managers (with statements), unclosed connections/files - **Type Safety**: Missing type hints, incorrect type usage, unjustified Any types - **Performance**: Inefficient algorithms (O(n²) or worse), unnecessary loops, memory leaks - **Code Quality**: Magic numbers, unclear variable names, unused imports/variables - **API Design**: Missing input validation, no error responses, required field checks - **Architecture**: Single responsibility violations, tight coupling, global state usage (6) Focus on concrete, actionable issues with specific code examples and fix recommendations. """ num_code_suggestions = 5 inline_code_comments = true ask_and_reflect = true [pr_description] extra_instructions = "Generate clear and comprehensive descriptions" generate_ai_title = true add_original_user_description = false publish_description_as_comment = true [pr_code_suggestions] extra_instructions = "Provide actionable code suggestions with examples" commitable_code_suggestions = true demand_code_suggestions_self_review=true
Enter fullscreen mode Exit fullscreen mode

Step 4: Testing the Setup

4.1 Create Test PR

Create a PR with intentionally problematic code.

# Missing type hints def fizz_buzz(n): for i in range(1, n): if i % 15 == 0: print("FizzBuzz") elif i % 3 == 0: print("Fizz") elif i % 5 == 0: print("Buzz") else: printo(i) # Intentional bug  # Security issue def add_user(name, password): users = [] user = {"name": name, "password": password} # Plaintext password  users.append(user)
Enter fullscreen mode Exit fullscreen mode

4.2 Expected Output

PR-Agent will provide.

Auto-generated PR Description:

  • Summary of changes
  • File-by-file change details
  • Impact analysis

Code Review Comments:

  • Fix for printo(i) → print(i)
  • Type hint suggestions
  • Security warning about plaintext passwords
  • Performance improvement suggestions

Step 5: Advanced Configuration

5.1 Cost Optimization

Use different models for different tasks.

[config] model = "openai/gpt-4o-mini" # For lightweight tasks model_turbo = "openai/gpt-5" # For complex analysis fallback_models = ["anthropic/claude-3.7-sonnet", "google/gemini-2.5-pro"]
Enter fullscreen mode Exit fullscreen mode

5.2 Team-Specific Rules 

[pr_reviewer] extra_instructions = """\ (1) Company-specific checks: - Use structured logging for all log outputs - External API calls must have timeout settings - Database access must use transactions - Never hardcode secrets (API keys, passwords, etc.) """
Enter fullscreen mode Exit fullscreen mode

5.3 File Exclusions 

[ignore] glob = ['dist/**', '*.min.js', 'node_modules/**', '__pycache__/**'] [config] ignore_pr_title = ["^\\[Auto\\]", "^WIP:", "^Draft:"] ignore_pr_labels = ['work-in-progress', 'do-not-review']
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

Common Issues and Solutions

1. Model not recognized

Error: Model openai/gpt-5 is not defined in MAX_TOKENS Solution: Add custom_model_max_tokens = 200000 to config
Enter fullscreen mode Exit fullscreen mode

This error occurs when using a model that PR-Agent doesn't recognize. The models supported by PR-Agent are listed in this file. If your chosen model isn't in this list, you need to set the custom_model_max_tokens configuration.

2. Inconsistent language responses

[config] response_language = "en-US" # Be specific [pr_reviewer] extra_instructions = "Always respond in English and explain technical terms clearly"
Enter fullscreen mode Exit fullscreen mode

3. GitHub Actions permission errors

permissions: issues: write pull-requests: write contents: write # Add this line id-token: write
Enter fullscreen mode Exit fullscreen mode

4. Timeouts on large PRs

[config] ai_timeout = 600 # Extend to 10 minutes large_patch_policy = "clip" # Split large PRs
Enter fullscreen mode Exit fullscreen mode

Cost Management Best Practices

1. Appropriate Model Selection

Task Recommended Model Pricing (approx.)
Simple reviews gpt-4o-mini $0.15/1M tokens
Detailed analysis gpt-4o $2.5/1M tokens
Highest quality openai/gpt-5 $1.25/1M tokens

2. PR Size Limits 

if: github.event.pull_request.changed_files < 50 # Limit file count
Enter fullscreen mode Exit fullscreen mode

3. Control Automatic Execution 

[config] ignore_pr_labels = ['draft', 'wip', 'skip-ai-review'] ignore_pr_title = ["^\\[WIP\\]", "^Draft:", "^Auto"]
Enter fullscreen mode Exit fullscreen mode

Impact Measurement

Before/After Comparison Metrics

Quantitative Metrics:

  • Reduced code review time
  • Increased bug detection
  • Earlier security issue identification
  • Faster PR merge times

Qualitative Metrics:

  • Reduced reviewer burden
  • Improved code quality
  • Learning effect (developer skill improvement)

Real-World Results Example 

Before: Average review time 2 hours/PR After: Average review time 45 minutes/PR (62% reduction) Security issue detection: - Before: 2 issues/month average - After: 8 issues/month average (4x increase)
Enter fullscreen mode Exit fullscreen mode

📝 Conclusion

Setting up PR-Agent with OpenRouter provides these benefits.

✅ Cost Performance - Use multiple models through OpenRouter for cost optimization

✅ Quality Improvement - Consistent review standards reduce oversights

✅ Efficiency - Significant reduction in review time

✅ Learning Effect - Developers learn best practices from AI feedback

✅ Flexibility - Customizable for team-specific requirements

We recommend starting with a small project for testing and gradually expanding. The configuration file allows extensive customization to match your team's needs.

📚 References

Hope this guide helps improve your code review process! Feel free to share questions or suggestions in the comments.

Top comments (0)