DEV Community

Cover image for How to Build an AI-Powered Code Review Bot with GitHub Action, Slack, and LLMs
cuongnp
cuongnp

Posted on

How to Build an AI-Powered Code Review Bot with GitHub Action, Slack, and LLMs

#ai #devops #productivity #opensource

🚀 How to Build an AI-Powered Code Review Bot with GitHub, Slack, and LLMs

Are you tired of manual code reviews? Want to supercharge your workflow with AI? In this post, I’ll show you how I built a fully automated code review system using GitHub Actions, Slack, and a Large Language Model (LLM)—and how you can do it too! It's just a demo, you change customize by your case.

🧩 What You’ll Build

  • Automatic code reviews for every pull request using AI
  • Slack notifications with interactive Approve/Request Changes buttons (I just implemented for Approve feature)
  • Seamless integration between GitHub Action, Slack, and your backend (I choosed Nodejs, too simple)
  • Open source, extensible, and team-friendly!

🛠️ Tech Stack

  • GitHub Actions for automation
  • Node.js/Express backend for Slack interactivity
  • Slack App for notifications and actions (You should create a trial workspace, and 2 accounts for testing)
  • CloudFare
  • OpenAI (or any LLM) for code review intelligence (OpenAI, Gemini, etc)

Prerequisites

Before you start, make sure you have:

  • A GitHub account with access to the target repository.
  • A Slack workspace where you can create and manage apps.
  • Node.js (v18 or later) and npm installed on your machine.
  • A personal access token for GitHub (with repo and pull_requests scopes).
  • A Slack bot token with chat:write, commands, and interactivity permissions.
  • OpenAI API key (or another LLM provider key) for AI code review.
  • (Optional) ngrok or CloudFare if testing Slack interactivity locally.

⚡️ How It Works 

sequenceDiagram participant Dev as Developer participant GH as GitHub participant LLM as LLM Review Service participant BE as Backend participant Slack as Slack Dev->>+GH: Create new PR/request GH->>+LLM: Send code for review using Github Action LLM->>+GH: Return review feedback GH->>+Slack: Post review message with Approve|Comment buttons https://slack.com/api/chat.postMessage [Auth Bot Token] Slack->>+Senior: Display review and buttons Senior-->>-Slack: Click Approve or Comment Slack->>+BE: Send interaction payload [Cloudfare: 3000] alt Approve BE->>GH: Approve PR/commit via API else Comment BE->>Senior: Prompt for comment (optional) Senior->>BE: Submit comment BE->>GH: Post comment via API end
Enter fullscreen mode Exit fullscreen mode

Project folder 

code-review-llm/ ├── .github/ │ └── workflows/ │ └── code-review.yml # GitHub Actions workflow for AI code review ├── src/ │ └── services/ │ ├── reviewRunner.js # Main script to run LLM code review │ ├── llmService.js # LLM API integration │ ├── reviewService.js # Review logic │ └── ... # Other utilities/services ├── public/ │ └── vite.svg # Static assets ├── App.jsx, main.jsx, App.css # Frontend entry and styles ├── package.json, README.md # Project metadata and docs └── slack-backend/ ├── index.js # Express backend for Slack interactivity ├── package.json # Backend dependencies ├── .env # Backend environment variables (not committed) └── README.md # Backend setup instructions
Enter fullscreen mode Exit fullscreen mode

Let's do it

1. Set Up the GitHub Action

  • Purpose: Triggers on every pull request and runs your AI code review script and posts results to Slack.
  • Todo: set OPENAI_API_KEY and SLACK_BOT_TOKEN as Repository secrets in Github setting 
name: AI Code Review on: pull_request: types: [opened, synchronize, reopened] jobs: code-review: runs-on: ubuntu-latest permissions: contents: read pull-requests: write actions: read issues: read steps: - name: Checkout code uses: actions/checkout@v4 with: fetch-depth: 0 token: ${{ secrets.GITHUB_TOKEN }} - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install dependencies run: npm ci - name: Get PR diff id: diff run: | git fetch origin ${{ github.event.pull_request.base.ref }} DIFF=$(git diff origin/${{ github.event.pull_request.base.ref }}...HEAD --unified=3) echo "diff<<EOF" >> $GITHUB_OUTPUT echo "$DIFF" >> $GITHUB_OUTPUT echo "EOF" >> $GITHUB_OUTPUT - name: Run AI Code Review env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} PR_NUMBER: ${{ github.event.pull_request.number }} REPO_OWNER: ${{ github.repository_owner }} REPO_NAME: ${{ github.event.repository.name }} PR_DIFF: ${{ steps.diff.outputs.diff }} run: node src/services/reviewRunner.js - name: Send review summary log to Slack env: SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} run: | SUMMARY=$(cat review-summary.txt 2>/dev/null || echo "No summary file found.") curl -X POST https://slack.com/api/chat.postMessage \ -H "Authorization: Bearer $SLACK_BOT_TOKEN" \ -H "Content-type: application/json" \ --data "{\"channel\": \"test-bot\", \"text\": \"$SUMMARY\"}" - name: Send LLM review to Slack with buttons env: SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} run: | curl -X POST https://slack.com/api/chat.postMessage \ -H "Authorization: Bearer $SLACK_BOT_TOKEN" \ -H "Content-type: application/json" \ --data '{ "channel": "test-bot", "text": "Code review for PR #${{ github.event.pull_request.number }}: ${{ github.event.pull_request.title }}", "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "*Code Review Request*\n*PR #${{ github.event.pull_request.number }}:* ${{ github.event.pull_request.title }}\n\n*LLM Review:*\nReview output here\n\nPlease review and take action below:" } }, { "type": "actions", "elements": [ { "type": "button", "text": { "type": "plain_text", "text": "✅ Approve" }, "value": "approve_${{ github.event.pull_request.number }}", "action_id": "approve_action", "style": "primary" }, { "type": "button", "text": { "type": "plain_text", "text": "💬 Request Changes" }, "value": "comment_${{ github.event.pull_request.number }}", "action_id": "comment_action", "style": "danger" } ] } ] }'
Enter fullscreen mode Exit fullscreen mode

2. Backend for Slack Interactivity

  • Purpose: Handles button clicks from Slack and updates GitHub PRs.
  • Todo: use a dedicated GitHub bot account for the API token to avoid self-approval issues. 
app.post('/slack/interactive', async (req, res) => { const payload = JSON.parse(req.body.payload); const { actions, user, response_url } = payload; const action = actions[0]; const prNumber = action.value.replace(/^(approve|comment)_/, ''); if (action.action_id === 'approve_action') { // Approve PR via GitHub API } else if (action.action_id === 'comment_action') { // Post comment via GitHub API } // Respond to Slack });
Enter fullscreen mode Exit fullscreen mode

3. Slack App Setup

  • Enable Interactivity and set the Request URL to your backend.
  • Add your bot to the target channel.
  • Created a target channel and add reviewer as member

4. Create a new branch and push a commit

  • Let's create the test commit and merge request, youc will see what happend

🔗 Try It Yourself!
Check out the GitHub repo and start automating your code reviews today!

🙌 Final Thoughts

Building an AI-powered code review bot is easier than you think—and it can save your team hours every week.
If you found this helpful, please share, comment, and follow for more DevOps and AI tips!

Top comments (0)