|
1 | 1 | # Promptfoo Code Scan GitHub Action |
2 | 2 |
|
3 | | -Automatically scan pull requests for security vulnerabilities using AI-powered analysis. |
4 | | - |
5 | | -## Features |
6 | | - |
7 | | -- 🔍 **AI-Powered Security Analysis**: Leverages Claude Agent SDK to identify security issues |
8 | | -- 💬 **Inline PR Comments**: Posts findings directly on the relevant lines of code |
9 | | -- ⚙️ **Configurable Severity**: Filter findings by minimum severity level |
10 | | -- 🎯 **Flexible Failure Modes**: Choose whether to fail builds on findings |
11 | | -- 📝 **YAML Configuration**: Use existing config files or generate on-the-fly |
| 3 | +Scan pull requests for LLM security vulnerabilities using AI-powered analysis. |
12 | 4 |
|
13 | 5 | ## Usage |
14 | 6 |
|
|
20 | 12 | types: [opened, synchronize] |
21 | 13 |
|
22 | 14 | permissions: |
23 | | - id-token: write # Required for GitHub OIDC authentication |
24 | | - contents: read # Required to checkout code |
25 | | - pull-requests: write # Required to post comments |
| 15 | + id-token: write |
| 16 | + contents: read |
| 17 | + pull-requests: write |
26 | 18 |
|
27 | 19 | jobs: |
28 | 20 | scan: |
29 | 21 | runs-on: ubuntu-latest |
30 | 22 | steps: |
31 | | - - name: Checkout code |
32 | | - uses: actions/checkout@v4 |
| 23 | + - uses: actions/checkout@v4 |
33 | 24 | with: |
34 | 25 | fetch-depth: 0 |
35 | 26 |
|
36 | | - - name: Run security scan |
37 | | - uses: promptfoo/code-scan-action@v1 |
| 27 | + - uses: promptfoo/code-scan-action@v0 |
38 | 28 | with: |
39 | | - server-url: 'https://scanner.example.com' |
40 | | - minimum-severity: 'medium' |
41 | | - fail-on-vulnerabilities: 'high' |
42 | | - github-token: ${{ secrets.GITHUB_TOKEN }} |
| 29 | + minimum-severity: medium |
43 | 30 | ``` |
44 | 31 |
|
45 | 32 | ## Inputs |
46 | 33 |
|
47 | | -| Input | Description | Required | Default | |
48 | | -|-------|-------------|----------|---------| |
49 | | -| `server-url` | URL of the code-scan server | Yes | - | |
50 | | -| `github-token` | GitHub token for posting comments | Yes | - | |
51 | | -| `minimum-severity` | Minimum severity to report (low, medium, high, critical) | No | `medium` | |
52 | | -| `fail-on-vulnerabilities` | Fail the build if vulnerabilities are found (false, high, critical) | No | `false` | |
53 | | -| `config-path` | Path to YAML config file | No | Auto-generated | |
54 | | - |
55 | | -### Input Details |
56 | | - |
57 | | -#### `server-url` |
58 | | -The URL of your deployed code-scan server. For production use, this should be a public HTTPS endpoint. |
59 | | - |
60 | | -#### `github-token` |
61 | | -The GitHub token used to post review comments. Use `${{ secrets.GITHUB_TOKEN }}` which is automatically provided by GitHub Actions. |
62 | | - |
63 | | -#### `minimum-severity` |
64 | | -Controls which findings are reported. Options: |
65 | | -- `low`: Report all findings |
66 | | -- `medium`: Report medium, high, and critical findings |
67 | | -- `high`: Report only high and critical findings |
68 | | -- `critical`: Report only critical findings |
69 | | - |
70 | | -#### `fail-on-vulnerabilities` |
71 | | -Controls whether the action fails the workflow. Options: |
72 | | -- `false`: Never fail (default, only post comments) |
73 | | -- `high`: Fail if high or critical vulnerabilities are found |
74 | | -- `critical`: Fail only if critical vulnerabilities are found |
75 | | - |
76 | | -#### `config-path` |
77 | | -Optional path to a YAML configuration file. If not provided, a temporary config will be generated using the `minimum-severity` input. |
78 | | - |
79 | | -Example config file: |
80 | | -```yaml |
81 | | -# code-scan.yml |
82 | | -minimumSeverity: medium |
83 | | -customRules: |
84 | | - - name: "Check for SQL injection" |
85 | | - severity: high |
86 | | -``` |
87 | | - |
88 | | -## Local Testing with Act |
89 | | - |
90 | | -You can test the action locally using [act](https://github.com/nektos/act): |
91 | | - |
92 | | -### Prerequisites |
93 | | - |
94 | | -1. Install act: |
95 | | - ```bash |
96 | | - # macOS |
97 | | - brew install act |
98 | | -
|
99 | | - # Linux |
100 | | - curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash |
101 | | - ``` |
102 | | - |
103 | | -2. Start the code-scan server locally: |
104 | | - ```bash |
105 | | - cd code-scan/server |
106 | | - npm run dev |
107 | | - ``` |
108 | | - |
109 | | -### Running Tests |
110 | | - |
111 | | -```bash |
112 | | -# Run the test workflow |
113 | | -act pull_request -W .github/workflows/test-scan.yml |
114 | | -
|
115 | | -# With event payload |
116 | | -act pull_request -W .github/workflows/test-scan.yml \ |
117 | | - -e .github/workflows/test-event.json |
118 | | -``` |
119 | | - |
120 | | -### Test Event Payload |
121 | | - |
122 | | -Create `.github/workflows/test-event.json`: |
123 | | - |
124 | | -```json |
125 | | -{ |
126 | | - "pull_request": { |
127 | | - "number": 1, |
128 | | - "head": { |
129 | | - "ref": "feature-branch", |
130 | | - "sha": "abc123" |
131 | | - }, |
132 | | - "base": { |
133 | | - "ref": "main" |
134 | | - } |
135 | | - }, |
136 | | - "repository": { |
137 | | - "name": "test-repo", |
138 | | - "owner": { |
139 | | - "login": "test-owner" |
140 | | - } |
141 | | - } |
142 | | -} |
143 | | -``` |
144 | | - |
145 | | -### Local Server Configuration |
146 | | - |
147 | | -The test workflow uses `http://host.docker.internal:2095` to connect to your local server from within act's Docker container. Make sure: |
148 | | - |
149 | | -1. Your server is running on port `2095` (or update the workflow) |
150 | | -2. GitHub OIDC auth is disabled in development: |
151 | | - ```bash |
152 | | - # In server/.env |
153 | | - GITHUB_OIDC_ENABLED=false |
154 | | - ``` |
155 | | - |
156 | | -## Architecture |
157 | | - |
158 | | -This action: |
159 | | - |
160 | | -1. **Validates Context**: Ensures it's running in a pull request |
161 | | -2. **Authenticates**: Obtains GitHub OIDC token for server authentication |
162 | | -3. **Generates Config**: Creates or uses provided YAML configuration |
163 | | -4. **Runs CLI**: Executes `@promptfoo/code-scan-cli` with `--json` output |
164 | | -5. **Parses Results**: Extracts structured comments from JSON response |
165 | | -6. **Posts Comments**: Uses Octokit to post inline PR comments |
166 | | -7. **Handles Failures**: Optionally fails the workflow based on findings |
167 | | - |
168 | | -The server handles all the heavy lifting: |
169 | | -- Clones and analyzes the repository |
170 | | -- Runs AI-powered security analysis |
171 | | -- Generates structured findings in XML format |
172 | | -- Parses XML to JSON for consumption |
173 | | - |
174 | | -## Requirements |
175 | | - |
176 | | -### Server Setup |
177 | | - |
178 | | -You need a running code-scan server. See the [server documentation](../server/README.md) for deployment instructions. |
179 | | - |
180 | | -Required environment variables on the server: |
181 | | -```bash |
182 | | -ANTHROPIC_API_KEY=your_api_key |
183 | | -GITHUB_OIDC_ENABLED=true |
184 | | -GITHUB_OIDC_AUDIENCE=https://scanner.example.com |
185 | | -``` |
186 | | - |
187 | | -### GitHub Repository Permissions |
188 | | - |
189 | | -The action requires these permissions in your workflow: |
190 | | -- `id-token: write` - For GitHub OIDC authentication |
191 | | -- `contents: read` - To checkout the code |
192 | | -- `pull-requests: write` - To post review comments |
193 | | - |
194 | | -## Troubleshooting |
195 | | - |
196 | | -### Action fails with "Not in a pull request context" |
197 | | - |
198 | | -Make sure your workflow triggers on `pull_request` events: |
199 | | -```yaml |
200 | | -on: |
201 | | - pull_request: |
202 | | - types: [opened, synchronize] |
203 | | -``` |
204 | | - |
205 | | -### Comments not posting |
206 | | - |
207 | | -Check that: |
208 | | -1. The workflow has `pull-requests: write` permission |
209 | | -2. The `github-token` input is provided |
210 | | -3. The bot account has access to the repository |
211 | | - |
212 | | -### Server authentication fails |
213 | | - |
214 | | -Verify: |
215 | | -1. Server has `GITHUB_OIDC_ENABLED=true` |
216 | | -2. `GITHUB_OIDC_AUDIENCE` matches your server URL |
217 | | -3. Workflow has `id-token: write` permission |
218 | | - |
219 | | -### No findings reported |
220 | | - |
221 | | -This likely means no vulnerabilities were detected! Check: |
222 | | -1. The `minimum-severity` setting isn't filtering too aggressively |
223 | | -2. The server logs for scan results |
224 | | -3. The CLI output in action logs (set `ACTIONS_STEP_DEBUG=true`) |
225 | | - |
226 | | -## Development |
227 | | - |
228 | | -### Building |
229 | | - |
230 | | -```bash |
231 | | -cd code-scan/action |
232 | | -npm run build |
233 | | -``` |
234 | | - |
235 | | -This uses `@vercel/ncc` to bundle the action into `dist/index.js`. |
236 | | - |
237 | | -**Important**: The `dist/` directory is committed to the repository (not gitignored) because GitHub Actions require the compiled code to run the action. |
238 | | - |
239 | | -### Testing Locally with Act |
240 | | - |
241 | | -Test the action locally using [act](https://github.com/nektos/act): |
242 | | - |
243 | | -```bash |
244 | | -# From action/ directory |
245 | | -npm run dev |
246 | | -
|
247 | | -# Or from code-scan/ directory |
248 | | -npm run dev:action |
249 | | -``` |
250 | | - |
251 | | -**Prerequisites:** |
252 | | - |
253 | | -1. **Install act** (one-time setup): |
254 | | - ```bash |
255 | | - # macOS |
256 | | - brew install act |
257 | | -
|
258 | | - # Linux |
259 | | - curl -s https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash |
260 | | -
|
261 | | - # Windows |
262 | | - choco install act-cli |
263 | | - ``` |
264 | | - See [act installation docs](https://github.com/nektos/act#installation) for more options. |
265 | | - |
266 | | -2. **Start the code-scan server**: |
267 | | - ```bash |
268 | | - cd ../server |
269 | | - npm run dev |
270 | | - ``` |
271 | | - |
272 | | -3. Act will connect to `http://host.docker.internal:2095` from within the Docker container |
273 | | - |
274 | | -**What happens during local testing:** |
275 | | -- The CLI is automatically built from `../cli` (local source) |
276 | | -- Act uses a mock PR event (`.github/workflows/test-event.json`) |
277 | | -- The action detects `ACT=true` and uses the local CLI instead of `npx` |
278 | | -- The scan runs against your current working directory |
279 | | -- Comments are logged but not posted to GitHub (ACT mode detected) |
280 | | - |
281 | | -**Note**: Full GitHub App integration testing (including comment posting) requires testing on a real repository. See [GitHub App Setup Guide](../server/docs/github-app-setup.md) for webhook configuration. |
282 | | - |
283 | | -### Testing Changes |
284 | | - |
285 | | -After making changes: |
286 | | -1. Run `npm run build` to rebuild the action |
287 | | -2. Commit the updated `dist/index.js` (required for GitHub Actions) |
288 | | -3. Test locally with `npm run test:local` or push to a test repository |
| 34 | +| Input | Description | Default | |
| 35 | +|-------|-------------|---------| |
| 36 | +| `minimum-severity` | Minimum severity to report: `low`, `medium`, `high`, `critical` | `high` | |
| 37 | +| `server-url` | Code scan server URL | `https://api.promptfoo.dev` | |
289 | 38 |
|
290 | 39 | ## License |
291 | 40 |
|
|
0 commit comments