Skip to content

Commit ea3ab76

Browse files
committed
Simplify README documentation
1 parent 819a9e8 commit ea3ab76

File tree

1 file changed

+11
-262
lines changed

1 file changed

+11
-262
lines changed

README.md

Lines changed: 11 additions & 262 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,6 @@
11
# Promptfoo Code Scan GitHub Action
22

3-
Automatically scan pull requests for security vulnerabilities using AI-powered analysis.
4-
5-
## Features
6-
7-
- 🔍 **AI-Powered Security Analysis**: Leverages Claude Agent SDK to identify security issues
8-
- 💬 **Inline PR Comments**: Posts findings directly on the relevant lines of code
9-
- ⚙️ **Configurable Severity**: Filter findings by minimum severity level
10-
- 🎯 **Flexible Failure Modes**: Choose whether to fail builds on findings
11-
- 📝 **YAML Configuration**: Use existing config files or generate on-the-fly
3+
Scan pull requests for LLM security vulnerabilities using AI-powered analysis.
124

135
## Usage
146

@@ -20,272 +12,29 @@ on:
2012
types: [opened, synchronize]
2113

2214
permissions:
23-
id-token: write # Required for GitHub OIDC authentication
24-
contents: read # Required to checkout code
25-
pull-requests: write # Required to post comments
15+
id-token: write
16+
contents: read
17+
pull-requests: write
2618

2719
jobs:
2820
scan:
2921
runs-on: ubuntu-latest
3022
steps:
31-
- name: Checkout code
32-
uses: actions/checkout@v4
23+
- uses: actions/checkout@v4
3324
with:
3425
fetch-depth: 0
3526

36-
- name: Run security scan
37-
uses: promptfoo/code-scan-action@v1
27+
- uses: promptfoo/code-scan-action@v0
3828
with:
39-
server-url: 'https://scanner.example.com'
40-
minimum-severity: 'medium'
41-
fail-on-vulnerabilities: 'high'
42-
github-token: ${{ secrets.GITHUB_TOKEN }}
29+
minimum-severity: medium
4330
```
4431
4532
## Inputs
4633
47-
| Input | Description | Required | Default |
48-
|-------|-------------|----------|---------|
49-
| `server-url` | URL of the code-scan server | Yes | - |
50-
| `github-token` | GitHub token for posting comments | Yes | - |
51-
| `minimum-severity` | Minimum severity to report (low, medium, high, critical) | No | `medium` |
52-
| `fail-on-vulnerabilities` | Fail the build if vulnerabilities are found (false, high, critical) | No | `false` |
53-
| `config-path` | Path to YAML config file | No | Auto-generated |
54-
55-
### Input Details
56-
57-
#### `server-url`
58-
The URL of your deployed code-scan server. For production use, this should be a public HTTPS endpoint.
59-
60-
#### `github-token`
61-
The GitHub token used to post review comments. Use `${{ secrets.GITHUB_TOKEN }}` which is automatically provided by GitHub Actions.
62-
63-
#### `minimum-severity`
64-
Controls which findings are reported. Options:
65-
- `low`: Report all findings
66-
- `medium`: Report medium, high, and critical findings
67-
- `high`: Report only high and critical findings
68-
- `critical`: Report only critical findings
69-
70-
#### `fail-on-vulnerabilities`
71-
Controls whether the action fails the workflow. Options:
72-
- `false`: Never fail (default, only post comments)
73-
- `high`: Fail if high or critical vulnerabilities are found
74-
- `critical`: Fail only if critical vulnerabilities are found
75-
76-
#### `config-path`
77-
Optional path to a YAML configuration file. If not provided, a temporary config will be generated using the `minimum-severity` input.
78-
79-
Example config file:
80-
```yaml
81-
# code-scan.yml
82-
minimumSeverity: medium
83-
customRules:
84-
- name: "Check for SQL injection"
85-
severity: high
86-
```
87-
88-
## Local Testing with Act
89-
90-
You can test the action locally using [act](https://github.com/nektos/act):
91-
92-
### Prerequisites
93-
94-
1. Install act:
95-
```bash
96-
# macOS
97-
brew install act
98-
99-
# Linux
100-
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
101-
```
102-
103-
2. Start the code-scan server locally:
104-
```bash
105-
cd code-scan/server
106-
npm run dev
107-
```
108-
109-
### Running Tests
110-
111-
```bash
112-
# Run the test workflow
113-
act pull_request -W .github/workflows/test-scan.yml
114-
115-
# With event payload
116-
act pull_request -W .github/workflows/test-scan.yml \
117-
-e .github/workflows/test-event.json
118-
```
119-
120-
### Test Event Payload
121-
122-
Create `.github/workflows/test-event.json`:
123-
124-
```json
125-
{
126-
"pull_request": {
127-
"number": 1,
128-
"head": {
129-
"ref": "feature-branch",
130-
"sha": "abc123"
131-
},
132-
"base": {
133-
"ref": "main"
134-
}
135-
},
136-
"repository": {
137-
"name": "test-repo",
138-
"owner": {
139-
"login": "test-owner"
140-
}
141-
}
142-
}
143-
```
144-
145-
### Local Server Configuration
146-
147-
The test workflow uses `http://host.docker.internal:2095` to connect to your local server from within act's Docker container. Make sure:
148-
149-
1. Your server is running on port `2095` (or update the workflow)
150-
2. GitHub OIDC auth is disabled in development:
151-
```bash
152-
# In server/.env
153-
GITHUB_OIDC_ENABLED=false
154-
```
155-
156-
## Architecture
157-
158-
This action:
159-
160-
1. **Validates Context**: Ensures it's running in a pull request
161-
2. **Authenticates**: Obtains GitHub OIDC token for server authentication
162-
3. **Generates Config**: Creates or uses provided YAML configuration
163-
4. **Runs CLI**: Executes `@promptfoo/code-scan-cli` with `--json` output
164-
5. **Parses Results**: Extracts structured comments from JSON response
165-
6. **Posts Comments**: Uses Octokit to post inline PR comments
166-
7. **Handles Failures**: Optionally fails the workflow based on findings
167-
168-
The server handles all the heavy lifting:
169-
- Clones and analyzes the repository
170-
- Runs AI-powered security analysis
171-
- Generates structured findings in XML format
172-
- Parses XML to JSON for consumption
173-
174-
## Requirements
175-
176-
### Server Setup
177-
178-
You need a running code-scan server. See the [server documentation](../server/README.md) for deployment instructions.
179-
180-
Required environment variables on the server:
181-
```bash
182-
ANTHROPIC_API_KEY=your_api_key
183-
GITHUB_OIDC_ENABLED=true
184-
GITHUB_OIDC_AUDIENCE=https://scanner.example.com
185-
```
186-
187-
### GitHub Repository Permissions
188-
189-
The action requires these permissions in your workflow:
190-
- `id-token: write` - For GitHub OIDC authentication
191-
- `contents: read` - To checkout the code
192-
- `pull-requests: write` - To post review comments
193-
194-
## Troubleshooting
195-
196-
### Action fails with "Not in a pull request context"
197-
198-
Make sure your workflow triggers on `pull_request` events:
199-
```yaml
200-
on:
201-
pull_request:
202-
types: [opened, synchronize]
203-
```
204-
205-
### Comments not posting
206-
207-
Check that:
208-
1. The workflow has `pull-requests: write` permission
209-
2. The `github-token` input is provided
210-
3. The bot account has access to the repository
211-
212-
### Server authentication fails
213-
214-
Verify:
215-
1. Server has `GITHUB_OIDC_ENABLED=true`
216-
2. `GITHUB_OIDC_AUDIENCE` matches your server URL
217-
3. Workflow has `id-token: write` permission
218-
219-
### No findings reported
220-
221-
This likely means no vulnerabilities were detected! Check:
222-
1. The `minimum-severity` setting isn't filtering too aggressively
223-
2. The server logs for scan results
224-
3. The CLI output in action logs (set `ACTIONS_STEP_DEBUG=true`)
225-
226-
## Development
227-
228-
### Building
229-
230-
```bash
231-
cd code-scan/action
232-
npm run build
233-
```
234-
235-
This uses `@vercel/ncc` to bundle the action into `dist/index.js`.
236-
237-
**Important**: The `dist/` directory is committed to the repository (not gitignored) because GitHub Actions require the compiled code to run the action.
238-
239-
### Testing Locally with Act
240-
241-
Test the action locally using [act](https://github.com/nektos/act):
242-
243-
```bash
244-
# From action/ directory
245-
npm run dev
246-
247-
# Or from code-scan/ directory
248-
npm run dev:action
249-
```
250-
251-
**Prerequisites:**
252-
253-
1. **Install act** (one-time setup):
254-
```bash
255-
# macOS
256-
brew install act
257-
258-
# Linux
259-
curl -s https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
260-
261-
# Windows
262-
choco install act-cli
263-
```
264-
See [act installation docs](https://github.com/nektos/act#installation) for more options.
265-
266-
2. **Start the code-scan server**:
267-
```bash
268-
cd ../server
269-
npm run dev
270-
```
271-
272-
3. Act will connect to `http://host.docker.internal:2095` from within the Docker container
273-
274-
**What happens during local testing:**
275-
- The CLI is automatically built from `../cli` (local source)
276-
- Act uses a mock PR event (`.github/workflows/test-event.json`)
277-
- The action detects `ACT=true` and uses the local CLI instead of `npx`
278-
- The scan runs against your current working directory
279-
- Comments are logged but not posted to GitHub (ACT mode detected)
280-
281-
**Note**: Full GitHub App integration testing (including comment posting) requires testing on a real repository. See [GitHub App Setup Guide](../server/docs/github-app-setup.md) for webhook configuration.
282-
283-
### Testing Changes
284-
285-
After making changes:
286-
1. Run `npm run build` to rebuild the action
287-
2. Commit the updated `dist/index.js` (required for GitHub Actions)
288-
3. Test locally with `npm run test:local` or push to a test repository
34+
| Input | Description | Default |
35+
|-------|-------------|---------|
36+
| `minimum-severity` | Minimum severity to report: `low`, `medium`, `high`, `critical` | `high` |
37+
| `server-url` | Code scan server URL | `https://api.promptfoo.dev` |
28938

29039
## License
29140

0 commit comments

Comments
 (0)