- Notifications
You must be signed in to change notification settings - Fork 1.7k
Open
Labels
Description
Describe the bug
ModSecurity v3.0.12 audit logs contain raw binary/hex-encoded data that completely breaks JSON validation. This causes:
- kubectl logs: Problematic logs become INVISIBLE - they don't appear in output at all
- Log processors: CrowdSec, Vector, Fluent-bit fail to parse with JSON errors
- Security monitoring: Critical security events are completely lost from visibility
Logs and dumps
Example of corrupted log that disappears from kubectl output:
{ "transaction": { "messages": [{ "details": { "match": "Matched against variable `ARGS_NAMES:�d allow_url_include=1 �d auto_prepend_file=php://input'", "data": "Matched Data: found within ARGS_NAMES:�d allow_url_include=1 �d auto_prepend_file=php://input", "ruleId": "933100", "severity": "2" } }] } }To Reproduce
- Setup ModSecurity with OWASP CRS 4.4.0:
SecAuditLogParts ABHJZ SecAuditLogFormat JSON SecAuditLog /dev/stdout SecRule REQUEST_URI ".*" \ "phase:1,pass,nolog,ctl:ruleRemoveTargetById=1-9999999;ARGS,ctl:ruleRemoveTargetById=1-9999999;ARGS_NAMES"- Send attack with binary data:
curl "http://target/?%ADd+allow_url_include%3d1+%ADd+auto_prepend_file%3dphp://input"- Check kubectl logs:
kubectl logs <modsecurity-pod> | grep "933100"Result: No output - the log entry is completely missing
- Check raw logs directly:
kubectl logs <modsecurity-pod> --previous > raw.logResult: File contains corrupted JSON with binary � characters
Expected behavior
- All audit logs should be visible via
kubectl logs - JSON should be valid and parseable by all standard tools
- Security events should not be lost
Actual behavior
- kubectl logs: Problematic logs DISAPPEAR from output entirely
- CrowdSec: Crashes parsing and stops processing all logs
- Vector: Drops entries with JSON parse errors
- Fluent-bit: Fails to process and forward
- Security events: Completely lost from monitoring
Server
- ModSecurity: v3.0.12 + nginx-connector v1.0.3
- WebServer: nginx/1.25.5
- OS: Kubernetes (ingress-nginx)
Rule Set
- OWASP CRS 4.4.0
Additional context
CRITICAL DATA LOSS:
This is worse than just corrupted logs - we're losing security events completely:
- kubectl logs: Shows only "good" logs, hides all ModSecurity alerts with binary data
- No debugging: Cannot see what attacks are being blocked
- Security blindspot: Attack detection events vanish from all monitoring
- Silent failure: No indication that logs are being lost
Evidence of data loss:
- Normal application logs appear in
kubectl logs - ModSecurity logs with binary data are COMPLETELY MISSING
- Only by examining raw container logs do we see the corrupted entries
- Security teams have no visibility into attacks being blocked
Impact:
- ❌ Zero visibility into ModSecurity blocks via standard kubectl
- ❌ Security monitoring completely broken - attacks are invisible
- ❌ Compliance failure - cannot audit security events
- ❌ Production debugging impossible for WAF issues
This makes ModSecurity unusable for production security - we're blocking attacks but can't see them happening.