Skip to content

ModSecurity v3.0.12 audit logs contain unescaped binary data making logs invisible to kubectl and unparseable by log processors #3463

@kolesaev

Description

@kolesaev

Describe the bug
ModSecurity v3.0.12 audit logs contain raw binary/hex-encoded data that completely breaks JSON validation. This causes:

  1. kubectl logs: Problematic logs become INVISIBLE - they don't appear in output at all
  2. Log processors: CrowdSec, Vector, Fluent-bit fail to parse with JSON errors
  3. Security monitoring: Critical security events are completely lost from visibility

Logs and dumps

Example of corrupted log that disappears from kubectl output:

{ "transaction": { "messages": [{ "details": { "match": "Matched against variable `ARGS_NAMES:�d allow_url_include=1 �d auto_prepend_file=php://input'", "data": "Matched Data: found within ARGS_NAMES:�d allow_url_include=1 �d auto_prepend_file=php://input", "ruleId": "933100", "severity": "2" } }] } }

To Reproduce

  1. Setup ModSecurity with OWASP CRS 4.4.0:
SecAuditLogParts ABHJZ SecAuditLogFormat JSON SecAuditLog /dev/stdout SecRule REQUEST_URI ".*" \ "phase:1,pass,nolog,ctl:ruleRemoveTargetById=1-9999999;ARGS,ctl:ruleRemoveTargetById=1-9999999;ARGS_NAMES"
  1. Send attack with binary data:
curl "http://target/?%ADd+allow_url_include%3d1+%ADd+auto_prepend_file%3dphp://input"
  1. Check kubectl logs:
kubectl logs <modsecurity-pod> | grep "933100"

Result: No output - the log entry is completely missing

  1. Check raw logs directly:
kubectl logs <modsecurity-pod> --previous > raw.log

Result: File contains corrupted JSON with binary characters

Expected behavior

  • All audit logs should be visible via kubectl logs
  • JSON should be valid and parseable by all standard tools
  • Security events should not be lost

Actual behavior

  • kubectl logs: Problematic logs DISAPPEAR from output entirely
  • CrowdSec: Crashes parsing and stops processing all logs
  • Vector: Drops entries with JSON parse errors
  • Fluent-bit: Fails to process and forward
  • Security events: Completely lost from monitoring

Server

  • ModSecurity: v3.0.12 + nginx-connector v1.0.3
  • WebServer: nginx/1.25.5
  • OS: Kubernetes (ingress-nginx)

Rule Set

  • OWASP CRS 4.4.0

Additional context

CRITICAL DATA LOSS:
This is worse than just corrupted logs - we're losing security events completely:

  1. kubectl logs: Shows only "good" logs, hides all ModSecurity alerts with binary data
  2. No debugging: Cannot see what attacks are being blocked
  3. Security blindspot: Attack detection events vanish from all monitoring
  4. Silent failure: No indication that logs are being lost

Evidence of data loss:

  • Normal application logs appear in kubectl logs
  • ModSecurity logs with binary data are COMPLETELY MISSING
  • Only by examining raw container logs do we see the corrupted entries
  • Security teams have no visibility into attacks being blocked

Impact:

  • Zero visibility into ModSecurity blocks via standard kubectl
  • Security monitoring completely broken - attacks are invisible
  • Compliance failure - cannot audit security events
  • Production debugging impossible for WAF issues

This makes ModSecurity unusable for production security - we're blocking attacks but can't see them happening.

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.xRelated to ModSecurity version 3.xWIP

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions