Skip to content

Conversation

@BrendanNurmi
Copy link
Contributor

Proposed commit message

Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline.

This change seeks to resolve defect #11514, where the dissect will fail in the event that a 400 bad request occurs by modify the ingest pipeline for Apache Tomcat localhost access logs.

The pipeline currently does not cater for a null (-) %r, First line of the request (method and request URI) see Tomcat 9 - Access Logging.

To resolve this, the %r will be split out as a whole and dissected again in the event it's not null (-).

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Added 400 test case to Integration Log sample.
  • Updated pipeline within ECE and tested using log samples within integration tests.
  • Updated Test Cases and results to expected vaules.

How to test this PR locally

  • Cloned existing pipeline "logs-apache_tomcat.access.1.5.1" within ECE deployment and modified to add in dissects and tested using the following records:

Input Docs:

[ { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 + 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.2\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 302 - 81.2.69.145 + 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.2\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 X 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 50 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 40 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.3\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 - \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:19:01:17 +0530] \"GET /manager/status HTTP/1.1\" 200 4654 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:19:02:25 +0530] \"GET / HTTP/1.1\" 200 11235" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - - [24/Oct/2024:14:18:49 +1100] \"-\" 400 - 81.2.69.145 + 0.000 \"-\" \"-\" X-Forwarded-For=\"-\"" } } ]

Results:

{ "docs": [ { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 400, "ip": { "local": "81.2.69.145" }, "connection_status": "+" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145", "127.0.0.1", "127.0.0.2" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "client": { "ip": [ "127.0.0.1", "127.0.0.2" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.240317575Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 400, "ip": { "local": "81.2.69.145" }, "connection_status": "+" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145", "127.0.0.1", "127.0.0.2" ] }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 302 } }, "client": { "ip": [ "127.0.0.1", "127.0.0.2" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.240331968Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 400, "connection_status": "X" } }, "related": { "ip": [ "81.2.69.144", "127.0.0.1" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "client": { "ip": [ "127.0.0.1" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.240334601Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 50 } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240339761Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 40, "ip": { "local": "81.2.69.145" } } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145", "127.0.0.1", "127.0.0.3" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "client": { "ip": [ "127.0.0.1", "127.0.0.3" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.2403418Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" } } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240343772Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "ip": { "local": "81.2.69.145" }, "connection_status": "-" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240345692Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:31:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" } } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 4654 }, "http": { "request": { "method": "GET", "referrer": "-" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/manager/status", "original": "/manager/status" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240348732Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:32:25.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" } } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 11235 }, "http": { "request": { "method": "GET" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/", "original": "/" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240350585Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2024-10-24T03:18:49.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "-", "ident": "-" }, "response_time": 0, "ip": { "local": "81.2.69.145" }, "connection_status": "+" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145" ] }, "http": { "request": { "referrer": "-" }, "response": { "status_code": 400 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "failure", "module": "apache_tomcat" }, "user_agent": { "original": "-" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240352292Z" } } } ] }

Related issues

@BrendanNurmi BrendanNurmi requested a review from a team as a code owner October 25, 2024 00:13
@cla-checker-service
Copy link

cla-checker-service bot commented Oct 25, 2024

💚 CLA has been signed

@andrewkroh andrewkroh added bugfix Pull request that fixes a bug issue Integration:apache_tomcat Apache Tomcat needs CLA User must sign the Elastic Contributor License before review. Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] and removed needs CLA User must sign the Elastic Contributor License before review. labels Oct 25, 2024
@BrendanNurmi BrendanNurmi changed the title [Bug] Adding Test case for 400 in test-access.log updated injest pipeline t… [Bug] Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline. Oct 25, 2024
@muthu-mps
Copy link
Contributor

/test

@muthu-mps
Copy link
Contributor

Update the version to 1.8.1 in the manifest.yml file as well.

@muthu-mps
Copy link
Contributor

/test

@harnish-crest-data harnish-crest-data self-requested a review October 30, 2024 09:51
…ix reference, removed null lines from tests.
@BrendanNurmi
Copy link
Contributor Author

Sorry @muthu-mps that should be fixed.
I missed the ctx reference, removed whitespace from the examples and updated and re-ordered the expected results.

Tests passed locally

{A7310BFB-60C1-4C69-A2F0-86ABDA4AA5BF}

@muthu-mps
Copy link
Contributor

/test

Copy link
Contributor

@harnish-crest-data harnish-crest-data left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor suggestion. LGTM!

@elastic-vault-github-plugin-prod

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>
@muthu-mps
Copy link
Contributor

/test

@elasticmachine
Copy link

💚 Build Succeeded

History

Copy link
Contributor Author

@BrendanNurmi BrendanNurmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes applied as requested.

Copy link
Contributor

@muthu-mps muthu-mps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@muthu-mps muthu-mps merged commit a2ae7fa into elastic:main Nov 5, 2024
5 checks passed
@elastic-vault-github-plugin-prod

Package apache_tomcat - 1.8.1 containing this change is available at https://epr.elastic.co/search?package=apache_tomcat

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
…ailure on pipeline. (elastic#11515) * Adding Test case for 400 in test-access.log updated ingest pipeline to cater for bad requests.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
…ailure on pipeline. (elastic#11515) * Adding Test case for 400 in test-access.log updated ingest pipeline to cater for bad requests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:apache_tomcat Apache Tomcat Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]

5 participants