- Notifications
You must be signed in to change notification settings - Fork 519
[Bug] Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline. #11515
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…o cater for bad requests.
| 💚 CLA has been signed |
packages/apache_tomcat/data_stream/access/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved
| /test |
| Update the version to |
| /test |
…ix reference, removed null lines from tests.
| Sorry @muthu-mps that should be fixed. Tests passed locally |
| /test |
harnish-crest-data left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor suggestion. LGTM!
🚀 Benchmarks reportTo see the full report comment with |
Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>
| /test |
💚 Build Succeeded
History
|
|
BrendanNurmi left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes applied as requested.
packages/apache_tomcat/data_stream/access/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved
muthu-mps left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
| Package apache_tomcat - 1.8.1 containing this change is available at https://epr.elastic.co/search?package=apache_tomcat |
…ailure on pipeline. (elastic#11515) * Adding Test case for 400 in test-access.log updated ingest pipeline to cater for bad requests.
…ailure on pipeline. (elastic#11515) * Adding Test case for 400 in test-access.log updated ingest pipeline to cater for bad requests.





Proposed commit message
Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline.
This change seeks to resolve defect #11514, where the dissect will fail in the event that a 400 bad request occurs by modify the ingest pipeline for Apache Tomcat localhost access logs.
The pipeline currently does not cater for a null (-) %r, First line of the request (method and request URI) see Tomcat 9 - Access Logging.
To resolve this, the %r will be split out as a whole and dissected again in the event it's not null (-).
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Input Docs:
[ { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 + 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.2\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 302 - 81.2.69.145 + 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.2\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 X 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 50 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 40 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.3\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 - \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:19:01:17 +0530] \"GET /manager/status HTTP/1.1\" 200 4654 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - admin [02/Mar/2023:19:02:25 +0530] \"GET / HTTP/1.1\" 200 11235" } }, { "_index": "index", "_id": "id", "_source": { "message": "81.2.69.144 - - [24/Oct/2024:14:18:49 +1100] \"-\" 400 - 81.2.69.145 + 0.000 \"-\" \"-\" X-Forwarded-For=\"-\"" } } ]Results:
{ "docs": [ { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 400, "ip": { "local": "81.2.69.145" }, "connection_status": "+" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145", "127.0.0.1", "127.0.0.2" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "client": { "ip": [ "127.0.0.1", "127.0.0.2" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.240317575Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 400, "ip": { "local": "81.2.69.145" }, "connection_status": "+" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145", "127.0.0.1", "127.0.0.2" ] }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 302 } }, "client": { "ip": [ "127.0.0.1", "127.0.0.2" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.240331968Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 400, "connection_status": "X" } }, "related": { "ip": [ "81.2.69.144", "127.0.0.1" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "client": { "ip": [ "127.0.0.1" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.240334601Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 50 } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240339761Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "response_time": 40, "ip": { "local": "81.2.69.145" } } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145", "127.0.0.1", "127.0.0.3" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "client": { "ip": [ "127.0.0.1", "127.0.0.3" ] }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "_value": null, "timestamp": "2024-10-24T23:59:02.2403418Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" } } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240343772Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:28:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" }, "ip": { "local": "81.2.69.145" }, "connection_status": "-" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145" ] }, "destination": { "bytes": 20486 }, "http": { "request": { "method": "POST", "referrer": "http://localhost:8080/host-manager/html" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/host-manager/images/asf-logo.svg", "extension": "svg", "original": "/host-manager/images/asf-logo.svg" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240345692Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:31:17.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" } } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 4654 }, "http": { "request": { "method": "GET", "referrer": "-" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/manager/status", "original": "/manager/status" }, "user_agent": { "name": "Chrome", "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "device": { "name": "Other" }, "version": "109.0.0.0" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240348732Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2023-03-02T13:32:25.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "admin", "ident": "-" } } }, "related": { "ip": [ "81.2.69.144" ] }, "destination": { "bytes": 11235 }, "http": { "request": { "method": "GET" }, "version": "1.1", "response": { "status_code": 200 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "success", "module": "apache_tomcat" }, "url": { "path": "/", "original": "/" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240350585Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "id", "_source": { "@timestamp": "2024-10-24T03:18:49.000Z", "ecs": { "version": "8.7.0" }, "apache_tomcat": { "access": { "http": { "useragent": "-", "ident": "-" }, "response_time": 0, "ip": { "local": "81.2.69.145" }, "connection_status": "+" } }, "related": { "ip": [ "81.2.69.144", "81.2.69.145" ] }, "http": { "request": { "referrer": "-" }, "response": { "status_code": 400 } }, "source": { "ip": "81.2.69.144" }, "event": { "category": [ "web" ], "type": [ "access" ], "kind": "event", "outcome": "failure", "module": "apache_tomcat" }, "user_agent": { "original": "-" } }, "_ingest": { "timestamp": "2024-10-24T23:59:02.240352292Z" } } } ] }Related issues