[Azure Logs] Fix event.original handling on Application Gateway and the generic Event Hub integration

[zmoog]

What does this PR do?

Fix handling of the event.original in the last two Azure Logs that were missing it.

Some event forwarders (for example, Logstash) send both message and event.original fields, causing an error in the ingest pipeline.

With this change, the pipeline ensures there will only be the event.original field early in the pipeline.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Open the Dev Tools and send the following request to simulate the pipeline execution:

POST _ingest/pipeline/logs-azure.application_gateway-1.5.10/_simulate { "docs": [ { "tags": "_source": { "event": { "original": "{\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/PEERINGTEST/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/Application-Gateway-Name\",\"operationName\":\"ApplicationGatewayAccess\",\"timestamp\":\"2017-04-26T19:27:38Z\",\"category\":\"ApplicationGatewayAccessLog\",\"properties\":{\"instanceId\":\"ApplicationGatewayRole_IN_0\",\"clientIP\":\"67.43.156.7\",\"clientPort\":46886,\"httpMethod\":\"GET\",\"requestUri\":\"/phpmyadmin/scripts/setup.php\",\"requestQuery\":\"X-AzureApplicationGateway-CACHE-HIT=0&SERVER-ROUTED=10.4.0.4&X-AzureApplicationGateway-LOG-ID=874f1f0f-6807-41c9-b7bc-f3cfa74aa0b1&SERVER-STATUS=404\",\"userAgent\":\"-\",\"httpStatus\":404,\"httpVersion\":\"HTTP/1.0\",\"receivedBytes\":65,\"sentBytes\":553,\"timeTaken\":205,\"sslEnabled\":\"off\",\"host\":\"www.contoso.com\",\"originalHost\":\"www.contoso.com\"}}" }, "message": "{\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/PEERINGTEST/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/Application-Gateway-Name\",\"operationName\":\"ApplicationGatewayAccess\",\"timestamp\":\"2017-04-26T19:27:38Z\",\"category\":\"ApplicationGatewayAccessLog\",\"properties\":{\"instanceId\":\"ApplicationGatewayRole_IN_0\",\"clientIP\":\"67.43.156.7\",\"clientPort\":46886,\"httpMethod\":\"GET\",\"requestUri\":\"/phpmyadmin/scripts/setup.php\",\"requestQuery\":\"X-AzureApplicationGateway-CACHE-HIT=0&SERVER-ROUTED=10.4.0.4&X-AzureApplicationGateway-LOG-ID=874f1f0f-6807-41c9-b7bc-f3cfa74aa0b1&SERVER-STATUS=404\",\"userAgent\":\"-\",\"httpStatus\":404,\"httpVersion\":\"HTTP/1.0\",\"receivedBytes\":65,\"sentBytes\":553,\"timeTaken\":205,\"sslEnabled\":\"off\",\"host\":\"www.contoso.com\",\"originalHost\":\"www.contoso.com\"}}" } } ] } 

The pipeline will run with no error and return a document.

Related issues

Relates

• Add additional checks for event.original already existing into Azure firewall logs integration ingest pipeline #5334
• [Azure Logs] Handle event.original field from upstream forwarders #3639

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-24T11:39:47.706+0000

• Duration: 16 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	124
Skipped	0
Total	124

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (10/10)	💚
Files	86.364% (19/22)	👎 -13.636
Classes	86.364% (19/22)	👎 -13.636
Methods	83.333% (155/186)	👎 -12.568
Lines	84.93% (2801/3298)	👎 -3.888
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Package azure - 1.5.10 containing this change is available at https://epr.elastic.co/search?package=azure