Remove regex from ingest pipeline

[kaiyan-sheng]

Proposed commit message

This PR removed all regex usage in the awsfirehose ingest pipeline. The script now uses standard Painless string methods (.contains(), .split(), .length(), etc.) to identify log types in order to lower the risk of high CPU consumption.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

elastic-package stack down elastic-package stack up -d --services=elasticsearch $(elastic-package stack shellinit) cd integrations/packages/awsfirehose elastic-package test pipeline --data-streams logs 

Benchmark results

Both results are ran 4 times and calculated an average with command:

elastic-package benchmark pipeline --data-streams logs 

With the change in this PR

╭─────────────────────────╮ │ parameters │ ├──────────────────┬──────┤ │ source_doc_count │ 16 │ │ doc_count │ 1000 │ ╰──────────────────┴──────╯ ╭───────────────────────────╮ │ pipeline_performance │ ├─────────────────┬─────────┤ │ processing_time │ 0.08s │ │ eps │ 12440.54│ ╰─────────────────┴─────────╯ ╭────────────────────────────────────╮ │ procs_by_total_time │ ├───────────────────────────┬────────┤ │ script @ default.yml:10 │ 78.63% │ │ reroute @ default.yml:205 │ 1.24% │ │ reroute @ default.yml:189 │ 1.24% │ │ set @ default.yml:7 │ 1.24% │ ╰───────────────────────────┴────────╯ ╭─────────────────────────────────────╮ │ procs_by_avg_time_per_doc │ ├───────────────────────────┬─────────┤ │ script @ default.yml:10 │ 65µs │ │ reroute @ default.yml:205 │ 5.291µs │ │ reroute @ default.yml:189 │ 5.291µs │ │ set @ default.yml:7 │ 1µs │ ╰───────────────────────────┴─────────╯ 

Without the change in this PR

╭─────────────────────────╮ │ parameters │ ├──────────────────┬──────┤ │ source_doc_count │ 16 │ │ doc_count │ 1000 │ ╰──────────────────┴──────╯ ╭────────────────────────────╮ │ pipeline_performance │ ├─────────────────┬──────────┤ │ processing_time │ 0.11s │ │ eps │ 9007.51 │ ╰─────────────────┴──────────╯ ╭───────────────────────────────────╮ │ procs_by_total_time │ ├──────────────────────────┬────────┤ │ script @ default.yml:10 │ 79.52% │ │ reroute @ default.yml:78 │ 0.90% │ │ reroute @ default.yml:62 │ 0.90% │ │ set @ default.yml:7 │ 0.90% │ ╰──────────────────────────┴────────╯ ╭────────────────────────────────────╮ │ procs_by_avg_time_per_doc │ ├──────────────────────────┬─────────┤ │ script @ default.yml:10 │ 89.5µs │ │ reroute @ default.yml:78 │ 5.291µs │ │ reroute @ default.yml:62 │ 5.291µs │ │ set @ default.yml:7 │ 1µs │ ╰──────────────────────────┴─────────╯ 

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[Kavindu-Dodan]

Thanks for working on it. Per shown results it's almost ~40% performance increase

[mykola-elastic]

should it be an enhancement and 1.9.0?

[gizas]

Personally I dont have a strong preference as this is an improvement of previous main change. I will let Kaiyan decide on this

[kaiyan-sheng]

I debated on this too... But I think this is more of a bugfix on the 1.8.0 version instead of a new feature. Thats why I kept it a bugfix

[gizas]

Nice!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 10b2380

History

• 💚 Build #29949 succeeded 35b6d6a
• 💚 Build #29947 succeeded cd8a7ca

cc @kaiyan-sheng

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package awsfirehose - 1.8.2 containing this change is available at https://epr.elastic.co/package/awsfirehose/1.8.2/