[Ti_AbuseCH] - Added support for optional "Auth Key" request header across all data streams #13261

ShourieG · 2025-03-24T06:59:26Z

Type of change

Enhancement

Proposed commit message

Added support for optional "Auth Key" request header across all data streams

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [AbuseCH]: Add Auth-Key field #13237

Screenshots

elasticmachine · 2025-03-24T07:01:08Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-03-24T07:23:01Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2025-03-24T07:50:47Z

packages/ti_abusech/_dev/build/docs/README.md

+As of February 2025, AbuseCH recommends using an optional `Auth Key` (API Key) in the requests to avoid rate limiting issues. 
+More details on this topic can be found [here](https://abuse.ch/blog/community-first/). As of version 2.7.0, we have added an optional `Auth Key`
+configuration option that can be used to avoid rate limiting.


Suggested change

As of February 2025, AbuseCH recommends using an optional `Auth Key` (API Key) in the requests to avoid rate limiting issues.

More details on this topic can be found [here](https://abuse.ch/blog/community-first/). As of version 2.7.0, we have added an optional `Auth Key`

configuration option that can be used to avoid rate limiting.

From February 2025, AbuseCH recommends using an optional `Auth Key` (API Key) in the requests to avoid rate limiting issues.

More details on this topic can be found [here](https://abuse.ch/blog/community-first/). In version 2.7.0, we have added an optional `Auth Key`

configuration option that can be used to avoid rate limiting.

efd6 · 2025-03-24T07:53:14Z

packages/ti_abusech/data_stream/malware/agent/stream/cel.yml.hbs

+state:
+ auth_key: {{auth_key}}


Suggested change

state:

auth_key: {{auth_key}}

{{#if auth_key}}

state:

auth_key: {{auth_key}}

{{/if}}

efd6 · 2025-03-24T08:03:03Z

packages/ti_abusech/data_stream/malware/agent/stream/cel.yml.hbs

+ ?"Auth-Key": has(state.auth_key) && !(state.auth_key == "") ? 
+ optional.of([state.auth_key]) 
+ :
+ optional.none(),


Suggested change

?"Auth-Key": has(state.auth_key) && !(state.auth_key == "") ?

optional.of([state.auth_key])

:

optional.none(),

?"Auth-Key": has(state.auth_key) ?

optional.of([state.auth_key])

:

optional.none(),

or

Suggested change

?"Auth-Key": has(state.auth_key) && !(state.auth_key == "") ?

optional.of([state.auth_key])

:

optional.none(),

?"Auth-Key": state.auth_key.optMap(k, [k]),

efd6 · 2025-03-24T08:03:59Z

packages/ti_abusech/data_stream/malware/agent/stream/cel.yml.hbs

Same comments in the other data streams.

ShourieG · 2025-03-24T09:03:27Z

@efd6, addressed all the suggestions

ShourieG · 2025-03-25T08:09:13Z

@efd6, if all looks good can you approve ?

CameronVIE · 2025-03-25T12:34:57Z

Hi everyone!
Thank you for your efforts – we really appreciate it!
Would it be possible to get the new integration files to replace the 2.6.0 ones?
Thanks a lot!
Cere

ShourieG · 2025-03-25T12:43:44Z

Hi everyone! Thank you for your efforts – we really appreciate it! Would it be possible to get the new integration files to replace the 2.6.0 ones? Thanks a lot! Cere

Hi @CameronVIE, once the 2.8.0 update is out, you will be able to seamlessly upgrade. Could you clarify a bit on the part where you said Would it be possible to get the new integration files to replace the 2.6.0 ones ? Do you want to manually replace the 2.6.0 integration with the updated ones or something like that ?

CameronVIE · 2025-03-25T12:52:42Z

Hi @CameronVIE, once the 2.8.0 update is out, you will be able to seamlessly upgrade. Could you clarify a bit on the part where you said Would it be possible to get the new integration files to replace the 2.6.0 ones ? Do you want to manually replace the 2.6.0 integration with the updated ones or something like that ?

Hi ShourieG!
Yes, exactly! 😊 I was just wondering if it might be possible to manually replace the 2.6.0 integration files with the updated ones ahead of the official 2.8.0 release. Just wanted to check if that's something that could work.

ShourieG · 2025-03-25T13:13:37Z

Hi everyone! Thank you for your efforts – we really appreciate it! Would it be possible to get the new integration files to replace the 2.6.0 ones? Thanks a lot! Cere

Hi @CameronVIE, once the 2.8.0 update is out, you will be able to seamlessly upgrade. Could you clarify a bit on the part where you said Would it be possible to get the new integration files to replace the 2.6.0 ones ? Do you want to manually replace the 2.6.0 integration with the updated ones or something like that ?

Hi ShourieG! Yes, exactly! 😊 I was just wondering if it might be possible to manually replace the 2.6.0 integration files with the updated ones ahead of the official 2.8.0 release. Just wanted to check if that's something that could work.

You could try it, but there's a high possibility it might break the natural upgrade process, at that point you would have to revert for the upgrade to work. However if you are running a local stack by having cloned the integrations repo it would be straight forward to revert and upgrade.

CameronVIE · 2025-03-25T13:46:09Z

You could try it, but there's a high possibility it might break the natural upgrade process, at that point you would have to revert for the upgrade to work. However if you are running a local stack by having cloned the integrations repo it would be straight forward to revert and upgrade.

Yes, that’s exactly what I had in mind – especially since there’s no fixed date for the 2.8.0 release yet 😅
But totally get your point – I’ll be careful not to break anything (famous last words, I know 😄).
Thanks for the heads-up!

ShourieG · 2025-03-25T13:59:00Z

Hi everyone! Thank you for your efforts – we really appreciate it! Would it be possible to get the new integration files to replace the 2.6.0 ones? Thanks a lot! Cere

Hi @CameronVIE, once the 2.8.0 update is out, you will be able to seamlessly upgrade. Could you clarify a bit on the part where you said Would it be possible to get the new integration files to replace the 2.6.0 ones ? Do you want to manually replace the 2.6.0 integration with the updated ones or something like that ?

Hi ShourieG! Yes, exactly! 😊 I was just wondering if it might be possible to manually replace the 2.6.0 integration files with the updated ones ahead of the official 2.8.0 release. Just wanted to check if that's something that could work.

You could try it, but there's a high possibility it might break the natural upgrade process, at that point you would have to revert for the upgrade to work. However if you are running a local stack by having cloned the integrations repo it would be straight forward to revert and upgrade.

Yes, that’s exactly what I had in mind – especially since there’s no fixed date for the 2.8.0 release yet 😅 But totally get your point – I’ll be careful not to break anything (famous last words, I know 😄). Thanks for the heads-up!

So 2.8.0 will be published quite soon after the PR is merged mostly within an hour or so, which can be either tomorrow or the day after, as soon as it's approved. The bigger issue is that you have to wait for 8.18 for the upgrade to show up which is not yet released. So the manual replacement is worth a try >_< . Hope it works out :).

CameronVIE · 2025-03-25T14:23:18Z

So 2.8.0 will be published quite soon after the PR is merged mostly within an hour or so, which can be either tomorrow or the day after, as soon as it's approved. The bigger issue is that you have to wait for 8.18 for the upgrade to show up which is not yet released. So the manual replacement is worth a try >_< . Hope it works out :).

Think I’ll give it a try! I’ll share an update here once I’ve tested it – fingers crossed 🤞😊

efd6

Nits only, then LGTM

efd6 · 2025-03-26T05:22:35Z

packages/ti_abusech/data_stream/malwarebazaar/agent/stream/cel.yml.hbs

+ "events": {
+ "error": {
+ ?"id": body.?query_status,
+ "message": "POST:"+(


Suggested change

"message": "POST:"+(

"message": "POST "+state.url+": "+(

efd6 · 2025-03-26T05:23:43Z

packages/ti_abusech/data_stream/malwarebazaar/agent/stream/cel.yml.hbs

+ optional.none(),
 }
+ }).do_request().as(resp, resp.StatusCode == 200 ?
+ bytes(resp.Body).decode_json().as(body, body.?query_status.orValue("") == "ok" ?


Suggested change

bytes(resp.Body).decode_json().as(body, body.?query_status.orValue("") == "ok" ?

bytes(resp.Body).decode_json().as(body, body.?query_status == optional.of("ok") ?

efd6 · 2025-03-26T05:24:32Z

packages/ti_abusech/data_stream/malwarebazaar/agent/stream/cel.yml.hbs

+ }),
+ "url": state.url
+ }
+ : body.?query_status.orValue("") == "no_results" ?


Suggested change

: body.?query_status.orValue("") == "no_results" ?

: body.?query_status == optional.of("no_results") ?

mainly doing this because we're otherwise potentially doing orValue twice.

elastic-sonarqube · 2025-03-26T06:07:12Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-03-26T06:07:17Z

💚 Build Succeeded

Buildkite Build
Commit: f270808

History

💚 Build #23746 succeeded 996113e
💚 Build #23741 succeeded 99a83cc

cc @ShourieG

elastic-vault-github-plugin-prod · 2025-03-26T13:27:46Z

Package ti_abusech - 2.8.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/2.8.0/

ShourieG · 2025-03-28T03:43:44Z

Hey @CameronVIE, We just recently released these changes as a back-port in version 2.6.1 which is now available, merged in this PR. This means there's no need to wait for 8.18.

…cross all data streams (#13261) * auth key param working in malwarebaazar with system tests, docs updated * added auth key support for all data streams and relevant system tests * updated changelog * addressed Dan's suggestions * addressed Dan's suggestions

ShourieG added 3 commits March 21, 2025 22:51

auth key param working in malwarebaazar with system tests, docs updated

104256a

added auth key support for all data streams and relevant system tests

fe0d7cf

resolved merge conflicts

321842b

ShourieG requested a review from a team as a code owner March 24, 2025 06:59

updated changelog

99a83cc

ShourieG self-assigned this Mar 24, 2025

ShourieG added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Mar 24, 2025

ShourieG added Integration:ti_abusech abuse.ch enhancement New feature or request labels Mar 24, 2025

ShourieG requested a review from kcreddy March 24, 2025 07:01

ShourieG changed the title ~~[Ti_AbuseCH] - Added support for "Auth Key" request header across all data streams~~ [Ti_AbuseCH] - Added support for optional "Auth Key" request header across all data streams Mar 24, 2025

efd6 reviewed Mar 24, 2025

View reviewed changes

addressed Dan's suggestions

996113e

ShourieG removed the request for review from kcreddy March 24, 2025 09:01

ShourieG requested a review from efd6 March 26, 2025 05:17

efd6 approved these changes Mar 26, 2025

View reviewed changes

addressed Dan's suggestions

f270808

efd6 approved these changes Mar 26, 2025

View reviewed changes

ShourieG merged commit 59c11ee into elastic:main Mar 26, 2025
7 checks passed

This was referenced Mar 26, 2025

[backport][Ti_AbuseCH] - Added support for optional "Auth Key" request header across all data streams #13317

Merged

[AbuseCH] - Backport AbuseCH "Auth-key" changes to a prior version before stack bump to 8.18.0 #13319

Closed

ShourieG deleted the ti_abusech/malwarebazaar branch March 27, 2025 04:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Ti_AbuseCH] - Added support for optional "Auth Key" request header across all data streams #13261

[Ti_AbuseCH] - Added support for optional "Auth Key" request header across all data streams #13261

Uh oh!

ShourieG commented Mar 24, 2025 •

edited

Loading

elasticmachine commented Mar 24, 2025

elastic-vault-github-plugin-prod bot commented Mar 24, 2025

efd6 Mar 24, 2025

efd6 Mar 24, 2025

efd6 Mar 24, 2025

efd6 Mar 24, 2025

ShourieG commented Mar 24, 2025

ShourieG commented Mar 25, 2025

CameronVIE commented Mar 25, 2025

ShourieG commented Mar 25, 2025 •

edited

Loading

CameronVIE commented Mar 25, 2025 •

edited

Loading

ShourieG commented Mar 25, 2025 •

edited

Loading

CameronVIE commented Mar 25, 2025 •

edited

Loading

ShourieG commented Mar 25, 2025

CameronVIE commented Mar 25, 2025 •

edited

Loading

efd6 left a comment

efd6 Mar 26, 2025

efd6 Mar 26, 2025

efd6 Mar 26, 2025

elastic-sonarqube bot commented Mar 26, 2025

elasticmachine commented Mar 26, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 26, 2025

ShourieG commented Mar 28, 2025 •

edited

Loading

Labels

4 participants

	bytes(resp.Body).decode_json().as(body, body.?query_status.orValue("") == "ok" ?
	bytes(resp.Body).decode_json().as(body, body.?query_status == optional.of("ok") ?

	: body.?query_status.orValue("") == "no_results" ?
	: body.?query_status == optional.of("no_results") ?

[Ti_AbuseCH] - Added support for optional "Auth Key" request header across all data streams #13261

[Ti_AbuseCH] - Added support for optional "Auth Key" request header across all data streams #13261

Uh oh!

Conversation

ShourieG commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 24, 2025

elastic-vault-github-plugin-prod bot commented Mar 24, 2025

🚀 Benchmarks report

efd6 Mar 24, 2025

Choose a reason for hiding this comment

efd6 Mar 24, 2025

Choose a reason for hiding this comment

efd6 Mar 24, 2025

Choose a reason for hiding this comment

efd6 Mar 24, 2025

Choose a reason for hiding this comment

ShourieG commented Mar 24, 2025

ShourieG commented Mar 25, 2025

CameronVIE commented Mar 25, 2025

ShourieG commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CameronVIE commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ShourieG commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CameronVIE commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ShourieG commented Mar 25, 2025

CameronVIE commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

efd6 Mar 26, 2025

Choose a reason for hiding this comment

efd6 Mar 26, 2025

Choose a reason for hiding this comment

efd6 Mar 26, 2025

Choose a reason for hiding this comment

elastic-sonarqube bot commented Mar 26, 2025

Quality Gate passed

elasticmachine commented Mar 26, 2025

💚 Build Succeeded

History

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 26, 2025

ShourieG commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

4 participants

ShourieG commented Mar 24, 2025 •

edited

Loading

ShourieG commented Mar 25, 2025 •

edited

Loading

CameronVIE commented Mar 25, 2025 •

edited

Loading

ShourieG commented Mar 25, 2025 •

edited

Loading

CameronVIE commented Mar 25, 2025 •

edited

Loading

CameronVIE commented Mar 25, 2025 •

edited

Loading

ShourieG commented Mar 28, 2025 •

edited

Loading