Skip to content

[Google Threat Intelligence] Add linux, malicious_network_infrastructure, malware, mobile and osx data streams #13190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 31 commits into from
Jul 10, 2025
Merged

[Google Threat Intelligence] Add linux, malicious_network_infrastructure, malware, mobile and osx data streams #13190

merged 31 commits into from
Jul 10, 2025

Conversation

@niraj-crest
Copy link
Contributor

@niraj-crest niraj-crest commented Mar 20, 2025 •
edited
Loading

Proposed commit message

This release includes five new data streams: linux, malicious_network_infrastructure, malware, mobile, and osx. Each data stream includes its own data collection logic, ingest pipeline, and associated dashboards and visualizations.

Fields are mapped to their corresponding ECS fields where applicable.

Test samples were derived from live logs and subsequently sanitized.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Clone integrations repo.
  • Install elastic-package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/ti_google_threat_intelligence) directory.
  • Run the following command to run tests. elastic-package test
@andrewkroh andrewkroh added dashboard Relates to a Kibana dashboard bug, enhancement, or modification. New Integration Issue or pull request for creating a new integration package. labels Mar 20, 2025
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Mar 26, 2025

Hey @jamiehynds & @andrewkroh
Can anyone please add reviewers to this PR?
@andrewkroh andrewkroh removed the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Mar 26, 2025
@andrewkroh andrewkroh requested a review from a team March 27, 2025 15:36
@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Mar 27, 2025
@elasticmachine
Copy link

elasticmachine commented Mar 27, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@andrewkroh andrewkroh added the Crest Contributions from Crest developement team. label Mar 27, 2025
@kcreddy
Copy link
Contributor

kcreddy commented Apr 2, 2025

/test
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Apr 2, 2025

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
kcreddy
kcreddy reviewed Apr 2, 2025
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@niraj-crest, can you check if you can improve Sonar quality?
Looks like none of these pipelines are being covered (hence at 0% coverage):
Screenshot 2025-04-02 at 2 45 14 PM
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Apr 3, 2025

@niraj-crest, can you check if you can improve Sonar quality? Looks like none of these pipelines are being covered (hence at 0% coverage): Screenshot 2025-04-02 at 2 45 14 PM

@kcreddy Have removed the pipelines and transforms as they are covered in this PR.
@kcreddy
Copy link
Contributor

kcreddy commented Apr 4, 2025

/test
kcreddy
kcreddy reviewed Apr 4, 2025
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments in #13189 (review) and #13189 (review)
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Apr 4, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube
efd6
efd6 reviewed Apr 7, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed CEL code only at this stage.
@kcreddy
Copy link
Contributor

kcreddy commented Apr 7, 2025

@niraj-crest, please add all of @efd6 review suggestions to #13189 as well. Thanks!
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Apr 8, 2025

@niraj-crest, please add all of @efd6 review suggestions to #13189 as well. Thanks!

Sure, currently working on it.
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Apr 15, 2025

Waiting for reply on this comment. Once we have a finalized approach, I will push all the changes accordingly.
@botelastic
Copy link

botelastic bot commented May 15, 2025

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!
@botelastic botelastic bot added the Stalled label May 15, 2025
@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025
efd6
efd6 reviewed Jul 2, 2025
View reviewed changes
...google_threat_intelligence/data_stream/cryptominer/elasticsearch/ingest_pipeline/default.yml Outdated
- set:
field: threat.feed.dashboard_id
tag: set_threat_feed_dashboard_id
value: ['ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a', 'ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531', 'ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e']
Copy link
Contributor

@efd6 efd6 Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments and YAML.
Copy link
Contributor

@efd6 efd6 Jul 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this not work?

 - set: field: threat.feed.dashboard_id tag: set_threat_feed_dashboard_id value: - ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a - ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531 - ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e
kcreddy
kcreddy reviewed Jul 4, 2025
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after #13190 (comment) and #13190 (comment)
@kcreddy
Copy link
Contributor

kcreddy commented Jul 4, 2025

/test
@niraj-crest niraj-crest requested review from efd6 and kcreddy July 7, 2025 06:31
@niraj-crest niraj-crest mentioned this pull request Jul 7, 2025
Merged
4 tasks
@kcreddy
Copy link
Contributor

kcreddy commented Jul 8, 2025

/test
kcreddy
kcreddy approved these changes Jul 8, 2025
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but please wait for @efd6 approval before merging.
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Jul 9, 2025

Hello @efd6,
We’ve addressed all the comments, could you please take look at the PR?
If everything looks good, we’d appreciate if we could proceed with merging.

Thank you!
efd6
efd6 reviewed Jul 9, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit in #13190 (comment)
@niraj-crest niraj-crest requested a review from efd6 July 9, 2025 17:06
efd6
efd6 approved these changes Jul 9, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks
@efd6
Copy link
Contributor

efd6 commented Jul 9, 2025

/test
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Jul 9, 2025

Quality Gate failed Quality Gate failed

Failed conditions
49.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
@elasticmachine
Copy link

elasticmachine commented Jul 9, 2025

💚 Build Succeeded

History
@efd6
Copy link
Contributor

efd6 commented Jul 9, 2025 •
edited
Loading

@niraj-crest Please use the PR template that exists in the repo so that we have a proposed commit message. I'll put one together for now. Though we do need to know where the test samples come from for this, so I'll wait for that.
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Jul 10, 2025

@efd6 we have Updated PR Template,
Thanks!
@efd6 efd6 merged commit 8d29cdc into elastic:main Jul 10, 2025
8 of 9 checks passed
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jul 10, 2025

Package ti_google_threat_intelligence - 0.3.0 containing this change is available at https://epr.elastic.co/package/ti_google_threat_intelligence/0.3.0/
@andrewkroh andrewkroh added the enhancement New feature or request label Jul 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Crest Contributions from Crest developement team. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

6 participants

@niraj-crest @elasticmachine @kcreddy @efd6 @jamiehynds @andrewkroh