Skip to content

[Google Threat Intelligence] Add IOC Stream data stream #13449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 19 commits into from
Jul 30, 2025
Merged

[Google Threat Intelligence] Add IOC Stream data stream #13449

merged 19 commits into from
Jul 30, 2025

Conversation

@niraj-crest
Copy link
Contributor

@niraj-crest niraj-crest commented Apr 7, 2025 •
edited
Loading

  • Enhancement

Proposed commit message

This release includes five new data streams: ioc_stream data stream includes its own data collection logic, ingest pipeline, and associated dashboards and visualizations.

Fields are mapped to their corresponding ECS fields where applicable.

Test samples were derived from live logs and subsequently sanitized.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Clone integrations repo.
  • Install elastic-package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/ti_google_threat_intelligence) directory.
  • Run the following command to run tests. elastic-package test
@niraj-crest niraj-crest requested a review from a team as a code owner April 7, 2025 15:31
@andrewkroh andrewkroh added dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Crest Contributions from Crest developement team. New Integration Issue or pull request for creating a new integration package. labels Apr 7, 2025
@botelastic
Copy link

botelastic bot commented May 7, 2025

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!
@botelastic botelastic bot added the Stalled label May 7, 2025
@niraj-crest niraj-crest requested a review from a team as a code owner May 19, 2025 12:14
@botelastic botelastic bot removed the Stalled label May 19, 2025
@niraj-crest niraj-crest mentioned this pull request May 30, 2025
Merged
4 tasks
@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jun 23, 2025
@elasticmachine
Copy link

elasticmachine commented Jun 23, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@andrewkroh andrewkroh added the Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) label Jun 23, 2025
@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025
efd6
efd6 reviewed Jul 6, 2025
View reviewed changes
@niraj-crest niraj-crest requested a review from a team as a code owner July 21, 2025 09:19
@andrewkroh andrewkroh removed the New Integration Issue or pull request for creating a new integration package. label Jul 21, 2025
@niraj-crest niraj-crest requested a review from efd6 July 23, 2025 13:18
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Jul 23, 2025

Hello @efd6,
We have addressed all the comments, can you please look into it.
@kcreddy
Copy link
Contributor

kcreddy commented Jul 25, 2025

/test
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jul 25, 2025

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
efd6
efd6 reviewed Jul 28, 2025
View reviewed changes
@niraj-crest niraj-crest requested a review from efd6 July 28, 2025 13:27
@efd6
Copy link
Contributor

efd6 commented Jul 28, 2025

/test
efd6
efd6 reviewed Jul 29, 2025
View reviewed changes
...ingest_pipeline/ti_google_threat_intelligence-correlation_detection_rule_ioc_st-pipeline.yml Outdated
Comment on lines 40 to 42
// Define regex patterns
def gtiPattern = /^temp_fields/; // Matches keys starting with 'temp_fields'
def underscorePattern = /^_/; // Matches keys starting with '_'
Copy link
Contributor

@efd6 efd6 Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this better than simply using startsWith? I'd be surprised if a regex is faster than a simple single-purpose machine, particularly for prefixes this short.
Copy link
Contributor Author

@niraj-crest niraj-crest Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have updated script to use startsWith.
Thanks for suggestion!
@niraj-crest niraj-crest requested a review from efd6 July 29, 2025 12:01
@efd6
Copy link
Contributor

efd6 commented Jul 30, 2025

/test
@elasticmachine
Copy link

elasticmachine commented Jul 30, 2025

💚 Build Succeeded

History
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Jul 30, 2025

Quality Gate failed Quality Gate failed

Failed conditions
43.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
efd6
efd6 approved these changes Jul 30, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks
@efd6 efd6 merged commit b07d748 into elastic:main Jul 30, 2025
8 of 9 checks passed
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jul 30, 2025

Package ti_google_threat_intelligence - 0.6.0 containing this change is available at https://epr.elastic.co/package/ti_google_threat_intelligence/0.6.0/
@andrewkroh andrewkroh added the enhancement New feature or request label Aug 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Crest Contributions from Crest developement team. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

5 participants

@niraj-crest @elasticmachine @kcreddy @efd6 @andrewkroh