Skip to content

[Google Threat Intelligence] Add phishing, ransomware, threat_actor, trending and vulnerability_weaponization data streams #13236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 23 commits into from
Jul 22, 2025
Merged

[Google Threat Intelligence] Add phishing, ransomware, threat_actor, trending and vulnerability_weaponization data streams #13236

merged 23 commits into from
Jul 22, 2025

Conversation

@niraj-crest
Copy link
Contributor

@niraj-crest niraj-crest commented Mar 21, 2025 •
edited
Loading

  • Enhancement

Proposed commit message

This release includes five new data streams: phishing, ransomware, threat_actor, trending, and vulnerability_weaponization. Each data stream includes its own data collection logic, ingest pipeline, and associated dashboards and visualizations.

Fields are mapped to their corresponding ECS fields where applicable.

Test samples were derived from live logs and subsequently sanitized.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Clone integrations repo.
  • Install elastic-package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/ti_google_threat_intelligence) directory.
  • Run the following command to run tests. elastic-package test
@andrewkroh andrewkroh added New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Mar 21, 2025
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Mar 26, 2025

Hey @jamiehynds & @andrewkroh
Can anyone please add reviewers to this PR?
@andrewkroh andrewkroh added Crest Contributions from Crest developement team. and removed dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Mar 26, 2025
@jamiehynds jamiehynds added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Mar 31, 2025
@elasticmachine
Copy link

elasticmachine commented Mar 31, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@niraj-crest niraj-crest requested a review from a team as a code owner April 8, 2025 06:05
@botelastic
Copy link

botelastic bot commented May 8, 2025

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!
@botelastic botelastic bot added the Stalled label May 8, 2025
@niraj-crest niraj-crest requested a review from a team as a code owner May 19, 2025 12:15
@botelastic botelastic bot removed the Stalled label May 19, 2025
@niraj-crest niraj-crest mentioned this pull request May 30, 2025
Merged
4 tasks
@andrewkroh andrewkroh added the Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) label Jun 23, 2025
@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025
@niraj-crest niraj-crest requested a review from a team as a code owner July 11, 2025 13:15
@andrewkroh andrewkroh removed the New Integration Issue or pull request for creating a new integration package. label Jul 11, 2025
efd6
efd6 reviewed Jul 13, 2025
View reviewed changes
.github/CODEOWNERS
Copy link
Contributor

@efd6 efd6 Jul 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this. If it needs to be fixed, please make a separate PR or issue.
@niraj-crest niraj-crest requested a review from efd6 July 14, 2025 06:02
@andrewkroh andrewkroh added the enhancement New feature or request label Jul 14, 2025
efd6
efd6 approved these changes Jul 14, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks
@efd6
Copy link
Contributor

efd6 commented Jul 14, 2025

/test
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Jul 17, 2025 •
edited
Loading

Hello @efd6,
If everything looks fine to you, can we please merge this pr!
@efd6
Copy link
Contributor

efd6 commented Jul 17, 2025

@niraj-crest We are waiting on review.
@efd6
Copy link
Contributor

efd6 commented Jul 17, 2025

/test
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jul 17, 2025 •
edited
Loading

🚀 Benchmarks report

Package ti_google_threat_intelligence 👍(7) 💚(1) 💔(6)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
iot 8474.58 5555.56 -2919.02 (-34.44%) 💔
linux 9174.31 6289.31 -2885 (-31.45%) 💔
malicious_network_infrastructure 10101.01 5555.56 -4545.45 (-45%) 💔
malware 10638.3 6097.56 -4540.74 (-42.68%) 💔
mobile 11111.11 6369.43 -4741.68 (-42.68%) 💔
osx 10752.69 6711.41 -4041.28 (-37.58%) 💔

To see the full report comment with /test benchmark fullreport
kcreddy
kcreddy reviewed Jul 18, 2025
View reviewed changes
...a_stream/vulnerability_weaponization/_dev/test/pipeline/test-vulnerability-weaponization.log
Comment on lines +1 to +4
{"data":{"type":"domain","id":"1freeprivacytoolsforyou.xyz","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":60},"severity":{"value":"SEVERITY_MEDIUM"}},"creation_date":1627032803,"jarm":"29d29d16d29d29d21c29d29d29d29d21ce4eed32174656499cf8d445b1a6e3","last_analysis_stats":{"malicious":9,"suspicious":2,"undetected":30,"harmless":53},"last_modification_date":1740123063,"positives":9,"tld":["xyz"]},"relationships":{"threat_actors":{"data":[{"id":"threat-actor--b6c5648f-b348-59e9-8caf-874f8fc8424a","type":"collection","attributes":{"name":"unc3525","collection_type":"threat-actor"}}]}}}}
{"data":{"type":"file","id":"ffbd26ad0a22a5bd9e1b473c576c1676b8b8c9f52dda67e8eb267cd00e4fcc4a","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":30},"severity":{"value":"SEVERITY_LOW"}},"first_submission_date":1681947913,"last_analysis_date":1740121234,"last_analysis_stats":{"malicious":45,"undetected":19,"typeUnsupported":13},"last_modification_date":1740121446,"last_submission_date":1740121234,"md5":"50ab6dc1d7e49cb738e27582b20a2eea","meaningful_name":"32397fadc2bd6e066eec5a8c5ff4838e2cc43c2e0a3e2620f2c9e9d0c259a369-1729405840159547980.gz","names":["32397fadc2bd6e066eec5a8c5ff4838e2cc43c2e0a3e2620f2c9e9d0c259a369-1733801445501932628.gz","32397fadc2bd6e066eec5a8c5ff4838e2cc43c2e0a3e2620f2c9e9d0c259a369-1729405840159547980.gz"],"positives":45,"tags":["gzip","contains-pe","spreader"],"times_submitted":305,"type_tags":["gzip","compressed"],"vhash":"90c5bf15514532c93441e4c9364efc5a"},"relationships":{}}}
{"data":{"type":"ip_address","id":"1.128.0.0","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":30},"severity":{"value":"SEVERITY_LOW"}},"asn":13335,"as_owner":"cloudflarenet","last_analysis_stats":{"malicious":1,"undetected":31,"harmless":62},"last_modification_date":1740131985,"network":"104.20.0.0/15","positives":1},"relationships":{"malware_families":{"data":[{"id":"threatfox_apk_hook","type":"collection","attributes":{"name":"hook","collection_type":"malware-family"}}]}}}}
{"data":{"type":"url","id":"13cbbd412525292fc352eae7b544f11461d11e3270b0640391239cfc639c3c5a","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":60},"severity":{"value":"SEVERITY_MEDIUM"}},"categories":["not recommended site/known infection source","malicious (alphamountain.ai)","malware sites","malicious web sites"],"first_submission_date":1738281953,"last_analysis_date":1738648120,"last_analysis_stats":{"malicious":13,"suspicious":2,"undetected":24,"harmless":57},"last_http_response_code":404,"last_modification_date":1738663304,"last_submission_date":1738648120,"positives":13,"tags":["ip"],"times_submitted":2,"title":"404 not found","url":"http://185.215.113.97/files/fck_windows/random.ps1"},"relationships":{"malware_families":{"data":[{"id":"malware--cc1d563a-cf32-55e8-bae3-cbdf8266c794","type":"collection","attributes":{"name":"amadey","collection_type":"malware-family"}}]}}}}
Copy link
Contributor

@kcreddy kcreddy Jul 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the sample logs, all the data seem to be just threat indicators rather than vulnerabilities.
Can you remove event.category: vulnerability mapping ?
@niraj-crest niraj-crest requested a review from kcreddy July 18, 2025 12:39
@kcreddy
Copy link
Contributor

kcreddy commented Jul 19, 2025

/test
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Jul 19, 2025

Quality Gate failed Quality Gate failed

Failed conditions
48.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
@elasticmachine
Copy link

elasticmachine commented Jul 19, 2025

💚 Build Succeeded

History
@niraj-crest
Copy link
Contributor Author

niraj-crest commented Jul 22, 2025

Hello @kcreddy,
We have resolved all the comments, could you please look into it.
kcreddy
kcreddy approved these changes Jul 22, 2025
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!
@kcreddy kcreddy merged commit 5815022 into elastic:main Jul 22, 2025
8 of 9 checks passed
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jul 22, 2025

Package ti_google_threat_intelligence - 0.5.0 containing this change is available at https://epr.elastic.co/package/ti_google_threat_intelligence/0.5.0/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Crest Contributions from Crest developement team. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

6 participants

@niraj-crest @elasticmachine @efd6 @kcreddy @andrewkroh @jamiehynds