[Google Threat Intelligence] Add phishing, ransomware, threat_actor, trending and vulnerability_weaponization data streams

packages/ti_google_threat_intelligence/_dev/build/docs/README.md

@@ -11,6 +11,7 @@ Google Threat Intelligence uses the **[Threat List API](https://gtidocs.virustot
 The Threat List API provides the following types of threat feeds:
 
 - **Cryptominers**
+- **Daily Top Trending**
 - **First Stage Delivery Vectors**
 - **Infostealers**
 - **Internet of Things (IoT)**
@@ -19,6 +20,10 @@ The Threat List API provides the following types of threat feeds:
 - **Malware**
 - **Mobile**
 - **OS X**
+- **Phishing**
+- **Ransomware**
+- **Threat Actor**
+- **Vulnerability Weaponization**
 
 ## GTI Subscription Tiers
 
@@ -30,7 +35,7 @@ Customers can access a subset of the available threat lists based on their **Goo
 
 ## Data Streams
 
-Data collection is available for all nine feed types: `cryptominer`, `first_stage_delivery_vectors`, `infostealer`, `iot`, `linux`, `malicious_network_infrastructure`, `malware`, `mobile` and `osx`, each with a separate data stream. By default, **Malicious Network Infrastructure** is enabled. Users can enable additional data streams based on their GTI subscription tier. If a user enables data collection for a data stream they do not have access to, it will result in an error log on the **Discover** page.
+Data collection is available for all threat feed types: `cryptominer`, `first_stage_delivery_vectors`, `infostealer`, `iot`, `linux`, `malicious_network_infrastructure`, `malware`, `mobile`, `osx`, `phishing`, `ransomware`, `threat_actor`, `trending` and `vulnerability_weaponization`, each with a separate data stream. By default, **Ransomware** and **Malicious Network Infrastructure** is enabled. Users can enable additional data streams based on their GTI subscription tier. If a user enables data collection for a data stream they do not have access to, it will result in an error log on the **Discover** page.
 
 ## Requirements
 
@@ -81,7 +86,7 @@ The `labels.is_transform_source` field indicates log origin:
 - **False** for transformed index
 - **True** for source index
 
-Currently, four transforms are available across all 9 data streams.
+Currently, four transforms are available across all 14 data streams.
 
 The following are four transforms along with their associated pipelines:
 
@@ -266,3 +271,53 @@ This is the `OS X` dataset.
 {{event "osx"}}
 
 {{fields "osx"}}
+
+### Phishing
+
+This is the `Phishing` dataset.
+
+#### Example
+
+{{event "phishing"}}
+
+{{fields "phishing"}}
+
+### Ransomware
+
+This is the `Ransomware` dataset.
+
+#### Example
+
+{{event "ransomware"}}
+
+{{fields "ransomware"}}
+
+### Threat Actor
+
+This is the `Threat Actor` dataset.
+
+#### Example
+
+{{event "threat_actor"}}
+
+{{fields "threat_actor"}}
+
+### Daily Top trending
+
+This is the `Daily Top trending` dataset.
+
+#### Example
+
+{{event "trending"}}
+
+{{fields "trending"}}
+
+### Vulnerability Weaponization
+
+This is the `Vulnerability Weaponization` dataset.
+
+#### Example
+
+{{event "vulnerability_weaponization"}}
+
+{{fields "vulnerability_weaponization"}}

packages/ti_google_threat_intelligence/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: 0.5.0
+ changes:
+ - description: Add data streams - phishing, ransomware, threat_actor, trending and vulnerability_weaponization.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/13236
 - version: 0.4.0
  changes:
  - description: Enable Agentless deployment.

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/first_stage_delivery_vectors/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/infostealer/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/iot/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/linux/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/malicious_network_infrastructure/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/malware/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/mobile/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/osx/agent/stream/cel.yml.hbs

@@ -33,7 +33,7 @@ program: |
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.4.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.5.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/phishing/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,3 @@
+fields:
+ tags:
+ - preserve_original_event

packages/ti_google_threat_intelligence/data_stream/phishing/_dev/test/pipeline/test-phishing.log

@@ -0,0 +1,4 @@
+{"data":{"type":"domain","id":"1freeprivacytoolsforyou.xyz","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":60},"severity":{"value":"SEVERITY_MEDIUM"}},"creation_date":1627032803,"jarm":"29d29d16d29d29d21c29d29d29d29d21ce4eed32174656499cf8d445b1a6e3","last_analysis_stats":{"malicious":9,"suspicious":2,"undetected":30,"harmless":53},"last_modification_date":1740123063,"positives":9,"tld":["xyz"]},"relationships":{"threat_actors":{"data":[{"id":"threat-actor--b6c5648f-b348-59e9-8caf-874f8fc8424a","type":"collection","attributes":{"name":"unc3525","collection_type":"threat-actor"}}]}}}}
+{"data":{"type":"file","id":"ffbd26ad0a22a5bd9e1b473c576c1676b8b8c9f52dda67e8eb267cd00e4fcc4a","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":30},"severity":{"value":"SEVERITY_LOW"}},"first_submission_date":1681947913,"last_analysis_date":1740121234,"last_analysis_stats":{"malicious":45,"undetected":19,"typeUnsupported":13},"last_modification_date":1740121446,"last_submission_date":1740121234,"md5":"50ab6dc1d7e49cb738e27582b20a2eea","meaningful_name":"32397fadc2bd6e066eec5a8c5ff4838e2cc43c2e0a3e2620f2c9e9d0c259a369-1729405840159547980.gz","names":["32397fadc2bd6e066eec5a8c5ff4838e2cc43c2e0a3e2620f2c9e9d0c259a369-1733801445501932628.gz","32397fadc2bd6e066eec5a8c5ff4838e2cc43c2e0a3e2620f2c9e9d0c259a369-1729405840159547980.gz"],"positives":45,"tags":["gzip","contains-pe","spreader"],"times_submitted":305,"type_tags":["gzip","compressed"],"vhash":"90c5bf15514532c93441e4c9364efc5a"},"relationships":{}}}
+{"data":{"type":"ip_address","id":"1.128.0.0","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":30},"severity":{"value":"SEVERITY_LOW"}},"asn":13335,"as_owner":"cloudflarenet","last_analysis_stats":{"malicious":1,"undetected":31,"harmless":62},"last_modification_date":1740131985,"network":"104.20.0.0/15","positives":1},"relationships":{"malware_families":{"data":[{"id":"threatfox_apk_hook","type":"collection","attributes":{"name":"hook","collection_type":"malware-family"}}]}}}}
+{"data":{"type":"url","id":"13cbbd412525292fc352eae7b544f11461d11e3270b0640391239cfc639c3c5a","attributes":{"gti_assessment":{"verdict":{"value":"VERDICT_MALICIOUS"},"threat_score":{"value":60},"severity":{"value":"SEVERITY_MEDIUM"}},"categories":["not recommended site/known infection source","malicious (alphamountain.ai)","malware sites","malicious web sites"],"first_submission_date":1738281953,"last_analysis_date":1738648120,"last_analysis_stats":{"malicious":13,"suspicious":2,"undetected":24,"harmless":57},"last_http_response_code":404,"last_modification_date":1738663304,"last_submission_date":1738648120,"positives":13,"tags":["ip"],"times_submitted":2,"title":"404 not found","url":"http://185.215.113.97/files/fck_windows/random.ps1"},"relationships":{"malware_families":{"data":[{"id":"malware--cc1d563a-cf32-55e8-bae3-cbdf8266c794","type":"collection","attributes":{"name":"amadey","collection_type":"malware-family"}}]}}}}