[Google Threat Intelligence] Add linux, malicious_network_infrastructure, malware, mobile and osx data streams

packages/ti_google_threat_intelligence/_dev/build/docs/README.md

@@ -14,6 +14,11 @@ The Threat List API provides the following types of threat feeds:
 - **First Stage Delivery Vectors**
 - **Infostealers**
 - **Internet of Things (IoT)**
+- **Linux**
+- **Malicious Network Infrastructure**
+- **Malware**
+- **Mobile**
+- **OS X**
 
 ## GTI Subscription Tiers
 
@@ -25,7 +30,7 @@ Customers can access a subset of the available threat lists based on their **Goo
 
 ## Data Streams
 
-Data collection is available for four feed types: `cryptominer`, `first_stage_delivery_vectors`, `infostealer`, and `iot`, each provided through a separate data stream. Users can enable data streams based on their GTI subscription tier. If a user enables data collection for a data stream they do not have access to, it will result in an error log on the **Discover** page.
+Data collection is available for all nine feed types: `cryptominer`, `first_stage_delivery_vectors`, `infostealer`, `iot`, `linux`, `malicious_network_infrastructure`, `malware`, `mobile` and `osx`, each with a separate data stream. By default, **Malicious Network Infrastructure** is enabled. Users can enable additional data streams based on their GTI subscription tier. If a user enables data collection for a data stream they do not have access to, it will result in an error log on the **Discover** page.
 
 ## Requirements
 
@@ -70,7 +75,7 @@ The `labels.is_transform_source` field indicates log origin:
 - **False** for transformed index
 - **True** for source index
 
-Currently, four transforms are available across all 4 data streams.
+Currently, four transforms are available across all 9 data streams.
 
 The following are four transforms along with their associated pipelines:
 
@@ -137,10 +142,10 @@ The following transform and its associated pipelines are used to filter relevant
 4. Under the **Destination configuration** section, set the **Ingest Pipeline**:
  - Each transform in the **Google Threat Intelligence** integration has a corresponding ingest pipeline.
  - Refer to the **Transforms table** above for the appropriate pipeline name associated with transform.
- - Prefix the pipeline name with the integration version. 
- For example: 
+ - Prefix the pipeline name with the integration version.
+ For example:
  ```
- 0.1.0-ti_google_threat_intelligence-latest_ip_ioc-transform-pipeline
+ 0.2.0-ti_google_threat_intelligence-latest_ip_ioc-transform-pipeline
  ```
  - Click **Update** to save the changes.
 5. Click the **three dots** again next to the transform and select **Start** to activate it.
@@ -205,3 +210,53 @@ This is the `Internet of Things` dataset.
 {{event "iot"}}
 
 {{fields "iot"}}
+
+### Linux
+
+This is the `Linux` dataset.
+
+#### Example
+
+{{event "linux"}}
+
+{{fields "linux"}}
+
+### Malicious Network Infrastructure
+
+This is the `Malicious Network Infrastructure` dataset.
+
+#### Example
+
+{{event "malicious_network_infrastructure"}}
+
+{{fields "malicious_network_infrastructure"}}
+
+### Malware
+
+This is the `Malware` dataset.
+
+#### Example
+
+{{event "malware"}}
+
+{{fields "malware"}}
+
+### Mobile
+
+This is the `Mobile` dataset.
+
+#### Example
+
+{{event "mobile"}}
+
+{{fields "mobile"}}
+
+### OS X
+
+This is the `OS X` dataset.
+
+#### Example
+
+{{event "osx"}}
+
+{{fields "osx"}}

packages/ti_google_threat_intelligence/_dev/deploy/docker/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '2.3'
 services:
  ti_google_threat_intelligence:
- image: docker.elastic.co/observability/stream:v0.15.0
+ image: docker.elastic.co/observability/stream:v0.18.0
  hostname: ti_google_threat_intelligence
  ports:
  - 8090

packages/ti_google_threat_intelligence/changelog.yml

@@ -1,5 +1,10 @@
 # newer versions go on top
-- version: "0.2.0"
+- version: 0.3.0
+ changes:
+ - description: Add data streams - linux, malicious_network_infrastructure, malware, mobile, osx.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/13190
+- version: 0.2.0
  changes:
  - description: Use `terminate` processor instead of `fail` processor to handle agent errors.
  type: enhancement

packages/ti_google_threat_intelligence/data_stream/cryptominer/_dev/test/pipeline/test-cryptominer.log-expected.json

@@ -71,7 +71,12 @@
  ],
  "threat": {
  "feed": {
- "name": "Cryptominer"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI Cryptominer"
  },
  "indicator": {
  "first_seen": "2021-07-23T09:33:23.000Z",
@@ -154,7 +159,12 @@
  ],
  "threat": {
  "feed": {
- "name": "Cryptominer"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI Cryptominer"
  },
  "indicator": {
  "file": {
@@ -241,7 +251,12 @@
  ],
  "threat": {
  "feed": {
- "name": "Cryptominer"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI Cryptominer"
  },
  "indicator": {
  "as": {
@@ -343,7 +358,12 @@
  ],
  "threat": {
  "feed": {
- "name": "Cryptominer"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI Cryptominer"
  },
  "indicator": {
  "id": [

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

@@ -24,16 +24,16 @@ redact:
  fields:
  - access_token
 program: |
- (
- state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format("2006010215"))
+ state.?cursor.last_timestamp.orValue(
+ (now - duration(state.initial_interval)).format("2006010215")
  ).as(start_time, state.with(
  request(
  "GET",
  state.url.trim_right("/") + "/api/v3/threat_lists/cryptominer/" + start_time + "?" + {
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.1.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.3.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/cryptominer/elasticsearch/ingest_pipeline/default.yml

@@ -63,7 +63,14 @@ processors:
  - set:
  field: threat.feed.name
  tag: set_threat_feed_name
- value: Cryptominer
+ value: GTI Cryptominer
+ - set:
+ field: threat.feed.dashboard_id
+ tag: set_threat_feed_dashboard_id
+ value:
+ - ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a
+ - ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531
+ - ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e
  - rename:
  field: json.data.attributes.as_owner
  tag: rename_data_attributes_as_owner

packages/ti_google_threat_intelligence/data_stream/cryptominer/sample_event.json

@@ -1,22 +1,22 @@
 {
  "@timestamp": "2025-01-27T19:51:31.000Z",
  "agent": {
- "ephemeral_id": "6e343759-95b9-417e-856e-ace3c03853e6",
- "id": "0a800239-cd1a-495d-88fc-db204c4fdc07",
- "name": "elastic-agent-87123",
+ "ephemeral_id": "4df7e827-0e86-458c-8ce4-750acbc29154",
+ "id": "a9506a30-0a26-4a32-ae73-5ddde67eab3f",
+ "name": "elastic-agent-56830",
  "type": "filebeat",
  "version": "8.16.0"
  },
  "data_stream": {
  "dataset": "ti_google_threat_intelligence.cryptominer",
- "namespace": "74935",
+ "namespace": "71400",
  "type": "logs"
  },
  "ecs": {
  "version": "8.17.0"
  },
  "elastic_agent": {
- "id": "0a800239-cd1a-495d-88fc-db204c4fdc07",
+ "id": "a9506a30-0a26-4a32-ae73-5ddde67eab3f",
  "snapshot": false,
  "version": "8.16.0"
  },
@@ -26,7 +26,7 @@
  "threat"
  ],
  "dataset": "ti_google_threat_intelligence.cryptominer",
- "ingested": "2025-06-03T07:03:00Z",
+ "ingested": "2025-07-07T05:47:28Z",
  "kind": "enrichment",
  "original": "{\"data\":{\"attributes\":{\"first_submission_date\":1582817050,\"gti_assessment\":{\"severity\":{\"value\":\"SEVERITY_NONE\"},\"threat_score\":{\"value\":1},\"verdict\":{\"value\":\"VERDICT_UNDETECTED\"}},\"last_analysis_date\":1582817050,\"last_analysis_stats\":{\"harmless\":55,\"malicious\":8,\"undetected\":8},\"last_http_response_code\":200,\"last_modification_date\":1738007491,\"last_submission_date\":1582817050,\"positives\":8,\"times_submitted\":1,\"tld\":\"ru\",\"url\":\"http://securepasswel.ru/files/grapes_encrypted_87ed10f.bin\"},\"id\":\"0146b3be6e724b10e620e8090821a8253772af779a4996145cdf295c01e0900c\",\"relationships\":{},\"type\":\"url\"}}",
  "type": [
@@ -89,7 +89,12 @@
  ],
  "threat": {
  "feed": {
- "name": "Cryptominer"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI Cryptominer"
  },
  "indicator": {
  "id": [

packages/ti_google_threat_intelligence/data_stream/first_stage_delivery_vectors/_dev/test/pipeline/test-first-stage-delivery-vectors.log-expected.json

@@ -71,7 +71,12 @@
  ],
  "threat": {
  "feed": {
- "name": "First Stage Delivery Vectors"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI First Stage Delivery Vectors"
  },
  "indicator": {
  "first_seen": "2021-07-23T09:33:23.000Z",
@@ -154,7 +159,12 @@
  ],
  "threat": {
  "feed": {
- "name": "First Stage Delivery Vectors"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI First Stage Delivery Vectors"
  },
  "indicator": {
  "file": {
@@ -241,7 +251,12 @@
  ],
  "threat": {
  "feed": {
- "name": "First Stage Delivery Vectors"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI First Stage Delivery Vectors"
  },
  "indicator": {
  "as": {
@@ -343,7 +358,12 @@
  ],
  "threat": {
  "feed": {
- "name": "First Stage Delivery Vectors"
+ "dashboard_id": [
+ "ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a",
+ "ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531",
+ "ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e"
+ ],
+ "name": "GTI First Stage Delivery Vectors"
  },
  "indicator": {
  "id": [

packages/ti_google_threat_intelligence/data_stream/first_stage_delivery_vectors/agent/stream/cel.yml.hbs

@@ -24,16 +24,16 @@ redact:
  fields:
  - access_token
 program: |
- (
- state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format("2006010215"))
+ state.?cursor.last_timestamp.orValue(
+ (now - duration(state.initial_interval)).format("2006010215")
  ).as(start_time, state.with(
  request(
  "GET",
  state.url.trim_right("/") + "/api/v3/threat_lists/first-stage-delivery-vectors/" + start_time + "?" + {
  ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
  "limit": ["4000"],
  "x-tool": ["Elastic"],
- "User-Agent": ["v0.1.0"], // Keep this in sync with 'version' in package level manifest.yml.
+ "User-Agent": ["v0.3.0"], // Keep this in sync with 'version' in package level manifest.yml.
  }.format_query()
  ).with({
  "Header": {

packages/ti_google_threat_intelligence/data_stream/first_stage_delivery_vectors/elasticsearch/ingest_pipeline/default.yml

@@ -63,7 +63,14 @@ processors:
  - set:
  field: threat.feed.name
  tag: set_threat_feed_name
- value: First Stage Delivery Vectors
+ value: GTI First Stage Delivery Vectors
+ - set:
+ field: threat.feed.dashboard_id
+ tag: set_threat_feed_dashboard_id
+ value:
+ - ti_google_threat_intelligence-0b0fb6b4-d250-4e31-a56a-bb872e4c7c4a
+ - ti_google_threat_intelligence-9e8de699-a623-4a1b-9f63-7d641116f531
+ - ti_google_threat_intelligence-95187e5c-b4a2-45ad-b6a4-d6ce68e1f43e
  - rename:
  field: json.data.attributes.as_owner
  tag: rename_data_attributes_as_owner