Skip to content

Commit 60be0e3

Browse files
kcreddykcreddy
committed
update ordering of data streams
1 parent 8a71b78 commit 60be0e3

File tree

2 files changed

+121
-123
lines changed

2 files changed

+121
-123
lines changed

‎packages/ti_abusech/_dev/build/docs/README.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,6 @@ For more details, check the abuse.ch [Community First - New Authentication](http
101101
2. In **Search field names**, search and add fields `error.message` and `data_stream.dataset` into the **Discover** view. For more details on adding fields inside **Discover**, check [Discover getting started](https://www.elastic.co/docs/explore-analyze/discover/discover-get-started).
102102
3. Search for the dataset(s) that are enabled by this integration. For example, in the KQL query bar, use the KQL query `data_stream.dataset: ti_abusech.url` to search on specific dataset or KQL query `data_stream.dataset: ti_abusech.*` to search on all datasets.
103103
4. Search for presence of any errors that are captured into `error.message` field using KQL query `error.message: *`. You can combine queries using [KQL boolean expressions](https://www.elastic.co/docs/explore-analyze/query-filter/languages/kql#_combining_multiple_queries), such as `AND`. For example, to search for any errors inside `url` dataset, you can use KQL query: `data_stream.dataset: ti_abusech.url AND error.message: *`.
104-
105104
- Since this integration supports Expiration of Indicators of Compromise (IOCs) using Elastic latest transform, the indicators are present in both source and destination indices. While this seem like duplicate ingestion, it is an implmentation detail which is required to properly expire indicators.
106105
- Because the latest copy of indicators is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly. The ILM policies on source indices can be tuned to manage their data retention period. For more details, check the [Reference](#ilm-policy).
107106
- For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).
@@ -114,24 +113,24 @@ For more information on architectures that can be used for scaling this integrat
114113

115114
### ECS field reference
116115

117-
{{fields "url"}}
118-
119116
{{fields "malware"}}
120117

121118
{{fields "malwarebazaar"}}
122119

123120
{{fields "threatfox"}}
124121

125-
### Example event
122+
{{fields "url"}}
126123

127-
{{event "url"}}
124+
### Example event
128125

129126
{{event "malware"}}
130127

131128
{{event "malwarebazaar"}}
132129

133130
{{event "threatfox"}}
134131

132+
{{event "url"}}
133+
135134
### Inputs used
136135

137136
These inputs can be used in this integration:

‎packages/ti_abusech/docs/README.md‎

Lines changed: 117 additions & 118 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,6 @@ For more details, check the abuse.ch [Community First - New Authentication](http
101101
2. In **Search field names**, search and add fields `error.message` and `data_stream.dataset` into the **Discover** view. For more details on adding fields inside **Discover**, check [Discover getting started](https://www.elastic.co/docs/explore-analyze/discover/discover-get-started).
102102
3. Search for the dataset(s) that are enabled by this integration. For example, in the KQL query bar, use the KQL query `data_stream.dataset: ti_abusech.url` to search on specific dataset or KQL query `data_stream.dataset: ti_abusech.*` to search on all datasets.
103103
4. Search for presence of any errors that are captured into `error.message` field using KQL query `error.message: *`. You can combine queries using [KQL boolean expressions](https://www.elastic.co/docs/explore-analyze/query-filter/languages/kql#_combining_multiple_queries), such as `AND`. For example, to search for any errors inside `url` dataset, you can use KQL query: `data_stream.dataset: ti_abusech.url AND error.message: *`.
104-
105104
- Since this integration supports Expiration of Indicators of Compromise (IOCs) using Elastic latest transform, the indicators are present in both source and destination indices. While this seem like duplicate ingestion, it is an implmentation detail which is required to properly expire indicators.
106105
- Because the latest copy of indicators is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly. The ILM policies on source indices can be tuned to manage their data retention period. For more details, check the [Reference](#ilm-policy).
107106
- For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).
@@ -114,43 +113,6 @@ For more information on architectures that can be used for scaling this integrat
114113

115114
### ECS field reference
116115

117-
**Exported fields**
118-
119-
| Field | Description | Type |
120-
|---|---|---|
121-
| @timestamp | Event timestamp. | date |
122-
| abusech.url.blacklists.spamhaus_dbl | If the indicator is listed on the spamhaus blacklist. | keyword |
123-
| abusech.url.blacklists.surbl | If the indicator is listed on the surbl blacklist. | keyword |
124-
| abusech.url.deleted_at | The timestamp when the indicator is (will be) deleted. | date |
125-
| abusech.url.id | The ID of the indicator. | keyword |
126-
| abusech.url.larted | Indicates whether the malware URL has been reported to the hosting provider (true or false). | boolean |
127-
| abusech.url.last_online | Last timestamp when the URL has been serving malware. | date |
128-
| abusech.url.reporter | The Twitter handle of the reporter that has reported this malware URL (or anonymous). | keyword |
129-
| abusech.url.tags | A list of tags associated with the queried malware URL. | keyword |
130-
| abusech.url.threat | The threat corresponding to this malware URL. | keyword |
131-
| abusech.url.url_status | The current status of the URL. Possible values are: online, offline and unknown. | keyword |
132-
| abusech.url.urlhaus_reference | Link to URLhaus entry. | keyword |
133-
| cloud.image.id | Image ID for the cloud instance. | keyword |
134-
| data_stream.dataset | Data stream dataset name. | constant_keyword |
135-
| data_stream.namespace | Data stream namespace. | constant_keyword |
136-
| data_stream.type | Data stream type. | constant_keyword |
137-
| event.dataset | Event dataset | constant_keyword |
138-
| event.module | Event module | constant_keyword |
139-
| host.containerized | If the host is a container. | boolean |
140-
| host.os.build | OS build information. | keyword |
141-
| host.os.codename | OS codename, if any. | keyword |
142-
| input.type | Type of Filebeat input. | keyword |
143-
| labels.interval | User-configured value for `Interval` setting. This is used in calculation of indicator expiration time. | keyword |
144-
| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword |
145-
| log.flags | Flags for the log file. | keyword |
146-
| log.offset | Offset of the entry in the log file. | long |
147-
| threat.feed.dashboard_id | Dashboard ID used for Kibana CTI UI | constant_keyword |
148-
| threat.feed.name | Display friendly feed name | constant_keyword |
149-
| threat.indicator.first_seen | The date and time when intelligence source first reported sighting this indicator. | date |
150-
| threat.indicator.last_seen | The date and time when intelligence source last reported sighting this indicator. | date |
151-
| threat.indicator.modified_at | The date and time when intelligence source last modified information for this indicator. | date |
152-
153-
154116
**Exported fields**
155117

156118
| Field | Description | Type |
@@ -257,88 +219,44 @@ For more information on architectures that can be used for scaling this integrat
257219
| threat.indicator.modified_at | The date and time when intelligence source last modified information for this indicator. | date |
258220

259221

260-
### Example event
222+
**Exported fields**
261223

262-
An example event for `url` looks as following:
224+
| Field | Description | Type |
225+
|---|---|---|
226+
| @timestamp | Event timestamp. | date |
227+
| abusech.url.blacklists.spamhaus_dbl | If the indicator is listed on the spamhaus blacklist. | keyword |
228+
| abusech.url.blacklists.surbl | If the indicator is listed on the surbl blacklist. | keyword |
229+
| abusech.url.deleted_at | The timestamp when the indicator is (will be) deleted. | date |
230+
| abusech.url.id | The ID of the indicator. | keyword |
231+
| abusech.url.larted | Indicates whether the malware URL has been reported to the hosting provider (true or false). | boolean |
232+
| abusech.url.last_online | Last timestamp when the URL has been serving malware. | date |
233+
| abusech.url.reporter | The Twitter handle of the reporter that has reported this malware URL (or anonymous). | keyword |
234+
| abusech.url.tags | A list of tags associated with the queried malware URL. | keyword |
235+
| abusech.url.threat | The threat corresponding to this malware URL. | keyword |
236+
| abusech.url.url_status | The current status of the URL. Possible values are: online, offline and unknown. | keyword |
237+
| abusech.url.urlhaus_reference | Link to URLhaus entry. | keyword |
238+
| cloud.image.id | Image ID for the cloud instance. | keyword |
239+
| data_stream.dataset | Data stream dataset name. | constant_keyword |
240+
| data_stream.namespace | Data stream namespace. | constant_keyword |
241+
| data_stream.type | Data stream type. | constant_keyword |
242+
| event.dataset | Event dataset | constant_keyword |
243+
| event.module | Event module | constant_keyword |
244+
| host.containerized | If the host is a container. | boolean |
245+
| host.os.build | OS build information. | keyword |
246+
| host.os.codename | OS codename, if any. | keyword |
247+
| input.type | Type of Filebeat input. | keyword |
248+
| labels.interval | User-configured value for `Interval` setting. This is used in calculation of indicator expiration time. | keyword |
249+
| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword |
250+
| log.flags | Flags for the log file. | keyword |
251+
| log.offset | Offset of the entry in the log file. | long |
252+
| threat.feed.dashboard_id | Dashboard ID used for Kibana CTI UI | constant_keyword |
253+
| threat.feed.name | Display friendly feed name | constant_keyword |
254+
| threat.indicator.first_seen | The date and time when intelligence source first reported sighting this indicator. | date |
255+
| threat.indicator.last_seen | The date and time when intelligence source last reported sighting this indicator. | date |
256+
| threat.indicator.modified_at | The date and time when intelligence source last modified information for this indicator. | date |
263257

264-
```json
265-
{
266-
"@timestamp": "2025-07-16T06:32:41.644Z",
267-
"abusech": {
268-
"url": {
269-
"deleted_at": "2025-07-16T07:31:14.625Z",
270-
"id": "2786904",
271-
"threat": "malware_download",
272-
"url_status": "online"
273-
}
274-
},
275-
"agent": {
276-
"ephemeral_id": "8039c627-ea96-4027-8751-2ff7db77251b",
277-
"id": "9106f11b-d54d-46d0-8ace-39e4fff1157b",
278-
"name": "elastic-agent-41888",
279-
"type": "filebeat",
280-
"version": "8.18.0"
281-
},
282-
"data_stream": {
283-
"dataset": "ti_abusech.url",
284-
"namespace": "49664",
285-
"type": "logs"
286-
},
287-
"ecs": {
288-
"version": "8.11.0"
289-
},
290-
"elastic_agent": {
291-
"id": "9106f11b-d54d-46d0-8ace-39e4fff1157b",
292-
"snapshot": true,
293-
"version": "8.18.0"
294-
},
295-
"event": {
296-
"agent_id_status": "verified",
297-
"category": [
298-
"threat"
299-
],
300-
"dataset": "ti_abusech.url",
301-
"ingested": "2025-07-16T06:32:44Z",
302-
"kind": "enrichment",
303-
"original": "{\"dateadded\":\"2024-03-19 11:34:09 UTC\",\"id\":\"2786904\",\"last_online\":\"2024-03-19 11:34:09 UTC\",\"reporter\":\"lrz_urlhaus\",\"tags\":[\"elf\",\"Mozi\"],\"threat\":\"malware_download\",\"url\":\"http://115.55.244.160:41619/Mozi.m\",\"url_status\":\"online\",\"urlhaus_link\":\"https://urlhaus.abuse.ch/url/2786904/\"}",
304-
"type": [
305-
"indicator"
306-
]
307-
},
308-
"input": {
309-
"type": "cel"
310-
},
311-
"labels": {
312-
"interval": "1h"
313-
},
314-
"tags": [
315-
"preserve_original_event",
316-
"forwarded",
317-
"abusech-url",
318-
"elf",
319-
"Mozi"
320-
],
321-
"threat": {
322-
"indicator": {
323-
"first_seen": "2024-03-19T11:34:09.000Z",
324-
"last_seen": "2024-03-19T11:34:09.000Z",
325-
"name": "http://115.55.244.160:41619/Mozi.m",
326-
"provider": "lrz_urlhaus",
327-
"reference": "https://urlhaus.abuse.ch/url/2786904/",
328-
"type": "url",
329-
"url": {
330-
"domain": "115.55.244.160",
331-
"extension": "m",
332-
"full": "http://115.55.244.160:41619/Mozi.m",
333-
"original": "http://115.55.244.160:41619/Mozi.m",
334-
"path": "/Mozi.m",
335-
"port": 41619,
336-
"scheme": "http"
337-
}
338-
}
339-
}
340-
}
341-
```
258+
259+
### Example event
342260

343261
An example event for `malware` looks as following:
344262

@@ -612,6 +530,87 @@ An example event for `threatfox` looks as following:
612530
}
613531
```
614532

533+
An example event for `url` looks as following:
534+
535+
```json
536+
{
537+
"@timestamp": "2025-07-16T06:32:41.644Z",
538+
"abusech": {
539+
"url": {
540+
"deleted_at": "2025-07-16T07:31:14.625Z",
541+
"id": "2786904",
542+
"threat": "malware_download",
543+
"url_status": "online"
544+
}
545+
},
546+
"agent": {
547+
"ephemeral_id": "8039c627-ea96-4027-8751-2ff7db77251b",
548+
"id": "9106f11b-d54d-46d0-8ace-39e4fff1157b",
549+
"name": "elastic-agent-41888",
550+
"type": "filebeat",
551+
"version": "8.18.0"
552+
},
553+
"data_stream": {
554+
"dataset": "ti_abusech.url",
555+
"namespace": "49664",
556+
"type": "logs"
557+
},
558+
"ecs": {
559+
"version": "8.11.0"
560+
},
561+
"elastic_agent": {
562+
"id": "9106f11b-d54d-46d0-8ace-39e4fff1157b",
563+
"snapshot": true,
564+
"version": "8.18.0"
565+
},
566+
"event": {
567+
"agent_id_status": "verified",
568+
"category": [
569+
"threat"
570+
],
571+
"dataset": "ti_abusech.url",
572+
"ingested": "2025-07-16T06:32:44Z",
573+
"kind": "enrichment",
574+
"original": "{\"dateadded\":\"2024-03-19 11:34:09 UTC\",\"id\":\"2786904\",\"last_online\":\"2024-03-19 11:34:09 UTC\",\"reporter\":\"lrz_urlhaus\",\"tags\":[\"elf\",\"Mozi\"],\"threat\":\"malware_download\",\"url\":\"http://115.55.244.160:41619/Mozi.m\",\"url_status\":\"online\",\"urlhaus_link\":\"https://urlhaus.abuse.ch/url/2786904/\"}",
575+
"type": [
576+
"indicator"
577+
]
578+
},
579+
"input": {
580+
"type": "cel"
581+
},
582+
"labels": {
583+
"interval": "1h"
584+
},
585+
"tags": [
586+
"preserve_original_event",
587+
"forwarded",
588+
"abusech-url",
589+
"elf",
590+
"Mozi"
591+
],
592+
"threat": {
593+
"indicator": {
594+
"first_seen": "2024-03-19T11:34:09.000Z",
595+
"last_seen": "2024-03-19T11:34:09.000Z",
596+
"name": "http://115.55.244.160:41619/Mozi.m",
597+
"provider": "lrz_urlhaus",
598+
"reference": "https://urlhaus.abuse.ch/url/2786904/",
599+
"type": "url",
600+
"url": {
601+
"domain": "115.55.244.160",
602+
"extension": "m",
603+
"full": "http://115.55.244.160:41619/Mozi.m",
604+
"original": "http://115.55.244.160:41619/Mozi.m",
605+
"path": "/Mozi.m",
606+
"port": 41619,
607+
"scheme": "http"
608+
}
609+
}
610+
}
611+
}
612+
```
613+
615614
### Inputs used
616615

617616
These inputs can be used in this integration:

0 commit comments

Comments
 (0)