[System][Core] Add dimensions for the core data_stream

packages/system/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.34.1"
+ changes:
+ - description: Add dimension fields to the core data_streams
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/6454
 - version: "1.34.0"
  changes:
  - description: Enable time series data streams for the metrics datasets, except core and process datasets. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html

packages/system/data_stream/core/fields/agent.yml

@@ -13,18 +13,21 @@
 
  Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.'
  example: 666777888999
+ dimension: true
  - name: availability_zone
  level: extended
  type: keyword
  ignore_above: 1024
  description: Availability zone in which this host is running.
  example: us-east-1c
+ dimension: true
  - name: instance.id
  level: extended
  type: keyword
  ignore_above: 1024
  description: Instance ID of the host machine.
  example: i-1234567890abcdef0
+ dimension: true
  - name: instance.name
  level: extended
  type: keyword
@@ -42,12 +45,14 @@
  ignore_above: 1024
  description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
  example: aws
+ dimension: true
  - name: region
  level: extended
  type: keyword
  ignore_above: 1024
  description: Region in which this host is running.
  example: us-east-1
+ dimension: true
  - name: project.id
  type: keyword
  description: Name of the project in Google Cloud.
@@ -66,6 +71,7 @@
  level: core
  type: keyword
  ignore_above: 1024
+ dimension: true
  description: Unique container id.
  - name: image.name
  level: extended
@@ -90,12 +96,6 @@
  ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.'
  type: group
  fields:
- - name: architecture
- level: core
- type: keyword
- ignore_above: 1024
- description: Operating system architecture.
- example: x86_64
  - name: domain
  level: extended
  type: keyword
@@ -121,64 +121,6 @@
  As hostname is not always unique, use values that are meaningful in your environment.
 
  Example: The current usage of `beat.name`.'
- - name: ip
- level: core
- type: ip
- description: Host ip addresses.
- - name: mac
- level: core
- type: keyword
- ignore_above: 1024
- description: Host mac addresses.
- - name: name
- level: core
- type: keyword
- ignore_above: 1024
- description: 'Name of the host.
-
- It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.'
- - name: os.family
- level: extended
- type: keyword
- ignore_above: 1024
- description: OS family (such as redhat, debian, freebsd, windows).
- example: debian
- - name: os.kernel
- level: extended
- type: keyword
- ignore_above: 1024
- description: Operating system kernel version as a raw string.
- example: 4.4.0-112-generic
- - name: os.name
- level: extended
- type: keyword
- ignore_above: 1024
- multi_fields:
- - name: text
- type: text
- norms: false
- default_field: false
- description: Operating system name, without the version.
- example: Mac OS X
- - name: os.platform
- level: extended
- type: keyword
- ignore_above: 1024
- description: Operating system platform (such centos, ubuntu, windows).
- example: darwin
- - name: os.version
- level: extended
- type: keyword
- ignore_above: 1024
- description: Operating system version as a raw string.
- example: 10.14.1
- - name: type
- level: core
- type: keyword
- ignore_above: 1024
- description: 'Type of host.
-
- For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.'
  - name: containerized
  type: boolean
  description: >

packages/system/data_stream/core/fields/ecs.yml

@@ -8,6 +8,7 @@
  name: host.mac
 - external: ecs
  name: host.name
+ dimension: true
 - external: ecs
  name: host.os.family
 - external: ecs
@@ -22,3 +23,6 @@
  name: host.os.version
 - external: ecs
  name: host.type
+- external: ecs
+ name: agent.id
+ dimension: true

packages/system/data_stream/core/fields/fields.yml

@@ -3,6 +3,7 @@
  fields:
  - name: id
  type: keyword
+ dimension: true
  description: |
  CPU Core number.
  - name: user.pct

packages/system/docs/README.md

@@ -1155,6 +1155,7 @@ This data should be available without elevated permissions.
 | Field | Description | Type | Unit | Metric Type |
 |---|---|---|---|---|
 | @timestamp | Event timestamp. | date | | |
+| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword | | |
 | cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | |
 | cloud.availability_zone | Availability zone in which this host is running. | keyword | | |
 | cloud.image.id | Image ID for the cloud instance. | keyword | | |
@@ -1180,7 +1181,7 @@ This data should be available without elevated permissions.
 | host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | |
 | host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | |
 | host.ip | Host ip addresses. | ip | | |
-| host.mac | Host mac addresses. | keyword | | |
+| host.mac | Host MAC addresses. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen. | keyword | | |
 | host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | |
 | host.os.build | OS build information. | keyword | | |
 | host.os.codename | OS codename, if any. | keyword | | |
@@ -1189,7 +1190,7 @@ This data should be available without elevated permissions.
 | host.os.full.text | Multi-field of `host.os.full`. | match_only_text | | |
 | host.os.kernel | Operating system kernel version as a raw string. | keyword | | |
 | host.os.name | Operating system name, without the version. | keyword | | |
-| host.os.name.text | Multi-field of `host.os.name`. | text | | |
+| host.os.name.text | Multi-field of `host.os.name`. | match_only_text | | |
 | host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | |
 | host.os.version | Operating system version as a raw string. | keyword | | |
 | host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | |

packages/system/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 1.0.0
 name: system
 title: System
-version: 1.34.0
+version: 1.34.1
 license: basic
 description: Collect system logs and metrics from your servers with Elastic Agent.
 type: integration