Added metric_type mapping for datastream consumer_group

Author: agithomas

packages/kafka/changelog.yml

@@ -1,4 +1,14 @@
 # newer versions go on top
+- version: "1.5.1"
+ changes:
+ - description: Add metric_type mapping for consumer_group datastream.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/5925
+- version: "1.5.0"
+ changes:
+ - description: Add metric_type mapping for broker datastream.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/5923
 - version: "1.4.1"
  changes:
  - description: Added categories and/or subcategories.

packages/kafka/data_stream/consumergroup/fields/fields.yml

@@ -6,12 +6,14 @@
  description: Consumer Group ID
  - name: offset
  type: long
+ metric_type: gauge
  description: consumer offset into partition being read
  - name: meta
  type: keyword
  description: custom consumer meta data string
  - name: consumer_lag
  type: long
+ metric_type: gauge
  description: consumer lag for partition/topic calculated as the difference between the partition offset and consumer offset
  - name: error.code
  type: long

packages/kafka/docs/README.md

@@ -129,13 +129,21 @@ An example event for `broker` looks as following:
 | Field | Description | Type |
 |---|---|---|
 | @timestamp | Event timestamp. | date |
+<<<<<<< HEAD
+=======
+| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
+>>>>>>> 53136fb29... Added metric_type mapping for datastream consumer_group
 | cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |
 | cloud.availability_zone | Availability zone in which this host is running. | keyword |
 | cloud.image.id | Image ID for the cloud instance. | keyword |
 | cloud.instance.id | Instance ID of the host machine. | keyword |
 | cloud.instance.name | Instance name of the host machine. | keyword |
 | cloud.machine.type | Machine type of the host machine. | keyword |
+<<<<<<< HEAD
 | cloud.project.id | Name of the project in Google Cloud. | keyword |
+=======
+| cloud.project.id | The cloud project identifier. Examples: Google Cloud Project id, Azure Project id. | keyword |
+>>>>>>> 53136fb29... Added metric_type mapping for datastream consumer_group
 | cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
 | cloud.region | Region in which this host is running. | keyword |
 | container.id | Unique container id. | keyword |
@@ -259,62 +267,62 @@ An example event for `consumergroup` looks as following:
 
 **Exported fields**
 
-| Field | Description | Type |
-|---|---|---|
-| @timestamp | Event timestamp. | date |
-| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |
-| cloud.availability_zone | Availability zone in which this host is running. | keyword |
-| cloud.image.id | Image ID for the cloud instance. | keyword |
-| cloud.instance.id | Instance ID of the host machine. | keyword |
-| cloud.instance.name | Instance name of the host machine. | keyword |
-| cloud.machine.type | Machine type of the host machine. | keyword |
-| cloud.project.id | Name of the project in Google Cloud. | keyword |
-| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
-| cloud.region | Region in which this host is running. | keyword |
-| container.id | Unique container id. | keyword |
-| container.image.name | Name of the image the container was built on. | keyword |
-| container.labels | Image labels. | object |
-| container.name | Container name. | keyword |
-| data_stream.dataset | Data stream dataset. | constant_keyword |
-| data_stream.namespace | Data stream namespace. | constant_keyword |
-| data_stream.type | Data stream type. | constant_keyword |
-| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |
-| event.dataset | Event dataset | constant_keyword |
-| event.module | Event module | constant_keyword |
-| host.architecture | Operating system architecture. | keyword |
-| host.containerized | If the host is a container. | boolean |
-| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |
-| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |
-| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |
-| host.ip | Host ip addresses. | ip |
-| host.mac | Host mac addresses. | keyword |
-| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |
-| host.os.build | OS build information. | keyword |
-| host.os.codename | OS codename, if any. | keyword |
-| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |
-| host.os.kernel | Operating system kernel version as a raw string. | keyword |
-| host.os.name | Operating system name, without the version. | keyword |
-| host.os.name.text | Multi-field of `host.os.name`. | text |
-| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |
-| host.os.version | Operating system version as a raw string. | keyword |
-| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword |
-| kafka.broker.address | Broker advertised address | keyword |
-| kafka.broker.id | Broker id | long |
-| kafka.consumergroup.client.host | Client host | keyword |
-| kafka.consumergroup.client.id | Client ID (kafka setting client.id) | keyword |
-| kafka.consumergroup.client.member_id | internal consumer group member ID | keyword |
-| kafka.consumergroup.consumer_lag | consumer lag for partition/topic calculated as the difference between the partition offset and consumer offset | long |
-| kafka.consumergroup.error.code | kafka consumer/partition error code. | long |
-| kafka.consumergroup.id | Consumer Group ID | keyword |
-| kafka.consumergroup.meta | custom consumer meta data string | keyword |
-| kafka.consumergroup.offset | consumer offset into partition being read | long |
-| kafka.partition.id | Partition id. | long |
-| kafka.partition.topic_broker_id | Unique id of the partition in the topic and the broker. | keyword |
-| kafka.partition.topic_id | Unique id of the partition in the topic. | keyword |
-| kafka.topic.error.code | Topic error code. | long |
-| kafka.topic.name | Topic name | keyword |
-| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword |
-| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword |
+| Field | Description | Type | Metric Type |
+|---|---|---|---|
+| @timestamp | Event timestamp. | date | |
+| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
+| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
+| cloud.image.id | Image ID for the cloud instance. | keyword | |
+| cloud.instance.id | Instance ID of the host machine. | keyword | |
+| cloud.instance.name | Instance name of the host machine. | keyword | |
+| cloud.machine.type | Machine type of the host machine. | keyword | |
+| cloud.project.id | Name of the project in Google Cloud. | keyword | |
+| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
+| cloud.region | Region in which this host is running. | keyword | |
+| container.id | Unique container id. | keyword | |
+| container.image.name | Name of the image the container was built on. | keyword | |
+| container.labels | Image labels. | object | |
+| container.name | Container name. | keyword | |
+| data_stream.dataset | Data stream dataset. | constant_keyword | |
+| data_stream.namespace | Data stream namespace. | constant_keyword | |
+| data_stream.type | Data stream type. | constant_keyword | |
+| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | |
+| event.dataset | Event dataset | constant_keyword | |
+| event.module | Event module | constant_keyword | |
+| host.architecture | Operating system architecture. | keyword | |
+| host.containerized | If the host is a container. | boolean | |
+| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | |
+| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | |
+| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | |
+| host.ip | Host ip addresses. | ip | |
+| host.mac | Host mac addresses. | keyword | |
+| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | |
+| host.os.build | OS build information. | keyword | |
+| host.os.codename | OS codename, if any. | keyword | |
+| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | |
+| host.os.kernel | Operating system kernel version as a raw string. | keyword | |
+| host.os.name | Operating system name, without the version. | keyword | |
+| host.os.name.text | Multi-field of `host.os.name`. | text | |
+| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | |
+| host.os.version | Operating system version as a raw string. | keyword | |
+| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | |
+| kafka.broker.address | Broker advertised address | keyword | |
+| kafka.broker.id | Broker id | long | |
+| kafka.consumergroup.client.host | Client host | keyword | |
+| kafka.consumergroup.client.id | Client ID (kafka setting client.id) | keyword | |
+| kafka.consumergroup.client.member_id | internal consumer group member ID | keyword | |
+| kafka.consumergroup.consumer_lag | consumer lag for partition/topic calculated as the difference between the partition offset and consumer offset | long | gauge |
+| kafka.consumergroup.error.code | kafka consumer/partition error code. | long | |
+| kafka.consumergroup.id | Consumer Group ID | keyword | |
+| kafka.consumergroup.meta | custom consumer meta data string | keyword | |
+| kafka.consumergroup.offset | consumer offset into partition being read | long | gauge |
+| kafka.partition.id | Partition id. | long | |
+| kafka.partition.topic_broker_id | Unique id of the partition in the topic and the broker. | keyword | |
+| kafka.partition.topic_id | Unique id of the partition in the topic. | keyword | |
+| kafka.topic.error.code | Topic error code. | long | |
+| kafka.topic.name | Topic name | keyword | |
+| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
+| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | |
 
 
 ### partition