- Notifications
You must be signed in to change notification settings - Fork 513
[AmazonMQ] Add RabbitMQ metrics dataset #12924
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits Select commit Hold shift + click to select a range
87f26d9 AmazonMQ Add RabbitMQ metrics dataset
agithomas cf5102e Update PR link
agithomas 0cc12d5 Updated the aggregation for RabbitMQDiskFree
agithomas bc50d70 Minor issue fixes
agithomas 3903e8d Adjusted the RabbitMQDiskFree as minimum aggregation
agithomas ec875a1 Fixed typo
agithomas 04d9912 Updated broker field mappings
agithomas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
77 changes: 77 additions & 0 deletions 77 packages/aws_mq/data_stream/rabbitmq_metrics/agent/stream/stream.yml.hbs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| metricsets: ["cloudwatch"] | ||
| period: {{period}} | ||
| {{#if data_granularity}} | ||
| data_granularity: {{data_granularity}} | ||
| {{/if}} | ||
| {{#if include_linked_accounts}} | ||
| include_linked_accounts: {{include_linked_accounts}} | ||
| {{#if owning_account}} | ||
| owning_account: "{{owning_account}}" | ||
| {{/if}} | ||
| {{/if}} | ||
| {{#if access_key_id}} | ||
| access_key_id: {{access_key_id}} | ||
| {{/if}} | ||
| {{#if secret_access_key}} | ||
| secret_access_key: {{secret_access_key}} | ||
| {{/if}} | ||
| {{#if session_token}} | ||
| session_token: {{session_token}} | ||
| {{/if}} | ||
| {{#if credential_profile_name}} | ||
| credential_profile_name: {{credential_profile_name}} | ||
| {{/if}} | ||
| {{#if shared_credential_file}} | ||
| shared_credential_file: {{shared_credential_file}} | ||
| {{/if}} | ||
| {{#if role_arn}} | ||
| role_arn: {{role_arn}} | ||
| {{/if}} | ||
| {{#if default_region}} | ||
| default_region: {{default_region}} | ||
| {{/if}} | ||
| {{#if regions}} | ||
| regions: | ||
| {{#each regions as |region|}} | ||
| - {{region}} | ||
| {{/each}} | ||
| {{/if}} | ||
| {{#if latency}} | ||
| latency: {{latency}} | ||
| {{/if}} | ||
| {{#if tags_filter}} | ||
| tags_filter: {{tags_filter}} | ||
| {{/if}} | ||
| {{#if proxy_url }} | ||
| proxy_url: {{proxy_url}} | ||
| {{/if}} | ||
| {{#if processors}} | ||
| processors: | ||
| {{processors}} | ||
| {{/if}} | ||
| metrics: | ||
| - namespace: AWS/AmazonMQ | ||
| statistic: ["Maximum"] | ||
| name: | ||
| - ExchangeCount | ||
| - QueueCount | ||
| - ConnectionCount | ||
| - ChannelCount | ||
| - ConsumerCount | ||
| - MessageCount | ||
| - MessageReadyCount | ||
| - MessageUnacknowledgedCount | ||
| - PublishRate | ||
| - ConfirmRate | ||
| - AckRate | ||
| - SystemCpuUtilization | ||
| - RabbitMQMemLimit | ||
| - RabbitMQMemUsed | ||
| - RabbitMQDiskFreeLimit | ||
| - RabbitMQFdUsed | ||
| - RabbitMQIOReadAverageTime | ||
| - RabbitMQIOWriteAverageTime | ||
| - namespace: AWS/AmazonMQ | ||
| statistic: ["Minimum"] | ||
| name: | ||
| - RabbitMQDiskFree | ||
45 changes: 45 additions & 0 deletions 45 packages/aws_mq/data_stream/rabbitmq_metrics/elasticsearch/ingest_pipeline/default.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| --- | ||
| description: Pipeline of RabbitMQ metrics | ||
| processors: | ||
| - dot_expander: | ||
| field: "*" | ||
| ignore_failure: true | ||
| - drop: | ||
| description: "To drop the documents having RabbitMQ metrics" | ||
| if: > | ||
| ctx.aws?.amazonmq?.metrics?.size() == 1 && | ||
| ctx.aws.amazonmq.metrics.ConsumerCount?.max != null | ||
| ignore_failure: true | ||
gpop63 marked this conversation as resolved. Show resolved Hide resolved | ||
| - rename: | ||
| field: "aws.amazonmq.metrics" | ||
| target_field: "aws.amazonmq.metrics.rabbitmq.broker" | ||
| ignore_missing: true | ||
| if: > | ||
| ctx.aws?.dimensions?.Broker != null && | ||
| (ctx.aws?.dimensions?.Node == null && ctx.aws?.dimensions?.Queue == null ) | ||
| - rename: | ||
| field: "aws.amazonmq.metrics" | ||
| target_field: "aws.amazonmq.metrics.rabbitmq.node" | ||
| ignore_missing: true | ||
| if: > | ||
| ctx.aws?.dimensions?.Broker != null && | ||
| (ctx.aws?.dimensions?.Node != null ) | ||
| - rename: | ||
| field: "aws.amazonmq.metrics" | ||
| target_field: "aws.amazonmq.metrics.rabbitmq.queue" | ||
| ignore_missing: true | ||
| if: > | ||
| ctx.aws?.dimensions?.Broker != null && | ||
| (ctx.aws?.dimensions?.Queue != null ) | ||
| | ||
| on_failure: | ||
| - set: | ||
| field: event.kind | ||
| value: pipeline_error | ||
| - append: | ||
| field: tags | ||
| value: preserve_original_event | ||
| allow_duplicates: false | ||
| - set: | ||
| field: error.message | ||
| value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' | ||
16 changes: 16 additions & 0 deletions 16 packages/aws_mq/data_stream/rabbitmq_metrics/fields/base-fields.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| - name: data_stream.type | ||
| type: constant_keyword | ||
| description: Data stream type. | ||
| - name: data_stream.dataset | ||
| type: constant_keyword | ||
| description: Data stream dataset. | ||
| - name: data_stream.namespace | ||
| type: constant_keyword | ||
| description: Data stream namespace. | ||
| - name: '@timestamp' | ||
| type: date | ||
| description: Event timestamp. | ||
| - name: event.module | ||
| type: constant_keyword | ||
| external: ecs | ||
| value: aws |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| - external: ecs | ||
| name: cloud.account.id | ||
| dimension: true | ||
| - external: ecs | ||
| name: cloud.region | ||
| dimension: true | ||
| - name: agent.id | ||
| external: ecs | ||
| dimension: true |
176 changes: 176 additions & 0 deletions 176 packages/aws_mq/data_stream/rabbitmq_metrics/fields/fields.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,176 @@ | ||
| - name: aws | ||
| type: group | ||
| fields: | ||
| - name: dimensions | ||
| type: group | ||
| fields: | ||
| - name: Broker | ||
| description: The name of the broker. | ||
| type: keyword | ||
| dimension: true | ||
| - name: Node | ||
| description: The name of the node. | ||
| type: keyword | ||
| dimension: true | ||
| - name: Queue | ||
| description: The name of the queue. | ||
| type: keyword | ||
| dimension: true | ||
| - name: VirtualHost | ||
| description: The name of the virtual host. | ||
| type: keyword | ||
| dimension: true | ||
| - name: cloudwatch.namespace | ||
| type: keyword | ||
| description: The namespace specified when query cloudwatch api. | ||
agithomas marked this conversation as resolved. Show resolved Hide resolved | ||
| - name: amazonmq.metrics.rabbitmq | ||
| type: group | ||
| fields: | ||
| - name: broker | ||
| type: group | ||
| fields: | ||
| - name: ExchangeCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of exchanges configured on the broker. | ||
| - name: QueueCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of queues configured on the broker. | ||
| - name: ConnectionCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of connections established on the broker. | ||
| - name: ChannelCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of channels established on the broker. | ||
| - name: ConsumerCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of consumers connected to the broker. | ||
| - name: MessageCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of messages in the queues. | ||
| - name: MessageReadyCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of ready messages in the queues. | ||
| - name: MessageUnacknowledgedCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of unacknowledged messages in the queues. | ||
| - name: PublishRate.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The rate at which messages are published to the broker. | ||
| - name: ConfirmRate.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The rate at which the RabbitMQ server is confirming published messages. | ||
| - name: AckRate.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The rate at which messages are being acknowledged by consumers. | ||
| - name: SystemCpuUtilization.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: percent | ||
| description: The percentage of allocated Amazon EC2 compute units that the broker currently uses. For cluster deployments, this value represents the aggregate of all three RabbitMQ nodes' corresponding metric values. | ||
| - name: RabbitMQMemLimit.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The RAM limit for a RabbitMQ broker. For cluster deployments, this value represents the aggregate of all RabbitMQ nodes' corresponding metric values. | ||
| - name: RabbitMQMemUsed.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The volume of RAM used by a RabbitMQ broker. For cluster deployments, this value represents the aggregate of all RabbitMQ nodes' corresponding metric values. | ||
| - name: RabbitMQDiskFreeLimit.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The disk limit for a RabbitMQ broker. For cluster deployments, this value represents the aggregate of all RabbitMQ nodes' corresponding metric values. | ||
| - name: RabbitMQDiskFree.min | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The total volume of free disk space available in a RabbitMQ broker. For cluster deployments, this value represents the aggregate of all RabbitMQ nodes' corresponding metric values. | ||
| - name: RabbitMQFdUsed.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The number of file descriptors used. For cluster deployments, this value represents the aggregate of all RabbitMQ nodes' corresponding metric values. | ||
| - name: RabbitMQIOReadAverageTime.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: ms | ||
| description: The average time for RabbitMQ to perform one read operation. | ||
| - name: RabbitMQIOWriteAverageTime.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: ms | ||
| description: The average time for RabbitMQ to perform one write operation. | ||
| - name: node | ||
| type: group | ||
| fields: | ||
| - name: SystemCpuUtilization.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: percent | ||
| description: The percentage of allocated Amazon EC2 compute units that the broker currently uses. | ||
| - name: RabbitMQMemLimit.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The RAM limit for a RabbitMQ node. | ||
| - name: RabbitMQMemUsed.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The volume of RAM used by a RabbitMQ node. | ||
| - name: RabbitMQDiskFreeLimit.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The disk limit for a RabbitMQ node. | ||
| - name: RabbitMQDiskFree.min | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: The total volume of free disk space available in a RabbitMQ node. | ||
| - name: RabbitMQFdUsed.max | ||
agithomas marked this conversation as resolved. Show resolved Hide resolved | ||
| metric_type: gauge | ||
| type: long | ||
| unit: byte | ||
| description: Number of file descriptors used. | ||
| - name: RabbitMQIOReadAverageTime.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: ms | ||
| description: The average time for RabbitMQ to perform one read operation. | ||
| - name: RabbitMQIOWriteAverageTime.max | ||
| metric_type: gauge | ||
| type: long | ||
| unit: ms | ||
| description: The average time for RabbitMQ to perform one write operation. | ||
| - name: queue | ||
| type: group | ||
| fields: | ||
| - name: ConsumerCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The number of consumers subscribed to the queue. | ||
| - name: MessageReadyCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The number of messages that are currently available to be delivered. | ||
| - name: MessageUnacknowledgedCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The number of messages for which the server is awaiting acknowledgement. | ||
| - name: MessageCount.max | ||
| metric_type: gauge | ||
| type: long | ||
| description: The total number of MessageReadyCount and MessageUnacknowledgedCount, referred to as queue depth. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.