Skip to content

[AWS] Add EMR metrics data stream #6120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Jun 16, 2023
Merged

[AWS] Add EMR metrics data stream #6120

merged 13 commits into from
Jun 16, 2023

Conversation

@gpop63
Copy link
Contributor

@gpop63 gpop63 commented May 9, 2023 •
edited
Loading

What does this PR do?

Adds AWS EMR (ElasticMapReduce) metrics data stream.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots
@elasticmachine
Copy link

elasticmachine commented May 9, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-06-15T12:28:22.305+0000

  • Duration: 52 min 0 sec

Test stats 🧪

Test Results
Failed 0
Passed 192
Skipped 4
Total 196

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@elasticmachine
Copy link

elasticmachine commented May 9, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (15/15) 💚
Files 93.75% (15/16) 👎 -6.25
Classes 93.75% (15/16) 👎 -6.25
Methods 85.714% (240/280) 👍 60.714
Lines 85.925% (7387/8597) 👎 -14.075
Conditionals 100.0% (0/0) 💚
@gpop63 gpop63 force-pushed the add_aws-emr_metrics branch from 5f0cef1 to 9b91ea9 Compare May 15, 2023 18:03
@gpop63 gpop63 force-pushed the add_aws-emr_metrics branch from 9b91ea9 to b6a4e7d Compare May 15, 2023 21:16
@gpop63 gpop63 marked this pull request as ready for review May 16, 2023 00:30
@gpop63 gpop63 requested a review from a team as a code owner May 16, 2023 00:30
ritalwar
ritalwar reviewed May 16, 2023
View reviewed changes
Copy link
Contributor

@ritalwar ritalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please define the metric type as either counter or gauge for applicable fields? This would be beneficial from a TSDB perspective in the future.
@gpop63 gpop63 force-pushed the add_aws-emr_metrics branch from 6dc09e7 to 694620e Compare May 29, 2023 23:40
@gpop63 gpop63 force-pushed the add_aws-emr_metrics branch from a32b736 to bd23a90 Compare May 29, 2023 23:42
@gpop63 gpop63 requested a review from ritalwar May 30, 2023 14:29
lucian-ioan
lucian-ioan reviewed Jun 6, 2023
View reviewed changes
packages/aws/_dev/build/docs/emr.md
Comment on lines +1 to +7
# Amazon EMR (ElasticMapReduce)

The Amazon EMR integration allows you to monitor [Amazon EMR](https://aws.amazon.com/emr/) — a fully managed big data processing and analytics service.

Use the Amazon EMR integration to collect metrics related to your EMR instances. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference the metrics when troubleshooting an issue.

For example, you could use this data to track Amazon EMR cluster progress and cluster storage. Then you can alert when utilization for an instance crosses a predefined threshold.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about clarifying the Hadoop version for this data stream in this doc?

Idea: Creating a meta issue for AWS integrations that we own, to improve documentation clarity. What are your thoughts on this? @zmoog @lalit-satapathy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is a good idea. @lucianpy, could you create an issue with a detailed list of information to add to this integration?

This way, we can ship the integration as it is and start working on an actionable list right after the current iteration ends.
zmoog
zmoog reviewed Jun 14, 2023
View reviewed changes
packages/aws/data_stream/emr_metrics/agent/stream/stream.yml.hbs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recently we added a new cloudwatch metricset setting called include_linked_accounts to collect data from linked account. I guess we could add this to this integration as well.

See #6459 for more context about this topic.
@gpop63
Copy link
Contributor Author

gpop63 commented Jun 15, 2023

/test
@elasticmachine
Copy link

💚 Build Succeeded

History

cc @gpop63
zmoog
zmoog approved these changes Jun 16, 2023
View reviewed changes
Copy link
Contributor

@zmoog zmoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, @gpop63! Could you create an issue and add the steps required to move this integration from BETA to GA? Many thanks!
@gpop63 gpop63 merged commit a2f0dd9 into elastic:main Jun 16, 2023
@elasticmachine
Copy link

Package aws - 1.45.0 containing this change is available at https://epr.elastic.co/search?package=aws
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:aws AWS

6 participants

@gpop63 @elasticmachine @zmoog @lucian-ioan @ritalwar @andrewkroh