Skip to content

[hadoop][cluster_metrics] Add cluster_metrics data stream for hadoop #2953

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
May 3, 2022
Merged

[hadoop][cluster_metrics] Add cluster_metrics data stream for hadoop #2953

merged 13 commits into from
May 3, 2022

Conversation

@yug-rajani
Copy link
Contributor

@yug-rajani yug-rajani commented Mar 31, 2022

What does this PR do?

  • Generated the skeleton of Hadoop integration package.
  • Added 1 data stream ( Cluster Metrics )
  • Added data collection logic.
  • Added the ingest pipelines.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
  • Added system test cases.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^8.0.0).

How to test this PR locally

  • Clone integrations repo.
  • Install elastic-package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/hadoop directory.
  • Run the following command to run tests.

elastic-package test
@yug-rajani yug-rajani added enhancement New feature or request Team:Integrations Label for the Integrations team New Integration Issue or pull request for creating a new integration package. Integration:hadoop Hadoop labels Mar 31, 2022
@yug-rajani yug-rajani requested a review from a team as a code owner March 31, 2022 06:33
@yug-rajani yug-rajani self-assigned this Mar 31, 2022
@elasticmachine
Copy link

elasticmachine commented Mar 31, 2022

Pinging @elastic/integrations (Team:Integrations)
yug-rajani
yug-rajani commented Mar 31, 2022
View reviewed changes
@yug-rajani
Copy link
Contributor Author

yug-rajani commented Mar 31, 2022

This PR is a split of #2614 as discussed over the comment #2614 (comment)
@elasticmachine
Copy link

elasticmachine commented Mar 31, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-04-28T06:46:21.302+0000

  • Duration: 17 min 19 sec

Test stats 🧪

Test Results
Failed 0
Passed 9
Skipped 0
Total 9

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@yug-rajani yug-rajani requested a review from mtojek March 31, 2022 07:10
@yug-rajani yug-rajani mentioned this pull request Mar 31, 2022
Closed
4 tasks
mtojek
mtojek reviewed Mar 31, 2022
View reviewed changes
@yug-rajani yug-rajani marked this pull request as draft April 1, 2022 06:04
@yug-rajani yug-rajani marked this pull request as ready for review April 5, 2022 06:58
@yug-rajani yug-rajani requested a review from mtojek April 5, 2022 06:58
mtojek
mtojek reviewed Apr 5, 2022
View reviewed changes
@yug-rajani yug-rajani requested a review from a team as a code owner April 11, 2022 05:12
@cla-checker-service
Copy link

cla-checker-service bot commented Apr 11, 2022 •
edited
Loading

💚 CLA has been signed
@yug-rajani yug-rajani force-pushed the package_hadoop_cluster_metrics branch from d4d35c0 to 9eb64fa Compare April 11, 2022 05:15
@mtojek mtojek self-requested a review April 15, 2022 09:47
mtojek
mtojek suggested changes Apr 15, 2022
View reviewed changes
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yug-elastic Could you please rebase this PR?
@yug-rajani
Merge branch 'main' of https://github.com/yug-elastic/integrations in…
fd2163a 
…to package_hadoop_cluster_metrics Conflicts:	packages/hadoop/_dev/build/docs/README.md	packages/hadoop/changelog.yml	packages/hadoop/docs/README.md	packages/hadoop/manifest.yml
@yug-rajani yug-rajani requested a review from mtojek April 18, 2022 13:27
lalit-satapathy
lalit-satapathy reviewed Apr 27, 2022
View reviewed changes
packages/hadoop/data_stream/cluster/manifest.yml
show_user: false
default:
- forwarded
- name: ssl
Copy link
Contributor

@lalit-satapathy lalit-satapathy Apr 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the ssl configuration placeholder now or working/used currently?
Copy link
Contributor Author

@yug-rajani yug-rajani Apr 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a placeholder and is being used currently.
@elasticmachine
Copy link

elasticmachine commented Apr 28, 2022

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (2/2) 💚
Files 100.0% (4/4) 💚
Classes 100.0% (4/4) 💚
Methods 100.0% (22/22) 💚
Lines 98.328% (294/299) ❕
Conditionals 100.0% (0/0) 💚
@yug-rajani yug-rajani merged commit 3a8bf98 into elastic:main May 3, 2022
@yug-rajani yug-rajani linked an issue May 10, 2022 that may be closed by this pull request
Create Hadoop integration #1543
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:hadoop Hadoop New Integration Issue or pull request for creating a new integration package. Team:Integrations Label for the Integrations team

5 participants

@yug-rajani @elasticmachine @masci @mtojek @lalit-satapathy