[rubrik] Add Monitoring Jobs datastream

[devamanv]

Proposed commit message

The PR adds a new Job Monitoring data stream to the new developed Rubrik package. This fetches job summary information for protection and recovery jobs separated by job type that have been scheduled, are currently running, or completed in the past 24 hours. Following are the metrics collected:

• total count of all archival jobs
• total count of all backup jobs
• total count of all conversion jobs
• total count of all log backup jobs
• total count of all recovery jobs
• total count of all replication jobs
• total count of all jobs

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes https://github.com/elastic/obs-integration-team/issues/124

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[gpop63]

LGTM left some small questions/suggestions. README is also missing a metrics section for this new data stream.

[muthu-mps]

Please remove the ignore_failure and ignore_empty_value for ecs.version and event.kind processors in the managed volumes.

[muthu-mps]

@devamanv - The metrics includes the count for the jobs. Is there a way we could associate this metrics with the Rubrik cluster name or id?
Including sample-events file to the data stream would help on the above. Is it not possible to add system tests to the package?

[devamanv]

@muthu-mps it's possible to gather cluster wide metrics to know how many jobs of a certain kind are running on that cluster, something like this:

 { "name": "cluster1", "id": "b1a9e557-0b47-4bc1-841d-e6379e733016", "archiveJobs": { "count": 24 }, "backupJobs": { "count": 646 }, "conversionJobs": { "count": 0 }, "logBackupJobs": { "count": 0 }, "recoveryJobs": { "count": 5 }, "replicationJobs": { "count": 15 } }

Is there a way we could associate this metrics with the Rubrik cluster name or id?

This would require a change in the query, but is possible. Does that align with what you suggested?

[muthu-mps]

@muthu-mps it's possible to gather cluster wide metrics to know how many jobs of a certain kind are running on that cluster, something like this:

Is there a way we could associate this metrics with the Rubrik cluster name or id?

This would require a change in the query, but is possible. Does that align with what you suggested?

Yes @devamanv. Adding cluster metadata is essential for monitoring each cluster.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f573f91fbc61ec81b9ae1a12ba11f8642e979265

History

• 💔 Build #20182 failed bfc6ce1dbec4b36e9e74e60726c01ad1c308acfb
• 💚 Build #20181 succeeded 97983aadeb78aae32e729e0e1744df87d053a792
• 💔 Build #20177 failed 19786bac317582ddbf7548663782c65e977373e8
• 💔 Build #20174 failed 8eb875c4ef82f8326ade5ca93626f23112aa8b26
• 💚 Build #20053 succeeded 0ac5aaebc85a31bd13db1e6e7fc1c077b483dc43

cc @devamanv

[elastic-sonarqube]

Quality Gate failed

Failed conditions
60.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[muthu-mps]

LGTM!

[elastic-vault-github-plugin-prod]

Package rubrik - 0.2.0 containing this change is available at https://epr.elastic.co/package/rubrik/0.2.0/