Update (most) obs-cloud-monitoring owned packages to format_version 3.0.0 (elastic#8041)

* run 'elastic-package format' across all obs-cloud-monitoring owned packages * remove duplicate and unsupported fields from obs-cloud-monitoring owned packages * [aws_logs] - removed dotted YAML keys from manifest The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/aws_logs * [awsfargate] - removed dotted YAML keys from manifest The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfargate * [awsfirehose] - removed dotted YAML keys from manifest The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfirehose * [azure_application_insights] - removed dotted YAML keys from manifest The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_application_insights * [azure_billing] - removed dotted YAML keys from manifest The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_billing * [gcp_metrics] - removed dotted YAML keys from manifest The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/gcp_metrics * additional manual fixes for 'azure_metrics' package * [azure_metrics] - removed dotted YAML keys from manifest The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_metrics * update PR number on affected packages * revert changes to 'azure' package - these will be done in a sepearate PR * fix weird auto-formatting introduced into embedded painless scripts in yaml files. * update awsfargate mappings for package-spec v3 compatibility * update awsfirehose mappings for package-spec v3 compatibility * update azure_application_insights validation.yml for package-spec v3 compatibility * update azure_billing validation.yml for package-spec v3 compatibility * revert changes to azure_metrics * remove ingest pipeline that seems to have somehow creeped its way into this PR by mistake

Author: tommyers-elastic

packages/aws_logs/changelog.yml

@@ -1,3 +1,8 @@
+- version: 0.6.0
+ changes:
+ - description: Update the package format_version to 3.0.0.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/8041
 - version: "0.5.1"
  changes:
  - description: Remove duplicated number_of_workers settings

packages/aws_logs/manifest.yml

@@ -1,17 +1,18 @@
-format_version: 1.0.0
+format_version: "3.0.0"
 name: aws_logs
 title: Custom AWS Logs
 description: Collect raw logs from AWS S3 or CloudWatch with Elastic Agent.
 type: integration
-version: 0.5.1
-release: beta
-license: basic
+version: "0.6.0"
 categories:
  - observability
  - custom
  - aws
 conditions:
- kibana.version: ^8.0.0
+ kibana:
+ version: ^8.0.0
+ elastic:
+ subscription: basic
 policy_templates:
  - name: aws_logs
  title: Custom AWS Logs
@@ -89,3 +90,4 @@ vars:
  description: URL to proxy connections in the form of http\[s\]://<user>:<password>@<server name/ip>:<port>
 owner:
  github: elastic/obs-cloud-monitoring
+ type: elastic

packages/awsfargate/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: 0.4.0
+ changes:
+ - description: Update the package format_version to 3.0.0.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/8041
 - version: 0.3.0
  changes:
  - description: Enable TSDB for task stats data stream. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html.

packages/awsfargate/data_stream/task_stats/fields/ecs.yml

@@ -56,4 +56,4 @@
 # agent
 - name: agent.id
  external: ecs
- dimension: true
+ dimension: true

packages/awsfargate/data_stream/task_stats/fields/fields.yml

@@ -1,6 +1,5 @@
 - name: awsfargate.task_stats
  type: group
- release: beta
  fields:
  - name: cluster_name
  type: keyword
@@ -23,8 +22,6 @@
  - name: cpu
  type: group
  fields:
- - name: core
- type: object
  - name: core.*.pct
  type: scaled_float
  metric_type: gauge
@@ -303,7 +300,7 @@
  description: |
  Total memory usage.
  - name: stats.*
- type: object
+ type: unsigned_long
  description: |
  Raw memory stats from the cgroups memory.stat interface
  - name: usage

packages/awsfargate/data_stream/task_stats/sample_event.json

@@ -6,7 +6,22 @@
  "task_known_status": "RUNNING",
  "task_desired_status": "RUNNING",
  "cpu": {
- "core": null,
+ "core": {
+ "1": {
+ "pct": 0,
+ "norm": {
+ "pct": 0
+ },
+ "ticks": 1520000000
+ },
+ "2": {
+ "pct": 0,
+ "norm": {
+ "pct": 0
+ },
+ "ticks": 1420180000000
+ }
+ },
  "kernel": {
  "norm": {
  "pct": 0

packages/awsfargate/docs/README.md

@@ -304,7 +304,6 @@ If you want to learn more about Amazon ECS metrics, take a look at the blog post
 | @timestamp | Event timestamp. | date | |
 | agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword | |
 | awsfargate.task_stats.cluster_name | Cluster name | keyword | |
-| awsfargate.task_stats.cpu.core | | object | |
 | awsfargate.task_stats.cpu.core.\*.norm.pct | Percentage of time per CPU core normalized by the number of CPU cores. | scaled_float | gauge |
 | awsfargate.task_stats.cpu.core.\*.pct | Percentage of time per CPU core. | scaled_float | gauge |
 | awsfargate.task_stats.cpu.core.\*.ticks | CPU ticks per CPU core. | long | counter |
@@ -351,7 +350,7 @@ If you want to learn more about Amazon ECS metrics, take a look at the blog post
 | awsfargate.task_stats.memory.rss.usage.max | Max memory usage. | long | counter |
 | awsfargate.task_stats.memory.rss.usage.pct | Memory usage percentage. | scaled_float | gauge |
 | awsfargate.task_stats.memory.rss.usage.total | Total memory usage. | long | gauge |
-| awsfargate.task_stats.memory.stats.\* | Raw memory stats from the cgroups memory.stat interface | object | |
+| awsfargate.task_stats.memory.stats.\* | Raw memory stats from the cgroups memory.stat interface | unsigned_long | |
 | awsfargate.task_stats.memory.usage.max | Max memory usage. | long | counter |
 | awsfargate.task_stats.memory.usage.pct | Memory usage percentage. | scaled_float | gauge |
 | awsfargate.task_stats.memory.usage.total | Total memory usage. | long | gauge |
@@ -403,7 +402,22 @@ An example event for `task_stats` looks as following:
  "task_known_status": "RUNNING",
  "task_desired_status": "RUNNING",
  "cpu": {
- "core": null,
+ "core": {
+ "1": {
+ "pct": 0,
+ "norm": {
+ "pct": 0
+ },
+ "ticks": 1520000000
+ },
+ "2": {
+ "pct": 0,
+ "norm": {
+ "pct": 0
+ },
+ "ticks": 1420180000000
+ }
+ },
  "kernel": {
  "norm": {
  "pct": 0

packages/awsfargate/manifest.yml

@@ -1,18 +1,20 @@
-format_version: 1.0.0
+format_version: "3.0.0"
 name: awsfargate
 title: AWS Fargate
-version: 0.3.0
-license: basic
+version: "0.4.0"
 description: Collects metrics from containers and tasks running on Amazon ECS clusters with Elastic Agent.
 type: integration
 categories:
  - observability
  - aws
-release: beta
 conditions:
- kibana.version: "^8.8.0"
+ kibana:
+ version: "^8.8.0"
+ elastic:
+ subscription: basic
 owner:
  github: elastic/obs-cloud-monitoring
+ type: elastic
 screenshots:
  - src: /img/awsfargate-integration-overview.png
  title: awsfargate integration overview

packages/awsfargate/validation.yml

@@ -0,0 +1,3 @@
+errors:
+ exclude_checks:
+ - SVR00002 # expected filter in dashboard: no filter found

packages/awsfirehose/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: 0.5.0
+ changes:
+ - description: Update the package format_version to 3.0.0.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/8041
 - version: 0.4.0
  changes:
  - description: Add routing rules for cloudfront log, elb log and s3access log.