Make ecs.version a constant_keyword field

[axw]

Motivation/summary

Instead of adding ecs.version as a field in every document written by APM Server, make it a constant_keyword field and set its version in the data stream mappings. This means that ecs.version will not be available in _source, and will only be available through the Elasticsearch fields API. Nothing uses ecs.version yet, so it doesn't matter in practice.

The data stream's ingest pipeline is responsible for upgrading documents to the expected format, so it follows that the integration package (ingest pipeline, mappings) should define the ECS version. Given that nothing currently relies on ecs.version, the pipelines now simply drop any ecs.version field defined in the incoming document, in favour of the constant_keyword value.

We're currently pinning to a recent commit of ECS due to our reliance on the unreleased device.* fields. When they are released, we should update to the released version.

Checklist

• Update CHANGELOG.asciidoc
• Update package changelog.yml (only if changes to apmpackage have been made)
  - [ ] Documentation has been updated

How to test these changes

1. Install the integration, send some data to APM Server
2. Ensure the documents have ecs.version set

Related issues

Closes #9043

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-28T09:04:14.921+0000

• Duration: 29 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	149
Skipped	0
Total	149

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ghost]

📚 Go benchmark report

Diff with the main branch

name old time/op new time/op delta pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64 FetchAndAdd/FetchFromCache-12 41.4ns ± 2% 46.3ns ± 1% +11.71% (p=0.008 n=5+5) FetchAndAdd/FetchAndAddToCache-12 96.7ns ± 3% 106.4ns ± 1% +10.02% (p=0.008 n=5+5) pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64 ContextResetContentEncoding/empty-12 118ns ± 0% 137ns ± 2% +15.78% (p=0.016 n=4+5) ContextResetContentEncoding/uncompressed-12 141ns ± 1% 163ns ± 1% +15.83% (p=0.008 n=5+5) pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64 ModelIndexer/BestCompression-12 14.8µs ±51% 8.4µs ± 3% -43.12% (p=0.008 n=5+5) pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64 BackendProcessor/events.ndjson-12 178µs ±18% 130µs ±19% -27.04% (p=0.032 n=5+5) RUMV3Processor/rum_errors.ndjson-12 9.25µs ±12% 7.21µs ±12% -22.11% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel2/invalid-json-event.ndjson-12 3.81µs ± 6% 4.04µs ± 4% +6.04% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel2/span-links.ndjson-12 3.87µs ± 2% 4.13µs ± 4% +6.86% (p=0.016 n=4+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel4/logs.ndjson-12 4.23µs ±12% 4.82µs ± 6% +13.96% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/otel-bridge.ndjson-12 3.94µs ± 1% 3.79µs ± 1% -3.73% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/ratelimit.ndjson-12 6.68µs ± 1% 6.18µs ± 1% -7.48% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/span-links.ndjson-12 1.61µs ± 1% 1.53µs ± 0% -4.45% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/spans.ndjson-12 11.5µs ± 1% 10.9µs ± 1% -5.17% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12 5.72µs ± 1% 5.55µs ± 2% -2.96% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12 1.99µs ± 0% 1.96µs ± 1% -1.56% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12 7.48µs ± 2% 7.30µs ± 1% -2.42% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12 7.60µs ± 1% 7.40µs ± 2% -2.59% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12 7.05µs ± 2% 6.82µs ± 2% -3.24% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/events.ndjson-12 13.7µs ± 1% 13.1µs ± 1% -4.41% (p=0.016 n=5+4) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/heavy.ndjson-12 535µs ± 2% 515µs ± 3% -3.86% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12 3.48µs ± 1% 3.36µs ± 1% -3.39% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-event.ndjson-12 1.14µs ± 1% 1.08µs ± 1% -5.65% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12 1.98µs ± 1% 1.89µs ± 0% -4.28% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/logs.ndjson-12 1.95µs ± 1% 1.90µs ± 2% -2.74% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metadata.ndjson-12 1.30µs ± 3% 1.27µs ± 1% -2.02% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal-service.ndjson-12 988ns ± 2% 966ns ± 1% -2.16% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal.ndjson-12 2.03µs ± 3% 1.95µs ± 1% -3.80% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12 1.48µs ± 1% 1.45µs ± 2% -2.44% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/otel-bridge.ndjson-12 3.29µs ± 2% 3.20µs ± 1% -2.93% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/ratelimit.ndjson-12 5.81µs ± 2% 5.58µs ± 1% -3.91% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12 1.21µs ± 1% 1.17µs ± 1% -2.83% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions.ndjson-12 9.81µs ± 4% 9.30µs ± 2% -5.22% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans.ndjson-12 9.88µs ± 4% 9.36µs ± 3% -5.34% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum.ndjson-12 1.64µs ± 1% 1.60µs ± 1% -2.80% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum_2.ndjson-12 1.56µs ± 1% 1.52µs ± 1% -2.51% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/unknown-span-type.ndjson-12 6.37µs ± 1% 6.14µs ± 1% -3.57% (p=0.008 n=5+5) ReadBatch/invalid-metadata-2.ndjson-12 21.8µs ±12% 26.0µs ± 9% +19.60% (p=0.016 n=5+5) ReadBatch/transactions_spans_rum_2.ndjson-12 22.2µs ±23% 15.9µs ±14% -28.47% (p=0.016 n=5+5) pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64 Process-12 2.29µs ±39% 1.24µs ±17% -45.82% (p=0.016 n=5+5) pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64 WriteTransaction/json_codec-12 4.05µs ± 7% 12.84µs ±17% +216.83% (p=0.008 n=5+5) WriteTransaction/json_codec_big_tx-12 4.97µs ±11% 11.75µs ±40% +136.16% (p=0.008 n=5+5) ReadEvents/json_codec/0_events-12 299ns ± 3% 350ns ± 6% +16.94% (p=0.008 n=5+5) ReadEvents/json_codec/399_events-12 2.90ms ± 4% 2.79ms ± 2% -4.06% (p=0.032 n=5+5) ReadEvents/json_codec_big_tx/0_events-12 315ns ± 4% 337ns ± 4% +6.83% (p=0.016 n=5+5) ReadEvents/nop_codec/0_events-12 306ns ± 3% 339ns ± 6% +10.98% (p=0.008 n=5+5) IsTraceSampled/sampled-12 67.6ns ± 2% 74.4ns ± 2% +9.96% (p=0.008 n=5+5) IsTraceSampled/unsampled-12 70.2ns ± 1% 76.6ns ± 0% +9.11% (p=0.016 n=5+4) name old alloc/op new alloc/op delta pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64 ModelIndexer/BestCompression-12 2.76kB ± 5% 2.66kB ± 1% -3.70% (p=0.016 n=5+5) pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64 BackendProcessor/events.ndjson-12 40.3kB ± 1% 39.6kB ± 0% -1.65% (p=0.008 n=5+5) BackendProcessor/invalid-metadata-2.ndjson-12 2.74kB ± 1% 2.71kB ± 1% -1.25% (p=0.032 n=5+5) BackendProcessor/otel-bridge.ndjson-12 11.2kB ± 1% 11.0kB ± 1% -1.89% (p=0.008 n=5+5) BackendProcessor/spans.ndjson-12 29.3kB ± 1% 28.8kB ± 1% -1.71% (p=0.032 n=5+5) BackendProcessor/transactions_spans_rum.ndjson-12 5.70kB ± 2% 5.56kB ± 1% -2.43% (p=0.008 n=5+5) BackendProcessor/transactions_spans_rum_2.ndjson-12 5.65kB ± 3% 5.51kB ± 2% -2.54% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel2/transactions_spans.ndjson-12 25.2kB ± 1% 25.6kB ± 1% +1.57% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel4/invalid-json-metadata.ndjson-12 5.88kB ± 2% 6.00kB ± 1% +2.09% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel4/spans.ndjson-12 30.4kB ± 1% 30.8kB ± 1% +1.07% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12 3.91kB ± 0% 3.95kB ± 1% +1.12% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-event.ndjson-12 4.49kB ± 1% 4.59kB ± 1% +2.13% (p=0.008 n=5+5) ReadBatch/errors.ndjson-12 20.6kB ± 0% 20.6kB ± 0% -0.12% (p=0.008 n=5+5) ReadBatch/metricsets.ndjson-12 15.6kB ± 0% 15.5kB ± 1% -0.54% (p=0.032 n=5+5) pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64 WriteTransaction/json_codec-12 3.00kB ± 0% 3.00kB ± 0% +0.03% (p=0.008 n=5+5) WriteTransaction/json_codec_big_tx-12 3.90kB ± 0% 3.90kB ± 0% +0.04% (p=0.032 n=5+5) ReadEvents/nop_codec/1000_events-12 2.17MB ± 0% 2.18MB ± 0% +0.45% (p=0.032 n=5+5) ReadEvents/nop_codec_big_tx/10_events-12 25.7kB ± 0% 25.7kB ± 0% -0.12% (p=0.016 n=5+4) ReadEvents/nop_codec_big_tx/100_events-12 250kB ± 0% 250kB ± 0% -0.05% (p=0.048 n=5+5) name old allocs/op new allocs/op delta pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64 BackendProcessor/events.ndjson-12 717 ± 0% 716 ± 0% -0.14% (p=0.000 n=5+4) BackendProcessor/heavy.ndjson-12 22.3k ± 0% 22.3k ± 0% -0.01% (p=0.024 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel4/spans.ndjson-12 573 ± 0% 574 ± 0% +0.17% (p=0.029 n=4+4) pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64 pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64 name old speed new speed delta pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64 BackendProcessor/events.ndjson-12 42.6MB/s ±20% 58.3MB/s ±20% +36.96% (p=0.032 n=5+5) RUMV3Processor/rum_errors.ndjson-12 104MB/s ±12% 134MB/s ±13% +28.24% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel2/invalid-json-event.ndjson-12 154MB/s ± 6% 145MB/s ± 4% -5.78% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel4/logs.ndjson-12 316MB/s ±12% 276MB/s ± 6% -12.66% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/otel-bridge.ndjson-12 478MB/s ± 1% 496MB/s ± 1% +3.87% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/ratelimit.ndjson-12 631MB/s ± 1% 682MB/s ± 1% +8.08% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/span-links.ndjson-12 425MB/s ± 1% 444MB/s ± 0% +4.64% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/spans.ndjson-12 701MB/s ± 1% 740MB/s ± 1% +5.45% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12 554MB/s ± 1% 571MB/s ± 2% +3.05% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12 562MB/s ± 0% 571MB/s ± 1% +1.59% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12 442MB/s ± 2% 453MB/s ± 1% +2.47% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12 835MB/s ± 1% 857MB/s ± 2% +2.67% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12 669MB/s ± 2% 691MB/s ± 2% +3.35% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/events.ndjson-12 542MB/s ± 1% 567MB/s ± 1% +4.62% (p=0.016 n=5+4) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/heavy.ndjson-12 746MB/s ± 2% 776MB/s ± 3% +4.03% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12 220MB/s ± 1% 228MB/s ± 1% +3.51% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-event.ndjson-12 513MB/s ± 1% 544MB/s ± 1% +5.98% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12 226MB/s ± 1% 236MB/s ± 0% +4.46% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/logs.ndjson-12 678MB/s ± 1% 697MB/s ± 2% +2.80% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metadata.ndjson-12 958MB/s ± 3% 978MB/s ± 1% +2.06% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal-service.ndjson-12 430MB/s ± 2% 440MB/s ± 1% +2.21% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal.ndjson-12 508MB/s ± 3% 528MB/s ± 1% +3.89% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12 692MB/s ± 1% 709MB/s ± 2% +2.52% (p=0.016 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/otel-bridge.ndjson-12 571MB/s ± 2% 588MB/s ± 1% +3.01% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/ratelimit.ndjson-12 725MB/s ± 2% 755MB/s ± 1% +4.06% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12 565MB/s ± 1% 581MB/s ± 1% +2.91% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions.ndjson-12 576MB/s ± 4% 607MB/s ± 2% +5.47% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans.ndjson-12 589MB/s ± 4% 622MB/s ± 3% +5.61% (p=0.032 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum.ndjson-12 702MB/s ± 1% 723MB/s ± 1% +2.87% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum_2.ndjson-12 715MB/s ± 1% 734MB/s ± 1% +2.57% (p=0.008 n=5+5) BackendProcessorParallel/BenchmarkBackendProcessorParallel200/unknown-span-type.ndjson-12 519MB/s ± 1% 538MB/s ± 1% +3.70% (p=0.008 n=5+5) ReadBatch/events.ndjson-12 35.1MB/s ± 4% 48.7MB/s ±31% +38.78% (p=0.016 n=4+5) ReadBatch/invalid-metadata-2.ndjson-12 20.2MB/s ±13% 16.8MB/s ±10% -16.80% (p=0.016 n=5+5) ReadBatch/transactions_spans_rum_2.ndjson-12 51.7MB/s ±26% 70.9MB/s ±16% +37.28% (p=0.016 n=5+5) 

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

[mergify]

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream git checkout -b integration-ecs-version upstream/integration-ecs-version git merge upstream/main git push upstream integration-ecs-version 

[simitt]

Generally LGTM!

@axw checking to see if I fully grasp the intended upgrade flow here: The apm package sets the ecs version, whenever the package gets upgraded, the templates get created and data streams are rolled over; therefore the ecs version is set on package upgrade. If there are still older apm-server instances sending to ES, there are no conflicts expected, as ecs related changes are expected to be backwards compatible and follow the stack versioning. Correct?

[axw]

If there are still older apm-server instances sending to ES, there are no conflicts expected, as ecs related changes are expected to be backwards compatible and follow the stack versioning. Correct?

Right. Hypothetically if there were a breaking change in ECS, then we would update our ingest pipelines to "upgrade" the docs coming from older APM Servers so they conform to the ecs.version set in the mapping. We could use the observer.version field from the docs to do that.

[mergify]

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream git checkout -b integration-ecs-version upstream/integration-ecs-version git merge upstream/main git push upstream integration-ecs-version 

[simitt]

Tested with BC5 that the index templates have the ecs.version set accordingly.

[elasticmachine]

Package apm - 8.6.0-preview-1670294014 containing this change is available at https://epr.elastic.co/search?package=apm