Skip to content

Conversation

@eyalkoren
Copy link
Contributor

APM Server uses timestamp.us at documents root level. When sending data with such field to a logs-*-* data stream, indexing fails with: failed to parse field [timestamp] of type [date] in document with id '<id>'.
This happens because it clashes with the ECS dynamic template that maps timestamp fields to a date.

The preferable solution would be to still automatically map such fields to date if they are of suitable candidate type (string, long or double). We would have relied on match_mapping_type, but this setting doesn't support multiple type values and creating multiple dynamic templates to same patterns with different match_mapping_type make them clash with each other. Two possible enhancements can work well for our purpose:

  • make match_mapping_type support array of types
  • add unmatch_mapping_type, which we would use to unmatch object type

For now, the quick fix would be to remove only the path_match: timestamp mapping, thus still support nested fields (like code_signature.timestamp)

@eyalkoren eyalkoren self-assigned this Jul 5, 2023
@eyalkoren eyalkoren requested review from carsonip and felixbarny July 5, 2023 15:13
@elasticsearchmachine elasticsearchmachine added Team:Data Management Meta label for data/management team external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 5, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@eyalkoren eyalkoren added >breaking >bug and removed Team:Data Management Meta label for data/management team external-contributor Pull request authored by a developer outside the Elasticsearch team >breaking labels Jul 5, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Jul 5, 2023
@elasticsearchmachine
Copy link
Collaborator

Hi @eyalkoren, I've created a changelog YAML for you.

@eyalkoren eyalkoren changed the title Fix timestamp as object as root level in APM Server Fix timestamp as object at root level as in APM Server Jul 6, 2023
@eyalkoren eyalkoren changed the title Fix timestamp as object at root level as in APM Server Fix timestamp as object at root level for APM Server Jul 6, 2023
@eyalkoren
Copy link
Contributor Author

I used the ECS testing utility to verify that all current ECS fields are still covered with the modified state of the ECS dynamic templates.

Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eyalkoren eyalkoren merged commit 9ba40d4 into elastic:main Jul 6, 2023
@elasticsearchmachine
Copy link
Collaborator

💔 Backport failed

The backport operation could not be completed due to the following error:

An unhandled error occurred. Please consult the logs 

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 97401

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport pending >bug :Data Management/Data streams Data streams and their lifecycles Team:Data Management Meta label for data/management team v8.9.0 v8.10.0

6 participants