Skip to content

Make ingestion more resilient by ignoring malformed and avoid field explosion #11531

@simitt

Description

@simitt

We want to reduce the risk of mapping conflicts. The APM solution should adopt ES features to make ingestion more resilient:

(1) Add ignore_malformed: true to the root of the mappings. See also elastic/elasticsearch#95329. Ignores fields with type conflicts instead of rejecting the whole document.
Validation Criteria: Ingesting a label foo: 42, then foo: "bar" doesn't lead to data loss

(2) Add dynamic: until_limit to avoid that field explosions lead to data loss. Relates to elastic/elasticsearch#96235.
Validation Criteria: Exceeding the field limit by creating many unique labels doesn't lead to data loss

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions