normalize_for_stream: Incompatible with flexible field access pattern #134116
Description
When using normalize_for_stream together with field_access_pattern: flexible, it doesn't behave right:
POST _ingest/pipeline/_simulate { "docs": [ { "_source": { "@timestamp": "2024-01-01T00:00:00.000Z", "message": "test", "log": { "level": "info", "logger": "nginx" } } } ], "pipeline": { "processors": [ { "normalize_for_stream": {} } ] } } returns
{ "attributes": { "log.logger": "nginx" }, "@timestamp": "2024-01-01T00:00:00.000Z", "body": { "text": "test" }, "resource": { "attributes": {} }, "severity_text": "info" } which is correct. However, setting "field_access_pattern": "flexible" will result in this:
{ "attributes": { "log.logger": "nginx", "body.text": "test" }, "@timestamp": "2024-01-01T00:00:00.000Z", "resource": { "attributes": {} }, "severity_text": "info" } POST _ingest/pipeline/_simulate { "docs": [ { "_source": { "@timestamp": "2024-01-01T00:00:00.000Z", "message": """{ "@timestamp": "2024-01-01T00:00:00.000Z", "log.level": "info", "log.logger": "nginx", "message": "test" }""" } } ], "pipeline": { "processors": [ { "normalize_for_stream": {} } ], "field_access_pattern": "flexible" } } will return
{ "type": "illegal_argument_exception", "reason": "field [log] not present as part of path [log]" } normalize_for_stream shouldn't change behavior no matter the current field_access_mode