- Notifications
You must be signed in to change notification settings - Fork 513
Closed
Labels
Integration:cloudflare_logpushCloudflare LogpushCloudflare Logpush
Description
Related GitHub issues and PR:
- Cloudflare Logpush - http_request ingest pipeline does not set @Timestamp #5571
- cloudflare_logpush: use EdgeStartTimestamp event timestamp #5599
Issue: The @timestamp field for docs ingested from Logpush to Elasticsearch is pulled from EdgeStartTimestamp and the pipeline processor is defined to accept any of the following formats:
Lines 40 to 59 in 35f9966
| - set: | |
| if: ctx.json?.EdgeStartTimestamp != null | |
| field: '@timestamp' | |
| copy_from: json.EdgeStartTimestamp | |
| - date: | |
| field: json.EdgeEndTimestamp | |
| if: ctx.json?.EdgeEndTimestamp != null && ctx.json.EdgeEndTimestamp != '' | |
| formats: | |
| - ISO8601 | |
| - uuuu-MM-dd'T'HH:mm:ssX | |
| - uuuu-MM-dd'T'HH:mm:ss.SSSX | |
| - yyyy-MM-dd'T'HH:mm:ssZ | |
| - yyyy-MM-dd'T'HH:mm:ss.SSSZ | |
| - UNIX_MS | |
| timezone: UTC | |
| target_field: cloudflare_logpush.http_request.edge.end_time | |
| on_failure: | |
| - append: | |
| field: error.message | |
| value: '{{{_ingest.on_failure_message}}}' |
However, per the Logpull documentation, the available formats that can be configured are as follows:
timestamp_format: string to specify format for timestamps, such as unixnano, unix, or rfc3339. Default unixnano. As a result, if the data comes through as UNIX for example (seconds from 1970) it would be interpreted as a UNIX_MS timestamp (milliseconds from 1970) resulting in a completely inaccurate timestamp.
Possible workarounds may include:
- Refactoring the Elasticsearch pipeline processor to use the given format.
- Reconfiguring the log output options in Cloudflare to accommodate the values are accepted by the pipeline (not recommended).
Metadata
Metadata
Assignees
Labels
Integration:cloudflare_logpushCloudflare LogpushCloudflare Logpush