Skip to content

Cloudflare Logpush integration does not support the correct available timestamp formats to match the output format options from the source #7762

@travisestill

Description

@travisestill

Related GitHub issues and PR:

Issue: The @timestamp field for docs ingested from Logpush to Elasticsearch is pulled from EdgeStartTimestamp and the pipeline processor is defined to accept any of the following formats:

- set:
if: ctx.json?.EdgeStartTimestamp != null
field: '@timestamp'
copy_from: json.EdgeStartTimestamp
- date:
field: json.EdgeEndTimestamp
if: ctx.json?.EdgeEndTimestamp != null && ctx.json.EdgeEndTimestamp != ''
formats:
- ISO8601
- uuuu-MM-dd'T'HH:mm:ssX
- uuuu-MM-dd'T'HH:mm:ss.SSSX
- yyyy-MM-dd'T'HH:mm:ssZ
- yyyy-MM-dd'T'HH:mm:ss.SSSZ
- UNIX_MS
timezone: UTC
target_field: cloudflare_logpush.http_request.edge.end_time
on_failure:
- append:
field: error.message
value: '{{{_ingest.on_failure_message}}}'

However, per the Logpull documentation, the available formats that can be configured are as follows:

timestamp_format: string to specify format for timestamps, such as unixnano, unix, or rfc3339. Default unixnano. 

As a result, if the data comes through as UNIX for example (seconds from 1970) it would be interpreted as a UNIX_MS timestamp (milliseconds from 1970) resulting in a completely inaccurate timestamp.

Possible workarounds may include:

  • Refactoring the Elasticsearch pipeline processor to use the given format.
  • Reconfiguring the log output options in Cloudflare to accommodate the values are accepted by the pipeline (not recommended).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions