- Notifications
You must be signed in to change notification settings - Fork 180
Open
Labels
enhancementFeature requestFeature request
Description
We use UTC timestamps when we write logs in Kafka, and date partitioning on S3.
When I tried to parse a timestamp column like { "@t": "2020-11-11T07:58:58.7291622Z" }, generated filename was 20201110_22xx (GMT+9), not 20201111_07xx!
in-kafka-group creates TimeParser with just time_format, since there is no parameter like utc.
fluent-plugin-kafka/lib/fluent/plugin/in_kafka_group.rb
Lines 134 to 140 in fe1d71e
| if @time_source == :record and @time_format | |
| if defined?(Fluent::TimeParser) | |
| @time_parser = Fluent::TimeParser.new(@time_format) | |
| else | |
| @time_parser = Fluent::TextParser::TimeParser.new(@time_format) | |
| end | |
| end |
TimeParser uses local time by default. (localtime = true)
https://github.com/fluent/fluentd/blob/6757d70b7fdfcf5e4a6316b1ecd802dfeb0e326e/lib/fluent/time.rb#L208
Can we have utc parameter like parsers? When I set time_source as kafka, it worked properly. The timestamp was UTC.
Metadata
Metadata
Assignees
Labels
enhancementFeature requestFeature request