Skip to content

Different per-topic timeout configuration #1671

@swehner

Description

@swehner

Description

I'm looking to use a single Kafka producer for two slightly differnt use cases/message types. One that is just fire-and-forget and that will simply enqueue a message without wait for any acknowledgment.
For the second use case I'd like to make sure that the message was written to broker before returning a response - similar to what's described in #1174

It seems that the way to achieve this would be to set a low message timeout and wait for poll() to acknowledge message delivery via the callback.

Having a low global message timeout wouldn't be very good for the fire-and-forget use cause, so it seems that the best option would be to use the per-topic configuration.
Was there any progress on implementing this since #328 (comment) ? It looks like none of the APIs are exposed through the python client.

If there's still no way to provide per-topic config I'd be open to work on it and make a pull request, but would like to understand what the preferred approach is. maybe exposing the topic creation methods on the produce with a config dict would be enough?

Thanks in advance

Checklist

Please provide the following information:

  • confluent-kafka-python==2.20 librdkafka==2.2.0
  • Apache Kafka broker version: 3.4.0
  • Client configuration: {...}
  • Operating system:
  • Provide client logs (with 'debug': '..' as necessary)
  • Provide broker log excerpts
  • Critical issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    api-changeReferencing changes to the API speccomponent:producerIssues tied specifically to producer logic or code pathspriority:highMaintainer triage tag for indicating high impact or criticality issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions