Validate consumer offset in SimpleConsumer #296
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
This slightly changes the SimpleConsumer interface adding the default option use_latest_offsets. The fetch behaviour is also changed since it does not raise OffsetOutOfRangeError anymore. Resetting the offsets automatically is especially useful in MultiprocessConsumer, where an explicit seek call is not possible.
Contributor
| 👍 |
…_consumer_offset Conflicts: kafka/consumer/simple.py
Contributor Author
| I changed the code to be more like KafkaConsumer. It now uses the keyword argument auto_offset_reset, which can be either "smallest" or "largest" (default), and falls back to the old behavior otherwise. |
| Can we get this merged in? It would be nice to be able to use the Multi consumer without this problem. |
Owner
| LGTM |
dpkp added a commit that referenced this pull request Mar 3, 2015
Validate consumer offset in SimpleConsumer
This was referenced Mar 31, 2015
Closed
This was referenced Jun 11, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
I slightly changed the SimpleConsumer interface adding the default option use_latest_offsets. Resetting the offsets automatically is especially useful in MultiprocessConsumer, where an explicit call to the method seek is not possible.
I manually tested it in my kafka cluster, but there are no integration tests for this feature, yet.
I think that tuning "log.retention.bytes" I should be able to implement two tests.
Fix for #263