This project is an Apache Storm Spout for consuming from Redis Streams.
- Ability to consume from Redis Streams while maintaing state.
- Parallelism supported via unique Consumer Ids.
Include the dependency in your POM.xml file:
NOTE: This project has not been published to MavenCentral yet.
<dependency> <groupId>org.sourcelab.storm.spout</groupId> <artifactId>redis-stream-spout</artifactId> </dependency>The spout is configured using the RedisStreamSpoutConfig class.
| Property | Required | Description |
|---|---|---|
Host | Required | The hostname to connect to Redis at. |
Port | Required | The port to connect to Redis at. |
Password | optional | Password to connect to Redis using. |
Group Name | Required | The Consumer group name the Spout should use. |
Consumer Id Prefix | Required | A prefix to use for generating unique Consumer Ids within the Consumer Group. To support multiple parallel consumers, the Spout instance will be appended to the end of this value. |
Stream Key | Required | The Redis key to consume messages from. |
Tuple Converter | Required | Defines how messages are transformed between being consumed from Redis, and being emitted into the topology |
Failure Handler | Required | Defines how the spout handles failed tuples. See note below. |
// Create config final RedisStreamSpoutConfig.Builder config = RedisStreamSpoutConfig.newBuilder() // Set Connection Properties .withHost("localhost") .withPort(6179) // Consumer Properties .withGroupName("StormConsumerGroup") .withConsumerIdPrefix("StormConsumer") .withStreamKey("RedisStreamKeyName") // Tuple Handler Class .withTupleConverter(..Your TupleConvertor implementation...) // Failure Handler .withFailureHandler(new RetryFailedTuples(10)); // Create Spout final ISpout redisStreamSpout = new RedisStreamSpout(config);In order to convert from the values consumed from RedisStream into Tuple values that can be emitted into the Storm Topology, an implementation of the TupleConverter must be defined and passed to the configuration.
TestTupleConverter is provided as an example implementation.
The FailureHandler interface defines how the Spout will handle Failed Tuples. The following implementations are provided out of the box:
| Implementation | Description |
|---|---|
| NoRetryHandler | Will never retry failed tuples. |
| ExponentialBackoffFailureHandler | Will attempt to retry failed messages using an exponential backoff strategy. |
| RetryFailedTuples | Rudimentary implementation that can be configured to replay failed tuples for a configured number of attempts. |
ExampleLocalTopology is provided as a working example running on a Local Storm Topology.
Steps for performing a release:
- Update release version:
mvn versions:set -DnewVersion=X.Y.Z - Validate and then commit version:
mvn versions:commit - Update CHANGELOG and README files.
- Merge to master.
- Deploy to Maven Central:
mvn clean deploy -P release-redis-spout - Create release on Github project.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.