Starting with Redis 5, redis supports streams. A stream is a
list of messages. Streams can be used as a reliable alternative to the
older Redis PUB/SUB (Publish Subscribe) mechanism that has no memory,
i.e., if a node is down when a message arrives the message is missed. In
addition, they can be used to have each message processed by a
consumer that belongs to a consumer group. Both facilities
are supported by library(redis_streams) (section
3)
Redis streams provide all the low-level primitives to realise message
brokering. Putting it all together is non-trivial though. Notably:
- We must take care of messages that have been sent to some consumer
but the consumer fails to process the message and (thus) ACK it is
processed. This is handled by xlisten_group/5
using several options. Good defaults for these options are hard to give
as it depends on the required processing time for a message, how common
failures are and an acceptable delay time in case of a failure, what to
do in case of a persistent failure, etc.
- Streams are independent from consumer groups and acknowledged
messages remain in the stream. xstream_set/3
can be used to limit the length of the stream, discarding the oldest
messages. However, it is hard to give a sensible default. The required
queue length depends on the the size of the messages, whether messages
come in more or less randomly or in bursts (that cause the stream to
grow for a while), available memory, how bad it is if some messages get
lost, etc.
The directory doc/packages/examples/redis in the
installation provides an example using streams and consumer groups to
realise one or more clients connected to one or more compute nodes.