NAME Queue::Q - Mix-and-match Queue Implementations and Backends SYNOPSIS # Pick you queue type and a back-end: # Very primitive FIFO queue (abstract interface) use Queue::Q::NaiveFIFO; # ... and it's pure-Perl, in-memory implementation use Queue::Q::NaiveFIFO::Perl; # ... it's Redis-based implementation use Queue::Q::NaiveFIFO::Redis; # "Reliable" enqueue, claim, mark-as-done FIFO queue: use Queue::Q::ClaimFIFO; # abstract interface # ... and again, it's pure-Perl, in-memory implementation use Queue::Q::ClaimFIFO::Perl; # ... and it's Redis-based implementation use Queue::Q::ClaimFIFO::Redis; # A composite, distributed queue, built from shards of # the above types with weak global and strict local # ordering, scaling beyond single nodes. use Queue::Q::DistFIFO; # Usage is somewhat similar for most queue types: my $queue = Queue::Q::ClaimFIFO::Redis->new( server => 'redis-01', port => '6379', queue_name => "image_queue" ); $q->enqueue_item( $img_url ); # In another process on another machine: while (defined( my $item = $q->claim_item )) { # Do some work! $q->mark_item_as_done($item); } # n-ary versions exist, too: $q->enqueue_items( @img_urls ); my @items = $q->claim_items(10); # 10 at a time, padded with undef $q->mark_items_as_done(@items); DESCRIPTION This is an experimental module. The interface may change without notice. Before using it in production, please get in touch with the authors! "Queue::Q" is a collection of queue implementations each with multiple backends. Right now, it comes with three basic queues: Queue::Q::NaiveFIFO A naive FIFO queue "NaiveFIFO". Supports enqueuing data and later claiming it in the order it was enqueued. Strict ordering, low-latency, high-throughput, no resilience against crashing workers. Queue::Q::ClaimFIFO A claim/mark-as-done FIFO queue "ClaimFIFO". Supports enqueuing data (as Queue::Q::ClaimFIFO::Item objects), claiming items, and keeping track of items-being-worked-on until they are reported as done. Strict ordering, slightly higher latency and lower throughput than the naive FIFO queue. Resilience against crashing workers when combined with a clean-up script that checks for old, claimed items. Queue::Q::ReliableFIFO Similar interface to the "ClaimFIFO" queue type, but only comes with a Redis-backed implementation now. Much faster for the basic operations then "ClaimFIFO" since it doesn't always have to execute Lua scripts. See Queue::Q::ReliableFIFO for details. The first two basic queues come with two back-end implementations each right now: One is a very simple, in-memory, single-process implementation Queue::Q::NaiveFIFO::Perl and Queue::Q::ClaimFIFO::Perl respectively. The other is an implementation based on Redis: Queue::Q::NaiveFIFO::Redis and Queue::Q::ClaimFIFO::Redis. As noted above, the "ReliableFIFO" queue has a Redis-based backend only for now. In addition to the basic queue types, the distribution contains Queue::Q::DistFIFO, an implementation of a distributed queue that can use the basic queues as shards (but only one type of shard per distributed queue). It supports all operations of the basic queues, but does not enforce strict global ordering, but weak global and strict local ordering. In an early test, two mid-range servers running multiple instances of Redis each sustained 800k-1M transactions per second with a naive-type distributed queue. "DistFIFO" has not been tested with the "ReliableFIFO" implementation as building blocks yet! ACKNLOWLEDGMENT This module was originally developed for Booking.com. With approval from Booking.com, this module was generalized and put on CPAN, for which the authors would like to express their gratitude. AUTHOR Herald van der Breggen, Steffen Mueller, COPYRIGHT AND LICENSE Copyright (C) 2012, 2013 by Steffen Mueller This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.1 or, at your option, any later version of Perl 5 you may have available.