Slides for the Drupal 7 Queues talk I gave at Drupal7Camp Leeds, May 2011.

1. Drupal 7 Queues

2. Drupal Queues New to Drupal 7 First in first out data structure Used internally by Drupal Uses OO principles Fully customisable

3. Why Use Queues? Batch processing of large amounts of data/items Delay processing of complex calculations so system load is more stable Sequential processing of items Preventing API service black listing

4. Drupal 7 Implementations Batch API Cron Aggregator Module Update Module

5. Drupal Queues Found in the file: /modules/system/system.queue.inc Classes are used to wrap queue functionality Reliable vs Non-reliable

6. Reliable Generally kept in database table Every item will be executed at least once Items exist over several requests If the request fails then queue remains intact

7. Non-reliable Generally kept in memory only All items might exist in a single request No guarantee that all queue items will be executed No guarantee that all items will be executed in order If request fails then queue might be lost

8. Drupal Queue Classes

9. DrupalQueue Single static method get() DrupalQueue::get('my_queue'); Returns a queue object of a given name and a given type (reliable or non-reliable) Force reliable queue by passing TRUE as second parameter Default returned class is SystemQueue Essentially a queue object factory

10. SystemQueue Example of a reliable queue class Implements DrupalReliableQueueInterface Uses the database table queue to store and retrieve the queue Items are 'leased' to ensure no two processes get the same queue item Default class for new queues

11. queue Table Created at Drupal install

12. MemoryQueue Example of an non-reliable queue class All queue items are stored in memory $queue parameter in class contains the queue Also implements item leasing

13. System Variables Three system variables are used by DrupalQueue to decide what sort of object to return 'queue_class_' . $name Default is NULL queue_default_class Default is SystemQueue queue_default_reliable_class Default is SystemQueue

14. Using Drupal Queues

15. Create A Queue $queue = DrupalQueue::get('my_queue', TRUE); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1

16. Create A MemoryQueue variable_set('queue_default_class', 'MemoryQueue'); $queue = DrupalQueue::get('my_queue'); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1

17. Get Item From Queue $queue = DrupalQueue::get('my_queue'); $got_item = $queue->claimItem(); echo $got_item->data['dataitem1'];

18. Retrieved Item Structure stdClass Object ( [data] => Array ( [dataitem1] => something [qwe] => 123 ) [item_id] => 89 )

19. Change Lease Length Can pass the lease time (in seconds) to claimItem() $got_item = $queue->claimItem( 100 );

20. Release Or Delete releaseItem() resets the lease time $queue->releaseItem($got_item); deleteItem() removes item from queue $queue->deleteItem($got_item);

21. Customizing Create custom queue class Create variable called “queue_class_” . $name Value is the class name of your queue class Call DrupalQueue::get() with the variable $name as a string

22. Customizing Reliable class MUST implement DrupalReliableQueueInterface Can also extend SystemQueue All Queue classes should at least implement DrupalQueueInterface

23. Using Custom Classes variable_set('queue_class_mycustom', 'MyCustomQueueClass'); $queue = DrupalQueue::get('mycustom');

24. Stack Class Last in first out implementation class Stack extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name ORDER BY created DESC ', 0, 1, array(':name' => $this->name))->fetchObject(); // ...

25. EventQueue Class Required as part of the NWDUG website Any items added would not be available for 45 minutes to give a period of grace after creating an event node class EventQueue extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name AND created >= UNIX_TIMESTAMP(DATE_ADD(NOW(), INTERVAL 2700 SECOND)) ORDER BY created ASC', 0, 1, array(':name' => $this->name))->fetchObject();

26. WatchdogSystemQueue Extends SystemQueue and creates a log every time anything is done Maintains SystemQueue functionality

27. WatchdogSystemQueue class WatchdogSystemQueue extends SystemQueue { public function __construct($name) { watchdog('queue', '%name queue created', array('%name' => $name)); parent::__construct($name); } public function createItem($data) { watchdog('queue', 'Item created : %data', array('%data' => print_r($data, TRUE))); parent::createItem($data); } public function claimItem($lease_time = 3600) { $return_value = parent::claimItem($lease_time); watchdog('queue', 'Item claimed %item (lease time = %lease)', array( '%item' => print_r($return_value, TRUE), '%lease' => $lease_time)); return $return_value; } //....

28. RandomMemoryQueue Extends the MemoryQueue class Items are added in order but are retrieved in random order

29. RandomMemoryQueue class RandomMemoryQueue extends MemoryQueue { public function claimItem($lease_time = 30) { $available_items = array(); // Extract the remining available items foreach ($this->queue as $key => $item) { if ($item->expire == 0) { $available_items[] = $item; } } // Randomly select one (if available) if (count($available_items) > 0) { $queue_length = count($this->queue); $rand_item = rand(0, $queue_length - 1); $item = $available_items[$rand_item]; $item->expire = time() + $lease_time; return $item; } return FALSE; } }

30. Creating And Destroying DrupalQueueInterface has two methods available for creating and destroying the queue createQueue() Should be called within an install hook deleteQueue() Should be called within an uninstall hook

31. Tips Unless you really need to rewrite the entire class it is best to extend SystemQueue or MemoryQueue For open source projects try to keep the same retrieved item structure as the system queues No checks for unique items in default system queues

32. Resources Drupal Queues API http://api.drupal.org/api/drupal/modules--system--system.queue.inc/group/queue/7 http://bit.ly/kuphnc Queue UI Module http://drupal.org/project/queue_ui Source code is well documented /modules/system/system.queue.inc Full write up of this talk on #! code http://www.hashbangcode.com/

33. Questions? $queue = DrupalQueue::get('questions'); $question = $queue->claimItem(); echo $question->data['question'];

34. Blog http:///www.norton42.org.uk/ Twitter @philipnorton42 #! code http://www.hashbangcode.com/ #! code on Twitter @hashbangcode Philip Norton