Download as PDF, PPTX
![Some implementations Queue D6 D7 D8 Memory core core Database OK core core AdvancedQueue OK Not yet Amazon SQS (aws_sqs) OK Not yet Beanstalkd OK OK evQueue Private Queue D6 D7 D8 Apache Kafka OK Started Gearman OK OK Not yet MongoDB OK Started PHPResque OK Not yet RabbitMQ OK OK Redis (redis[_queue]) OK OK Alpha](https://image.slidesharecdn.com/faster-sites-queue-api-160625091642/75/Faster-Drupal-sites-using-Queue-API-13-2048.jpg)
![D6/D7 Queue API D7: core D6: drupal_queue module Declaring queues: hook_cron_queue_info[_alter]() ● “Skip on cron”: enable decoupling from cron runs ● Time: max lifetime allocated to process items during a cron run, useless with skip on cron = TRUE ● Worker callback: an implementation of callback_queue_worker (mixed queue_item): void API useable without cron Default Runner: ● In the cron subsystem ● Pokemon exception handling](https://image.slidesharecdn.com/faster-sites-queue-api-160625091642/75/Faster-Drupal-sites-using-Queue-API-15-2048.jpg)
![Queue API methods: Queue QueueInterface ● Q::createItem(mixed $data: void ● Q::claimItem($lease_time = 3600: mixed $item ○ FALSE | stdClass + [item_id => int, data => mixed, created => timestamp] ○ $lease_time → Assumptions for runner, currently not used ● Q::deleteItem($item): void -> work done ● Q::releaseItem($item): bool ● Q::numberOfItems(): int → best guess, unreliable ● Q::createQueue() / Q::deleteQueue() ReliableQueueInterface: ordering, single execution](https://image.slidesharecdn.com/faster-sites-queue-api-160625091642/75/Faster-Drupal-sites-using-Queue-API-17-2048.jpg)
Uploaded byOSInet
Faster Drupal sites using Queue API
Frédéric G. Marand and Yuriy Gerasimov discuss the implementation of Queue API to enhance website performance, scalability, and efficiency in processing time-consuming tasks for Drupal projects. They detail various use cases and solutions, including content generation, image processing, and job monitoring while comparing the capabilities of different queue systems. The document also touches on the architecture of Queue API, its methods, limitations, and implementation across different Drupal versions.
Faster Drupal sites using Queue API
- 1. Frédéric G. MARAND(fgm) Yuriy GERASIMOV (ygerasimov) OSInet ygerasimov Faster web sites with Queue API
- 2. Frédéric G. Marand ●OSInet: performance/architecture consulting for internal teams at larger accounts ● Core contributor 4.7 to 8.2.x, MongoDB + XMLRPC maintainer + others ● Already 7 D8 customer projects , 4 before 8.0.0 ● Customer D8 in production since 07/2015 ● Frequently adds queueing to larger Drupal projects : Beanstalkd, RabbitMQ, Apache Kafka... fgm
- 3. Yuriy Gerasimov ● FFW ●Drupal architect & developer ● Contrib 7 modules: services, draggableviews ● Founder at Backtrac.io ygerasimov
- 4. Why use queues? To have websites which are : ● Faster for visitors ● Snappier for editors ● More scaleable To process time-consuming jobs : ● Video encoding ● High-resolution gallery uploads and processing
- 5. Actual use cases ●Prepare content for non-Drupal front-ends ● Anticipate content generation ● Deferred submits, e.g. comments handling ● Slow operations: node saves, previews, image processing ● External data sources: pull, push ● Multi-step operations: batch
- 6. Cooking content forfront-ends Front end
- 7. Anticipated content generation Blocks Ctoolscontent types Controllers etc. Contrib : http://github.com/FGM/lazy Content created Served from cache Fresh Stale Expiredt0 t1 t2 Served from cache Regenerate cache time Usual Drupal Content created Served from cache Fresh Stale Fresht0 t1 t2 Served from cache + request update Store Served from cache time Anticipated content generation
- 8.
- 9. “Pull” data sources(aggregator)
- 10.
- 11.
- 12. Job servers ● Howto get results ● Rerun failed jobs ● Separate queue for failed jobs ● Monitoring queues, workers ● Supervisor
- 13. Some implementations Queue D6D7 D8 Memory core core Database OK core core AdvancedQueue OK Not yet Amazon SQS (aws_sqs) OK Not yet Beanstalkd OK OK evQueue Private Queue D6 D7 D8 Apache Kafka OK Started Gearman OK OK Not yet MongoDB OK Started PHPResque OK Not yet RabbitMQ OK OK Redis (redis[_queue]) OK OK Alpha
- 14. Queue API: concepts Queue:a minimally-featured FIFO Worker: the code actually doing the work Item: a piece of workload submitted to the queue Runner: the process triggering/monitoring workers Batch subsystem: a high-level API on top of Queue API D8: Manager, Plugins
- 15. D6/D7 Queue API D7:core D6: drupal_queue module Declaring queues: hook_cron_queue_info[_alter]() ● “Skip on cron”: enable decoupling from cron runs ● Time: max lifetime allocated to process items during a cron run, useless with skip on cron = TRUE ● Worker callback: an implementation of callback_queue_worker (mixed queue_item): void API useable without cron Default Runner: ● In the cron subsystem ● Pokemon exception handling
- 16. D8 Queue API APIuseable without cron Declaring queue workers: Service: plugin.manager.queue_worker Instantiates QueueWorker plugins Definition: ● Cron, not enabled by default ○ Time: max lifetime allocated to process items during a cron run ● Core examples : AggregatorRefresh, LocaleTranslation ● hook_queue_info_alter() Default Runner: In the cron subsystem: DrupalCoreCron::processQueues() SuspendQueueException: $q- >releaseItem()
- 17. Queue API methods:Queue QueueInterface ● Q::createItem(mixed $data: void ● Q::claimItem($lease_time = 3600: mixed $item ○ FALSE | stdClass + [item_id => int, data => mixed, created => timestamp] ○ $lease_time → Assumptions for runner, currently not used ● Q::deleteItem($item): void -> work done ● Q::releaseItem($item): bool ● Q::numberOfItems(): int → best guess, unreliable ● Q::createQueue() / Q::deleteQueue() ReliableQueueInterface: ordering, single execution
- 18. Queue API methods:others Queue service → QueueFactory::get($name, $reliable) QueueManager: a vanilla plugin manager ● In charge of hook_queue_info_alter() ● createInstance($plugin_id, $configuration) QueueWorkerInterface: ● processItem (mixed data) : void @throws SuspendQueueException
- 19. Queue Runners Core /Contrib ● Core Cron / Elysia Cron / Queue_Runner ● Drush: queue-list / queue-run ● Similar limitations: ○ Default on in D6 / D7, default off in D8 ○ Limited timeout support: non preemptive ○ Single threaded, single process across queues Custom runners ● Provided by queue modules or per-project one-offs ● Preemption, parallel execution...
- 20. Queue API limitations LimitedFIFO paradigm ● D8: non-Reliable QueueInterface: datagram No monitoring No queue disciplines ● Priority management ● Tagging ● Delay, burying ... Implementations may provide more ● Item structure is free-form: add richer interfaces No Peek(), no LIFO, no deduplication: hacks
- 21. Performance edge Runners: ● Avoidactive polling à la core DB ● Use a blocking layer + select() ● Parallel handling of multiple queues → multiple runners, scheduling Workers: read after write ● Write in the queue → cache invalidated ● Read again→ cache primed
- 23. Sprints: all week https://www.flickr.com/photos/amazeelabs/ 9965814443/in/faves-38914559@N03/ Sprintwith the Community until Sunday We have tasks for every skillset. Mentors are available for new contributors. Follow @drupalmentoring.