O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Architecting with Queues - Northeast PHP 2015

847 visualizações

Publicada em

You may know that queues can help with long-running tasks, but did you know they can help you make your application easier to debug, more performant, and scale in the cloud? Taking the real-world example of a contest app, we’ll see how easy queues can be to implement. You’ll see how the smart use of queues can enable your application to handle many more users with the same code, break components across servers, and help you keep your app responsive.

Publicada em: Software
  • Entre para ver os comentários

Architecting with Queues - Northeast PHP 2015

  1. 1. Architecting with Queues for Scale and Separation Sandy Smith Northeast PHP 2015
  2. 2. Insert Text Here Architecting with Queues - Sandy Smith - Northeast PHP 2015 Setting Expectations 2
  3. 3. Architecting with Queues - Sandy Smith - Northeast PHP 2015 What this talk is NOT •A queues HOWTO (though there is some) •Benchmark bonanza •An Azure talk (though there is some) 3
  4. 4. Architecting with Queues - Sandy Smith - Northeast PHP 2015 What this talk IS •An architecture talk •A challenge to think differently about your applications •A story about rapidly developing an app 4
  5. 5. Architecting with Queues - Sandy Smith - Northeast PHP 2015 The Challenge 5
  6. 6. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Social Contest •Show off Azure + PHP •People submit tweets to enter contest •Pull specified keywords from Twitter queue (prefiltered by Node.js app) •Human admins filter out inappropriate content •Humans or computer pulls out winner from approved entries, on timer or arbitrarily •Display latest entries and latest winners to public 6
  7. 7. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Stretch goals •Allow any size contest •Assume global contest with distributed moderators 7
  8. 8. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Quick refresher Performance vs. Scaling Vertical vs. Horizontal Scaling 8
  9. 9. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Initial design 9
  10. 10. Architecting with Queues - Sandy Smith - Northeast PHP 2015 System-level view 10 Twitter Firehose Entries node.js app Contest App DB Public Admins
  11. 11. Architecting with Queues - Sandy Smith - Northeast PHP 2015 The real bottleneck What’s the slowest and most variable part of any application? 11
  12. 12. Insert Text Here Architecting with Queues - Sandy Smith - Northeast PHP 2015 The real bottleneck 12
  13. 13. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Traditional database design •Group by kind •Keep metadata with object or in metadata tables •Objects (Document, Person, Account) are most important •Reinforced by TableGateway, ActiveRecord patterns, and ORM and framework module generator defaults •Works for 80+% of cases 13
  14. 14. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Tradeoffs of traditional design •Everything’s in one table, even when you routinely only need a subset •Typically one master writes, replicants read •Design is focused around what something is, not its state or other attribute •Requires creative solutions for horizontal scaling •Encourages (but does not require) centralizing logic for a given type of data 14
  15. 15. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Worst of all… •This design really didn’t show off all the stuff in Azure 15
  16. 16. (We had a week left) Architecting with Queues - Sandy Smith - Northeast PHP 2015 Redesign time 16
  17. 17. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Redesign goals •Include as much Azure stuff as is reasonable •Implement the stretch goals of scalability 17
  18. 18. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Queues to the rescue! (Plus some other cool stuff) 18
  19. 19. Architecting with Queues - Sandy Smith - Northeast PHP 2015 New approach •Keep long term but fast storage •Central concern is not the Thing (Entry) but Status – Unapproved – Approved – Denied – Winner •Separate updates to long term storage from status changes •Minimal impact to working code 19
  20. 20. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Azure queuing options Azure Queues (duh) •Simple •“Short”-lived (<7 days) •Used within Azure •Uses REST •Can track message processing 20 Azure Service Bus •Enterprisey •Long-lived •In Azure or private cloud •Can use AMQP, REST, or API •Can publish/subscribe •Can batch requests •Can guarantee FIFO •etc. See https://msdn.microsoft.com/en-us/library/azure/hh767287.aspx
  21. 21. Architecting with Queues - Sandy Smith - Northeast PHP 2015 More options •Anything you can install on Linux or Windows (RabbitMQ, ZeroMQ, Kafka, Kestrel, ActiveMQ, etc.) •Any relational or NoSQL database •Azure Tables - Simple REST NoSQL store with a twist 21
  22. 22. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Solutions •Long term storage and display retrieval: Azure Table •Since Node.js app already used it, use Service Bus to store changes in status for consistency •Have daemons pull incoming status changes out and write them to the Table 22
  23. 23. Architecting with Queues - Sandy Smith - Northeast PHP 2015 New design 23 __construct(EntryAccessor $mapper) Entry EntryAccessor __construct($proxy, $table) EntryAccessorTable EntryRepository __construct($dbh) EntryRepositoryTable Entries Incoming Daemon Denied Winner Approv- ed In- coming Approved Daemon Denied Daemon Winner Daemon
  24. 24. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Azure Table basics 24 require_once 'vendorautoload.php';
 use WindowsAzureCommonServicesBuilder;
 use WindowsAzureCommonServiceException;
 use WindowsAzureTableModelsEntity;
 use WindowsAzureTableModelsEdmType;
 
 // Create table REST proxy.
 $tableRestProxy = ServicesBuilder::getInstance()->createTableService($connectionString);
 
 try {
 // Create table.
 $tableRestProxy->createTable("mytable");
 } catch(ServiceException $e){
 // Handle exception based on error codes and messages.
 }
 
 $entity = new Entity();
 $entity->setPartitionKey("pk");
 $entity->setRowKey("1");
 $entity->addProperty("PropertyName", EdmType::STRING, "Sample");
 
 try {
 $tableRestProxy->insertEntity("mytable", $entity);
 } catch(ServiceException $e){
 // Handle exception based on error codes and messages.
 }
  25. 25. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Create and send with Service Bus 25 use WindowsAzureServiceBusModelsQueueInfo;
 use WindowsAzureCommonServiceException;
 use WindowsAzureCommonServicesBuilder;
 
 $serviceBusRestProxy = ServicesBuilder::getInstance()- >createServiceBusService($connectionString);
 
 try {
 $queueInfo = new QueueInfo("myqueue");
 $serviceBusRestProxy->createQueue($queueInfo);
 } catch(ServiceException $e) {
 // handle error
 }
 
 try {
 // Create message.
 $message = new BrokeredMessage();
 $message->setBody("my message");
 
 // Send message.
 $serviceBusRestProxy->sendQueueMessage("myqueue", $message);
 } catch(ServiceException $e) {
 // handle error
 }
  26. 26. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Receive with Service Bus 26 use WindowsAzureServiceBusModelsQueueInfo;
 use WindowsAzureCommonServiceException;
 use WindowsAzureCommonServicesBuilder;
 
 $serviceBusRestProxy = ServicesBuilder::getInstance()- >createServiceBusService($connectionString);
 
 try {
 // Set the receive mode to PeekLock (default is ReceiveAndDelete).
 $options = new ReceiveMessageOptions();
 $options->setPeekLock(true);
 
 // Receive message.
 $message = $serviceBusRestProxy->receiveQueueMessage("myqueue", $options);
 echo "Body: ".$message->getBody()."<br />";
 echo "MessageID: ".$message->getMessageId()."<br />";
 
 // *** Process message here ***
 
 // Delete message.
 $serviceBusRestProxy->deleteMessage($message);
 } catch(ServiceException $e){
 // handle error
 }
  27. 27. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Benefits •Queues add safety: if processing fails, main store is unchanged •App doesn’t wait for process-update-delete cycle – Concerns more separated – Scaling is simpler •Can move queue processing to separate machines •Trivial to move to different stores for each status •Very performant with up-to-second data 27
  28. 28. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Challenges •No full ACID •Safety is largely in the application layer •Potential for race conditions – Humans suck 28
  29. 29. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Takeaways •Look for more than just long processes •Use queues to decouple functionality •Look for status changes, e.g. workflow •Is the type of data the most important aspect of your data? – It usually is! •Design for replacement of components – Makes changes easier (not easy) 29
  30. 30. Architecting with Queues - Sandy Smith - Northeast PHP 2015 Links •Azure: http://azure.microsoft.com/en-us/ •Social Contest:
 https://github.com/MusketeersMe/SocialContest •Me – @SandyS1 – http://phparch.com/ •Feedback! https://joind.in/14720 •php[world] (Nov 16–20) world.phparch.com •Slides will be at: slideshare.net/SandySmith 30

×