1. High-throughput eScience mixing Grids and Clouds: an experience with the Nimrod tool family Presenter: Blair Bethwaite MonasheScience and Grid Engineering Lab

2. MeSsAGE Lab team: David Abramson Colin Enticott SlavisaGaric and others... Acknowledgements NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

3. Agenda NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

4. The Nimrod tool family NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

5. Parametric computing with the Nimrod tools Vary parameters Execute programs Copy code/data in/out X, Y, Z could be: Basic data types; ints, floats, strings Files Random numbers to drive Monte Carlo modelling X Y Parameter Space Solution Space Z User Job EII Cloud Workshop - AWS Intro Blair Bethwaite - MeSsAGE Lab, Monash Uni

6. Nimrod Applications messagelab.monash.edu.au/EScienceApplications NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

7. From Clusters, to Grids, to Clouds Jobs / Nimrod experiment Nimrod Actuator, e.g., SGE, PBS, LSF, Condor Local Batch System NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

8. From Clusters, to Grids, to Clouds Jobs / Nimrod experiment Portal Nimrod-O/E/K Nimrod/G Actuator, e.g., Globus Servers Upper middleware Lower middleware Pilot jobs / agents Agents Grid Middleware Grid Middleware Grid Middleware Agents Grid Middleware NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

9. From Clusters, to Grids, to Clouds NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni The Grid Global utility computing mk.1-(beta) Somewhere in-between Infrastructure and Platform as-a-Service For Nimrod Increased computational scale – massively parallel New scheduling and data challenges Computational economy proposed Problems Interoperability Barriers to entry

10. From Clusters, to Grids, to Clouds NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

11. From Clusters, to Grids, to Clouds NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni Cloud opportunities for HTC Virtualisation helps interoperability and scalability Cloud bursting Scale-out to supplement locally and nationally available resources Test computational economy and scheduling, in anger Deadline driven Budget driven What’s missing? Grids provide services above IaaS E.g., you can build a grid on EC2 Grids provide job and data handling services, more like PaaS

12. From Clusters, to Grids, to Clouds def process_queue(self): """Prepare allocation of commands/agents to instances. This might mean requesting new instances from the web service and/or allocating available slots from existing instances. ""“ if not self._queued_cmds and not self.proxy: return False self._update_available_instances() req_slots = len(self._queued_cmds) new_slots = req_slots - self.free_slots num_insts = new_slots / self.slots_per_instance # if we need the proxy we might have to force # launching an instance to host it if self.proxy and num_insts < 1 br /> and len(self.instances) < 1: num_insts = 1 rsv = None ... ... if num_insts > 0: try: rsv = self.ec2conn.run_instances(self.ami_id, min_count=1, max_count=num_insts, key_name=self.ws_label, security_groups=[self.secgroup.name], instance_type=self.ec2InstanceType) except EC2ResponseError,e: if ec2.parse_response_error(e, 'Code') == br />u'InstanceLimitExceeded': self.at_instance_limit = True print "[%s] Instance limit exceeded" % self.label else: print "[%s] Error running instances:%s" % br /> (self.label, t5exc.exception()) raise if rsv: self._pending_reservations.append(rsv) ... NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

13. Integrating with IaaS Jobs / Nimrod experiment Portal Nimrod-O/E/K Nimrod/G Actuator: Globus,... Services New actuators: EC2, IBM, Azure, OCCI?,...? RESTfulIaaS API Grid Middleware VM Agents Agents VM VM Agents Agents NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

14. Integrating with IaaS Advantage: Nimrod is already a meta-scheduler Creates an ad-hoc grid dynamically overlaying the available resource pool Don’t need Grid-like job processing services to stand-up resource pool Requires explicit management of infrastructure Extra level of scheduling – when to initialise infrastructure? NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

15. Integrating with IaaS 1 2 3 NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

16. Application Examples A lot of existing grid based infrastructure So, mix it together “Mixing Grids and Clouds: High-Throughput Science Using the Nimrod Tool Family,” in Cloud Computing, vol. 0 (Springer London, 2010) Markov Chain Monte Carlo methods for recommender systems For better results, insert coins here... NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

17. Application Examples Modelling ash dispersion – NG-TEPHRA IEEE e-Science 2010 Supplement local infrastructure for deadline sensitive analysis NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

18. Work-in-progress What’s keeping me awake... Spot-price scheduling Smarter data handling Windows support On EC2 And integrating with Azure Rose NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

19. Nimrod utilising NeCTAR RC Host MeSsAGE Lab tools Dev and test environment Excess capacity supporting HTC NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni

20. Thank you! Presentation by: Blair Bethwaite Researcher, Developer, SysAdmin Monash eScience and Grid Engineering Lab Feedback/queries: blair.bethwaite@monash.edu david.abramson@monash.edu NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni