App Engine offers developers the opportunity to deploy systems on Google's robust and scalable server-farms. App Engine provides a higher-level platform than Amazon Web Services,with automated scaling and true pay-per-use billing. The poster-child of App Engine, "BuddyPoke", has gained over thirty million users. With App Engine, Google has released the first public API to BigTable, its planetary datastore, which performs successfully at petabyte scale across diverse applications from search to finance to Google Earth. This presentation will cover App Engine's features and limitations, and how to exploit this new and evolving platform.

1. Castles in the Cloud:
Developing with Google App Engine
Kevin Noonan
Calbane Ltd.
IT@Cork, 20th April, 2009

2. Castles in the Cloud
Introduction

3. A working definition of Cloudware
• Outsourced Virtualization
o Virtual server running on outsourced server farm

4. The Cloud Computing Pyramid
•Michael Sheehan of GoGrid devised this classification

5. Main Players

6. Other Players
• Microsoft Windows Azure
o Including .NET Live Services
• Force.com from Salesforce
• SUN (recently announced)
• Mosso
• GoGrid
• FlexiScale
• Hosting365
• ...

7. What's Cloudware good for? (I)‫‏‬
• Guerilla development
o Pay-per-use model is ideal for startups
o Also good for guerillas inside the enterprise...

8. What's Cloudware good for? (II)‫‏‬
• Scaling up
o Clone the virtual server
o pay for extra resources (bandwidth, storage, RAM, etc.)‫‏‬

9. What's Cloudware good for? (III)‫‏‬
• In general, outsourcing IT infrastructure
o pay someone else to run your servers, network, database...

10. Castles in the Cloud
Who's using Cloudware & why not?

11. Startups & Large Firms on AWS
• Startups
o Twitter
o SmugMug
o Pixenate (Walter Higgin's site, from here in Ireland)‫‏‬
o Many Facebook apps, e.g. iLIke
• Large firms
o NASDAQ
o Activision
o Business Objects
o Hasbro
o New York Times
• See: http://aws.amazon.com/solutions/case-studies/

12. Startups on Google App Engine
• Startups
o BuddyPoke! (30M daily pageviews on App Engine)‫‏‬
o Pixverse (PixChat)‫‏‬
o Hyperic (Cloudstatus)‫‏‬
• One large company's experiment:
o GiftTag from BestBuy
• http://code.google.com/appengine/casestudies.html

13. Barriers to Enterprise Adoption
• Immaturity of the technology
o Although AWS is over three years old
• Concerns about reliability
• Concerns about portability
o no two clouds are the same
• Bureaucracy & inertia
• Lock-in to legacy systems
• Fear of loss-of-control & of job-losses
• ...

14. Black days in the Cloud
• Amazon S3 went down for two consecutive hours in Feb 2008.
• Amazon S3 (& Amazon SQS) went down for eight consecutive US daytime hours in
July 2008.
• App Engine had a five-hour partial-outage in June '08
• App Engine had a problem of increased latency (for web-serving) over a few days in
early Mar '09.
• Group for notification of App Engine downtime
o http://groups.google.com/group/google-appengine-downtime-notify

15. Castles in the Cloud
Google App Engine

16. Python: The Original of the Species

17. Java: the All New Flavour

18. Languages on App Engine's JVM
• Java
• JRuby
• Jython
• BeanShell
• JavaScript
• Scala
• Groovy
• …
• Will it Play in App Engine?
http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

19. What is Google App Engine?
• Complete web stack
o Soon to be more general computing platform
• Runs on Google's infrastructure
• Bigtable Datastore
o Like a sorted hashtable
o Not a relational database
o Use GQL (with Python) instead of SQL
o Use JDO or JPA (with Java) for datastore work
• Memcache
• Image API
• Mail API
• Python library available for Force.com web services
• SDK for development on Windows/Mac/Linux
o Run & test locally before uploading to Cloud

20. A fabulous app: 'Gaebar'
Alternative: use AppRocket to replicate an App Engine datastore to an SQL database
http://code.google.com/p/approcket/

21. Limitations of Google App Engine
• Applications run in “The Sandbox”:
http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox
• Native Python modules (written in C) do not work.
• Many Java libraries will not work (but many have been patched or will be).
See the JRE whitelist: http://code.google.com/appengine/docs/java/jrewhitelist.html
• No “long-lived” processing (requests must return in 30 seconds).
• Limited “offline processing” (this is coming)
o no forks, no spawned processes, no threads
o Just scheduled tasks, which work like “cron jobs” on UNIX systems
• HTTPS only on XYZ.appspot.com
o and only with Google's certificate.
o Due to limitation of SSL protocol, as applied to Google's Cloud.

22. Roadmap to June for App Engine
• Service for storing and serving large files
• Task queues for performing background processing
• Incoming email support
• XMPP (Jabber) API

23. 'Batch processing' on App Engine today
• Use App Engine's task scheduling for “cron jobs” spaced a minute apart (or more seldom)
• A hack for more frequent jobs: run a cron-job on your PC or server to periodically hit a URL on App
Engine and kick off a task
• An outsourced hack for more frequent jobs: use webcron.org
o do the same as above (perhaps more reliably!)
o Cost: one hundredth of one Euro cent per 'hit'
• More sophisticated: hitch the clouds together
o 'App Engine Patch' has support for 'Boto'
o 'Boto' is a Python library for Amazon
o Use Boto (on App Engine) to queue work-packets on Amazon's Simple Queue Service (SQS)
o Kick off tasks on App Engine with a method above.
o Cost: $0.01 per 10,000 requests
o (+ $0.10 per GB transferred in...)

24. Web tools for App Engine
• “Xmas gifts” - announced in late December 2008
• Zoho Creator
o generates code and web-pages for deployment
o http://creator.zoho.com/
• App Engine Site Creator
o “Lightweight CMS” (with GUI interface)‫‏‬
o http://code.google.com/p/app-engine-site-creator/
o Apache 2.0 license

25. Billing on Google App Engine
• True pay-per-use
o free quotas & no bill to pay if quotas not exceeded.
o quite different model to AWS
• Developer can assign budget per resource.
• Prices
o $0.10 per CPU core-hour
o $0.15 per GB-month of storage
o $0.12 per GB outgoing bandwidth
o $0.10 per GB incoming bandwidth
o $0.0001 per email recipient for emails sent by the application

26. Free Quotas on App Engine
•Daily outgoing bandwidth: 10 GB
•Daily incoming bandwidth 10 GB
•Daily requests 1.3 million
• Daily CPU-hours: 46
• More quotas (mail, datastore, images, memcache)
http://code.google.com/appengine/docs/quotas.html

27. Other limitations on App Engine
• 1 GB of free storage per app.
•100 MB Memcache storage limit
• App Engine apps can now receive requests and send responses of up to 10MB.
• Caveat: 1 MB limit on datastore/memcache API call
• You can deploy up to 1,000 code and 1,000 static files with an app.
Workaround: use ZipImport (with Python) to store files in a Zip-archive:
http://code.google.com/appengine/articles/django10_zipimport.html
• 150MB cumulative limit on an app's code and static files
(as opposed to storage in the datastore).

28. Atypical Uses of App Engine
• Run Open Source software (e.g. blog / CMS):
http:/groups.google.com/group/google-appengine/web/google-app-engine-open-source-projects
• Store your static files on App Engine
o You can use App Engine as rudimentary 'Content Delivery Network'
• Expose a web-service on App Engine (while doing your main development elsewhere)
REST library for Django & App Engine: http://code.google.com/p/python-rest/
• Expose an interface to BigTable and exploit its scalable storage:
REST interface to datastore: http://code.google.com/p/app3/
• Use RESTlets for Java equivalent of the two above:
http://wiki.restlet.org/docs_1.2/13-restlet/252-restlet.html

29. Brief Case Study: LongURLPlease
• LongURLPlease expands “tiny URLs” to the original long versions.
• Original version written in a day by Darragh Curran, with barely any previous Python experience.
• Launched publicly on December 23rd, 2008.
• Rapidly became a platform in itself: at least nine other applications built on the API within 8 weeks
• Over four thousand downloads of the Firefox plugin
• 100K – 200K requests per day (each request can batch up to 4 URLs)
• Running costs: zero (since its usage is well below the free quotas)

30. Where to go from here (Python)
• App Engine Helper tutorial
o http://blog.pas.net.au/2009/02/17/django-and-google-app-engine-tutorial/
• App Engine Patch article (I recommend AEP over the “helper” above)
o http://code.google.com/appengine/articles/app-engine-patch.html
• App Engine cheatsheet: http://short.ie/cheatongae
• Code samples
o http://code.google.com/p/google-app-engine-samples/downloads/list

31. Where to go from here (Java)
• Sign up for the Java “early look”
http://appengine.google.com/promo/java_runtime
• Install the Eclipse plugin (& SDK)
http://code.google.com/appengine/docs/java/tools/eclipse.html
(The Java samples are in the SDK.)
• Read Google's docs and follow the tutorial:
http://code.google.com/appengine/docs/java/overview.html
• Or pick your own language and get up-and-running on the JVM:
http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

32. Castles in the Cloud
Reference
2/27/2009

33. Articles & Papers
• 10 Reasons Enterprises Aren't Ready to Trust the Cloud
o http://gigaom.com/2008/07/01/10-reasons-enterprises-arent-ready-to-trust-the-cloud/
• The Information Factories, by George Gilder
o http://www.wired.com/wired/archive/14.10/cloudware_pr.html
o quot;When the network becomes as fast as the processor, the computer hollows out and spreads across the network.quot;
- in 1993, from Sun Microsystems' CTO, Eric Schmidt (now CEO of Google)
• Above the Clouds: A Berkeley View of Cloud Computing
o http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.html

34. Resources for Google App Engine
• Google App Engine: http://appengine.google.com
• Videos from conference Google held in 2008
o http://sites.google.com/site/io/
• Chat with developers from the App Engine team
o first Wednesday of the month: 3am next day (Dublin/London time).
o third Wednesday of the month: 5pm (Dublin/London time).
o IRC channel: #appengine on irc.freenode.net
o http://en.wikipedia.org/wiki/List_of_IRC_clients
• Google Group
o http://groups.google.com/group/google-appengine/
• Paper on Bigtable:
o http://labs.google.com/papers/bigtable.html

35. Book (I)‫‏‬
Developing with Google App Engine
by Eugene Ciurana
Published:
Feb 2009
http://apress.com/

36. Book (II)‫‏‬
Google App Engine in Action
by Noah Gift & Michael Orr
To be published:
May 2009
http://www.manning.com/gift/

37. Book (III)‫‏‬
Google App Engine
by Dan Sanderson
To be published:
Sept 2009
http://oreilly.com/

38. Castles in the Cloud
Wrapping up
2/27/2009

39. Credits
• Photos used under Creative Commons license:
• Photo Credits
o http://www.flickr.com/photos/notsogoodphotography/
o http://www.flickr.com/photos/barto/
o http://www.flickr.com/photos/kitoy/
o http://www.flickr.com/photos/wolfgangstaudt/
o http://www.flickr.com/photos/motumboe/
o http://www.flickr.com/photos/ragnar1984/
o http://www.flickr.com/photos/janekm/
o http://www.flickr.com/photos/raylopez/
o http://www.flickr.com/photos/pagedooley/
o http://www.flickr.com/photos/gustty/
o http://www.flickr.com/photos/mike9alive/
o http://www.flickr.com/photos/tambako/
o http://www.flickr.com/photos/nate
o http://www.flickr.com/photos/freewine/

40. Cloudware is the arsenal of freedom
• Go guerilla
o Experiment with cloudware
o Consider using cloudware for your new projects
o Test-drive your favourite language on the App Engine SDK
o Await availability for access to the production Java runtime

41. Contact Information
Kevin Noonan
kevin_noonan on Twitter
http://www.calbane.com