An invited talk I gave at the Runtime Verification Conference in Sept 2011 in San Francisco, about the Google stack w.r.t. RV and analysis.

1. Runtime Analysis in the Cloud:Challenges and Opportunities Wolfgang Grieskamp Software Engineer, Google Corp

2. About Me < 2000: Formal Methods Research Europe (TU Berlin) 2000-2006: Microsoft Research: Languages and Model-Based Testing Tools 2007-2011: Microsoft Windows Interoperability Program: protocol testing and tools Since 4/2011: Google: Google+ platform and tools DISCLAIMER: This talk does not necessarily represent Google’s opinion or direction.

3. Content of this talk General blah blah about cloud computing Monitoring the Cloud Testing the Cloud Using the Cloud for Development A formal method guy’s dream… Conclusion

4. About the Cloud

5. What is Cloud Computing? From Wikipedia, the free encyclopedia Cloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software and information are provided to computers and other devices as a utility (like the electricity grid) over a network (typically the Internet).

6. What is Cloud Computing? From Wikipedia

8. It is device independent

9. It has a unified access control model

10. You pay for only what you use

11. It can scale-up or scale-down on demand

13. Cloud Stack

14. IAAS (Infrastructure As A Service) Basic building blocks Storage Networking Computation Homogenous, easy migration (based on VMs) Distributed Data Centers over Geographical Zones Players: Amazon, GoGrid, Rackspace, Microsoft, Google Estimated revenue 2010: $1b (Source: Economist)

15. Platform As A Service (PAAS) Basic Building Blocks Operating System Frameworks and Development Tools Deployment and Monitoring Tools Players: Microsoft, Google, IBM, SAP, … Estimated Revenue 2010: $311m (Source: Economist)

16. Software As A Service (SAAS) Device and location independent applications, typically running in a browser (Email, Social, Retail, Enterprise Apps, etc.) Many different players Estimated revenue 2010: $11.7b (Source: Economist)

17. Monitoring the Cloud

18. Monitoring vs RV vs Testing What’s the difference? A (strict) take: Monitoring collects and presents information for human analysis Runtime verification collects and transforms information for automated analysis which ultimately leads to a verdict Testing does the above things in an isolated, staged or mocked, environment. In particular, stimuli from the environment are simulated.  In practice, boundaries are not so clear. For this talk RV = Monitoring (adapting to Google conventions)

19. Anatomy of a Data Center Data Center A Data Center B …… Controller Server Controller Server Server Server Server … Storage Storage Storage Storage Storage Note: abstracted and simplified

20. Anatomy of a Server Data Center A Data Center B …… Controller Server (VM) Controller Server Controller Job Job Job Server Server Server Server … Monitor Monitor Monitor Storage Storage Storage Storage Logs Alert Note: abstracted and simplified

21. Anatomy of a Service Data Center A Data Center B …… Controller Service (across Servers) Job Job Server Controller Job Job Job Server Server Server Server … Job Storage Storage Storage Storage Storage Storage Note: abstracted and simplified

22. Monitoring Types @ Google Black Box Monitoring White Box Monitoring Log Analysis

23. Black Box Monitoring Job Monitor Frequently send requests and analyze the response Possible because server jobs are ‘stateless’ and always input enabled If failure rate over a certain time interval exceeds a given ratio, raise an alert and page an engineer Engineers aim for minimizing paging and avoiding false positives

24. Black Box Monitoring: How its done @ Google There are rule based languages for defining request/responses. Each rule: Synthesizes an HTTP request Analyzes the response using a regular expression Specifies frequency and allowed failure ratio Rules are like tests: a simple trigger and a simple response analysis Monitors can be also custom code Job Monitor

25. Black Box Monitoring: Issues? Is the ‘stateless’ hypothesis feasible? Nothing is really stateless -- state is passed as parameters in cookies, continuation tokens, etc. However, as these are health tests, state can be ignored What is the relation to testing? In theory very similar, only that the environment is not mocked. In practice uses quite different frameworks/languages What about service/system level monitoring? Its only about one job. Doesn’t give failure root cause (it only measures a symptom) Job Monitor

26. ChallengeIntegrate Black-Box Monitoring and Testing Job Monitor Black-box monitoring can be seen as particular way of executing tests end-to-end on the live product such that the impact on performance can be neglected. Frameworks which integrate design and execution of monitoring rules and test cases are promising Mainly an engineering challenge

27. Job Job ChallengeSystem/Service Level Black-Box Monitoring Monitor Monitor Not commonly done Main purpose would be failure cause analysis and failure prevention Simple local monitoring already discovers failures Is there a strong point of doing it at runtime (vs. log analysis)? Only if real-time prevention and potentially repair is important Monitor

28. ChallengeProtocolcontract verification Job Monitor At Google, all communication between jobs happens via a single homogeneous RPC mechanism based on a message format definition language (called protocol buffers) Also all (= terra bytes) of data is stored in formats specified by protocol buffers One could formulate data invariant and protocol sequencing contracts over protocol buffers and enforce them at runtime

29. White-Box Monitoring Server exports collection of probe points (variables) Memory, # RPCs, # Failures, etc. Monitor collects time series of those values and computes functions over them Dashboards prepare information graphically Mostly used for diagnosis by humans Job Monitor

30. White-Box Monitoring:How its done @ Google Job Monitor Declarative language for time series computations Collects samples from the server by memory scraping Merging of similar data from multiple servers running the same job Rich support for diagram rendering in the browser

31. White-Box Monitoring:Issues? Design for monitorability/testability? Its already ubiquitous throughout, since software engineers are themselves on-call… Distributed collection/network load? Not really an issue because it’s sample based Relation to testing? Same as with black-box – should be a common framework. Automatic root cause analysis and self-repair? Current systems mostly build for human analysis and repair. Self-repair would be a big thing. Job Monitor

32. ChallengeSelf-repair Job Monitor Cloud system are homogenous and operate with redundancy Many VMs with exactly the same properties Self-repair could identify and ‘drain’ faulty parts of the system, apply fallbacks, rollback software updates, etc. One major cause of cloud failures are outages Another major cause are software updates A semi-automated approach suggesting actions to a human would be already very useful Ever got paged at 2am in the morning?

33. ChallengeHybrid White-Box Monitoring / RV Foundations Job Monitor The data collected in white-box monitoring represents continuous and often stochastic functions over time. The triggers for discrete actions (like alerts) are thresholds over integrated values of those functions. Sounds like hybrid systems/automaton. Has anybody looked at it like this in RV community?

34. Log Analysis Job Logs Collect data from each server’s run containing information like operation flow, exceptions, etc. Store data over a window of time (say for last 24h) Access data from various sources programmatically to analyze issues (post-mortem, performance, etc.) Allows for correlation of system/service wide information

35. Log Analysis:How its done @ Google Job Logs Very fined grained logging on job side; huge amounts of data collected Logs are stored in Bigtable (Google’s large-scale storage solution) Logs are analyzed using parallel (cloud) computing, e.g. with Sawzall, a declarative language based on map-reduce Logs are most often used for failure cause analysis/debugging

36. Log Analysis:Issues? Job Logs Amount of data and accessibility? Not really an issue because of highly performing distributed file systems Format of the data? Logs are structured data (at Google, protocol buffers) Encryption? A big issue: if it can’t be decrypted, not much may be diagnosable. If it can be decrypted, the access to this now decrypted data needs to be restricted.

37. ChallengePrivacy and Encryption Job Monitor Data logged (or otherwise analyzed) during monitoring may contain encrypted proprietary information A problem may not be diagnosable without decryption Decrypted clear text data (in particular if logged) needs to be highly protected Automatic obfuscation and/or anonymization would be highly desirable A protocol may need to be designed for this in the first place

38. Testing the Cloud

39. Testing @Google

40. ChallengeIntegration Testing Job Job Job Storage Two or more components are plugged together with a partially mocked environment These tests are usually very ‘flaky’ (unreliable) because: Difficulty to construct mocked component’s precise behavior (its more than a simple mock in a unit test) Difficulty to synthesize mocked component’s initial state (it may have a complex state) Potential solution: model-based testing

41. Model-Based Testing in a Nutshell Requirements Feedback Author Model Feedback Feedback Generate Test Suite Issue Expected Outputs (Test Oracle) Inputs (Test Sequences) Verdict Control Observe Feedback Implementation

42. Technical Document Testing Program of Windows: A Success Story for MBT 222 protocols/technical documents tested 22,847 pages studied and converted into requirements 36,875 testable requirements identified and converted into test assertions 69% tested using MBT 31% tested using traditional test automation 66,962 person days (250+ years) Hyderabad: 250 test engineers Beijing: 100 test engineers 38

44. Vendor 2 modeled 85% of all test suites, performing relatively much better than Vendor 139 Grieskamp et. al: Model-based quality assurance of protocol documentation: tools and methodology. Softw. Test., Verif. Reliab. 21(1): 55-71 (2011)

45. Exploiting the Cloud for Development

47. Consequences for Testing, Program Analysis, etc. Need to rethink the base assumptions of some of the existing approaches for testing and program analysis for massive coarse-grained parallelism: Early divide-and-conquer ideal, e.g. initial random seed than run-till-end and collect and compare Try different heuristics on the same problem; see which one wins Techniques like SMT and concolic execution can largely benefit from this

48. A formal method guy’s dream…

49. Many Languages, one Subject… Service These languages are all about the same semantic thing! They use the same data structures (messages), protocols, architecture, etc.

50. Unify some of those Languages! Service

51. SSL – Service Specification Language

52. Conclusions The cloud brings nothing really new, but it changes priorities. Some of the problems traditionally researched in RV and testing are non-brainers (at least at Google). Others do wait for a solution. Have ideas? Apply for a Google Research Award (Google it). Deadline is February 1, 2012.