Continuous Integration using Hudson and Fitnesse Speaker: Vasu Durgavarjhula , Jennifer Wong , Norman Boccone Level: Intermediate | Room: 4221 | 11:15 AM Saturday Learn about Continuous Integration (CI) and Continuous Deployment(CD) and how Ingenuity Systems moved from a traditional release process to a more agile frequent release model. In this talk we will discuss specifics and show demos on: using Hudson as a framework for continuous integration, deployment, and build promotion deployment and configuration management changes we made to make our architecture more service-oriented our automated test strategy using JUnit, FitNesse, and Selenium migrating our build and deployment process from Ant to Maven challenges to overcome and lessons learned in implementing a successful CI system

1. Proprietary and Confidential Continuous Integration using Hudson and Fitnesse Norman Boccone | Lead Engineer | twitter: @dropslowly Jennifer Wong | Lead QE Engineer | twitter: @jenlwong Vasu Durgavarjhula | Director of Development | twitter: @d_vasu

2. Proprietary and Confidential 2 Overview ► Who we are ? ► The Challenge ► Tools and Processes ► Test Strategy ► Conclusion

3. Ingenuity Systems We are a leading provider of information and analytics solutions for life science researchers Our goal is to help life science researchers get biological insights from their data quickly and reliably Our products are used by thousands of researchers worldwide Partial Customer List Proprietary and Confidential 3

4. ? Scientists Need a Tool to Make Good Decisions From Complex Biological Data Toxicology Biomarkers Pharmacogenomics Discovery Use understanding of disease mechanisms to identify and validate targets Elucidate biological mechanisms of drug action and toxicity Identify and prioritize novel biomarkers by understanding role in disease pathways Understand mechanisms behind differential response to R(x) Hard to synthesize the picture of what that data means in a broader biological context, and how the pieces work together… Traditional publishing models are good for learning but not applying knowledge … Enormous volume and complexity of biological and chemical data… Cancer Publishing and Collaboration Experimental Data Search Proprietary and Confidential 4

5. The Ingenuity® Knowledge Base Is the core technology behind all of Ingenuity’s products and services THE INGENUITY KNOWLEDGE BASE ANALYSIS HYPOTHESIS GENERATION VISUALIZATION ECOMMERCE ENABLEMENT PATHWAY, REAGENT & GENE SEARCH ENABLES SCIENTIFIC UNDERSTANDING Discover existing relationships and function, as well as inference of relationships that may not have been studied in a particular context yet Generatetestable hypotheses, build pathways, ability to inference Get answers by asking detailed and in-depth scientific questions

6. The Ingenuity® Knowledge Base How is it different? THE INGENUITY KNOWLEDGE BASE ► Structured-to capture relevant details Scientific statements are modeled into Findings using the Ingenuity Ontology ► Expert Review Process- for accuracy Findings go through QC process ► Comprehensive- leverage knowledge in one place Largest scientific knowledge base of its kind with modeled relationships between proteins, genes, complexes, cells, tissues, drugs, pathways and diseases ► Timely- for access to up-to-date knowledge Findings are added weekly

7. Proprietary and Confidential The Challenge 7

8. Proprietary and Confidential 8 Single product

9. Proprietary and Confidential 9 multiple products Ingenuity Variant Analysis Next Generation Sequencing Analysis Reports Ingenuity Product Search Reagent Portals for BD, Sigma, Thermo Fischer

10. Proprietary and Confidential 10 Large monolithic code base

11. Proprietary and Confidential 11 Modular Services Single Sign On User Management Views & Reports User Data Content Service iReport Master Graph

12. Proprietary and Confidential 12 Long release cycles

13. Proprietary and Confidential 13 Faster release cycles

14. Proprietary and Confidential 14 Speeding up the feedback loop Developer Stakeholders

15. Proprietary and Confidential 15 Do all of this while supporting the existing product

16. Proprietary and Confidential 16 Approach Move to a Service Oriented Architecture Don’t miss Jeff’s talk on Sunday 10:45 am “Learning to Love Services: Experiences from the Transition to a Service-oriented Architecture” Develop a test strategy focused on automated testing Build a continuous integration system Adopt agile product management Improve infrastructure to create and deploy services Foster a Dev/Ops culture

17. Proprietary and Confidential 17 Continuous Integration Workflow Application Bundle Deploy Application Run Fitnesse Tests (Nightly suite) Nightly Build (Clover) Run JUnit Tests publish summary Fitnesse Bundle Deploy Fitnesse publish publish Hudson Dashboard (JUnit, Fitnesse summary, Code Coverage) Fitnesse Wiki (Test history, Details, Test Case Management) Link SVN Commit (Test Cases)

18. Proprietary and Confidential 18 Tools and Processes

19. Proprietary and Confidential 19 Service Oriented Architecture

20. Proprietary and Confidential 20 SOA: Before lots of modules, separate but dependent easy to develop on but also easy to add circular dependencies hard to keep track of purpose/value of all the modules product consisted of bunches of modules put together reference was typically to latest devcode to ensure "latest," individual modules of a product were rebuilt for every release junittests were run every time as well new product version meant build everything, test everything builds were slow

21. Proprietary and Confidential 21 SOA: “Current” components: module grouped together. Access through api module only less chance of circular dependencies easier to comprehend modules more flexibility to change code: just do not change api components built/tested separately faster builds less repetitive testing some shared modules still exist (they are not part of any component)

22. Proprietary and Confidential 22 SOA: Conversion Notes find a logical group of modules Convert groupput group into hierarchical structure, hide things behind an apilayer convert outside modules to use apis see Jeff's talk

23. Proprietary and Confidential 23 Build System

24. Proprietary and Confidential 24 Build System: Before in-house ant scripts lots of control but it required lots of resources no real industry standard global scripts/property files, overrides allowed offered consistency but (busy) people would override instead of follow pattern scripts got complicated anyone could add a new feature if anyone wanted a new feature, just tell them to add it inefficient additions, different code styles made scripts hard to read very little dependency management multiple repositories for binary files

25. Proprietary and Confidential 25 Build System: “Current” maven not much control, and very rigid (a good thing, in a way) steep learning curve different from previous way of doing things standard structure; difficult to not follow structure hierarchical structure easily followed versions for all dependencies well defined easy to add new features (after a short burst of intense agony) only one repository

26. Proprietary and Confidential 26 Build System: Conversion Notes get a maven repository manager (nexus? Turn on search!) add antrun plugin to publish artifacts to old system choose a simple component to start, make lots of little submodules(1 module per artifact, plus assembly) manually add needed dependencies to repository build up company parent pom, use hierarchy Put all plugin management into parent Key maven plugins: buildnumber, buildhelper, release, assembly,

27. Proprietary and Confidential 27 Build System: TODO cleaner release plugin continue learning maven contribute to plugins Improve internal maven FAQ Look into git

28. Proprietary and Confidential 28 Continuous Builds

29. Proprietary and Confidential 29 Continuous Builds: Before Hudson (no plugins) auto build start for each module, with artifacts published manual build start for products shell scripts to copy artifacts to release dir

30. Proprietary and Confidential 30 Continuous Builds: “Current” Hudson devmaster for auto build of modules/submodules, auto published to repository manager (dev area) release master for components/products, auto published to repository manager (release area) modules/submodulesbuilt on checkin components/products built on schedule or on demand process includes various tests and deployments (Jen)

31. Proprietary and Confidential 31 Continuous Builds: Conversion Notes use Jenkins, not Hudson (more activity for Jenkins) startup new, clean servers copy/create job that we want Try to make Jenkins be the portal to everything fingerprinting Plugins we use: clover, fitnesse, findbugs, checkstyle, disk-usage, downstream build view, (custom) dashboard view, promoted builds, ssh slaves

32. Proprietary and Confidential 32 Continuous Builds: TODO better job rollup job templates more maven/hudson plugins submit hudson plugin(s) to community

33. Proprietary and Confidential 33 Deployment: Before proprietary shell/perl scripts targeted to install app only (not machine set up) multiple proprietary property config scripts

34. Proprietary and Confidential 34 Deployment: “Current” Puppet Configures machines and apps Work on our machines and EC2 Use of .erb files everywhere for property configuration

35. Proprietary and Confidential 35 Deployment: Conversion Notes modify tar format moved to one tomcat used erb files split properties into app

36. Proprietary and Confidential 36 Deployment: TODO standard server configuration types (e.g. “build machine”, “app machine”, …) more puppet scripts

37. Proprietary and Confidential 37 Visibility

38. Proprietary and Confidential 38 Visibility: Before cfmap (IT server diagnostic tool) Hudson status page FitNesse server

39. Proprietary and Confidential 39 Visibility: “Current” cfmap Hudson status page Hudson dashboard extra tests built in to Hudson FitNesse server Service status page

40. Proprietary and Confidential 40 Visibility: Conversion Notes use Hudson plugins, with drill down get everything to report in a standard way (junit xml)

41. Proprietary and Confidential 41 Visibility: TODO more widgets for the dashboard Integrate outside reports into dashboard cfmap status page sdm? (service discovery manager)

42. Proprietary and Confidential 42 Test Strategy

43. Proprietary and Confidential 43 Testing For CI/CD Test Automation is key How and what to automate Mezaros Cohn Crispin

44. Proprietary and Confidential 44 Test Challenges Testability: some products are difficult to test at lower levels Legacy Apps: one of our main products was a ‘Legacy App’. Tests used to look like this:

45. Proprietary and Confidential 45 Test Challenges varying levels of coverage some of our newer components have great coverage other components have lower coverage: legacy, proof of concept

46. Proprietary and Confidential 46 Solutions Test and code refactoring Legacy Apps: Strangulation approach automate new and refactored features incremental work on tests: reserve time in each iteration for adding to tests Change in culture: team ownership of tests and status Cycle time for ipa was 2.5 weeks Now for most of our services, it is 20-40 minutes

47. Proprietary and Confidential 47 Test Infrastructure and Environments

49. Deployment Tests

50. System and performance tests

52. functional testsService1 Service1 Service2 Service2 Service3 Service3 Service4 Service4 Service5 Service5 Service6

53. Proprietary and Confidential 50 Test Types

54. Proprietary and Confidential 51 Test Types: Deployment and Health Each of our components has a built-in status page

55. Proprietary and Confidential 52 Test Types: Deployment and Health Status page Reflects app status Resource availability: DB Connections, Services

56. Proprietary and Confidential 53 Test Types: Deployment and Health Information is used for: Deployment Health Monitoring Service Compatibility and Dependency checking

57. Proprietary and Confidential 54 Test Types: FitNesse FitNesse is a wiki-based web server test tool Helps abstract test definition from technical implementation Provides visual reporting and result history tracking

58. Proprietary and Confidential 55 Test Types: FitNesse The FitNesse Server http://localhost:8080/FitNesse.UserGuide.TwoMinuteExamplehttp://localhost:8080/FitNesse.UserGuide.TwoMinuteExample?pageHistory Wiki view Editing the wiki Test Results

60. Proprietary and Confidential 57 Drill downs on fitnesse test results in Hudson

61. Proprietary and Confidential 58 FitNessePlugin Configuration Convert results to Junit format

62. Proprietary and Confidential 59 Publish FitNesse Results and Coverage

64. Functional tests

65. Integration tests

66. UI Tests (com.jbergin.HtmlFixture)

67. DB Tests (dbfit)What we’ve done with it that is different Use as execution framework for more complex tests Extension of fitnesse server for data-driven tests json fixture – pass in javascript Execution of Selenium tests

68. Proprietary and Confidential 61 Test Types: FitNesse Lessons learned FitNesse is good for straightforward verification of data To do more, you have to get creative Fixture and test ownership needs to be a shared responsibility

70. Proprietary and Confidential 63 Test Types: Selenium Hudson job configuration: trigger execution on Selenium grid using Ant

71. Proprietary and Confidential 64 Test Types: Selenium Second way we fire off Selenium tests is using a FitNesse fixture: WebTest

72. Proprietary and Confidential 65 Test Types: Backward Compatibility No staging environment How do we know that when we release a new service version to prod that it won’t break? Service version compatibility Leverage Service Discover Manager (SDM) and Status pages to check for availability of services Jar compatibility static code analysis could detect changes in method signature but not underlying object or serialization changes

73. Proprietary and Confidential 66 What’s Next: Getting to the End of the Road

75. Culture:Continue to support, reinforce, educate

76. Proprietary and Confidential 68 The Finish Line Continuous deployment Reduced Cycle Time Improved quality Predictable metrics Reduced risk

77. Proprietary and Confidential 69 Slides are posted here: http://www.slideshare.net/jenlwong/ingenuity-svcc-ci-presentation-20111007 Plugs Learning to Love Services: Experiences from the Transition to a Service-oriented Architecture Speaker: Jeff Green 10:45 AM Sunday Ingenuity is hiring: http://www.ingenuity.com/company/careers.html Q&A References Todd. Test Automation Pyramid – review. Retrieved September 29, 2011 from: http://blog.goneopen.com/2010/08/test-automation-pyramid-review/ Humble, Jez, and Farley, David. Continuous Delivery. Boston: Pearson Education Inc, 2011. Print. Crispin, Lisa, and Gregory, Janet. Agile Testing: A Practical Guide for Testers and Agile Teams. Boston: Pearson Education Inc, 2011. Print.