Improving Engineering Processes using Hudson - Spark IT 2010
1. Improving Engineering Process
Using Hudson
<Insert Picture Here>
Arun Gupta, Java EE & GlassFish Guy
Oracle Corp
http://blogs.sun.com/arungupta, @arungupta
4. Rise of Continuous Integration
Offload from people, push to computers
$
computers
people
Time
5. Spend more CPU power to help you
First on your laptops and workstations
●
● IDEs are at the forefront
And then to the servers
●
● a.k.a. “Continuous Integration”
● More frequent build/test executions
● Static code analysis tools
● And more to come
6. Hudson (http://hudson-ci.org)
● Open-source CI server
● Emphasis on ease of installation and use
● “java -jar hudson.war” execution
● Or your OS-specific package
● Configure everything from browser
● Extensibility
● 230+ community-developed public plugins
● By 230+ contributors
● Estimated 13,000 installations
11. Why do I care?
Life before Hudson
●
● Dev forgets to commit a file
● Nightly build breaks, Testing blocked
● Resources not utilized efficiently
Life after Hudson
●
● Hudson will tell you in 5 mins if a build broke
● No one else notices that you forgot to commit a
file ;-)
12. Why do I care?
● Life before Hudson
● Dev makes a change Monday AM
● Test nightly finds a bug Monday night
● Dev fixes it Tuesday
● Again, under utilization of resources
Life after Hudson
●
● Dev makes a change Monday AM
● Hudson finds a regression few mins later
● Dev can fix it before lunch
13. Why do I care?
Life before Hudson
●
● Test runs every night, results in e-mail
● Excitement fades after 1 week
● Regressions go unnoticed until it’s too late
Life after Hudson
●
● Tests run Hudson after every commit
● E-mail sent out only when tests start failing
● So it manages to keep people’s attention
14. Matrix projects
Run the same thing on
●
different environments
● JDKs
● Databases
● OSes
● App servers
● XML parsers
● ...
Results aggregated
●
18. Inter-Project Support
Host many projects on 1 Hudson
●
● 1+1 > 2
You can…
●
● Execute tests separately and correlate them
with builds
● Mark builds as “promoted” based on test
outcome
● Track which build of libraries are used where
● . . .
19. Test/Build Separation
●Tests often dominate the build time
●Multiple tests are run in parallel
● Improves turn-around time time
JAX-WS #10 JAX-WS #11 JAX-WS #12 …
Unit test with GF
Unit test with Tomcat
SQE test
Test w/o
3OS x 2JDK = 6 tests
container
20. Build Promotion
CI produces a lot of “successful” builds
●
● Often overwhelming to downstream
consumers
●Run tests as fast as you can
●If a build passes tests, promote it
20
21. Got the idea?
Automation
Reduce turn-around time
Make things transparent
Remove people from the loop
Save people’s time
Push jobs to servers, keep workstations
idle for you
23. Distributed builds with Hudson
Master
●
● Serves HTTP requests slave
● Stores all important info 1
slave slave
Slaves
●
8 2
● 170KB single JAR
● Assumed to be
slave slave
unreliable 7 Master 3
● Scale to at least 100
Link
●
slave slave
6 4
● Single bi-di byte stream slave
● No other requirements 5
24. Automated System Installations
● Hudson + PXE plugin
● ISO images of OS
Your corporate IT guy
& his DHCP server
● Slaves
● Power on, hit F12
● PC boots from network (PXE)
● Choose OS from menu
● Installs non-interactively
25. Automated Tool Installation - JDK
JDK from http://java.sun.com/
●
● Automatically chooses the right bundle
● Always up to date with new releases
25
27. Heterogeneous Cluster Challenge
Builds/tests need to run in specific environment
Dependency on individual nodes hurts utilization
jobs slaves
Wombat Windows
Windows test #1
GlassFish Windows
Windows test #2
Hudson Solaris
Windows test #1
Hudson
Solaris test
28. Labels to rescue
Label is a group of slaves
Tie jobs to labels
jobs slaves
Wombat Windows
Windows test #1
Windows
GlassFish Windows
Windows test #2
Hudson Solaris
Windows test #1
Solaris
Hudson Windows
Solaris test #3
29. Forecasting failures
Hudson monitors key health metrics of
●
slaves
● Low disk space, insufficient swap
● Clock out of sync
● Extensible
Slaves go offline automatically
●
32. Hudson EC2 plugin
Automatically provisions slaves on EC2 on
●
demand
● Based on the current load
●Picks the right AMI depending on demand
●Starts slave agent
●Shuts down unused instances
Can run Hudson master in the cloud too
●
33. And a lot more . . .
IDE plugins
iPhone/Android apps
REST API and CLI
Deployment Automation
Hadoop, Selenium
…
35. Conclusions
CI is here to stay
●
● Using lots of PCs is a key
Hudson is very easy to get started
●
● Once started, there is a lot you can do
hudson-ci.org
●