The talk is focused on open-source Groovy/Java libraries that we have developed in order to solve our daily DevOps problems including SSH connectivity, cloud instance management, OS image management and integration testing of Puppet modules with the help of Groovy and Junit.

1. 01

2. Groovy DevOps in the
Cloud
02

3. About us
03

4. Luciano Fiandesio
Bio: Developer, speaker, author, kitchen hacker
Company: Aestas/IT (http://aestasit.com)
E-mail: luciano@aestasit.com
Linkedin: http://www.linkedin.com/in/lucianofiandesio
•
•
•
•
04

5. Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: andrey@aestasit.com
Linkedin: http://www.linkedin.com/in/andreyadamovich
•
•
•
•
05

6. What's this presentation about?
Our take on:
DevOps
Software Provisioning
Continuous Integration
Continuous Delivery
•
•
•
•
06

7. Technologies
AWS - http://aws.amazon.com
Groovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://jenkins-ci.org
Puppet - http://puppetlabs.com
•
•
•
•
•
07

8. Developers +
Operations =
?08

9. Silos
09

10. Conflicts
10

11. Risk
11

12. Agile
12

13. What is DevOps?
13

14. C.A.M.S.
Culture: People over processes and tools. Software is made by and
for people.
Automation: Automation is essential for DevOps to gain quick
feedback.
Measurement: DevOps finds a specific path to measurement. Quality
and shared (or at least aligned) incentives are critical.
Sharing: Creates a culture where people share ideas, processes, and
tools.
•
•
•
•
14

15. Though...
15

16. It's not about
tools!
16

17. It's about
culture and
process!17

18. But without
tools...
18

19. ...it's
definitely
harder!19

20. DevOps imply
automation!
20

21. DevOps imply
structure!
21

22. Infrastructure as Code
Automate the provisioning and maintenance of servers:
Build from source control
Utilize Open Source tools
Ensure testability
•
•
•
22

23. Configuration propagation
23

24. Changes
No Manual Changes!
24

25. Building an automation toolkit
Automation is key
We are JVM hackers
Fragmented ecosystem
Full-stack approach
•
•
•
•
25

26. Tooling
Infrastructure connectivity
Infrastructure provisioning
Infrastructure virtualization
Infrastructure testing
•
•
•
•
26

27. Sshoogr
27

28. Sshoogr Features
Remote command execution
File uploading/downloading
Tunneling
•
•
•
28

29. Sshoogr Example I
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
29

30. Sshoogr Example II
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
30

31. Sshoogr Example III
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ->
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
31

32. Why Groovy?
Groovy is perfect choice for scripting
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a book about it!
•
•
•
•
32

33. Shameless plug
33

34. Puppet
34

35. Why Puppet?
More mature than competition
Large community
Readable DSL
No need to learn Ruby ;)
•
•
•
•
35

36. Puppet example
36

37. Puppet provisioning
37

38. Puppet provisioning
38

39. Puppet provisioning
39

40. Puppet provisioning
40

41. Puppet state management
41

42. Puppet state management
42

43. Puppet state management
43

44. Puppet modules
44

45. Puppet modules
45

46. Puppet modules
46

47. Continuous Integration
47

48. Why Jenkins?
De-facto standard
Stable
There is a plugin for that!
•
•
•
48

49. Gradle
49

50. Gradle Example I
task uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
50

51. Gradle Example I
...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
51

52. Gradle Example II
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
52

53. Jenkins example
53

54. In the meanwhile...
We started developing complex Puppet modules
Modules needs proper testing
...on different platforms
•
•
•
54

55. Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•
•
•
55

56. PuppetUnit
56

57. PUnit
Simple testing tool for verifying remote server state
Uses sshoogr and JUnit
Reuse reporting features of JUnit and Jenkins
As simple as ...
•
•
•
•
57

58. PUnit Example I
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
58

59. PUnit Example II
@Test
def void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
59

60. PUnit Example III
@Test
def void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
60

61. PUnit Example III
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should return at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
61

62. PUnit Example III
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
62

63. PUnit
63

64. More examples
session {
tunnel ('127.0.0.1', 8080) { int localPort ->
// Login to Jenkins.
def driver = new HtmlUnitDriver(false)
driver.manage().timeouts().pageLoadTimeout(300, TimeUnit.
driver.get("http://127.0.0.1:${localPort}/login")
...
01.
02.
03.04.
05.
06.
07.
08.
09.
64

65. More examples
...
def input = driver.findElement(By.name('j_username'))
input.sendKeys('john')
input = driver.findElement(By.name('j_password'))
input.sendKeys('123456')
input.submit()
...
01.
02.
03.
04.
05.
06.
07.
65

66. More examples
...
// Verify login.
def wait = new WebDriverWait(driver, 30)
wait.until ExpectedConditions.presenceOfElementLocated (B
...
}
}
01.
02.
03.
04.
05.
06.
07.
66

67. More examples
session {
tunnel ('127.0.0.1', 80) { int localPort ->
// Initilize repository connection data.
DAVRepositoryFactory.setup()
def url = SVNURL.create('http', null, '127.0.0.1', localP
def repository = SVNRepositoryFactory.create(url)
println "Verifying SVN repository at ${url}"
...
01.
02.
03.04.
05.
06.
07.
08.
09.
67

68. More examples
...
// Setup credentials.
def authManager = SVNWCUtil.createDefaultAuthenticationMa
repository.setAuthenticationManager(authManager)
// Verify repository is at revision 0.
assertEquals 0, repository.getLatestRevision()
...
01.
02.
03.
04.
05.06.
07.
08.
68

69. More examples
...
// Commit first revision.
ISVNEditor editor = repository.getCommitEditor("Initial c
editor.with {
openRoot(-1)
addFile('dummy.txt', null, -1)
applyTextDelta('dummy.txt', null)
def deltaGenerator = new SVNDeltaGenerator()
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
69

70. More examples
...
def checksum = deltaGenerator.sendDelta('dummy.txt', ne
closeFile('dummy.txt', checksum)
def commitInfo = closeEdit()
println commitInfo
}
...
01.
02.
03.
04.
05.
06.
07.
70

71. More examples
...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03.
04.05.
06.
71

72. Next
problem?
72

73. Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•
•
•
73

74. Amazon Web
Services
74

75. Elastic Compute Cloud
Mature
Great API
Virtual hardware variety
OS variety
•
•
•
•
75

76. Gramazon
76

77. Gramazon
Groovy based API for interacting with EC2
Integrates with the rest of the stack
•
•
77

78. Gramazon Example I
task startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gramazon/cloud-do'
stateFileName 'cloud-do.json'
ami 'ami-6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
78

79. Gramazon Example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
79

80. Imgr
80

81. Imgr
A tool for building images
Inspired by Packer
•
•
81

82. Images?
82

83. Supports
Shell
Puppet
•
•
83

84. Configuration Example
84

85. The flow
Start instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
85

86. Images, manifests, tasks
86

87. The big picture
87

88. Aetomation
88

89. Conclusions
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Don't be afraid to try new technologies
Automate!
•
•
•
•
•
89

90. Reading
material
90

91. Groovy 2 Cookbook
91

92. The Phoenix Project
92

93. Continuous Delivery
93

94. Release It
94

95. Programming Amazon EC2
95

96. Gradle in Action
96

97. Technologies to follow
Vagrant - http://www.vagrantup.com/
Docker - https://www.docker.io/
Packer - http://www.packer.io/
Qemu - http://wiki.qemu.org/Main_Page
jclouds - http://jclouds.apache.org/
Cloudbees - http://www.cloudbees.com/
•
•
•
•
•
•
97

98. One more
thing...
98

99. It's all Open
Source!
99

100. Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit: https://github.com/aestasit/puppet-unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•
•
•
•
•
100

101. Seeking
contributors!
101

102. Questions?
102

103. Thank you!
103

104. 104