O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Taking Jenkins Pipeline to the Extreme

Slide deck from Jenkins User Conference Tel Aviv 2018.
Talking about suggested (best?) practices, tips and tricks, using Jenkins pipeline scripts with shared libraries, managing shared libraries, using docker compose, and more.

  • Entre para ver os comentários

Taking Jenkins Pipeline to the Extreme

  1. 1. Taking Jenkins Pipeline to the Extreme Yinon Avraham Tech Lead @ eBay Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.com
  2. 2. AGENDA ● Why ● What ● How ● WOW! Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference
  3. 3. WHY Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference
  4. 4. MICROSERVICES Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference Service Service Service Service SQL DB Service SQL DB Service NoSQL DB HTTPHTTP HTTP HTTP 3rd Party Service
  5. 5. WHAT Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference
  6. 6. IMPROVED CI/CD PROCESS CI: For each active branch: 1. Build 2. Unit Tests 3. Publish Artifacts 4. Integration Tests Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference M F RF D CI M F RF D CI Deploy & TestDeploy & Test Deploy to Production Deploy to Production Repo 2Repo 1 Deploy to Acceptance Environment & Test
  7. 7. REQUIREMENTS 1. Configuration as Code 2. Multibranch Coverage 3. Visibility 4. Reuse Job Configuration (D.R.Y) 5. Integration Tests Isolation 6. Self Contained Configuration Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference
  8. 8. REQUIREMENTS Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference
  9. 9. HOW Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference
  10. 10. CODE REUSE - GLOBAL PIPELINE LIBRARIES Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference Jenkinsfile #!groovy @Library('buildtools@1.x') _ properties([ parameters([forcePublishParam()]), discardOldBuildsProperty() ]) withMavenDockerNode(jdk: 'JDK8') { buildTools -> stage('Setup') { buildTools.init() dir('app') { checkout scm } } dir('app') { mavenBuildTestPublish() } failAsUnstable { withDockerEx { dockerEx -> def kafkaIp stage('Start Sidecars') { kafkaIp = startKafkaSidecar(dockerEx) } stage('Integration Tests') { dir('app') { runIntegrationTests(kafkaIp) } } } } } def call() { stage('Build') { sh 'mvn clean verify –U –B –fae –Dmaven.test.failure.ignore' } stage('Unit Tests') { junit '**/target/surefire-reports/TEST-*.xml' } stage('Publish') { def msg = canDeploy() if (msg == 'ok' || params.FORCE_PUBLISH) { sh 'mvn deploy' } else { echo "Publish skipped, cannot deploy: ${msg}" } } } vars/mavenBuildTestPublish
  11. 11. ISOLATION – “SIDECAR” PATTERN Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference Jenkins Master Jenkins Slave Jenkins Slave Kafka Broker Schema Registry Zookeeper kafka sidecar Kafka Broker Schema Registry Zookeeper kafka sidecar
  12. 12. ISOLATION – “SIDECAR” PATTERN Docker Extension – Support Docker Compose Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference import com.ebay.tools.DockerEx def call(body) { def dockerEx = new DockerEx(this) try { body(dockerEx) } finally { dockerEx.close() } } vars/withDockerEx def call(dockerEx) { def kafka = dockerEx.compose('kafka', [file: 'buildtools/docker/kafka.yml']) kafka.defaultNetwork.create() def ip = kafka.inspectGateway() echo "Kafka IP: ${ip}" withEnv(["KAFKA_IP=${ip}"]) { kafka.up().ps() } waitForPort host: ip, port: 29092 // Kafka broker waitForEndpoint url: "http://${ip}:8081" // Schema Registry ip } vars/startKafkaSidecar withDockerEx { dockerEx -> def kafkaIp stage('Start Sidecars') { kafkaIp = startKafkaSidecar(dockerEx) } stage('Integration Tests') { dir('app') { runIntegrationTests(kafkaIp) } } } class DockerEx implements Serializable { protected final def script private final ArrayMap compositions = new ArrayMap() DockerEx(script) { this.script = script } DockerExCompose compose(String name, Map config = [:]) { DockerEx dockerEx = this compositions.get(name, { -> new DockerExCompose(docker, name, config) } } void close() { closeAllCompositions() } void closeAllCompositions() { compositions.forEach { _, compose -> try { compose.down() } catch (e) { script.echo "Destroy '${compose.name}' failed: ${e}" } } } } src/com/ebay/tools/DockerEx
  13. 13. ISOLATION – “SIDECAR” PATTERN Docker Extension – Support Docker Compose Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference class DockerExCompose implements Serializable { private final DockerEx dockerEx private final def script private final ArrayMap config final String name DockerExCompose(DockerEx dockerEx, String name, Map config = [:]) { this.dockerEx = dockerEx this.script = dockerEx.script this.name = name this.config = config } DockerExCompose up() { script.sh "docker-compose –project-name $name " + "${fileArg()} up -d" this } DockerExCompose down() { script.sh "docker-compose ${fileArg()} down" this } src/com/ebay/tools/DockerExCompose DockerExCompose ps() { script.sh "docker-compose ${fileArg()} ps" this } String inspectGateway() { defaultNetwork.inspectGateway() } DockerExNetwork getDefaultNetwork() { docker.network("${name}_default".toString()) } private String fileArg() { def file = config.get('file') file ? "--file ${file}" : '' } } String inspectGateway() { def res = script.sh(script: "docker network inspect -f " + "'{{range .IPAM.Config}}{{.Gateway}}{{end}}' $name", returnStdout: true) res.trim().split('/')[0].toString() }
  14. 14. SELF CONTAINED CONFIGURATION Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference withMavenDockerNode import com.ebay.tools.BuildTools def call(Map args, body) { timestamps { def timeoutArgs = args.timeout ?: [time: 1, unit: 'HOURS'] timeout (timeoutArgs) { def label = 'maven-docker-builder' mavenDockerPodTemplate(label) { node(label) { def buildTools = new BuildTools(this, args) body(buildTools) } } } } } private void mavenDockerPodTemplate(String label, body) { podTemplate( label: label, containers: [ containerTemplate( name: 'jnlp', image: ... )] ... ) { body() } } Jenkinsfile #!groovy @Library('buildtools@1.x') _ properties([ parameters([forcePublishParam()]), discardOldBuildsProperty() ]) ... src/com/ebay/tools/BuildTools class BuildTools implements Serializable { private final def script private final ArrayMap args void init() { gitCloneBuildTools() setJavaHome(args.get('jdk')) printToolsVersion() } private void printToolsVersion() { script.sh script: 'mvn --version', returnStatus: true script.sh script: 'java -version', returnStatus: true script.sh script: 'docker version', returnStatus: true } }
  15. 15. TIPS & TRICKS ● Do not load global libraries implicitly by default. This is sometimes redundant and it is not visible ● Do load global libraries explicitly with specific stable development branch, e.g. “1.x”. This way you get fixes and enhancements transparently, but keep the option for “2.x” ● Do treat your global pipeline library as any other library you write (e.g. versioning, testing, etc.) Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference Global Pipeline Library Management
  16. 16. TIPS & TRICKS Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference Global Pipeline Library Management - Testing #!groovy @Library('buildtools@1.x') _ import static com.ebay.tools.test.PipelineTestUtils.* withMavenDockerNode { pipelineTests(this) { testSuite('withBuildTools, shEx') { test("withBuildTools - Test build tools with JDK8") { withBuildTools(jdk: 'JDK8') { buildTools -> def result = shEx(script: 'java -version') assertStringContains expected: 'version "1.8.', actual: result.get('err'), message: 'Java version not as expected' assertEquals expected: 0, actual: result.get('status'), message: "Return status not as expected" } } ... } } }
  17. 17. TIPS & TRICKS ● Do prototype, test and play using the online script editor ● Do use the “Replay” functionality ● Do echo, echo, echo – write decisions and state to the console ● Trick: interactiveShell Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference Pipeline Script Development and Debugging def call() { timeout(time: 10, unit: 'MINUTES') { echo "[Shell] ***** Starting interactive shell *****" String cmd = '' while (cmd != 'exit') { cmd = input(id: 'cmd', message: 'Command:', parameters: [ [$class: 'TextParameterDefinition', name: 'cmd', description: 'Enter shell command to run, "exit" to stop the shell'] ]) def ret = sh(script: cmd, returnStatus: true) echo "[Shell] Return Code: $ret" } echo "[Shell] Bye!" } }
  18. 18. TIPS & TRICKS ● Do organize in stages to reflect your pipeline flow, think visibility ● Do echo, echo, echo (yes, I already wrote it, it’s important) ● Do control the result status to reflect the actual result ● Do write your global library to express your DSL Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference Increase Visibility and Readability
  19. 19. Any Questions? THANK YOU! Yinon Avraham yavraham@ebay.com @yinonavraham Copyright @ 2018 JFrog - All rights reserved | juc-il.jfrog.comUser Conference

×