Rise of the Machines - Automate your Development

Rise of the machines Automate your development @svenpet

Disclaimer Scripts Chef, Puppet, etc better build pipelines

Disclaimer Ideas through examples

Rise of the machines

ROBOTS are coming to our day to day lives

Rise of the machines IN Software Development

Sven Peters Atlassian Evangelist @svenpet svenpet.com

Automation in Software Development my personal story build scripts (make) 1996 JUnit 2001 2004 ant -> later maven 2005 Eclipse (code generation) Hudson 2006 2009 Puppet 1998 GUI builder 2013 Docker

Build & Tests Deployments Small scripts What are we automating right now?

What are we automating right now? Code Reviews Stand Ups Ops Support Merging Engineering Health

Bots everywhere Coding bots Ops bots Service bots Doc bots Report botsTest bots

Welcome… or I’m new

Add ons Postgres Scala IntelliJ Git JDK Happy installing & setup… for the next 8 hours

Agent CharlieMeet

Download Install Conﬁgure Agent CharlieMeet

Puppet, Shell & Ruby Works with OSX & Ubuntu Atlassian Plugin SDK Bitbucket SSH Keys Git Mercurial HipChat HomeBrew IntelliJ IDEA JDK Maven PostgreSQL Stash Keys and much more…

Without Agent Charlie With Agent Charlie

Coding bots Automate your Coding phase

27’’ is not enough

Chat Email Repo manager IDE Command line Issue Tracker

Issue tracker workflow TODO IN PROGRESS IN REVIEW DONE

BRANCH CODE REFACTOR PULL MERGE Developer workflow

BRANCH CODE REFACTOR PULL MERGE What you should do TODO IN PROGRESS IN REVIEW DONE

BRANCH CODE REFACTOR PULL MERGE What you do TODO IN PROGRESS IN REVIEW DONE

Keep the team updated!

Smart commits mention issue_key and #trigger the issue tracker Github JIRA YouTrack Trello Bitbucket

Control the issue by commits TODO IN PROGRESS IN REVIEW DONE JRA-566 #start-Progress JRA-566 #start-review ﬁxes #566

No tool switching Robot is just receiving commands

TODO IN PROGRESS IN REVIEW DONE branch commit Controlled by action pull create review request merge close review

Our workflow TODO IN PROGRESS IN REVIEW DONE branch pull merge

branch pull merge master JRA-554-bugﬁx

branch pull merge master commits code directly to the master branch!

branch pull merge master

branch pull merge master JRA-554-bugﬁx

bad surprise sh*t happens, but can be avoided

bad surprise sh*t doesn’t happen that often no

branch pull merge don’t branch from a red build master JRA-554-bugﬁx

branch pull merge master JRA-554-bugﬁx invite the best reviewers

Recent comitters or author

auto suggest reviewer

branch pull merge minimum 2 approvals master JRA-554-bugﬁx

branch pull merge unapprove reviewers when code changes master JRA-554-bugﬁx

The need to update older versions

master Version 1.1 Version 1.0 Support older versions

master Version 1.1 Version 1.0 JRA-234-hotﬁx Support older versions

master Version 1.1 Version 1.0 JRA-234-bugﬁx Support older versions boring!

Huhu I can do it!

master Version 1.1 Version 1.0 JRA-234-bugﬁx auto merges

Test bots SMARTER Test Automation

&:hover { text-decoration: none; color: @invertedBodyTextColor background-color: #366ca6; cursor: pointer; }

&:hover { text-decoration: none; color: @invertedBodyTextColor background-color: #366ca6; cursor: pointer; } Static code analysis can catch this! Findbugs - Checkstyle - PMD

Listen to what Freud tells you about source codeyour

Freud Bot &:hover { text-decoration: none; color: @invertedBodyTextColor background-color: #366ca6; cursor: pointer; } You should use a variable instead for . I suggest that you use a variable of similar color:#366CA6 @globalHeaderPrimaryActionFallbackColor @globalHeaderTopColor @primaryLinkColor

Freud Bot for Pull Requests

Freud Bot less review work makes suggestions

We are trying hard to write awesome code

package com.miguelcatalan.materialsearchview; import android.app.Activity; import android.widget.ListView; import android.widget.RelativeLayout; import android.widget.TextView; import java.lang.reflect.Field; import java.util.List; /** * @author Miguel Catalan Bañuls */ public class MaterialSearchView extends FrameLayout implements Filter.FilterListener { private MenuItem mMenuItem; private boolean mIsSearchOpen = false; public MaterialSearchView(Context context) { this(context, null); } public MaterialSearchView(Context context, AttributeSet attrs) { this(context, attrs, 0); @Override public void onTextChanged(CharSequence s, int start, int before, int count) { mUserQuery = s; startFilter(s); MaterialSearchView.this.onTextChanged(s); } } LOC: 32 MyAwesomeClass

One month later

package com.miguelcatalan.materialsearchview; import android.app.Activity; import android.content.Context; import android.content.Intent; import android.content.pm.PackageManager; import android.content.pm.ResolveInfo; import android.content.res.TypedArray; import android.graphics.Rect; import android.graphics.drawable.Drawable; import android.os.Build; import android.os.Parcel; import android.os.Parcelable; import android.speech.RecognizerIntent; import android.text.Editable; import android.text.TextUtils; import android.text.TextWatcher; import android.util.AttributeSet; import android.util.Log; import android.view.KeyEvent; import android.view.LayoutInflater; import android.view.MenuItem; import android.view.View; import android.view.inputmethod.InputMethodManager; import android.widget.AdapterView; import android.widget.EditText; import android.widget.Filter; import android.widget.Filterable; import android.widget.FrameLayout; import android.widget.ImageButton; import android.widget.ListAdapter; import android.widget.ListView; import android.widget.RelativeLayout; import android.widget.TextView; import com.miguelcatalan.materialsearchview.utils.AnimationUtil; …one month later LOC: counting…. MyAwesomeClass

mSearchSrcTextView.setText(null); mSearchSrcTextView.requestFocus(); if (animate) { AnimationUtil.fadeInView(mSearchLayout, AnimationUtil.ANIMATION_DURATION_MEDIUM, new AnimationUtil.AnimationListener() { @Override public boolean onAnimationStart(View view) { return false; } @Override public boolean onAnimationEnd(View view) { if (mSearchViewListener != null) { mSearchViewListener.onSearchViewShown(); } return false; } @Override public boolean onAnimationCancel(View view) { return false; } }); } else { mSearchLayout.setVisibility(VISIBLE); if (mSearchViewListener != null) { mSearchViewListener.onSearchViewShown(); } } mIsSearchOpen = true; } /** * Close search view. */ public void closeSearch() { …one month later LOC: 986 MyAwesomeClass

Code over time 86% 20 1045 27 test coverage deprecated methods old unit tests (JUnit 3) compiler warnings

Code over time 83% test coverage 23 deprecated methods 1136 old unit tests (JUnit 3) 36 compiler warnings

Code over time 75% test coverage 32 deprecated methods 1289 old unit tests (JUnit 3) 45 compiler warnings

Developers are trying to solve problems Developers don’t write bad code on purpose

Calling Dr. Codemeasure constantly the engineering health

measure constantly the engineering health Calling Dr. Code

jira.stats.tests.junit.3.count No more JUnit 3 tests! JUnit 3 JUnit 4

jira.stats.bundled.jars.in.plugins Inspect all the things jira.stats.tests.junit.3.count

Hall of Shame! You’ll enter the JS not minified API compatibility :( New JUnit3 test Caught by Dr Code?

Hall of Shame! You’ll enter the Wallboard - so everyone sees it!

Hall of Fame! You’ll enter the Fixed a matrix?

public abstract class Task { public final void executeWith(Callback callback) { execute(); if (callback != null) { callback.call(); } } public abstract void execute(); } public abstract class Task { public final void executeWith(Callback callback) { execute(); if (callback != null) { callback.call(20); } } public abstract void execute(); } Old version New version

Old version New version

Old version New version Hard to spot?

Image Compare Bot Compare images (on build) Alert on differences (build warning) Easy reporting (build report)

Our functional tests ran for h ursso we ran them parallel

Test block 1 Test block 2 Test block 3 Test block 4 Test block 5 statically sized test blocks waiting for the longest test block

Hallelujah Functional Test Balancer

T1 T 2 T 3 T4 T5 T6 T7 T8 T9 T10 T11 T 12 T13 T14 Hallelujah

build status

Flaky Test Detector

Flaky Test Detector failing test rerun test successful test test in quarantaine continue build create issue

1888 flaky tests detected 817 quarantined tests fixed

Flaky Test Detector no log file searching forces fixing flaky tests no manual reruns no more ‘this one again’ moments

OPS bots Automated Operations

Ops nowproblem

ops Slow feedback loop devprob

Faster feedback loop devprob

Faster feedback loop knows which team -> service responsibility

Faster feedback loop But that’s a lot of configuration!

Faster feedback loop add team in deployment descriptor add connection to pager duty

Own your code end-to-end

Prepared for the release? WAIT… Release Notes?

New Feature Improvement Bugfix Do you know which features really ship? Release Branch

Merged JRA-345 Commit messages + Issue information = Release notes Release Notes Bot

Label in GitHub Issues Label in GitHub Issues

New features Improvements Fixed bugs

Issue Description as Release Notes Human readable issue descriptions Think about customer value No surprises

Service bots Automate your Service

Support Developer scanning log filesask for log file system knowledge

pause work

Support Developer scanning log filesask for log file system knowledge

Support Hercules scanning log filesask for log file system knowledge

Teaching the robot

Hercules get’s smarter suggest read update

{ Well f*%k me THE ROBOT was right! }

Your boss has a really important task for your team

Everyone is working from home today

With chat you can reach everyone immediately

All these questions! How do I download the Sprint report? How do I set up a build for my feature branch? What Java versions do we support?

!ask I’ll look it up for you Self service for chat

!ask the knowledge base first!

REPORT bots Manual Reporting sucks

Stand Up Automating s ?

too long setup VC calls timezones remind people Stand Up Problems

Never miss a stand up Music - Start at 09:29:00 Crontab!

Video - Open at 09:30 No video set up time

Stand up report bots colocated distributed different timezones

Status update

Team update

Stand up bot

Exception Build failure New issue Deployment Chat Bots everywhere

ΩΩΩΩ Builds

ΩΩΩΩ Deployments

ΩΩΩΩ Exceptions

Chat Bots everywhere Fast feedback loops Team knowledge Start discussions

The build failed New deployment

New important issue New deployment

New important issue New deployment Get the big picture

Get the big picture

Every team has one or more http://atlasboard.bitbucket.org/

Every team has one or more http://atlasboard.bitbucket.org/ see what’s going on point directly

All kind of boards AdvocatesDeveloper Support

Bots everywhere Coding bots Ops bots Service bots Doc bots Report botsTest bots

Reinvent the wheel? Lots of unmaintained projects Writing automation code Very specific problem

No. 1 Problem

10% for better engineering health work time Technical debt

is serious fun! Automation Go - Ruby - PHP - Python Languages Puppet - Chef - Ansible - Docker Technologies Fast feedback Success Your Problems Motivation

From scripts on a developer hard disc

to professional automation

Source Control Add code to

Code Reviews Do for every change

Tests Write

Deployment Scripts Have

Keep your automated systems under control

Keep your automated systems under control Lots of different places

Instances for automation services

costcentre ow ner service We use labels for better overview

Micros <our own PaaS> Docker Java Go Python

micros service:create <service-name> Service descriptor Owner PagerDuty Cost Centre MicrosSimple to create & register

Were are we heading?

Better customer insides better conclusions collecting more data Were are we heading?

Faster dev feedback loops less time updating & searching integrated tools Were are we heading?

Better Quality deeper code analysis smarter tests Were are we heading?

Building software Parts can be automated craft is a Creating customer value is an art Hard to automate

Rise of the machines …Skynet became self aware

@svenpet svenpet.com/talks I’ll be back… find slides here

Photo Credits “VW Golf TDI Clean Diesel was 2010 8983" by Mariordo Mario Roberto Duran Ortiz - Own work. Licensed under CC BY-SA 3.0 via Commons - https:// commons.wikimedia.org/wiki/ Backyard Series by syauqee mohamad "http:// www.flickr.com/photos/7901820@N07/6945260735" Tesla Model S by Chrishmt0423 from flickr Burnt & Abandoned Computer "http://www.flickr.com/ photos/99649389@N02/13262802714"