For the last few months we've been implementing a Continuous Delivery pipeline for the redesign of Time Out. In this talk I will demonstrate a real life example of what our pipeline looks like, the different tools we've used to get it done (phing, github, jenkins, ansible, AWS S3, ...), and peculiarities for PHP and Symfony2 projects. Most importantly, I'll be looking at things we've struggled with along the way and the lessons we've learnt.

1. ONE COMMIT
ONE RELEASE
CONTINUOUSLY DELIVERING A PROJECT

2. Javier López @loalf
Senior Platform Engineer @Rightster!
formerly Software Architect @TimeOut
Certified Symfony Developer
Zend Certified PHP
Co-organizer @desymfony

3. DISCLAIMER

4. AGENDA
CONTINUOUS DELIVERY IN A NUTSHELL
BUILDING THE PIPELINE
BEFORE AND AFTER
1
2
3

5. CONTINUOUS DELIVERY …
… IN A NUTSHELL
1

6. COMMIT BUILD TEST DEPLOY
When and how a change in your code is
going to trigger the pipeline

7. COMMIT BUILD TEST DEPLOY
Checks the integerity of your code.
Produces an artifact.

8. COMMIT BUILD TEST DEPLOY
Checks the integerity of your project.
!
After this step you should be 100% confident that
you could deploy your artifact.

9. COMMIT BUILD TEST DEPLOY
Does what it says in the tin. Triggering this
step should be “a click away”.
Manual

10. COMMIT BUILD TEST DEPLOY
Does what it says in the tin. Triggering this
step should be “a click away”.
Manual

11. THE BOOK

12. THE PIPELINE
BUILDING
2

13. TOOLS!
OF THE TRADE

19. BUILDING!
THE PIPELINE

20. COMMIT BUILD TEST DEPLOY
master
6ebb017

21. COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017

22. COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017
PULL REQUEST

23. COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017 9046c48

24. COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017 9046c48
BUILD TEST DEPLOY

25. COMMIT BUILD TEST DEPLOY

26. 1. Checkout out master branch
COMMIT BUILD TEST DEPLOY

27. 1. Checkout out master branch
2. Generate parameters.ini for each environment
COMMIT BUILD TEST DEPLOY

28. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
COMMIT BUILD TEST DEPLOY

29. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
COMMIT BUILD TEST DEPLOY
—optimizer-­‐autoload —prefer-­‐dist

30. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
COMMIT BUILD TEST DEPLOY

31. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
COMMIT BUILD TEST DEPLOY

32. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
COMMIT BUILD TEST DEPLOY

33. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
COMMIT BUILD TEST DEPLOY

34. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
COMMIT BUILD TEST DEPLOY

35. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
COMMIT BUILD TEST DEPLOY
discard unneeded files

36. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
COMMIT BUILD TEST DEPLOY

37. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
8. Upload artifact to S3
COMMIT BUILD TEST DEPLOY

38. 1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
8. Upload artifact to S3
COMMIT BUILD TEST DEPLOYbuild.xml

39. <?xml version="1.0" encoding="UTF-­‐8" ?>
<project name="Time Out Miyagi" description="Time Out Website V4" default="build"
!
<!-­‐-­‐ Install PHP dependencies (composer) -­‐-­‐>
<target name="build:php-­‐dependencies" depends="build:params">
<echo msg="Installing PHP dependencies (composer install)" />
<exec command="/opt/composer/composer.phar install -­‐-­‐optimize-­‐autoloader -­‐-­‐prefer-­‐dist"
logoutput="true"
checkreturn=“true"
/>
</target>
!
<!-­‐-­‐ Test JS & Generate JS/CSS assets (grunt) -­‐-­‐>
<target name="build:frontend-­‐dependencies">
<echo msg="Test JS and Generate JS/CSS assets (grunt)" />
<exec command="npm install"
logoutput="true" checkreturn="true" dir="./web-­‐src" />
<exec command="xvfb-­‐run grunt -­‐-­‐env=dist"
logoutput="true" checkreturn="true" dir="./web-­‐src" />
</target>
</project>
build.xml

40. COMMIT BUILD TEST DEPLOY
Split into two jobs in Jenkins
1. Test deployment script
2. Run automated tests

41. COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT

42. COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server

43. COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3

44. COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact

45. COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact
4. Set right permissions

46. COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact
4. Set right permissions
5. Update virtual host

47. COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact
4. Set right permissions
5. Update virtual host
6. Reload PHP-FPM and nginx

48. -­‐-­‐-­‐
-­‐ hosts: qa:beta:prod
sudo: true
vars:
amazon_bucket : "releases-­‐miyagi"
base_dir : "/var/www/v4.timeout.com"
project_dir : "{{ base_dir }}/{{ git_hash }}"
web_dir : "{{ project_dir }}/web"
tasks:
-­‐ name: Fetch artefact from S3
s3: bucket={{ amazon_bucket }} aws_access_key={{ amazon_key }}
aws_secret_key={{ amazon_secret }} object={{ git_hash }}.gzip dest="{{ project_dir
}}/{{artefact_zip}}" mode="get"
!
-­‐ name: Uncompress zip file
command: chdir="{{ project_dir }}" tar -­‐xf {{ artefact_zip }}
!
-­‐ name: Apply right permissions to project root
file: dest="{{ project_dir }}" state=directory mode=0755 group=nginx
owner=nginx recurse=true
-­‐ name: Restart PHP-­‐FPM
service: name=php-­‐fpm state=reloaded
deploy.yml

49. COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS

50. COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS
1. Checkout 9046c48

51. COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS
1. Checkout 9046c48
2. Fetch PHP dependencies

52. COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS
1. Checkout 9046c48
2. Fetch PHP dependencies
3. Run behat against QA box

53. COMMIT BUILD TEST DEPLOY
Split into two jobs in Jenkins
1. Run deployment script (production)
2. Run automated tests agains production

54. BUILD DEPLOY
AUTOMATED!
TESTS
6ebb017
6ebb017
QA
6ebb017
QA
DEPLOY
AUTOMATED!
TESTS
6ebb017
PROD
6ebb017
PROD
6ebb017
6ebb017.zip

55. BUILD DEPLOY
AUTOMATED!
TESTS
6ebb017
6ebb017
QA
6ebb017
QA
DEPLOY
AUTOMATED!
TESTS
6ebb017
STAGING
6ebb017
STAGING
DEPLOY
AUTOMATED!
TESTS
6ebb017
PROD
6ebb017
PROD
6ebb017
6ebb017.zip
SAFETY NET

56. PARAMETERS.INI!
DILEMMA

57. 1. One parameters.ini per environment
2. Inject environment specific parameters on virtual host
http://symfony.com/doc/current/cookbook/configuration/external_parameters.html

58. TIME

59. BUILD DEPLOY
AUTOMATED!
TESTS
~3 mins ~30 secs ~5 mins

60. VISIBILITY
EVERYBODY IN THE TEAM SHOULD BE
AWARE OF THE STATUS OF THE PIPELINE

61. BUILD DEPLOY
AUTOMATED!
TESTS
9046c48
9046c48
QA
9046c48
QA
9046c48
BUILD DEPLOY
AUTOMATED!
TESTS
6ebb017
6ebb017
QA
6ebb017
QA
6ebb017
BUILD DEPLOY
AUTOMATED!
TESTS
b0b325
b0b325
QA
b0b325
QA
b0b325
BUILD DEPLOY
AUTOMATED!
TESTS
99e6d6
99e6d6
QA
99e6d6
QA
99e6d6

63. TRACEABILITY
EVERYBODY SHOULD KNOW WHAT VERSION
IS DEPLOYED IN WHICH ENVIRONMENT

64. curl -­‐I http://www.timeout.com/las-­‐vegas
!
HTTP/1.1 200 OK
Server: nginx/1.4.7
Vary: Accept-­‐Encoding
Cache-­‐Control: no-­‐cache
Content-­‐Type: text/html; charset=UTF-­‐8
Date: Fri, 19 Sep 2014 06:07:29 GMT
Transfer-­‐Encoding: chunked
Access-­‐Control-­‐Allow-­‐Origin: http://media.timeout.com
Connection: Keep-­‐Alive
X-­‐TIMEOUT-­‐V: d645127afb423e543d90ab5a7b8eae94f248b137
X-­‐Powered-­‐By: PHP/5.5.14
VERSION NUMBER

65. https://github.com/symfony/symfony/commits/a469c56

66. ROLLING!
BACK

67. YOU’LL NEED!
ALLIES

68. YOUR ALLIES

69. DEV TEAM
YOUR ALLIES

70. DEV TEAM
QA TEAM
YOUR ALLIES

71. DEV TEAM
QA TEAM
DEVOPS
YOUR ALLIES

72. DEV TEAM
QA TEAM
DEVOPS
PRODUCT
YOUR ALLIES

73. DEV TEAM
QA TEAM
DEVOPS
PRODUCT
THE BRASS
YOUR ALLIES

74. BEFORE AFTER&
3

75. ONE QA BOX PER FEATURE
SAME QA BOX FOR EVERYONE
BEFORE
AFTER

76. ~ !WEEKS FROM DEVELOPMENT TO RELEASE
~ !DAYS FROM DEVELOPMENT TO RELEASE
BEFORE
AFTER

77. 5 PEOPLE TO RELEASE TO PRODUCTION
1 PERSON TO RELEASE TO PRODUCTION
BEFORE
AFTER

78. ~30 MINUTES TO RUN DEPLOYMENT SCRIPT
~30 SECONDS TO RUN DEPLOYMENT SCRIPT
BEFORE
AFTER

79. RELEASING WAS AN EVENT
RELEASING WAS A NO EVENT
BEFORE
AFTER

80. @loalf