1.
CHASING CODE QUALITY IN HUGE MULTI-
LOCATION TEAM PROJECT
October 15, 2012 www.ExigenServices.com

2.
GLOBAL EXIGEN SERVICES
Riga St. Petersburg Nizhniy Novgorod Kazan
R&D Center R&D Center R&D Center R&D Center
San Francisco New York
Headquarters
Europe
Eastern Europe
Wuxi
US London Asia-Pacific R&D Center
Minsk Suzhou
R&D Center R&D Center
Frankfurt
Odessa
R&D Center Dnepropetrovsk
• Founded in 2000 R&D Center Adelaide
• 1’500 professionals
Vilnius Johannesburg
• Revenue approx. $70M San Paulo R&D Center
Christchurch
2 www.ExigenServices.com

3.
EXIGEN SUITE COMPONENTS
Sales and Customer Service DISTRIBUTIONCORE Distribution Management
Business Invoice & Payment New
New Policy
Policy Product FNOL Adjudicaton
Business Servicing
Processes Collection Processes Business Servicing Management & Settlement
&
Core BILLINGCORE POLICYCORE CLAIMCORE
Functions
Bill Plan Payment PRODUCT Under- Claim
Claim Claims
Claim
FACTORY
Management Management Rating Writing Processing Financials
Processing Financials
Common Accounts Reporting & Tasks & 3rd Party Correspondence
Sub-Ledger
Services & CRM Compliance Decisions Mgmt Management
Data & Sys Billing Claims Commission Self-Service Reinsurance Analytics 3rd Party
Integration Interface Interface Interface Portal Interface Interface Interface Interface
Operations Configuration Document Content Business Process & Business
Security Activity
Management & Scheduling Generation Management Rules Task Mgmt Monitoring
3 www.ExigenServices.com

4.
WHEN YOU START FAILING
Agile works says:
 One location, team, scrum master and product owner
 Everybody are universal and senior;
But in sometimes we have
 Multi location, teams, scrum masters and PO’s
 And mixed level;
And here problems arise.
4 www.ExigenServices.com

5.
THIS PRESENTATION TOPIC
Stable Continuous Integration
Effective Code review
Useful Static code quality
5 www.ExigenServices.com

6.
TECHNOLOGIES WE SELECTED
 Source control (DSVC) - hg (Mercurial) mercurial.selenic.com
 Build tool - Jenkins jenkins-ci.org
 Review tool - Reviewboard reviewboard.org
 Static code quality - Sonar sonarsource.org
6 www.ExigenServices.com

7.
STABLE CONTINUOUS INTEGRATION
7 www.ExigenServices.com

8.
CI FOR ONE TEAM
8 www.ExigenServices.com

9.
CI WITH MULTIPLE TEAMS
9 www.ExigenServices.com

10.
WHAT TO DO?
Options
 Fight with continuous integration failures every day
 Fail fast. Setup your continuous integration to no accept failing
code.
10 www.ExigenServices.com

11.
FAILING FAST
 “Failing fast” for development means getting response about your
code quickly, from:
 compiler
 deployment
 unit and integration test
 UI tests
 static quality parameters
And all those answers should be given before code gets to source
control.
11 www.ExigenServices.com

12.
MULTIPLE TEAMS – MULTIPLE
CONTINUOUS ENVIRONMENTS
12 www.ExigenServices.com

13.
TEAM BRANCH
13 www.ExigenServices.com

14.
BENEFITS
 Team continuous integration environment is effected only by team
changes.
 Only finished and tested features will get to central. Gatekeeper
would be responsible to merge.
14 www.ExigenServices.com

15.
STILL TO BE IMPROVED
 There is still possibility that gatekeeper will push bad code
 We don’t reach our ultimate goal - STABILITY of central.
15 www.ExigenServices.com

16.
ONE STEP FURTHER - TEAM-MERGE
 To achieve our goal we created CI job that doesn't allow bad code.
It does automatic code merge, compile, test and push.
– rights write to central are revoked for anyone and only "special"
"team-merge" user can do merge.
– only possible way to commit code from team-branch to central
is through "team-merge" script.
16 www.ExigenServices.com

17.
HOW IT WORKS
17 www.ExigenServices.com

18.
HOW TO START IT
18 www.ExigenServices.com

19.
FINAL TIPS
 Your environment should be stable enough to do this approach.
 Better have small number of working test rather than huge number
of failing test.
 Each team has own CI plans – hardware is required
19 www.ExigenServices.com

20.
EFFECTIVE CODE REVIEW
20 www.ExigenServices.com

21.
CODE REVIEW OR NOT CODE REVIEW
 Code review, if:
– You plan long life cycle for your product
– Your customer insist on high quality and you have maintenance
contract.
Don’t code review, if:
– Your product has short life cycle (demos, POC, etc.)
– You don’t care about maintenance after it’s done 
21 www.ExigenServices.com

22.
CODE REVIEW
 There are various proposals on code review –
 Pair Programming
 Review about do less than 100%, with rules, like
Review only Selected commits.
Review only commits from juniors
Random reviews
 Review 100% of code
We chose 100% review.
22 www.ExigenServices.com

23.
CODE REVIEW AFTER COMMIT
 Guarantee that reviewed code is committed
 No special actions required if reviewed code passes
Issues:
 In case code doesn’t pass you have bad code in repository until it’s
fixed.
 Bad code effects others
23 www.ExigenServices.com

24.
CODE REVIEW BEFORE COMMIT
 Review patches or review branch.
 Bad code doesn’t effect others.
Issues:
 You have no guarantee that good code reviewed will get to
repository. As somebody needs to merge it.
What we needed is:
Automated merge on code review pass.
24 www.ExigenServices.com

25.
100% CODE REVIEW
What we want is
1. Work as a team.
2. Review all code changes
3. Do not effect others
Therefore we need code review systems what would assure 100%
review.
Other requirements:
 DVCS-backed review system
 Acts as a front end for to central DVCS-repo
 When review is complete auto merge with default branch
25 www.ExigenServices.com

26.
CODE REVIEW TOOLS
 For git users there is good tool called gerrit
http://code.google.com/p/gerrit/
 If you use mercurial (hg), it’s bit more complicated – as where is
no analogue for gerrit for hg.
 Our approach was to use reviewboard.org and mercurial enhanced
with python scripts what are mimicking gerrit approach.
 To initiate review you should call “hg postreview”
26 www.ExigenServices.com

27.
REVIEWBOARD
27 www.ExigenServices.com

28.
REVIEW CYCLE
28 www.ExigenServices.com

29.
USEFUL STATIC CODE CONTROL
29 www.ExigenServices.com

30.
CONTROL CODE QUALITY
 We want to control basic parameters of code quality
 Static code violation
 Code unit and integration test coverage
 Other code parameters
We can do it effectively with tool called Sonar
30 www.ExigenServices.com

31.
METRICS
 What you want to measure?
– Test Coverage?
– Static code violations?
– Code metrics?
Answer to this is SONAR
Automates metrics and creates reports.
Allows to analyze code to the code line details.
sonarsource.org
31 www.ExigenServices.com

32.
SONAR PROJECT VIEW
32 www.ExigenServices.com

33.
CODE COVERAGE
33 www.ExigenServices.com

34.
CODE DRILLDOWN
34 www.ExigenServices.com

35.
CODE VIOLATIONS
35 www.ExigenServices.com

36.
VIOLATION DRILLDOWN
36 www.ExigenServices.com

37.
WHAT’S NEXT WITH SONAR
 Good is that we have tool to control
 Bad is that what people are still people and tend to do crappy
code.
 number of violations is increasing.
 still repeating same errors.
 code is unmaintainable
37 www.ExigenServices.com

38.
OUR APPROACH
 Decide that violations are critical for you and what you can live
with.
 What we do:
 Using multistage commit we don’t accept code with blocker
violations.
 Commit is declined and developer should rework their code
before next commit.
 Use personal violations view to work with particular developers
coding style.
38 www.ExigenServices.com

39.
PERSONAL VIOLATIONS DRILLDOWN
39 www.ExigenServices.com

40.
PREVENT VIOLATIONS
40 www.ExigenServices.com

41.
TIPS
 Declare violations as critical first.
 Allow teams to fix them.
 Increase severity to blocker after all existing are fixed
 Decline new violations of same type
41 www.ExigenServices.com

42.
QUESTIONS?
42 www.ExigenServices.com