Talk given by Vladimir Gerasimov (Product Management Senior Manager) and Joyce Yeh (Software Engineer) at Salesforce, at STPcon in September 2016 Salesforce delivers three major feature releases a year, made possible with strong collaboration among its team members. In this session we will talk about how Developers and Quality Engineers collaborate in an Agile environment on a daily basis. It all starts with a User Story and ends with satisfied customers. We will walk you through everything in between, from the moment the story is created to the release time when the code is deployed to production. We will use the lifecycle of a User Story to show how different team members are enabled through our Agile process and different tools. Session Takeaways: How Salesforce leverages collaboration between Developers and Quality Engineers to deliver 3 major feature releases a year. How Salesforce maintains the highest quality standards. What quality and development practices are used in scrum team. General lifecycle of a User Story from idea to production at Salesforce.

1. How Developers and Quality
Engineer Collaborate at Salesforce
Using Agile to deliver three major feature releases a year
Vladimir Gerasimov, Sr. Software Engineer
vgerasimov@salesforce.com
/in/vgerasimov87
Joyce Yeh, Software Engineer
jyeh@salesforce.com
/in/joycevyeh

2. Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such
uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ
materially from the results expressed or implied by the forward-looking statements we make. All statements other than
statements of historical fact could be deemed forward-looking, including any projections of product or service availability,
subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of
management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or
technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and
delivering new functionality for our service, new products and services, our new business model, our past operating losses,
possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our
security measures, the outcome of any litigation, risks associated with completed and any possible mergers and
acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain,
and motivate our employees and manage our growth, new releases of our service and successful customer deployment,
our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further
information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report
on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter.
These documents and others containing important disclosures are available on the SEC Filings section of the Investor
Information section of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not
currently available and may not be delivered on time or at all. Customers who purchase our services should make the
purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does
not intend to update these forward-looking statements.
Safe Harbor

3. Our Goals Today:
• Show you how Developers and Testers collaborate
• Show you how Salesforce Technology Organization leverages
Agile methodologies to deliver 3 major releases a year
• Explain the engineering practices used to maintain the highest
quality standards
• Provide practical tips on how to start your own agile
transformation today

4. Salesforce Agile Implementation
Principles
(Lean & Agile)
Management
Practices
(Scrum & Kanban)
Technical Practices
(XP)

5. We’ve Built a Scalable Mode
400 +
Agile Teams @ Salesforce

6. Developers at Salesforce are Responsible for:
• Test-Driven Development
• Unit and Functional Test Automation
• Fixing Regression bugs

7. Quality Engineers at Salesforce are Responsible for:
• Test Design and Test Planning
• Identifying Test Coverage gaps
• Developing Test Frameworks
• Writing Test Automation

8. Agile Teams at Salesforce
QE
Engineers
Product
Owner
Developers
Tech Writer
Scrum
Master

9. It’s All About User Story

10. Collaboration Between Devs and QEs
New Triaged
In
Progress
Ready For
Review
Fixed
QE in
Progress
Closed
How Devs and QEs collaborate through User Stories
• User Story cannot be closed without QE
• Devs and QEs working on the stories together
Devs and QEs
discussed the
work
Devs and QEs
reviewed the
code
Code
checked in
Definition of
Done is met

11. Continuous Planning And Prioritization
Vision
V2MOM
2 Release
Review
1 Release
Review
Sprint
Reviews
Stand
ups
3-5 years
1 year
2 releases
1 release
2 weeks
Daily
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

12. Early communication is the key
• Devs and QEs sit together before work on a user story is started
• It allows to keep both sides on the same page
• Gives Dev first-look test prospective on User Story
• Gives QE and idea about what’s coming next
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

13. Freedom to adjust the process
Teams decide what work or doesn’t work for them
• Team has the freedom to decide
best practices
• Continuous improvement part of the
DNA
• Encourages everyone to own the
process
• Reduces amount of bureaucracy
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

14. Agile Principles to Keep in Mind
For quality prospective
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed
• Build quality in. Quality over more
features. Trust in #1 priority.
• Eliminate waste. Fix test issue, not
disable tests.

15. Dog-fooding Our Own Product
Using your product on daily basis
• Bug and User story tracking
• Internal and external
communication
• Discovering issues earlier
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

16. Code Reviews
• Code reviews help to catch problems
before code is checked in
• Both Dev and QE engineers participate
• Code Review is required for check in
• Code Reviewer knows if check in failed
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

17. Pre-Checkin Validation
Keeps the Build healthy
• Sharing the same code branch for current
release work
• Important to keep build healthy at all
times
• Pre-checkin validation ensures the
change you are submitting is not
breaking the build and major functionality
• Takes about one hour
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

18. Test Failures, Flappers and Long Running Tests
Lock teams with Test Failures older
than 7 days
Opening Bugs for non-stable tests
Opening Bugs for Long Running
Tests
Keeping test healthy and stable
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

19. Lock Out Email
Team gets locked out if they don't fix their Test Failures
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

20. Test Failure Auto Assign
Auto-Assign manages test failures
effectively
It can link Test Failure to the change list
that most likely caused the failure
It groups similar test failures under one
Bug
It eliminates manual work of running
reports and opening bugs
Even when wrong, it still opens bugs
and starts the process of fixing Test
Failure
Helping teams to address test failures
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

21. How Does Definition of Done help the quality?
• Single Definition of Done ensures
understanding within & across teams
• Increases team accountability for
quality and commitment
• Reduces Code debt and possible
points of failure
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

22. Example of Definition of Done
Criteria Features
1 2 3
Code checked in and follows department
standards
☑ ☑
No open regressions. Automated tests
written and reviewed for all regressions
☑ ☑
No open P1 & P2 bugs
☑
Code Coverage 78
%
80
%
100% of test cases logged and executed in
a QA environment, and all P1/P2 cases
passing
☑ ☑
All resolved bugs verified and closed
☑ ☑ ☑
UE has reviewed any new features; P1 and
P2 UI bugs fixed
☑ ☑ ☑
Usability testing completed when
necessary, and feedback incorporated into
backlog
☑ ☑
Code and UI reviewed for 508 compliance;
UE team notified of any non-compliant
features
☑ ☑
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

23. Frequent Product Demos & Reviews
• Demo completed work with team
every sprint
• Demo and review release plan
progress monthly with Executive
participation
• Provide radical transparency
• No surprises come release time
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

24. Frequent Major Releases
Feature
Freeze
Release
Freeze
Done Done Done Release to
Internal Sandbox
& Production
Instances
SB/R0 Release
• 2/3 sandbox
instances
• Production instance
where Salesforce has
largest orgs
R1/R2 Release
• 25% of prod
instances
• All remaining
instances
• Branch locked
• Check-in approval
required
• Incomplete features
disabled
• Code line open for next
release
Monthly Sprint Reviews Release Sprint Staggered Release
Scrum Teams
and Functional Areas
Sign Off
Scrum Teams
Sign Off
Dec Jan Feb Mar MayApr Jun
• Continuous integration w/
600k JUnit and Selenium
tests
• Performance testing
• 110M Apex customer
tests
• Other production tests
• Final performance
testing
New Triaged
In
Progress
Ready
For
Review
Fixed
QE in
Progress
Closed

25. Next Steps
What you can do today
Get Developers to work closer with QEs
• Discuss User Story scope and requirements together
• Participate in Code Reviews
• Have Dev and QE pair on story completion
Test Automation
• Add Automated Test Coverage to Definition of Done
• Create backlog to track remaining Automation work
• Involve Developers in writing Automation and helping
QEs with Test Frameworks
• Test Failures locks when too high
• Enforce Test Automation health
• Invest in Auto-Assign tools and reports

26. Thank you!