This document summarizes a keynote presentation on testing and quality in the Scaled Agile Framework (SAFe) given by Mette Bruhn-Pedersen and Derk-Jan de Grood. It discusses how SAFe addresses testing and quality, but that more guidance could be provided, especially around roles. It also provides suggestions for defining a quality strategy and embedding quality practices at each level of the framework, from the team to portfolio levels. The presentation emphasizes building quality in from the start and establishing a chief quality officer role.

1. Mette Bruhn-Pedersen (Safe Journey)
Derk-Jan de Grood (Valori)
Testing and Quality in the
Scaled Agile Framework for
Lean Enterprises
Keynote on the TestNet Fall Event 2018
10-10-2018 The Netherlands

2. Mette Bruhn-Pedersen
2
QA & Testing
+10 years of software testing and
QA experience
• Testing
• Test Management
+10 years in DSTB and ISTQB
• Chair of DSTB
• ISTQB Release Manager
• ISTQB Governance Officer
• Vice Chair of ISTQB Examination
Working Group
+10 years of agile experience
• Agile Team Member
• Test Manager
+4 years of Scaled Agile
Framework
• Agile Team Member
• Community of Practice (CoP)
Facilitator
• Agile Coach
• Agile Transformation Leader
Agile

3. Derk-Jan de Grood
3
QA & Testing
Helping organisations to get more
out of testing and embedding quality in
the organisation
20+ years of Testing
Test strategies & management
Author, Trainer & Speaker
Testing Excellence Award 2016
Helping organisations with embracing
agile, increasing the value stram and
embed the way of Working in the
organisation
Agile Transition Coach
Team coach
Workshop facilitator & trainer
Author of “Starting with Scrum”
Agile

4. Key takeaways
• Test and QA professionals
can add value at all
organisational levels in ways
that are (not) described in
SAFe.
• Defining a quality strategy is
key to handle typical issues
with test and QA activities
spanning multiple teams and
even multiple Release
Trains.
• Getting quality and testing
anchored on Portfolio Level
ensures that quality can be
built-in from top to bottom.
4

5. Why Scale
your Agile?
5

6. Business agility is key as disruption of sectors and
industries becomes the new normal.
6
Organiza(ons more and more start to
understand that business agility and
responsiveness is a key factor to survive and
stay ahead of their compe(tors. In order to yield
value, the work of single agile teams should
therefore be integrated and embedded in larger
business processes. The adop(on of agile is
shi?ing from a single team focus to a wider
organiza6onal approach

7. TestNet Fall Event 2018
Transformation Challenges
7
• Body Level One
• Body Level Two
• Body Level Three
• Body Level Four
• Body Level Five
Agile difficult to implement
Integrating non-development functions
Change resistance
Requirements engineering challenges
Hierarchical management and organizational boundaries
Lack of investment
Coordination challenges in multi-team environment
Different approaches emerge in a multi-team environment
Quality assurance challenges
0 5 10 15 20 25 30 35 40 45 50
Percentage of case organisations
* Challenges and success factors for large-scale agile
transformations: A systematic literature review, Digert, K.
et al, The Journal of Systems and Software, 119 (2016)
87–108
Challenge categories

8. Agile
Frameworks
show the
way !
8

9. DAD
SoS
Agile Methodologies at a Glance
9
Scrum
XP
Kanban
Spotify LeSS
SAFe®
Scalability
Prescriptiveness
Nexus
DSDM
Information combined from several sources:
Agile Scaling Knowledgeable (ASK), Agile Scaling, agilescaling.org
12th annual State of Agile Report, Collabnet VersionOne, collab.net
World Quality Report 2017-18, Ninth Edition, Capgemini, Micro Focus, Sogeti, capgemini.com
Kanban and Scrum making the most of both, Henrik Kniberg & Mattias Skarin, 2010, C4Media

10. SAFe® Portfolio Big Picture
10See latest description on www.scaledagileframework.com/what-is-safe/

11. 11
TestNet Fall Event 2018

12. Quality in
SAFe
12

13. SAFe 4.6
13
TestNet Fall Event 2018

14. What does SAFe say about testing
14
SAFE 4.6
SAFe still says little about:
• Roles in QA & testing
Increased guidance on quality and testing in the new version 4.6 to
support technical agility:
• Emphasis on using Behaviour-Driven Development (BDD)
• Include more types of testing in the sprint
• Modeling stories
• Design with quality
• Implement with quality

15. Agile Software Engineering
15
TestNet Fall Event 2018

16. Check for yourself what’s new !
TestNet Fall Event 2018
16
http://v46.scaledagileframework.com/whats-
new-in-safe-46/

17. Team Layer
17
Strive for a continues flow of
integrated quality solutions

18. What SAFe says: Build in Quality
18

19. SAFe Practises
19
Guidance in SAFe Practices
Collective Ownership
Advocate that anyone can adopt and refactor
code, that code is understandable for everyone
and quality is a collective responsibility
Unit Testing & Test-First Approach
Advocate, pilot and contribute to TDD, ATDD
and BDD
Pair Work Pair with e.g. Users, Developers, PO, BA
Automate CI/CD
Focus on Ci/CD in SAFe 4.6

20. Focus on CI/CD in SAFe 4.6
20
TestNet Fall Event 2018

21. What to test…
21
TestNet Fall Event 2018
Process of creating executable code
- Syntax of source code is correct
User Acceptance Test
- System supports customer journey
Exploratory tests
- Validation check Integration test
- System works within system of other
systems
Functional and NF tests
- System delivers value as intended
- System complies to capacity,
security, availability, etc.
requirements
Smoke test and deployment test
- Systems is correctly and installed and
configured on target environment
Configuration test
- 1: ensure references to external
service in configuration setting are
OK
- 2: Run smoke test
Code Quality Criteria
- e.g. test coverage and code metrics
Unit testing
- Code behaves as expected

22. Review the DOD
• Define quality
measures
• Identify tests
outside the sprint
• Cross-team
alignment
Or more practical…
22

23. Or more practical…
23
Story impacts Programmer Tester User
Web interface
• •
Calculation
• •
Business rule
• •
Letter/e-mail
• •
TEST MATRIX AS STRATEGY ON USER STORY LEVEL
Feedback loop:
found issues to
improve testing
Focus on NFTs
Refinement: Discuss
examples, exceptions and
raise the “how to demo/test” question

24. Program Layer
24
Divide work between individual teams and merge
completed features into a continuous delivery pipeline.
The Agile Release Train (ART) bundles the work of the
teams into controlled product increments.

25. Brian Marick’s Agile Testing Matrix
25

26. Test Strategy
26
Explain how test
activities contribute to
build a quality
solution
Ensure that the
right tests are done (at
the right time, by the
right team)
Assure that the
testing is of sufficient
quality (coaching and
auditing)
Explain how to
gather intelligence on
progress and quality

27. The last sprint of the PI
27

28. The last sprint of the PI
28
• Participate in product or
technical experiments
• Root Cause Analysis
• Identify technical and
business risks
• Improve development
practises
• e2e integration starting
point for planning
Integration
&
Acceptance

29. Other suggestions
29
Operational andorganisationalreadiness
Governance
Emphasize the
quality mind-
set so that
integration
testing is taken
into account at
PI Planning
Test
environments

30. Large Solution Layer
30
Extending Built-In Quality Practices,
Support system and solution integration,
continuous compliance verification

31. 31
TestNet Fall Event 2018
See latest description on http://v46.scaledagileframework.com/compliance

32. Build Quality and Compliancy in
32
TestNet Fall Event 2018
See latest description on http://v46.scaledagileframework.com/compliance

33. 33
TestNet Fall Event 2018
See latest description on http://v46.scaledagileframework.com/compliance

34. There is value in the Solution Layer
34
TestNet Fall Event 2018
See latest description on http://v46.scaledagileframework.com/compliance
• Not many Organisations use the Solution layer
• SAFe offers solutions for embedding Quality
• Benefit is gained by using the solution layer practises
• Testers could emphasis and direct the organisations
towards what is available in the framework
• There are good things that are useful even when you
don’t use the large solution layer

35. Portfolio Layer
35
Translate strategic themes into value
streams and Epics

36. If we want to
embed quality in
to the
organisa(on, it
should be done
at this level.
36

37. 37
In Order to embed quality in our
development practices we are looking for
an
CHIEF QUALITY OFFICER (M/F)
He/She wil be responsible for………

38. 38
In Order to embed quality in our
development practices we are looking for
an
CHIEF QuALITY OFFICER (M/F)
He/She wil be responsible for………
Prioritise
Compliance
Epics
Defines KPIs to
assess UX and
Value streams
Defining strategic
themes with a
quality focus

39. Our Quality Hero on C level
• The Chief Quality Officer is a representa(on of quality
management within the company.
• Keeps quality in the forefront of the company
• Is a strong advocate for the quality process.
• Supports the QA team and stresses the importance of
quality in the organisa(on.
• Ensures that metrics such as Cost of Quality, Rework Rates,
and equipment u(lisa(on and (me-to-market enhances
the transparency of development and opera(ons.
• Defines quality themes at a strategic level (all the way up
to the SAFe PorPolio level)
• Strives for sustainable quality of delivery and development
(and even beRer constantly improving)
39

40. Next steps !
40

41. More info……
41
Download at https://
huddle.eurostarsoftwaretesting.com/
download-testing-quality-scaled-
agile-framework-lean-enterprises/
Grab a copy of the Bits&Chips
magazine
Scaled Agile
Insider
Must Read

42. • Body Level One
• Body Level Two
• Body Level Three
• Body Level Four
• Body Level Five
42
Frameworks
develop
themselves
(e.g. SAFe 5.x)
Test
Community
defines Add-
Ons

43. Derk-Jan de Grood
Thoughtleader Testing & Agile Transition Coach
E: derkjandegrood@valori.nl
W: www.valori.nl
Contact Information
43
Mette Bruhn-Pedersen
Agile Transformation Leader and Partner
E: mette@safejourney.dk
W: safejourney.dk