1. Mixing
Waterfall, Agile and Outsourcing
at Dutch Rail
Bob Harnisch, ProRail
Tim Koomen, Consultant
www.eurostarconferences.com
@esconfs
#esconfs

2.  Who am I
◦ Bob Harnisch
◦ Manager Testservices at ProRail
 ProRail
◦ Railway company in The Netherlands
◦ Provides a secure, reliable, punctual and sustainable rail network
◦ Develops and maintains rail traffic control systems for rail traffic
controllers
◦ Problem:
 Control systems must Go-Live first-time-right!
 Multiple vendors and suppliers, mixed project approaches
ICT Services
ICT Projects
Testservices
Project
managers

4. Planning
Control
Messaging
Security
Infra
elements
• Real-time control systems
• Safe and reliable
• High available systems 24/7
• Multiple system chains
• Legacy and new
• Mixed platform (VAX VMS, Linux)

5.  Testservices
◦ policy: formal testing department for IT projects
◦ involved at project start-up
◦ responsible for acceptance testing
◦ knowledgeable and experienced test experts
 Structured test approach tailored to ProRail
 Acceptance Test Centre Modelpost (close to 100% representative)
 Technical, complex, real-time, high-availability system
environments
 Non-functional testing very important
 But also office automation (SAP, Sharepoint, GIS)

6.  Different project approaches
◦ Policy: outsourcing of SW development
◦ Agile/scrum approach
◦ Waterfall approach
 (Many) external SW suppliers
◦ Little business knowledge
◦ No representative test environment
 Changing architectures
◦ From old legacy to SOA
 Testing is vital for operation
◦ Installation window 01:00 – 04:00 hrs weekend night
 1st time right => next window in 6 months
◦ Any installation or operational failure:
 We’re instantly on the news!

7.  How to cope with issues …?
 Anyone here that can give me advice……?

8.  Tim Koomen, independent testconsultant
8

9.  Baseline of testing is good
 Last few years explosive increase of
agile/scrum development at suppliers. Questions:
◦ Is separate acceptance testing still necessary?
◦ If so, how to align planning of sprints with acceptance tests at test centre
◦ How to coordinate all testing activities?
◦ Controlling the quality of testing?
 Shift to Service Oriented Architectures (ESB, Tibco)
◦ Different testing?
◦ Other measures?
9

10. Business
Beleid Acceptatietest
Pilot STD STR
Plan
Subsysteem
Ontwerp
Systeem
Specificatie
Systeem
Ontwerp
Gebruikers
wensen
Subsystemen
Integratie
(Pilot)
Busi-ness
Case
Context
VL
(OCD
Post21)
Proces
boek
VL
Proces
boek
VL
- Proces-sen
(SSS
Post21)
-KPI’s
Proces
boek
VL
-Bedrijfs-objecten
(SSDD
Post21)
-Externe
eisen
- wetgeving
- e.d.
OCD
-Organi-satie
en
Geautoma-tiseerde
Onderst.
Beheer
Proce
dures
SSS
(Use
Cases)
IRS
(externe
systemen)
SSDD
IDD
Project
Brief
Subsysteem
Specificatie
Syste
em
Systeem =>
Subsystemen
SRS
IRS
sub
systemen
Subsysteme
n
Locatie
Specifieke
Test
Subsysteme
n
SDD DBDD IDD
STD STR
STD STR
STD STR
STD STR
STD STR
STD STR
STD STR
Vrijgave
Business
Requirements
Business
Ontwerp
MTP
systeem
STP
systeem
Unit
Coderen &
Test
Mega
Integratie
Test
Validatietest
Systeem
STP
Systeem/
Sub-systemen
Decharge leverancier
Syste
em
Knip
HLOB
IRS
sub
systemen
Systeem
Kwalifica
tie
IntegrTateiset
Test
Subsystemen
Subsysteem
Kwalificatie
Test
Review
Review
Legenda:
• ProRail architecten,
ontwerpers of andere ProRail
onderdelen, in ieder geval
buiten het werkgebied van TS
• project, ontwerpers
• leverancier
• integrator
• Test Services
10

11. 11
Business
policy
Business
requirements
Business
design
Architecture
design
Acceptance
test
Going
live
Scrum
development
and system
test

12. YES, so far ProRail still needs it
 Representative test environment
 Testing non-functionals
 Testing the chain of systems:
 at ProRail 1st time large scale system integration
 Operational management tests
 Many suppliers, many scrum teams:
 quality not always predictable
 Few demands on supplier testing:
 mostly deliverables like test plan, design and report
 Suppliers have little business knowledge
 Configuration management of application/environment:
 not always as it should
12

13. Business
policy
Business
requirements
Business
design
Architecture
design
Scrum
development
and system
test
Acceptance
test
Going
live
PoC
or
Pilot
Shadow run
13

14. Business
policy
Business
requirements
Business
design
Architecture
design
Scrum
development
and system
test
Acceptance
test
Going
live
PoC
or
Pilot
Shadow run
14
Typical scrum tests:
Unit test
Code review
Functional test (user
story test)
(Service) integration test
Non-functional tests
Regression test
Typical acceptance tests:
(Service) integration test
Validation test
Chain test
UAT
Non-functional tests
Location specific test
Operational acc.test

15. 15
Business
policy
Business
requirements
Business
design
Architecture
design
Going
live
PoC
or
Pilot
Shadow run
System
requirements,
user stories
System-design
&
specification
Coding
Unit tests
Code reviews
Validation
(automated)
Regression
Non-functional
tests
test
(Service)
integration
Functional
(qualification)
tests
tests
Test
Chain test
Location
Specific
Test
User
acceptance
test
Non-functional
tests
Deliverables:
• Testplan
• Test scripts
• Test report
(Service)
Integration
Tests
Business architect
Architect / designer
Deliverables:
• Project Start Architecture
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
Testservices
Definition of Done / deliverables:
• Test scripts
• Test Report
• User Manual
• Installation Guide
cafetaria

16. Architectural changes to SOA
 Measure 1: test effort on services, non-functionals, platform changes
◦ Experience:
 non-functional requirements missing or not well described (architecture documentation )
 (Lack of attention for) non-functionals is a major cause for project delay
Improve quality at beginning of project
 Measure 2: improve review process
◦ We have initiated an improved review process, together with architects
◦ Review the Product from 4 Directions:
 Up: Norms, standards, reference architecture
 Left: Consistency with other system documentation
 Right: Usable (for users), testable (testers)
 Down: Designable, Buildable, Implementable
◦ Use of roles versus directions
 (Peer) architect reviews architecture document from Up and Left
 Designer/ Tester/ Developer: reviews from Down, Left, and Right
 Sr user reviews from Right
◦ Reviews form includes checklist per role
16
4DR

17. 17
Business
policy
Business
requirements
Business
design
Architecture
design
Going
live
PoC
or
Pilot
Shadow run
System
requirements,
user stories
System-design
&
specificatio
n
Coding
Unit tests
Code reviews
Validation
Test
Chain test
(automated)
Regression
test
Non-functional
tests
(Service)
integration
tests
Functional
(qualification)
tests
Location
Specific
Test
Non-functional
tests
User
acceptance
test
(Service)
Integration
Tests
Architect /
designer
Business
architect
Product Owner / Supplier/ ProRail DevelopmentTeam
Test services
Integrator
4DR
4DR
4DR
cafetaria
Deliverables:
• Project Start Architecture
Deliverables:
• Testplan
• Test scripts
• Test report
Definition of Done / deliverables:
• Test scripts
• Test Report
• User Manual
• Installation Guide

18. Business
policy
Business
requirements
Business
design
System
requirements,
user stories
System-design
&
specification
Coding
Unit tests
Code
reviews
(automated)
Regression
test
Non-functional
tests
Functional
(qualification)
tests
4DR
(Service)
integration
tests
Validation
Test
Chain test
Location
Specific
Test
Non-functional
tests
User
acceptance
test
Going
live
PoC
or
Pilot
Shadow running
Test services
Business
architect
Architect /
designer
Users
Deliverables:
Testplan,
testscripts,
Testreport
Definition of Done / deliverables:
Testscripts, testreport, user maual,
installation guide
Legenda:
• ProRail architects, designers or other
ProRail roles, outside scope Test
services
• project, developers
• Supplier, ProRail product owner
• Test Services
(Service)
Integration
Tests
Architecture
design
4DR
4DR
Deliverables:
Project Start
Architecture
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
18
cafetaria

19. Business
policy
Business
requirements
Business
design
System
requirements,
user stories
System-design
&
specification
Coding
Code reviews
Unit tests
(automated)
Regression test
Non-functional
tests
Functional
(qualification)
tests
4RR
(Service)
integration tests
Validation
Test
Chain test
Location
Specific
Test
Non-functional
tests
User
acceptance
test
Going
live
PoC
or
Pilot
Shadow running
Test services
Business
architect
Architect /
designer
Users
Master Test Plan
MTP
Legenda:
• ProRail architects, designers or
other ProRail roles, outside scope
Test services
• project, developers
• Supplier, ProRail product owner
• Test Services
(Service)
Integration
Tests
Architecture
design
4RR
4RR
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
19
cafetaria
Deliverables:
Testplan,
testscripts,
Testreport
Definition of Done / deliverables:
Testscripts, testreport, user maual,
installation guide
Deliverables:
Project Start
Architecture

20.  Implement it
 Train testmanagers, implement in projects
 Build up experience
 1st Project
 2nd Project
 ……
 Next Project
 Some time later….
20

21. • Satisfied with
Stakeholders
– Roll-out and acceptance of model
– Good fit to current practice, clarifies testing
– Early identification & alignment of test activities at suppliers and ProRail
gives flexibility in finding test solutions
– Early involvement
– Coaching of test managers
– Largest changes: choice and frequency of tests, tighter control of
supplier tests and working in or with scrum teams
• To be improved
– 4 Direction Reviews, as Testservices is not the natural owner
– Influencing and controlling the supplier tests
• Not a test issue
– Agile / scrum in general: product owner is the (weakest!) link …
– … and the hardest role!
21
Product
owner
Scrum team

22.  Waterfall
 Agile
 Combined
 United
 Outsourcing
 Testing
 ProRail
 …
22
WAUW

23. Business
policy
Business
requirements
Business
design
System
requirements,
user stories
System-design
&
specification
Coding
Code reviews
Unit tests
(automated)
Regression test
Non-functional
tests
Functional
(qualification)
tests
4RR
(Service)
integration tests
Validation
Test
Chain test
Location
Specific
Test
Non-functional
tests
User
acceptance
test
Going
live
PoC
or
Pilot
Shadow running
Test services
Business
architect
Architect /
designer
Users
Master test plan
MTP
Legenda:
• ProRail architects, designers or
other ProRail roles, outside scope
Test services
• project, developers
• Supplier, ProRail product owner
• Test Services
(Service)
Integration
Tests
Architecture
design
4RR
4RR
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
23
cafetaria
Deliverables:
Testplan,
testscripts,
Testreport
Definition of Done / deliverables:
Testscripts, testreport, user maual,
installation guide
Deliverables:
Project Start
Architecture

24.  ProRail’s test solution for hybrid situations: WAU! - testmodel
 Acceptance testing is a part of this solution
 but only if it delivers value
 Flexibility in choosing the required tests
 coordinated by a Master Test Plan
 Early test involvement in reviews
 4DR
 Model for communication at start project and for awareness
 for testers and rest of project organisation
 In future to 100% agile?
 We’re ready!
24

25. 25

26. bob.harnisch@prorail.nl
+31(0)647268603
www.prorail.nl
info@timkoomen.nl
+31(0)634139260
www.timkoomen.nl

27.  Between 40-60% of IT-projects use scrum
 Always a project manager
 Product owners: usually 2, one big project 5
 Product owner: mostly an information analist or architect
 Number of suppliers: 1-2
 Number of scrum teams: 1-4
 Test role(s) in scrum team? Usually, but not always. If yes, then
usually from supplier, sometimes from ProRail
 Frequency of acceptance testing: from 1 x at the end to almost
each sprint
27