Successful project managers manage achievements, not activities. What this means is that the readiness and acceptability of project deliverables are what must be monitored, rather than the completion status of activities. But most managers track time, cost and effort only because they are easy to measure. The decision making in projects is critically dependent on accurate information and this tutorial describes how QA and testing is the main source of Project Intelligence (PI). Delegates will see how PI can be gathered, analysed and presented to senior management to support the crunch decisions that often go unmade because of lack of information. This tutorial offers a new approach to test strategy, using techniques based on the emerging discipline called Benefits Realisation Management. Results Chain models will be used to show the fundamental relationships between project goals, risks, project activities and dependencies being used as a framework for test strategy. PI promotes better communication within projects and supports the need of senior management to make better decisions when setbacks occur early, when change becomes inevitable and testing is squeezed. The techniques of PI will be illustrated and practised using several exercises during the day. A case study will be used to explain how PI can work in a realistic setting and the way PI can be used to support decision making are discussed. The Project Intelligence approach does not come naturally. Individuals may or may not embrace change, but organisational culture always resists it. Organisational culture allows an organisation to bend when a changed approach is required. But that same culture tends to restore the old ways, when the pressure to work differently is removed. So often, change programmes fail because initial successes cannot be sustained. The tutorial outlines a change process specifically targeted at changing the culture of a business to demand better intelligence from its project management and establish a permanent need for IT to engage with business to secure business benefits. The PI approach transforms management's perception of testing, promotes closer collaboration between development and test and better informs management decision making.

1. Managing Software Projects With
Intelligence
Paul Gerrard
Gerrard Consulting
Silvaplana
35 Higher Fence Road
Macclesfield
Cheshire
SK10 1QF
e: paul@gerrardconsulting.com
w: http://gerrardconsulting.com
t: +44 (0)7940 547894Slide 1 Assurance with Intelligence

2.  Why Projects Fail
 Introduction to Project Intelligence
 Results Chain Analysis
 PI Strategy
 Test Strategy Completion
 Goal, Risk and Requirements Based Testing
 Close
Agenda
Slide 3 Assurance with Intelligence

3. Why Projects Fail
Slide 4 Assurance with Intelligence

4.  Won’t bore you with yet another survey of IT projects that
fail - take it for granted, most do
 But why is this?
 Hundreds of modes of project failure – all very well
documented
 But a large proportion of project failures can be traced to
poor decision making:
- Constraints to decision making - “you are not allowed to decide”
- Decisions already made cannot be changed
- Decisions that are ‘too big’ to make now – it’s simply too late
 Sometimes, the worst decisions are those that are never
made at all.
Why projects fail
Slide 5 Assurance with Intelligence

5.  Imagine a world where the project plan is perfect, and
everything goes well
 There are few decisions to make except exit/acceptance
reviews
- These would be easy to make, wouldn’t they?
 But plans aren’t perfect and things don’t go to plan
 Most decision-making is required when things don’t go to plan
- Late delivery – we are late or expect to be late
- Poor quality or stability – deliverables not meeting requirements, fail to
stabilise or are defective
- Technical challenges that cannot be met – technology doesn’t work,
can’t demonstrate working systems, cannot achieve objectives with
proposed solution.
Decision-making when things go wrong
Slide 6 Assurance with Intelligence

6.  When called upon to make a call on delays, quality or solution
viability…
- Stakeholders and project managers usually lack the right information at
the right time
- Most of the information actually required is TEST EVIDENCE
 Early and continuous review and testing will highlight
- Items not ready (we can’t test them)
- Poor quality (they are buggy)
- Developers that cannot deliver
 I call this evidence Project Intelligence
 THE TEST STRATEGY IS A STRATEGY FOR GATHERING
PROJECT INTELLIGENCE.
Decision-making dependent on good
information
Slide 7 Assurance with Intelligence

7.  The way we plan projects and monitor/manage
progress is flawed
 Project plans set out a prediction of the future
 A network of interdependent activities aligned
with a calendar
 We identify activities, costs and timelines
 We monitor activities, spend and the calendar
 We act when progress doesn’t meet the plan.
Our project management approach is
critically flawed
Slide 8 Assurance with Intelligence

8.  We monitor progress by checking…
- The time – “it’s 12.45pm – we must be half way”
- The fuel gauge – “we are nearly out of gas – we must
be there!”
 This is nonsense of course
 We are monitoring INPUTS, NOT OUTPUTS
 We should be monitoring ACHIEVEMENTS, not
ACTIVITY
 What are achievements?
 How do we monitor them?
But STOP! Imagine we planned a
journey this way
Slide 9 Assurance with Intelligence

9.  The achievements of most significance are
- Deliverables (on time and of good quality)
- Decisions made (on the acceptability of deliverables)
 But wait! The REAL measure of achievement is the
acceptance decision, based on evidence
- Requirements
- Designs
- Code
- Systems
- Etc…
 Testing provides that evidence.
The measurement of achievement
Slide 10 Assurance with Intelligence

10. Slide 11
Pawnbroker model of IT development
RequiredDesigned
DeliveredUsed!
• Developers usually complain
that requirements change
• Were the users focused on the
business goal?
• Developers focus on technical
deliverables, as usual
• But who keeps them on the
right track?
• Required, designed, not delivered (cuts through deadline pressure?)
• Not required, designed, delivered (an analyst's bright idea?)
• Required, not designed, delivered (the user bribed a programmer?)
Slide 11 Assurance with Intelligence

11. Slide 12
Four-eyed plans
Increasingly
Inaccurate
IT- focused
Initial Plans
The plan is a model of the project. The real project
consists of people, organisation, goals and risks.
Slide 12 Assurance with Intelligence

12.  Our deliverables are intermediate project goals
- Are they available? Are they acceptable? Do they meet
requirements?
 Testing can answer these questions
- No tests completed? The deliverable isn’t available
- Test successes/failures indicate quality and whether
deliverables meet requirements
 We manage product risks and goals, not activities:
- We analyse the risks associated with this deliverable
- We plan tests to address uncertainty
- We monitor tests planned, passed and failed.
Testing provides the “intelligence”
Slide 13 Assurance with Intelligence

13.  Inputs to project plan for delivery
- The Contract
- The Requirements (business or technical level)
 The requirements may be good or bad
 But if the contract is poor or inflexible
- Things get forgotten, underestimated
- Testing, risk management, reporting all suffer
- Project Management attention on timescales and costs
(inputs) not on deliverables, quality, benefits (outputs)
- Acceptance process and criteria are loose, so plans are
woolly when precision is most important.
Poor contracts result in poor plans and
poor project management
Slide 14 Assurance with Intelligence

14. Introduction to Project Intelligence
Slide 15 Assurance with Intelligence

15. Slide 16
Testing throughout the life cycle: the
W model
Slide 16 Assurance with Intelligence
Acceptance
Test
Code
Build
Software
Install
Unit Test
Write
Logical
Design
Physical
Design
Test the
Test the
Design
Test the
Specs
Integration
Test
System
Test
Build
System
RequirementsRequirements

16.  Almost all project activities generate deliverables
 Project goals depend on deliverables
 Every deliverable has an associated test activity
- Static tests: reviews, inspections, static analysis etc.
- Dynamic tests: component, link, system, acceptance tests
etc.
 Testing provides the evidence of achievement
- Of intermediate goals throughout a project
- Of final project goal
 (This is standard PRINCE approach, but is
appropriate for all methodologies, by the way).
Testing through the lifecycle
Slide 17 Assurance with Intelligence

17. Slide 19
What is a test?
Deliverable Test
Deliverable
(unchanged)
intelligence
Testing doesn’t change
the product in any way.
We do it to gather
‘intelligence’.
Slide 19 Assurance with Intelligence

18.  If you don’t know where you are, a map won’t help
 If you don’t know the status of the deliverables, the
best plan won’t help you manage your project
 A project is like driving off-road in the dark
- Testing provides the headlights for the journey
 Testing provides data on the status of deliverables
and generates “PROJECT INTELLIGENCE”
 Stakeholders and management need intelligence to
make decisions, command and control the project.
Testing generates “project intelligence”
Slide 20 Assurance with Intelligence

19. Slide 21 Assurance with IntelligenceMilitary Intelligence?

20.  What are our project goals?
 Our goals depend on the deliverables
- Which are the key deliverables?
- What are the dependencies between them?
 How could our deliverables fail?
- Evidence required to be sure they won’t fail?
 How can I be sure our goals are met?
- Evidence required to demonstrate they are met?
Project Intelligence starts with goals
Slide 22 Assurance with Intelligence

21.  The value of project intelligence depends on how
well we achieve and communicate:
1. Coverage – planned and achieved
2. Product Risks – and the status of those risks
3. Business Goals – and the status of those goals
4. Test Status – passes, failures, incidents
 Time to deliver intelligence depends on the test
process
 Willingness of the organisation to accept and act
upon project intelligence is key.
Key sources of project intelligence
Slide 23 Assurance with Intelligence

22. PI Phase
 Results Chain Analysis
 PI Strategy
 Test Strategy Completion
 (Assessment and Testing)
 PI Reporting
Slide 25 Assurance with Intelligence

23. Slide 26
PI and the project lifecycle
Project
Initiation
Project Planning
PI Management
Acceptance
GoalAssessment
GoalAssessment
GoalAssessment
Key:
Stakeholder Involvement/Project Assurance/Governance
Project planning and initiation Project Intelligence Activities
Development activities Review and Test activities
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Project Intelligence Planning
Slide 26 Assurance with Intelligence

24. Results Chain Analysis

25. Slide 28
Results Chain Analysis
Project
Initiation
Project Planning
PI Management
Acceptance
GoalAssessment
GoalAssessment
GoalAssessment
Key:
Stakeholder Involvement/Project Assurance/Governance
Project planning and initiation Project Intelligence Activities
Development activities Review and Test activities
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Project Intelligence Planning
RCA defines the
relationship between
goals, activities and risks
Slide 28 Assurance with Intelligence

26. Slide 30
The Results Chain
ImpactOutcomesOutputsActivitiesInputs
Administrative By-products
Spending the budget, meeting
deadlines, filling timesheets,
compiling RAG reports, ticking
boxes, endless meetings
Waste Products?
Business Results
Achievement of business goals as a
consequence of the investment in the
programme/project.
Programme/Project Management
Slide 30 Assurance with Intelligence

27. Results Chain Analysis (RCA)
 Takes place as part of the overall project planning
process
 Intended to document the relationship between:
- Activities (aka initiatives)
- Deliverables (aka outcomes)
- Dependencies (aka contributions)
- Risks (aka assumptions)
- Intermediate and final project goals (outcomes again)
 Used to identify risks early and tie activities to goals
and risks.
Slide 31 Assurance with Intelligence

28. RCA Diagram elements
Outcome - an output (deliverable), outcome
(intermediate goal) or impact (final goal) of
the project
Initiative - an activity intended to create or
acquire a deliverable, achieve a desired
outcome etc.
Contribution - identifies a contribution of an
initiative, assumption or outcome to
another initiative, or outcome.
Assumption - identifies a dependency on
an external situation, not usually under
the control of the project
Slide 32 Assurance with Intelligence

29. Slide 33
Simple Results Chain example
Slide 33 Assurance with Intelligence

30. Using RCA
 RCA models the benefits realisation process
 Stakeholders and management can see the road map
leading to the business goal
 Derived from interviews and workshops
 Promotes discussion, consensus and commitment
 A shared understanding of the linkages between
business and IT initiatives
 Makes implicit thinking explicit, brings hidden
assumptions to the surface and enables better
communication and decision making.
Slide 34 Assurance with Intelligence

31. PI Strategy

32. Slide 36
PI Strategy
Project
Initiation
Project Planning
PI Management
Acceptance
GoalAssessment
GoalAssessment
GoalAssessment
Key:
Stakeholder Involvement/Project Assurance/Governance
Project planning and initiation Project Intelligence Activities
Development activities Review and Test activities
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Project Intelligence Planning
PI Strategy generates the
assessment process
Slide 36 Assurance with Intelligence

33. Goals and Risks as PI Drivers
 Strategy for goal-based progress reporting:
- Intermediate and final business goals
- Deliverables: documents, software, infrastructure, training,
resources etc.
- Coverage goals: functional/non-functional requirements,
scenarios, regulatory requirements etc.
- “To demonstrate that these goals have been met or
identify the problems that block the goal”
 Strategy for risk-based progress reporting
- Product risks: “to demonstrate that risks have
materialised, or have been addressed”.
Slide 38 Assurance with Intelligence

34. Slide 39
Project Intelligence Strategy
Risks
Requirements
Business goals
PI Drivers Ass. Obj.
Project Phase
Reqs Design Build Integ Systest UAT Trial Prod.
Objectives for each
test phase are easily
identified
Slide 39 Assurance with Intelligence

35. Test Strategy Completion

36. Slide 41
Test Strategy Completion
Project
Initiation
Project Planning
PI Management
Acceptance
GoalAssessment
GoalAssessment
GoalAssessment
Key:
Stakeholder Involvement/Project Assurance/Governance
Project planning and initiation Project Intelligence Activities
Development activities Review and Test activities
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Project Intelligence Planning
Test Strategy Completion
provides the test process
Slide 41 Assurance with Intelligence

37. Test Strategy Completion
 E.g. System test…
- Objectives (from PI strategy)
- Environment (technology, mainly)
- Responsibility (plan, spec, execution, approval)
- Deliverables (plan, spec, incidents, reports)
- Techniques (test design, intelligence reporting)
- Tools (plan, spec, execution, data prep, reporting etc.)
- Entry/exit criteria (goals met, risks addressed)
- Etc. etc.
Slide 42 Assurance with Intelligence

38. Test Strategy Completion 2
 Coverage policy
 Timescales/estimates (or the policy for
estimation)
 Regression testing policy
 Incident management process
 Reporting approach
 Usual stuff…etc. etc.
Slide 43 Assurance with Intelligence

39. Slide 44
PI Process – designed to handle change
Test Process Management
Project Risk
Management
PI Reporting
Test Phase
Coverage Goals
Change
Goals and Risks
3
Project or
Process
Risks
4
Evaluation &
Analysis
13
Impact Analysis
1
Requirements
Analysis/
coverage
2
Classification
5
Product Risks
Out of Scope
7
Goals Outstanding
Risks Outstanding
Coverage Outstanding
12
Goals Achieved
Risks Addressed
Coverage Achieved
6
Allocation to
Test Phase
9
Run
Test
10
Identify
Regression
Test
11
Run
Regression
Test
8
Define/
Design Tests
Risk is not product-related,
or is programme-critical
Risk will not be
tested (no test
objective)
Test
objective
defined
All new goals/risks
are “outstanding”
initially
Tests have
revealed a new
risk or changes
to a risk
Change affects
requirements
(and tests)
Change requires
regression tests
Failed tests must
be repeated
when fixes
received
New risk identified
Slide 44 Assurance with Intelligence

40. Goal, Risk and Requirements Based
Testing
Slide 45 Assurance with Intelligence

41. Tests
Requirements 1 2 3 4 5 6 7 8 9 10
Manage Customers X X X
Manage Products X X X
Create Requisition X X X X
Authorise Requisition
Raise Sales Order
Allocate Stock
Fulfil Order and Distribute
Raise Invoice
Match Payment
And so on…
Requirements Coverage
– Typical System Test
Large number of tests,
each focusing on a
limited scope
Slide 46 Assurance with Intelligence

42. Tests
Requirements 1 2 3 4 5 6 7 8 9 10
Manage Customers X X
Manage Products X X
Create Requisition X X
Authorise Requisition X X
Raise Sales Order X X X X
Allocate Stock X X X X
Fulfil Order and Distribute X X X X
Raise Invoice X X X
Match Payment X X
And so on…
Requirements Coverage
– Typical User/End to End Test
Smaller number
of tests, each
focusing on a
business
process path
Slide 47 Assurance with Intelligence

43.  Product risks are speculative failures and incidents
 The risk impact scale is the SAME as incident severity
 Some modes of failure are unacceptable so we select tests
to demonstrate selected failure modes are unlikely
- NB: these tests COULD be in any project phase
- Means we focus on the tests that MUST pass
- We know defects that cause these failures MUST be corrected
 If we can’t correct failures of risk-based tests we can say
these risks have materialised
 Critical risks block acceptance
- Force a change of plan: re-schedule, redesign, re/de-scoping or
abandonment!
Product risks as potential modes of
failure
Slide 48 Assurance with Intelligence

44. Slide 49
Risk-Based Testing complements
requirements-based
High
Medium
Low
Product Risks Test Obj.
Project Phase
Reqs Design Build Integ Systest UAT Trial Prod.
Objectives for each
test phase are easily
identified
Slide 49 Assurance with Intelligence

45. Slide 50
Risk-based reporting
Progress through the test plan
today
Planned
end
residual risks
of releasing
TODAY
start
all risks
‘open’ at
the start
Slide 50 Assurance with Intelligence

46.  All projects have underpinning goals/benefits
 Stakeholders want to know:
- The status of these goals
- The status of the risks that block those goals
- The confidence with which they may make a
decision (test coverage)
 These are the three areas where test evidence
is required: Project Intelligence.
Goal-Based Testing completes the
picture – Project Intelligence
Slide 51 Assurance with Intelligence

47. Slide 52
Project Intelligence and Coverage
Goal
Coverage
The tests in scope cover
requirements, risks and
goals.
Any test may be designed to
address the needs of one,
two or all three drivers.
Corollary: a test that doesn’t
address any driver is a
pointless test.
Slide 52 Assurance with Intelligence

48. Slide 53
Project Intelligence Strategy
Risks
Requirements
Business goals
PI Drivers Ass. Obj.
Project Phase
Reqs Design Build Integ Systest UAT Trial Prod.
Objectives for each
test phase are easily
identified
Slide 53 Assurance with Intelligence

49. Slide 54
PI reporting
Risks
Requirements
Goals
PI Drivers Ass. Obj.
Project Phase
Reqs Design Build Integ Systest UAT Trial Prod.
Progress of achievement is
clearly seen. Outstanding
objectives are highly visible.
Slide 54 Assurance with Intelligence

50. Goal-based test reporting
Open
Closed
Risks
Open
Open
Closed
Closed
Open
Objective
Objective
Benefit
Objective
Goal
Goal
Benefit
Goal
Goal
Goal achieved,
Benefits available
Objective
BenefitClosed
Slide 55 Assurance with Intelligence

51. Close
Slide 56 Assurance with Intelligence

52.  Historically, coverage (requirements, code etc.) was the
only game in town
 Risk-based introduced a second dimension
 Goal-based is the third
 Testing is transformed from just defect-detection to
intelligence gathering and decision-support
 Project Intelligence
- Extends from business case through to post deployment
assessment
- Involves document, code, system, user, service testing
 Not necessarily more tests, but more focused tests,
much better control and stakeholder reporting.
PI Summary
Slide 57 Assurance with Intelligence

53.  Best PM practice: manage goals, not activities
 PI gives management visibility of risks, goals and progress
towards them
 Project managers and stakeholders are PI customers
 The test process is the nervous system of your project,
designed to provide useful PI in a timely manner
 What’s new?
- Goal and risk-based testing
- New styles of test reporting throughout the lifecycle
- Appreciation of testing as a key source of Project Intelligence
 PI is usable in ALL methodologies
 A mainstream TM tool - QualityCenter - can now support PI.
Finally
Slide 58 Assurance with Intelligence

54. gerrardconsulting.com
ukalf.com
Thank-You!
Managing Software Projects With
Intelligence
Slide 59 Assurance with Intelligence