Mais conteúdo relacionado Semelhante a Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat Egozi Levi (20) Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat Egozi Levi1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Leveraging cloud data to
optimize your Agile
processes
Ronen Aseo, Product Manager
Efrat Egozi-Levi, Senior Data Scientist, CTO office
2. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
3. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
HP Agile Manager defined
Agile project management
Release & User Story mgmt.
Defect management
ALM/PPM synchronization
ALI integrations
Key Capabilities
A SaaS solution for Agile
Instant-on service
Regular, incremental
releases driven by
user feedback
Web-based, intuitive UI
Complements HP ALM/QC
Key
Characteristics
4. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Ø 3x number of users within the last year
Ø Monthly drops to production
Ø Vast internal adoption throughout HP
Our Story
http://www.tonycooke.org/
5. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
HP Agile Manager Development Teams
Agile Manager Product
characteristics
• 3 applications
• Teams are co-located
• On premise & SaaS product
Products
ALM Synchronizer
Application
Lifecycle
Intelligence
Agile Manager
Sunnyvale,
CAProduct
Management
Prague,
CZDevelopment
Yehud,
ILDevelopment
Ho-chi minh, Vi
Development
6. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
AgM Delivery Model
SaaS Production Farms (External)
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6
SaaS Production Farm (Internal – early adapters)
Test Farm (trunk)
Nightly
deployment
trunk
Deploy after two weeks
on internal farm, including
required bug fixes
RELEASE/PSI
Internal branch
Bug fixes
Test Farm – internal branch
Bug fixes
Deploy after two weeks
on internal farm, including
required bug fixes
Drop 1 Drop 2
Drop 1 Drop 2
Sprint length: 2 weeks
7. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Source: http://www.redeemthecommute.com/
8. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Challenges
Roadmap
Features
Coverag
e
Resource
s
Customer
Requests
9. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
BIG DATA -
TO THE RESCUE
Source: http://coffeebreakwithgamechangers.tumblr.com/
10. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Data Science Office @ HP Software
Add data-
science and
analytic
capabilities in
HP products
machine
learning
graph
analysi
s
static /
dynamic
code
analysis
other
technique
s
11. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
• We want to make informed product/dev decision based on real
users behavior
• We have real user monitoring today e.g. Google Analytics
• Information is event based not flow based
• Flow based is hard to understand
• There is code coverage but this doesn’t reflect real user usage
• What we need is real user coverage and insights
Problem scope
Event-based analysis in Google analytics
12. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Problem scope
External Flows
Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n
Test Flows
Feature:
Use Chat
Login
Open Chat
Write Msg
Send
Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n
13. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Real user test coverage algorithm
• Difference between two flows based on their similarity score
definition
• The similarity score combines 2 criteria (Law & Order):
– the identical steps, which exist in both flows
– the similarity in the step order in each flow
• Match flows based on the difference function with an adaptive
sensitivity threshold
• Compute the weighted similarity for a set of matched production
flows per test
Production Flows Internal / Test Flows
1. Login > 2. Open Chat > 3. Write Msg > 4. Send
1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5.
Send
1. Login > 2. Open Chat > 3. Send
1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5. Write Msg > 6.
Write Msg > 7. Attach > 8. Write Msg > 9. Send
1. Login > 2. Open Chat > 3. Write Msg > 4. Send > 5.
Attach
Usage
: 50%
Coverag
e: 78%
Feature X
84
78
70
75
14. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Key Findings
Ø The automated tests cover
35% of production flows
Ø 50% of production events are
not covered by any automatic
test
Ø Interesting insights from
comparing internal vs.
external user flows
From the motion picture “Moneyball”,2011
15. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Significant increase in real user test coverage
after gathering insights
0
10
20
30
40
50
60
70
80
100 150 200 250 300 350 400 450
Drop 1
stabilization
Drop 2
New
Content
#tests
Realusercoverageintests
16. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Example: Dev Summary
External Flows Internal Flows
Release mngmnt > Taskboard > Zoom > ali summary > all > builds
Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective
Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard> project
configuration teams > Taskboard > Zoom > ali summary > all > shared items
• Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective > Taskboard >
Zoom > Kanban > Kanban: Configure Team Storyboard> project configuration teams > tab:
Storyboard
• Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard>
project configuration teams > Taskboard > Zoom > ali summary > all > shared items
• Release mngmnt > Taskboard > Zoom > ali summary > all > builds
• Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective
• Compared: external
customers to internal
customers
• Surprise: page is used
similarly, regardless of
dev configuration
• => Cancel “prompt to
config” plans
Feature:
Dev
Summary
Same usage
in external
and Internal
17. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
Example: Planning board
• Planning board
event is rarely
used internally –
Should we
invest?
• Surprise:
Planning board
is used as the
1st step in many
flows
Planning board Kanban
Release
backlog
[?]
Sprint Backlog …
18. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
• Surprise: “Sprint Closure” flows have high usage (>20%) but low coverage
(3%)
• Sprint closure should have high priority and additional test automation
Example: Sprint closure
19. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19
Benefits - Recap
Data driven decision making
Better understanding & evaluation of customer needs
Improved real user test coverage
Optimize resource allocations
20. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank You
Contact:
ronen.aseo@hp.com
eegozi@hp.com
21. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
Release Planning
• Feature Ranking
• Feature Scoping - MMF
• Initial Feature Sizing
• Define Feature Lead
• Define Release Scope
22. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22
Definition Of Done - DoD
ü Unit tests are written and green
ü Acceptance tests are defined by QA and passed on top of a build
ü Main functionality of the user story is covered by automated acceptance tests
ü Sanity and Regression tests are passed green after the user story is checked in
0 C/H open defects
ü Majority of Medium defects are fixed
ü All fixed defects are validated
ü The user story is pushed to production, monitored and green
23. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
Delivery Model to Production
Release Criteria
Progress Status Target Current Status
Test Coverage On Track 100% • 100%
Quality On Track No Critical defects
No C/H New features defects
No C/H customer encountered defects
No Functional Regression defects
<10 High defects
70% M defects fixed
• 0 Critical defects
• 0 C/H New features defects
• 1 C/H customer encountered defects
• 3 Functional regression
• 10 open High defects
• 61% of the M defects were fixed
Performance On Track No C/H issues No C/H issues
Security On Track No C/H issues No C/H issues
24. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24
Product Owner Challenges
• How to prioritize features and enhancement requests?
• Can I cut a story from a feature and it will still have value for the user?
(MMF)
• How will a new feature impact current usage?
• Does my acceptance criteria cover the end user’s flow?
• Are features being used the way they are “supposed” to?
• Is there a difference between how features are used internally vs.
externally?
25. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25
Team Challenges
• How much to invest in automation?
• What areas to automate and how to automate?
• What to include in our Regression testing?