Enterprise Software
Performance Engineering
Building systems that scream so your users don’t:
A key component of risk management
Presented By: Walter Kuketz
Hosted By: Frank Days
© 2012 Collaborative Consulting

Housekeeping
Presentation will last 45 Submit questions via the Slides will be made
minutes chat window available
2
2012 Collaborative Consulting

Purpose of today’s webinar
Review the current challenges of being a performance engineer
Overview of the five knowledge areas of Software performance engineering body of knowldge
Where to inject PE tasks and activities into the SDLC to build more performant applications
Overview of application performance management (end-to-end user experience)
Roles and career path overview
Case study – 10X event
3
2012 Collaborative Consulting

Performance engineering challenges today
Highly • Complex and tangled system of systems
distributed • Distributed Infrastructure
apps • Distributed production operations
• Performance engineering and testing
Distributed
• Architecture and development
teams
• Business
Evolving & • Mobile devices
changing • Third party services
application • SaaS products
architectures • Cloud
Changing • Driven by mobile anywhere
usage
• Flash traffic events
patterns
4
2012 Collaborative Consulting

Performance engineering challenges
User
Enterprise Budget Team skills
experience
• Release • User populations • Constant cost • Performance
schedules, mont are exploding and budget organization;
hly, quarterly pressure skillsets, multi
• Web site disciplined,
• Project based performance • Maintaining a communication,
directly impacts meaningful split team
revenue performance lab
(production-like) • Training and
career paths
5
2012 Collaborative Consulting

No clear standards in the market for PE
Test automation technician
Capacity planner
?
Performance tester
?
? Performance Engineer
Performance analyst
?
Performance Architect
?
Benchmarks
?
Troubleshooting
Performance Data engineer
?
6
2012 Collaborative Consulting

Goal of the SPE Body of Knowledge
Define the profession of performance
Define profession engineering
Establish Body of Establish a body of knowledge for
Knowldge performance engineering
Establish Establish a common language for roles,
common responsibilities, and practice areas using
language the form of Knowledge Areas
Define Define the competencies shared across
competencies the knowledge areas
Capture common Capture common techniques for sharing
techniques across the industry
7
2012 Collaborative Consulting

The SPE Body of Knowledge
Enterprise Software Performance Engineering
Planning, coordination, information sharing, and control
Knowledge areas
Performance
validation and testing Problem
Application
SDLC & (PVT) detection
performance
architecture and
management
(SA) Capacity planning (APM)
resolution
(PDR)
(CP)
Underlying competencies and roles for delivering
Techniques supporting the knowledge areas
8
2012 Collaborative Consulting

The Enterprise PE team: There must be a practice leader
A shared service and clearing house for all performance, scalability and
capacity planning information. Transforms performance information to
business knowledge and advantage.
Communicates across the teams to share information
For instance, the production workload and the
Operations PT performance test workload – are they in sync?
Architecture and Make sure the architecture team knows how the
production new release, or newest build behave in production.
The business risks are clearly identified and
Business direction communicated.
Can provide performance guidelines to the
Development teams developers, must influence the unit testing process.
9
2012 Collaborative Consulting

Information sharing across the knowledge areas
Enterprise PE
Capacity
planning and planning Triage
monitoring
SPE impacts Architects,
Results,
Developers, testing, and
Forecast Workload operations
Workload scalability
NFR’s Performance User Exp
Volumetrics, validation, Workload
Utilization
workload testing
SDLC, Results
Architecture Observations APM
, NFR
Growth plans
Workload
NFR’s
10
2012 Collaborative Consulting

PE: Knowledge areas
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
11
2012 Collaborative Consulting

PE: Software development lifecycle and architecture
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
12
2012 Collaborative Consulting

PE: SA (software development lifecycle)
Waterfall methodologies
Functional Performance
Define Design Develop Deploy
test test
Need to plan for Build code to performance
performance testing requirements
Define
Deploy Agile Design
methods
Test Build
Can a system be designed to support 10 TPS and 1 second response time work @ 100 TPS and 1 second?
13
2012 Collaborative Consulting

SDLC: Define and design
Define Design
1. Project risk scorecard 1. Technology Architecture validated
- Key business transactions to the performance Usage
identified Scenarios. How will it achieve the
- Workflow constraints key performance requirements?
Sub-second
- SLA’s and penalties response time. Define the scalability approach.
Flash marketing
2. Workload model and usage event. 2. Scalability to support flash events
scenarios
3. Environments: include any
3. Business volumes with peak
required changes to develop and
factors
test the performance usage
4. Business performance scenarios.
requirements
Communication to your downstream processes and people
Prepare to deal with flash events
Design to avoid flash events
(they are expensive. E.g. Work with business why does the outbound email
have to send email to all xM customers at the same time
14
2012 Collaborative Consulting

SDLC: Develop and functional testing
Functions are tested. Performance needs to be analyzed.
Develop Functional test
• The functions are mapped to software • Verify business functions and key
components. The performance req’s are performance functions.
assigned to the components.
• Verify the key performance workflows
• Unit testing for early validation of the and functions.
performance req’s for the software
components. Response time • Define and execute micro-performance
measurements. testing in the QA process.
• Integration testing include micro- • Produce report on for each key
performance testing of key functions. performance Usage Scenario with
Code review of the key performance response time and throughput.
functions.
• Enhance unit and integration test for
early validation of key performance
functions
Greatest impact on performance and scalability, but you must know the goals.
Communication to your downstream processes and people
15
2012 Collaborative Consulting

Performance budgets
Do your developers know how much time they have for their piece of the transaction?
Code runs
everywhere.
SERVER
WEB APP SVR
PROXY
MQ/ESB
LDAP CORBA
DCOM
Respond to the user
in 2 seconds. Web
Services
Datacenter
16
2012 Collaborative Consulting

PE: Performance testing and validation
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
17
2012 Collaborative Consulting

PE: PTV (performance testing and validation)
What is changing? Where does it need to be validated? What (all) is being measured?
The project: Why are we testing and what are the success criteria?
Prerequisites:
Business goals and non-functional requirements are defined
Workload and scenarios defined
Environment defined (where are we testing)
Application in a stable state
Configuration management and release management ready
Core performance team identified
Supporting team committed
18
2012 Collaborative Consulting

Organization: Performance team value
You still need to explain to the rest of the organization what you do
Business and Early involvement
application area with the applications Test execution Environment
knowledge team
• Deep understanding of • Performance engineers • Well-understood • Testing environment is
the application being involved in design workload model and a close approximation
tested sessions characterization of production, including
database size
• Deep understanding of • Understand the nature • All components in the
the technical of the data required for test are monitored and • Differences between
architecture of the testing reported on test and production are
application clearly known
• Able to understand • A defined triage process
• Understanding of the technical gaps to meet for troubleshooting • Release and
business area requirements not configuration
• A process to manage
supported by the supported by the tool management in place
the inevitable
applications unplanned tests and and procedures
enforced
test archive exists
Performance team value
19
2012 Collaborative Consulting

Performance team reference model
Primary test, design, and execution team Performance engineering support skills
Planning and
Application
project
Performance architect
management
engineer
Technology
SME
Test
Results
automation
analyst Data
technician
engineering
Must determine the core team members vs. on-demand team
20
2012 Collaborative Consulting

Enterprise performance team
PT problem Strategy and
PT technology
resolution planning
PMO
board (architecture)**
Infrastructure**
Performance
Performance test engineering and
execution team optimization
Application
Vendor Application
support
management deployment
Vendor IBM Oracle EMC Informatica Vendor Vendor
Vendors
21
2012 Collaborative Consulting

Model test case execution
In the end, the test has to be run.
Prepare test
Prepare test Confirm
Select test harness online Initiate test
0 case or batch
monitoring
components
environment
readiness
case 1
messages
No
Problem
analysis and Evaluate
triage team impact of
Select test Normal test
1 case completion?
change
Yes
Yes Gather test
Analysis for Create test
results
specific report; publish
Txn RT, CPU,
DB
change? to repository 2
No
Are Categorize Can you
Implement
changes change – enter retest with Rerun the test
2 required for
Yes
change to CM
configuration
change
the
Yes
case
retest? tracking tool change?
No No
Move to next Move to next
test case 0 test scenario
22
2012 Collaborative Consulting

PE: Capacity planning
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
23
2012 Collaborative Consulting

PE: CP (capacity planning)
The right system at the right price!
Plan for • Forecast workloads
future • Plan for usage
Analyze
• Track utilization of
existing systems current
capacity
• Track workload
Determine • Define system to
capacity support workloads
requirements • Test
• Define service
levels
Workloads drive system transactions, which drive resource utilization
24
2012 Collaborative Consulting

Capacity: Use of resources
The overall system has a capacity; each component has a capacity.
SERVER
WEB APP SVR
PROXY
MQ/ESB
LDAP CORBA
DCOM
Web
Services
Datacenter
25
2012 Collaborative Consulting

Workload model validation
Actual workload Synthetic workload
Model calibration
System System
Measured response times, Measured response times,
throughputs, utilizations, etc. throughputs, utilizations, etc.
Acceptable No
Yes
26
2012 Collaborative Consulting

PE: Application performance management
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
27
2012 Collaborative Consulting

Five elements of APM (Gartner)
Monitoring the performance of complex distributed applications
End user experience measurement – browser/mobile
1 device
2 Create a model of the run-time environment (discovery)
Profile the performance and behavior of user-defined
3 transactions
Performance metrics from each of the
4 applications/systems technical components (Webserver,
App server, Database, etc.)
5 Application performance management database
28
2012 Collaborative Consulting

End user experience monitoring
29
2012 Collaborative Consulting

Track key user experience metrics
Mobile Total end-user response time
Browser rendering time
Network latency
Mobile Real User Monitoring
Rendering
30
2012 Collaborative Consulting

And see every hop of the transaction
Mobile
Proxy
server
Web server
App server
Data warehouse gateway
Mainframe
Database
Dynamic Transaction
Path Detection
Rendering
31
2012 Collaborative Consulting

PE: Problem detection and resolution
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
32
2012 Collaborative Consulting

Performance issues and root-cause analysis
This leverages several knowledge areas
A solution must be found quickly (under pressure)
Performance issues are found in:
• Preproduction environment
• Production and operations environments
Key business transactions are not meeting their service level objectives:
• End user experience is degrading (acute or chronic)
• Overnight processing is taking longer
• Stability and performance issues
33
2012 Collaborative Consulting

When problems occur
Flash web site events (all at once)
New code released causes problems
Infrastructure upgrade
Consolidation of applications and servers
New workload model (batch and online)
34
2012 Collaborative Consulting

Underlying competencies
Analytical
Project Communication
thinking
management skills
problem solver
Business Monitoring and
Statistics
Knowledge tuning
Workload
Behavioral Application and
model
characteristics tool knowledge
Underlying development
competencies
Test data Test execution
Forecasting
management process
35
2012 Collaborative Consulting

Technical competencies
Code Deep dive Resource
Database profiling tools
utilization
monitoring tools (HP Diagnostics,
tools
CA Wily, etc, system
(Static analysis, Jprobe, etc)
diagnostics) level
End to end Comprehensive
Statistics and
performance results collection
Distributions
tools and reporting
Technical
competencies Test design Queuing
and and Test data
Little’s law
execution modeling creation
tools tools
36
2012 Collaborative Consulting

PE Techniques
Brainstorming Data modeling
Creating a workload model Performance testing
Triage approach Static code analysis
Logging and
System metrics collection
instrumentation
37
2012 Collaborative Consulting

PE roles: The team – many possible roles
Managing and developing the core team members
1 Technical Project manager
2 Performance Architect
3 Sr. Performance Engineer
4 Performance Engineer
5 Sr. Test automation technician
6 Test automation technician
7 Performance data engineer
8 Sr. Capacity planner
9 Capacity planner
10 Modeler
38
2012 Collaborative Consulting

Career path: Performance engineer foundation
Performance
Engineer
Grow your Performance
testing
leadership and Lead
communication technical Workload
role models
Technical Production
architecture triage
Core Additional
technical tech
competency Communication
competency
Grow your breadth of capabilities
39
2012 Collaborative Consulting

Case study: Preparing for a flash marketing event
Planning a large event to introduce a new product
• Send out email to all registered customers (500,000 currently), there is a link in the email.
• Run multiple TV ad’s encouraging people to register and then come to the event, results in
100,000 new registrations
• No new functionality required to support the event
• Potential 600,000 users of the web site
The business is forecasting at least a 10X increase during the
event, in terms of product sales
• How does this translate into system workload and utilization?
• Do we need more servers?
• The current system was not really designed for such a flash event
40
2012 Collaborative Consulting

PE: Process to prepare for the event (10X)
CP CP PVT SA
Forecast the
workload Tune application
based on results
APM PDR
Measure the current Plan and execute
workload and PT for forecast
resource utilization workload
41
2012 Collaborative Consulting

CP: Workload model
CP
The CP team uses this information to forecast the
new workload 10X.
Step Transaction – by product Percentage
1 Login 100
2 Search product 100
3 View product detail 100
4 Add to shopping cart 50
5 Confirm purchase method 50
6 Shipping information 50
7 Send confirmation email 50
42
2012 Collaborative Consulting

CP: Analyze current capacity
CP Review current Production workload, and the under lying system
utilization.
• Average workload
• Current peak workload
Response
time
Service level
100%
System utilization
43
2012 Collaborative Consulting

PVT: planning, executing, and analysis
PVT
• Use the workload model from CP
• Enable APM during testing
• Work with the Application Architect
Methodology is iterative and tactical
Define business
Review infrastructure
activity profiles & Design & build tests
& architecture
service levels
• Identify risk areas • Types & numbers of • Test data generation
• Review configuration users • Create test scripts
settings, topology, & sizing • Business activities & • User & transaction profiles
• Define points of frequencies • Infrastructure configuration
measurement
Iterate testing & tuning
44
2012 Collaborative Consulting

Questions?
Thank you!
Get your free copy
SharePath RUM today!
More information:
www.collaborative.com
info@correlsense.com
www.correlsense.com/demo
www.real-user-monitoring.com
45
2012 Collaborative Consulting