This presentation talks about the common pitfalls from a systems performance perspective and what development teams could do to engineer performance proactively into the system design, build and test process. It also throws light on the practices of SPE (Systems Performance Engineering) across the Development Life Cycle.

1. Challenges Engineering Applications
for High Performance
Practical Performance Analyst
http://www1.practicalperformanceanalyst.com

2. • Started mucking around with Open Source 15 years ago
• Fell in love with Gnu/Linux and more importantly the Free Software / Open Source movement
• Travelled across the country advocating the use of Linux and Free Software
• Went to work for Red Hat. Sold & deployed Free Software
• Worked at the Performance R&D Labs at TCS and then moved to Consulting
• Moved to Accenture Technology Consulting and now part of Tech Arch, Performance Engineering
• Founded Practical Performance Analyst (www1.practicalperformanceanalyst.com)
• Revived CMGA (Computer Management Group, Australia)
A few words about me

3. How Do You Define Performance
Throughput &
Congestion across
the last mile
Poor End User
Experience
Client Device
Performance
Congestion on
the Internet
Infrastructure
Performance
Application
Performance
Backbone Network
Throughput &
Performance
Data Center
Network
Performance

4. • All of us have cribbed about poor performing solutions at one point or the other.
• Unfortunately, poor performance seems to be the norm and we’ve all accepted that it’s just how things are.
Lets look at some examples:
• Web Store: The web is abound with case studies about online retailers loosing revenue as users turn to
competitors offerings. Users today aren’t very tolerant of poor performing websites and with easy access to
google finding an alternate offering is no more than a few clicks away. There’s tons of cheap tools available to
monitor system these days but engineering systems to performance is a completely different kettle of fish.
• Ticketing / Help Desk Application: You tried reaching out to your internal help desk for support and it
seemed like eternity before you got through. But even after you got through to the help desk, the wait
continued because the help desk technician just couldn’t get to the data they needed quickly to help you out.
The tech team looking into the issue needed insight into the performance bottleneck but the tools they had
at their disposal only gave them a siloed view of performance which wasn’t good enough to get to the root
cause quickly.
• Batch Job Performance : You’ve had to go back and confess to business that the batch jobs your teams had
designed weren’t performing the way you intended them to perform. Un-fortunately, the developers of the
batch job hadn’t taken into account the volume of data that the system would be need to tested for. Your
development environments happen to be a fraction of your production environments and so was the data
store for performance testing.
How Does Poor Performance Manifest Itself

5. What is SPE
• Definition – Systems Performance Engineering (SPE) is a systematic and quantitative
approach for the cost-effective development of software systems to meet stringent
Non Functional Requirements i.e. Performance, Capacity, Scalability, Availability,
Reliability, etc.
• Systems Performance Engineering is focused on optimal selection of application
architecture, design, build and implementation choices with the objective of
meeting Non Functional Requirements.
• Systems Performance Engineering can also be defined functionally as the set of
tasks or activities that need to be performed across the Software Development Life
Cycle (SDLC) to meet the documented Non Functional Requirements.
• Software Performance Engineering is often viewed as the art of building systems
that meeting Non Functional requirements within the allocated time frame and
budget constraints.

6. Where Do Programs Generally Go Wrong
• Lack of Non Functional Requirements - 99% of programs I’ve worked on have no or very poorly define Non Functional Requirements.
Ask 5 different people on the team their interpretation of on Non Functional and you will end up getting 5 different answers. As a
result –
• Architects have no guidance on what are the right design choices
• Developers don’t have unit performance or tier performance targets
• Infrastructure designers have no idea of what specs they need to be building for
• Inability to regression test nightly builds against unit performance or tier performance targets
• Performance testers have no idea of what and how the various application workloads should be evaluated
• Lack of investment in right capability – Performance isn’t just about response times or server utilization. Engineering performance
goes far beyond all of that and it happens to be a way of thinking. Programs rarely invest in the right capability. The fact is most
program leads lack an understanding of what is required to build performance into the solution. Functional capability is easy to nail
down, performance is rather complicated since it traverses so many domains across the delivery chain.
• Lack of investment in appropriate tools – Tooling is a very complicated subject and performance tools happen to be expensive. Most
projects think that Nagios (performance monitoring) and Jmeter (performance testing) is good enough to get them over the line.
• Lack of investment in the required infrastructure – To validate performance you need appropriately sized infrastructure. Testing on
infrastructure that is <50% of production and drawing conclusions based on assumptions of linear scalability is quite frequent and a
perfect recipe for disaster.
• Lack of funding to design, build, validate and manage performance across the delivery cycle – Performance starts at the
requirements gathering stage and continues onto architecture, design, build, test and then capacity planning. Each of these functions
requires the right capability across the board. Projects seldom invest in the right capability across the board.

7. SPE Across the SDLC
Software Development Life Cycle
Functional Requirements Gathering
Architecture & Design
Build Application
System Test,
System Integrated Test & UAT
Deploy Into Production
Performance Engineering Life Cycle
Non Functional Requirements
Gathering
Design for Performance &
Performance Modelling
Unit Performance Test &
Code Optimization
Performance Test
Monitoring & Capacity Management

8. Performance By Design
• Requirements Gathering –
• Document Non Functional Requirements
• Work with Business & IT to get NFR’s signed off
• Decompose NFR targets across various application tiers
• Identify & budget for relevant tools for monitoring, diagnostics,
performance testing and capacity planning
• Identify & budget relevant capability required to integrate
performance across the SDLC i.e. Performance at Design, Performance
at Build, Performance Testing, Diagnostics and Tuning, Monitoring &
Capacity Management
• On programs that can afford it, have an owner for Performance across
the program
• Design –
• Work with Architecture teams on adopting the relevant design
patterns
• PoC design / implementation options and identify potential
constraints
• Document Overall Performance Engineering strategy
• Document Overall Performance Testing Strategy
• Document Overall Capacity Management & Monitoring Strategy
Performance
Requirements
Analysis
Performance
Modelling &
Capacity
Planning
Build &
Optimization
Performance
Testing
Performance
Monitoring
Capacity
Management

9. Performance By Design
• Build –
• Translate the NFR’s to tier wise performance targets
• Work with development teams and establish process to track regression
of code performance
• Enforce unit performance testing of code and work with developers to
track tier wise performance targets
• Work with development teams and assist with usage of profiling &
diagnostics tools. Use diagnostics tools to identify potential bottlenecks
and optimize code
• Test –
• Build stubs and begin early performance testing
• Document workload models and get them signed off
• Execute end to end performance testing
• Use Diagnostics tools to identify bottlenecks
• Analyst results, tune, optimize and re-run
• Go Live -
• Implement system, application, transaction and end user monitoring
• Collect relevant business/transaction performance metrics
• Forecast & model performance impacts using performance metrics
Performance
Requirements
Analysis
Performance
Modelling &
Capacity
Planning
Build &
Optimization
Performance
Testing
Performance
Monitoring
Capacity
Management

10. SPE Roles Across The SDLC
Phase Role Performed
Requirements Gathering Determine Business Volumes & Growth plans
Determine Non Functional Requirements
Design Validate Application Architecture
Validate Infrastructure Architecture
Validate Infrastructure Design
Determine Infrastructure Capacity Requirements
Build & Optimization Set performance targets for developers
Validate outcome of Unit Performance Tests
Recommend optimization to Code
Recommend optimization to Application Design
Recommend optimization to Infrastructure Design
SIT & UAT Begin validating Application Performance for functionality that’s available
Begin Tier Performance Tests for functionality that’s available
Test, tune & optimize Performance for functionality that has been released
SVT Execute End to End Performance Tests
Validate Performance End to End
Identify applicable bottlenecks
Use Diagnostics tools to identify bottlenecks, tune & Optimize application
Ensure application meets its Non Functional Requirements
Evolve Performance Monitoring Requirements
Pre-Go live Setup Performance Monitors for Network, OS, Application, Business Txn
Set alerting for various Performance metrics
Setup capture Performance metrics for purposes of Capacity Management
Go-Live & Post Go Live Monitor application for potential breaches in SLA’s
Identify hotspots using low overhead transactional tracing & diagnostics tools
Model application performance and predict capacity impacts for growing business workload

11. Devops, Continous Integration, Agile, etc.
• Devops, Agile, Continous Integration are paradigms that are have begun to influence development practices around the
world. Each of these paradigms have their own strengths and weaknesses.
• Whatever you development paradigms you may adopt for development of your systems you will need to integrate
Proactive Performance Management or SPE principles into it
• Un-fortunately most of the bad habits (mentioned in earlier slides) with regards to poor SPE practices are carried into
the new paradigms as well. It’s quite rare that we find programs adopting proactive SPE practices into their Agile
development processes.
• Agile, Devops, CI are paradigms have a lot of good to offer but don’t forget to integrate Proactive Performance
Management practices and more importantly SPE practices into your implementation of those processes.

12. What Am I Doing To Change The World
• Practical Performance Analyst -
• What is PPA - Practical Performance is a community portal that I started work on in 2012. Practical Performance
Analyst has grown significantly over the last couple of years with a growing readership base. Our aim is for Practical
Performance Analyst to be a open platform for sharing and collaboration on standards, practices and guidelines
around Systems Performance Engineering (SPE). Practical Performance Analyst has a growing list of international
authors who are experts in their own domains of Performance. Practical Performance Analyst aims to be a hub for all
things SPE (Systems Performance Engineering). Drop in and find out.
• Accessing PPA – http://www1.practicalperformanceanalyst.com
• Contributing & Supporting PPA – trevor@practicalperformanceanalyst.com
• Computer Management Group, Australia –
• What is CMGA - Computer Management Group Australia is a professional body that focuses on bringing together
professionals from across Australia who have interests in Systems Performance Engineering (SPE). CMG is an
international organization of professionals (CMG.org) and CMGA is the Australian Chapter of CMG International.
Spent the last 6 months reviving CMGA and getting CMGA going again here in Australia.
• Accessing CMGA – http://www.cmga.org.au
• Joining CMGA – Joining CMGA is quite easy. Hop over to the CMG website and join the group using the Meetup link
on the right hand side corner. We use the meetup group to keep in touch and co-ordinate all national events.

13. What Can You Do To Help
• Learning & Sharing – If you are interested in learning more about Proactive
Performance Management & Systems Performance Engineering head off to
Practical Performance Analyst (http://www.practicalperformanceanalyst.com).
• There’s tons of resources out there on all topics related to SPE.
• You’ll also find access to books and tools on all aspects of SPE
• Grab your daily dose of SPE news from around the world at PPA
• Contributing & Helping Build the Community – If you are interested in helping
building the SPE Community and contribute your knowledge from a Design,
Architecture, Build, Optimization, APM, Performance Testing, Capacity
Management, Modelling, etc. perspective please drop me an email at
trevor@practicalperformanceanalyst.com. We are keen to expand out list of
authors and grow our community of readers.
• Networking & Lobbying – Come and join us at Computer Management Group
Australia. CMG Australia is a professional organization with a focus on evolving
standards, fostering innovation and most importantly advocating adoption of SPE
across programs. CMG Australia has various initiatives focused on SPE across
industry, academia and research. Come join us and help build appreciation for good
development practices.

14. Thank You
Practical Performance Analyst
http://www1.practicalperformanceanalyst.com
trevor@practicalperformanceanalyst.com