Mais conteúdo relacionado

Apresentações para você(20)

Mais de Correlsense(18)


An Introduction to Software Performance Engineering

  1. Enterprise Software Performance Engineering Building systems that scream so your users don’t: A key component of risk management Presented By: Walter Kuketz Hosted By: Frank Days © 2012 Collaborative Consulting
  2. Housekeeping Presentation will last 45 Submit questions via the Slides will be made minutes chat window available 2 2012 Collaborative Consulting
  3. Purpose of today’s webinar Review the current challenges of being a performance engineer Overview of the five knowledge areas of Software performance engineering body of knowldge Where to inject PE tasks and activities into the SDLC to build more performant applications Overview of application performance management (end-to-end user experience) Roles and career path overview Case study – 10X event 3 2012 Collaborative Consulting
  4. Performance engineering challenges today Highly • Complex and tangled system of systems distributed • Distributed Infrastructure apps • Distributed production operations • Performance engineering and testing Distributed • Architecture and development teams • Business Evolving & • Mobile devices changing • Third party services application • SaaS products architectures • Cloud Changing • Driven by mobile anywhere usage • Flash traffic events patterns 4 2012 Collaborative Consulting
  5. Performance engineering challenges User Enterprise Budget Team skills experience • Release • User populations • Constant cost • Performance schedules, mont are exploding and budget organization; hly, quarterly pressure skillsets, multi • Web site disciplined, • Project based performance • Maintaining a communication, directly impacts meaningful split team revenue performance lab (production-like) • Training and career paths 5 2012 Collaborative Consulting
  6. No clear standards in the market for PE Test automation technician Capacity planner ? Performance tester ? ? Performance Engineer Performance analyst ? Performance Architect ? Benchmarks ? Troubleshooting Performance Data engineer ? 6 2012 Collaborative Consulting
  7. Goal of the SPE Body of Knowledge Define the profession of performance Define profession engineering Establish Body of Establish a body of knowledge for Knowldge performance engineering Establish Establish a common language for roles, common responsibilities, and practice areas using language the form of Knowledge Areas Define Define the competencies shared across competencies the knowledge areas Capture common Capture common techniques for sharing techniques across the industry 7 2012 Collaborative Consulting
  8. The SPE Body of Knowledge Enterprise Software Performance Engineering Planning, coordination, information sharing, and control Knowledge areas Performance validation and testing Problem Application SDLC & (PVT) detection performance architecture and management (SA) Capacity planning (APM) resolution (PDR) (CP) Underlying competencies and roles for delivering Techniques supporting the knowledge areas 8 2012 Collaborative Consulting
  9. The Enterprise PE team: There must be a practice leader A shared service and clearing house for all performance, scalability and capacity planning information. Transforms performance information to business knowledge and advantage. Communicates across the teams to share information For instance, the production workload and the Operations PT performance test workload – are they in sync? Architecture and Make sure the architecture team knows how the production new release, or newest build behave in production. The business risks are clearly identified and Business direction communicated. Can provide performance guidelines to the Development teams developers, must influence the unit testing process. 9 2012 Collaborative Consulting
  10. Information sharing across the knowledge areas Enterprise PE Capacity planning and planning Triage monitoring SPE impacts Architects, Results, Developers, testing, and Forecast Workload operations Workload scalability NFR’s Performance User Exp Volumetrics, validation, Workload Utilization workload testing SDLC, Results Architecture Observations APM , NFR Growth plans Workload NFR’s 10 2012 Collaborative Consulting
  11. PE: Knowledge areas SA PVT CP APM PDR Software development Performance testing and Capacity planning Application performance Problem detection and lifecycle and architecture validation management resolution 11 2012 Collaborative Consulting
  12. PE: Software development lifecycle and architecture SA PVT CP APM PDR Software development Performance testing and Capacity planning Application performance Problem detection and lifecycle and architecture validation management resolution 12 2012 Collaborative Consulting
  13. PE: SA (software development lifecycle) Waterfall methodologies Functional Performance Define Design Develop Deploy test test Need to plan for Build code to performance performance testing requirements Define Deploy Agile Design methods Test Build Can a system be designed to support 10 TPS and 1 second response time work @ 100 TPS and 1 second? 13 2012 Collaborative Consulting
  14. SDLC: Define and design Define Design 1. Project risk scorecard 1. Technology Architecture validated - Key business transactions to the performance Usage identified Scenarios. How will it achieve the - Workflow constraints key performance requirements? Sub-second - SLA’s and penalties response time. Define the scalability approach. Flash marketing 2. Workload model and usage event. 2. Scalability to support flash events scenarios 3. Environments: include any 3. Business volumes with peak required changes to develop and factors test the performance usage 4. Business performance scenarios. requirements Communication to your downstream processes and people Prepare to deal with flash events Design to avoid flash events (they are expensive. E.g. Work with business why does the outbound email have to send email to all xM customers at the same time 14 2012 Collaborative Consulting
  15. SDLC: Develop and functional testing Functions are tested. Performance needs to be analyzed. Develop Functional test • The functions are mapped to software • Verify business functions and key components. The performance req’s are performance functions. assigned to the components. • Verify the key performance workflows • Unit testing for early validation of the and functions. performance req’s for the software components. Response time • Define and execute micro-performance measurements. testing in the QA process. • Integration testing include micro- • Produce report on for each key performance testing of key functions. performance Usage Scenario with Code review of the key performance response time and throughput. functions. • Enhance unit and integration test for early validation of key performance functions Greatest impact on performance and scalability, but you must know the goals. Communication to your downstream processes and people 15 2012 Collaborative Consulting
  16. Performance budgets Do your developers know how much time they have for their piece of the transaction? Code runs everywhere. SERVER WEB APP SVR PROXY MQ/ESB LDAP CORBA DCOM Respond to the user in 2 seconds. Web Services Datacenter 16 2012 Collaborative Consulting
  17. PE: Performance testing and validation SA PVT CP APM PDR Software development Performance testing and Capacity planning Application performance Problem detection and lifecycle and architecture validation management resolution 17 2012 Collaborative Consulting
  18. PE: PTV (performance testing and validation) What is changing? Where does it need to be validated? What (all) is being measured? The project: Why are we testing and what are the success criteria? Prerequisites: Business goals and non-functional requirements are defined Workload and scenarios defined Environment defined (where are we testing) Application in a stable state Configuration management and release management ready Core performance team identified Supporting team committed 18 2012 Collaborative Consulting
  19. Organization: Performance team value You still need to explain to the rest of the organization what you do Business and Early involvement application area with the applications Test execution Environment knowledge team • Deep understanding of • Performance engineers • Well-understood • Testing environment is the application being involved in design workload model and a close approximation tested sessions characterization of production, including database size • Deep understanding of • Understand the nature • All components in the the technical of the data required for test are monitored and • Differences between architecture of the testing reported on test and production are application clearly known • Able to understand • A defined triage process • Understanding of the technical gaps to meet for troubleshooting • Release and business area requirements not configuration • A process to manage supported by the supported by the tool management in place the inevitable applications unplanned tests and and procedures enforced test archive exists Performance team value 19 2012 Collaborative Consulting
  20. Performance team reference model Primary test, design, and execution team Performance engineering support skills Planning and Application project Performance architect management engineer Technology SME Test Results automation analyst Data technician engineering Must determine the core team members vs. on-demand team 20 2012 Collaborative Consulting
  21. Enterprise performance team PT problem Strategy and PT technology resolution planning PMO board (architecture)** Infrastructure** Performance Performance test engineering and execution team optimization Application Vendor Application support management deployment Vendor IBM Oracle EMC Informatica Vendor Vendor Vendors 21 2012 Collaborative Consulting
  22. Model test case execution In the end, the test has to be run. Prepare test Prepare test Confirm Select test harness online Initiate test 0 case or batch monitoring components environment readiness case 1 messages No Problem analysis and Evaluate triage team impact of Select test Normal test 1 case completion? change Yes Yes Gather test Analysis for Create test results specific report; publish Txn RT, CPU, DB change? to repository 2 No Are Categorize Can you Implement changes change – enter retest with Rerun the test 2 required for Yes change to CM configuration change the Yes case retest? tracking tool change? No No Move to next Move to next test case 0 test scenario 22 2012 Collaborative Consulting
  23. PE: Capacity planning SA PVT CP APM PDR Software development Performance testing and Capacity planning Application performance Problem detection and lifecycle and architecture validation management resolution 23 2012 Collaborative Consulting
  24. PE: CP (capacity planning) The right system at the right price! Plan for • Forecast workloads future • Plan for usage Analyze • Track utilization of existing systems current capacity • Track workload Determine • Define system to capacity support workloads requirements • Test • Define service levels Workloads drive system transactions, which drive resource utilization 24 2012 Collaborative Consulting
  25. Capacity: Use of resources The overall system has a capacity; each component has a capacity. SERVER WEB APP SVR PROXY MQ/ESB LDAP CORBA DCOM Web Services Datacenter 25 2012 Collaborative Consulting
  26. Workload model validation Actual workload Synthetic workload Model calibration System System Measured response times, Measured response times, throughputs, utilizations, etc. throughputs, utilizations, etc. Acceptable No Yes 26 2012 Collaborative Consulting
  27. PE: Application performance management SA PVT CP APM PDR Software development Performance testing and Capacity planning Application performance Problem detection and lifecycle and architecture validation management resolution 27 2012 Collaborative Consulting
  28. Five elements of APM (Gartner) Monitoring the performance of complex distributed applications End user experience measurement – browser/mobile 1 device 2 Create a model of the run-time environment (discovery) Profile the performance and behavior of user-defined 3 transactions Performance metrics from each of the 4 applications/systems technical components (Webserver, App server, Database, etc.) 5 Application performance management database 28 2012 Collaborative Consulting
  29. End user experience monitoring 29 2012 Collaborative Consulting
  30. Track key user experience metrics Mobile Total end-user response time Browser rendering time Network latency Mobile Real User Monitoring Rendering 30 2012 Collaborative Consulting
  31. And see every hop of the transaction Mobile Proxy server Web server App server Data warehouse gateway Mainframe Database Dynamic Transaction Path Detection Rendering 31 2012 Collaborative Consulting
  32. PE: Problem detection and resolution SA PVT CP APM PDR Software development Performance testing and Capacity planning Application performance Problem detection and lifecycle and architecture validation management resolution 32 2012 Collaborative Consulting
  33. Performance issues and root-cause analysis This leverages several knowledge areas A solution must be found quickly (under pressure) Performance issues are found in: • Preproduction environment • Production and operations environments Key business transactions are not meeting their service level objectives: • End user experience is degrading (acute or chronic) • Overnight processing is taking longer • Stability and performance issues 33 2012 Collaborative Consulting
  34. When problems occur Flash web site events (all at once) New code released causes problems Infrastructure upgrade Consolidation of applications and servers New workload model (batch and online) 34 2012 Collaborative Consulting
  35. Underlying competencies Analytical Project Communication thinking management skills problem solver Business Monitoring and Statistics Knowledge tuning Workload Behavioral Application and model characteristics tool knowledge Underlying development competencies Test data Test execution Forecasting management process 35 2012 Collaborative Consulting
  36. Technical competencies Code Deep dive Resource Database profiling tools utilization monitoring tools (HP Diagnostics, tools CA Wily, etc, system (Static analysis, Jprobe, etc) diagnostics) level End to end Comprehensive Statistics and performance results collection Distributions tools and reporting Technical competencies Test design Queuing and and Test data Little’s law execution modeling creation tools tools 36 2012 Collaborative Consulting
  37. PE Techniques Brainstorming Data modeling Creating a workload model Performance testing Triage approach Static code analysis Logging and System metrics collection instrumentation 37 2012 Collaborative Consulting
  38. PE roles: The team – many possible roles Managing and developing the core team members 1 Technical Project manager 2 Performance Architect 3 Sr. Performance Engineer 4 Performance Engineer 5 Sr. Test automation technician 6 Test automation technician 7 Performance data engineer 8 Sr. Capacity planner 9 Capacity planner 10 Modeler 38 2012 Collaborative Consulting
  39. Career path: Performance engineer foundation Performance Engineer Grow your Performance testing leadership and Lead communication technical Workload role models Technical Production architecture triage Core Additional technical tech competency Communication competency Grow your breadth of capabilities 39 2012 Collaborative Consulting
  40. Case study: Preparing for a flash marketing event Planning a large event to introduce a new product • Send out email to all registered customers (500,000 currently), there is a link in the email. • Run multiple TV ad’s encouraging people to register and then come to the event, results in 100,000 new registrations • No new functionality required to support the event • Potential 600,000 users of the web site The business is forecasting at least a 10X increase during the event, in terms of product sales • How does this translate into system workload and utilization? • Do we need more servers? • The current system was not really designed for such a flash event 40 2012 Collaborative Consulting
  41. PE: Process to prepare for the event (10X) CP CP PVT SA Forecast the workload Tune application based on results APM PDR Measure the current Plan and execute workload and PT for forecast resource utilization workload 41 2012 Collaborative Consulting
  42. CP: Workload model CP The CP team uses this information to forecast the new workload 10X. Step Transaction – by product Percentage 1 Login 100 2 Search product 100 3 View product detail 100 4 Add to shopping cart 50 5 Confirm purchase method 50 6 Shipping information 50 7 Send confirmation email 50 42 2012 Collaborative Consulting
  43. CP: Analyze current capacity CP Review current Production workload, and the under lying system utilization. • Average workload • Current peak workload Response time Service level 100% System utilization 43 2012 Collaborative Consulting
  44. PVT: planning, executing, and analysis PVT • Use the workload model from CP • Enable APM during testing • Work with the Application Architect Methodology is iterative and tactical Define business Review infrastructure activity profiles & Design & build tests & architecture service levels • Identify risk areas • Types & numbers of • Test data generation • Review configuration users • Create test scripts settings, topology, & sizing • Business activities & • User & transaction profiles • Define points of frequencies • Infrastructure configuration measurement Iterate testing & tuning 44 2012 Collaborative Consulting
  45. Questions? Thank you! Get your free copy SharePath RUM today! More information: 45 2012 Collaborative Consulting

Notas do Editor

  1. Performance is often at odds with other key quality attributes, such as flexibility, portablility, maintainability, etc.
  2. The performance team fits inside a large context model. How does it interact with other groups?
  3. The performance database is required to support performance analysis and analytics. This is a resource for the CP team.
  4. You have the ability to watch/monitor every transaction for every user. What Is A Transaction?Request issued by an end-userGoes through multiple componentsEach component may be activated multiple timesNote that:Different transaction types take different paths and flowsSometimes instances of the same transaction type execute across different flows
  5. What Is A Transaction?Request issued by an end-userGoes through multiple componentsEach component may be activated multiple timesNote that:Different transaction types take different paths and flowsSometimes instances of the same transaction type execute across different flows
  6. The process for PDR is need for production and preproduction environments. This is required in Performance testing, critical for a key POC that demonstrates business advantage. Most POC’s experience problems along the way. Having a well defined detection and resolution process is key to success. Problems will happen and you must be ready for them, and move past them quickly.They can occur gradually or suddenly. A small change can cause big problems or disruptions. How do they manifest themselves to the end-user, or business workflow?
  7. Flash events can wreak havoc on a system that was not designed for scaling. Database contention can come into play at much higher transaction levels. Slow memory leaks can suddenly become exposed at 10x level.Consolidation of servers or applications is a difficult tasks. Combining workloads onto one physical server that is hosting many Virtual Servers always has few unintended consequences.
  8. How do we support the delivery of the five knowledge areas?Analytic skills are required; Decision making, Creative thinking, Learning, Problem solving, Systems thinkingBusiness knowledge – awareness of your organization and how it makes decisions (especially on budgeting)Interaction skills - Negotiation and facilitation, Leadership, Consultative/Influencing, Teamwork. Test data management – a very critical part of the Body of Knowledge.
  9. You should correlate the TV ad to the system utilization. Did the ad drive traffic to your web site?
  10. Each step is multiple system transactions. For instance, I have seen some login processes display too much information. In a low volume site, that might be ok. But under stress, the information for the login screen could crush the system.
  11. Understand when the current system violates the service level. How many transactions and how far away from the Flash event goal are we? This relies on the APM tools and procedures in place. Collecting the system CPU metrics and database usage information.
  12. Test planning, results review.Stress testing approach for the application. Most application break well before the 10X level. During the testing process, the development team will be making coding changes to help scale the application.Results reporting, detailed metrics on utilization are critical, also, often times deep diagnostics are required to find the problems.