Hints Complex System Performance Optimization

Hints on complex system performance
optimization
Ainars Galvans
Exigen Services Latvia

December 2, 2013

www.ExigenServices.com

System Performance
Introduction: context, terms and basics

3

CONFIDENTIAL – NOT FOR DISTRIBUTION


Terminology

4



Performance is:

Requirement
Number of users
Hardware spec.
“Response time”

Measurement
Throughput
CPU utilization
Response time

i.e.
time user wait for
page to load
in the browser

5



A simple 3-tier Web Application

Client

App.

DB

Prepare request
Response time

User Experience

Parse, process
Select or update
More Logic

Select or update
Render HTML

Render UI
Repeated
6



A simple 3-tier Web Application

App.
Client1

7

DB

Client2



With 3rd party WebServices

Client1

App.

DB

Client2

8



Other Apps share the same DB?

App.

9

DB


Other
(Corp.)
App.


Performance is:

Requirement
Number of users
Hardware spec.
“Response time”

10

Measurement
Throughput
CPU utilization
Response time



Rules of thumb

 Response time
– >2 seconds: is it all CPU time?
– 0.5-2.0 seconds: why?
– <0.5 seconds : OK

 Concurrent users
– The term is misleading
– What matters technically is throughput

 Resource utilization:
– DB CPU should be below 50%
11



My issue localization techniques
 First step: where (what host) is the bottleneck?
– Single user repeat the same action (i.e. with Selenium)
– Monitor “CPU time” delta (secs.), not CPU (%)
– Use excel to understand how it scales
 If it’s the App Server: analyze HTML
– FireFox “Save Page As” -> complete page
– Look for large, unused data
 If it’s DB or Web Services: analyze traffic, logs
– Put milliseconds to log files, see what takes most of it
– Look for repeated requests for single page rendering
 If it’s client (browser): profile JavaScript
– IE -> Dev. Tools (F12) -> Profiler
– Are all the JavaScripts cached (or compiled each time)?
12



System Performance
Performance optimization stories

13



Locating database call issues

App.
Client1

14

DB

Client2



Story #1: Just a few tables
 Problem description
– Web Page shows a number of tables
– Page response time is a bit above 2 seconds
– Under load DB becomes overloaded quickly
– All DB queries takes <0.1 sec
 Investigation results:
– Single page load cause about 100 select executions
– Each took 0.02 secs
– There were two distinct statements
 Solution
– After adjusting DB caching
– Response times reduced to 0.4
– Almost no load to DB under load
15



Locating WebService call issues

Client1

App.

DB

Client2

16



Story #2: One call only takes 0.5 secs.
 Problem description
– Web Page shows user’s data overview
– Typical customer page response time 5 seconds
– Response time up to 30 seconds for specific customers
– Under load response times don’t increase too much
 Investigation results:
– Data in tables is requested via web services
– Separate call made on each business item
– Call response time is 0.5 secs
 Solution
– Protocol adjusted to request all info with one call
– Response times reduced to 0.9
17



Application Logic (HTML processing)

App.
Client1

18

DB

Client2



Story #3: Tables, again: .NET
 Problem description:
– Page shows main page and a dynamic table with hot links
– Response times >5 secs.
– App server loaded significantly
– Under load response times increase
 Investigation
– In page source (HTML) I’ve discovered huge callbackstate
 Solution
– http://www.devexpress.com/Support/Center/Question/Details/Q42
5597
[Browsable(false)]
– Response times < 3secs. now

19

public string
UnboundLargeData
{ get; set; }



Client Side (rare case though)

App.
Client1

20

DB

Client2



Story #4: Client side
 Problem
– Visually page loads 8-12 seconds and vary
– Looks like response time depend on client workstation
 Investigation
– Server response time is only 3 seconds
– It takes browser 1-2 seconds to send request
– It takes browser 3-4 seconds to render response
– It takes more if workstation CPU is slow (<3 GHz)
 Solution
– Optimize JavaScript and remove features from page
– Browser choice recommendation
– Unfortunately: optimal client CPU recommendation
21



… everything. The business itself

App.
Client1

22

DB

Client2



Story #5. The reports and queries
 Production problem
– Special feature “accounting reports” usually work 1-2 mins
– Sometimes 10 times as much
– Suspected reason a “morning lag” (all users)
 Investigation
– One report loads DB CPU to 100% for 1-2 minutes
– Only technical solution: rewrite whole app, changing DB
structure and logic
 The actual solution
– Nightly batch jobs on demand
– The default nightly batch jobs
– Daily execution only with special privileges and only one at a
time
23



Conclusions
Localizing the problem

24



My issue localization techniques
 First step: where (what host) is the bottleneck?
– Single user repeat the same action (i.e. with Selenium)
– Monitor “CPU time” delta (secs.), not CPU (%)
– Use excel to understand how it scales
 If it’s the App Server: analyze HTML
– FireFox “Save Page As” -> complete page
– Look for large, unused data
 If it’s DB or Web Services: analyze traffic, logs
– Put milliseconds to log files, see what takes most of it
– Look for repeated requests for single page rendering
 If it’s client (browser): profile JavaScript
– IE -> Dev. Tools (F12) -> Profiler
– Are all the JavaScripts cached (or compiled each time)?
25



Happy to share more details

Skype name: ainarsgalvans
Phone: +371 29432698
Facebook: Ainars Galvans
Email: Ainars.Galvans@gmail.com
Blog: find me at SoftwareTestingClub.com
26



Hints Complex System Performance Optimization

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (18)

Semelhante a Hints Complex System Performance Optimization

Semelhante a Hints Complex System Performance Optimization (20)

Mais de DevConFu

Mais de DevConFu (20)

Último

Último (20)

Hints Complex System Performance Optimization

Notas do Editor