1. Advanced Troubleshooting and
Performance Testing
#SPSBE SPSBE31
Donald Hessing
April 26th, 2014

2. Who am I?
Donald Hessing
 Principal Consultant | Thought Leader SharePoint @Capgemini
Netherlands
 Microsoft Certified Master (MCM) - SharePoint
 (Virtual) Technology Solution Professional for Microsoft
 Work full time on SharePoint since 2007 | #DEV | #ITPRO | #STRATEGY
 donald.hessing@capgemini.com | @dhessing | #SPSBE SPSBE31

5. It’s not all about SharePoint!
Load Balancer

6. Understand response time

7. Response Time
1414 KB
1024 KB (10Mb)
20
ms
(44
/2)
300 ms 300 ms 2.6 sec.
1414KB
100 KB (1Mb)
150
ms
(44
/2)
300 ms 300 ms
15.42
sec
IE8, FireFox 3 – 6
connections max.
TCP Slow Start

8. SQL IO Characteristics
Operation Random / Sequential Read/Write Size Range
OLTP – LOG Sequential Write 512 bytes – 64KB
OLTP- Data Random Read / Write 8KB
Bulk Insert Sequential Write 8KB – 128KB (in
multiples of 8KB)
Read Ahead Sequential Read 8KB – 128KB (in
multiples of 8KB)
Backup Sequential Read/Write 1MB
Restore Sequential Read/Write 64KB

9. Validate configurations prior to deployment

10. Customer Example – IOMonitor.exe
Design TEST - 2 KB Random - Read/Write (67%/33%)
Drive
Expected
IO
Total
IO
Average
Read Time
(ms)
Average Write
Response Time
(ms)
Maximum
Read Time
(ms)
Maximum
Write
Time(ms)
C: 60 314 3 1 197 6
L: 300 419 2 3 31 11
S: 600 386 3 3 115 28
T: 1200 51 35 52 89 2501
U: 1000 43 5 59 41 1817

11. Monitoring Pending Disk IO
http://henkvandervalk.com/sql-under-the-hood-part-1-monitoring-current-pending-disk-ios

13. Let’s investigate….
 Who
 What
 What
 When when
 Where
 Why Why

14. …defines the scope
 LabUser8
 30 seconds render time
 Mon-Fri between 0900-1100 CST, but not on weekends
 WFE1
 Typically achieves results between 5-7 seconds
 When user is searching for “Widget” and hits WFE1

15. ULS Viewer

16. Fiddler

17. Fiddler – Troubleshooting WCF services
http://127.0.0.1:8888

18. Fiddler – Troubleshooting Search

20. LOGPARSER 2.2

21. LOGPARSER

22. IIS LOGS

23. Example – Find slow pages

26. Customer Case

27. Questions?

28. LOGPARSER "SELECT QUANTIZE(TO_TIMESTAMP(date, TO_LOCALTIME(time)), 3600) AS Hour,
DIV(SUM(sc-bytes),1048576) AS TotalBytesSent, DIV(SUM(cs-bytes),1048576) AS
TotalBytesReceived FROM *.log GROUP BY Hour ORDER BY Hour" -i:w3c
Determine network capacity..
Number of users 5.000 10.000 20.000 50.000
Bandwidth (Mbit/s) 25 50 100 250

29. Response times
SC-STATUS AVGTIME MAXTIME HITS
200 215 33882 30581
302 56 3104 2949
400 58 78 13
401 16 3166 14811
403 31 31 1
404 81 2745 1472
500 63 109 12

30. Response Times Web Frontend Servers

31. Response times per DOMAIN

32. Response times for DOMAINA per WFE

33. Why are my responses for DOMAINA
users SLOW on WFE2?

35. Win32Status

36. SC-WIN32-STATUS
Win32-Status Description Hits
1 Incorrect function. 17
2 The system cannot find the file specified. 4788
5 Access is denied. 1
64 The specified network name is no longer available. 1663
121 The semaphore timeout period has expired. 964
995
The I/O operation has been aborted because of either a thread exit or an application
request.
85
1236 The network connection was aborted by the local system. 3934
1330 The password for this account has expired. 6
2148074248 The token supplied to the function is invalid 120
2148074252 The logon attempt failed 271
2148074254 No credentials are available in the security package 117831
2148074257 No authority could be contacted for authentication. 9379

38. Performance Testing

39. Custom Web Test Request Plugin

41. Valid testing data is essential for performance tests

42. Content Generation
• Download a copy of the wiki database
http://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia
• English ~40GB XML Data
• Dutch ~10GB XML Data
• Wrote custom wiki parser for generating

43. Load Testing

44. Performance Counters
% CPU SharePoint Servers Threshold
% CPU SQL Server <75%
% CPU SharePoint Servers <75%
# Available Mbytes SharePoint Servers < 20%
Avg. Disk Write Queue Length < 2 Per Spindel
Avg. Disk Read Queue length < 2 Per Spindel
Avg. Disk Sec/Read SQL Server Disks < 12 ms – 0.012
Avg. Disk Sec/Write SQL Server Disks < 12 ms – 0.012

45. Load Results
Avg=48%
Avg=29%
Avg=11%
Avg=0.013
Avg=6.18

46. Thank you!

47. Thanks to our sponsors!
Gold
Silver