Сегодня никого не удивишь высоконагруженными системами. И мало кого в нашей индустрии удивишь отдельно выделенным человеком который занимается нагрузочным тестирование. Большинство людей думают, что они все могут автоматизировать и тесты будут запускаться автоматически. Вот только мало кто знает, что львиная доля времени уходит на анализ результатов (логов и графиков). В докладе будут рассказаны подходы, как возможно сократить время для поиска узких мест при анализе различных логов. Как можно применить простейшие модели машинного обучения для поиска узких мест. И как получить требования с помощью исторических данных.

1. Methodology of
MACHINE LEARNING
in
PERFORMANCE TESTING
(how to find bottlenecks automatically)
t
Krychko Nikita

2. Access to logs
Logs storage
Knowledge of programming language and algorithms
t
KYIV 2018
Prerequisites

3. Decrease time of looking for bottleneck
Decrease time of preparing feedback
Describe in details what and where happens
t
KYIV 2018
Why we need it

4. If everything ok – skip this topic
If not, you should DEBUG
And then find LOAD and BOTTLENECK (place or
resource)
t
KYIV 2018
NOTE

5. BUG Description Or What developer needs to
reproduce :
Test steps: function (operation) name
Conditions: load in RPS (TPS)
Actual results (what happens, what was changed)
Hardware: server name
t
KYIV 2018
BUG REPORT

7. Performance testing tool log
Resources monitoring log
All logs with traceID
t
KYIV 2018
DEPENDENCIES

8. Тема доклада
Тема доклада
Тема доклада
KYIV 2018
Performance
testing
tool log

9. Problem: Sudden increase of response time
Solution: Split a request by layers and test together
t
KYIV 2018
PROBLEM

10. Sample time = 60
60= 20 + 20 + 20
On layers
60 40 20

11. LOAD (rps)
Response times (MS)
REST SOAP SQL
10 60 40 20
20 70 50 30
30 80 60 40
40 90 70 50
50 120 100 60

13. Тема доклада
Тема доклада
Тема доклада
KYIV 2018
Get expected
results based on
history timeseries

14. Problem: Absence of expected results
Solution: Generate them based on historical data
t
KYIV 2018
PROBLEM

18. library(forecast)
fit <- auto.arima(timeseries)
predValue <- forecast(fit,h=1)
Predict
and compare
if ((predValue+10%) < actualValue) {…}
…compare everything you want

19. library(forecast)
fit <- auto.arima(timeseries)
predValue <- forecast(fit,h=1)
Predict
and compare
if ((predValue+10%) < actualValue) {…}
…compare everything you want

20. Тема доклада
Тема доклада
Тема доклада
KYIV 2018
Resource monitoring
tool

21. Problem: Detect conditions when system reaches max limit of
resource usage
Solution: Use k-means clustering to find resources plateau
t
KYIV 2018
PROBLEM

22. CPU %
RAM %
DISK IO mb
Network mb
(depend on process)
t
KYIV 2018
MONITORING RESORCE LOG

24. MONITORING
RESORCE LOG

25. clusters <- kmeans(timeseries, 5, nstart=25)
dt <- data.table("value"=timeseries$value,
"cluster"=as.factor(clusters $cluster))
ggplot(dt, aes(x=1:420, y=value, col=cluster))
+ geom_line()
WITH PASSION TO QUALITY
CLUSTERING

26. clusters <- kmeans(timeseries, 5, nstart=25)
dt <- data.table("value"=timeseries$value,
"cluster"=as.factor(clusters $cluster))
ggplot(dt, aes(x=1:420, y=value, col=cluster))
+ geom_line()
WITH PASSION TO QUALITY
SET CLASS

27. clusters <- kmeans(timeseries, 5, nstart=25)
dt <- data.table("value"=timeseries$value,
"cluster"=as.factor(clusters $cluster))
ggplot(dt, aes(x=1:420, y=value, col=cluster))
+ geom_line()
WITH PASSION TO QUALITY
DRAWING RESULTS

28. MONITORING
RESORCE LOG

29. Algorithm:
•Number of clusters should be less than number of
steps
•Find a cluster with max median
•If timeframe is bigger than time of load step – you
will find your resources plateau (candidate for
bottleneck)
•Filter your log by time and find the load
t
KYIV 2018
CLUSTERING

30. BUG Description Or What developer need to
reproduce this situation:
Test steps: function (operation) name
Conditions: load in RPS (TPS)
Actual results (what happens, what was changed)
Hardware: server name
t
KYIV 2018
BUG REPORT

31. Тема доклада
Тема доклада
Тема доклада
KYIV 2018
Magic of
correlation
What and where was changed

32. Problem: Something changed, and it is unknown who is
responsible for this impact
Solution: Use correlation matrix to detect dependencies and
correlations
t
KYIV 2018
PROBLEM

34. Old build New build Difference

36. Тема доклада
Тема доклада
Тема доклада
KYIV 2018
TRACE ID in
LOGS

37. Add logging with traceId on all levels for each request
(e.g. http header)

38. PT_A.GA_20180921131555_10
t
KYIV 2018
TRACE ID FORMAT

39. PT_A.GA_20180921131555_10
t
KYIV 2018
TRACE ID FORMAT
Performance
testing prefix
Service
initials
Method
initials
Date
Time
Thread
number

40. 1. Gather log from servers
2. Clean logs
3. Merge logs
4. Visualize
t
KYIV 2018
PLAN

41. LOAD (rps) Response times (MS)
REST SOAP SQL
10 60 40 20
20 70 50 30
30 80 60 40
40 90 70 50
50 120 100 60

43. No difference with the first approach, but more accurate.
Possibility to get all internal operations, not only known.
t
KYIV 2018
NOTE

44. Тема доклада
Тема доклада
Тема доклада
KYIV 2018
Questions