Since its beginning, the Performance Advisory Council aims to promote engagement between various experts from around the world, to create relevant, value-added content sharing between members. For Neotys, to strengthen our position as a thought leader in load & performance testing. During this event, 12 participants convened in Chamonix (France) exploring several topics on the minds of today’s performance tester such as DevOps, Shift Left/Right, Test Automation, Blockchain and Artificial Intelligence.

1. Switching Performance
Left & Right
Leandro Melendez

2. Who am I
• Performance guy that likes to explain stuff
• Night time crusader fighting bottlenecks as Señor Performo
• QualiTester @QualiTest
• Contributor @ PerfBytes team and soon to be podcaster
• www.srperf.com
• leandro@srperf.com & leandro@perfbytes.com
• TW @SrPerf
• LI www.linkedin.com/in/leandromelendez
• FB www.facebook.com/srperformo
• Group www.facebook.com/groups/PBytesEsp

3. When I discovered Agile

4. Some Agile Differences
• Technical debt should be swiped often
• People do many things
• Few people
• The scope changes often
• Release stuff fast
• Continuous release
• No siloes
• Collaborate and communicate often
• Everything happens always

5. Problems for performance
• Automation requires frozen stuff
• Requires almost reverse engineering or hacking
• Takes time
• Changes happen too often
• Maintenance of automation gets ugly
• Performance results become invalid
• Can’t stop everything to run tests
• Multiple moving pieces

6. Problems for performance
• Automation requires frozen stuff
• Requires almost reverse engineering or hacking
• Takes time
• Changes happen too often
• Maintenance of automation gets ugly
• Performance results become invalid
• Can’t stop everything to run tests
• Multiple moving pieces

7. Problems for performance
• The approach has been to automate more and faster on every release
• Sprint to sprint more changes come
• Changes impact more automation almost exponentially
• The focus in the center (Test phase) requires to distribute work

8. Problems for performance
• The approach has been to automate more and faster on every release
• Sprint to sprint more changes come
• Changes impact more automation almost exponentially
• The focus in the center (Test phase) requires to distribute work

9. But wait!

10. Agile huge advantages
• The test team is involved and contributes from
day one

11. Agile huge advantages
• The test team is involved and contributes from
day one

12. Agile huge advantages
• The test team is involved and contributes from
day one
• Testers and developers work together
• Code access

13. Agile huge advantages
• The test team is involved and contributes from
day one
• Testers and developers work together
• Code access
• Blurry lines in roles, devs can pre work test

14. Then that gave me some ideas!
• Some terrific ideas!

15. Then that gave me some ideas!
• Some terrific ideas!
• I got some wonderful and not awful ideas!

16. Switch Left First!

17. Automationmanners
• Since there is a say from the customer meetings and design

18. Automationmanners
• Since there is a say from the customer meetings and design

19. Automationmanners
• Since there is a say from the customer meetings and design
• Think of requests from QA’s POV
• The design can take automation into account
• Think of the tester & automation

20. Automationmanners
• Since there is a say from the customer meetings and design
• Think of requests from QA’s POV
• The design can take automation into account
• Think of the tester & automation

21. Automationmanners
• Test manners VS Innovation
Tester- Developer -Customer

22. Automationmanners
• Test manners VS Innovation
• After the innovative design, ease tester’s life
Tester- Developer -Customer

23. Automationmanners
• Test manners VS Innovation
• After the innovative design, ease tester’s life
• Don’t be a bad person
Tester- Developer -Customer

24. Untestableon purpose

25. Untestableon purpose

26. Untestableon purpose

27. Untestableon purpose

28. Test enablement
• Since ages ago there has been a practice to help

29. Test enablement
• Since ages ago there has been a practice to help

30. Test enablement
• Since ages ago there has been a practice to help
• Why has this gotten out of fashion?

31. Test enablement
• Performance requires especially load generation
cheat codes

32. Test enablement
• Performance requires especially load generation
cheat codes
• Direct function calls, hidden front end objects,
test mode, modular calls (such as micro services),
unit like triggering, Internal triggering modules.
• What would be easier?

33. Test enablement
• Performance requires especially load generation
cheat codes
• Direct function calls, hidden front end objects,
test mode, modular calls (such as micro services),
unit like triggering, Internal triggering modules.
• What would be easier?

34. Test enablement
• It is not so new or innovative

35. Test enablement
• It is not so new or innovative
No TE

36. Test enablement
• It is not so new or innovative
No TE
Yes TE

37. Integratedinstrumentation
• You and the dev are almost one
• Code access!
• Implement and enable coding practices

38. Integratedinstrumentation
• You and the dev are almost one
• Code access!
• Implement and enable coding practices
• Code poetry is beautiful!

39. Integratedinstrumentation
• You and the dev are almost one
• Code access!
• Implement and enable coding practices
• Code poetry is beautiful!
• Understandable is maintainable and testable
((x2-x1) < 200) && (var=true;)
// Love at first sight
if (me.getDistanceTo(you.position) < 200) {
me.setFeelings({
inLove: true,
});
}

40. Integratedinstrumentation
• Add timers
• Add counters
• Relevant info
• Error logging w/details
• Code should report from DEV
• No poor performers check in
• Enable disable mechanism
• Send to ELK, Splunk, Graylog, etc.
• DASHBOARDS!!

41. Integratedinstrumentation
• Add timers
• Add counters
• Relevant info
• Error logging w/details
• Code should report from DEV
• No poor performers check in
• Enable disable mechanism
• Send to ELK, Splunk, Graylog, etc.
• DASHBOARDS!!
Method()
{
Stopwatch.start();
CODE;
CODE;
CODE;
END;
Stopwatch.stop();
Log(Stopwatch,
user,
method,
dataUsed,
DevUID//Only for devenv
);
//To report to the developer and
//add to logs.
}

42. Integratedinstrumentation
• Work may take a bit at first, but must focus on making it simple and quick

43. Integratedinstrumentation
• Work may take a bit at first, but must focus on making it simple and quick
• Afterwards it is like daily hygiene. Implement your code’s hygiene

44. Integratedinstrumentation
• Work may take a bit at first, but must focus on making it simple and quick
• Afterwards it is like daily hygiene. Implement your code’s hygiene
• Everything checked-in should have it

45. Integratedinstrumentation
• Work may take a bit at first, but must focus on making it simple and quick
• Afterwards it is like daily hygiene. Implement your code’s hygiene
• Everything checked-in should have it
• Don’t be lazy, validate
• Leave without shower?

46. Integratedinstrumentation
• Work may take a bit at first, but must focus on making it simple and quick
• Afterwards it is like daily hygiene. Implement your code’s hygiene
• Everything checked-in should have it
• Don’t be lazy, validate
• Leave without shower?

47. Instrumentation
Ford Model T 1923

48. Instrumentation
Ford Model T 1923

49. Instrumentation
Ford Model T 1923 Ford Taurus 2013

50. IN THE CENTER

51. QA processes receivebenefits
• Test mode allows to jump steps

52. QA processes receivebenefits
• Test mode allows to jump steps
• Module tests automated calls
• Functional automation is eased

53. QA processes receivebenefits
• Test mode allows to jump steps
• Module tests automated calls
• Functional automation is eased
• Change impacts are shown faster
• Modular fix

54. We already get performancemetrics
• Code instrumentation reports performance when other tests are executed

55. We already get performancemetrics
• Code instrumentation reports performance when other tests are executed

56. We already get performancemetrics
• Code instrumentation reports performance when other tests are executed
• Metrics at many levels

57. We already get performancemetrics
• Code instrumentation reports performance when other tests are executed
• Metrics at many levels
• It is like a chain reaction!

58. AutomateautomatedLoadtests
• Load tests creation can be automated
• Load generation through direct calls
• Front end calls designed from dev

59. AutomateautomatedLoadtests
• Load tests creation can be automated
• Load generation through direct calls
• Front end calls designed from dev
• Automated scenario creation
• Execution in separate pipe

60. AutomateautomatedLoadtests
• Load tests creation can be automated
• Load generation through direct calls
• Front end calls designed from dev
• Automated scenario creation
• Execution in separate pipe
• Deviation analysis per runs & releases
0
2
4
6
8
10
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43
Response time
Response
New code
Check in

61. Switching right

62. Inheritance
• All the instrumentation and invoking is inherited

63. Inheritance
• All the instrumentation and invoking is inherited
• Can be switched on/off/partial/selective
• Prioritize reporting

64. Inheritance
• All the instrumentation and invoking is inherited
• Can be switched on/off/partial/selective
• Prioritize reporting
• Enables alert system with detailed tracking
• Ops know of flaw before customer reports

65. Performanceon theright
• Leverage organic load tests

66. Performanceon theright
• Leverage organic load tests
• Blue green release structure
• Forecast and comparison from green

67. Performanceon theright
• Leverage organic load tests
• Blue green release structure
• Forecast and comparison from green
• Implement probes, agents and APMs

68. Analysis Ops->Dev
• Generates deep usage data
• Data can be leveraged

69. Analysis Ops->Dev
• Generates deep usage data
• Data can be leveraged
• Prod utilization patterns

70. Analysis Ops->Dev
• Generates deep usage data
• Data can be leveraged
• Prod utilization patterns
• ML and QA automation

71. Analysis Ops->Dev
• Generates deep usage data
• Data can be leveraged
• Prod utilization patterns
• ML and QA automation
• Feed automated scenario creator
• AI auto create test cases

72. Analysis Ops->Dev
• Generates deep usage data
• Data can be leveraged
• Prod utilization patterns
• ML and QA automation
• Feed automated scenario creator
• AI auto create test cases
• Prioritize QA risk impact

73. Gracias!

74. Thank you