Presentation I delivered at ISSA UK "Application Security - London Chapter Meeting" https://www.eventbrite.co.uk/e/application-security-london-chapter-meeting-tickets-42284085839

1. Using Security to drive
Chaos Engineering
Dinis Cruz, CISO, Photobox
April 2018

2. https://pbx-group-security.com

3. I’m a CISO focused on
securing our client’s Magic moments
by creating secure environments
that enable and accelerate the business
and contribute to the
top and bottom line

4. Here are my challenges
How to make rational risk based decisions
How to create high performance teams
How to scale Security knowledge
How to drive and enable change
How to map data as graphs

5. We are also hiring :)
Head of AppSec
Head of Cloud Security

6. What is chaos
engineering

7. Success story
Netflix recommendations

9. Chaos Engineering is

11. Evolution of Testing
https://www.slideshare.net/NoraJones1/choose-your-own-adventure-qcon-2017-1

12. Let’s look at a number of
Chaos Engineering
definitions

13. Chaos Engineering
Building Confidence
in System Behaviour
through experiments

14. Chaos Engineering is
about trying controlled
changes to observe system
availability deviation

15. Chaos Engineering is
carefully injecting harm into
our systems
to test the system’s ability to
respond to it.

16. Chaos Engineering
is the discipline of experimenting on a
distributed system
in order to build confidence
in the system’s capability to withstand
turbulent conditions in production

17. (most business friendly)
Chaos Engineering is
limited scope,
continuous,
disaster recovery

18. PrinciplesOfChaos.org

19. 1. Start by defining ‘steady state’ as some measurable output of a system that
indicates normal behaviour.
2. Hypothesise that this steady state will continue in both the control group and
the experimental group.
3. Introduce variables that reflect real world events like servers that crash, hard
drives that malfunction, network connections that are severed, etc.
4. Try to disprove the hypothesis by looking for a difference in steady state
between the control group and the experimental group.
Chaos in practice - 4 experiments
http://principlesofchaos.org/

20. 1.Build a Hypothesis around Steady State Behavior
2.Vary Real-world Events
3.Run Experiments in Production
4.Automate Experiments to Run Continuously
5.Minimise Blast Radius
Advanced Principles
http://principlesofchaos.org/

21. the idea that “Chaos engineering is
not Testing”
Is caused by
the failure to make TDD (Test-Driven
development) Scale

22. TDD Demo
1) Real-time
Test Execution
2) Real time
code coverage

23. Chaos Engineering
is testing
the different are the test abstractions
and the extra random layer

24. Security as chaos
generator

25. Cyber Security is a ‘change
generation factory’

26. Developers and Techops are
chaos creators

27. Security testing (and users)
are chaos creators

28. I have been called
Director of chaos :)

29. The myth of the
singe point of failure
(i.e. attackers only need to run
code and find a weak spot)

30. Do you understand what is
going on in your network?

31. Biggest threat is not the issue,
but is not having visibility

32. When do you know about
security incidents?
(or changes)

33. You need to know what the
attackers are
doing on your system
(and users)

34. If you don’t know what is on
the pentest report …
you have a bigger problem
(i.e. your SOC should be able to tell you)

35. Best Security model is one
based on
the attacker making a mistake
(i.e. a change)

36. Use risks to understand reality
and to make the business
owners responsible for their
decisions

37. Use Threat Models to
understand how your system
works and to document it

38. Use tests to replicate
known behaviours, attacks
and simulate changes
(with and without random events)

39. Which can also called
Security tests
(which pass on vulnerable
state and on regression test)

40. Some scenarios

41. If a server on your cloud is
mining bitcoins
Would you know about it?

42. If a server or app misbehaves ?
When do you know about it?

43. If your servers start running
30% slower?
What happens to your apps?

44. If your servers fails to reboot
after a patch
What happens to your system?

46. When (not if) you have
malicious or api breaking
dependencies?
How do you know about them?

47. If 3rd parties are using your
APIs (official or not) to dump
your user’s data (aka Facebook)
Would you know about it?

48. Properties of resilient
and secure systems

49. Availability

50. Plan for Failure
Ability to sustain failures

51. Validate and Sanitise
all requests

52. Authenticate and Authorise
all requests

53. Reduce capabilities and
features gracefully

54. Hostile to
insecure traffic
and
insecure code

55. Have error budgets
(from Google SRE)

56. Are easy to change

57. Are easy to refactor
(make changes with confidence)

58. Pushes to production
happen minutes
(fully tested and 100x a day (if needed))

59. The bigger they
get the faster they go
(it is smooth and safe to make changes)

60. Have 99% change coverage

61. Change coverage

62. It is not about
test code coverage

63. What matters is
change coverage

64. If you make changes
and they are not detected

65. you are just making
random changes

66. Basically
you are an
agent of chaos

67. Every change you make
has to have a respective
test change
(much better pair programming model)

68. Chaos Engineering
is modern change
management

69. Chaos Engineering
is the programatic
introduction of changes

70. one more thing

71. Following from 2017 edition
https://owaspsummit.org

72. Collaboration @ 16x per day for 5x days

73. Open Security Summit 2018 - London
https://open-security-summit.org/4th - 8th of June

80. Any questions
@DinisCruz