Slides from TDC POA 2016 talk about dealing with failures in distributed systems with circuit breaker pattern.

1. Resilient
Applications
with Circuit Breakers

2. failure• n. (pl. failures)
1 inability of a system or component to perform its required
functions within specified performance requirements

3. Extracted from Netflix’s Github
Dependencies

4. Extracted from Netflix’s Github
Dependencies

5. Extracted from Netflix’s Github
Dependencies

7. Error propagation

8. Embrace failure
● Faults will occur
● Faults are tolerable
“We should write software that is cynical and expects
bad things to happen” - Michael T. Nygard

9. 1 an automatic device for stopping the flow of a
current in an electric circuit as a safety measure
circuit-breaker • n. (pl. circuit-breakers)

10. Resilience
● Fail fast

11. Resilience
● Fail fast
● Rapidly recover

12. The Pattern

22. Netflix Hystrix

23. How it works
● Wrap calls
● Thread pool
● Timeout
● Measuring
● Fallback

24. The Code

25. class SleepCommand extends HystrixCommand<String> {
SleepCommand() {
super(
withGroupKey('sleep-group').
andCommandPropertiesDefaults(
withCircuitBreakerSleepWindowInMilliseconds(wait).
withExecutionTimeoutInMilliseconds(timeout)
)
)
}
...

26. class SleepCommand extends HystrixCommand<String> {
SleepCommand() {
super(
withGroupKey('sleep-group').
andCommandPropertiesDefaults(
withCircuitBreakerSleepWindowInMilliseconds(wait).
withExecutionTimeoutInMilliseconds(timeout)
)
)
}
...

27. class SleepCommand extends HystrixCommand<String> {
SleepCommand() {
super(
withGroupKey('sleep-group').
andCommandPropertiesDefaults(
withCircuitBreakerSleepWindowInMilliseconds(wait).
withExecutionTimeoutInMilliseconds(timeout)
)
)
}
...

28. class SleepCommand extends HystrixCommand<String> {
SleepCommand() {
super(
withGroupKey('sleep-group').
andCommandPropertiesDefaults(
withCircuitBreakerSleepWindowInMilliseconds(wait).
withExecutionTimeoutInMilliseconds(timeout)
)
)
}
...

29. class SleepCommand extends HystrixCommand<String> {
SleepCommand() {
super(
withGroupKey('sleep-group').
andCommandPropertiesDefaults(
withCircuitBreakerSleepWindowInMilliseconds(wait).
withExecutionTimeoutInMilliseconds(timeout)
)
)
}
...

30. static final Integer timeout = 100
@Override
String run() throws Exception {
def time = Math.abs random.nextInt(150)
Thread.sleep time
"Sleeped for ${time} miliseconds"
}
@Override
String getFallback() {
"Didn't sleep at all"
}

31. static final Integer timeout = 100
@Override
String run() throws Exception {
def time = Math.abs random.nextInt(150)
Thread.sleep time
"Sleeped for ${time} miliseconds"
}
@Override
String getFallback() {
"Didn't sleep at all"
}

32. static final Integer timeout = 100
@Override
String run() throws Exception {
def time = Math.abs random.nextInt(150)
Thread.sleep time
"Sleeped for ${time} miliseconds"
}
@Override
String getFallback() {
"Didn't sleep at all"
}

33. static final Integer timeout = 100
@Override
String run() throws Exception {
def time = Math.abs random.nextInt(150)
Thread.sleep time
"Sleeped for ${time} miliseconds"
}
@Override
String getFallback() {
"Didn't sleep at all"
}

35. References

36. References

37. References

38. References

39. References

40. References
Release It, Design and Deploy Production-ready Software, Michael T. Nygard
https://pragprog.com/book/mnee/release-it
Application Resilience Engineering at Netflix, Ben Christensen
https://www.youtube.com/watch?v=RzlluokGi1w
Operational Excellence with Netflix Hystrix, Billy Yuen
https://www.youtube.com/watch?v=VuCYkY7pFk8
Making Netflix more resilient, Ben Schmaus
http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html
Hystrix, Defend Your App, Netflix Github
https://github.com/Netflix/Hystrix/wiki/How-it-Works

41. Josué Neis
jneis josueneis josueneis