Apache Helix DevOps & LSPE-IN Meetup , June'15

1. Simplifying
Distributed
Systems
Shahnawaz
Saifi,
@shahsaifi
h:p://www.linkedin.com/in/shahsaifi
June’2015

2. Outline
• Introduc)on
• Architecture
• How
to
use
Helix
• Helix
usage
• Q
&
A
2

3. Examples
of
distributed
data
systems
3

4. Lifecycle
Single
Node
MulM
node
• ParMMoning
• Discovery
• Co-­‐locaMon
Fault
tolerance
• ReplicaMon
• Fault
detecMon
• Recovery
Cluster
Expansion
• Thro:le
data
movement
• Re-­‐distribuMon
4

5. Typical
Architecture
5

6. Distributed
Search
Service
6

7. Distributed
data
Store
7

8. Message
consumer
group
• Similar
to
Message
groups
in
AcMveMQ
–
guaranteed
ordering
of
the
processing
of
related
messages
across
a
single
queue
–
load
balancing
of
the
processing
of
messages
across
mulMple
consumers
–
high
availability
/
auto-­‐failover
to
other
consumers
if
a
JVM
goes
down
• Applicable
to
many
messaging
pub/sub
systems
like
ka]a,
rabbitmq
etc
8

9. Message
consumer
group
9

10. We
need
high
level
primiMves.
ApplicaMon
Zookeeper
• Filesystem
• Lock
• Ephemeral
Zookeeper
provides
low
level
primiMves.
ApplicaMon
Framework
Consensus
System
• Node
• ParMMon
• Replica
• State
• TransiMon
10

11. 11

12. Outline
• IntroducMon
• Architecture
• How
to
use
Helix
• Helix
usage
12

13. Terminologies
13

14. Core
concept
14

15. Helix
soluMon
15

16. IDEALSTATE
16

17. CURRENT
STATE
17

18. EXTERNAL
VIEW
18

19. Helix
Based
System
Roles
19

20. Logical
deployment
20

21. Outline
• IntroducMon
• Architecture
• How
to
use
Helix
• Helix
usage
21

22. Helix
based
soluMon
1. Define
2. Configure
3. Run
22

23. Step
1,
Define:
State
model
definiMon
• States
–
All
possible
states
–
Priority
• TransiMons
–
Legal
transiMons
–
Priority
• Applicable
to
each
parMMon
of
a
resource
• e.g.
MasterSlave
ConMnued…
23

24. Define:
State
model
Builder = new StateModelDefinition.Builder(“MASTERSLAVE”);
//Add states and their rank to indicate priority.
builder.addState(MASTER, 1);
builder.addState(SLAVE, 2);
builder.addState(OFFLINE);
//Set the initial state when the node starts
builder.initialState(OFFLINE);
//Add transitions between the states.
builder.addTransition(OFFLINE, SLAVE);
builder.addTransition(SLAVE, OFFLINE);
builder.addTransition(SLAVE, MASTER);
builder.addTransition(MASTER, SLAVE);
ConMnued…
24

25. Define:
constraints
State
Transi)on
ParMMon
Y
Y
Resource
-­‐
Y
Node
Y
Y
Cluster
-­‐
Y
State
Transi)on
ParMMon
M=1,S=2
-­‐
ConMnued…
25

26. Define:
constraints
// static constraint
builder.upperBound(MASTER, 1);
// dynamic constraint
builder.dynamicUpperBound(SLAVE, “R”);
// Unconstrained
builder.upperBound(OFFLINE, -1);
ConMnued…
26

27. Define:
parMcipant
plug-­‐in
code
27

28. Step
2:
configure
28

29. Zookeeper
view
29

30. Step
3:
Run
30

31. zookeeper
view
31

32. Znode
content
32

33. Spectator
Plug-­‐in
code
33

34. Helix
ExecuMon
modes
34

35. IDEALSTATE
35

36. ExecuMon
modes
• Who
controls
what
36

37. Auto
rebalance
v/s
Auto
37

38. In
acMon
38

39. Custom
mode:
example
39

40. Custom
mode:
handling
failure
• Custom
code
invoker
– Code
that
lives
on
all
nodes,
but
acMve
in
one
place
– Invoked
when
node
joins/leaves
the
cluster
– Computes
new
idealstate
– Helix
controller
fires
the
transiMon
without
viola)ng
constraints

41. Helix
usage
at
LinkedIn
Primary
DB
Data
Change
Events
Standardi
zaMon
Standardi
zaMon
Standardi
zaMon
Standardi
zaMon
Standardi
zaMon
Search
Index
Standardi
zaMon
Standardi
zaMon
Graph
Index
Standardi
zaMon
Standardi
zaMon
Read
Replicas
Updates
Databus
Espresso
41

42. Summary
• Helix:
A
Generic
framework
for
building
distributed
systems
• Modifying/enhancing
system
behavior
is
easy
–
AbstracMon
and
modularity
is
key
• Simple
programming
model:
declaraMve
state
machine
42

43. website
h:p://helix.apache.org
codeRepo
h:p://github.com/apache/helix
user
user@helix.apache.org
dev
dev@helix.apache.org
twi:er
@apachehelix,
@shahsaifi