Implement Reliable, Isolated & Unified Job Submission - Zili Chen

1. Implement Reliable,
Isolated & Unified Job
Submission
Zili Chen – Software Engineer, Flink Committer

2. 2
• Background
• Reliability
• Isolation
• Unification
Outline

3. 3
• Background
• Reliability
• Isolation
• Unification
Outline

4. Background
4
1

5. 5
Background1
Oceanus: Tencent Real-Time Compute Platform
Data Distribution Monitoring Real-Time BI Online Learning
1200
Jobs
35 Trillion
Messages in a day
4.5 PB
Data in a day

6. 6
Background1
The Problem
Job submission & execution is unreliable.
Client becomes bottleneck & suffers security risks.
Flexible programmatic interface is missing.
Start with the workflow of job submission!

7. 7
Workflow1
Cluster Deployment(Session Mode)
Deployer
client side cluster side Resource
Manager
(YARN)
2. Launch Application
Job Manager Container
Dispatcher

8. 8
Workflow1
Job Submission(Session Mode)
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
1. Generate
JobGraph

9. 9
Workflow1
Job Submission(Session Mode)
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
3. Check
JobStatus
4. Persist
JobGraph
(otherwise)

10. 10
Workflow1
Job Submission(Session Mode)
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
5. Start
Job Manager
Job Manager
6. Report
Started

11. 11
Workflow1
Job Execution(Session Mode)
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
Job Manager
1. Check
JobStatus
2. Finished
By Others
2. Set JobStatus
RUNNING

12. 12
Workflow1
Job Execution(Session Mode)
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
Job Manager
3. Set JobStatus
DONE
4. Finished

13. 13
Workflow1
Job Execution(Session Mode)
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
5. Remove
JobGraph
6. Clear
JobStatus

14. 14
Workflow1
Cluster Deployment(Job Mode)
Deployer
client side cluster side Resource
Manager
(YARN)
3. Launch Application
Job Manager Container
Dispatcher
1. Generate
JobGraph

15. 15
Workflow1
Job Execution(Job Mode)
cluster side
Dispatcher
JobGraph
Store
JobRegistry
2. Start
Job Manager
Job Manager
1. Recover
JobGraph

16. 16
Workflow1
Job Execution(Job Mode)
cluster side
Dispatcher
JobGraph
Store
JobRegistry
Job Manager
3. Check
JobStatus
4. Finished
By Others
4. Set JobStatus
RUNNING

17. 17
Workflow1
Job Execution(Job Mode)
cluster side
Dispatcher
JobGraph
Store
JobRegistry
Job Manager
5. Set JobStatus
DONE
6. Finished

18. 18
Workflow1
Job Execution(Job Mode)
cluster side
Dispatcher
JobGraph
Store
JobRegistry
7. Remove
JobGraph
8. Clear
JobStatus

19. 19
• Background
• Reliability
• Isolation
• Unification
Outline

20. Reliability
20
2

21. 21
Reliability2
What is reliability?
Submited exactly once!
Executed exactly once!

22. 22
Reliability2
Failure Case: Job Manager failed to start in time
22
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
Job Manager
Starting
Job Manager

23. 23
Reliability2
Failure Case: Job Manager crashed after job finished
23
cluster side
Dispatcher
JobGraph
Store
JobRegistry
Job Manager
Set JobStatus
DONE

24. 24
Reliability2
Failure Case: Job Manager crashed after job finished
24
cluster side
Dispatcher
JobGraph
Store
JobRegistry
Job Manager
Crash
Finished
Clear
JobStatus

25. 25
Reliability2
Failure Case: Standby starts after JobStatus cleared
25
cluster side
Dispatcher
JobGraph
Store
JobRegistry
Check
JobStatus

26. 26
Reliability2
Failure Case: Standby starts after JobStatus cleared
26
cluster side
Dispatcher
JobGraph
Store
JobRegistry
Job Manager
Start
Job Manager Execute twice!
(FLINK-11813)

27. 27
Reliability2
What to define job submitted & executed
27
JobRegistry
JobGraphStore
(internal) JobManagerRunnerFuture Handled in Dispatcher
Modified by Dispatcher
Modified by Dispatcher & Job Manager
Many factors to define job status!
Goal: Achieve atomic job submission and execution

28. 28
Reliability2
How to achieve atomic job submission
28
• What is the sign of a successful submission?
JobGraph persisted in JobGraphStore

29. 29
Reliability2
Successful job submission defined by JobGraph
29
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
4. Persist
JobGraph

30. 30
Reliability2
How to achieve atomic job submission
30
• What is the sign of a successful submission?
JobGraph persisted in JobGraphStore
• What is the sign of a successful execution?
DONE in JobRegistry(only Dispatcher modifies it; not cleared after written)

31. 31
Reliability2
Successful job execution defined by DONE
31
Dispatcher
JobGraph
Store
JobRegistry
Job Manager
6. Set JobStatus
RUNNING
cluster side
7.Start
Job Manager

32. 32
Reliability2
Successful job execution defined by DONE
32
Dispatcher
JobGraph
Store
JobRegistry
Job Manager
8. Finished
9. Set JobStatus
DONE
cluster side

33. 33
• Background
• Reliability
• Isolation
• Unification
Outline

34. Isolation
34
3

35. 35
Isolation3
Where is JobGraph generated?
35

36. 36
Isolation3
Job Submission(Session Mode)
Client
Dispatcher
client side cluster side
JobGraph
Store
JobRegistry
Generate
JobGraph

37. 37
Isolation3
Cluster Deployment(Job Mode)
Deployer
Resource
Manager
(YARN)
client side cluster side
Generate
JobGraph

38. 38
Isolation3
Where is JobGraph generated?
38
Compilation always happens at client side!

39. 39
Isolation3
Drawbacks
39
Require user code dependencies
Execute arbitrary user code
Bottleneck at client
Goal: Achieve isolated job compilation

40. 40
Isolation3
Compilation on cluster side
Application Mode

41. 41
Isolation3
Application Mode: Package User Program
41
ShipFiles
MainClass
Parallelism
SavepointSettings
Arguments
...
Deployer
client side
CommandLine
Interface
Pass Program
Metadata

42. 42
Isolation3
Application Mode: Package User Program
42
Deployer
client side
CommandLine
Interface
MainClass
Parallelism
SavepointSettings
Arguments
...
Package
User Program

43. 43
Isolation3
Application Mode: Prepare Dependencies
43
Deployer
Distributed Storage
client side cluster side
Upload
Dependencies

44. 44
Isolation3
Application Mode: Prepare Dependencies
44
Deployer
Distributed Storage
client side cluster side
Return URIs
ShipFiles

45. Distributed Storage
45
Isolation3
Application Mode: Prepare Dependencies
45
Deployer
client side cluster side
Configure
Dependencies
URIs
SharedLibs
ShipFiles

46. 46
Isolation
client side cluster side Resource
Manager
(YARN)
3
Application Mode: Cluster Deployment
Deployer

47. 47
Isolation
client side cluster side Resource
Manager
(YARN)
3
Application Mode: Cluster Deployment
Launch Application
Job Manager Container
Deployer

48. 48
Isolation3
Application Mode: Download Dependencies
48
cluster side
Job Manager ContainerDistributed Storage
SharedLibs
ShipFiles
Localize(YARN)
InitContainer(K8s)

49. Job Manager Container
49
Isolation3
Application Mode: Execute in Isolation
49
cluster side
DispatcherClient
Submit JobGraph
Generate
JobGraph

50. 50
Isolation3
Deployment: Recap
50
Session Mode Job Mode
User Program execute as is abort on execute
Client Perspective cluster deployed &
job submitted
cluster deployed
with bundled job
High Availability configured
JobGraphStore
special
JobGraphStore
Application Mode
execute as is
cluster deployed &
(local) job submitted
configured
JobGraphStore

51. 51
• Background
• Reliability
• Isolation
• Unification
Outline

52. Unification
52
4

53. 53
Unification4
Client Interface: The Problem
Flink does not provide public & stable client interface
Various customized submission requires programmatic interface
Goal: Expose unified layered client interface!

54. 54
Unification4
Client Interface: Three-Layers Architecture
Deployment: Deployer
Cluster: Cluster Client
Job: Job Client
Target: Kubernetes, YARN, etc.
Target: Flink Cluster
Target: Job Manager

55. 55
Unification4
Client Interface: Deployer
Target: YARN, Kubernetes
Deployer
Resource
Manager
Submit Application:
ApplicationDescriptor → ApplicationID
Shutdown Application:
ApplicationID → Acknowledge
Get Application Status:
ApplicationID → ApplicationStatus

56. 56
Unification4
Client Interface: Cluster Client
Submit Job:
JobGraph → JobID
Shutdown Cluster:
() → Acknowledge
Get Cluster Overview:
() → ClusterOverview
Target: Flink Cluster
Cluster Client Dispatcher

57. 57
Unification4
Client Interface: Job Client
Get Job Status:
() → JobStatus
Trigger Savepoint
Savepoint Path → Savepoint Path
Cancel Job:
() → Acknowledge
Target: (Flink) Job Manager
Job Client Job Manager

58. 58
Unification4
User Case: Platform
Deployer
Resource
Manager
Submit Session Application
Session Mode
Cluster Client
Submit Multiple JobGraphs
Dispatcher

59. 59
Unification4
User Case: Platform
Deployer
Resource
Manager
Submit Multiple Applications
Application Mode

60. 60
Unification4
User Case: End-User
Deployer
Resource
Manager
Submit Application
env.executeAsync(): JobClient
Cluster Client
Submit JobGraph
Dispatcher
(locally in Application Mode)
(remotely in Session Mode)

61. 61
Unification4
User Case: End-User
env.executeAsync(): JobClient
Job Client Job Manager
1. Monitor Job Status
2. Trigger Savepoint
3. Manipulate Job

62. Thanks!