A brief overview of several methods to optimize data movement in distributed multi-query processing applications.

1. Minimizing Communication Cost in Distributed Multi-query Processing Jian Li, Amol Deshpande, Samir Khuller Department of Computer Science, University of Maryland Presented by: Luis Galárraga Saarland University July 7th, 2010

3. Problem formulation

5. Tree topology

6. Arbitrary graph topologies

10. Tree topology

11. Arbitrary graph topologies

14. Publish-subscribe systems

15. Distributed stream processing applications Common need: Minimize data movement!!

17. Justification

19. Tree topology

20. Arbitrary graph topologies

24. No restrictions on data size.

25. Query plans are part of the input.

26. Intermediate results sizes are known.

28. Topology, undirected weighted graph

29. Assignment of relations to nodes in the topology.

31. Each query comes with a plan in the form of a directed tree. Destination node Data sources involved Data size S i S j S i x S j w z(S i ) z(S j ) z(S i x S j )

32. More formally Given the topology graph G c and a set of trees representing the query plans, our goal is to find a data movement plan that minimizes the total communication cost incurred while executing the queries.

33. Problem formulation Topology G c Queries (10) S 1 S 2 S 1 x S 2 C (10) (7) S 4 S 1 x S 2 x S 4 (5) (100) (100) S 2 S 6 S 2 x S 6 D (10) (5) S 2 S 5 S 2 x S 5 B (10) (6) (8) B A C D E F S 2 S 1 S 3 S 4 S 6 S 5 (10) (10) (100) (8) (100)

37. Problem formulation

39. Arbitrary graph topologies

42. For general topologies, aproximation algorithms are known.

48. Hyperedges can group any number of vertices.

50. Find a flow or mapping of maximum value:

51. Max-flow/Min-cut 3 / 3 2 / 3 2 / 2 3 / 3 0 / 2 1 / 4 2 / 2 3 / 3 Flow Capacity

53. The maximum value of an s-t flow is equal to the minimum capacity of an s-t cut.

54. Max-flow/Min-cut 3 / 3 2 / 3 3 / 3 0 / 2 1 / 4 2 / 2 3 / 3 2 / 2 Flow Capacity

57. Max-flow/Min-cut in hypergraphs w

58. w Max-flow/Min-cut in hypergraphs

60. We want to find a partition (S, T) on V such that:

63. Problem formulation

71. Tree topology – Step 2 – Single query G c u G c v S 1 S 2 C S 4 (10) S 1 x S 2 (10) (7) S 1 x S 2 x S 4 (5) (100) H D B A C D E F S 2 S 1 S 3 S 4 S 6 S 5 (10) (10) (100) (8) (100) C C D C

75. Tree topology – Step 2 – Multiple queries S 1 S 1 x S 2 C (10) (7) S 4 S 1 x S 2 x S 4 (5) (100) (100) S 6 S 2 x S 6 D (5) S 2 S 5 S 2 x S 5 B (10) (6) (8)

79. Then add the edges according to this rule (for every edge in G c ):

82. Problem formulation

88. A query-overlap graph H is a graph where the vertices correspond to the set of data items and each edge corresponds to a pair query.

89. The pairs problem

92. Otherwise to Connected Facility Location Problem Isn't this problem NP-Hard? Yes, but good approximation algorithms do exist!!

94. Output: Bought edges (Steiner cost) Rented edges (Connection cost)

95. Connected Facility Location Problem Demands, D Facilities, F Rented edges

99. Problem formulation

101. Tree topology

104. Multi-query approach compared with isolated optimization per query.

105. Normalized with the cost of the naïve approach.

106. Identical data size vs tri-modal distribution

107. Random query overlap.

108. Tree topology

109. Arbitrary topology – Pairs problem

112. Tree topology

113. Arbitrary graph topologies

118. Thank you!!