Presentation from Matt Wood, Amazon Web Services #dataconf More at http://event.gigaom.com/structuredata/

1. THE MISSING MANUAL FOR DATA SCIENCE: REMIX.
RESUSE. REPRODUCE
SPEAKER: Matt Wood
Principal Data Scientist
Amazon Web Services
Monday, April 1, 13

2. The Missing Manual:
Reproduce, Reuse, Remix
Dr. Matt Wood
matthew@amazon.com
@mza
Monday, April 1, 13

3. Monday, April 1, 13

4. Hello.
Monday, April 1, 13

5. Monday, April 1, 13

6. Data.
Monday, April 1, 13

7. Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Monday, April 1, 13

8. Monday, April 1, 13

9. Generation challenge.
Monday, April 1, 13

10. Amazing data generators: cell phones tracking cholera in Haiti
Linus Bengtsson et al. PLoS Medicine, 2011
Monday, April 1, 13

11. Amazing data generators: social networks tracking influenza
You Are What You Tweet: Analyzing Twitter for Public Health. M. J. Paul and M. Dredze, 2011
Monday, April 1, 13

12. Amazing data generators: web app logs targeting advertising
500% return on ad spend
Monday, April 1, 13

13. Monday, April 1, 13

14. Monday, April 1, 13

15. Chromosome 11 : ACTN3 : rs1815739
Monday, April 1, 13

16. Chromosome X : rs6625163
Monday, April 1, 13

17. Chromosome 19 : FUT2 : rs601338
Monday, April 1, 13

18. Chromosome 2 : rs10427255
Monday, April 1, 13

19. Chromosome 10 : rs7903146
TYPE II
Monday, April 1, 13

20. Chromosome 15 : rs2472297
+0.25
Monday, April 1, 13

21. Monday, April 1, 13

22. Generation challenge.
Monday, April 1, 13

23. Generation challenge.
X
Monday, April 1, 13

24. Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Monday, April 1, 13

25. Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Monday, April 1, 13

26. Monday, April 1, 13

27. Utility computing.
Monday, April 1, 13

28. Monday, April 1, 13

29. Monday, April 1, 13

30. Monday, April 1, 13

31. Remove constraints.
Monday, April 1, 13

32. Monday, April 1, 13

33. Analytics challenge.
Monday, April 1, 13

34. Analytics challenge.
X
Monday, April 1, 13

35. Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Monday, April 1, 13

36. Monday, April 1, 13

37. Beautiful, unique.
Monday, April 1, 13

38. Monday, April 1, 13

39. Impossible to recreate.
Monday, April 1, 13

40. Monday, April 1, 13

41. Snowflake Data Science
Monday, April 1, 13

42. Monday, April 1, 13

43. Reproducibility.
Monday, April 1, 13

44. Monday, April 1, 13

45. Reproducibility scales data science.
Monday, April 1, 13

46. Monday, April 1, 13

47. Reproduce. Reuse. Remix.
Monday, April 1, 13

48. Monday, April 1, 13

49. Value++
Monday, April 1, 13

50. Monday, April 1, 13

51. Monday, April 1, 13

52. How do we get from
here to there?
IPLESF
5 PR INC O
REPRO DUCIBILITY
Monday, April 1, 13

53. PRINCIPLESF
5
O
REPRODUCIBILITY
Monday, April 1, 13

54. PRINCIPLESF
5
O
REPRODUCIBILITY
1. Data has Gravity
Monday, April 1, 13

55. Monday, April 1, 13

56. Increasingly large data collections.
Monday, April 1, 13

57. Monday, April 1, 13

58. Challenging to obtain and manage.
Monday, April 1, 13

59. Monday, April 1, 13

60. Expensive to experiment.
Monday, April 1, 13

61. Monday, April 1, 13

62. Large barrier to reproducibility.
Monday, April 1, 13

63. Monday, April 1, 13

64. Move data to the users.
Monday, April 1, 13

65. Move data to the users.
X
Monday, April 1, 13

66. Monday, April 1, 13

67. Move tools to the data.
Monday, April 1, 13

68. Monday, April 1, 13

69. Place data where it can be
consumed by tools.
Monday, April 1, 13

70. Monday, April 1, 13

71. Place tools where they
can access data.
Monday, April 1, 13

72. Monday, April 1, 13

73. Monday, April 1, 13

74. Monday, April 1, 13

75. Monday, April 1, 13

76. Monday, April 1, 13

77. More data,
more users,
more uses,
more locations
Monday, April 1, 13

78. Monday, April 1, 13

79. Cost
Monday, April 1, 13

80. Monday, April 1, 13

81. Force multiplier.
Monday, April 1, 13

82. Monday, April 1, 13

83. Cost and complexity
kill reproducibility.
Monday, April 1, 13

84. PRINCIPLESF
5
O
REPRODUCIBILITY
Monday, April 1, 13

85. PRINCIPLESF
5
O
REPRODUCIBILITY
2. Ease of use is a prerequisite
Monday, April 1, 13

86. http://headrush.typepad.com/creating_passionate_users/2005/10/getting_users_p.html
Monday, April 1, 13

87. Monday, April 1, 13

88. Help overcome the suck threshold.
Monday, April 1, 13

89. Monday, April 1, 13

90. Easy to embrace and extend.
Monday, April 1, 13

91. Monday, April 1, 13

92. Choose the right abstraction for the user.
Monday, April 1, 13

93. Monday, April 1, 13

94. $ ec2-run-instances
Monday, April 1, 13

95. Monday, April 1, 13

96. $ starcluster start
Monday, April 1, 13

97. Monday, April 1, 13

98. Monday, April 1, 13

99. Package and automate.
Monday, April 1, 13

100. Monday, April 1, 13

101. Expert-as-a-service.
Monday, April 1, 13

102. Monday, April 1, 13

103. Monday, April 1, 13

104. 1000 Genomes
Project
Cloud BioLinux
Monday, April 1, 13

105. Monday, April 1, 13

106. 1000 Genomes
Project + your
genomic data
Illumina Basespace
Monday, April 1, 13

107. Cassandra Aegisthus Hadoop, Hive, Pig
Amazon S3
Legacy data warehousing
http://www.youtube.com/watch?v=oGcZ7WVx6EI
Monday, April 1, 13

108. Sting
Microstrategy
R
Cassandra Aegisthus Hadoop, Hive, Pig
Amazon S3
Legacy data warehousing
http://www.youtube.com/watch?v=oGcZ7WVx6EI
Monday, April 1, 13

109. Monday, April 1, 13

110. PRINCIPLESF
5
O
REPRODUCIBILITY
Monday, April 1, 13

111. PRINCIPLESF
5
O
REPRODUCIBILITY
3. Reuse is as important as reproduction
Monday, April 1, 13

112. Seven Deadly sins of Bioinformatics: http://www.slideshare.net/dullhunk/the-seven-deadly-sins-of-bioinformatics
Monday, April 1, 13

113. Seven Deadly sins of Bioinformatics: http://www.slideshare.net/dullhunk/the-seven-deadly-sins-of-bioinformatics
Monday, April 1, 13

114. Monday, April 1, 13

115. Data scientists are hackers.
Monday, April 1, 13

116. Monday, April 1, 13

117. They have their own way of working.
Monday, April 1, 13

118. Monday, April 1, 13

119. Beware the Big Red Button.
Monday, April 1, 13

120. Monday, April 1, 13

121. Fire and forget reproduction
is a good first step, but limits
longer term value.
Monday, April 1, 13

122. Monday, April 1, 13

123. Monolithic, one-stop-shop.
Monday, April 1, 13

124. Monday, April 1, 13

125. Work well for intended purpose.
Monday, April 1, 13

126. Monday, April 1, 13

127. Challenging to install,
dependency heavy.
Monday, April 1, 13

128. Monday, April 1, 13

129. Difficult to grok.
Monday, April 1, 13

130. Monday, April 1, 13

131. Data scientists are hackers:
embrace it.
Monday, April 1, 13

132. Monday, April 1, 13

133. Small things. Loosely coupled.
Monday, April 1, 13

134. Monday, April 1, 13

135. Easier to grok, reuse and integrate.
Monday, April 1, 13

136. Monday, April 1, 13

137. Lower barrier to entry.
Monday, April 1, 13

138. PRINCIPLESF
5
O
REPRODUCIBILITY
Monday, April 1, 13

139. PRINCIPLESF
5
O
REPRODUCIBILITY
4. Build for collaboration
Monday, April 1, 13

140. Monday, April 1, 13

141. Workflows are memes.
Monday, April 1, 13

142. Monday, April 1, 13

143. Reproduction is just the first step.
Monday, April 1, 13

144. Monday, April 1, 13

145. Bill of materials:
code, data, configuration, infrastructure.
Monday, April 1, 13

146. Monday, April 1, 13

147. Full definition for reproduction.
Monday, April 1, 13

148. Monday, April 1, 13

149. Utility computing provides a
playground for data science.
Monday, April 1, 13

150. Code + AMI +
custom datasets + public datasets +
databases + compute + result data
Monday, April 1, 13

151. Code + AMI +
custom datasets + public datasets +
databases + compute + result data
Monday, April 1, 13

152. Code + AMI +
custom datasets + public datasets +
databases + compute + result data
Monday, April 1, 13

153. Code + AMI +
custom datasets + public datasets +
databases + compute + result data
Monday, April 1, 13

154. PRINCIPLESF
5
O
REPRODUCIBILITY
Monday, April 1, 13

155. PRINCIPLESF
5
O
REPRODUCIBILITY
5. Provenance is a first class object
Monday, April 1, 13

156. Monday, April 1, 13

157. Versioning becomes really important.
Monday, April 1, 13

158. Monday, April 1, 13

159. Especially in an active community.
Monday, April 1, 13

160. Monday, April 1, 13

161. Doubly so with loosely coupled tools.
Monday, April 1, 13

162. Monday, April 1, 13

163. Provenance metadata is a
first class entity.
Monday, April 1, 13

164. Monday, April 1, 13

165. Distributed provenance.
Monday, April 1, 13

166. IPLESF
5
PRI NC O
Y
RODUCIBILIT
REP
Monday, April 1, 13

167. IPLESF
5 PRI NC O
Y
RODUCIBILIT
REP
1. Data has gravity
2. Ease of use is a prerequisite
3. Reuse is as important as reproduction
4. Build for collaboration
5. Provenance is a first class object
Monday, April 1, 13

168. Monday, April 1, 13

169. Thank you
matthew@amazon.com
aws.amazon.com
@mza
Monday, April 1, 13

170. Monday, April 1, 13