This talk goes over a concepts of data mining and data analysis using open source tools, mainly Python and R with interesting libraries and the tools I have used and currently use at Engine Yard.

1. The Artful Business
of Data Mining
Computational Statistics
with Open Source Tool
Wednesday 20 March 13

2. David Coallier
@davidcoallier
Wednesday 20 March 13

3. Data Scientist
At Engine Yard (.com)
Wednesday 20 March 13

4. Find Data
Wednesday 20 March 13

5. Clean Data
Wednesday 20 March 13

6. Analyse Data?
Wednesday 20 March 13

7. Analyse Data
Wednesday 20 March 13

8. Question Data
Wednesday 20 March 13

9. Report Findings
Wednesday 20 March 13

10. Data Scientist
Wednesday 20 March 13

11. Data Janitor
Wednesday 20 March 13

12. Actual
Tasks
Wednesday 20 March 13

13. “If your model
is elegant, it’s
probably wrong”
Wednesday 20 March 13

14. “The Times
they are
a-Changing”
— Bob Dylan
Wednesday 20 March 13

15. Python & R
Wednesday 20 March 13

16. SciPy
http://www.scipy.org
Wednesday 20 March 13

17. scipy.stats
Wednesday 20 March 13

18. scipy.stats
Descriptive Statistics
Wednesday 20 March 13

19. from scipy.stats
import describe
s = [1,2,1,3,4,5]
print describe(s)
Wednesday 20 March 13

20. scipy.stats
Probability Distributions
Wednesday 20 March 13

21. Example
Poisson Distribution
Wednesday 20 March 13

22. λ e
k −k
f (k; λ ) =
k!
for k >= 0
Wednesday 20 March 13

23. import scipy.stats.poisson
p = poisson.pmf([1,2,3,4,1,2,3], 2)
Wednesday 20 March 13

24. print p.mean()
print p.sum()
...
Wednesday 20 March 13

25. NumPy
http://www.numpy.org/
Wednesday 20 March 13

26. NumPy
Linear Algebra
Wednesday 20 March 13

27. ⎛ 1 0 ⎞
⎜ 0 1 ⎟
⎝ ⎠
Wednesday 20 March 13

28. import numpy as np
x = np.array([ [1, 0], [0, 1] ])
vec, val = np.linalg.eig(x)
np.linalg.eigvals(x)
Wednesday 20 March 13

29. >>> np.linalg.eig(x)
(
array([ 1., 1.]),
array([
[ 1., 0.],
[ 0., 1.]
])
)
Wednesday 20 March 13

30. Matplotlib
Python Plotting
Wednesday 20 March 13

31. statsmodels
Advanced Statistics Modeling
Wednesday 20 March 13

32. NLTK
Natural Language Tool Kit
Wednesday 20 March 13

33. scikit-learn
Machine Learning
Wednesday 20 March 13

34. from sklearn import tree
X = [[0, 0], [1, 1]]
Y = [0, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, Y)
clf.predict([[2., 2.]])
>>> array([1])
Wednesday 20 March 13

35. PyBrain
... Machine Learning
Wednesday 20 March 13

36. PyMC
Bayesian Inference
Wednesday 20 March 13

37. Pattern
Web Mining for Python
Wednesday 20 March 13

38. NetworkX
Study Networks
Wednesday 20 March 13

39. MILK
MOAR machine LEARNING!
Wednesday 20 March 13

40. Pandas
easy-to-use
data structures
Wednesday 20 March 13

41. from pandas import *
x = DataFrame([
{"age": 26},
{"age": 19},
{"age": 21},
{"age": 18}
])
print x[x['age'] > 20].count()
print x[x['age'] > 20].mean()
Wednesday 20 March 13

42. R
Wednesday 20 March 13

43. RStudio
The IDE
Wednesday 20 March 13

44. lubridate
and zoo
Dealing with Dates...
Wednesday 20 March 13

45. yy/mm/dd mm/dd/yy
YYYY-mm-dd HH:MM:ss TZ
yy-mm-dd 1363784094.513425
yy/mm different timezone
Wednesday 20 March 13

46. reshape2
Reshape your Data
Wednesday 20 March 13

47. ggplot2
Visualise your Data
Wednesday 20 March 13

48. RCurl, RJSONIO
Find more Data
Wednesday 20 March 13

49. HMisc
Miscellaneous useful functions
Wednesday 20 March 13

50. forecast
Can you guess?
Wednesday 20 March 13

51. garch
And ruGarch
Wednesday 20 March 13

52. quantmod
Statistical Financial Trading
Wednesday 20 March 13

53. xts
Extensible Time Series
Wednesday 20 March 13

54. igraph
Study Networks
Wednesday 20 March 13

55. maptools
Read & View Maps
Wednesday 20 March 13

56. map('state', region = c(row.names(USArrests)), col=cm.colors(16, 1)[floor(USArrests$Rape/max(USArrests$Rape)*28)], fill=T)
Wednesday 20 March 13

57. Sto
rage
Wednesday 20 March 13

58. Oppose
“big” Data
Wednesday 20 March 13

59. “Learn how
to sample”
Wednesday 20 March 13

60. Experim
ents
Wednesday 20 March 13

61. What Do
You Want to Answer?
Wednesday 20 March 13

62. Understand
Your Audience
Wednesday 20 March 13

63. Scientific
Reporting
Wednesday 20 March 13

64. Busy-ness
Time is money
Wednesday 20 March 13

65. Public
Visualisation
Wednesday 20 March 13

66. Best
Visualisation,
Bad
Data
Wednesday 20 March 13

67. Best
Forecasting
models...
Bad
Visualisation
Wednesday 20 March 13

68. Wednesday 20 March 13

69. Wednesday 20 March 13

70. Sean
chaí
Wednesday 20 March 13

71. Wednesday 20 March 13

72. Feel
it
Wednesday 20 March 13

73. Wednesday 20 March 13

74. Wednesday 20 March 13

75. Wednesday 20 March 13

76. “Don’t be scared of
bar charts.”
Wednesday 20 March 13

77. Mathematical
Statistics
Engineering
Business
Economics
Curiosity
Wednesday 20 March 13

78. davidcoallier.github.com
@davidcoallier on Twitter
Wednesday 20 March 13