The document provides an overview of various Python machine learning libraries and tools, including Orange, MDP, PyMC, PyML, hcluster, NLTK, mlpy, LIBSVM, PyEvolve, FANN, Theano, PyBrain, Shogun, ffnet. For each library, it gives information on the homepage, dependencies, installation/source options, key developers and details. It also discusses machine learning and Python in general terms, noting the large amount of activity but also varying documentation quality and lack of packaging.

1. Python and Machine Learning by Semen A. Trygubenko

3. Spam filtering

4. Player ranking in online games

5. Vehicle stability systems

6. Computer vision (barcode, fingerprint and number plate readers)

7. Optimisation

9. The act of remembering: data / experience

10. Generalisation, similarity and new inputs

11. Learning and flexibility: adaptation

14. Semi-supervised learning

15. Evolutionary learning, fitness

16. Unsupervised

19. Graphical models

20. Artificial neural networks

21. Kernel methods

22. Dimensionality reduction

23. Optimisation

24. Interdisciplinary & as a branch of CS

25. ML is a vaaast field

27. FANN

28. Orange

29. PyMC

30. PyML

31. LIBSVM

32. PyBrain

34. Shogun toolbox

35. Theano

36. mlpy

37. Elefant

38. Bayes Blocks

39. Monte Python

41. Pycplex

42. pymorph

43. We need total coverage ...

47. (3) Establish where the project is in its lifecycle

48. (4) Demo stuff that I think is cool

53. Visual programming framework + scriptable environment

54. Orange canvas: widgets and channels

55. Communication via tokens

56. Implements sampling, filtering, scaling, discretisation, regression, classification, clustering, scoring functions, SVMs

58. AI lab, University of Ljubljana, Slovenia

59. (L)GPL

60. C++ components accessible from Python

61. 98K LOCs in Python, 66K LOCs in C++

62. 700 revisions in 2010, 14 developers

63. 10000 commits since 2003 when migrated from cvs

64. Orange # Homepage: http://www.ailab.si/orange/ # Dependencies: Python, PythonWin, NumPy, PyQt, PyQwt... .PHONY: install install: repo sudo apt-get install orange-svn python-orange-svn .PHONY: source source: svn checkout http://www.ailab.si/svn/orange/trunk Orange .PHONY: repos repos: sudo echo -e "deb http://www.ailab.si/orange/debian lenny maindeb-src http://www.ailab.si/orange/debian lenny maindeb http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu lucid maindeb-src http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu lucid main"|sudo tee -a /etc/apt/sources.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5BB92C09DB82666C && sudo apt-get update sudo apt-get install python2.5

65. import orange data = orange.ExampleTable("in") print data.domain for item in data: print item orange.saveTabDelimited("out.tab",data) I/O

66. selection = orange.MakeRandomIndices2(data, 0.03) sample = data.select(selection, 0) sample.save("sample.tab") Basic statistics

67. Learners and classifiers classifier = orange.BayesLearner(data) print classifier(newItem)

70. Supervised and unsupervised

71. PCA and ICA, Slow feature analysis, LLE, restricted Boltzmann machine

72. Hierarchical networks

73. Support vector machines (through wrappers around Shogun and LIBSVM)

74. Concept of nodes and flows; basic are parallelized

76. Institute for Theoretical Biology of the Humboldt University, Berlin

77. Schuppner, Wilbert and Zito are active, 150 commits this year

78. LGPL, version 2.6 released in May this year

79. 30K LOCs of Python, 3K of comments in the code!

80. MDP # Homepage: http://mdp-toolkit.sourceforge.net/ # Dependencies: NumPy, SciPy .PHONY: install install: sudo apt-get install python-mdp .PHONY: source source: git clone git://mdp-toolkit.git.sourceforge.net/gitroot/mdp-toolkit/mdp-toolkit

81. Dimensionality reduction import mdp x = mdp.numx_rand.random((100, 25)) # 25 variables, 100 observations y = mdp.pca(x) z = mdp.fastica(x, dtype='float32')

82. Nodes: training and usage n = mdp.nodes.PCANode() n.train(x) # learn PC of x n.stop_training() print n1.output_dim print n1.explained_variance z = n.execute(y) # project y on PC learned in training

83. Inverting the flow print n.is_invertible() # true for PCA node print n.inverse(z) # get y back

84. Flows: feed-forward architectures flow = mdp.Flow([ mdp.nodes.PCANode(output_dim=5) ,mdp.nodes.CuBICANode() ]) flow = mdp.nodes.PCANode(output_dim=5)+ mdp.nodes.CuBICANode() flow.train(x)

86. FlowNode = a node with internal structure

87. Switchboard = DS for arbitrary routing

88. Hierarchical networks a = mdp.nodes.PCANode() b = mdp.nodes.SFANode() c = mdp.hinet.FlowNode( mdp.Flow([a,b]) ) layer = mdp.hinet.Layer([n, c])

90. Can fit Bayesian statistical models with MCMC

91. Large suite of statistical distributions

92. Building blocks to construct probability models: stochastic, deterministic and potential

93. Python for scalar vars, numpy and hand-optimised Fortran code for arrays

95. USA

96. MIT lic

97. 3 developer, 100 commits this year, all in Jan

98. Version: 2.1 released in Jan

99. 26K LOCs of Python, 10K of Fortran and 3K LOCs of C

100. PyMC # Homepage: http://code.google.com/p/pymc/ # Dependencies: NumPy,SciPy,matplotlib,pytables,pydot,nose .PHONY: install # grr... 2.1alpha is missing gp_submodel.py and step_methods.py ##dir:=pymc-2.1alpha dir:=pymc install: source ##${dir}.tar.gz sudo apt-get install python-dev gcc gfortran ## tar -xzvf $< cd ${dir} && python setup.py config_fc --fcompiler gnu95 build && sudo python setup.py install ##pymc-2.1alpha.tar.gz: ## wget ##http://pymc.googlecode.com/files/pymc-2.1alpha.tar.gz .PHONY: source source: svn checkout http://pymc.googlecode.com/svn/trunk/ pymc

101. Model module, part 1 x = numpy.array([-.86,-.3,-.05,.73]) alpha = pymc.Normal('alpha',mu=0,tau=.01) beta = pymc.Normal('beta',mu=0,tau=.01) @pymc.deterministic def theta(a=alpha,b=beta): return pymc.invlogit(a+b*x)

102. Model module, part 2 # Binomial likelihood for data d = pymc.Binomial( 'd' ,n=numpy.ones(4,dtype=int)*5 ,p=theta ,value=numpy.array([0.,1.,3.,5.]) ,observed=True )

103. Sampling from a distribution import pymc import model S=pymc.MCMC(model,db='pickle') S.sample(iter=10000,burn=5000,thin=2)

105. Interactive OO framework, focuses on SVM and other kernel methods

106. tools for feature selection, model selection, syntax for combining classifiers

107. methods for assessing classifier performance

109. Depends: matplotlib, numpy

110. 1 developer (+ research group?) + NSF funding

111. 0.7.4.1 out last month

112. 18K LOCs of Python, 8K LOCs of C++

113. LGPL

114. PyML # Homepage: http://pyml.sourceforge.net/ # Dependencies: numpy, matplotlib .PHONY: install install: PyML-0.7.4.1.tar.gz tar -xzvf $< cd PyML-0.7.4.1 && python setup.py build && sudo python setup.py install PyML-0.7.4.1.tar.gz: wget https://sourceforge.net/projects/pyml/files/PyML-0.7.4.1.tar.gz/download

116. 11K LOCs of Python, 8K of C and 500 of C++

117. 2500 lines of documentation

118. 1 developer, active 2007-2008

119. New BSD lic

120. 8 routines for agglomerative clustering, routines that compute statistics on hierarchies, 21 distance functions

121. hcluster # Homepage: http://code.google.com/p/scipy-cluster/ .PHONY: install install: sudo apt-get install python-hcluster .PHONY: source source: svn checkout http://scipy-cluster.googlecode.com/svn/trunk/ hcluster

122. NLTK # Homepage: http://www.nltk.org/ # Dependencies: PyYAML .PHONY: install install: sudo apt-get install python-nltk .PHONY: source source: sudo apt-get source python-nltk

123. mlpy # Homepage: https://mlpy.fbk.eu/ # Dependencies: NumPy, GSL .PHONY: install install: sudo echo &quot;deb http://ppa.launchpad.net/davide-albanese/ppa/ubuntu jaunty main&quot;|sudo tee -a /etc/apt/sources.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4AEC3064 sudo apt-get update sudo apt-get install python-mlpy .PHONY: source source: svn co https://mlpy.fbk.eu/svn/ mlpy

124. LIBSVM # Homepage: http://www.csie.ntu.edu.tw/~cjlin/libsvm/ .PHONY: install install: sudo apt-get install libsvm2 python-libsvm .PHONY: source source: sudo apt-get source libsvm2

125. PyEvolve # Homepage: http://pyevolve.sourceforge.net/ # Dependencies: matplotlib .PHONY: install install: wget http://downloads.sourceforge.net/pyevolve/Pyevolve-0.5-py2.6.egg sudo easy_install Pyevolve-0.5-py2.6.egg .PHONY: source source: svn co https://pyevolve.svn.sourceforge.net/svnroot/pyevolve

126. FANN library # Homepage: http://www.nltk.org/ # Dependencies: PyYAML .PHONY: install install: sudo apt-get install libfann1-dev .PHONY: source source: sudo apt-get source libfann1 .PHONY: pythonBindings: pythonBindings: # need to build from source!

127. Theano # Homepage: http://deeplearning.net/software/theano # Dependencies: blas .PHONY: install install: sudo apt-get install libblas-dev cd Theano && sudo python setup.py install .PHONY: source source: hg clone http://hg.assembla.com/theano Theano

128. PyBrain # Homepage: http://www.pybrain.org/pages/download .PHONY: install install: cd pybrain && sudo python setup.py install .PHONY: source source: git clone git://github.com/pybrain/pybrain.git

129. Shogun # Homepage: http://www.shogun-toolbox.org .PHONY: install install: sudo apt-get install shogun-doc-en shogun-cmdline shogun-python .PHONY: source source: svn checkout https://svn.tuebingen.mpg.de:/shogun/trunk shogun

130. ffnet # Homepage: http://ffnet.sourceforge.net .PHONY: install install: cd ffnet && sudo python setup.py install .PHONY: source source: svn co https://ffnet.svn.sourceforge.net/svnroot/ffnet/trunk ffnet

132. Not everything is packaged for your favourite OS

133. But: it is pretty easy to get going / help

134. Documentation is of varying quality, heterogeneous

135. But: there's source

136. Good libraries are usually centred on research groups / backed up by universities

138. http://videolectures.net

139. http://archive.ics.uci.edu/ml/

141. by Stephen Marsland, 2009

142. Artificial Intelligence: A Modern Approach

143. by Stuart Russell and Peter Norvig, 2009

144. Programming Collective Intelligence

145. by Toby Segaran, 2007

146. Information Theory, Inference, and Learning Algorithms

147. by David McKay, 2005

148. The End

150. using a component from a library we achieve reduction in size

151. components maybe whatever form of abstraction we may invent: subroutines, generators of subroutines, generators of generators of sub ...

153. libraries allow us to “compress the incompressible” by taking the advantage of the commonality exhibited by programs within a problem domain

154. problem domain entropy is defined as a measure of program diversity

156. “better tools and culture can have only marginal impact on reuse rates if the domain is inherently resistant to reuse”

157. Machine learning is a high-entropy domain

159. “You are a creative genius. Your creative genius is so accomplished that it appears, to you and to others, as effortless. Yet it far outstrips the most valiant efforts of today's fastest supercomputers. To invoke it, you need only open your eyes.” – Donald D. Hoffman