The document discusses machine learning in Rust using the Leaf and Collenchyma frameworks. Collenchyma provides portable, parallel computation capabilities and memory management across devices using SharedTensors. It defines frameworks like Rust and backends like CPU/GPU. Plugins extend backends with operations. Leaf builds neural networks from layers that use Collenchyma operations, and features solvers for training using backpropagation and stochastic gradient descent. An example classifies MNIST digits using a single-layer perceptron on GPU with these frameworks.
Machine Learning in Rust with Leaf and Collenchyma
1. MACHINE LEARNING IN RUST
WITH LEAF AND COLLENCHYMA
RUST TALK BERLIN | Feb.2016
> @autumn_eng
2. “In machine learning, we seek methods by which the computer will
come up with its own program based on examples that we provide.”
MACHINE LEARNING ~ PROGRAMMING BY EXAMPLE
[1]: http://www.cs.princeton.edu/courses/archive/spr08/cos511/scribe_notes/0204.pdf
3. AREAS OF ML
[1]: http://cs.jhu.edu/~jason/tutorials/ml-simplex
DOMAIN KNOWLEDGE LOTS OF DATA PROOF. TECHNIQUES
BAYESIAN DEEP CLASSICAL
INSIGHTFUL MODEL FITTING MODEL ANALYZABLE MODEL
4. > DEEP LEARNING.
TON OF DATA + SMART ALGOS + PARALLEL COMP.
[1]: http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/
6. > DEEP NEURAL NETWORK
UNIVERSAL FUNCTION APPROXIMATOR,
REPRESENTING HIRARCHICAL STRUCTURES IN
LEARNED DATA
KEY CONCEPTS | ALGORITHMS
[1]: http://cs231n.github.io/neural-networks-1/
9. KEY CONCEPTS | PARALLEL COMPUTATION
> MULTI CORE DEVICES (GPUs)
HIGH-DIMENSIONAL MATHEMATICAL OPERATIONS
CAN BE EXECUTED MORE EFFICIENTLY ON
SPECIAL-PURPOSE CHIPS LIKE GPUS OR FPGAS.
13. A COLLENCHYMA-FRAMEWORK DESCRIBES A
COMPUTATIONAL LANGUAGE LIKE RUST, OPENCL,
CUDA.
A COLLENCHYMA-BACKEND DESCRIBES A SINGLE
COMPUTATIONAL-CAPABLE HARDWARE (CPU, GPU,
FPGA) WHICH IS ADDRESSABLE BY A FRAMEWORK.
COLLENCHYMA | PORTABLE COMPUTATION
14. /// Defines a Framework.
pub trait IFramework {
/// Initializes a new Framework.
///
/// Loads all the available hardwares
fn new() -> Self where Self: Sized;
/// Initializes a new Device from the provided hardwares.
fn new_device(&self, &[Self::H]) -> Result<DeviceType, Error>;
}
/// Defines the main and highest struct of Collenchyma.
pub struct Backend<F: IFramework> {
framework: Box<F>,
device: DeviceType,
}
COLLENCHYMA | PORTABLE COMPUTATION
15. // Initialize a CUDA Backend.
let backend = Backend::<Cuda>::default().unwrap();
// Initialize a CUDA Backend - the explicit way
let framework = Cuda::new();
let hardwares = framework.hardwares();
let backend_config = BackendConfig::new(framework, hardwares[0]);
let backend = Backend::new(backend_config).unwrap();
COLLENCHYMA | PORTABLE COMPUTATION
16. COLLENCHYMA-PLUGINS ARE CRATES, WHICH EXTEND
THE COLLENCHYMA BACKEND WITH FRAMEWORK
AGNOSTIC, MATHEMATICAL OPERATIONS E.G. BLAS
OPERATIONS
COLLENCHYMA | OPERATIONS
[1]: https://github.com/autumnai/collenchyma-blas [2]: https://github.com/autumnai/collenchyma-nn
17. /// Provides the functionality for a backend to support Neural Network related operations.
pub trait NN<F: Float> {
/// Initializes the Plugin.
fn init_nn();
/// Returns the device on which the Plugin operations will run.
fn device(&self) -> &DeviceType;
}
/// Provides the functionality for a Backend to support Sigmoid operations.
pub trait Sigmoid<F: Float> : NN<F> {
fn sigmoid(&self, x: &mut SharedTensor<F>, result: &mut SharedTensor<F>) -> Result<(), ::co::error::Error>;
fn sigmoid_plain(&self, x: &SharedTensor<F>, result: &mut SharedTensor<F>) -> Result<(), ::co::error::Error>;
fn sigmoid_grad(&self, x: &mut SharedTensor<F>, x_diff: &mut SharedTensor<F>) -> Result<(), ::co::error::Error>;
fn sigmoid_grad_plain(&self, x: &SharedTensor<F>, x_diff: &SharedTensor<F>) -> Result<(), ::co::error::Error>;
}
COLLENCHYMA | OPERATIONS
[1]: https://github.com/autumnai/collenchyma-nn/blob/master/src/plugin.rs
22. fn main() {
// Initialize a CUDA Backend.
let backend = Backend::<Cuda>::default().unwrap();
// Initialize two SharedTensors.
let mut x = SharedTensor::<f32>::new(backend.device(), &(1, 1, 3)).unwrap();
let mut result = SharedTensor::<f32>::new(backend.device(), &(1, 1, 3)).unwrap();
// Fill `x` with some data.
let payload: &[f32] = &::std::iter::repeat(1f32).take(x.capacity()).collect::<Vec<f32>>();
let native = Backend::<Native>::default().unwrap();
x.add_device(native.device()).unwrap();
write_to_memory(x.get_mut(native.device()).unwrap(), payload); // Write to native host memory.
x.sync(backend.device()).unwrap(); // Sync the data to the CUDA device.
// Run the sigmoid operation, provided by the NN Plugin, on your CUDA enabled GPU.
backend.sigmoid(&mut x, &mut result).unwrap();
// See the result.
result.add_device(native.device()).unwrap(); // Add native host memory
result.sync(native.device()).unwrap(); // Sync the result to host memory.
println!("{:?}", result.get(native.device()).unwrap().as_native().unwrap().as_slice::<f32>());
}
COLLENCHYMA | BRINGING IT TOGETHER
[1]: https://github.com/autumnai/collenchyma#examples
26. CONNECTED LAYERS FORM A NEURAL NETWORK
BACKPROPAGATION VIA TRAITS
=> GRADIENT CALCULATION EASILY SWAPABLE
LEAF | Layers
27. T = DATATYPE OF SHAREDTENSOR (e.g. f32).
B = BACKEND
/// A Layer that can compute the gradient with respect to its input.
pub trait ComputeInputGradient<T, B: IBackend> {
/// Compute gradients with respect to the inputs and write them into `input_gradients`.
fn compute_input_gradient(&self,
backend: &B,
weights_data: &[&SharedTensor<T>],
output_data: &[&SharedTensor<T>],
output_gradients: &[&SharedTensor<T>],
input_data: &[&SharedTensor<T>],
input_gradients: &mut [&mut SharedTensor<T>]);
}
LEAF | Layers
28. POOLING LAYER
EXECUTE ONE OPERATION
OVER A REGION OF THE INPUT
LEAF | Layers
[image]: http://cs231n.github.io/convolutional-networks/
30. STOCHASTIC GRADIENT DESCENT
REQUIRES BACKPROPAGATION
MIGHT NOT FIND THE GLOBAL
MINIMUM BUT WORKS FOR
A HUGH NUBMER OF WEIGHTS
LEAF | Solvers
[image]: https://commons.wikimedia.org/wiki/File:Extrema_example_original.svg by Wikipedia user KSmrq
![“In machine learning, we seek methods by which the computer will
come up with its own program based on examples that we provide.”
MACHINE LEARNING ~ PROGRAMMING BY EXAMPLE
[1]: http://www.cs.princeton.edu/courses/archive/spr08/cos511/scribe_notes/0204.pdf](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)