SlideShare uma empresa Scribd logo

Jupyter, A Platform for Data Science at Scale

•
•
Matthias Bussonnier
Matthias Bussonnier

This document discusses Jupyter, an open-source tool for interactive data science and scientific computing. Jupyter allows for interactive exploration, development, and communication through code, equations, visualizations and narrative text. It supports over 50 programming languages and has found widespread adoption in academia and industry for individual and collaborative work across the entire workflow of a scientific idea from data collection to publication. The document outlines Jupyter's history and architecture, ecosystem of related projects, and future development plans to enhance collaboration and software engineering capabilities.

Tecnologia
1 de 45
Baixar para ler offline
1 A tool for datascience at scale. Matthias Bussonnier (UC Berkeley mbussonnier@berkeley.edu) Slides examples on GitHub: https://github.com/Carreau/talks/tree/master/labtech-2015
Jupyter A bit of History The Notebook Application The document format & Publication Multi-User & scaling The Ecosystem 2
The Lifecycle of a Scientific Idea 1. Individual exploratory work 2. Collaborative development 3. Parallel production runs (HPC, cloud, …) 4. Publication & communication (reproducibly!) 5. Education 6. Goto 1 3
The Lifecycle of a Scientific Idea 1. Individual exploratory work – Matlab command line 2. Collaborative development – email scripts back and forth ? 3. Parallel production runs (HPC, cloud, …) – rewrite Fortran/MPI 4. Publication & communication (reproducibly!) – Copy Past in PPT 5. Education – Specific tools 6. Goto 1 4
Can we have a single tool that cover all the lifecycle of a scientific idea, from data collection to publication ? 5
A bit of History Fernando Perez, 2001, CU Boulder (instead of writing a physics dissertation): Python can replace the collection of bash, perl, C/C++ Script. But the Python REPL can be better. 6
NOVEMBER 2001: "JUST AN AFTERNOON HACK" 259 Line Python script. (https://gist.github.com/fperez/1579699) sys.ps1 -> In [N]. sys.displayhook -> Out[N], caches results. Plotting, Numeric, etc. 2014 (OPENHUB STATS) 19,279 commits 442 contributors Total Lines: 187,326 Number of Languages : 7 (JS, CSS, HTML, …) 7
Improve over the terminal ❖ The REPL as a network protocol ❖ Kernels ❖ execute code ❖ Clients ❖ Read input ❖ Present output Simple abstractions enable rich, sophisticated clients 8
❖ Rich web client ❖ Text & math ❖ Code ❖ Results ❖ Share, reproduce. 2011: The IPython Notebook 9
The Team (people that spend a noticeable amount of time on the project, subjective of course) Fernando Perez (UC Berkeley LBL) Brian Granger (CalPoly) Oberon Lopez (summer student) Cameron Oelsen (summer student) SimonVurens (summer student) Ryan Morshed (summer student) Min Ragan-Kelley (Simula) Thomas Kluyver (UK) Matthias Bussonnier (UC Berkeley) Jon Frederic (Cal Poly) Jess Hamrick (UC Berkeley) Kyle Kelley (Rackspace) Jason Grout (Bloomberg) Sylvain Corlay (Bloomberg) Kester Tong (Google) Nicholas Bollweg Will Whitney (MIT) Damián Avila (Continuum) Steven Silvester (Continuum) Chris Colbert (Continuum) David Willmer (Continuum) Peter Parente (IBM) Dan Gisolfi (IBM) Gino Bustelo (IBM) All 400+ GitHub contributors. Bold:Working full time on IPython/Jupyter, underline: Contribute to Jupyter/IPython with corporate agreement 10
Funding 11
Jupyter vs IPython Network protocol for interactive computing Clients for protocol Console Qt Console Notebook Notebook file format & tools (nbconvert…) JupyterHub Nbviewer NbGrader Tmpnb … Interactive Python shell at the terminal Kernel for this protocol in Python Tools for Cross-Language integration Tools for Interactive Parallel computing The “reference” kernel for Jupyter 12
Why ? Don’t reinvent the wheel: reimplement 1 piece, get the rest for free. You don’t like the frontend, write a new one for Python get 50+ languages that work out of the box with it. (https://github.com/ ipython/ipython/wiki/IPython-kernels-for-other-languages) You don’t like a language, write your own kernel, get all the IDEs, conversion tools. Etc.. 13
The Notebook Try it on https://try.jupyter.org Demo Notebook app, also have a terminal, text editor, increasing number of plugins, and of course support 50 languages. 14
The notebook Web Application, that allow code to produce web-rich representation (images, sound, video, math, …) The Browser, Server, and kernel(s) can be on separate machines. The default application to edit `.ipynb` files. `.ipynb` file are JSON based files embeding input and output, so which can be read & converted without a running kernel. 15
The Notebook Fileformat (`.ipynb`) 16
NbViewer Zero-install reading of notebooks Just share a URL nbviewer.org Under the hood: get raw URL and convert to HTML on the fly. Sharing: git push, or dropbox sync. 17
Nbviewer on GitHub Since May GitHub renders Notebooks Powered by `nbconvert`, the library that deals with `.ipynb` -> * Over 200,000 notebooks on GitHub 18
What content as notebook ? http://www.nature.com/ismej/journal/v7/n3/full/ ismej2012123a.html http://qiime.org/home_static/nih-cloud-apr2012 Papers with code as AMI/VMs 19
Blogs Jake van der Plas @ UW http://blogs.scientificamerican.com/ sa-visual/2014/09/16/visualizing-4- dimensional-asteroids 20
Course, MOOCS 21
Books By Cameron Davidson-PilonBy Matthew Russell By José Unpingco You can download and execute the books locally. 22
Check The Gallery https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks23
Replicating, simpler for readers http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 What if you didn’t had to install anything ? Docker Container Just for you. 24
Replicating, simpler for authors 1. Fixed set of notebooks/dependencies 1. Tmpnb, (https://lambdaops.com/ipythonjupyter-tmpnb-debuts/) 2. CodeNeuro, (http://codeneuro.org/) 2. Build on demand 1. Everware, (https://github.com/everware) 2. Binder, (MyBinder.org) 25
https://github.com/binder-project by Jeremy Freeman (Demo mybinder.org ) 26
The networking architecture (single user) Https/ websocket ZMQ 27
MULTI-USER Jupyter Notebook is Single-User by design. Multi-User enable through JupyterHub - Allow Better scalability - resources monitoring/user - Per-user configuration/version of IPython - Better integration with existing infrastructure 28
Hub Https/websocket proxy Auth & Security Hub 29
Everything* is a plugin • Auth: • Unix PAM (default), OAuth, LDAP… (ie, not yet another thing to manage) • Spawner - Start a single user server for each user. • Localhost, Rackspace, EC2, Docker • Meant for sysadmin, • Deployments relatively involved • Recent software stack (Node.js/Python3) Hub 30
Quick Demo 31
JupyterHub in education Jess Hamrick @ Cal K. Kelley Rackspace M. Ragan-Kelley Simula B. Granger Cal Poly https://developer.rackspace.com/blog/deploying-jupyterhub-for-education ❖ Computationally intensive course, ~220 students ❖ Fully hosted environment, zero-install ❖ Integration with autograding. 32
Deploy at larger scale this fall at UC Berkeley - Data Science 101 - Everyone with CalNet account. New Jupyter in Education Mailing List: https://groups.google.com/forum/#!forum/jupyter-education 33
Ecosystem 34
Non-Notebook projects IDE/Frontends: Atom Hydrogen EIN VIM IPython Rodeo PyCharm MicrosoftVisual Studio 35
Non-Notebook projects RISE (interactive slideshow) runipy (notebooks are report templates) ipymd (store notebook as markdown) NbGrader (grade assignments) Jupyter-Drive (store notebooks on G-drive) pgcontents (store notebooks on PostGres) urth (declarative widget, + dashboard from notebook) 36
Google CoLaboratory Kayur Patel, Kester Tong, Mark Sanders, Corinna Cortes @ Google Matt Turk @ NCSA/UIUC Currently being merge into Jupyter itself. 37
O’Reilly: authoring and delivering executable books Atlas, ipymd and Thebe beta.oreilly.com 38
The Future 39
40
Future work Interactive Computing Notebooks as interactive applications Modular, reusable UI/UX Software engineering with notebooks Computational Narratives nbconvert Element filtering Documentation Collaboration Real time collaboration JupyterHub Sustainability People Events 41
Future work Component and tiled-layout are oft requested feature. Collaboration with Continuum Analytics. Plan on adding panels for Text editor, output, variable inspectors, debuggers, … Discussion with Microsoft PTVS team for “debugger protocol” 42
43
Hiring At UC Berkeley Two new postdocs Project manager Web developer, tech writer (short contracts) One administrative assistant. At Cal Poly Three software engineers (one already hired) One designer One administrative assistant. 44
Time for questions ? Thanks 45

Recomendados

Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...
Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...
Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...Edureka!
 
Multi-armed Bandits
Multi-armed BanditsMulti-armed Bandits
Multi-armed BanditsDongmin Lee
 
Machine Learning with Decision trees
Machine Learning with Decision treesMachine Learning with Decision trees
Machine Learning with Decision treesKnoldus Inc.
 
Clustering
ClusteringClustering
ClusteringM Rizwan Aqeel
 
Machine Learning vs. Deep Learning
Machine Learning vs. Deep LearningMachine Learning vs. Deep Learning
Machine Learning vs. Deep LearningBelatrix Software
 
Deep neural networks
Deep neural networksDeep neural networks
Deep neural networksSi Haem
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep LearningOswald Campesato
 
Intro to deep learning
Intro to deep learning Intro to deep learning
Intro to deep learning David Voyles
 

Recomendados

Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...
Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...
Scikit Learn Tutorial | Machine Learning with Python | Python for Data Scienc...Edureka!
 
Multi-armed Bandits
Multi-armed BanditsMulti-armed Bandits
Multi-armed BanditsDongmin Lee
 
Machine Learning with Decision trees
Machine Learning with Decision treesMachine Learning with Decision trees
Machine Learning with Decision treesKnoldus Inc.
 
Clustering
ClusteringClustering
ClusteringM Rizwan Aqeel
 
Machine Learning vs. Deep Learning
Machine Learning vs. Deep LearningMachine Learning vs. Deep Learning
Machine Learning vs. Deep LearningBelatrix Software
 
Deep neural networks
Deep neural networksDeep neural networks
Deep neural networksSi Haem
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep LearningOswald Campesato
 
Intro to deep learning
Intro to deep learning Intro to deep learning
Intro to deep learning David Voyles
 
Random forest
Random forestRandom forest
Random forestMusa Hawamdah
 
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...Simplilearn
 
Decision trees for machine learning
Decision trees for machine learningDecision trees for machine learning
Decision trees for machine learningAmr BARAKAT
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent methodProf. Neeta Awasthy
 
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Preferred Networks
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual IntroductionLukas Masuch
 
Fine tune and deploy Hugging Face NLP models
Fine tune and deploy Hugging Face NLP modelsFine tune and deploy Hugging Face NLP models
Fine tune and deploy Hugging Face NLP modelsOVHcloud
 
Propositional And First-Order Logic
Propositional And First-Order LogicPropositional And First-Order Logic
Propositional And First-Order Logicankush_kumar
 
Introduction to Chainer 11 may,2018
Introduction to Chainer 11 may,2018Introduction to Chainer 11 may,2018
Introduction to Chainer 11 may,2018Preferred Networks
 
Neural Networks and Deep Learning Basics
Neural Networks and Deep Learning BasicsNeural Networks and Deep Learning Basics
Neural Networks and Deep Learning BasicsJon Lederman
 
Classification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsClassification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsMd. Main Uddin Rony
 
Optimizers
OptimizersOptimizers
OptimizersIl Gu Yi
 
Fine tuning large LMs
Fine tuning large LMsFine tuning large LMs
Fine tuning large LMsSylvainGugger
 
And then there were ... Large Language Models
And then there were ... Large Language ModelsAnd then there were ... Large Language Models
And then there were ... Large Language ModelsLeon Dohmen
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkKnoldus Inc.
 
TensorFlow and Keras: An Overview
TensorFlow and Keras: An OverviewTensorFlow and Keras: An Overview
TensorFlow and Keras: An OverviewPoo Kuan Hoong
 
General introduction to AI ML DL DS
General introduction to AI ML DL DSGeneral introduction to AI ML DL DS
General introduction to AI ML DL DSRoopesh Kohad
 
Markov decision process
Markov decision processMarkov decision process
Markov decision processHamed Abdi
 
CNN Attention Networks
CNN Attention NetworksCNN Attention Networks
CNN Attention NetworksTaeoh Kim
 
Bert
BertBert
BertAbdallah Bashir
 
LSESU a Taste of R Language Workshop
LSESU a Taste of R Language WorkshopLSESU a Taste of R Language Workshop
LSESU a Taste of R Language WorkshopKorkrid Akepanidtaworn
 
Multi-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorMulti-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorLevi Waldron
 

Mais conteúdo relacionado

Mais procurados

K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...Simplilearn
 
Decision trees for machine learning
Decision trees for machine learningDecision trees for machine learning
Decision trees for machine learningAmr BARAKAT
 
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Preferred Networks
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual IntroductionLukas Masuch
 
Fine tune and deploy Hugging Face NLP models
Fine tune and deploy Hugging Face NLP modelsFine tune and deploy Hugging Face NLP models
Fine tune and deploy Hugging Face NLP modelsOVHcloud
 
Propositional And First-Order Logic
Propositional And First-Order LogicPropositional And First-Order Logic
Propositional And First-Order Logicankush_kumar
 
Introduction to Chainer 11 may,2018
Introduction to Chainer 11 may,2018Introduction to Chainer 11 may,2018
Introduction to Chainer 11 may,2018Preferred Networks
 
Neural Networks and Deep Learning Basics
Neural Networks and Deep Learning BasicsNeural Networks and Deep Learning Basics
Neural Networks and Deep Learning BasicsJon Lederman
 
Classification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsClassification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsMd. Main Uddin Rony
 
Optimizers
OptimizersOptimizers
OptimizersIl Gu Yi
 
Fine tuning large LMs
Fine tuning large LMsFine tuning large LMs
Fine tuning large LMsSylvainGugger
 
And then there were ... Large Language Models
And then there were ... Large Language ModelsAnd then there were ... Large Language Models
And then there were ... Large Language ModelsLeon Dohmen
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkKnoldus Inc.
 
TensorFlow and Keras: An Overview
TensorFlow and Keras: An OverviewTensorFlow and Keras: An Overview
TensorFlow and Keras: An OverviewPoo Kuan Hoong
 
General introduction to AI ML DL DS
General introduction to AI ML DL DSGeneral introduction to AI ML DL DS
General introduction to AI ML DL DSRoopesh Kohad
 
Markov decision process
Markov decision processMarkov decision process
Markov decision processHamed Abdi
 
CNN Attention Networks
CNN Attention NetworksCNN Attention Networks
CNN Attention NetworksTaeoh Kim
 

Mais procurados (20)

Random forest
Random forestRandom forest
Random forest
 
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
 
Decision trees for machine learning
Decision trees for machine learningDecision trees for machine learning
Decision trees for machine learning
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent method
 
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual Introduction
 
Fine tune and deploy Hugging Face NLP models
Fine tune and deploy Hugging Face NLP modelsFine tune and deploy Hugging Face NLP models
Fine tune and deploy Hugging Face NLP models
 
Propositional And First-Order Logic
Propositional And First-Order LogicPropositional And First-Order Logic
Propositional And First-Order Logic
 
Introduction to Chainer 11 may,2018
Introduction to Chainer 11 may,2018Introduction to Chainer 11 may,2018
Introduction to Chainer 11 may,2018
 
Neural Networks and Deep Learning Basics
Neural Networks and Deep Learning BasicsNeural Networks and Deep Learning Basics
Neural Networks and Deep Learning Basics
 
Classification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsClassification Based Machine Learning Algorithms
Classification Based Machine Learning Algorithms
 
Optimizers
OptimizersOptimizers
Optimizers
 
Fine tuning large LMs
Fine tuning large LMsFine tuning large LMs
Fine tuning large LMs
 
And then there were ... Large Language Models
And then there were ... Large Language ModelsAnd then there were ... Large Language Models
And then there were ... Large Language Models
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural Network
 
TensorFlow and Keras: An Overview
TensorFlow and Keras: An OverviewTensorFlow and Keras: An Overview
TensorFlow and Keras: An Overview
 
General introduction to AI ML DL DS
General introduction to AI ML DL DSGeneral introduction to AI ML DL DS
General introduction to AI ML DL DS
 
Markov decision process
Markov decision processMarkov decision process
Markov decision process
 
CNN Attention Networks
CNN Attention NetworksCNN Attention Networks
CNN Attention Networks
 
Bert
BertBert
Bert
 

Destaque

Multi-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorMulti-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorLevi Waldron
 
Analytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-PlattformAnalytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-PlattformRising Media Ltd.
 
Computational Approaches to Systems Biology
Computational Approaches to Systems BiologyComputational Approaches to Systems Biology
Computational Approaches to Systems BiologyMike Hucka
 
Computational Biology and Bioinformatics
Computational Biology and BioinformaticsComputational Biology and Bioinformatics
Computational Biology and BioinformaticsSharif Shuvo
 
Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102remko caprio
 
MongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open SourceMongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open SourceB1 Systems GmbH
 
The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4turingfan
 
Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen Harald Erb
 
Systems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems levelSystems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems levelLars Juhl Jensen
 
DNA Information and Creation (PDF)
DNA Information and Creation (PDF)DNA Information and Creation (PDF)
DNA Information and Creation (PDF)Hans Rudolf Tremp
 
IBM - Big Value from Big Data
IBM - Big Value from Big DataIBM - Big Value from Big Data
IBM - Big Value from Big DataWilfried Hoge
 
Systems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological systemSystems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological systemLars Juhl Jensen
 
Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?IoT User Group Hamburg
 
Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)Annex Publishers
 
Tutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer WorkshopTutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer WorkshopVivek Krishnakumar
 
System biology and its tools
System biology and its toolsSystem biology and its tools
System biology and its toolsGaurav Diwakar
 

Destaque (20)

LSESU a Taste of R Language Workshop
LSESU a Taste of R Language WorkshopLSESU a Taste of R Language Workshop
LSESU a Taste of R Language Workshop
 
Multi-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorMulti-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/Bioconductor
 
Analytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-PlattformAnalytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
 
Computational Approaches to Systems Biology
Computational Approaches to Systems BiologyComputational Approaches to Systems Biology
Computational Approaches to Systems Biology
 
Computational Biology and Bioinformatics
Computational Biology and BioinformaticsComputational Biology and Bioinformatics
Computational Biology and Bioinformatics
 
Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102
 
COMPUTATIONAL BIOLOGY
COMPUTATIONAL BIOLOGYCOMPUTATIONAL BIOLOGY
COMPUTATIONAL BIOLOGY
 
MongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open SourceMongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open Source
 
The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4
 
Donald Knuth
Donald KnuthDonald Knuth
Donald Knuth
 
Zwischen Browser, Code & Photoshop - aus dem Leben eines Webworkers
Zwischen Browser, Code & Photoshop - aus dem Leben eines WebworkersZwischen Browser, Code & Photoshop - aus dem Leben eines Webworkers
Zwischen Browser, Code & Photoshop - aus dem Leben eines Webworkers
 
Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen
 
Systems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems levelSystems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems level
 
DNA Information and Creation (PDF)
DNA Information and Creation (PDF)DNA Information and Creation (PDF)
DNA Information and Creation (PDF)
 
IBM - Big Value from Big Data
IBM - Big Value from Big DataIBM - Big Value from Big Data
IBM - Big Value from Big Data
 
Systems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological systemSystems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological system
 
Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?
 
Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)
 
Tutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer WorkshopTutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer Workshop
 
System biology and its tools
System biology and its toolsSystem biology and its tools
System biology and its tools
 

Semelhante a Jupyter, A Platform for Data Science at Scale

Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...Domino Data Lab
 
Using Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsUsing Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsLuciano Resende
 
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)PyData
 
Reproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and ZenodoReproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and ZenodoEGI Federation
 
Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroidsJose Enrique Ruiz
 
Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1inside-BigData.com
 
Behold the Power of Python
Behold the Power of PythonBehold the Power of Python
Behold the Power of PythonSarah Dutkiewicz
 
PyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsPyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsHenry Schreiner
 
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...Puppet
 
Reproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and AndurilReproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and AndurilChristian Frech
 
Python 101 For The Net Developer
Python 101 For The Net DeveloperPython 101 For The Net Developer
Python 101 For The Net DeveloperSarah Dutkiewicz
 
1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptx1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptxFEG
 
Python 101 for the .NET Developer
Python 101 for the .NET DeveloperPython 101 for the .NET Developer
Python 101 for the .NET DeveloperSarah Dutkiewicz
 
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Luciano Resende
 
Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.Carlos Miguel Ferreira
 
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinalProf. Wim Van Criekinge
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?Julien Pivotto
 
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayStrata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayLuciano Resende
 
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Codemotion
 

Semelhante a Jupyter, A Platform for Data Science at Scale (20)

Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...
 
Using Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsUsing Elyra for COVID-19 Analytics
Using Elyra for COVID-19 Analytics
 
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
 
Reproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and ZenodoReproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and Zenodo
 
Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroids
 
Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1
 
Behold the Power of Python
Behold the Power of PythonBehold the Power of Python
Behold the Power of Python
 
PyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsPyCon2022 - Building Python Extensions
PyCon2022 - Building Python Extensions
 
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
 
Reproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and AndurilReproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and Anduril
 
Python 101 For The Net Developer
Python 101 For The Net DeveloperPython 101 For The Net Developer
Python 101 For The Net Developer
 
1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptx1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptx
 
Python 101 for the .NET Developer
Python 101 for the .NET DeveloperPython 101 for the .NET Developer
Python 101 for the .NET Developer
 
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
 
Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.
 
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?
 
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayStrata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
 
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
 
London level39
London level39London level39
London level39
 

Último

🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 

Último (20)

🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 

Jupyter, A Platform for Data Science at Scale

  • 1. 1 A tool for datascience at scale. Matthias Bussonnier (UC Berkeley mbussonnier@berkeley.edu) Slides examples on GitHub: https://github.com/Carreau/talks/tree/master/labtech-2015
  • 2. Jupyter A bit of History The Notebook Application The document format & Publication Multi-User & scaling The Ecosystem 2
  • 3. The Lifecycle of a Scientific Idea 1. Individual exploratory work 2. Collaborative development 3. Parallel production runs (HPC, cloud, …) 4. Publication & communication (reproducibly!) 5. Education 6. Goto 1 3
  • 4. The Lifecycle of a Scientific Idea 1. Individual exploratory work – Matlab command line 2. Collaborative development – email scripts back and forth ? 3. Parallel production runs (HPC, cloud, …) – rewrite Fortran/MPI 4. Publication & communication (reproducibly!) – Copy Past in PPT 5. Education – Specific tools 6. Goto 1 4
  • 5. Can we have a single tool that cover all the lifecycle of a scientific idea, from data collection to publication ? 5
  • 6. A bit of History Fernando Perez, 2001, CU Boulder (instead of writing a physics dissertation): Python can replace the collection of bash, perl, C/C++ Script. But the Python REPL can be better. 6
  • 7. NOVEMBER 2001: "JUST AN AFTERNOON HACK" 259 Line Python script. (https://gist.github.com/fperez/1579699) sys.ps1 -> In [N]. sys.displayhook -> Out[N], caches results. Plotting, Numeric, etc. 2014 (OPENHUB STATS) 19,279 commits 442 contributors Total Lines: 187,326 Number of Languages : 7 (JS, CSS, HTML, …) 7
  • 8. Improve over the terminal ❖ The REPL as a network protocol ❖ Kernels ❖ execute code ❖ Clients ❖ Read input ❖ Present output Simple abstractions enable rich, sophisticated clients 8
  • 9. ❖ Rich web client ❖ Text & math ❖ Code ❖ Results ❖ Share, reproduce. 2011: The IPython Notebook 9
  • 10. The Team (people that spend a noticeable amount of time on the project, subjective of course) Fernando Perez (UC Berkeley LBL) Brian Granger (CalPoly) Oberon Lopez (summer student) Cameron Oelsen (summer student) SimonVurens (summer student) Ryan Morshed (summer student) Min Ragan-Kelley (Simula) Thomas Kluyver (UK) Matthias Bussonnier (UC Berkeley) Jon Frederic (Cal Poly) Jess Hamrick (UC Berkeley) Kyle Kelley (Rackspace) Jason Grout (Bloomberg) Sylvain Corlay (Bloomberg) Kester Tong (Google) Nicholas Bollweg Will Whitney (MIT) Damián Avila (Continuum) Steven Silvester (Continuum) Chris Colbert (Continuum) David Willmer (Continuum) Peter Parente (IBM) Dan Gisolfi (IBM) Gino Bustelo (IBM) All 400+ GitHub contributors. Bold:Working full time on IPython/Jupyter, underline: Contribute to Jupyter/IPython with corporate agreement 10
  • 12. Jupyter vs IPython Network protocol for interactive computing Clients for protocol Console Qt Console Notebook Notebook file format & tools (nbconvert…) JupyterHub Nbviewer NbGrader Tmpnb … Interactive Python shell at the terminal Kernel for this protocol in Python Tools for Cross-Language integration Tools for Interactive Parallel computing The “reference” kernel for Jupyter 12
  • 13. Why ? Don’t reinvent the wheel: reimplement 1 piece, get the rest for free. You don’t like the frontend, write a new one for Python get 50+ languages that work out of the box with it. (https://github.com/ ipython/ipython/wiki/IPython-kernels-for-other-languages) You don’t like a language, write your own kernel, get all the IDEs, conversion tools. Etc.. 13
  • 14. The Notebook Try it on https://try.jupyter.org Demo Notebook app, also have a terminal, text editor, increasing number of plugins, and of course support 50 languages. 14
  • 15. The notebook Web Application, that allow code to produce web-rich representation (images, sound, video, math, …) The Browser, Server, and kernel(s) can be on separate machines. The default application to edit `.ipynb` files. `.ipynb` file are JSON based files embeding input and output, so which can be read & converted without a running kernel. 15
  • 16. The Notebook Fileformat (`.ipynb`) 16
  • 17. NbViewer Zero-install reading of notebooks Just share a URL nbviewer.org Under the hood: get raw URL and convert to HTML on the fly. Sharing: git push, or dropbox sync. 17
  • 18. Nbviewer on GitHub Since May GitHub renders Notebooks Powered by `nbconvert`, the library that deals with `.ipynb` -> * Over 200,000 notebooks on GitHub 18
  • 19. What content as notebook ? http://www.nature.com/ismej/journal/v7/n3/full/ ismej2012123a.html http://qiime.org/home_static/nih-cloud-apr2012 Papers with code as AMI/VMs 19
  • 20. Blogs Jake van der Plas @ UW http://blogs.scientificamerican.com/ sa-visual/2014/09/16/visualizing-4- dimensional-asteroids 20
  • 22. Books By Cameron Davidson-PilonBy Matthew Russell By José Unpingco You can download and execute the books locally. 22
  • 24. Replicating, simpler for readers http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 What if you didn’t had to install anything ? Docker Container Just for you. 24
  • 25. Replicating, simpler for authors 1. Fixed set of notebooks/dependencies 1. Tmpnb, (https://lambdaops.com/ipythonjupyter-tmpnb-debuts/) 2. CodeNeuro, (http://codeneuro.org/) 2. Build on demand 1. Everware, (https://github.com/everware) 2. Binder, (MyBinder.org) 25
  • 27. The networking architecture (single user) Https/ websocket ZMQ 27
  • 28. MULTI-USER Jupyter Notebook is Single-User by design. Multi-User enable through JupyterHub - Allow Better scalability - resources monitoring/user - Per-user configuration/version of IPython - Better integration with existing infrastructure 28
  • 30. Everything* is a plugin • Auth: • Unix PAM (default), OAuth, LDAP… (ie, not yet another thing to manage) • Spawner - Start a single user server for each user. • Localhost, Rackspace, EC2, Docker • Meant for sysadmin, • Deployments relatively involved • Recent software stack (Node.js/Python3) Hub 30
  • 32. JupyterHub in education Jess Hamrick @ Cal K. Kelley Rackspace M. Ragan-Kelley Simula B. Granger Cal Poly https://developer.rackspace.com/blog/deploying-jupyterhub-for-education ❖ Computationally intensive course, ~220 students ❖ Fully hosted environment, zero-install ❖ Integration with autograding. 32
  • 33. Deploy at larger scale this fall at UC Berkeley - Data Science 101 - Everyone with CalNet account. New Jupyter in Education Mailing List: https://groups.google.com/forum/#!forum/jupyter-education 33
  • 36. Non-Notebook projects RISE (interactive slideshow) runipy (notebooks are report templates) ipymd (store notebook as markdown) NbGrader (grade assignments) Jupyter-Drive (store notebooks on G-drive) pgcontents (store notebooks on PostGres) urth (declarative widget, + dashboard from notebook) 36
  • 37. Google CoLaboratory Kayur Patel, Kester Tong, Mark Sanders, Corinna Cortes @ Google Matt Turk @ NCSA/UIUC Currently being merge into Jupyter itself. 37
  • 38. O’Reilly: authoring and delivering executable books Atlas, ipymd and Thebe beta.oreilly.com 38
  • 40. 40
  • 41. Future work Interactive Computing Notebooks as interactive applications Modular, reusable UI/UX Software engineering with notebooks Computational Narratives nbconvert Element filtering Documentation Collaboration Real time collaboration JupyterHub Sustainability People Events 41
  • 42. Future work Component and tiled-layout are oft requested feature. Collaboration with Continuum Analytics. Plan on adding panels for Text editor, output, variable inspectors, debuggers, … Discussion with Microsoft PTVS team for “debugger protocol” 42
  • 43. 43
  • 44. Hiring At UC Berkeley Two new postdocs Project manager Web developer, tech writer (short contracts) One administrative assistant. At Cal Poly Three software engineers (one already hired) One designer One administrative assistant. 44
  • 45. Time for questions ? Thanks 45