How to build collaborative environment using data science notebook.

1. What makes Data driven
environments more efficient and how to
build a data science toolchain around
Notebook technologies
Creator of Apache Zeppelin
Co-Founder, CTO
Moon soo Lee
moon@zepl.com

2. #GDSC 2018
Who am I
A true believer that data science notebook changes how
people collaborate
Creator of Apache Zeppelin
Co-founder
https://github.com/Leemoonsoo

3. #GDSC 2018
It was 2013, really wanted to have
interactive analytics interface for .

4. #GDSC 2018
Started an opensource project -
Zeppelin http://zeppelin-project.org/
data science notebook.Became an project in 2016.
http://zeppelin.apache.org

5. #GDSC 2018
Iterations REPL interface (2012)
Editor / Result interface (2013)
Notebook interface (2014)

6. #GDSC 2018
Pilot to Production in 1 day
Hey, take a look
I need an update every morning!

7. #GDSC 2018
More notebook consumers than producers

8. #GDSC 2018
At the same time
Opensource project receiving contributions like
Authentication
Access control

9. #GDSC 2018
Realized that notebook is a great collaboration tool
Why notebook?

10. #GDSC 2018
Notebook is
- Interactive
- Flexible
- Visualized
- Inline description
- Contain a story
- Shareable

11. #GDSC 2018
How to build collaborative environment
with notebook technology
Data sharing
Multi-user
environment
Notebook sharing

12. #GDSC 2018
Data scientist
Data engineer Data Analyst
Marketing
SW
engineer
Sales
Executive
You
Notebook Sharing

13. #GDSC 2018
You’re using only half of its
potential if not sharing

14. #GDSC 2018
Github
nbviewer
Zeppelin
Airbnb/knowledge-repo
Commercial services for notebook sharing
VCS
Open
source
Service

15. #GDSC 2018
Github
● Store notebook in github
● Versioning
● Github provides .ipynb viewer
● Fork / pull request / merge
● Private / Public / Team / Org
● Hard to apply Notebook level ACL
● Not easy for Non-engineers

16. #GDSC 2018
nbviewer
● Publishing notebook
● Share notebook by
sharing link
● Easy use
● No access control
Nbconvert (endering ipynb to static HTML) as a webservice

17. #GDSC 2018
Apache Zeppelin
● Share notebook with ACL, Read/Write/Execute
● In case of Jupyter notebook, need to convert .ipynb to
zeppelin format in command line.

18. #GDSC 2018
Airbnb/knowledge-repo
https://github.com/airbnb/knowledge-repo
● .ipynb, md as a post
● Git repo for version
control
● Feeds
● Search
● No access control

19. #GDSC 2018
Commercial services for notebook sharing
Google Colab
● Share notebook through google drive
● View/Edit/Run ipynb notebook using Colab
● Realtime collaboration
ZEPL
● Notebook level ACL
● View/Edit/Run .ipynb and Zeppelin notebook
● Realtime collaboration
● Import existing notebook from git/s3 storage
www.zepl.com

20. #GDSC 2018
Data Sharing

21. #GDSC 2018
DON’Ts
● Email attach
● Direct send
● Share through USB
● ...
Email attach
Local copy in laptop
USB drive

22. #GDSC 2018
DO’s
● Provide access to the same
dataset
● Access control capability
● Horizontal scalability

23. #GDSC 2018
Data catalog
● Provides location of data, what it means and how to load
○ e.g.
● Catalogue need to be accessible / searchable / annotatable
● Many different way to build depends on team / infra
○ Hive Metastore as a data catalog
○ Cloud infrastructure service (e.g. AWS glue data catalog, Azure data catalog)
○ Data catalog / publishing software (e.g. CKAN, DKAN)
○ Custom built on top of RDBMS, Nosql, Indexing engine
○ Build data catalog using Notebook
Dataset Location Schema Note
Activity s3://service/activity Date (DateTime), type (INT), action(String) Type is either RUN or STOP. ….
Images s3://service/images 512x256 pixel images Images are collected from profile photo...

24. #GDSC 2018
Build data catalog using Notebook
● Flexible enough to describe data
● Searchable, shareable, annotatable
● Programmatic generation

25. #GDSC 2018
Multi-user environment

26. #GDSC 2018
I like my notebook running on my laptop.
No you don’t.

27. #GDSC 2018
Sign in and Run
Install libraries and
Install notebook and
Configure driver, environments and
Request access to data and
Setup access to notebook repo and
….
Run

28. #GDSC 2018
Reverse Proxy
JupyterHub
/hub
Jupyter server
Kernel (Python, R)
Jupyter server
Kernel (Python, R)
/user/[name]
Authenticator
Spawner
Notebook
Storage
(Filesystem, Git, etc)
LDAP,
OAuth,
etc
Docker, k8s
Zeppelin Server
LDAP,
OAuth,
etc
Notebook
Storage
(Filesystem, Git, etc)
Interpreter Manager
Auth / ACL
Interpreter (kernel)
Interpreter (kernel)
Interpreter (kernel)

29. #GDSC 2018
● Easier to implement / manage
● Notebook sharing is decoupled with
execution environment
● Usually notebook sharing is basic or
restricted. (no notebook level ACL)
● e.g.
○ JupyterHub
○ AWS Sagemaker
Reverse Proxy
Single user
Notebook server
Kernel
Single user
Notebook server
Kernel
Notebook
Storage
Multi user
Notebook server
Notebook
Storage
Kernel Kernel Kernel
Browser
Browser
● More complex to implement / manage
● Notebook sharing is coupled with execution
environment
● Usually notebook sharing is more advanced
and fine grained
● e.g.
○ Apache Zeppelin
○ ZEPL
○ Google Colab

30. #GDSC 2018
Conclusion
Notebook Share
Data share
Multi-user environment
Collaboration

31. #GDSC 2018
Thanks