Jupyter is useful for DevOps. It enables collaboration between experts and novices to accumulate infrastructure knowledge, while automation via notebooks enhances traceability and reproducibility. Yoshi Nobu Masatani shows how to combine Jupyter with Ansible for reproducible infrastructure and explores knowledge, workflow, and customer support as literate computing practices. Session type: Session Topics: Usage and application

2. yoshi NOBU masatani, National Institute of Informatics
NII Cloud Operation
• NII Cloud Operation is a team supporting
researchers and teachers in our institute
• Struggling to adequately support; maintaining
private OpenStack, setting up software stacks,
consulting their configuration and optimization,
and managing almost everything
• We have been applying the Jupyter Notebook
to operation engineering for years
https://github.com/NII-cloud-operation

3. yoshi NOBU masatani, National Institute of Informatics
• our domain and motivation
• extension to Jupyter
• what have learned and where we are going to..

4. yoshi NOBU masatani, National Institute of Informatics
Literate Computing for Reproducible Infrastructure
• Automated Operation: assure traceability and
reproducibility utilizing computational narratives, i.e.
Jupyter Notebook
• Collaboration: share reproducible experience
participating both tech and non-tech alike regarding
infrastructure design, development and operation

5. yoshi NOBU masatani, National Institute of Informatics
Facing Practical Problems..
• Record daily operations for assuring ops quality
• Refactor proven but scattered procedures
into reusable forms
• Prepare knowledge for apprentice engineers
and serve educational materials for users
At all situations above computing and engineering experience
should be consistent and seamless... Beyond SILOs!!
• for exchanging and accumulating knowledge
• similar descriptive methods, i.e. language and representation, should be applicable
• efficient granularity and representation of schema should be managed
• would not depend upon subjects/objects matters, phases of processes..
Traceability
Reusability
Reproducibility

6. yoshi NOBU masatani, National Institute of Informatics
Reproducible Experience
Reproducible Operation Procedure
Make “Prerequisite” reproducible too
What we call as “Bindings” to the world.
• Moving targets, i.e. systems’ system
• Always involve human in a evolutional loop
Literate Computing
Skill 文芸 Art
How we maintain Notebooks
Reproducible Infrastructure
Share Knowledge
Share Reproducible
Experience
Jupyter Notebook ~ Reproducible Document

7. yoshi NOBU masatani, National Institute of Informatics
Literate Computing for Reproducible Infrastructure
Cloud
< Live Codes >
< Outputs >
refactor
operations
run notebooks as live codes
describe reproducible
operations as notebooks
keep outputs for tractability and reproducibility
communicate via notebookcommunicate via notebook
share and elaborate
narrative stories
share and elaborate
narrative stories
self administration
reproducible workflows
accelerated experiments
Ansible Playbook &
Jupyter Notebook
Reference Patterns
Expert EngineerApprentice Engineer Expert User
Participant User

8. yoshi NOBU masatani, National Institute of Informatics
Infrastructure Operation

9. yoshi NOBU masatani, National Institute of Informatics
Daily Operation through Jupyter
Evidences for traceable operations

10. yoshi NOBU masatani, National Institute of Informatics
Reusable Notebooks for Reproducible Infrastructure
25 Notebooks & 55 Playbooks
• Deployment
• Maintenance/Operation
• Test/Verify
https://github.com/NII-cloud-operation/Literate-computing-Hadoop

11. yoshi NOBU masatani, National Institute of Informatics
Our Deployment
Expert Engineer
@Operation Team
Production
@Private Cloud
Playground @Public Cloud
Participant User
Apprentice Engineer
@Operation Team
JupyterHub
Expert User
Jupyter
@Management
Server
Managed by Operators
Self Admin by Users
Acquire Skills from
Experiences on Sandbox
KVM
…
Jupyter
@Hypervisor

12. yoshi NOBU masatani, National Institute of Informatics
Share Reproducible Experiences
Transfer skill and knowledge through shared reproducible experiences
among engineers, users, tech and non-tech.
< Live Codes >
< Outputs >
Engineers
Expert Engineer
share
experiences
Apprentice Engineer
Expert User
Participant User
User
elaborate
notebookstest deployment of applications
production deployment for users
share experiences
for administration
share
experience
acquire skill and knowledge
through shared experiences
self
administration

13. yoshi NOBU masatani, National Institute of Informatics
Enhancement for Robust and Reliable Operation
The goals for Literate Computing tools are:
• Secure outputs; summarize massive output lines on GUI and save all original
output lines into an individual file
• Prevent miss-operation; once a cell has been executed, it “freezes” against
unintended execution. Also you can “lock” cells for unintended modification.
• Add a perspective; when markdown’s hierarchy is collapsed embedded cells
underneath are represented as “bricks” and also run through altogether.
Jupyter Notebook server which bundles a
set of tools for Literate Computing for
Reproducible Infrastructure. This bundle
shows what we have enhanced Jupyter.
docker run -it --rm -p 8888:8888 niicloudoperation/notebook

14. yoshi NOBU masatani, National Institute of Informatics
About NII Extensions
https://github.com/NII-cloud-operation

15. yoshi NOBU masatani, National Institute of Informatics
Secure Outputs – LC_wrapper kernel
• Summarize massive output lines
• At each cell’s execution all original
output lines are saved into an
individual file with a time stamp.
• You can review whole output and
compare with previous results from
different executions.
https://github.com/NII-cloud-operation/Jupyter-LC_wrapper

16. yoshi NOBU masatani, National Institute of Informatics
Enhanced Collapsible Headings
https://github.com/NII-cloud-operation/Jupyter-LC_run_through/

17. yoshi NOBU masatani, National Institute of Informatics
“Run through” Control

18. yoshi NOBU masatani, National Institute of Informatics
Notebook for Diagnostic & Variation

19. yoshi NOBU masatani, National Institute of Informatics
Visualize Relationship and Dependency
https://github.com/NII-cloud-operation/Literate-computing-Elasticsearch/blob/master/99_Summarizing%20notebooks.ipynb
http://blockdiag.com/en/

20. yoshi NOBU masatani, National Institute of Informatics
Visualize Relationship and Dependency

21. yoshi NOBU masatani, National Institute of Informatics
Visualize Relationship and Dependency

22. yoshi NOBU masatani, National Institute of Informatics
Exploring Best Way..

23. yoshi NOBU masatani, National Institute of Informatics
https: //github.com/NII-cloud-operation/Jupyter-LC_docker << Document
$ docker pull niicloudoperation/notebook
$ docker run -it --rm -p 8888:8888 niicloudoperation/notebook
demo

24. yoshi NOBU masatani, National Institute of Informatics
Automation Should Be Like Iron Man, Not Ultron
“The automation we created abided by the complementarity
principle. It was a partnership between human and machine. It
did not limit our ability to learn and grow. The control over the
system was shared between the automation and the humans
involved.
In other words, rather than creating a system that took over
the cluster and ran it, we created one that partnered with
humans to take care of most of the work. It did its job
autonomously, but we did not step on each other's toes.”
Thomas A. Limoncelli. 2015. Automation Should Be Like Iron Man, Not
Ultron. Queue 13, 8, pages 50 (September 2015), 10 pages.

25. yoshi NOBU masatani, National Institute of Informatics
Collaboration and Automated Operation
“Manual”
Operations
“Autonomous”
Operations
No Ops
“Automated”
Operations
DevOps
+ Programming
“Leftover Principle”
“Complementarity Principle”
Automation within a Silo
and in Blackboxes
Literate Computing for
Reproducible Infrastructure
+ Computational
Narratives
• Collaboration among human sharing reproducible experiences
including both “what went wrong” and “what went right”
• Collaboration with Automation keeping potential to respond,
monitor, learn and anticipate

26. yoshi NOBU masatani, National Institute of Informatics
Our Tools as Parts of Ourselves
“Automation weakens the bond between tool and user
not because computer-controlled systems are complex
but because they ask so little of us. They hide their
workings in secret code. They resist any involvement of
the operator beyond the bare minimum. They discourage
the development of skillfulness in their use. Automation
ends up having an anesthetizing effect. We no longer feel
our tools as parts of ourselves. ”
InCarr, Nicholas. The Glass Cage: How Our Computers Are
Changing Us (p. 223). W. W. Norton & Company. Kindle Edition.

27. yoshi NOBU masatani, National Institute of Informatics
Hardiman, General Electric in 1965
https://www.engadget.com/2014/01/26/ge-man-amplifying-robots/
Automation 自働化
≠
Automated Operation 機械化
“Our tools as parts of ourselves”

28. yoshi NOBU masatani, National Institute of Informatics
Hadoop – Ready! on CentOS6 /
Hadoop Swimlanes – Ready! on Tez / Hivemall – Ready! on Hive / …
Start / Stop the services / Hadoop – Decommission DataNode / …
Operational Policy Settings; Security.../ Hadoop Prerequisites – Ready! on CentOS6
KVM – Ready! on CentOS / KVM – Set! CentOS6 / KVM – Go! VM / …
CentOS6 CentOS6 CentOS6 CentOS6 CentOS6 CentOS6 CentOS6 CentOS6
VM /
Baremetal
VM /
Baremetal
VM /
Baremetal
VM /
Baremetal
VM /
Baremetal
VM /
Baremetal
VM /
Baremetal
VM /
Baremetal
ZooKe
eper
ZooKe
eper
ZooKe
eper
NameN
ode
NameN
ode
DataNo
de
DataNo
de
DataNo
de
DataNo
de
Resour
ce
Manag
er
Resour
ce
Manag
er
Node
Manag
er
Node
Manag
er
Node
Manag
er
Node
Manag
er
Spark
Tez-
tools
Hive
Hivem
all
Test Dataset Test Job
3. Hadoop Deployment Notebook
4. Hadoop Operation Notebook
2. Hadoop Prerequisites Notebook
Simple YARN for Test / Simple Spark script for Test / … 5. Hadoop Post Verification
Notebook
1. Hadoop Cluster Design NotebookSet! Inventory
YARN
HDFS
Elaborated Notebook’s Structure

29. yoshi NOBU masatani, National Institute of Informatics
Elaborated Control Flows

30. yoshi NOBU masatani, National Institute of Informatics
nblineage – Understand Life Cycle of Notebooks
How people share, revise, and evolve notebooks in a team.
A hypothesis:
- there should be a seed notebook and it would be reused
and revised several situations.
- Subsequently, a useful seed would generate a collection of
diverged notebooks,
- Then the collection would be refactored..
nblineage has been developed in order to study how usable
notebooks are evolved and distributed. This is on-going
practice..

31. yoshi NOBU masatani, National Institute of Informatics
https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
Cell’s meme ID at `lc_cell_meme` of cell metadata.
{
"lc_cell_meme": {
"current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008", ## The meme id for this cell
"previous": "8f5c5cea-71cc-11e7-9abe-02420aff0008", ## The context of this cell as reference to the
"next": "f2125b84-4669-11e7-958b-02420aff0006", ## previous and the next cells
"history": [ ## The history of this notebook's context
{
"current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008",
"previous": "8f5c5ee8-71cc-11e7-9abe-02420aff0008",
"next": "8f5c60aa-71cc-11e7-9abe-02420aff0008"
},
{
"current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008",
"previous": "f2125b84-4669-11e7-958b-02420aff0006",
"next": "8f5c5ee8-71cc-11e7-9abe-02420aff0008"
}
]
},
...
}

32. yoshi NOBU masatani, National Institute of Informatics
https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
Notebook's meme ID at `lc_notebook_meme` of notebook metadata.
{
"lc_notebook_meme": {
"current": "1904d564-71c6-11e7-8369-0242ac110002", ## The meme id for this notebook
"lc_server_signature": {
"current": {
"server_url": "https://xxxxx.nii.ac.jp/user/xxxx/", ## The current notebook server information is also stored
"notebook_path": "/",
"notebook_dir": "/notebooks",
"signature_id": "034b406c-71c8-11e7-a8bf-02420aff0008"
},
"history": [
{ ## Travel history where this notebook has been executed
"notebook_dir": "/notebooks",
"notebook_path": "/",
"server_url": "http://localhost:8888/",
"signature_id": "dc3b0162-71bb-11e7-8369-0242ac110002"
}
]
},
...
}

33. yoshi NOBU masatani, National Institute of Informatics
https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
The list of meme IDs of cells in this notebook is also recorded
{
"lc_notebook_meme": {
"current": "5fb59a5c-71d0-11e7-9acc-0242ac110002", ## newly assigned notebook's meme id
"lc_server_signature": {
"current": {
...
...
},
"root_cells": [
"5fb59caa-71d0-11e7-9acc-0242ac110002", ## The list of cells' meme IDs in this notebook
"5fb59dfe-71d0-11e7-9acc-0242ac110002", # In this case there are 6 cells in this notebook
"5fb59f2a-71d0-11e7-9acc-0242ac110002",
"5fb5a042-71d0-11e7-9acc-0242ac110002",
"5fb5a150-71d0-11e7-9acc-0242ac110002",
"5fb5a25e-71d0-11e7-9acc-0242ac110002"
],
"history": [
"1904d564-71c6-11e7-8369-0242ac110002" ## old meme id of this notebook that was <source.ipynb>
]
},
...
}

34. yoshi NOBU masatani, National Institute of Informatics
Lineage Network
Notebook
degree of
common memes

35. yoshi NOBU masatani, National Institute of Informatics
Lineage Network

36. yoshi NOBU masatani, National Institute of Informatics
Lineage Network
Collect reusable materials
as a baseline
CentOS 6 7
Import stacks
Deploy a seed
Production
Side work for production
Another concurrent
development

37. yoshi NOBU masatani, National Institute of Informatics
Retrospective Communication

38. yoshi NOBU masatani, National Institute of Informatics
Thank you
https://github.com/NII-cloud-operation