Vanessa Sochat, Research Software Engineer, Stanford Research Computing Center, and Gregory Kurtzer, High Performance Computing Services group at Lawrence Berkeley National Lab, present on their work with Singularity and Singularity Hub. ​ Dear reader, how should you disseminate your software? If you want your recipe to come out just right, we encourage you to put it in a container. One such container, Singularity, is the first of its kind to be securely deployed internationally on more than 40 shared cluster resources. Its registry, Singularity Hub, further supports reproducible science by building and making containers accessible to any user of the software. In this talk, Vanessa will review the primary use cases for both Singularity and Singularity Hub, and how both have been designed to support modern, common workflows. (Greg will participate remotely.) She will discuss current and future challenges for building, capturing metadata for, and organizing the exploding landscape of containers, and present novel work for assessing reproducibility of such containers. Containers are changing scientific computing, and this is something to be excited about.

1. SINGULARITY
CONTAINERS FOR SCIENCE
Vanessa Sochat, PhD
Research Software Engineer
Research Computing Stanford University

2. THE PERFECT SANDWICH

3. The Perfect Sandwich
1. Peanut Butter
2. Jelly
3. Bread
4. Spread on Bread
5. Eat

4. The Perfect Sandwich
1. Peanut Butter
2. Jelly
3. Bread
4. Spread on Bread
5. Eat

5. The Perfect Sandwich
1. Peanut Butter
2. Jelly
3. Bread
4. Spread on Bread
5. Eat

6. The Perfect Sandwich
1. Peanut Butter
2. Jelly
3. Bread
4. Spread on Bread
5. Eat

7. IS IT THE SAME SANDWICH?

8. SAME SAME, BUT DIFFERENT

9. SAME SAME, BUT DIFFERENT

10. WE COULD HAVE DONE WORSE...

11. Why does it taste different?

12. 1. Our recipe was not reproducible

13. 1. Our recipe was not reproducible
2. We had missing dependencies

14. 1. Our recipe was not reproducible
2. We had missing dependencies
3. The perfect sandwich might never be made again

15. 1. Our recipe was not reproducible
2. We had missing dependencies
3. The perfect sandwich might never be made again
no ability to easily distribute or validate work

16. Introducing Singularity

17. Introducing Singularity
Give them the sandwich.

18. Container:
encapsulation of system
environment

19. LIFE’S
WORK
Container:
encapsulation of system
environment

20. Why not Docker?

21. DOCKER IS (STILL) GREAT!

22. DOCKER IS (STILL) GREAT!
Docker
Well-known container platform

23. DOCKER IS (STILL) GREAT!
Docker
Well-known container platform
Micro-service virtualization

24. DOCKER IS (STILL) GREAT!
Docker
Well-known container platform
Micro-service virtualization
Create + distribute containers

25. DOCKER IS (STILL) GREAT!
Docker
Well-known container platform
Micro-service virtualization
Create + distribute containers
Reproducible

26. DOCKER IS (STILL) GREAT!
Docker
Well-known container platform
Micro-service virtualization
Create + distribute containers
Reproducible
Easy to use, well documented

27. WHY NOT DOCKER?
Docker is not designed for,

28. WHY NOT DOCKER?
Docker is not designed for,
efficient for,

29. WHY NOT DOCKER?
Docker is not designed for,
efficient for,
or even compatible with

30. WHY NOT DOCKER?
Docker is not designed for,
efficient for,
or even compatible with
traditional HPC architectures

31. WHY NOT DOCKER?
Docker is not designed for,
efficient for,
or even compatible with
traditional HPC architectures
No centers run Docker on their traditional HPC

32. +
HPC
HPC ADMIN

33. HPC USER

34. scientists
need
containers
too

35. 1. Singularity, three ways
2. Singularity Hub
3. Reproducible Science

36. Singularity
...three ways

37. How do I use it

38. THE SINGULARITY FLOW

39. Image Creation
$ singularity create ubuntu.img
$ singularity import ubuntu.img docker://ubuntu:14.04

40. Image Creation
$ singularity create ubuntu.img
$ singularity import ubuntu.img docker://ubuntu:14.04

41. Image Creation
$ singularity create ubuntu.img
$ singularity import ubuntu.img docker://ubuntu:14.04
$ singularity pull docker://ubuntu:14.04
ubuntu-14.04.img

42. Image Bootstrap
$ singularity create ubuntu.img
$ sudo singularity bootstrap ubuntu.img Singularity

43. Image Bootstrap
$ singularity create ubuntu.img
$ sudo singularity bootstrap ubuntu.img Singularity

44. Bootstrap: docker
From: python:latest
Singularity

45. Bootstrap: docker
From: python:latest
%post
apt-get update
apt-get install -y vim wget
mkdir /cave
Singularity

46. Bootstrap: docker
From: python:latest
%post
apt-get update
apt-get install -y vim wget
mkdir /cave
%labels
MAINTAINER vanessasaurus
Singularity

47. Bootstrap: docker
From: python:latest
%post
apt-get update
apt-get install -y vim wget
mkdir /cave
%labels
MAINTAINER vanessasaurus
%files
/home/vanessa/Desktop/rawr.sh /cave/rawr.sh
Singularity

48. Bootstrap: docker
From: python:latest
%post
apt-get update
apt-get install -y vim wget
mkdir /cave
%labels
MAINTAINER vanessasaurus
%files
/home/vanessa/Desktop/rawr.sh /cave/rawr.sh
%environment
DINOSAUR_HOME=/cave
export DINOSAUR_HOME
Singularity

49. Bootstrap: docker
From: python:latest
%post
apt-get update
apt-get install -y vim wget
mkdir /cave
%labels
MAINTAINER vanessasaurus
%files
/home/vanessa/Desktop/rawr.sh /cave/rawr.sh
%environment
DINOSAUR_HOME=/cave
export DINOSAUR_HOME
%runscript
exec /bin/bash /cave/rawr.sh “$@”
Singularity

51. Where does it live?

52. open
source
https://octodex.github.com/

53. client
(bash)

54. client
(bash)
src
(C)

55. client
(bash)
src
(C)
helper
(python)

56. /home/vanessa/.singularity
├── docker
├── metadata
└── shub
/usr/local/var/singularity/
└── mnt
├── container
├── overlay
└── session

57. /usr/local/
├── bin
├── etc
├── include
├── lib
├── libexec
└── var
./configure --prefix=/usr/local

58. client
/usr/local/
├── bin
├── etc
├── include
├── lib
├── libexec
└── var
./configure --prefix=/usr/local

59. /usr/local/
├── bin
├── etc
├── include
├── lib
├── libexec
└── var
src

60. /usr/local/
├── bin
├── etc
├── include
├── lib
├── libexec
└── var
python

61. /usr/local/
├── bin
├── etc
├── include
├── lib
├── libexec
└── var
mount

62. /usr/local/
├── bin
├── etc
├── include
├── lib
├── libexec
└── var
config

63. How does it work?

64. Installation
git clone https://www.github.com/singularityware/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

65. Customizable by the HPC Admin
SINGULARITY.CONF
- bind/mount points
- permissions
- overlayfs

66. Customizable by the HPC Admin
SINGULARITY.CONF
- bind/mount points
- permissions
- overlayfs
- config file must be root owned

67. Customizable by the HPC Admin
SINGULARITY.CONF
- bind/mount points
- permissions
- overlayfs
- config file must be root owned
- controls what user can/not do

68. Customizable by the HPC Admin
SINGULARITY.CONF
- bind/mount points
- permissions
- overlayfs
- config file must be root owned
- controls what user can/not do
- dis/allow different devices

69. Customizable by the HPC Admin
SINGULARITY.CONF
- bind/mount points
- permissions
- overlayfs
- config file must be root owned
- controls what user can/not do
- dis/allow different devices
- paths, session dirs all controlled

70. If you want to be root inside the container, you
must be root outside the container.

71. contained processes exit
all namespaces collapse
...leaving a cleaned system

72. The Singularity Command
singularity --debug run --contain sandwich.img

73. The Singularity Command
singularity --debug run --contain sandwich.img
<action>

74. The Singularity Command
singularity --debug run --contain sandwich.img
[global options]

75. The Singularity Command
singularity --debug run --contain sandwich.img
[command options]

76. The Singularity Command
singularity --debug run --contain sandwich.img
<image>

78. share
sandwich?

79. Singularity
Hub

80. 1. Singularity, three ways
2. Singularity Hub
3. Reproducible Science

81. WHERE IS THE BOTTLENECK?

82. SINGULARITY HUB: CONTAINER REGISTRY

83. COLLECTIONS

84. COLLECTION

85. COLLECTION
commit

86. CONTAINER BUILD

87. CONTAINER BUILD LOG

88. ESTIMATED OPERATING SYSTEMS

89. How does it work? :>

90. 1. Add bootstrap specification file to Github repo base

91. 1. Add bootstrap specification file to Github repo base
2. “Turn build on” in Singularity Hub

92. 1. Add bootstrap specification file to Github repo base
2. “Turn build on” in Singularity Hub
3. Commits are built automatically on Google Cloud

93. 1. Add bootstrap specification file to Github repo base
2. “Turn build on” in Singularity Hub
3. Commits are built automatically on Google Cloud
4. Accessible via command line

97. 1. Singularity, three ways
2. Singularity Hub
3. Reproducible Science

98. 1. Singularity, three ways
2. Singularity Hub
3. Reproducible Science
What happens next?

99. container...
predictions!
Change in the movement of information

100. Change in the movement of information
bits

101. Change in the movement of information
bits
file

102. Change in the movement of information
bits
file
folder

103. Change in the movement of information
bits
file
folder

104. Change in the movement of information
bits
file
folder
software

105. Change in the movement of information
bits
file
folder
software

106. Change in the movement of information
bits
file
folder
software
apt-get install -y party-animal
pip install party-animal

107. I’m missing dependencies.
I didn’t get the same result
What version of Python did you use?
It doesn’t compile on my system!

108. The unit of information isn’t good enough.

109. Change in the movement of information
bits
file
folder
software
os

110. Change in the movement of information
bits
file
folder
software
osos
containers

111. container...
predictions!
Change in the movement of information: put stuff in containers

112. Too many containers!

114. Which containers do genomic analysis?

115. Which containers do genomic analysis?
Which containers do it best? How do we define best?

116. Which containers do genomic analysis?
Which containers do it best? How do we define best?
Which ones have the most varying result? Why?

118. I don’t know how to measure that.

119. Our representation of containers isn’t good enough

121. Expectation:
This container makes the
perfect sandwich!

122. Reality:

123. container...
predictions!
Change in the movement of information: put stuff in containers
Change in the representation of containers: reproducibility metrics

124. 1. Singularity, three ways
2. Singularity Hub
3. Reproducibility Metrics

125. How is container C1
similar to container C2
?
C1
C2

129. C1
C2

130. Intersection of sets C1
and C2
C1
C2
Total sum of files in C1
and C2

131. C1
C2
Is container C1
similar to container C2
?

132. C1
C2
Is container C1
similar to container C2
?
It depends who is asking

133. SONIC MADE IT THROUGH
REPRODUCIBILITY LEVEL REPLICATE!

134. C1

135. Levels of Reproducibility
Identical: the exact same image file

136. Levels of Reproducibility
Identical: the exact same image file
Replicate: the same image built at different times

137. Levels of Reproducibility
Identical: the exact same image file
Replicate: the same image built at different times
Base: the core os is estimated to be the same

138. Levels of Reproducibility
Identical: the exact same image file
Replicate: the same image built at different times
Base: the core os is estimated to be the same
Runscript: the content of the runscript is the same
Environment: the environments are the same
Labels: the container labels are the same

139. What is a level of reproducibility?
A set of files between containers that are compared via content hash

141. Intersection of sets C1
and C2
C1
C2
Total sum of files in C1
and C2

142. C1
C2
Reproducibility
Assessment Algorithm
“Hash Content Comparison”
Intersection of sets C1
and C2
Total sum of files in C1
and C2

143. Do the levels behave as I would expect?
Compare an image to itself

144. Do the levels behave as I would expect?
Compare an image to itself
- At step 1, start with the image compared to its full self

145. Do the levels behave as I would expect?
Compare an image to itself
- At step 1, start with the image compared to its full self
- Subtract one file from the second image, recalculate, until empty

146. Do the levels behave as I would expect?
Compare an image to itself
- At step 1, start with the image compared to its full self
- Subtract one file from the second image, recalculate, until empty
a. Remove more recent files first

147. Do the levels behave as I would expect?
Compare an image to itself
- At step 1, start with the image compared to its full self
- Subtract one file from the second image, recalculate, until empty
a. Remove more recent files first
Do this across all levels

150. Reproducibility Metrics: Takeaways
1. “Operating system science” needs to be a thing

151. Reproducibility Metrics: Takeaways
1. “Operating system science” needs to be a thing
2. Definitions of levels important

152. Reproducibility Metrics: Takeaways
1. “Operating system science” needs to be a thing
2. Definitions of levels important
3. I learned things about the OS just looking at the graphs

153. Reproducibility Metrics: Takeaways
1. “Operating system science” needs to be a thing
2. Definitions of levels important
3. I learned things about the OS just looking at the graphs
4. A way to derive features for an operating system?

154. thinking
about
the
future

155. How can containers support reproducible science?

156. How can the HPC community support containers?

157. container sharing
integration
incentives

158. container sharing
integration
incentives

160. This is not optimized for scaled building!

161. Singularity Registry

162. Singularity Registry
a local registry for a cluster resource

165. Challenges
- Most resources can’t support web download links
- How to share images? manifests?
- Storage (for most) is a file system
- No Docker for orchestration
- Permissions?
- Integration with Singularity Hub?
- Management?

166. storage
- file system

167. storage
- file system
builders
- job queue
- build node
- virtual machines

168. storage
- file system
builders
- job queue
- build node
- virtual machines
manager
- singularity image
- command line
- web interface

169. /usr/local/libexec/sregistry
├── cli
│ ├── help.database
│ ├── help.init
│ ├── sregistry.build
│ ├── sregistry.database
│ ├── sregistry.help
│ └── sregistry.init
│
├── helpers
│ ├── args
│ ├── update
│ └── utils
└── singularity.registry
sudo ./install.sh --prefix=/usr/local

170. /opt/shub/
builder/
templates/
recipes/
.git/
.travis.yml
storage/
containers/
sudo sregistry init --base /opt/shub

171. /opt/shub/builder
recipes/
.git/
tensorflow/
tensorflow/
Singularity
Singularity.gpu

172. /opt/shub/builder
recipes/
.git/
tensorflow/
tensorflow/ ← collection tensorflow/tensorflow
Singularity
Singularity.gpu

173. /opt/shub/builder
recipes/
.git/
tensorflow/
tensorflow/ ← collection tensorflow/tensorflow
Singularity shub://tacc/tensorflow/tensorflow
Singularity.gpu

174. /opt/shub/builder
recipes/
.git/
tensorflow/
tensorflow/ ← collection tensorflow/tensorflow
Singularity shub://tacc/tensorflow/tensorflow
Singularity.gpu
registry

175. /opt/shub/builder
recipes/
.git/
tensorflow/
tensorflow/ ← collection tensorflow/tensorflow
Singularity shub://tacc/tensorflow/tensorflow
Singularity.gpu
container name

176. /opt/shub/builder
recipes/
.git/
tensorflow/
tensorflow/ ← collection tensorflow/tensorflow
Singularity shub://tacc/tensorflow/tensorflow:tag
Singularity.gpu
tag

177. registry:
container collection corresponds to a folder in repository

178. registry:
container collection corresponds to a folder in repository
Individual user:
container collection corresponds to an entire Github repo

179. registry:
container collection corresponds to a folder in repository
Individual user:
container collection corresponds to an entire GIthub repo
both
build multiple tags for one collection from within same repository

180. Connect to Singularity Hub

181. Connect to Singularity Hub
...permission to build, granted!

182. ...build away, Merrill.

183. 1. If setup to build locally
Launches local build job
2. If setup to only build on Singularity Hub
Pings Singularity Hub
3. Both
Launches local build job
Successful builds ping Singularity Hub

184. 1. run a build command
sudo sregistry build tensorflow
sudo sregistry build tensorflow/tensorflow
sudo sregistry build tensorflow/tensorflow:gpu

185. /opt/shub/builder
templates/
recipes/
.git/
tensorflow/
tensorflow/
Singularity
Singularity.gpu

186. container sharing
integration
incentives

187. container sharing
integration
incentives

188. THE CLOUD HPC RESOURCE

189. How can we work together?

190. Local
Development
Environment

191. Local
Development
Environment

192. Local
Development
Environment
Testing

193. Local
Development
Environment
Testing
Deploy and
Share

194. Run!Local
Development
Environment
Testing
Deploy and
Share
Run!
Run!

195. Run!Local
Development
Environment
Testing
Deploy and
Share
Run!
Run!
Result

196. Run!
Creation
Testing Publication
Run!
Run!
Reproduce

197. Run!
Creation
Testing Publication
Run!
Run!
Reproduce

198. How can we work together?

199. How can we work together?
Just try.

201. Thank you Google!

202. container sharing
integration
incentives

203. Academic Layer Cake
scientists

204. Academic Layer Cake
scientists
staff

205. scientist
“I need a custom tool”

206. scientist
“I need a custom tool”
staff
“I offer resources”

207. scientist
“I need a custom tool”
staff
“I offer resources”
scientist
“I’ll do it myself”

208. scientist
“I need a custom tool”
staff
“I offer resources”
scientist
“I’ll do it myself”

209. scientist
“I need a custom tool”
staff
“I offer resources”
scientist
“I’ll do it myself”

210. Why can’t we do better?

211. Academic Layer Cake
scientists
staff
software engineers

214. Lessons from Software Engineering
1. Continuous Integration (testing)
2. Version Control
3. Documentation
4. Logging, Handling Errors
5. Databases, Organization, and Storage

215. We need incentives and support for
Research Software Engineers

216. scientist
“I need a custom tool”

217. scientist
“I need a custom tool”
software engineer
“I can help with that”

218. containers-ftw

220. containers-ftw
crowdsourcing science with
competitive containers

221. containers-ftw
crowdsourcing science with
competitive containers
- package your challenge

222. containers-ftw
crowdsourcing science with
competitive containers
- package your challenge
- define metric of success

223. containers-ftw
crowdsourcing science with
competitive containers
- package your challenge
- define metric of success
- share it

224. containers-ftw
crowdsourcing science with
competitive containers
- package your challenge
- define metric of success
- share it
- ...may the best container win!

225. SHOW ME WHAT YOU GOT

226. SHOW ME WHAT YOU GOT
- Dave Godlove, NIH
- Stefan Kombrink
https://singularity-hub.org/demos/6/

229. 1. Singularity, three ways
2. Singularity Hub
3. Reproducible Science

230. [ Singularity ]

231. [ Singularity ]
reproducible tools

232. [ Singularity ]
reproducible tools
are a group effort

233. [ Singularity ]
reproducible tools
are a group effort
grow out of need

234. [ Singularity Hub ]

235. [ Singularity Hub ]
reproducible practices

236. [ Singularity Hub ]
reproducible practices
sharing containers, data, software

237. [ Singularity Hub ]
reproducible practices
sharing containers, data, software
working across lines

238. [ Reproducibility Metrics ]

239. [ Reproducibility Metrics ]
representation for understanding

240. [ Reproducibility Metrics ]
representation for understanding
container transparency

241. [ Incentives ]

242. [ Incentives ]
Research software engineering

243. Build for how you want the world to be

244. Party on, party dinosaur

245. [ the perfect sandwich ]

246. HPC Admin, Developers, and Scientists

247. http://singularityware.github.io
https://www.singularity-hub.org
https://www.github.com/singularityhub

248. Got messy code?
Need to use a node?
...jokes, help, for free!
#SRCC

249. Only the best
for your analysis mess
SRCC

250. vsochat@stanford.edu