Talk given at HashiConf 2016. Video at https://www.youtube.com/watch?v=xXWaECk9XqM

1. The Container Revolution
Reflections after the first decade
CTO
bryan@joyent.com
Bryan Cantrill
@bcantrill

2. The Container Revolution
Reflections after the first third decade
CTO
bryan@joyent.com
Bryan Cantrill
@bcantrill

3. Containers are born: March 18, 1982
• Containers are not a new idea, having originated via filesystem
containers with chroot in Seventh Edition Unix
• chroot originated with Bill Joy, but specifics are blurry; according
to Kirk McKusick, via Poul-Henning Kamp and Robert Watson:

4. Containers ca. 2000
• Seeking to provide a security mechanism, FreeBSD extended
chroot into jails:

5. • To provide workload consolidation, Sun introduced complete
operating system virtualization with zones (née Project Kevlar)
Containers ca. 2002

6. Containers ca. 2006

7. Elsewhere ca. 2006...

8. Aside: Hardware-level virtualization
• Since the 1960s, the preferred approach for operating legacy
stacks unmodified has been to virtualize the hardware
• A virtual machine is presented upon which each tenant runs an
operating system that they choose (but must also manage)
• Effective for running legacy stacks, but with a clear inefficiency:
there are as many operating systems on a machine as tenants:
• Operating systems are heavy and don’t play well with others with
respect to resources like DRAM, CPU, I/O devices, etc.!
• Still, hardware-level virtualization became de facto in the cloud

9. Containers ca. 2011

10. Containers ca. 2011, cont.
• Via SmartOS zones, Joyent has run containers in multi-tenant
production since ~2006...
• Adding support for hardware-based virtualization circa 2011
strengthened our resolve with respect to OS-based virtualization!
• We saw that the operational characteristics of OS containers —
performance, elasticity, tenancy — opened up new possibilities...

11. Containers ca. 2012/2013

12. Containers ca. 2012/2013, cont.
• Over 2012 and early 2013, we built Manta, a ZFS- and container-
based internet-facing object store offering in situ compute
• Containers allow the compute to be moved to the data instead of
having to backhaul data to transient compute
• And as a reminder, the OS — Unix! — was built around the notion
of ad hoc computation

13. Aside: Unix
• When Unix appeared in the early 1970s, it was not merely a new
system, but a new way of thinking about systems
• The Unix philosophy, as articulated by Doug McIlroy:
• Write programs that do one thing and do it well
• Write programs to work together
• Write programs that handle text streams, because that is a
universal interface
• Four decades later, this philosophy remains as relevant as ever!

14. Aside: Doug McIlroy v. Don Knuth
• In 1986, Jon Bentley posed the challenge that became the Epic
Rap Battle of computer science history:
Read a file of text, determine the n most frequently used words,
and print out a sorted list of those words along with their
frequencies.
• Don Knuth’s solution: a purpose-built algorithm in WEB, a Pascal-
like literate programming system of his own invention
• Doug McIlroy’s solution shows the power of the Unix philosophy:
tr -cs A-Za-z 'n' | tr A-Z a-z |
sort | uniq -c | sort -rn | sed ${1}q

15. Containers ca. 2012/2013, cont.
• Manta allows for an arbitrarily scalable variant of McIlroy’s
solution to Bentley’s challenge:
mfind -t o /bcantrill/public/v7/usr/man |
mjob create -o -m "tr -cs A-Za-z 'n' |
tr A-Z a-z | sort | uniq -c" -r
"awk '{ x[$2] += $1 }
END { for (w in x) { print x[w] " " w } }' |
sort -rn | sed ${1}q"
• As with McIlroy’s solution, the container allows for not just for
terser expression but faster execution!

16. Containers ca. 2012/2013
• Manta served to strengthen our belief in the power of containers
as not merely better infrastructure, but as transformative
• Our biggest challenge with Manta was that containers were not
well understood: it felt “easy” to us, but was confusing to many
• Would the world ever really figure out containers?!

17. Containers ca. 2013

18. Containers ca. 2013/2014: Revolution at last!
• Docker used the rapid provisioning + shared underlying
filesystem of containers to allow developers to think operationally
• Operational dependencies are encoded via an image
• Images can be reliably and reproducibly deployed as a container
• Images can be quickly deployed — and re-deployed
• Coupled with the mainstream adoption of open source and the
leading adoption of microservices...
• Containers accelerate software development

19. Containers ca. 2014/2015

20. Containers ca. 2014/2015, cont.
• Triton combines SmartOS and our cloud management software
with a Docker Remote API endpoint
• With Triton, the notion of a Docker host is virtualized: to the
Docker client, the datacenter is a large Docker host
• One never allocates VMs with Triton; one only allocates
containers — and all Triton containers run directly on-the-metal
• All of Triton is open source — you can download it and run it for
yourself (or run it on our public cloud)

21. Containers ca. 2015

22. Containers ca. 2015

23. Containers ca. 2015

24. Containers ca. 2015

25. Containers ca. 2015

26. 2016 and beyond: Frameworks v. Libraries
• The crowded container ecosystem can be viewed broadly as two
different approaches:
• A framework approach in which the framework is in control, but
alternatives are “pluggable”
• A library approach in which the developer/operator is in
control, composing functionality out of well-defined modules
• The “all in” nature of frameworks may make initial adoption easier,
but they ultimately sacrifice flexibility!

27. 2016 and beyond: Frameworks v. Libraries
• For example, a framework approach to orchestration may not
accommodate applications that demand local persistence
• We opted for orchestration that lives not in an all-knowing
framework, but rather with the application, in the container
• In-container logic allows for service registration/discovery, health
checks, etc. — but does not handle restart
• This is application-centric orchestration, and we have dubbed this
the autopilot pattern: https://autopilotpattern.io

28. 2016 and beyond: Containers on autopilot
• We have developed ContainerPilot, an embodiment of the
autopilot pattern for Docker containers
• All open source (https://github.com/joyent/containerpilot) — and
not Triton or Joyent specific; can be used anywhere

29. 2016 and beyond: Container failure modes

30. 2016 and beyond: Container failure modes

31. 2016 and beyond: Container failure modes

32. 2016 and beyond: Failure matters
• When deploying containers + microservices, there is an unstated
truth: you are developing a distributed system
• While more resilient to certain classes of force majeure failure,
distributed systems remain vulnerable to software defects
• We must be able to debug such systems; hope is not a strategy!
• Distributed systems are hard to debug — and are more likely to
exhibit behavior non-reproducible in development
• With container-based systems, we must think in terms of the
system, not merely the program!

33. 2016 and beyond: The Jevons paradox
• The Jevons paradox seems very likely to hold for containers:
greater efficiency will result in a net increase in consumption!
• Efficiency gains from containers are in terms of developer time...
• ...but requiring containers to be scheduled in VMs induces
operational inefficiencies: every operator must now think like a
cloud operator — maximizing density within fixed-cost VMs
• Greater consumption + operational inefficiencies threaten to slow
the container revolution — or make it explosive in terms of cost
• To realize the full economic promise of the container revolution,
we need container-native infrastructure!

34. 2016 and beyond: Public and on-prem
• Death of on-prem computing is greatly exaggerated!
• There are three key determinants for public v. on-premises:
• Economics: Rent vs. buy; OPEX vs. CAPEX
• Risk Management: Security/compliance — and also risk
factors associated with operator-as-threat
• Latency: The speed of light is a constant!
• Economics dominates: “private cloud” efforts that do not deliver
public cloud economics are doomed to (continue to) fail!

35. 2016 and beyond: #serverless
• On the one hand, this term is clearly meaningless…
• ...but on the other, it highlights that containers have liberated us to
think in terms of higher-level functionality
• Thinking in functions is great (viz. Manta), but confining functions
to containers to single-tenant VMs undermines their economics
• The virtual machine is a vestigial abstraction; we must reject
container-based infrastructure that implicitly assumes it!
• So don’t say “#serverless” when you mean...

36. 2016 and beyond
#vmless