A discussion about how to choose lines or planes of reasonable boundary for software abstractions.

1. A concrete talk about abstraction.
Theo Schlossnagle @postwait 1

2.  Founder: Circonus, OmniTI, Sparkpost, Fontdeck
 Distinguish ACM Member
 ACM Member at Large
 Avid Open Source Contributor
 Gentleman Farmer
Theo Schlossnagle @postwait 2

3. Boundaries in the software/hardware and software/human interface.
Also in the software/software interface.
Theo Schlossnagle @postwait 3

4. Transistors
Hardware Function
Micro Architecture
Instruction Set Architecture
Machine Code
Assembly Languages
Programming Languages
Algorithms/Data Structures
Applications
Theo Schlossnagle @postwait 4
Hardware
Software

5. Abstractions boundaries are at a
significantly lower dimension
(typically one)
than the space they separate
(typically greater than four).
Theo Schlossnagle @postwait 5
Task /
Problem
Skills
Efficiency /
Resources
Safety /
Correctness
Availability
Security /
Compliance

6. Time Series Data
Histogram
RocksDB
Raw Data
Column Families
Rollups
Tiers
Time Shards
LMDB
ZFS
Files
Abstractions are divides,
and the world isn’t always clean.
Performance requirements are
often a witness.
Result:
“Leaky Abstractions” or
“Abstraction Violations”
Theo Schlossnagle @postwait 6
Time Series Data
(for rebuild)
Time Series Data
(for rollup ETL)

7. Constraints Design
Performanc
e
AudienceScale
Resources
Techniques
API
Boundary
Abstraction
Core Problem
Abstractions aim to be generically useful.
The fewer initial consumers, the less likely this will be satisfied.
Theo Schlossnagle @postwait 7

8. Abstractions interfere with holistic
systems thinking.
Problems span layers.
Therefore analysis spans layers.
Abstractions are loans.
If abstraction boundaries bisect
common analysis tasks, those end
up being high-interest loans.
Theo Schlossnagle @postwait 8
Layer 1
client
communications
Layer 2
distributed
coordination
Layer 3
storage
layer
Expensive analysis
Inexpensiveanalysis

9. Abstractions in real-life can win, can lose, but most often do both.
Theo Schlossnagle @postwait 9

10.  Time Series Database
 Distributed
 Scalable
 High performance
• HTTP
• Lua
• CRDTs
• Math
• Resource Pools
• RocksDB
• C/C++/Asm
Speed Scale
AccessibilityCorrectness
Theo Schlossnagle @postwait 10

11. A sordid love story of layer violations and compromise.
Theo Schlossnagle @postwait 11

12.  libmtev is a multi-threaded, event-driven framework for
building high-performance service applications in C
 It makes it “easy”… for C developers that understand event-driven
code.
 Lua is an elegant and accessible functional language
 LuaJIT (is lua, but fast)
 LuaJIT’s embedding API make violating boundaries
convenient.
Theo Schlossnagle @postwait 12

13. Mistakes layered on mistakes layered on mistakes.
Theo Schlossnagle @postwait 13

14.  updating, appending, deleting
 reading randomly, sequentially, bulk
 It’s basically impossible to excel at all without layering
violations.
Theo Schlossnagle @postwait 14

15. Concurrency limitations and fair scheduling.
Theo Schlossnagle @postwait 15

16.  Resources (CPU, memory, disk, L2 cache, network, etc.) are
limited.
 Concurrency can help until… it hurts.
 For performance, we abstract on resource usage
boundaries/workload characteristics.
Theo Schlossnagle @postwait 16

17. Concrete lessons about abstractions.
Theo Schlossnagle @postwait 17

18. No one would get anything done. Ever.
Theo Schlossnagle @postwait 18

19. Microservices (in particular) afford
isolation choices beyond code
complexity: security, compliance,
availability, ownership, etc.
The dimension on which the abstraction
boundary is drawn should be selected to
yield the most benefit.
Theo Schlossnagle @postwait 19

20. The more things (and ultimately people)
that use the abstraction, the better it will
be designed (or subsequently
redesigned).
Never build an API without a real
consumer (TDD doesn’t count). This
should apply to all simplifications.
Theo Schlossnagle @postwait 20

21. They are not a goal, but they are not
evil. It should be intentional.
Some things behind an abstraction
should intuitively be hidden. Intuition
isn’t always right. When these leaks
consistently benefit multiple
stakeholders, it simply means the line of
abstraction has been redrawn.
Theo Schlossnagle @postwait 21

22. They happen.
They happen all the time.
I am unapologetic about this.
Violation carries cost burden.
Assess it, accept it, and move on.
Theo Schlossnagle @postwait 22

23. https://irondb.io/
https://github.com/circonus-labs/libmtev
Theo Schlossnagle @postwait 23