Presentation from ZFS Day, 2 October 2012 (http://zfsday.com)

1. Disposable Environments at Scale
/ or: How I Learned to Stop Worrying and Love ZFS

2. Eric Sproul
Build Engineer
Sysadmin
Consultant
Twitter:
@eirescot

3. Recipe for Success
enablement (n): the act of providing (someone)
with adequate power, means, opportunity or authority
(to do something)
This is a story of how ZFS enabled business success

4. Background
Etsy is the world's handmade marketplace
Experiencing rapid growth
Every department needed to understand
how the business was evolving
Asked OmniTI for help

5. The Situation
No data warehouse
Analytical queries against
PostgreSQL OLTP
Large initial size (~250 GB)
Forecast to reach 1 TB+
within a year

6. Problems
Long-running queries to OLTP
destroy web performance
Re-running reports
produces different results
Inflexible reporting interface

7. Requirements
Relieve pressure from
OLTP database
Make production data
continuously available
Enable correlation
of other sources
Flexible reporting UI

8. Solution
Create separate BI analytics DB
Build it on ZFS

9. Initial Capabilities
ETL to collate table-level data
from multiple databases
Run deep analytic queries
without impacting website
New web UI enables ad-hoc reporting

10. Reaping the Benefits of ZFS
Snapshots
Faster backups
Simple replica creation

11. Reaping the Benefits of ZFS
Compression
Extend usable life of storage:
PgSQL logical: 1.3T
On-disk: 653G (2.0x)
Intelligent resilver
Shorter rebuilds ==
Reduced risk of data loss

12. We Want More!
Monthly reports now on BI system
Occasional problems require
re-running reports
Still get different results,
same as before

13. We Want More!
Need to test report changes
Fine to dev with small mock-up
Staging requires something
that looks like production

14. The Next Level
Disposable environments
Run on slave replica
R/W copy of BI data
Discard when finished

15. Disposable Environment
Use a non-global zone
set zonepath=/zones/bistage
set autoboot=true
set limitpriv=default,dtrace_proc,dtrace_user
set ip-type=shared
add net
set address=10.1.2.3
set physical=bnx0
end
add dataset
set name=bi01tank/stage
end

16. Disposable Environment
Starting state: ZFS datasets
bi01tank/pgsql/data
bi01tank/pgsql/data/91
bi01tank/pgsql/wal_archive
bi01tank/pgsql/wal_archive/91

17. Disposable Environment
Take snapshots
zfs snapshot -r bi01tank/pgsql/data@stage
bi01tank/pgsql/data@stage
bi01tank/pgsql/data/91@stage
bi01tank/pgsql/wal_archive@stage
bi01tank/pgsql/wal_archive/91@stage

18. Disposable Environment
Create clones
zfs clone <src_dataset>@stage <dst_dataset>
bi01tank/pgsql/data@stage
bi01tank/pgsql/data/91@stage
bi01tank/pgsql/wal_archive@stage
bi01tank/pgsql/wal_archive/91@stage
bi01tank/stage/data
bi01tank/stage/data/91
bi01tank/stage/wal_archive
bi01tank/stage/wal_archive/91

19. Disposable Environment
Zone now sees a full, writable copy of data
Unchanged data is referenced to origin
Changes accumulate to clone

20. Disposable Environment
pgsql/data/91 pgsql/data/91@stage stage/data/91
Unchanged data
referenced from
snapshot
Change accounted
to clone
Live FS Snapshot Clone

21. Next-Level Results
Any report can be re-run
on the same data
Massage existing data or
bring more in for ad-hoc report
Test changes to reports and web UI

22. Next-Level Results
When finished with the environment:
Shut down zone
Delete clone & origin snap

23. Return on Investment
BI database runs on 2 machines
OLTP database lifetime extended
two years past expectation
Faster, more granular, and ad-hoc reporting
enables better decisions by management

24. Bonus!
With the same technique,
we can safely test:
PostgreSQL upgrades
Schema changes

25. Thank You
ZFS, Zones and much more
are available to the community via
illumos and its distributions
Go forth and enable your business!