Danil Zburivsky, Big Data Practice Lead, Pythian and Max Luebbe, SRE, Google presentation from HBaseConEast2016: http://www.meetup.com/HBase-NYC/events/233024937/

1. OpenTSDB + Bigtable
Integrating time series
database with
Google Cloud Bigtable
Danil Zburivsky, Big Data Practice Lead, Pythian — zburivsky@pythian.com
Max Luebbe, SRE, Google — maxluebbe@google.com

2. Pythian specializes in design, implementation, and management of
systems that directly contribute to revenue and business success.
History
19 years in business
Growing at 30+% per year
400+ employees
300+ customers worldwide
HQ Ottawa, Canada - global reach
Technology agnostic = trusted advisor
Deep expertise: Oracle, Oracle Apps, MySQL, AWS, SQL Server,
Cassandra/DataStax, Azure, PostgreSQL, Cloudera, MapR, Hortonworks etc.
Google Premier Partner Status (as of end Aug)
5 Certified Developers (soon to be 12)
Dedicated Google Technical Champion
Launch partner for: Kubernetes,
Dataflow, Cloud SQL, Dataproc
Integrated OpenTSDB with Bigtable
DW Explorers Program Partner
Upcoming BigQuery & Cloud ML Launch Partner

3. • (time, metric, value)
• OS and apps metrics
• Industrial equipment
• Web traffic
Time series data

4. • Volume can be explosive
• Data arrival and access
patterns are different
Storing time series data is a challenge

5. • Volume can be explosive
• Data arrival and access
patterns are different
Storing time series data is a challenge

6. OpenTSDB Architecture
Server Server Server Server
TSD TSD
Hbase or Bigtable
TSD RPC
Hbase RPC or
Hbase API
Web UI
Scripts/Alerting
http
TSD RPC

7. • Open source
• Uses HBase as a data store
• Data model optimized for TS
• REST API
OpenTSDB
<metric_uid><timestamp><tagk1><tagv1>[...<tagkN><tagvN>]
<col_t+1>[...<col_t+N>]

8. Origins of Bigtable
• Updating search index in bulk: too slow!
• Rather than a file system (GFS), needed random access
• Generic & scalable, now powers many diverse products

9. Bigtable decouples compute & storage...
Node
Distributed filesystem
Node
Client Client Client Client Client Client
Processing
Storage
Clients
A B C D E

10. ...making scalability fast & seamless
Node
Distributed filesystem
Node
Client Client Client Client Client Client
Processing
Storage
Clients
A B C D E
Node

11. Google Cloud Bigtable
Cloud Bigtable
Bigtable
Service

12. async-bigtable
● Uses standard HBase 1.0 API
● BufferedMutator
● Thread pool
https://github.com/OpenTSDB/asyncbigtable

13. Future work
● Native Bigtable API
● Fully asynchronous
● Performance

14. Demo time!