The architecture around Hadoop based on Cloud services in Treasure Data.

1. Maintainable Cloud
Architecture of Hadoop
Kai Sasaki
Treasure Data Inc.

2. Who am I?
• Kai Sasaki (佐々木 海)
• @Lewuathe at Twitter, GitHub
• Treasure Data Inc.
Software Engineer
• Contributing Hadoop, Spark.

3. Hadoop in Treasure Data

4. Cloud-based Data warehousing service

8. Hadoop is the core of Treasure Data

9. Hadoop on Cloud
1. Features provided by AWS, IDCF, Heroku etc
2. Fast growing reliability and integrity

10. Hadoop on Cloud
1. Features provided by AWS, IDCF, Heroku etc
2. Fast growing reliability and integrity
Maintainability of Middleware

11. Agenda
• Maintainability of Distributed System
• Our Challenges
• Stateless Hive Metastore
• Cloud Storage for Hadoop
• Multiple Hadoop Version Management
• Regression Test for Hive Queries
• REST API for Hadoop
• Workflow Integration
• What we should keep in mind

12. Maintainability
We think high maintainability is achieved by…
• Stateless

13. Maintainability
We think high maintainability is achieved by…
• Stateless
• Mobility

14. Maintainability
We think high maintainability is achieved by…
• Stateless
• Mobility
• Queueing

15. Stateless
• Stateless Hive metastore
• Cloud Storage for Hadoop

16. Stateless Hive MS

17. Stateful Hive MS
MySQL

18. Stateful Hive MS
Driver Metastore
MySQL

19. Stateful Hive MS
Driver Metastore
MySQL
Require Maintaining RDBMS for only Meta Store

20. Stateless Hive MS
Driver Metastore

21. Stateless Hive MS
Driver Metastore Derby

22. Stateless Hive MS
Driver Metastore Derby
Worker
Submit DDL
request

23. Stateless Hive MS
Driver Metastore Derby
Worker
Submit DDL
request
Aggregate Stateful points
Treasure Data
API

24. Cloud Storage for Hadoop

25. PlazmaDB
Data
Connector
S3, Redshift, MySQL,
PostgreSQL, Salesforce and more
SDK
iOS, Android, JavaScript
Unity
Bulk Import td client
...

26. PlazmaDB
Data
Connector
S3, Redshift, MySQL,
PostgreSQL, Salesforce and more
SDK
iOS, Android, JavaScript
Unity
Bulk Import td client
...
msgpack

27. PlazmaDB
Data
Connector
S3, Redshift, MySQL,
PostgreSQL, Salesforce and more
SDK
iOS, Android, JavaScript
Unity
Bulk Import td client
...
msgpack
Hadoop

28. PlazmaDB
Data
Connector
S3, Redshift, MySQL,
PostgreSQL, Salesforce and more
SDK
iOS, Android, JavaScript
Unity
Bulk Import td client
...
msgpack
Hadoop
Stateful

29. PlazmaDB
PostgreSQL
S3
or
Riak
S3
or
Riak
S3
or
Riak
S3
or
Riak
msgpack
Amazon RDS

30. PlazmaDB
PostgreSQL
S3
or
Riak
S3
or
Riak
S3
or
Riak
S3
or
Riak
msgpack
Amazon RDS
Transaction Immutable

31. Mobility
• Multiple Hadoop Version Management
• Regression Test for Hive Queries

32. Multiple Hadoop
Version Management

33. Multiple Version
Management
CDH HDP Apache

34. Multiple Version
Management
CDH HDP Apache
client client client

35. Multiple Version
Management
CDH HDP Apache
client client client
Tough Operation

36. Multiple Version
Management
CDH HDP Apache
Worker

37. Multiple Version
Management
CDH HDP Apache
Worker
switching

38. Multiple Version
Management
CDH HDP Apache
Worker
switching

39. Multiple Version
Management
CDH HDP Apache
Worker
CDH package
HDP package
Apache package
switching

40. Multiple Version
Management
CDH HDP Apache
Worker
CDH package
HDP package
Apache package
S3
switching

41. Multiple Version
Management
S3
/test
/stable
...

42. Multiple Version
Management
CDH package
HDP package
Apache package
S3
/test
/stable
...

43. Multiple Version
Management
CDH package
HDP package
Apache package
S3
/test
/stable
...
CDH
HDP
Apache
Worker
download

44. Regression Test for Hive
• Introducing new features, version up, migration
must be done without regression
• Running integration system test and regression test
for Hive queries

45. CDH
HDP
Apache
Worker
http://blog.circleci.com/meet-our-new-logo/
System Test
Repository

46. CDH
HDP
Apache
Worker
http://blog.circleci.com/meet-our-new-logo/
System Test
Repository

47. CDH
HDP
Apache
Worker
http://blog.circleci.com/meet-our-new-logo/
System Test
Repository
S3
Hadoop
Repository

48. CDH
HDP
Apache
Worker
http://blog.circleci.com/meet-our-new-logo/
System Test
Repository
S3
Apache package
Hadoop
Repository

49. CDH
HDP
Apache
Worker
http://blog.circleci.com/meet-our-new-logo/
System Test
Repository
S3
Apache package
Hadoop
Repository

50. Queueing
• REST API for Hadoop
• RDS based Queue management system

51. REST API for Hadoop

52. REST API for Hadoop
CDH HDP Apache
Worker

53. REST API for Hadoop
CDH HDP Apache
Worker PerfectQueue
Hadoop Job
Server
REST API

54. REST API for Hadoop
CDH HDP Apache
Worker PerfectQueue
Hadoop Job
Server
REST API
Presto
API

55. RDBMS-based Queue
Management System

56. RDBMS based
queue management
CDH HDP Apache
Worker
Client Client Client
PerfectQueue
Hadoop Job
Server

57. PerfectQueue
• Highly available distributed queue build on RDBMS
• Amazon SQS like API
• Resource scheduling for multi tenancy
• Graceful and Live Restarting
https://github.com/treasure-data/perfectqueue

58. What we should
keep in mind
• Stateless
Delegate responsibility to Cloud systems
• Mobility
Looking ahead for version up, migration
• Queueing
Make each request persistent

59. Recap
• Maintainability of Distributed System
• Our Challenges
• Stateless Hive Metastore
• Cloud Storage for Hadoop
• Multiple Hadoop version management
• Regression Test for Hive queries
• REST API for Hadoop
• Workflow Integration
• What we should keep in mind

60. https://www.treasuredata.com/