しまねOSS協議会 第106回オープンソースサロン・総会記念講演

1. Open Source Software,
Distributed Systems,
Database as a Cloud Service
第106回オープンソースサロン・総会記念講演
Jul 29, 2016
Satoshi Tagomori (@tagomoris)

2. Satoshi "Moris" Tagomori
(@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, ...
Treasure Data, Inc.

3. Topics
• What is Treasure Data?
• Who is tagomoris?
• Treasure Data: Database as a Service
• DB as a Service and Distributed Systems
• Distributed Systems and Open Source Software
• Open Source Software and Developers

5. http://rubybiz.jp/prize.html

7. API
Data
M
arts
O
DBC
/ JDBC
Sensor
ERP
CRM
RDBMS
Mobile
Web
Server
3 Complex ETL
4 End User System
2 Time consuming integration
1 Disparate data silos
Without
Treasure Data
Advanced
Analytics
Reporting
BI

8. API
Data
M
arts
O
DBC
/ JDBC
Sensor
ERP
CRM
RDBMS
Mobile
Web
Server
IoT
Connectors
Data
Connectors
JavaScript
SDK
Serverside
collector
Bulk
Loader
M
obile
SDK
With Treasure Data
3
Easy to Integrate
2 Zero Management
1
Easy to Collect
50+ Data Outputs
Multi-Tenant Cloud Service
300+ Data Sources
Advanced
Analytics
Reporting
BI
50+Integrations
Schema-flexible, Access via SQL,
Unlimited Users, Queries

9. HQ
Branch
Matsue

11. Treasure Data, Inc.
• Since Nov 2011
• Headquarters: Mountain View, CA, US
• Japan Branch: Marunouchi, Chiyoda, Tokyo
• Korea Branch: Gangnam, Seoul
• Some remote workers - US, UK, Costa Rica

12. Developers in TD
• Daily development in each offices
• Communication over Internet
• Slack, JIRA, Confluence & Zoom
• Frontend Team: mainly in US
• Console, Web services, etc
• Backend Team: mainly in JP
• Database, Distributed processing systems, etc

13. Satoshi "Moris" Tagomori
(@tagomoris)
Born in Matsue, Shimane
Living in Tokyo from 1999

14. Started to work
as an OSS developer
1. Asahi Net
Internal system developer
2. NTT DATA Intellilink
System consultant
3. livedoor - NHNJ - LINE
Infrastructure engineer
Data analytics platform
engineer
4. Treasure Data
Backend engineer
OSS developer

15. @tagomoris as
an Open Source Software Developer
• Author
• Norikra, Woothee, xbuild, Shib, Yabitz, Focuslight
• Many fluent-plugin-*
• And many libraries, tools, etc
• Committer, Maintainer
• Fluentd, MessagePack-Ruby, etc
• Contributor
• Docker (logging driver), etc

16. @tagomoris as
an Open Source Software Developer
• Talks
• Many programming conferences (local, global)
• Many small meetups
• Articles
• WEB+DB Magazine, Software Design
• Many blog posts
• Invented Event: ISUCON

17. OSS Developers in TD
• MessagePack, Fluentd, Embulk & Digdag founder
• Ruby committer
• Ruby & JRuby committer
• Fluentd & D-language committer
• Hadoop/Spark contributor, pyenv author, ...

18. Why Are OSS Developers
So Major in TD?

19. Treasure Data:
Database as a Cloud
Service

20. API
Data
M
arts
O
DBC
/ JDBC
Sensor
ERP
CRM
RDBMS
Mobile
Web
Server
IoT
Connectors
Data
Connectors
JavaScript
SDK
Serverside
collector
Bulk
Loader
M
obile
SDK
3
Easy to Integrate
2 Zero Management
1
Easy to Collect
50+ Data Outputs
Multi-Tenant Cloud Service
300+ Data Sources
Advanced
Analytics
Reporting
BI
50+Integrations

21. Database as a Cloud Service
• Collect data
• from remote site - customer side
• Store/Process data
• beyond cloud
• Integrate data
• to remote site - customer side

22. Two OSS Pattern in TD
• OSS to collect/integrate data from/to remote site
• OSS to store/process data

23. API
Data
M
arts
O
DBC
/ JDBC
Sensor
ERP
CRM
RDBMS
Mobile
Web
Server
IoT
Connectors
Data
Connectors
JavaScript
SDK
Serverside
collector
Bulk
Loader
M
obile
SDK
3
Easy to Integrate
2 Zero Management
1
Easy to Collect
50+ Data Outputs
Multi-Tenant Cloud Service
300+ Data Sources
Advanced
Analytics
Reporting
BI
50+Integrations

24. Make Input/Output Easy
• Agent installed in our customers systems
• OSS + Plugin to connect various systems
• No barrier to use TD
1.Make a great OSS product to do it
2.Make it major
3.Potential customer already uses it :)
• very easy to switch to use Treasure Data!

25. Multi-Tenant Cloud Service
API
Data
M
arts
O
DBC
/ JDBC
Sensor
ERP
CRM
RDBMS
Mobile
Web
Server
IoT
Connectors
Data
Connectors
JavaScript
SDK
Serverside
collector
Bulk
Loader
M
obile
SDK
3
Easy to Integrate
2 Zero Management
1
Easy to Collect
50+ Data Outputs
300+ Data Sources
Advanced
Analytics
Reporting
BI
50+Integrations

26. Database as a Service
and
Distributed Systems

27. Many Customers in a System
• Share computer resource
• Provide much more computer resource
• Reduce total cost :-)

28. Big Data in a System
• Manage big data from many customers
• Manage computing power for many customers
• Create a distributed system!
• for fast query processor
• for resource scheduler
• for high availability

29. Distributed Systems
and
Open Source Software

30. Distributed Systems

31. Distributed System Software
• Major software are all OSS
• Hadoop, Presto, Kafka, Storm, ...
• Concept and Implementation
• MapReduce concept was from Google
• Yahoo! engineers implemented it as Hadoop
• Many others made Hadoop better
• Data is always growing
-> Software MUST be growing too

32. Deploying Distributed System
• Many things make it hard to fix issues
• Big data, many computers, complex queries, ...
• We MUST fix our issues as soon as possible
• for our customers
• for our operation costs
DO IT YOURSELF! → OSS

33. Updating Distributed System
• It's very hard to update distributed systems
• many servers, no data lost, no downtime, ...
• Use OSS as-is without dirty fix
• to keep it easy to upgrade "software"
• Contribute your patch to community
• to use patched mainstream software as-is

34. Open Source Software
and
Developers

35. DIY Policy Makes "Tech" Company
• Do it yourself "At Your Own Risk": OSS
• Taking risk: more OSS
• OSS: more controllable than proprietary software
• We can read/contribute source code :)
• Technology problem: Can we take a risk? Or not?

36. Tech Company and Developers
• Taking risk for business success:
more focus on technology
• Quality of OSS depends on each developers
• Who is the committer of that product?
• Who can review quality of that product?
• Tech company needs great developers seriously!

37. OSS and Developers
• "OSS Committer", not "OSS Committing Company"
• the initiative by developer, not company
• Commit log shows everything about common things
• Who did contribute to that software?
• Who did develop that feature?
• Who did fix that problem?
• People can know who is a good software engineer
• it makes good developers happy!

38. Developers love OSS Company
• OSS Company: a kind of Tech Companies
• easy to find it: see committers/contributors
• Developers love:
• challenging "technical" tasks/issues to be solved
• great coworkers, like committers of great software
• nice salary brought by taking risk :P

39. Enjoy Engineering!
MOST IMPORTANT THING:
Thanks!