1. Overview
SCALE14x 2016

2. Agenda/Schedule
-Apache Bigtop Overview
-Apache Spark Overview/Getting Started
-Lunch Break
-Apache Ignite
-Workshop, tutorial, open time
http://workshops.bigtop.rocks
(click on Agenda button)

3. What is Bigtop?
Setting the standard for testing, packaging and
integration of leading big/fast data components

4. and many other…
Components as Building Blocks

5. --------------------------------------------------------------------
-----
Dependency Hell!!
hdfs
zookeeper
hbase
kafka
spark
.
.
.
mapred
oozie
hive
etc
-----------------------------------------------------
-----
-----------------------------------------------------
-----
-----------------------------------------------------
-----
-----------------------------------------------------
-----
-----------------------------------------------------
-----
-----------------------------------------------------
-----
Build all the
Things!!!

6. The BOM
Build of Materials (BOM)
* List of >=1 components
* Gradle for build/actions
* Produce sets of debs/rpms

7. Bigtop Origins
Yahoo!, 2010
Created, fostered early Hadoop community
Working on Hadoop 0.20 stack
2011
Yahoo!’s to Cloudera, solving early problems of packaging and
maintaining first commercial supported Hadoop distro

8. Early value add
Provide a common foundation for proper integration of
growing number of Hadoop family components
Foundation provides solid base for validating applications
running on top of the stack(s)
Provide neutral packaging and deployment/config

9. Early Mission Accomplished
Foundation for commercial Hadoop distros/services
Leveraged by app providers…

10. What now?
We are done right?1?!?

11. Industry/Ecosystem Evolution
&
New Community Needs/Ideas

12. Where should we spend our time?,
which users should benefit?

13. Moving beyond oob mapreduce…

14. Lambda/Stream Architectures
HDFS + Zookeeper +

15. Get out from the Apache dome

16. New focus and target end users
Data engineers vs distro
builders
Enhance
Operations/Deployment
Reference implementations
& tutorials

17. Laying new foundation with 1.0+
Self-starter, non-kitchen sink building
-Making gradle tooling smarter
-Jenkins job autogen
-leveraging containers for
parallelization

18. Data data data…
Smarter/Realistic test data
-bigpetstore
-bigtop-bazaar
-weather data gen
Tutorial/Learning Data sets
-githubarchive.org
-more tbd…

19. Deployment/Mgmt
Updated puppet modules
-newest best practices
-next level enhanced security options
Wider range of starter deployment topologies
Include some handling of test/tutorial data

20. More components…

21. Sounds interesting, how can I help?
*Join mailing list, ask questions, suggest features, etc
*Contribute (components, tutorials, docs)
*Report bugs

22. Thank You, Q&A
Nate D’Amico
kaiyzen@apache.org
@kaiyzen