This presentation gives an overview of the Apache Gobblin project. It explains Apache Gobblin in terms of it's architecture, data sources/sinks and it's work unit processing. Links for further information and connecting http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/ https://nz.linkedin.com/pub/mike-frampton/20/630/385 https://open-source-systems.blogspot.com/

1. What Is Apache Gobblin ?
● A big data integration framework
● To simplify integration issues like
– Data ingestion
– Replication
– Organization
– Lifecycle management
● For streaming and batch
● An Apache incubator project

2. Gobblin Execution Modes
● Gobblin has a number of execution modes
● Standalone
– Run on a single box / JVM / embedded mode
● Map Reduce
– Run as a map reduce application
● Yarn / Mesos ( proposed ? )
– Run on a cluster via a scheduler, supports HA
● Cloud
– Run on AWS / Azure, supports HA

3. Gobblin Sinks/Writers
● Gobblin supports the following sinks
– Avro HDFS
– Parquet HDFS
– HDFS byte array
– Console (StdOut)
– Couchbase
– HTTP
– JDBC
– Kafka

4. Gobblin Sources
Gobblin supports the following sources
● Avro files
● File copy
● Query based
● Rest API
● Google Analytics
● Google drive
● Google webmaster
● Hadoop text input
● Hive Avro to ORC
● Hive compliance purging
● JSON
● Kafka
● MySQL
● Oracle
● Salesforce
● FTP / SFTP
● SQL Server
● Teradata
● Wikipedia

5. Gobblin Architecture

6. Gobblin Architecture
● A Gobblin job is built on a set of plugable constructs
● Which are extensible
● A job is a set of tasks created from a workunit
● The workunit serves as a container at runtime
● Tasks are executed by the Gobblin runtime
– On the chosen deployment i.e. MapReduce
● Run time handles scheduling, error handling etc
● Utilities handle meta data, state, metrics etc

7. Gobblin Job

8. Gobblin Job
● Optional aquire lock (to stop next job instance)
● Create source instance
● From source work units create tasks
● Launch and run tasks
● Publish data if OK to do so
● Persist the job/task states into the state store
● Clean up temporary work data
● Release the job lock ( optional )

9. Gobblin Constructs

10. Gobblin Constructs
● Source partitions data into work units
● Source creates work unit data extractors
● Converter converts schema and data records
● Quality checker checks row and task level data
● Fork operator allows control to flow into multiple streams
● Writers sends data records to sink
● Publisher publishes job records

11. Gobblin Job Configuration
● Goblin jobs are configured via configuration files
● May be named .pull / .job plus .properties
● Source properties file defines
– Connection / converter / quality / publisher
● Job file defines
– Name / group / description / schedule
– Extraction properties
– Source properties

12. Gobblin Users

13. Available Books
● See “Big Data Made Easy”
– Apress Jan 2015
●
See “Mastering Apache Spark”
– Packt Oct 2015
●
See “Complete Guide to Open Source Big Data Stack
– “Apress Jan 2018”
● Find the author on Amazon
– www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
●
Connect on LinkedIn
– www.linkedin.com/in/mike-frampton-38563020

14. Connect
● Feel free to connect on LinkedIn
– www.linkedin.com/in/mike-frampton-38563020
● See my open source blog at
– open-source-systems.blogspot.com/
● I am always interested in
– New technology
– Opportunities
– Technology based issues
– Big data integration