IoT NY #24 Meetup Presentation by Google

1. IoT NY - Cloud services for IoT
James Chittenden
Google Cloud Platform Solutions Engineer
jameschi@google.com

2. +James Chittenden
(Big Data Cloud Engineer)
jameschi@google.com

3. Google confidential │ Do not
distribute
Agenda
Big Data the Cloud Way - Why would you ?
Fully Managed: NoOps Ingest, Process & Analyse
Hands On Demo: Building an Event Streaming Pipeline
1
2
3

4. Big Data at Google
aka. Data at Google

5. 20-?? BILLION
devices will be
connected by 2020
$4-11 Trillion
Economic Impact
54% of top performer
companies will invest
more in sensors this yr
Sources: Gartner, PwC, McKinsey

6. 20-?? BILLION
devices will be
connected by 2020
$4-11 Trillion
Economic Impact
54% of top performer
companies will invest
more in sensors this yr
Sources: Gartner, PwC, McKinsey

7. What is IoT?
IoT is a period of transformation
Phone IoT Phone

8. Wearables
Watches
Phones
Cars
Home
Appliances
Existing Business Owned
Equipment
Connected
IoT is a transition to connected
Not Connected

9. Back in the 70s ….

10. The PC

11. The Result

12. A datacenter is not a collection of computers,
a datacenter is a computer.
The same is happening in the Cloud today

13. State of the art Data Centers.
For the past 17 years, Google has been building out the
world’s fastest, most powerful, highest quality cloud
infrastructure on the planet.

14. 2002 2004 2006 2008 2010 2012
Dremel ColossusMapReduce
GFS Bigtable Spanner
2014
Dataflow
Google’s Big Data Innovations go far back Flumejava
BigQuery
Millwheel
Bigtable

15. Extends the Android platform to IoT devices

16. Weave - IoT Protocol and Schema

17. Google Glass at Work

18. Nest - solutions for the connected home

20. Health and Wearables

21. Confidential & ProprietaryGoogle Cloud Platform 21
Management
Mobile
Services
Compute
Big Data
Networking
Storage
Developer Tools

22. Fully Managed:
NoOps Ingest, Process & Analyze

23. StoreCapture Analyze
BigQuery
Process
DataflowCloud Storage
DatastoreCloud SQL
Hadoop/SparkKafka
Pub/Sub
Hadoop/Spark
Manage the Entire Lifecycle of Big Data

24. Dataflow
BigQuery
Fast ETL
Regex
JSON
UDFs
Spreadsheets
BI Tools
Coworkers
Applications + Reports
PubSub
Cloud Storage
BigTable
Your
Data
GCS-Hadoop
Connector
Hadoop on
Compute Engine Cloud Dataproc
unmanaged
managed
Big Data Architecture with Google managed services

25. Building what’s next 25
Scales automatically
No setup or administration
Stream up to 100,000 rows p/sec
Easily integrates with third-party software
Google BigQuery
makes complex data analysis simple

26. Question:
Find root cause why ad was or was
not delivered in the last 30 days.
select date,
rejection_reason, count(*)
from line_item_table.
last30days
where line_item_id=56781234
1.2B Rows scanned
Result in ~5 seconds!
BigQuery Use @Google: DoubleClick Support

27. BigQuery scales “Google scale”
Streaming ingest at peak
Largest Data Lake on BigQuery
Largest query by data size
Largest query by rows 10.5 Trillion rows
2.3 Million rows per second
38 Petabytes
2.1 Petabytes

28. What is BigQuery?
Externalization of Google Dremel
Convenience of SQL
Petabyte-Scale and Fast
Fully Managed, No-Ops Data Warehouse

29. Building what’s next 29
Merges batch and stream processing
Data processing pipelines
Monitoring interface
Significantly lower cost
Runs on Google or Cloudera Spark (Github)
Google Cloud Dataflow
makes complex data analysis simple

30. What is Cloud Dataflow?
Cloud Dataflow is a
collection of SDKs for
building batch or
streaming parallelized
data processing pipelines.
Cloud Dataflow is a fully
managed service for
executing optimized
parallelized data processing
pipelines.

31. Cloud Pub/Sub
• Globally redundant
• Low latency (sub sec.)
• Batched read/write
• Custom labels
• Push & Pull
• Auto expiration
Publisher A Publisher B Publisher C
Message 1
Topic A Topic B Topic C
Subscription XA Subscription XB
Subscription
YC
Subscription
ZC
Cloud
Pub/Sub
Subscriber X Subscriber Y
Message 2 Message 3
Subscriber Z
Message 1
Message 2
Message 3
Message 3

32. Dataflow goodies
Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Pipeline p = Pipeline.create();
p.begin()
.apply(TextIO.Read.from(“gs://…”))
.apply(ParDo.of(new ExtractTags())
.apply(Count.create())
.apply(ParDo.of(new ExpandPrefixes())
.apply(Top.largestPerKey(3))
.apply(TextIO.Write.to(“gs://…”));
p.run();

33. Dataflow goodies
Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Deploy
Schedule & Monitor

34. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Dataflow goodies
800 RPS 1200 RPS 5000 RPS 50 RPS

35. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Dataflow goodies

36. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Dataflow goodies
Pipeline p = Pipeline.create();
p.begin()
.apply(TextIO.Read.from(“gs://…”))
.apply(ParDo.of(new ExtractTags())
.apply(Count.create())
.apply(ParDo.of(new ExpandPrefixes())
.apply(Top.largestPerKey(3))
.apply(TextIO.Write.to(“gs://…”));
p.run();
.apply(PubsubIO.Read.from(“input_topic”))
.apply(Window.<Integer>by(FixedWindows.of(5, MINUTES))
.apply(PubsubIO.Write.to(“output_topic”));

37. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Dataflow goodies
Nighttime Mid-Day Nighttime

39. Demo Time
Pub/Sub
Ingest Process Analyse
Cloud Dataflow BigQuery
Git: https://github.com/james-google/event-streams-dataflow

40. Demo Time
Pub/Sub
Ingest Process Analyse
Cloud Dataflow BigQuery
Git: https://github.com/james-google/event-streams-dataflow

41. Questions?

42. Thank You
James Chittenden
jameschi@google.com