Analyzing and comparing your energy consumption with that of other consumers provides healthy peer pressure and useful insight leading to energy conservation and impacting the bottom line. We helped GridPocket (http://www.gridpocket.com/), a smart grid company developing energy management applications for electricity water and gas utilities, implement high scale anonymized energy comparison queries with an order of magnitude lower cost and higher performance than was previously possible. IoT use cases like that of GridPocket are swamping our planet with data, and drive demand for analytics on extremely scalable and low cost storage. Enter Spark SQL over Object Storage: highly scalable and low cost storage which provides RESTful APIs to store and retrieve objects and their metadata. Key performance indicators (KPIs) of query performance and cost are the number of bytes shipped from Object Storage to Spark and the number of incurred REST requests. We propose Pluggable Spark SQL Filters, which extend the existing Spark SQL partitioning mechanism with an ability to dynamically filter irrelevant objects during query execution. Our approach handles any data format supported by Spark SQL (Parquet, JSON, csv etc.), and unlike pushdown compatible formats such as Parquet which require touching each object to determine its relevance, it avoids accessing irrelevant objects altogether. We developed a pluggable interface for developing and deploying Filters, and implemented GridPocket’s filter which screens objects according to their metadata, for example geo-spatial bounding boxes which describe the area covered by an object’s data points. This leads to drastically lower KPIs since there is no need to ship the entire dataset from Object Storage to Spark if you are only comparing yourself with your neighborhood. We demonstrate GridPocket analytics notebooks, report on our implementation and resulting 10-20x speedups, explain how to implement a Pluggable File Filter, and how we applied this to other use cases.

1. © 2017 IBM Corporation
Using Pluggable Apache Spark SQL Filters
to help GridPocket users keep up with the
Jones’ (and save the planet)
Paula Ta-Shma,
Guy Gerson
IBM Research
Contact: paula@il.ibm.com
Joint work with:
Tal Ariel, IBM
Filip Gluszak, GridPocket
Gal Lushi, IBM
Yosef Moatti, IBM
Papa Niamadio, GridPocket
Nathaël Noguès, GridPocket

2. © 2017 IBM Corporation2
You have 5 minutes to pack your bags…

3. © 2017 IBM Corporation3
?

4. © 2017 IBM Corporation4
?

5. © 2017 IBM Corporation
Using Pluggable Apache Spark SQL Filters
to help GridPocket users keep up with the
Jones’ (and save the planet)
Paula Ta-Shma,
Guy Gerson
IBM Research
Contact: paula@il.ibm.com
Joint work with:
Tal Ariel, IBM
Filip Gluszak, GridPocket
Gal Lushi, IBM
Yosef Moatti, IBM
Papa Niamadio, GridPocket
Nathaël Noguès, GridPocket

6. © 2017 IBM Corporation6
What would convince you to reduce your energy usage ?
#EUres2

7. © 2017 IBM Corporation7
What would convince you to reduce your energy usage ?
§ Save $ ?
#EUres2

8. © 2017 IBM Corporation8
What would convince you to reduce your energy usage ?
§ Save $ ?
§ Save the planet ?
#EUres2

9. © 2017 IBM Corporation9
Competing to Save Energy
§ Research shows that utility
customers are most influenced by
peer pressure to save energy
§ => Help utilities compare their
customers’ energy consumption
with that of their neighbours
– Queries are anonymized
#EUres2

10. © 2017 IBM Corporation10
GridPocket
§ A smart grid company developing
energy management applications
and cloud services for electricity,
water and gas utilities
§ HQ based in France
§ http://www.gridpocket.com/
§ Developed open source data
generator
§ Provided industry specific use
cases and algorithms
The GridPocket Dataset
§ Our target: 1 million meters
reporting every 15 minutes
§ Records are ~100 bytes
§ Generated ~1TB in 3 months
§ Allowing for 1 order of magnitude
growth in each dimension gives
1PB in 3 months
§ => Use object storage
§ Typically at least order of
magnitude lower storage cost
than NoSQL databases
§ NoSQL dataset – one large
denormalized table containing
meter reading information
– SQL query for nearest neighbours
#EUres2

11. © 2017 IBM Corporation11
What is Object Storage ?
§ Objects contain data and metadata
§ Written once and not modified
– Cannot append or update data
- Can overwrite completely
- Can update metadata
– No rename operation
§ Accessed through RESTful HTTP
– PUT/GET/POST/DELETE object/bucket
– Flat namespace
§ High capacity, low cost
§ Storage of choice for Big datasets
– Analytics works best on equally sized
objects
§ Examples
– Amazon S3, IBM COS, Google Cloud
Storage, OpenStack Swift
Swift

12. © 2017 IBM Corporation12
Object Storage and Spark are separately managed
microservices

13. © 2017 IBM Corporation13
Object Storage and Spark are separately managed
microservices
Want to query data on object storage directly

14. © 2017 IBM Corporation14
Object Storage and Spark are separately managed
microservices
Want to query data on object storage directly

15. © 2017 IBM Corporation15
Object Storage and Spark are separately managed
microservices
Want to query data on object storage directly
Our goal:Minimize
1. Number of bytes shipped
2. Number of REST requests
THE key factors affecting cost
(and performance)

16. © 2017 IBM Corporation16
How is this done today ?
1. Use specialized column based
formats such as Parquet, ORC
– Column wise compression
– Column pruning
– Specialized metadata

17. © 2017 IBM Corporation17
How is this done today ?
1. Use specialized column based
formats such as Parquet, ORC
– Column wise compression
– Column pruning
– Specialized metadata
We want one solution for all data
formats

18. © 2017 IBM Corporation18
How is this done today ?
1. Use specialized column based
formats such as Parquet, ORC
– Column wise compression
– Column pruning
– Specialized metadata
We want one solution for all data
formats
2. Use Hive Style Partitioning to layout
Object Storage data
– Partition pruning

19. © 2017 IBM Corporation19
Hive Style Partitioning and Partition Pruning
§ Relies on a dataset naming convention
§ Object storage has flat namespace, but can create a virtual folder hierarchy
– Use ‘/’ in object name
§ Data can be partitioned according to a column e.g. dt
– Information about object contents is encoded in object name e.g. dt=2015-09-14
§ Spark SQL can query fields in object name as well as in data e.g. “dt”
– Filters the objects which need to be read from Object Storage and sent to Spark

20. © 2017 IBM Corporation20
Limitations of Today’s Hive Style Partitioning in Spark
§ Only one hierarchy is possible
– Like database primary key, no secondary keys
§ Changing partitioning scheme requires rewriting entire dataset !
– Hierarchy cannot be changed without renaming all objects
§ No range partitioning
– Only supports partitioning with discrete types e.g. Gender:M/F, date, age etc.
– doesn't work well for timestamps, arbitrary floating point numbers etc.
§ A deep hierarchy may result in small and non uniform object sizes
– May reduce performance

21. © 2017 IBM Corporation21
Can we do more ?
NY

22. © 2017 IBM Corporation22
Can we do more ?
§ Generate metadata per
object column and index it
§ Various index types
– Min/max, bounding boxes,
value lists
§ Filter objects according to
this metadata
§ Applies to all formats e.g.
json, csv
NY

23. © 2017 IBM Corporation23
Can we do more ?
§ Generate metadata per
object column and index it
§ Various index types
– Min/max, bounding boxes,
value lists
§ Filter objects according to
this metadata
§ Applies to all formats e.g.
json, csv
NY
Min/max values

24. © 2017 IBM Corporation24
Can we do more ?
§ Generate metadata per
object column and index it
§ Various index types
– Min/max, bounding boxes,
value lists
§ Filter objects according to
this metadata
§ Applies to all formats e.g.
json, csv
NY
One or more bounding boxes

25. © 2017 IBM Corporation25
Can we do more ?
§ Generate metadata per
object column and index it
§ Various index types
– Min/max, bounding boxes,
value lists
§ Filter objects according to
this metadata
§ Applies to all formats e.g.
json, csv
NY
A set of values

26. © 2017 IBM Corporation26
Filter According to Metadata
§ Store one metadata record per object
– Unlike database fully inverted index
§ Various index types
– Min and max value for ordered columns
– Bounding boxes for geospatial data
– Bloom filters as space efficient value lists
§ Users can choose which columns to index
and index type per column
– Can index additional columns later on
§ Main requirement: no false negatives
§ Avoids touching irrelevant objects altogether
§ Handles updates (PUT/DELETE object)
– Filtering out irrelevant objects always works
#EUres2
NY

27. © 2017 IBM Corporation27
Spark SQL query execution flow
Query
Prune
partitions
Read data
Query
Prune
partitions
Optional file
filter
Read data
Metadata
Filter

28. © 2017 IBM Corporation28
The Interface
Filters should extend this trait and implement application specific filtering logic.
trait ExecutionFileFilter {
/**
* @param dataFilters query predicates for data columns
* @param f represents an object which exists in the file catalog
* @return true if the object needs to be scanned during execution
*/
def isRequired(dataFilters: Seq[Filter], f: FileStatus) : Boolean
}
Turn filter on by specifying Filter class:
sqlContext.setConf("spark.sql.execution.fileFilter", “TestFileFilter”)
< 20 lines of code to integrate into Spark
#EUres2

29. © 2017 IBM Corporation29
Data Ingest: How to best Partition the Data
§ Organizing data in
objects in a smart way
generates more effective
metadata
§ Partition during data
ingestion
§ Example: Kafka allows
user to define partitioning
function
§ Needs to scale
horizontally – want
stateless partitioner
#EUres2
Client 1
Partition 0
Client 2
Client 3
Partition 1
Partition 2
Partition 3

30. © 2017 IBM Corporation30
Geospatial Partitioning: Grid Partitioner
§ Divide the world map into a grid
– Precision depends on use case
§ Each data point belongs to a cell
§ Each cell is hashed to a partition
§ A partition can contain multiple
cells
§ Partitions periodically generate
objects
§ Each object is described using a
list of bounding boxes
– 1 per participating cell
#EUres2
1 2 3 4
5 6 7 8
Partition 0
Partition 1
Partition 2
8
2
6
4
5
1
7
3
6 6
5
1
2 7
Objects

31. © 2017 IBM Corporation31
Experimental Results
The GridPocket Dataset
§ 1 million meters reporting every 15 minutes
§ Records are ~100 bytes
§ Generated ~1TB in 3 months
§ Partitioned using Grid Partitioner using precision 1 (cells are roughly 10 km2)
§ Compared using 50 and 100 Kakfa partitions
#EUres2
Get my neighbours’ average usage
SELECT AVG(usage)
FROM (
SELECT vid as meter_id,
(MAX(index)-MIN(index)) as usage
FROM dataset
WHERE (lat BETWEEN 43.300 AND 44.100)
AND (lng BETWEEN 6.800 AND 7.600)
GROUP BY vid
)

32. © 2017 IBM Corporation32
Experimental Results
The GridPocket Dataset
§ 1 million meters reporting every 15 minutes
§ Records are ~100 bytes
§ Generated ~1TB in 3 months
§ Partitioned using Grid Partitioner using precision 1 (cells are roughly 10 km2)
§ Compared using 50 and 100 Kakfa partitions
#EUres2
Get my neighbours’ average usage
SELECT AVG(usage)
FROM (
SELECT vid as meter_id,
(MAX(index)-MIN(index)) as usage
FROM dataset
WHERE (lat BETWEEN 43.300 AND 44.100)
AND (lng BETWEEN 6.800 AND 7.600)
GROUP BY vid
)

33. © 2017 IBM Corporation33
#GB Transferred for bounding box queries
#EUres2
0
200
400
600
800
1000
1200
20x20 km 10x10 km 5x5 km
no filter filter
#GB transferred
50 partitions 100 partitions
1 TB dataset, grid partitioner, precision 1, average of 10 randomly located queries

34. © 2017 IBM Corporation34
#GET Requests for bounding box queries
#EUres2
1 TB dataset, grid partitioner, precision 1, took average of 10 randomly located queries
0
2000
4000
6000
8000
10000
12000
20x20 km 10x10 km 5x5 km
no filter filter
#GET requests
50 partitions 100 partitions

35. © 2017 IBM Corporation35
Time (sec) for bounding box queries
#EUres2
1 TB dataset, precision 1, took average of 10 randomly located queries
0
500
1000
1500
2000
2500
20x20 km 10x10 km 5x5 km
no filter filter
Time (sec)
50 partitions 100 partitions

36. © 2017 IBM Corporation36
Demo
§ Runs on IOSTACK testbed
– 3 Spark and 3 Object
Storage nodes
§ Demo dataset
– 1 million meters
– Report every 15 mins
– 1 day’s worth of data
– 11 GB (csv)
§ Grid Partitioner Config
– Cells have precision 1
- ~10 km2
– 100 partitions
Get my neighbours’ average usage
SELECT AVG(usage)
FROM (
SELECT vid as meter_id,
(MAX(index)-MIN(index)) as usage
FROM dataset
WHERE (lat BETWEEN 43.300 AND 44.100)
AND (lng BETWEEN 6.800 AND 7.600)
GROUP BY vid
)

37. © 2017 IBM Corporation37
Conclusions: Don’t let your vacation turn into a relocation
§ Running SQL queries directly
on Big Datasets in Object
Storage is viable using
metadata
§ A small change to Spark
enables dramatic
performance improvements
§ We focused here on
geospatial data and the
GridPocket use case,
although it also applies to
other data types and use
cases
§ IBM is investing in Spark
SQL as a backbone for SQL
processing services in the
cloud
#EUres2

38. © 2017 IBM Corporation38
Thanks!
Contact:
paula@il.ibm.com
https://www.linkedin.com/in/paulatashma/
guyger@il.ibm.com
https://www.linkedin.com/in/guy-gerson-82619164/

39. © 2017 IBM Corporation39
Backup
#EUres2

40. © 2017 IBM Corporation40
GridPocket Use Case and Dataset
§ NoSQL dataset – one large denormalized table
– date, index, sumHC, sumHP, type, vid, size, temp, city, region, lat, lng
– Index = meter reading
– sumHC = total of energy consumed since midnight in off-hours
– sumHP = total of energy consumed since midnight in rush-hours
– Type = elec/gas/water
– Vid = meter id
– Size = apartment size in square meters

41. © 2017 IBM Corporation41
Example Filter Scenario
§ Want to analyze data from active sensors only
§ External DB contains sensor activity info
Data Layout
Archives/dt=01-01-2014
Archives/dt=01-01-2014/sensor1.json (500MB)
Archives/dt=01-01-2014/sensor2.json (500MB)
Archives/dt=01-01-2014/sensor3.json (500MB)
Archives/dt=02-01-2014
Archives/dt=02-01-2014/sensor1.json (500MB)
Archives/dt=02-01-2014/sensor2.json (500MB)
Archives/dt=02-01-2014/sensor3.json (500MB)
more...
#EUres2
sensor active
sensor1 FALSE
sensor2 TRUE
sensor3 FALSE

42. © 2017 IBM Corporation42
Example Filter
class LiveSensorFilter extends ExecutionFileFilter {
//get a list of live sensors from an external Service
val activeSensors = SensorService.getLiveSensors
//returns true if object represents a live sensor
@Override
def isRequired(dataFilters:Seq[org.apache.spark.sql.sources.Filter],
fileStatus: FileStatus) : Boolean = {
activeSensors.contains(Utils.getSensorIdFromPath(fileStatus))
}
Turn filter on:
sqlContext.setConf("spark.sql.execution.fileFilter", “LiveSensorFilter ”)
#EUres2