New feature overview of Cubes 1.0 – lightweight Python OLAP and pluggable data warehouse. Video: https://www.youtube.com/watch?v=-FDTK80zsXc Github sources: https://github.com/databrewery/cubes

1. Cubes 1.0 Overview
light data warehouse and conceptual modelling
Štefan Urbánek, @Stiivi
stefan.urbanek@gmail.com November 2014

2. understanding
through metadata

3. model data
reporting apps / modules
metadata

4. ❄
logical
physical

5. Categorical Data
Σ =

6. OLAP
(online analytical processing)
lightweight framework for
conceptual modelling and
analytics

7. Original Cubes
before 1.0

8. Workspace 1 × 1 ×
store
process or server
|
model

9. We needed more!

10. Models
Stores
file
Σ
database API
Postgres Mongo API
multiple model parts,
different sources
multiple data sources,
heterogenous

11. Cubes 1.0

12. Python ≥ 3.4
works with ≥ 2.7 too for the “two” series

13. ■ analytical workspace
■ model providers
■ new and improved backends
■ better extensibility
■ authorisation

14. Analytical
Workspace

15. Model Providers
Cubes
Stores
Static Model
Provider
API Model
Provider
sales churn activations events
BI Data
(Postgres)
BI Data 2
(Mongo)
Events
(API)

16. Workspace
Model Providers
Cubes
Stores
Static Model
Provider
sales churn activations events
BI Data
(Postgres)
BI Data 2
(Mongo)
crm sales events
[workspace]
models_path: /var/lib/cubes/models
[models]
crm: crm.cubesmodel
sales: sales.cubesmodel
events: events.cubesmodel
[store crm]
type: sql
url: postgresql://localhost/crm
[store events]
type: mongo
host: localhost
collection: events

17. BYOB
bring your own backend
Slicer

18. Backend

19. |
Browser
"
Store
#
Provider

20. Logical Physical
create model connect
physical data store
(database or API)
|
Browser
"
Store
#
Provider
Σ
aggregate
model
cubes
dimensions
model
backend objects

21. Model Provider
model
cubes
dimensions

23. Model Provider
■ metadata on-the-fly
■ local or external source
■ might be linked to a store
model
cubes
dimensions

24. Model
required
required
automatic
automatic
automatic
Dimensions
column (table)
key/attribute
property
dimension
Cubes / Facts
table
collection
event
metric
Backend
SQL
MongoDB
Mixpanel
Google Analytics
Slicer cube dimension

25. Model Improvements

26. Model
■ measures → aggregates
■ more front-end metadata
cube categories, dimension role and cardinality
■ customised dimension linking

28. "measures": [
{
"name": "amount",
"label": "Sales Amount"
},
{
"name": "vat",
"label": "VAT"
}
]
"aggregates": [
{
"name": “total_sales",
"label": "Total Sales Amount",
"measure": "amount",
"function": "sum"
},
{
"name": “total_vat",
"label": "Total VAT",
"measure": "vat",
"function": "sum"
},
{
"name": "item_count",
"label": "Item Count",
"function": "count"
}
]

29. Aggregates
■ custom name
■ can refer to other aggregates
post-aggregation calculations
■ functions are backend-specific
SQL aggregations: sum, count, count_nonempty, count_distinct, min,
max, avg, stddev, variance, …

30. Contextual Dimensions
{
"measures": [ … ],
"dimensions": [
{"name": "date", "hierarchies": ["ym", "yqm"]},
{"name": "date", "alias": "contract_date"}
],
…
}
customisable linking properties:
alias, hierarchies, exclude_hierarchies,
default_hierarchy_name, cardinality,
nonadditive

31. Dimension Roles
dimension.role
time
level.role
year, month, day, …
hint for reporting applications or backends

32. Cardinality
overload precautions
dimension.cardinality
level.cardinality
tiny < low < medium < high
< <

33. Browser
Σ

34. Browser
■ uses logical model
■ implements aggregation
■ builds queries
■ retrieves data
Logical Physical
physical data store
(database or API)
|
Browser
"
Store
Σ
aggregate
model

35. Browser Methods
■ features()
■ aggregate(cell, drilldown,…)
■ members(cell, dimension, …)
■ facts(cell, …)
■ fact(id)
■ cell_details(cell, drilldown, …)

36. Split Cell
False True
aggregate(split=cell)
__within_split__
generated dimension

37. Post-aggregation
“statutils”
■ computed on aggregation result
in Python
■ moving averages, deviation, variance
wma, sma, sms, smstd, smsrd, smsvar
■ aggregate property: window_size

38. Store

39. Store
■ provides database or API connection
■ might provide a model
■ slicer tool actions (future)
validation, schema, optimization, ... Logical Physical
physical data store
(database or API)
|
Browser
connect
"
Store

40. SQL Backend
also known as ROLAP
or SQL query generator

41. SQL Overview
■ new query builder
■ join optimisation
■ support for outer-joins
■ support for “split” dimension
■ new aggregate functions

42. ❄
fact table
join optimisation

43. facts match date
master detail
facts detail date
master detail
facts master date
master detail
"joins" = [
{
"master": "fact_contracts.contract_date_id",
"detail": "dim_date.id",
"method": "detail"
}
]

44. Authentication and
Authorisation

47. {
“lidia”: {
“allowed_cubes”: [“sales”],
“cube_restrictions”: {
“sales”: [“store:3”]
}
},
“martin”: {
“allowed_cubes”: [“sales”],
“cube_restrictions”: {
“sales”: [“store:5”]
}
}
}

48. [workspace]
authorization: simple
[authorization]
rights_file: access_rights.json
!
Authorizer

49. Slicer
server
✂

51. Model Queries
■ GET /cubes
overview of cubes from all providers
■ GET /cube/sales/model
detailed cube model with described dimensions

52. Browser Queries
■ GET /cube/name/aggregate
■ GET /cube/name/members/dim
■ GET /cube/name/facts
■ GET /cube/name/fact
■ GET /cube/name/cell

53. Aggregate
GET /cube/sales/aggregate? cut=date:2010
& drilldown=date|region & split=status:1
& page=10 & page_size=100

54. {
"cell": [],
"total_cell_count": 2,
"drilldown": [
{
"record_count": 31,
"amount_sum": 550840,
“date.year": 2009
},
{
"record_count": 31,
"amount_sum": 566020,
“date.year": 2010
}
],
"summary": {
"record_count": 62,
"amount_sum": 1116860
}
}

55. Special Characters
“category:10-24”
→ “10-24”
“city:Nové Mesto nad Váhom”
→ “Nové Mesto nad Váhom"

56. Relative Time
uses dimension roles and Calendar
date:yesterday
date:90daysago-today
expiration_date:lastmonth-next2months

57. Output Format
format=csv
format=json
format=json_lines
*
*for facts and members

58. Deployment
reporting for your app or stand-alone

59. Public
HTML & JS
Application
Slicer server
store
HTTP request
JSON reply
model
Public
HTML & JS
Application
WSGI
store
HTTP request
JSON reply
Slicer Flask App
model
Public
HTML
Django, Flask, …
store
JSON reply
Cubes
Python API
model
Public
Public
store
Flask
HTML
HTML
Web Application
PHP, RoR, Django
Slicer server
Slicer Blueprint
model
Internal
store
HTTP request
JSON reply
model

60. Front-ends
generic ad-hoc reporting

62. ✂
Slicer

67. Cubes Viewer
Jose Juan Montes, ! jjmontesl/cubesviewer

68. checkgermany.de
*
Front-end by: Felix Ebert (@femeb)
Data by: Friedrich Lindenberg (@pudo) Cubes 0.10.2

69. Summary & Future

70. Summary
■ heterogenous pluggable environment
■ easier to extend
■ better SQL query generator

71. Not Mentioned
■ localisation
■ namespaces
■ calendar
■ query logging

72. Incubated
■ non-additive properties
■ periods-to-date
■ modeler app
■ cubes.js

73. Future
■ arithmetic expressions
■ SQL improvements
■ improved API for custom browsers
■ cubes.js

74. Nutrition Facts
Serving Size 1 cube
Amount Per Serving
Total Fat 0g
Saturated Fat 0g
Trans Fat 0g
% Daily Value
Total Carbohydrate 0g
Dietary Fiber 0g
Sugars 0g
0%
0%

75. Want to contribute?
#TODO, #FIXME, Issue #
https://github.com/DataBrewery/cubes/issues

76. Credits

77. Thanks for 1.0
Robin Thomas
Ryan Berlew
Jose Juan Montes
Squarespace
and all contributors on Github

78. Thank You
"Stiivi

79. github.com/DataBrewery/cubes
cubes.databrewery.org