This document discusses the evolution of data analytics and modeling. It describes three waves: the first with slow hardware and manual entry; the second with faster PCs but tool explosions; and the third wave now with big data, cloud warehouses, and data-driven tools like Looker and BigQuery. It argues that in this current wave, having a flexible yet performant data model built on SQL in a warehouse, and using a language like LookML to define relationships and translate questions, allows gaining reliable answers with agility without worrying about low-level syntax or tools.
9. 9
...what if?
1. Storing big data got cheap?
2. Querying big data was fast?
3. Warehouses scaled elastically?
4. Ops was handled?
5. You paid for what you used?
13. 13
...you would want?
1. A model that
a. aids performance
b. is flexible and easy to update
c. reflects the real world and is easy to understand
2. A tool that could leverage the warehouse directly
3. A language to abstract away low-level concerns
15. 1940 1950 1960 1970 1980 1990 2000 2010
Machine Code
Assembly
FORTRAN
COBOL
BASIC C C++
Objective-C
Python
Java
Javascript
PHP
C#
Go
Ruby
1940 1950 1960 1970 1980 1990 2000 2010
Data Written to Files
Roll Your Own b-tree
Codd SQL
Oracle V2
IBM DB2
T-SQL MySQL
PostgreSQL
??????
Programming Language Development
Data Language Development
16. 16
What do we want from data language?
1. Define relationships and definitions once
2. Retain agility
3. Translate business questions to data queries
4. Stay performant
5. Stop worrying about syntax
17. 17
“What about SQL? I love SQL!”
It’s proven, powerful and versatile
Except:
SELECT
DATE(orders.timestamp),
SUM(orders.total)
FROM
orders
WHERE
orders.status = ‘completed’
GROUP BY 1
18. 18
LookML is one solution: builds on SQL
● Reusable
● Collaborative
● Flexible
● Organized
● Version-controlled
20. 20
Third wave
● Big Data
Revolution
● Too many tools
● The Internet
Why? Advantages
● Reliable answers
● Agility
● Best-in-class
tools
● Full resolution
Disadvantages
● Shift in thinking
Need a powerful
warehouse
Insight
abundance