This is a talk given by Fabio Beltramini and Carl Anderson at the Looker JOIN conference (San Francisco, November 6th, 2019). Here, we review the ecosystem of open-sourced tooling around LookML development. We cover: - Parsers: Interpret LookML text into a more usable/structured format for other software. - Linters: Evaluate LookML against a set of rules, primarily to automate feedback to developers making LookML changes. - Visualizers & Informational: Provide alternate presentations of LookML contents for informational or navigational purposes. - Generators: Automate the writing of LookML from some other process or format. - Testers: Evaluate LookML within the context of a connection, Looker instance, or data, primarily to automate feedback to developers making LookML changes, or, to monitor for breaking external changes. A blog post as a companion to this talk was published the same day at: https://medium.com/@leapingllamas/2019-lookml-open-source-state-of-the-union-d0470012fed0

1. Extending LookML
Development
JOIN 2019

2. Fabio Beltramini
Customer Success @ Looker
© 2019 Looker. All rights reserved. Confidential.
Carl Anderson
Sr. Director, Data Science; Head BI Data Eng
@ WW (The new Weight Watchers)

3. This session is for visionaries
Welcome!
© 2019 Looker. All rights reserved. Confidential.

4. Contents
© 2019 Looker. All rights reserved. Confidential.
● Background
○ Fabio: LookML background
○ Carl: Development best practices background
● What already exists / “State of the Union”
○ rundown of existing / in-progress OSS
● What can you do next?
○ What could you contribute to existing / in-progress projects?
○ What categories of tools could you build?
○ “What would you need from Looker?”

5. But first, some background...
© 2019 Looker. All rights reserved. Confidential.

6. LookML Development Highlights
20162012
New LookML
introduced
© 2019 Looker. All rights reserved. Confidential.
YAML LookML LookML unit tests in
closed beta
2019 Present
Content Validator
out of Beta

7. From the beginning, LookML
tapped in to development best
practices, and continues to develop
rapidly
… but we can accelerate and extend this trajectory
© 2019 Looker. All rights reserved. Confidential.

8. A well-traveled path
1) Official trajectory is fast, but limited
2) What to do if my use case isn’t covered?
3) Leverage platform extensibility with open-source
© 2019 Looker. All rights reserved. Confidential.

9. Extending LookML Development:
The State of the Union
© 2019 Looker. All rights reserved. Confidential.

10. © 2019 Looker. All rights reserved. Confidential.
Generators
Visualizers /
Informational
Parsers
Linters
Testing

11. Parsers
Accept LookML strings/projects and easily interpret them

12. /joshtemple/lkml
Python LookML Parser & Generator
MIT License
Josh Temple
Milk Bar
Node.js is a trademark of Joyent, Inc. and is used with its permission. We are not endorsed by or affiliated with Joyent.
"Python" and the intertwined snake graphic are trademarks of the Python Software Foundation and are used in accordance with its policy. The Github Mark is a
trademark of GitHub, Inc and is used in accordance with its policy.
/fabio-looker/node-lookml-parser
Node.js LookML Parser
MIT License
Fabio Beltramini
Looker

13. Generators
Generate LookML from some other input

14. ● Works with in-memory Python arrays/objects
○ In particular, with the format from its parser
● Enables Parse ⇾ Mutate ⇾ Serialize workflows
lkml
/joshtemple/lkml
MIT License
Josh Temple
Milk Bar

15. ● Dynamically modify existing LookML
○ Add new key:values
○ Search for key and modify value
● Uses lkml parser
LookML Tools / Updater
/ww-tech/lookml-tools
Apache 2.0 License
Carl Anderson
WW

16. LookML Tools / Updater
/ww-tech/lookml-tools
Apache 2.0 License
Carl Anderson
WW

17. ● Build up LookML objects using Python methods
LookML Gen
/symphonyrm/lookml-gen
Apache 2.0 License
Joe Schmid
SymphonyRM

18. Honorable Mentions
Generators that are unpublished or unlicensed, but usable or instructive
Ro LookML Generator
Sami Yabroudi, Ro
FABS
Jesse St. Charles, Frame.ai
JSON ⇾ LookML
Leigha Jarret, Looker
EAV Generator
Fabio Beltramini, Looker

19. Visualizers & Informational
Understand your LookML through alternate presentations

20. ● Generates a “network diagram” of model-explore-views
● Highlight orphaned views
LookML Tools / Grapher
/ww-tech/lookml-tools
Apache 2.0 License
Carl Anderson
WW

21. LookML Tools / Grapher

22. LookML Tools / Grapher

23. ● Analyzes LookML together with i__looker data
● Helps identify stale & unused fields & explores
● Reports viewed via the command line
Henry
/looker-open-source/henry
MIT License
Joseph Axisa
Looker

24. Honorable Mentions
Visualizers that are unpublished or unlicensed, but usable or instructive
/marcellooker/model_diagram_web_app
Marcell Babai, Looker
/leighajarett/Looker_Explore_ERD_Generator
Leigha Jarret, Looker

25. Linters
Prescriptive analysis of LookML during code development

26. ● Includes 9 rules
● Output usable by CircleCI out of the box
● Written in Python, can be forked to add rules
Lookmlint
/WarbyParker/lookmlint
Apache 2.0 License
Ryan Tuck
Warby Parker

27. ● Includes 12 rules
● Outputs markdown files into the repo/Looker IDE
● Written in JS, and accepts custom rules via external JS, or via
in-project LISP expressions in addition to forking
Look At Me Sideways
/looker-open-source
/look-at-me-sideways
MIT License
Joseph Axisa & Fabio Beltramini

28. Look At Me Sideways

29. ● Includes 10 rules
● Outputs to a CSV file, or to a table in BigQuery
● Written in Python, can be forked to add rules
LookML Tools / Linter
/ww-tech/lookml-tools
Apache 2.0 License
Carl Anderson
WW

30. LookML Tools / Linter

31. Testers
Test LookML against an actual instance and/or data

32. ● Runs in CI pipeline
● Queries all explores & fields to find errors
Spectacles
/dbanalyticsco/spectacles
MIT License
Dylan Baker, Josh Temple

33. Contributing

34. © 2019 Looker. All rights reserved. Confidential.
Generators
Visualizers /
Informational
Parsers
Linters
Testing
Existing Projects

35. © 2019 Looker. All rights reserved. Confidential.
Generators
Visualizers /
Informational
Parsers
Linters
Testing
Something New - What’s Missing?
?

36. What will you share?