Using Recursive Common Table Expressions with Ecto

1. Maarten van Vliet
Backend developer @ Awkward
Email: maarten@awkward.co
Github: maartenvanvliet

2. Recursive
Common Table Expressions
and Ecto
PRESENTATION
By Maarten van Vliet

3. First:
an introduction to the problem

4. What is Sketch?
An intuitive vector editor for the
Mac. It’s used primarily by screen
designers who create websites,
icons, and user interfaces for
desktop and mobile devices.

5. Sketch Cloud
Sketch Cloud is a platform that allows
you to share documents easily and with
everyone. Many more features are
coming!
Sketch Cloud uses a GraphQL API built in
Elixir, we call it SketchQL

6. Prototyping
Sketch’s Prototyping features makes
it easy to create interactive
workflows and preview your designs
as your users will see them.
Released last year in Sketch and
Sketch Cloud
A user can now create a prototype in
the Sketch, upload it to Cloud and
interactively play with it

7. Prototyping Cloud
Building prototyping was challenging
• Fluent transitions across browsers
• Converting Sketch Prototypes to the
web
• And, there are simple prototypes such
as this one

8. And complex prototypes…

9. Problems
We needed to fluently transition from
one screen to the next for prototyping in
the browser.
This meant: (deep) preloading the
relations of one screen (artboard) with
all other artboards
So, when A is loaded, we need to load B
and C, but also D!

10. Simplest solution
Recursively query database for related
artboards from application
1. First query for artboard A
2. Query for artboards directly related to
A, returns [B, C]
3. Query for artboards directly related to
[B, C], but leave out already found
artboards [A], this returns [D]
4. Query for artboards directly related to
[D], but leave out already found
artboards [A, B, C], returns []
5. We stop when an empty set is
returned.
Problem:lots of queries

11. Solution:
Recursive Common
Table Expressions!
• Last year we migrated Sketch Cloud to
Mariadb 10.2
• Introduced support for (Recursive) Common
Table Expressions
• (R)CTE's are also available in Mysql 8.0
(since 2018), and Postgres 8.4 (since 2009)
• But what are they?

12. Common Table Expressions
A CTE is a temporary resultset
Think of it as a database view only
created and visible for one query.
Useful for making subqueries easier
to read
You can have multiple in one query
WITH FirstUser AS (
SELECT * FROM Users WHERE id = 1
)
SELECT * FROM FirstUser
— Equivalent to query with subquery
SELECT * FROM
(SELECT * FROM Users WHERE id = 1) AS F;

13. CTE’s can also do
recursion!
Recursive CTE’s are useful for querying
hierarchies, e.g. tables with a parent_id
column, so a row has can have a parent
or children
E.g. a CMS with pages, where a page can
have children
Pages:
WITH RECURSIVE PageGraph AS (
SELECT
P.id,
P.parent_id
FROM
Pages P
WHERE
P.parent_id IS NULL —start id
UNION
SELECT
P.id,
P.parent_id
FROM
Pages P
JOIN PageGraph PG
ON P.parent_id = PG.id
)
SELECT * FROM PageGraph
Id parent_id Name
1 NULL Page 1
2 1 Subpage 1
3 1 Subpage 2
4 2 Subpage 3

14. Dealing with cycles
How to deal with cycles? Hierarchies
with “loops” in them. E.g. page A has
page B as a parent, and page B has page
A as a parent
WITH RECURSIVE PageGraph AS (
SELECT
P.id,
P.parent_id
FROM
Pages P
WHERE
P.id = 1 #start id
UNION
SELECT
P.id,
P.parent_id
FROM
Pages P
JOIN PageGraph PG
ON P.parent_id = PG.id
)
SELECT * FROM PageGraph
Id parent_id Name
1 2 Page 1
2 1 Page 2
Union removes duplicates!

15. Back to the problem
In steps:
• First get artboards related to A, and
store them in “to”, returns [B, C]
• UNION this with the artboards where
the id matches those of [B, C]
• Get related artboards of [B, C], returns
[D]
• Again, UNION and get related
artboards of [D], returns [A].
• Nothing new found, so stop
WITH RECURSIVE RelatedArtboards AS (
SELECT
— A.id AS "from",
F.DestinationArtboardId AS "to"
FROM
Artboards A
JOIN Layers L ON L.ArtboardId = A.id
JOIN Flows F ON F.id = L.FlowId
WHERE
A.id = #Start ID, in this case Artboard A
UNION
SELECT
— A.id AS "from",
F.DestinationArtboardId AS "to"
FROM
Artboards A
JOIN Layers L ON L.ArtboardId = A.id
JOIN Flows F ON F.id = L.FlowId
JOIN RelatedArtboards ON A.id = RelatedArtboards.to
WHERE
A.id = RelatedArtboards.to
)
SELECT
R.to
FROM
RelatedArtboards R
From To
A B
A C
B D
C D
D A

16. Now we only need one query to load
all artboards for a prototype!
But how to use this in Elixir/Ecto?

17. Not supported in the query builder, yet…
Still open 😢

18. Once merged:
page_tree_initial_query =
Page
|> where([p], is_nil(p.parent_id))
page_tree_recursion_query =
Page
|> join(:inner, [p], pt in "page_tree", on: p.parent_id == pt.id)
page_tree_query =
page_tree_initial_queryv
|> union(^page_tree_recursion_query)
Page
|> recursive_ctes(true)
|> with_cte("page_tree", as: ^page_tree_query)
|> Repo.all

19. Until then…
Fragments gives us the
ability to extend Ecto
defmacro with_related_artboards(artboard_id) do
quote do
fragment(
"""
(
WITH RECURSIVE RelatedArtboards AS (
SELECT
F.DestinationArtboardId AS "to"
FROM
Artboards A
JOIN Layers L ON L.ArtboardId = A.id
JOIN Flows F ON F.id = L.FlowId
WHERE
A.id = ?
UNION
SELECT
F.DestinationArtboardId AS "to"
FROM
Artboards A
JOIN Layers L ON L.ArtboardId = A.id
JOIN Flows F ON F.id = L.FlowId
JOIN RelatedArtboards ON A.id = RelatedArtboards.to
WHERE
A.id = RelatedArtboards.to
)
SELECT
RelatedArtboards.to
FROM
RelatedArtboards
WHERE RelatedArtboards.to IS NOT NULL
)
""",
unquote(artboard_id)
)
end
end
import Sketchql.Utils.RelatedArtboards
artboard_id = 1
Artboard
|> join(:inner, [a], ra in with_related_artboards(^artboard_id)
|> Repo.all()
So, this will return a list of
%Artboard{} Ecto.Schema structs
related to the artboard with id 1.
• Keep composability of queries

20. 🎉 Conclusion
• With one query leveraging Ecto and
RCTE ’s we can query all artboards
related to the current one, no matter
how deep.
• In the app we also paginate these
calls. This way we can render much
larger prototypes in Sketch Cloud
• It really pays off to dive deep into the
tools your database can provide such
as RCTE’s.
• Ecto’s extensibility is great! Where we
could not use its native features we
could use SQL to make up for it