Scala can be used for developing both frontend (Scala.js) and backend (Scala JVM) applications. A missing piece has been bridging these two worlds using Scala. We built Airframe RPC, a framework that uses Scala traits as a unified RPC interface between servers and clients. With Airframe RPC, you can build HTTP/1 (Finagle) and HTTP/2 (gRPC) services just by defining Scala traits and case classes. It simplifies web application design as you only need to care about Scala interfaces without using existing web standards like REST, ProtocolBuffers, OpenAPI, etc. Scala.js support of Airframe also enables building interactive Web applications that can dynamically render DOM elements while talking with Scala-based RPC servers. With Airframe RPC, the value of Scala developers will be much higher both for frontend and backend areas.

1. © 2020 Treasure Data 1
Taro L. Saito
Treasure Data
November 2nd, 2021
ScalaCon 2021
Unifying Frontend and Backend
Development with Scala

2. Copyright 1995-2020 Treasure Data. All rights reserved.
About Me: Taro L. Saito
2
● Treasure Data
○ Principal software engineer
○ Building distributed query engine services
● OSS activities
○ Using Scala for 13+ years
■ Since Scala 2.7 (2008)
○ Airframe (This talk)
■ 20+ Scala utility modules
○ sbt-sonatype
■ Publishing to Maven central (Sonatype)
○ etc.
Airframe

3. Copyright 1995-2020 Treasure Data. All rights reserved.
Treasure Data: A Ready-to-Use Cloud Data Platform
3
Logs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS
Data
● Even if you are not an engineer, you can start collecting and analyzing your data

4. Copyright 1995-2020 Treasure Data. All rights reserved.
Even Backend Engineers Need Frontend Skills
● Creating a better visualization of services
○ To monitor millions of queries running every day
○ Created a UI for service monitoring and cluster management with Scala.js
4

5. Copyright 1995-2020 Treasure Data. All rights reserved.
Scala.js Application: Notebook Query Editor
● Rich SQL editor
● Frontend
○ Scala.js
○ Monaco editor of Visual Studio
Code
● Backend
○ Scala on JVM
○ RPC server
■ SQL runner
■ Task manager
● Written only in Scala
5

6. Copyright 1995-2020 Treasure Data. All rights reserved.
Too Many Frontend and Backend Frameworks
● How do we choose one of the frameworks to use?
● It’s almost impossible to find Scala engineers mastering all of these frontend and backend
technologies.
6
Web Browsers
Scala.js

7. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe RPC
● Using plain Scala as an RPC interface between frontend and backend applications
● Low-learning cost
○ Requires only basic Scala knowledge, which can be learned with one Scala book
● Our experience so far after introducing Airframe RPC:
○ A Java engineer in Treasure Data can start writing a Scala RPC service interacting with a
Scala.js UI in a week.
7
RPC Interface
Scala.js Client RPC Web Server
Generates
Scala.js
Web Application
API Documentation
RPC Call RPC Call
Implements
Airframe

8. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe
● GitHub: wvlet/airframe
● 20+ useful modules written in pure Scala
○ Cross-built for Scala 2.12, 2.13, and Scala.js
○ Scala 3 support is in progress
■ 85%~ completed
■ WIP:
● Tasty reader & Scala reflection
compatibility
● Core Modules (*: will be explained in this talk)
○ airframe-log*
○ airframe-codec*
■ Object serializer
○ airframe-http*
■ HTTP server/client with RPC support
○ airframe-rx*
■ Reactive DOM rendering
○ airframe-json/msgpack
■ JSON/MsgPack parser
○ airframe-di
■ Constructor injection helper
○ airframe-surface
■ Compile-time or runtime object inspector
○ AirSpec: Testing library integrated with Airframe DI
○ etc.
8
Airframe

9. Copyright 1995-2020 Treasure Data. All rights reserved.
● Various internal and third-party Scala/Java libraries were used
○ e.g., Many web frameworks, utilities, testing frameworks, etc.
● The cost of learning individual libraries and frameworks was high
Before Airframe: 5 Years Ago (2016)
9
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming Various Libraries Outcome
logger
launcher
object mapper
JDBC reader
json4s jackson
….

10. Copyright 1995-2020 Treasure Data. All rights reserved.
Current (2021): Consolidating the Best Practices into Airframe OSS
● Since 2016, we started defining standards in Airframe
○ Engineers can focus on developing applications instead of learning frameworks
10
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming OSS Outcome
Airframe

11. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-log: A Modern Logging Library for Scala and Scala.js
● There were too many choices just for generating human-readable logs
○ log4j, slf4j, scala-logging, java.util.logging, twitter/util, etc.
● airframe-log
○ Adding colorful logging methods to your class with LogSupport trait
○ Showing the source code location
11

12. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-log usage in sbt-sonatype
● Shows detailed logs while publishing projects to Maven Central (Sonatype OSS Repository)
● News: sbt-sonatype update
○ Since 2019, sbt-sonatype introduced a new bundle upload process, which can upload thousands of
artifact files (jars, source files) to Sonatype within a minute.
12

13. Copyright 1995-2020 Treasure Data. All rights reserved.
● Scala.js compiles Scala code into JavaScript, which runs on web browsers
● Logging for the developer console of web browsers
airframe-log: Debugging Scala.js Code
13

14. Copyright 1995-2020 Treasure Data. All rights reserved.
Frontend Development: Rendering DOM with Scala.js
● DOM element:
○ <div class=”container”> …. </div>
■ tag, attribute, enclosed elements
○ scalajs-dom library has all necessary functions for manipulating DOM nodes
● A previous approach for DOM rendering:
○ Converting Scala’s XML literal into DOM tree nodes
■ OSS: monadic-html, Binding.scala
■ Similar to JSX in Facebook React (XML syntax embedded to JavaScript)
○ But, XML literal will be deprecated in Scala 3
14

15. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-rx: Functional DOM Rendering
● The current best practice is defining
functions for rendering HTML elements
○ table, tr, td, body, div, span, etc.
● Then, compose these functions
● Style 1: Separating DOM attributes and
elements into different blocks
○ ScalaTags, Slinky
○ Some disadvantages
■ Too many parentheses!
■ Formatting with scalafmt is a bit
challenging
● Style 2: Enclosing both DOM attributes
and elements inside a single block
○ airframe-rx, scalajs-react
○ More Scala-friendly syntax
○ The begin/end of a tag are clear
15
ScalaTags
airframe-rx

16. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-rx: Rendering Complex DOM
● RxElement interface:
○ def render(): RxElement
○ Support nesting with apply(...)
■ Embedding Scala functions and collections
16

17. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-rx: Reactive Stream Interface Rx[A]
● Rx[A]
○ ReactiveX stream implementation
● Rewriting DOM interactively upon upstream data changes
○ e.g., mouse clicks, getting RPC responses (Future), interval
timer events, etc.
● Reactive Operators
○ map, flatMap, filter, join, zip, etc.
○ The usage is almost the same with Scala collection APIs
17
Airframe

18. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-rx: Rx.variable
● Rx.variable
○ Fire an event if the variable is updated
● Interactive DOM Rendering
○ Subscribing upstream changes
■ Update DOMs for each change
○ Cancellation
■ Unsubscribe the event processing
■ Necessary for cleaning up event
handlers added to the browser
● mouse click handler, timer, etc.
18

19. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-rx: Gallery
19

20. Copyright 1995-2020 Treasure Data. All rights reserved.
RPC: Bridging Scala and Scala.js Applications
Program Function
Call
Return
Local Function Call
(ideal)
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Remote Procedure Call
(reality)
Network
Client
Call
Return
Server
Call
Return
20
Scala.js
Scala.js

21. Copyright 1995-2020 Treasure Data. All rights reserved.
Traditional Approaches for Implementing RPC
● REST API
○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.)
○ REST web frameworks for Scala
■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc.
● gRPC
○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language
○ Generate server and client code stubs from proto files
○ Scala wrappers:
■ ScalaPB, muScala, akka-grpc, etc.
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Return
Server
Call
Return
21

22. Copyright 1995-2020 Treasure Data. All rights reserved.
New Approach: Defining RPC Interface with Scala
● Scala is a perfect fit for RPC
○ Scala functions = RPC methods
○ Scala objects (statically typed) = RPC request/response types
● Using the same interface and model classes between servers and clients
○ No need to think about REST nor ProtocolBuffers for implementing RPC
22
Airframe

23. Copyright 1995-2020 Treasure Data. All rights reserved.
Necessary Building Blocks for Implementing RPC
● 1. Message serializer and deserializer
● 2. Network data format
○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC)
● 3. RPC interface language
○ REST API, Protobuf, or Scala
● 4. HTTP client and server implementation
○ Code generator from RPC interface
23
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Return
Server
Call
Return

24. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-codec: MessagePack-Based Object Serialization
● MessagePack
○ A compact binary format compatible with
JSON
Object Object
Pack Unpack
Pack
Unpack
Server Side
Client Side
24
Scala.js
JSON

25. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-codec: Predefined Codecs
● Primitive Codec
○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
○ FloatCodec, DoubleCodec
○ StringCodec
○ BooleanCodec
○ TimeStampCodec
● Collection Codec
○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec
○ OptionCodec, EitherCodec
● Java-specific Codec
○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec
○ JDBCResultSetCodec
● etc.
25
Airframe

26. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-codec: Serializing Complex Objects
Pack
Unpack
IntCodec
StringCodec
DoubleCodec
MessageCodec.of[A]
26
Serialize
Deserialize
JSON
● airframe-surface
○ Inspecting object parameters using compile-time macros
○ Different implementations between Scala 2 and Scala 3

27. Copyright 1995-2020 Treasure Data. All rights reserved.
Object Serialization for Multiple Data Formats
● Data to Object Mapping
○ We need many data readers as well as object mappers
● A lot of existing libraries for parsing JSON
○ circe, json4s, play-json, jackson, etc.
YAML
JDBC
ResultSet
YAML Parser +
Object Mapper
Config
Object
Table
Object
Object-Relation
Mapper
JSON
JSON Parser +
Object Mapper
Object
27

28. Copyright 1995-2020 Treasure Data. All rights reserved.
airframe-codec: Using MessagePack As A Universal Data Format
Object
Unpack
Pack
JDBC
ResultSet
Pack/Unpack
YAML
JSON
28
● Once your data are converted into MessagePack, you can use the same object
serializer/deserializer (airframe-codec)
● Airframe RPC supports JSON format through MessagePack conversion
Airframe
airframe-codec

29. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe RPC: Serializing Function Call Request
● Serialize function call arguments as Map (arg_name -> arg_value)
● An example:
○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”)
■ Serialize this Map into MessagePack with airframe-codec
● Mapping to HTTP requests
○ HTTP method: POST
○ Path: /(package name).(class name)/(method name)
■ e.g., /hello.api.v1.MyService/hello
○ Content body: serialized function call arguments
29

30. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe RPC: Implementing RPC Servers
● Extending the RPC interface trait
30

31. Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe RPC: Backend Servers/Clients Are Pluggable
RPC Interface RPC Web Server
Generates API Documentation
31
Airframe
RPC Call RPC Call
Implements
● Finagle (HTTP/1)
● gRPC (HTTP/2)
RPC Clients

32. Copyright 1995-2020 Treasure Data. All rights reserved.
● Finagle: Twitter’s HTTP server written in Scala
○ For HTTP/1 services accessible from web browsers running Scala.js code
AIrframe RPC: Finagle Backend
32
Airframe
Scala.js
airframe-rx-html
airframe-http-finagle

33. Copyright 1995-2020 Treasure Data. All rights reserved.
● gRPC
○ grpc-java (using Netty as backend)
● Airframe RPC + gRPC
○ No .proto file is required
○ Only Scala
Airframe RPC: gRPC Backend
33
Airframe

34. Copyright 1995-2020 Treasure Data. All rights reserved.
RPC Performance Comparison: Greeter Service
● Airframe RPC
○ serde: MessagePack (airframe-codec) -
Scala case classes
○ Finagle (HTTP1) or gRPC (HTTP2)
● ScalaPB
○ serde: Protobuf - Scala case classes
○ gRPC (HTTP2)
● grpc-java
○ serde: Protobuf - Java classes
○ gRPC (HTTP2)
● Notes
○ Using Finagle with HTTP2 had almost no
benefit over HTTP1
○ Multiplexing RPC requests into a
single connection is the key
performance factor
○ Overhead of ScalaPB over grpc-java
■ Scala Future (10% overhead)
■ Mapping Protobuf to Scala case
classes (10% overhead)
34
HTTP/1 HTTP/2 (gRPC)

35. Copyright 1995-2020 Treasure Data. All rights reserved.
[Advanced] Extending gRPC for Other Data Formats
● gRPC is data-format agonistic
framework
○ You can use other than
ProtocolBuffers
■ e.g., JSON, MessagePack
● There are two extension points:
● MethodDescriptor
○ Define gRPC endpoints
corresponding to Scala functions
○ Register custom marshallers
● Request/ResponseMarshaller
○ Define how to encode/decode RPC
request/response data
○ We are using airframe-codec here
35

36. Copyright 1995-2020 Treasure Data. All rights reserved.
sbt-airframe: Generating RPC Clients
● sbt-airframe plugin
○ Read RPC interface classes and generate HTTP client code for the target backend
■ Using different code generators for Scala, Scala.js, and gRPC
sbt-airframe
Code
Generation
RPC Client
Scala.js
36
Scala.js Client
HTTP/gRPC Client
Open API Spec
Cross-Language
RPC Client

37. Copyright 1995-2020 Treasure Data. All rights reserved.
sbt-airframe: Generating Open API Schema
● RPC Interface -> Open API schema YAML file
● Generating RPC Clients & Swagger Documentation
37
RPC Interface
API Documentation
Open API Spec
(YAML)
Cross-Language
RPC Client
Generate
sbt-airframe
API Documentation

38. Copyright 1995-2020 Treasure Data. All rights reserved.
Making An RPC Call
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Response
Server
Call
Response
Airframe
Scala.js
38

39. Copyright 1995-2020 Treasure Data. All rights reserved.
Summary: Enable Scala-Oriented Development with Airframe
39
RPC Interface
Scala.js Client RPC Web Server
Generates
Scala.js
Web Application
API Documentation
RPC Call RPC Call
Implements
Airframe
● Kick start frontend/backend application development with Scala
○ Only one language to learn
○ No need to think about REST, ProtocolBuffers, other frontend frameworks at the beginning.
● Find example code at https://github.com/wvlet/airframe/ (examples folder)