Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My

Cameron Waeland
Software Engineer
Rethinking CodeGen:
IDL, Thrift, gRPC, Ohh My…

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
compass-rest-grpc

Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

12 Months Ago -
Adding a new API Endpoint
Lots of steps
and coordination
across the stack
Difﬁcult to
enforce standards
Proliferation
of similar APIs

Old API
Framework Issues
• Business logic spread
across Python API and
Java backend services
• Maintenance difﬁcult for
developers
• Python API coupled to
backend
• Numerous ad hoc
endpoints
• Adding an HTTP API falls
to service developer
INTRODUCTION

Fast Forward 12 Months
Add a method  
to the relevant
Thrift IDL.
I'm a Compass developer and I want to add an endpoint:
1.
Implement the
method in a
backend
service.
2.
Run a
command.
3.
Get a beer.
4.
a. Generate APIs
b. Generate SDKs
c. Generate Documentation

Cameron Waeland
14°32´55˝ N, 20°59´35˝ E
Software Engineer APIs
• Leads API development at Compass
• Developer frameworks and tooling
• Service development/architecture

COMPASS COMPANY OVERVIEW |
2017
Our
Nationwide
Network
1600+
Agents in
30
ofﬁces across
10
major regions
East
West
Boston
New York City
The Hamptons
Washington DC
Miami
San Francisco
Santa Barbara &
Montecito
Los Angeles
Orange County
Aspen

2017
7
Our Technology
The Compass technology suite
empowers agents to become trusted
advisors to their clients, build their
network and grow their business.
We provide value through technology
and data to our agents so they
can help their clients make smart real
estate decisions.
Advanced Search
Agent Mobile App
Listing Strategy
Markets App
Network
Performance Metrics Valuation
Insights
Deal Closer
Documents
Listing Presentation
Showsheets
Toursheets
Efﬁciency
Collections
Open House App
Client Portal
Relationship Building

2017
Drive Toward
Market Leadership
$180M
Compass nearly tripled its
2016 revenue year-over-year to
more than
98%
Our technology and agent support has
led Compass to achieve industry-
leading retention of
#1 #1 TOP
5
Compass is ranked as #1
overall brokerage in Washington
D.C.
Compass is ranked as the #1
brokerage in single-family home
sales in San Francisco
Compass is a top 5 brokerage
by market share in New York,
San Francisco, Los Angeles,
Miami, D.C. and Boston

CodeGen
is a valuable tool
and less difﬁcult
than you think

AGENDA
Birth of a CodeGen Framework
Code Generating an API
Going Further
Thrift to gRPC with CodeGen

What’s a Remote
Procedure Call?
• Interservice
communication over a
network
• Call remote method as
though it were local
• Low overhead, binary
format
• E.g. Thrift, GRPC, Avro
HTTP typically used for
external requests
ADDING GO SUPPORT FOR THRIFT SERVICES

Thrift @Compass
Developed by Facebook,
open sourced via Apache
Interfaces defined using
Interface Description Language
• Define services, methods,
types, etc.
• Inputs to a code
generation tool
• Clients and servers
generated for each
service in a target
language.
• Thrift IDL files are similar
to Protobuf
struct Blog {
1: optional i32 id; // The unique identifier of the blog post
2: optional i32 ownerId; // The userId of the post creator
3: optional string title; // The title of the blog post
4: optional string content; // The content of the blog post
}
struct ViewPostRequest {
1: optional i32 blogId;
}
struct ViewPostResponse {
1: optional base.ResponseStatus status;
2: optional Blog post;
}
struct CreatePostRequest {
1: optional i32 userId; // The user creating the new post
2: optional Blog blog;
}
struct CreatePostResponse {
}
service BlogService {
ViewPostResponse viewPost(1: ViewPostRequest request);
CreatePostResponse createPost(1: CreatePostRequest request);
}
Thrift IDL
• Similar to Protobuf
Blog Service
• Exposes a read
method and a
write method
• Single model
object
• Basic request/
response structs
ADDING GO SUPPORT FOR THRIFT SERVICES

Go Code Generator…
…or a gopher appears
—
Parallel initiative to add Go for backend services.
We were excited by Go’s:
• Concurrency primitives & performance
• Static typing & unambiguous syntax.
But:
• Support in the ofﬁcial Thrift project was poor.
• We needed to build our own generator for Thrift -> Go

Common Code Gen  
Misconceptions
—
• High level of complexity
• Produces messy unreadable code
• Poorly performing code
• Not as good as handwritten code

Thrift to Go Generator
—
• Used go-thrift - Thrift parser
• Library generates Abstract Syntax Trees
• ASTs provide Thrift ﬁle contents in a structured way -
Enums, Structs, Services, Methods, etc.
• Think compilers but not as scary!

Code Generation Pipeline
—
• Built Go template files compatible with the AST
generated.
• With template files and ASTs we were able to render
out idiomatic Go code.
• Generated files are compiled into a compass-thrift
binary for use by services.

AST Model
• Using a Parsing
Expression Grammar
(PEG) the file is broken
down into key sections
• Set of rules for string
recognition (think
regexes)
• Choice operator selects
the first match
• The file is represented as a
tree using Go data
structures.
• Field & method names,
type information become
easily accessible.
BIRTH OF CODEGEN FRAMEWORK

Looking at a Struct
AST
"Blog": {
"Name": "Blog",
"Fields": [
{
"ID": 1,
"Name": "id",
"Type": {"Name": "i32"}
},
{
"ID": 2,
"Name": "ownerId",
"Type": {"Name": "i32"}
},
{
"ID": 3,
"Name": "title",
"Type": {"Name": "string"}
},
{
"ID": 4,
"Name": "content",
"Type": {"Name": "string"}
}
]
}
struct Blog {
1: optional i32 id; // The unique identifier of the blog post
2: optional i32 ownerId; // The userId of the post creator
3: optional string title; // The title of the blog post
4: optional string content; // The content of the blog post
}

Go Templates for
Generation
type Blog struct {
ID *int32 `thrift:"1" json:"id,omitempty" param:"id" `
OwnerID *int32 `thrift:"2" json:"ownerId,omitempty" param:"ownerId" `
Title *string `thrift:"3" json:"title,omitempty" param:"title" `
Content *string `thrift:"4" json:"content,omitempty" param:"content" `
}
type {{$name|Export}} struct {
type {{$name|Export}} struct {
{{range . -}}
{{.Name|Export}} {{if .|ShouldDereference}}*{{end}}{{.|FieldType}} ` +
"`thrift:"{{.ID}}{{if .Type|IsSet}},set{{end}}"" +
" json:"{{.Name}},omitempty"" +
" param:"{{.Name}}"" +
" {{if Injectable .}}inject:"{{(Injectable .)}}"{{end -}}`" + `
{{end -}}
}
}

Dispelling the 
Misconceptions
—
Thrift's clear syntax:
• Straightforward templates
• Create clean and performant code
• Add the features we want

Motivation for a New
API Framework
—
• Monolithic Python app
• New endpoints built ad hoc by frontend devs
• Business logic spread across services and Python
app
• Inconsistent & difﬁcult to maintain

Where did
we start?
• Consistency - validation,
request/response formats,
URL patterns
• Discoverability -
complete, accurate and
generated API
documentation
• Productivity - generated
API clients, insights
(performance, errors,
availability)
CODE GENERATING AN API
Requirements Gathering!
Group discussions/
interviews with client/service
developers.
Key requirements:
Thrift RPC  
Everywhere
Python App
REST Proxy for
Backend Services
Go Code
Generator
Available Compass Tech
During Design Phase

A Common Refrain
—
How does this endpoint work?
Check the Thrift...
What does this parameter do?
Look at the Thrift...
Is there any documentation?
Read the Thrift…
A pattern emerges… Thrift is the source of truth

General Approach
—
• Thrift emphasis - should be source of truth for any API
• Now have access to a Thrift CodeGen framework
• Let's try an API framework generated directly from Thrift
• HTTP to Thrift reverse proxy
• Go Application - simpler to extend already generated code

The Prototypes
—
• 2 reverse proxy prototypes generated from Thrift
• First would be a more traditional JSON REST API
• Second would be a more experimental GraphQL API

What We Chose
—
Decision: JSON REST API
As much as we tried to justify using GraphQL…
• Risk mitigation - easier migration of clients
• Engineer familiarity with REST
• Relative immaturity of GraphQL

Three Key Aspects of an  
API CodeGen Framework
Source of Truth
In our case Thrift, but could be
database model, object model, etc.
1.
Request
Context
2.
Deployment,
Operations &
Developer
Tooling
3.
HTTP, session info, etc. Supporting development
Performance monitoring, high
availability, etc.

1. Source of Truth
—
• Already established that Thrift is our source of truth
• For an HTTP API we were just short of all the info we
would need
• The "Aha!" moment - Thrift annotations

Annotations
• Language feature of Thrift
• Applied to services,
methods, structs and ﬁelds
• Similar to Java - control
parser/compiler behavior
• Specify arbitrary key/value
metadata
• Small modiﬁcation to
parser to support
service BlogService extends base.BaseService {
ViewPostResponse viewPost(1: ViewPostRequest request)
(api.url="/blog/:blogId", api.method="GET");
CreatePostResponse createPost(1: CreatePostRequest request)
(api.url="/blog", api.method="POST", api.roles="Editor");
} (api.url="/blog")
Example
• Top level URL for the service
• Per-method URLs including
resource IDs
• HTTP request methods
• Endpoint permission
requirements

2. Request Context
—
• URL route + HTTP method to service method binding
• Endpoint authorization
• Python API style convenience features
• Current user
• File uploads
• User permissions
• HTTP headers
• etc.

Injectors
• Introduced special Thrift
annotation for struct fields
• API will inject relevant
information into annotated
fields.
• (api.path)
• The value of ":blogId" as
defined in the path will be
injected into this field.
• (api.inject="userId")
• ID of requesting user will
injected into this field.
struct ViewPostRequest {
1: optional i32 blogId (api.path);
}
struct ViewPostResponse {
2: optional Blog post;
}
struct CreatePostRequest {
1: optional i32 userId (api.inject="userId");
}
struct CreatePostResponse {
2: optional i64 result;
}

Key Beneﬁts
—
1. Reduces logic and custom code in Python mono app.
2. Thrift ﬁles are “source of truth” for all API interactions.
3. Consistent & predictable API.
4. Improved developer productivity

Code Generating
Clients
• Improve consistency for
frontend developers
• Clients deﬁne and maintain
their own templates
• Use structure or
frameworks that they want
• API team doesn't need to
maintain them
GOING FURTHER

Code Generating Documentation
—
• We added a documentation template
• The output is RESTful API Modeling Language
(RAML)
• Supports custom extensions and complex objects in
GET.
• Updated our parser to surface Thrift comments
• Hosted and easily accessible to developers

Key Beneﬁts
—
1. Reduces logic and custom code in Python mono app.
2. Thrift ﬁles are “source of truth” for all API interactions.
3. Consistent & predictable API.
4. Automatically generated client code - clients instantly get
backend changes
5. Improved developer productivity

Introducing gRPC @Compass
—
• Increased commitment to microservices
• Started to re-evaluate our usage of Thrift
• Experienced several frustrations:
• Low quality Python Thrift Server
• Reliability of connections
• General lack of Thrift development
• Investigated gRPC by Google after it was open sourced.

gRPC Advantages
—
• gRPC ﬁlls same role as Thrift
• RPC framework for polyglot architectures via CodeGen
• However it also provides:
• Bi-directional streaming using http/2
• Customizability - pluggable auth, tracing, load balancing
and health checking
• Built-in connection retrying
• Efﬁcient, idiomatic clients and servers.

Code Generated Migration
—
• A problem - gRPC recommends protobuf as its IDL
• Compass has a substantial number of Thrift ﬁles
• gRPC pluggability + CodeGen framework
• Migrate and keep our Thrift ﬁles
• Wrote our own Thrift codec which we plugged into gRPC.

gRPC Stub Example
—
Code generated client/server extensions in target languages
public class BlogServiceGRPC extends com.urbancompass.common.base.BaseServiceGRPC {
private static final Empty __EMPTY = new Empty();
private static final MethodDescriptor<CreatePostRequest, CreatePostResponse> __METHOD_DESCRIPTOR_CREATE_POST =
MethodDescriptor.create(
io.grpc.MethodDescriptor.MethodType.UNARY,
generateFullMethodName("BlogService", "createPost"),
new ThriftMarshaller<>(CreatePostRequest.class),
new ThriftMarshaller<>(CreatePostResponse.class)
);
private static final MethodDescriptor<ViewPostRequest, ViewPostResponse> __METHOD_DESCRIPTOR_VIEW_POST =
MethodDescriptor.create(
io.grpc.MethodDescriptor.MethodType.UNARY,
generateFullMethodName("BlogService", "viewPost"),
new ThriftMarshaller<>(ViewPostRequest.class),
new ThriftMarshaller<>(ViewPostResponse.class)
);

Where are we today?
Go API + gRPC active
in production -
mobile/web clients
Go API is
deployed as a
single service
Backend service
changes require
regen and
deployment

New Architecture
—
• All requests reverse proxied through Go API
• App services are annotated and available to clients
• Core/Internal services only accessed via App services

Where should we go next?
Future goal - deploy
an instance of API v3
per service
Service change only
affects its API v3
instance to regen
Each Go API instance
registers with load
balancer/reverse proxy

Other Future Opportunities
—
Robust JavaScript (or TypeScript) SDK
• Leverage Thrift type information in our JS
• Give our developers convenient static analysis
Moving to Protobuf
• Protobuf is the standard IDL format for gRPC
• Standardize our stack - leverage more of the ecosystem
 
Evaluating GRPC-WEB
• New protocol - web clients speak with gRPC using JSON over HTTP
• Eventually - browsers can use native gRPC over HTTP/2 via
upcoming whatwg fetch/streams API

Wrapping Up
• CodeGen is a valuable tool
• Cut 2 major time consuming steps for our devs
• We migrated to a new RPC framework
• Generated an API from a single source of truth
• It is easier than you think
• Straightforward to build
• Easy to extend to new applications
CodeGen at Compass has allowed us to rapidly transform our
stack over the last year.

Rapidly Growing and Hiring for multiple positions
www.compass.com/careers/jobs/
Thanks!

Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
compass-rest-grpc

Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My

Recommended

Recommended

More Related Content

More from C4Media

More from C4Media (20)

Recently uploaded

Recently uploaded (20)

Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My