O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Generating Unified APIs with Protocol Buffers and gRPC

1.285 visualizações

Publicada em

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2k1eZ00.

Chris Roche and Christopher Burnett discuss how they extended the Protocol Buffer (PB) IDL to create unified APIs and data models. From validation logic to automatic logging and statistics, PBs allows them to speed up development across Go, Python, and PHP stacks. They also show how they leverage the Envoy proxy to transparently upgrade HTTP/1.1 services to speak gRPC on the wire. Filmed at qconnewyork.com.

Chris Roche is a Core Libraries Engineer at Lyft focused on enriching the tools to create distributed systems. Christopher Burnett is Lead Core Libraries Engineer at Lyft, where he designs software for distributed systems.

Publicada em: Tecnologia

Generating Unified APIs with Protocol Buffers and gRPC

  1. 1. TYPES ALL THE WAY DOWN GENERATING APIS WITH PROTOBUFS AND GRPC
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ api-pb-grpc
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. HI! Christopher Burnett - Lead Core Libraries Formerly Twitter, VSCO @twoism Chris Roche - Core Libraries Formerly VSCO @rodaine
  5. 5. CORE LIBRARIES A little about us… • HTTP & gRPC Frameworks • Tooling for Go, Python, and PHP • Rolling out gRPC within Lyft
  6. 6. AGENDA • Why choose RPC in the first place? • Working with legacy systems • Disrupting workflows, nicely :) • What we’ve built
  7. 7. EVERY TEN YEARS… A furious bout of language and protocol design takes place and a new distributed computing paradigm is announced that is compliant with the latest programming model. - A Note On Distributed Computing, Waldo 1994
  8. 8. LOOK FAMILIAR? • CORBA • Thrift • SOAP • WDDX • JSON-RPC • XML-RPC • Avro • HyperMedia • REST • MessagePack
  9. 9. A LITTLE HISTORY Like any good story we begin with a PHP monolith… • Active decomp efforts • 100s of Python Microservices - Flask HTTP/REST And to keep things interesting… • gRPC Core and Compositional Services
  10. 10. DEFINING A CORE SERVICE • Organizational Primitives - User, Rides, Vehicles • Zero (Service) Dependencies - Databases, caches, etc • Highly Performant
  11. 11. SO, WHY GRPC?
  12. 12. LET’S TALK ABOUT REST
  13. 13. RESTful
  14. 14. RESTish
  15. 15. REST/JSON: S2S COMMUNICATION POST /api/updateUser HTTP/1.0 Content-Type: application/json { "id": 18446744073709551615, "username": “chris" }
  16. 16. ALRIGHT, LET’S PAINT THAT SHED… PUT /api/users HTTP/1.0 Content-Type: application/json { "id": 18446744073709551615, "username": “chris" }
  17. 17. PUTTING ON ANOTHER COAT… PUT /api/users/18446744073709551615 HTTP/1.0 Content-Type: application/json { "username": “chris" }
  18. 18. FINISHING TOUCHES… PUT /api/v1/users/18446744073709551615 HTTP/1.0 Content-Type: application/json { "username": “chris" }
  19. 19. IDLs are pretty great :)
  20. 20. IDLS ARE PRETTY GREAT • Single Source of Truth -Primitive definitions • Code Generation -APIs, Clients, Servers, Data Models, Docs, Observability • Extensibility -Plugins for everything else
  21. 21. IDL SERVICE DEFINITION package lyft.service.users.v1 service Users { rpc Update(UpdateRequest) UpdateResponse; } message UpdateRequest { uint64 id = 1; string name = 2; }
  22. 22. What about existing services?
  23. 23. IDL SERVICE DEFINITION — HTTP package lyft.service.users.v1 service Users { option (http.http_server_options).isHttpServer = true; rpc Update(UpdateRequest) returns UpdateResponse { // Override `path` for legacy URL support option (http.http_options).path = "/api/v1/users/:id"; option (http.http_options).method = "PUT"; } }
  24. 24. TYPES ON THE WIRE • Simplified API I/O - Structs In, Structs Out • Safety - Big wins for dynamic languages • Transfer Cost - Improved latencies
  25. 25. ENVOY TOPOLOGY IMAGE
  26. 26. Types on the wire eliminate an entire class of errors
  27. 27. TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'
  28. 28. HTTP/2.0
  29. 29. HTTP/2.0 • Full Duplex Streaming • Binary Transport • Push • Header Compression
  30. 30. WHAT’S NOT SO GREAT?
  31. 31. Introducing a new protocol or language can be highly traumatic for teams.
  32. 32. “How do I cURL this?”
  33. 33. WHAT CAN MAKE THIS BETTER? • Incremental Adoption -Allow teams to opt-in to the new shiny things • Familiarity -Tooling that feels welcoming -Standardized framework patterns • Roll Forward -Wire format first, then the protocol and frameworks
  34. 34. How can we make protocol and infra changes flexible and transparent?
  35. 35. Service Mesh ENVOY TOPOLOGY Internet Clients “Front” Envoy Legacy PHP Monolith Envoy Go Services Python Services Envoy Envoy MongoDB / DynamoDB Ratelimiting Stats / Tracing Discovery
  36. 36. SIDECAR?
  37. 37. ENVOY: GRPC BRIDGE FILTER Python HTTP Envoy Envoy HTTP/1 POST /GetUser 200 OK gRPC gRPCBridge
  38. 38. ENVOY: JSON PROXY FILTER Python HTTP Envoy Envoy HTTP/1 POST /GetUser 200 OK gRPC JSONFILTER
  39. 39. With Envoy, services and clients can evolve independently.
  40. 40. *Next Chris
  41. 41. How can we leverage IDLs beyond the API?
  42. 42. ODIE: IDLS MEET THE DATASTORE Type-Safe Repository Driver-Agnostic Client Expression Engine Decorator Middleware Database Driver IDL-Based Model Lifecycle Events MongoDB DynamoDB Spanner BoltDB
  43. 43. ODIE: MODELS AS PROTOCOL BUFFERS message User { option (odie.mongo).enabled = true; string id = 1 [(odie.mongo).primary = true, (odie.type).object_id = true]; string name = 2 [(odie.mongo).name = "username"]; int64 date = 3 [(odie.type).datetime = true]; uint32 vers = 4 [(odie.locking).revision = true]; }
  44. 44. ODIE: MODELS AS PROTOCOL BUFFERS type UserModel struct { Id bson.ObjectId `bson:"_id"` Name string `bson:"username"` Date time.Time Vers uint32 } func (pb *User) ToModel() *UserModel func (m *UserModel) ToProto() *User
  45. 45. ODIE: TYPE-SAFE REPOSITORIES type UserRepo interface { Events() *Events Get(ctx context.Context, id bson.ObjectId) *GetBuilder Put(ctx context.Context, m *UserModel) *PutBuilder Delete(ctx context.Context) *DeleteBuilder Update(ctx context.Context) *UpdateBuilder Query(ctx context.Context) *QueryBuilder }
  46. 46. ODIE: THE PLATONIC IDEAL • Zero/Empty value ≅ Nil value - False is semantically the same as null • Fields are not polymorphic - All attributes are singularly typed • Models are rigidly typed - Legacy data is already mappable
  47. 47. ODIE: THE REALITY • Zero/Empty value ≅ Nil value - False is semantically the same as null • Fields are not polymorphic - All attributes are singularly typed • Models are rigidly typed - Legacy data is already mappable • true and false …and nil - Allow proto2 syntax models • Need to decomp from existing PHP Doctrine models - Move source of truth without breaking legacy code. • Covariant polymorphic repeated discriminator maps - Enough said.
  48. 48. “How do you change the wheels of a moving train?”
  49. 49. MOVE THE SOURCE OF TRUTH Entity UserModel Schema Computed Properties Behavior Entity UserModel Computed Properties Behavior UserFields Schema Codegen from the IDL
  50. 50. MOVE THE SOURCE OF TRUTH Users Service Envoy Legacy PHP Envoy m.ToProto() pb.ToModel() OdieDoctrine Adapter Odie Repo Legacy ModelPB / gRPC Fields Class
  51. 51. Reduce trauma with transitional tooling
  52. 52. PYTHON ENVOY-GRPC CLIENTS // Envoy-gRPC Client class UsersClient(…): def get(self, request) def multiget(self, request) def update(self, request) // Official gRPC client class UsersServiceClient(…): def get(self, request) def multiget(self, request) def update(self, request)
  53. 53. Listen to your customers
  54. 54. “Protobufs are really hard to use”
  55. 55. OFFICIAL PYTHON MESSAGES User = _reflection.GeneratedProtocolMessageType('User', (_message.Message,), dict( DESCRIPTOR = _USER, __module__ = 'pb.lyft.user_pb2' )) _sym_db.RegisterMessage(User)
  56. 56. PYTHON MESSAGE PROXIES class UserProxy(object): def __init__(self, base_pb): self._base = base_pb @property def id(self): return self._base.id @id.setter def id(self, value): self._base.id = value
  57. 57. FURTHER CODE GENERATION • Go/PHP Envoy-aware clients • PB over HTTP clients/server • Observability interceptors • Ergonomics helpers • Response caching • CLI
  58. 58. That’s an awful lot of codegen…
  59. 59. PROTOC-GEN-STAR (PG*) Code generation framework
 • AST of Lyft primitives • Simplifies code generation • Highly testable Package File Service Method Enum EnumValue Message OneOf Field
  60. 60. PG*: WALK THE AST type Visitor interface { VisitPackage(Package) (v Visitor, err error) VisitFile(File) (v Visitor, err error) VisitMessage(Message) (v Visitor, err error) VisitEnum(Enum) (v Visitor, err error) VisitEnumValue(EnumValue) (v Visitor, err error) VisitField(Field) (v Visitor, err error) VisitOneOf(OneOf) (v Visitor, err error) VisitService(Service) (v Visitor, err error) VisitMethod(Method) (v Visitor, err error) }
  61. 61. How far can we take this?
  62. 62. SERVICE GENERATION Srv Framework HTTP Server gRPC Server Debug API App-Level Config Users Odie Gizmo Odie HTTP Service Config Users Service ConfigModel Repositories HTTP Handlers Users Handlers
  63. 63. FUTURE TOOLS Linting & Static Analysis • Enforce best practices • Protect production code • Core Libraries ≠ IDL Police Mocks & Test Fixtures • Scenarios of valid state • Reduce reliance on integration tests • Developer confidence gRPC on Mobile • Reduced payload size • Leverage streaming APIs • Global consistency
  64. 64. The incremental march continues…
  65. 65. Having an ideal is good
  66. 66. Awareness of reality is essential
  67. 67. THANKS! Christopher Burnett - Lead Core Libraries @twoism Chris Roche - Core Libraries @rodaine
  68. 68. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/api-pb- grpc

×