Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, and DSE Graph

Creating a Python Microservice
Tier in Four Sprints with
Cassandra, Kafka, and DSE
Graph
Jeff Carpenter, Director of Developer Advocacy, DataStax
community.datastax.com | @jscarp

© DataStax, All Rights Reserved.
Prhotoshopping
by Steve
Halladay

© DataStax, All Rights Reserved.© DataStax, All Rights Reserved.
Language of
the year 2018
https://www.tiobe.com/tiobe-index/

KillrVideo – A video sharing application
https://github.com/KillrVideohttps://killrvideo.github.io

KillrVideo High Level Architecture
KillrVideo
Services
Your Browser
Web Application
Technology Choices
• Node.js
• Falcor
• Java / C# / Node.js / Python
• GRPC
• Etcd
• DataStax Drivers
• DataStax Enterprise
including Apache Cassandra
& Spark, Graph
Deployment
• Download and run locally
via Docker
• Deployed in AWS using
DataStax Managed
Services:
http://killrvideo.com/

KillrVideo Python Services – Sprint 1
User
Management
Service
Web Application
Video
Catalog
Service
Comments
Service
Rating
Service
Search
Service
Statistics
Service
Suggested
Videos
Service
Generator
Integration
Tests

KillrVideo Python Services – Sprint 2a
User
Management
Service
Web Application
Video
Catalog
Service
Comments
Service
Rating
Service
Search
Service
Statistics
Service
Suggested
Videos
Service
Generator
Integration
Tests

KillrVideo Python Services – Sprint 2b
User
Management
Service
Web Application
Video
Catalog
Service
Comments
Service
Rating
Service
Search
Service
Statistics
Service
Suggested
Videos
Service
Generator
Integration
Tests

User
Management
Service
Web Application
Video
Catalog
Service
Comments
Service
Rating
Service
Search
Service
Statistics
Service
Suggested
Videos
Service
Generator
Integration
Tests

KillrVideo Python - Givens
• Overall Architecture
• Infrastructure Selection
• User Interface Design / Web Application
• Service Design / APIs
• Data Models - CQL and Gremlin
• Infrastructure Configuration / Docker Compose
• Integration tests
• Logo

WEB
WEB Client
bootstrap react redux falcor
Service
Registry
KillrVideo Services
CASSANDRA
Microservices
(Service
requests)
9091
SEARCH
GRAPH
ANALYTICS
DRIVERS
9042
2379
registrator
3000
Register videos
Killrvideo Integration
Test client
(cucumber)
WEB Server
falcor
Component
Runtime
Customer
Browser
GENERATOR
Killrvideo-generator

+ +

Service Interfaces?
• Pro
– Great for cross
language support
– Stub generation
– Compact
• Con
– Generation with
common files
– Debugging
• Pro
– Easy to understand
– Easy to test (curl)
• Con
– HTTP Status codes
– Consistency
• Pro
– Great at composing
across multiple
smaller services
• Con
– Overkill for CRUD
style services?

User Management Service – Protobuf service definition
https://github.com/KillrVideo/killrvideo-service-protos/blob/master/src/user-management/user_management_service.proto

GRPC / Protobuf to Python Source
https://github.com/KillrVideo/killrvideo-service-protos/tree/master/src/user-management
user_management_service_pb2.py
user_management_service_pb2_grpc.py
user_management_service_grpc.py
user_management_service.py
Protobuf
definition
Generated
stubs
GRPC-specific
code
Service
implementation
user_management_service.proto
Python grpc_tools
Extend
https://github.com/KillrVideo/killrvideo-python/tree/master/killrvideo/user_management
Call

Initializing GRPC Server / registering services in etcd (__init__.py)
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/__init__.py

Success! Integration tests all fail!
https://github.com/KillrVideo/killrvideo-integration-tests

Sprint 1 – The Plumbing
• Results:
– Services up and running, exposed via GRPC, registered in etcd, all integration tests FAILING!
• Lessons Learned:
– Decouple all the things
• Service interfaces, service implementations, service discovery, properties, etc.
– Multiple service interface styles 👍
– Consider monorepo for new work
– The levels of testing blur as services get smaller
• Microservices, serverless…
• Component testing vs. unit testing
– Vertical slices FTW!

+ +
++

Application Workflow in KillrVideo
User Logs
into site
Show basic
information
about user
Show videos
added by a
user
Show
comments
posted by a
user
Search for a
video by tag
Show latest
videos added
to the site
Show
comments for
a video
Show ratings
for a video
Show video
and its details

Queries in KillrVideo to Support Workflows
Users
User Logs into
site
Find user by email
address
Show basic
information
about user
Find user by id
Comments
Show
comments for
a video
Find comments by
video (latest first)
Show
comments
posted by a
user
Find comments by
user (latest first)
Ratings
Show ratings
for a video Find ratings by video

Designing Tables Based on Queries
Show video
and its details
Find video by id
Show videos
added by a
user
Find videos by user (latest
first)
CREATE TABLE videos (
videoid uuid,
userid uuid,
name text,
description text,
location text,
location_type int,
preview_image_location text,
tags set<text>,
added_date timestamp,
PRIMARY KEY (videoid)
);
CREATE TABLE user_videos (
userid uuid,
added_date timestamp,
videoid uuid,
name text,
preview_image_location text,
PRIMARY KEY (userid,
added_date, videoid)
)
WITH CLUSTERING ORDER BY (
added_date DESC,
videoid ASC);

KillrVideo Schema in DataStax Studio

Implementing Data Access and Business Logic
• Data Access
– Driver installation
– Driver initialization
– Writing and reading data
– Search
• Business Logic
– Input validation
– Email address format check
– Encoding / decoding page state
– Stop word removal
– Password encryption

Initializing DSE Driver and Mapper (__init__.py)
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/__init__.py

Video Catalog Service – Creating Prepared Statements
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/video_catalog/video_catalog_service.py

Video Catalog Service - Model classes used with Mapper

Video Catalog Service – adding a video

Video Catalog Service - Retrieving Latest Videos (paging)

Paging Blog Post
https://academy.datastax.com/content/paging-killrvideo

Search Service – Finding Videos based on Keyword
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/search/search_service.py

Sprint 2 – Data access and business logic
• Results
– DSE DB (Apache Cassandra) integration in multiple services
– DSE Search integration in Search Service
• Lessons Learned
– The Cassandra development methodology works!
• application data flows ➡️ data modeling ➡️ implementation
– Use multiple methods of driver interaction
• Mapper for simple reads/writes
• Prepared statements for paging
– Search is a powerful thing
• Added value beyond just avoiding denormalization
– Friction logging as a time capsule

+ +
+ +
+

KillrVideo Services Suggested
Videos
Service
DataStax Enterprise
DSE Graph
• UserCreated
• YouTubeVideoAdded
• UserRatedVideo • Populate graph
• Graph recommender
traversal
• Read and write
data
User Management, Video
Catalog, Ratings
KillrVideo Services and Events

Kafka / Zookeeper Docker Configuration
https://github.com/KillrVideo/killrvideo-python/blob/master/docker-compose.yaml

Video Catalog Service – Producing YouTubeVideoAdded
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/video_catalog/video_catalog_events_kafka.py

Suggested Video Service – Consuming Events
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_events_kafka.py

Uh oh
https://timber.io/blog/multiprocessing-vs-multithreading-in-python-what-you-need-to-know/

Sprint 3 – Kafka messaging
• Results
– Events produced by Comments Service, Ratings Service, User Management Service, Video
Catalog Service
– Events consumed by Suggested Videos Service
• Lessons Learned
– Example code – buyer beware!

+ +
+ +
+ +

Live Coding on Twitch
• Live coding sessions with advocates and
guests
• Working through the challenges of
building distributed systems
• Join the conversation and ask questions
• Twitch Rewind: KillrVideo Python Graph
Recommender
– https://www.youtube.com/watch?v=jK1J4sb
6qP0
https://www.twitch.tv/datastaxdevs

KillrVideo Graph Schema - Groovy

Adding vertexes to the graph
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_service.py

Generating Video Recommendations
https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_service.py

Sprint 4 – Graph Recommender
• Results
– Graph recommender MVP in Suggested Videos Service
• Lessons Learned
– Graph way of thinking
• One traversal, many steps
– One learning curve at a time
• Graph learning curve ➡️ DataStax Studio
• DataStax Python Driver Graph Extension ➡️ command line
• Then add traversals to app
– New language, new process?
• Scripting languages – use the command line to prototype
– Live coding is like pair programming++

KillrVideo Python – Remaining Work
• Running Python services in Docker / Integration with killrvideo-all-in-one
• Graph recommender optimization
• Authenticating to DSE
• Upgrade to Python 3
• Feedback to docs/driver teams

https://trello.com/b/nlbJQfup

Stepping back - what did I learn??
• Decouple all the things
• Multiple service interface styles 👍
• Consider monorepo for new work
• The levels of testing blur as services get
smaller
• Vertical slices FTW!
• Friction logging as a time capsule
• Example code – buyer beware!
• One learning curve at a time
• New language, new process?
• Live coding is like pair programming++
• The Cassandra development
methodology works!
• Use multiple methods of driver interaction
• Search is a powerful thing
• Graph way of thinking

Getting involved
• Download and run
– All in one
• File an issue / Fix an issue
• Ask a question
• Weekly Coding Twitch Streams
• Coordination – every other Wednesday

Blogs
• KillrVideo Python Pt. 1— the Backstory
• Python App Dev with ProtoBuf and GRPC
• Advertising Python services via etcd
• Who needs unit tests? I’m building microservices!
• When data access is the easiest part of a microservice
• Cassandra with Python: Simple to Complex
• Kafka + Cassandra — like peanut butter and chocolate?
• + Post in progress on graph recommender
– https://medium.com/@jscarp

Thanks / Credits
• Collaborators!
– David Gilardi - @SonicDMG
– Amanda Moran - @AmandaK_Data
• Python driver guru
– Alan Boudreault

THANK YOU

Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, and DSE Graph

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, and DSE Graph

Semelhante a Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, and DSE Graph (20)

Último

Último (20)

Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, and DSE Graph

Notas do Editor