Jean Maynier (Kpler): “Python, SQLalchemy and Scrapy for real-time data processing at Kpler” Bio: I lead the engineering and operational activities of Kpler (formerly eCO2market). Founded in Paris in 2009, Kpler is an intelligence company providing transparency solutions in energy markets. We develop proprietary technologies that systematically aggregate data from hundreds of sources, ranging from logistics and commercial, to governmental and shipping databases. By connecting the dots across fragmented information landscapes, we are able to deliver our clients with unique real-time market coverage.

1. Python, SQLAlchemy
and Scrapy at Kpler
Jean Maynier
CTO

2. Business : commodities markets
Domain: trading / maritime transport
Customers: physical traders / analysts
Needs: follow cargoes flows and prices impact

3. Commercial Database
“Meta” AIS (10 networks)
Ports Authorities
Gas Inventories
Methodology :4 data layers
Thefirst “Cargo-Tracking” system,
bridging thegapbetween LNGShipping
andTrading.

4. Why python ?
Simple, fast to prototype
Data libraries (sqlAlchemy, Pandas, Scikit-learn)
Data oriented community
Scrapy: best scraping framework
Step 1

5. Scrapy
Simplify crawling
Reusable extractors (xpath, css selectors)
PaaS: Scrapinghub
Blacklist: Tor, crawlera for IP rotation
Step 2

6. Process raw data
Clean and normalize data
Positions
Port calls
Contracts
Prices
Python PubSub system
Persist data : sqlAlchemy & Postgresql
Step 3

7. Enrich data
Done
Vessel dock/undock
events
Predict next destination
Find buyer/seller
Basic routing
Todo / WIP
Estimate price
Routing with weather
Portfolio optimisation
Fleet optimisation
Step 4

8. Architecture
Websites sources
API sources
Scraping
Python/scrapy
Cache
Elasticseach
Data
ETL + models
python
Web
REST API
scala
client
mobile client
Excel

9. Metrics trends
Present
DBs < 100Gb
50 sources
500 vessels
Positions every 3min
Future
1-10 Tb
100 sources
10k to 100k vessels
Position every 30s
Performance problem !

10. Batch to data streaming
Parallelization
Granularity : item vs source
Handle failure, no data loss, monitoring
Akka streaming, Spark streaming, Celery, Storm
Step 5

11. Storm at Kpler: POC
Websites sources
API sources
Scraping
Python/scrapy
ETL + models
Position Dock/undock
Custom prices Match prices
Next
destination
Trades
Cargo volumes

12. We are recruiting, join us !
jobs@kpler.com