PyData Paris 2015 - Track 4.2 Jean Maynier

•

0 likes•1,258 views

Jean Maynier (Kpler): “Python, SQLalchemy and Scrapy for real-time data processing at Kpler” Bio: I lead the engineering and operational activities of Kpler (formerly eCO2market). Founded in Paris in 2009, Kpler is an intelligence company providing transparency solutions in energy markets. We develop proprietary technologies that systematically aggregate data from hundreds of sources, ranging from logistics and commercial, to governmental and shipping databases. By connecting the dots across fragmented information landscapes, we are able to deliver our clients with unique real-time market coverage.

Data & Analytics

Python, SQLAlchemy
and Scrapy at Kpler
Jean Maynier
CTO

Business : commodities markets
Domain: trading / maritime transport
Customers: physical traders / analysts
Needs: follow cargoes flows and prices impact

Commercial Database
“Meta” AIS (10 networks)
Ports Authorities
Gas Inventories
Methodology :4 data layers
Thefirst “Cargo-Tracking” system,
bridging thegapbetween LNGShipping
andTrading.

Why python ?
Simple, fast to prototype
Data libraries (sqlAlchemy, Pandas, Scikit-learn)
Data oriented community
Scrapy: best scraping framework
Step 1

Scrapy
Simplify crawling
Reusable extractors (xpath, css selectors)
PaaS: Scrapinghub
Blacklist: Tor, crawlera for IP rotation
Step 2

Process raw data
Clean and normalize data
Positions
Port calls
Contracts
Prices
Python PubSub system
Persist data : sqlAlchemy & Postgresql
Step 3

Enrich data
Done
Vessel dock/undock
events
Predict next destination
Find buyer/seller
Basic routing
Todo / WIP
Estimate price
Routing with weather
Portfolio optimisation
Fleet optimisation
Step 4

Architecture
Websites sources
API sources
Scraping
Python/scrapy
Cache
Elasticseach
Data
ETL + models
python
Web
REST API
scala
client
mobile client
Excel

Metrics trends
Present
DBs < 100Gb
50 sources
500 vessels
Positions every 3min
Future
1-10 Tb
100 sources
10k to 100k vessels
Position every 30s
Performance problem !

Batch to data streaming
Parallelization
Granularity : item vs source
Handle failure, no data loss, monitoring
Akka streaming, Spark streaming, Celery, Storm
Step 5

Storm at Kpler: POC
Websites sources
API sources
Scraping
Python/scrapy
ETL + models
Position Dock/undock
Custom prices Match prices
Next
destination
Trades
Cargo volumes

We are recruiting, join us !
jobs@kpler.com

Viewers also liked

Have you heard "Python" being thrown around a bit more lately? You weren't mistaken. Python Programming Language, who deems itself as self-governing and slow-moving, has ironically reached celebrity status in countless disciplines. Brian Ray, well known for building the Python community, will give a fair assessment on: who uses Python, where this came from, what all the fuss is about, and what to expect to see next.

Python Hype?

Brian Ray

Using Morphlines for On-the-Fly ETL

Cloudera, Inc.

Wellcome to python

Nanra Sukedy

Bringing Down the House - How One Python Script Ruled Over AntiVirus

CTruncer

Speaker:  Althea Rizzo, Geologic Hazards Program Coordinator, Oregon Emergency Management Public education has consistently been shown to be an effective emergency preparedness and mitigation strategy when done well. This presentation will give the attendees an effective road map to developing, implementing and evaluating a public education and outreach program. Using real world examples, the presenter will show best practices and how to design your outreach campaign to create positive preventative action by the public.

Public Outreach - Running an effective campaign

Partners in Emergency Preparedness Conference

Intro to Python Programming Language

Dipankar Achinta

Building an EmPyre with Python

Will Schroeder

Introduction about Python by JanBask Training

Tic

Bio Data

Impak

CV Issam 2

10 powerful reasons to eat watermelon

Vikas Gupta

Qcl 15-v4 [challenge-no 2]_[imnu]_[shubham gupta]

Shubham Gupta

Derecho de propiedad 4

kristian solorzano hernandez

Bd26359376

IJERA Editor

Viewers also liked (16)

Python Hype?

Using Morphlines for On-the-Fly ETL

Wellcome to python

Bringing Down the House - How One Python Script Ruled Over AntiVirus

Public Outreach - Running an effective campaign

Intro to Python Programming Language

Building an EmPyre with Python

Introduction about Python by JanBask Training

Tic

Bio Data

Impak

CV Issam 2

10 powerful reasons to eat watermelon

Qcl 15-v4 [challenge-no 2]_[imnu]_[shubham gupta]

Derecho de propiedad 4

Bd26359376

More from Pôle Systematic Paris-Region

Short-range wireless communication technologies such as Bluetooth or ZigBee represent an important part of the Internet of Things ecosystem. By design, this category of smart devices has physically limited reachability inside their Wireless Personal Area Network (WPAN) and are not directly compatible with the TCP/IP stack. However, users may need to access them from anywhere at any moment. To address this problem, we design a new application-agnostic approach called RCM (Remote Connection Manager) enabling transparent communication between an application and out-of-range devices. It creates new IoT use cases by seamlessly mixing remote and local devices. We implemented an open-source prototype for Bluetooth Low Energy (BLE) technology on top of Linux and Android BLE stacks and demonstrated its efficiency through experiments performed on real devices.

OSIS19_IoT :Transparent remote connectivity to short-range IoT devices, by Na...

Pôle Systematic Paris-Region

OSIS19_Cloud : SAFC: Scheduling and Allocation Framework for Containers in a ...

Pôle Systematic Paris-Region

On parle d’observabilité des services lorsque ceux-ci exposent des états et métriques internes pour améliorer la disponibilité globale. Qu’en est-il de l’observabilité des infrastructures sur lesquelles ils sont déployés, configurés et maintenus ? Les différents logs (centralisés, agrégés) permettent un bon début d’analyse mais il faut aussi observer les systèmes au fil de l’eau pour tracer chaque changement et les corréler avec le monitoring. Aujourd’hui, ces étapes de configuration IT devraient être prises en charge par les outils de gestion de configuration, qui deviennent la passerelle vers l’observabilité des opérations. Nous montrerons l'intérêt de cette approche pour la gestion IT moderne avec un retour d’expérience sur les challenges de leur mise en place dans Rudder, notre solution libre d’audit et de gestion de configuration en continu.

OSIS19_Cloud : Qu’apporte l’observabilité à la gestion de configuration? par ...

Pôle Systematic Paris-Region

My research is in virtualized infrastructure domain. I aim at minimizing electricity consumption while improving application performance. To achieve the first goal, I work both at the entire datacenter level (by providing better VM placement strategies) and at the physical machine level (by providing better power management policies). Concerning the second goal, I work both at the VM monitor level (for minimizing its overhead) and at the VM's operating system (OS) level (for making it aware of the fact that it is virtualized). In this talk I present two contributions of my research team, one for each objective. The first contribution presents Drowsy-DC, a novel way to reduce data center power consumption inspired by smartphones. The second contribution presents XPV (eXtended Para-Virtualization), a new principle for well virtualizing NUMA machines.

OSIS19_Cloud : Performance and power management in virtualized data centers, ...

Pôle Systematic Paris-Region

L'expérience du développement de CRESON, support pour des objets distants fortement cohérents dans Infinispan, par Etienne Riviere (UCLouvain). Cet exposé présentera des résultats obtenus dans le cadre du projet européen LEADS que j'ai coordonné et où l'entreprise Red Hat était partenaire. Le code produit a été intégré dans le “staging" de la base de données NoSQL Infinispan, et évalué avec un équivalent open source de Dropbox développé par CloudSpaces, un autre projet européen.

OSIS19_Cloud : Des objets dans le cloud, et qui y restent -- L'expérience du ...

Pôle Systematic Paris-Region

OSIS19_Cloud : Attribution automatique de ressources pour micro-services, Alt...

Pôle Systematic Paris-Region

OSIS19_IoT : State of the art in security for embedded systems and IoT, by Pi...

Pôle Systematic Paris-Region

Pointers are a notorious "defect attractor", in particular when dynamic memory management is involved. Ada mitigates these issues by having much less need for pointers overall (thanks to first-class arrays, parameter modes, generics) and stricter rules for pointer manipulations that limit access to dangling memory. Still, dynamic memory management in Ada may lead to use-after-free, double-free and memory leaks, and dangling memory issues may lead to runtime exceptions. The SPARK subset of Ada is focused on making it possible to guarantee properties of the program statically, in particular the absence of programming language errors, with a mostly automatic analysis. For that reason, and because static analysis of pointers is notoriously hard to automate, pointers have been forbidden in SPARK until now. We are working at AdaCore since 2017 on including pointer support in SPARK by restricting the use of pointers in programs so that they respect "ownership" constraints, like what is found in Rust. In this talk, I will present the current state of the ownership rules for pointer support in SPARK, and the current state of the implementation in the GNAT compiler and GNATprove prover, as well as our roadmap for the future.

Osis19_IoT: Proof of Pointer Programs with Ownership in SPARK, by Yannick Moy

Pôle Systematic Paris-Region

Osis18_Cloud : Pas de commun sans communauté ?

Pôle Systematic Paris-Region

Osis18_Cloud : Projet Wolphin

Pôle Systematic Paris-Region

La virtualisation est une technologie mature dont le surcoût est aujourd’hui marginal sur les machines grand public. Néamoins, ce surcoût augmente radicalement pour les machines reposant sur une architecture Non Uniform Memory Access (NUMA), omniprésentes dans les data centers. Les techniques de virtualisation actuelles exploitent mal cette architecture et causent une dégradation des performances des applications allant jusqu’à 700%. Cette présentation détaille les causes de telles dégradations et propose une méthode qui permet la virtualisation efficace d’architectures NUMA. Une évaluation de cette méthode montre qu’il est possible de multiplier par 2 ou plus la performance de 9 des 29 applications testées.

Osis18_Cloud : Virtualisation efficace d’architectures NUMA

Pôle Systematic Paris-Region

Nous présentons une solution Open Source de stockage et d’archivage distribué des données dont l’objectif est la pérennité des données. Il est basé sur le protocole BitTorrent et intègre un haut niveau de redondance, ainsi que d’un mécanisme de régénération automatique des données. Il peut être déployé à grande échelle en LAN et en WAN. Les agents sont compatibles avec des serveurs et postes clients Linux, Window ou Mac OS.

Osis18_Cloud : DeepTorrent Stockage distribué perenne basé sur Bittorrent

Pôle Systematic Paris-Region

Le logiciel est au cœur de notre société numérique et le code source des logiciels contient une part croissante de nos connaissances scientifiques, techniques et organisationnelles, au point d'être devenu désormais une partie intégrante du patrimoine de l'Humanité. La mission de «Software Heritage» est de veiller à ce que cette précieuse masse des connaissances soit collectée, préservée, organisée et mise à la disposition de tous. Construire une telle infrastructure pose des défis importants, à la fois techniques et stratégiques, et nous pouvons tous contribuer à les résoudre.

Osis18_Cloud : Software-heritage

Pôle Systematic Paris-Region

Dans cet exposé, on présentera OMicroB, une machine virtuelle OCaml pour microcontrôleurs à faibles ressources, inspirée des travaux précédents sur le projet OCaPIC. Cette machine virtuelle, destinée à être exécutée sur diverses architectures matérielles (AVR, PIC, ARM, ...) permet ainsi de factoriser le développement d’applications, mais aussi de généraliser l’analyse et le débogage du bytecode associé, tout en permettant un usage précautionneux de la mémoire. On cible alors des programmes ludiques ou de domotiques destinés à être exécutés sur des microcontrôleurs à faibles ressources, en insistant sur les particularités inhérentes à la programmation de systèmes embarqués.

OSIS18_IoT: L'approche machine virtuelle pour les microcontrôleurs, le projet...

Pôle Systematic Paris-Region

OSIS18_IoT: La securite des objets connectes a bas cout avec l'os et riot

Pôle Systematic Paris-Region

OSIS18_IoT : Solution de mise au point pour les systemes embarques, par Julio...

Pôle Systematic Paris-Region

OSIS18_IoT : Securisation du reseau des objets connectes, par Nicolas LE SAUZ...

Pôle Systematic Paris-Region

The programming language Ada offers unique features to safely program a micro-controller. From the start, Ada was designed to make it difficult to introduce errors, and to make it easy to discover errors that were introduced. For example, language rules enforced at compile time make it possible to have safe concurrency by design. And run-time checking allows immediate detection of what would be "undefined behavior" in C/C++. In the first part of this presentation, we will present the benefits of using Ada for micro-controller programming, including support for debugging on a board. In the second part of this presentation, we will present how the Ada language and its subset SPARK provide a strong foundation for static analyzers, that make it possible to detect errors and provide guarantees on embedded software in Ada/SPARK.

OSIS18_IoT : Ada and SPARK - Defense in Depth for Safe Micro-controller Progr...

Pôle Systematic Paris-Region

OSIS18_IoT : RTEMS pour l'IoT professionnel, par Pierre Ficheux (Smile ECS)

Pôle Systematic Paris-Region

PyParis 2017 / Un mooc python, by thierry parmentelat

Pôle Systematic Paris-Region

More from Pôle Systematic Paris-Region (20)