SlideShare uma empresa Scribd logo
1 de 24
Baixar para ler offline
(c) Higher Frequency Trading
Who am I?

Australian living in UK. Three kids 5, 8 and 15

Five years designing, developing and supporting HFT
systems in Java

My blog, “Vanilla Java” gets 120K page views per month.

3rd
for Java on StackOverflow

Lead developer for OpenHFT which includes Chronicle
and Thread Affinity.
(c) Higher Frequency Trading
Introduction to Chronicle
Micro second latency persistence
Higher Frequency Trading
(c) Peter Lawrey
(c) Higher Frequency Trading
What is Chronicle?
Very fast embedded persistence for Java.
Functionality is simple and low level by design
(c) Higher Frequency Trading
Where does Chronicle come from

Low latency, high frequency trading
– Applications which are sub 100 micro-second
external to the system.
(c) Higher Frequency Trading
Where does Chronicle come from

High throughput trading systems
– Hundreds of thousand of events per second
(c) Higher Frequency Trading
Where does Chronicle come from

Modes of use
– GC free
– Lock-less
– Shared memory
– Text or binary
– Replicated over
TCP
– Supports thread
affinity
(c) Higher Frequency Trading
Is there a free version?

It is open source and free with an Apache 2.0
license.

You can pay for training and consulting
(c) Higher Frequency Trading
Use for Chronicle

Synchronous text logging

Synchronous binary data logging
(c) Higher Frequency Trading
Use for Chronicle

Messaging between processes
via shared memory

Messaging across systems
(c) Higher Frequency Trading
Use for Chronicle

Supports recording micro-second timestamps
across the systems

Replay for production data in test
(c) Higher Frequency Trading
Writing to Chronicle
IndexedChronicle ic = new IndexedChronicle(basePath);
Excerpt excerpt = ic.createExcerpt();
for (int i = 1; i <= runs; i++) {
excerpt.startExcerpt(17);
excerpt.writeUnsignedByte('M'); // message type
excerpt.writeLong(i); // e.g. time stamp
excerpt.writeDouble(i);
excerpt.finish();
}
ic.close();
(c) Higher Frequency Trading
Reading from Chronicle
IndexedChronicle ic = new IndexedChronicle(basePath);
ic.useUnsafe(true); // for benchmarks
Excerpt excerpt = ic.createExcerpt();
for (int i = 1; i <= runs; i++) {
while (!excerpt.nextIndex()) {
// busy wait
}
char ch = (char) excerpt.readUnsignedByte();
long l = excerpt.readLong();
double d = excerpt.readDouble();
assert ch == 'M';
assert l == i;
assert d == i;
excerpt.finish();
}
ic.close();
(c) Higher Frequency Trading
How does it perform
With one thread writing and another reading
* Chronicle 2.0
-XX:+AggressiveOpts -Xmx32m
Tiny
4 B
Small
16 B
Medium
64 B
Large
256 B
tmpfs 77 M/s 57 M/s 23 M/s 6.6 M/s
ext4 65 M/s 35 M/s 12 M/s 3.2 M/s
(c) Higher Frequency Trading
How does it recover?
Once finish()
returns, the OS will do
the rest.
If an excerpt is
incomplete, it will be
pruned.
(c) Higher Frequency Trading
Cache friendly
Data is laid out continuously, naturally packed.
You can compress some types. One entry
starts in the next byte to the previous one.
(c) Higher Frequency Trading
Consumer insensitive
No matter how slow the consumer is, the
producer never has to wait. It never needs to
clean messages before publishing (as a ring
buffer does)
You can start a consumer at the end of the day
e.g. for reporting. The consumer can be more
than the main memory size behind the
producer as a Chronicle is not limited by main
memory.
(c) Higher Frequency Trading
How does it collect garbage?
There is an assumption that your application has a daily
or weekly maintenance cycle.
This is implemented by
closing the files and
creating new ones.
i.e. the whole lot is moved,
compressed or deleted.
Anything which must be
retained can be copied
to the new Chronicle
(c) Higher Frequency Trading
Is there a lower level API?
Chronicle 2.0 is based on OpenHFT Java Lang
library which supports access to 64-bit native
memory.
●
Has long size and offsets.
●
Support serialization and deserialization
●
Thread safe access including locking
(c) Higher Frequency Trading
Is there a higher level API?
You can hide the low level details with an
interface.
(c) Higher Frequency Trading
Is there a higher level API?
There is a demo
program with a
simple interface.
This models a “hub”
process which take in
events, processes
them and publishes
results.
(c) Higher Frequency Trading
Is there a higher level API?
The interfaces look like this
public interface Gw2PeEvents {
public void small(MetaData metaData,
SmallCommand command);
}
public interface Pe2GwEvents {
public void report(MetaData metaData,
SmallReport smallReport);
}
(c) Higher Frequency Trading
What does Chronicle need
More documentation
More tutorials
More use case
examples
(c) Higher Frequency Trading
What is planned
Lower overhead per
message
80 ns → 15 ns.
Low latency XML and
FIX parser and
writer
Central Order Book
engine.
(c) Higher Frequency Trading
Q & A

Mais conteúdo relacionado

Mais procurados

Low latency for high throughput
Low latency for high throughputLow latency for high throughput
Low latency for high throughputPeter Lawrey
 
Deterministic behaviour and performance in trading systems
Deterministic behaviour and performance in trading systemsDeterministic behaviour and performance in trading systems
Deterministic behaviour and performance in trading systemsPeter Lawrey
 
Writing and testing high frequency trading engines in java
Writing and testing high frequency trading engines in javaWriting and testing high frequency trading engines in java
Writing and testing high frequency trading engines in javaPeter Lawrey
 
High Frequency Trading and NoSQL database
High Frequency Trading and NoSQL databaseHigh Frequency Trading and NoSQL database
High Frequency Trading and NoSQL databasePeter Lawrey
 
Low level java programming
Low level java programmingLow level java programming
Low level java programmingPeter Lawrey
 
Determinism in finance
Determinism in financeDeterminism in finance
Determinism in financePeter Lawrey
 
Java in High Frequency Trading
Java in High Frequency TradingJava in High Frequency Trading
Java in High Frequency TradingViktor Sovietov
 
Improving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberImproving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberYing Zheng
 
Europe Virtual ALT.NET - EventStore v3
Europe Virtual ALT.NET - EventStore v3Europe Virtual ALT.NET - EventStore v3
Europe Virtual ALT.NET - EventStore v3Jonathan Oliver
 
Handle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaHandle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaJiangjie Qin
 
Producer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache KafkaProducer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache KafkaJiangjie Qin
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
 
PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...
PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...
PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...DataStax
 
Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014
Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014
Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014Chen-en Lu
 
Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Otávio Carvalho
 
How Zhaopin contributes to Pulsar community
How Zhaopin contributes to Pulsar communityHow Zhaopin contributes to Pulsar community
How Zhaopin contributes to Pulsar communityStreamNative
 
Troubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolutionTroubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolutionJoel Koshy
 
Deep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumptionDeep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumptionAlexandre Tamborrino
 
Donatas Mažionis, Building low latency web APIs
Donatas Mažionis, Building low latency web APIsDonatas Mažionis, Building low latency web APIs
Donatas Mažionis, Building low latency web APIsTanya Denisyuk
 
Using SCTP with Scamper and Netty
Using SCTP with Scamper and NettyUsing SCTP with Scamper and Netty
Using SCTP with Scamper and NettyTim Boudreau
 

Mais procurados (20)

Low latency for high throughput
Low latency for high throughputLow latency for high throughput
Low latency for high throughput
 
Deterministic behaviour and performance in trading systems
Deterministic behaviour and performance in trading systemsDeterministic behaviour and performance in trading systems
Deterministic behaviour and performance in trading systems
 
Writing and testing high frequency trading engines in java
Writing and testing high frequency trading engines in javaWriting and testing high frequency trading engines in java
Writing and testing high frequency trading engines in java
 
High Frequency Trading and NoSQL database
High Frequency Trading and NoSQL databaseHigh Frequency Trading and NoSQL database
High Frequency Trading and NoSQL database
 
Low level java programming
Low level java programmingLow level java programming
Low level java programming
 
Determinism in finance
Determinism in financeDeterminism in finance
Determinism in finance
 
Java in High Frequency Trading
Java in High Frequency TradingJava in High Frequency Trading
Java in High Frequency Trading
 
Improving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberImproving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at Uber
 
Europe Virtual ALT.NET - EventStore v3
Europe Virtual ALT.NET - EventStore v3Europe Virtual ALT.NET - EventStore v3
Europe Virtual ALT.NET - EventStore v3
 
Handle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaHandle Large Messages In Apache Kafka
Handle Large Messages In Apache Kafka
 
Producer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache KafkaProducer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache Kafka
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
 
PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...
PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...
PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...
 
Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014
Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014
Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014
 
Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018
 
How Zhaopin contributes to Pulsar community
How Zhaopin contributes to Pulsar communityHow Zhaopin contributes to Pulsar community
How Zhaopin contributes to Pulsar community
 
Troubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolutionTroubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolution
 
Deep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumptionDeep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumption
 
Donatas Mažionis, Building low latency web APIs
Donatas Mažionis, Building low latency web APIsDonatas Mažionis, Building low latency web APIs
Donatas Mažionis, Building low latency web APIs
 
Using SCTP with Scamper and Netty
Using SCTP with Scamper and NettyUsing SCTP with Scamper and Netty
Using SCTP with Scamper and Netty
 

Semelhante a Introduction to chronicle (low latency persistence)

The 1990s Called. They Want Their Code Back.
The 1990s Called. They Want Their Code Back.The 1990s Called. They Want Their Code Back.
The 1990s Called. They Want Their Code Back.Jonathan Oliver
 
Elixir Into Production
Elixir Into ProductionElixir Into Production
Elixir Into ProductionJamie Winsor
 
Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1
Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1  Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1
Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1 securityxploded
 
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...apidays
 
Jdd2014: High performance logging - Peter Lawrey
Jdd2014: High performance logging - Peter LawreyJdd2014: High performance logging - Peter Lawrey
Jdd2014: High performance logging - Peter LawreyPROIDEA
 
Kamailio with Docker and Kubernetes
Kamailio with Docker and KubernetesKamailio with Docker and Kubernetes
Kamailio with Docker and KubernetesPaolo Visintin
 
Networking and Go: An Engineer's Journey (Strangeloop 2019)
Networking and Go: An Engineer's Journey (Strangeloop 2019)Networking and Go: An Engineer's Journey (Strangeloop 2019)
Networking and Go: An Engineer's Journey (Strangeloop 2019)Sneha Inguva
 
Flink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow
Flink Forward SF 2017: Eron Wright - Introducing Flink TensorflowFlink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow
Flink Forward SF 2017: Eron Wright - Introducing Flink TensorflowFlink Forward
 
Import golang; struct microservice
Import golang; struct microserviceImport golang; struct microservice
Import golang; struct microserviceGiulio De Donato
 
CocoaConf: The Language of Mobile Software is APIs
CocoaConf: The Language of Mobile Software is APIsCocoaConf: The Language of Mobile Software is APIs
CocoaConf: The Language of Mobile Software is APIsTim Burks
 
Devoxx 2015 - Building the Internet of Things with Eclipse IoT
Devoxx 2015 - Building the Internet of Things with Eclipse IoTDevoxx 2015 - Building the Internet of Things with Eclipse IoT
Devoxx 2015 - Building the Internet of Things with Eclipse IoTBenjamin Cabé
 
Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...
Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...
Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...confluent
 
Modern Web Security, Lazy but Mindful Like a Fox
Modern Web Security, Lazy but Mindful Like a FoxModern Web Security, Lazy but Mindful Like a Fox
Modern Web Security, Lazy but Mindful Like a FoxC4Media
 
Apache Kafka
Apache KafkaApache Kafka
Apache KafkaJoe Stein
 
Server Side JavaScript: Ajax.org O3
Server Side JavaScript: Ajax.org O3Server Side JavaScript: Ajax.org O3
Server Side JavaScript: Ajax.org O3Javeline B.V.
 
Server Side JavaScript: Ajax.org O3.
Server Side JavaScript: Ajax.org O3.Server Side JavaScript: Ajax.org O3.
Server Side JavaScript: Ajax.org O3.ejpbruel
 
CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...
CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...
CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...CanSecWest
 

Semelhante a Introduction to chronicle (low latency persistence) (20)

The 1990s Called. They Want Their Code Back.
The 1990s Called. They Want Their Code Back.The 1990s Called. They Want Their Code Back.
The 1990s Called. They Want Their Code Back.
 
Elixir Into Production
Elixir Into ProductionElixir Into Production
Elixir Into Production
 
Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1
Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1  Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1
Advanced Malware Analysis Training Session 2 - Botnet Analysis Part 1
 
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...
 
Jdd2014: High performance logging - Peter Lawrey
Jdd2014: High performance logging - Peter LawreyJdd2014: High performance logging - Peter Lawrey
Jdd2014: High performance logging - Peter Lawrey
 
Rust Hack
Rust HackRust Hack
Rust Hack
 
Kamailio with Docker and Kubernetes
Kamailio with Docker and KubernetesKamailio with Docker and Kubernetes
Kamailio with Docker and Kubernetes
 
Networking and Go: An Engineer's Journey (Strangeloop 2019)
Networking and Go: An Engineer's Journey (Strangeloop 2019)Networking and Go: An Engineer's Journey (Strangeloop 2019)
Networking and Go: An Engineer's Journey (Strangeloop 2019)
 
Flink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow
Flink Forward SF 2017: Eron Wright - Introducing Flink TensorflowFlink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow
Flink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow
 
Import golang; struct microservice
Import golang; struct microserviceImport golang; struct microservice
Import golang; struct microservice
 
CocoaConf: The Language of Mobile Software is APIs
CocoaConf: The Language of Mobile Software is APIsCocoaConf: The Language of Mobile Software is APIs
CocoaConf: The Language of Mobile Software is APIs
 
JS Event Loop
JS Event LoopJS Event Loop
JS Event Loop
 
Devoxx 2015 - Building the Internet of Things with Eclipse IoT
Devoxx 2015 - Building the Internet of Things with Eclipse IoTDevoxx 2015 - Building the Internet of Things with Eclipse IoT
Devoxx 2015 - Building the Internet of Things with Eclipse IoT
 
Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...
Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...
Streamsheets and Apache Kafka – Interactively build real-time Dashboards and ...
 
Modern Web Security, Lazy but Mindful Like a Fox
Modern Web Security, Lazy but Mindful Like a FoxModern Web Security, Lazy but Mindful Like a Fox
Modern Web Security, Lazy but Mindful Like a Fox
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
Server Side JavaScript: Ajax.org O3
Server Side JavaScript: Ajax.org O3Server Side JavaScript: Ajax.org O3
Server Side JavaScript: Ajax.org O3
 
Server Side JavaScript: Ajax.org O3.
Server Side JavaScript: Ajax.org O3.Server Side JavaScript: Ajax.org O3.
Server Side JavaScript: Ajax.org O3.
 
CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...
CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...
CSW2017 Henry li how to find the vulnerability to bypass the control flow gua...
 
Apache Thrift
Apache ThriftApache Thrift
Apache Thrift
 

Mais de Peter Lawrey

Chronicle Accelerate Crypto Investor conference
Chronicle Accelerate Crypto Investor conferenceChronicle Accelerate Crypto Investor conference
Chronicle Accelerate Crypto Investor conferencePeter Lawrey
 
Legacy lambda code
Legacy lambda codeLegacy lambda code
Legacy lambda codePeter Lawrey
 
Responding rapidly when you have 100+ GB data sets in Java
Responding rapidly when you have 100+ GB data sets in JavaResponding rapidly when you have 100+ GB data sets in Java
Responding rapidly when you have 100+ GB data sets in JavaPeter Lawrey
 
Reactive programming with examples
Reactive programming with examplesReactive programming with examples
Reactive programming with examplesPeter Lawrey
 
Streams and lambdas the good, the bad and the ugly
Streams and lambdas the good, the bad and the uglyStreams and lambdas the good, the bad and the ugly
Streams and lambdas the good, the bad and the uglyPeter Lawrey
 
Thread Safe Interprocess Shared Memory in Java (in 7 mins)
Thread Safe Interprocess Shared Memory in Java (in 7 mins)Thread Safe Interprocess Shared Memory in Java (in 7 mins)
Thread Safe Interprocess Shared Memory in Java (in 7 mins)Peter Lawrey
 
Using BigDecimal and double
Using BigDecimal and doubleUsing BigDecimal and double
Using BigDecimal and doublePeter Lawrey
 

Mais de Peter Lawrey (7)

Chronicle Accelerate Crypto Investor conference
Chronicle Accelerate Crypto Investor conferenceChronicle Accelerate Crypto Investor conference
Chronicle Accelerate Crypto Investor conference
 
Legacy lambda code
Legacy lambda codeLegacy lambda code
Legacy lambda code
 
Responding rapidly when you have 100+ GB data sets in Java
Responding rapidly when you have 100+ GB data sets in JavaResponding rapidly when you have 100+ GB data sets in Java
Responding rapidly when you have 100+ GB data sets in Java
 
Reactive programming with examples
Reactive programming with examplesReactive programming with examples
Reactive programming with examples
 
Streams and lambdas the good, the bad and the ugly
Streams and lambdas the good, the bad and the uglyStreams and lambdas the good, the bad and the ugly
Streams and lambdas the good, the bad and the ugly
 
Thread Safe Interprocess Shared Memory in Java (in 7 mins)
Thread Safe Interprocess Shared Memory in Java (in 7 mins)Thread Safe Interprocess Shared Memory in Java (in 7 mins)
Thread Safe Interprocess Shared Memory in Java (in 7 mins)
 
Using BigDecimal and double
Using BigDecimal and doubleUsing BigDecimal and double
Using BigDecimal and double
 

Último

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 

Último (20)

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 

Introduction to chronicle (low latency persistence)

  • 1. (c) Higher Frequency Trading Who am I?  Australian living in UK. Three kids 5, 8 and 15  Five years designing, developing and supporting HFT systems in Java  My blog, “Vanilla Java” gets 120K page views per month.  3rd for Java on StackOverflow  Lead developer for OpenHFT which includes Chronicle and Thread Affinity.
  • 2. (c) Higher Frequency Trading Introduction to Chronicle Micro second latency persistence Higher Frequency Trading (c) Peter Lawrey
  • 3. (c) Higher Frequency Trading What is Chronicle? Very fast embedded persistence for Java. Functionality is simple and low level by design
  • 4. (c) Higher Frequency Trading Where does Chronicle come from  Low latency, high frequency trading – Applications which are sub 100 micro-second external to the system.
  • 5. (c) Higher Frequency Trading Where does Chronicle come from  High throughput trading systems – Hundreds of thousand of events per second
  • 6. (c) Higher Frequency Trading Where does Chronicle come from  Modes of use – GC free – Lock-less – Shared memory – Text or binary – Replicated over TCP – Supports thread affinity
  • 7. (c) Higher Frequency Trading Is there a free version?  It is open source and free with an Apache 2.0 license.  You can pay for training and consulting
  • 8. (c) Higher Frequency Trading Use for Chronicle  Synchronous text logging  Synchronous binary data logging
  • 9. (c) Higher Frequency Trading Use for Chronicle  Messaging between processes via shared memory  Messaging across systems
  • 10. (c) Higher Frequency Trading Use for Chronicle  Supports recording micro-second timestamps across the systems  Replay for production data in test
  • 11. (c) Higher Frequency Trading Writing to Chronicle IndexedChronicle ic = new IndexedChronicle(basePath); Excerpt excerpt = ic.createExcerpt(); for (int i = 1; i <= runs; i++) { excerpt.startExcerpt(17); excerpt.writeUnsignedByte('M'); // message type excerpt.writeLong(i); // e.g. time stamp excerpt.writeDouble(i); excerpt.finish(); } ic.close();
  • 12. (c) Higher Frequency Trading Reading from Chronicle IndexedChronicle ic = new IndexedChronicle(basePath); ic.useUnsafe(true); // for benchmarks Excerpt excerpt = ic.createExcerpt(); for (int i = 1; i <= runs; i++) { while (!excerpt.nextIndex()) { // busy wait } char ch = (char) excerpt.readUnsignedByte(); long l = excerpt.readLong(); double d = excerpt.readDouble(); assert ch == 'M'; assert l == i; assert d == i; excerpt.finish(); } ic.close();
  • 13. (c) Higher Frequency Trading How does it perform With one thread writing and another reading * Chronicle 2.0 -XX:+AggressiveOpts -Xmx32m Tiny 4 B Small 16 B Medium 64 B Large 256 B tmpfs 77 M/s 57 M/s 23 M/s 6.6 M/s ext4 65 M/s 35 M/s 12 M/s 3.2 M/s
  • 14. (c) Higher Frequency Trading How does it recover? Once finish() returns, the OS will do the rest. If an excerpt is incomplete, it will be pruned.
  • 15. (c) Higher Frequency Trading Cache friendly Data is laid out continuously, naturally packed. You can compress some types. One entry starts in the next byte to the previous one.
  • 16. (c) Higher Frequency Trading Consumer insensitive No matter how slow the consumer is, the producer never has to wait. It never needs to clean messages before publishing (as a ring buffer does) You can start a consumer at the end of the day e.g. for reporting. The consumer can be more than the main memory size behind the producer as a Chronicle is not limited by main memory.
  • 17. (c) Higher Frequency Trading How does it collect garbage? There is an assumption that your application has a daily or weekly maintenance cycle. This is implemented by closing the files and creating new ones. i.e. the whole lot is moved, compressed or deleted. Anything which must be retained can be copied to the new Chronicle
  • 18. (c) Higher Frequency Trading Is there a lower level API? Chronicle 2.0 is based on OpenHFT Java Lang library which supports access to 64-bit native memory. ● Has long size and offsets. ● Support serialization and deserialization ● Thread safe access including locking
  • 19. (c) Higher Frequency Trading Is there a higher level API? You can hide the low level details with an interface.
  • 20. (c) Higher Frequency Trading Is there a higher level API? There is a demo program with a simple interface. This models a “hub” process which take in events, processes them and publishes results.
  • 21. (c) Higher Frequency Trading Is there a higher level API? The interfaces look like this public interface Gw2PeEvents { public void small(MetaData metaData, SmallCommand command); } public interface Pe2GwEvents { public void report(MetaData metaData, SmallReport smallReport); }
  • 22. (c) Higher Frequency Trading What does Chronicle need More documentation More tutorials More use case examples
  • 23. (c) Higher Frequency Trading What is planned Lower overhead per message 80 ns → 15 ns. Low latency XML and FIX parser and writer Central Order Book engine.
  • 24. (c) Higher Frequency Trading Q & A