This document discusses how CSX, a freight railroad company, used Elastic to gain real-time insights into their business data. It describes CSX's need to analyze data from various systems and sensors to improve visibility, maintain train-locomotive associations, and democratize access to data. The document outlines how CSX implemented Elastic to explore data without impacting systems, synthesize data streams, and develop proofs of concept like tracking PTC compliance and locomotive faults. This has helped CSX better understand their data and share insights with business partners.

1. Chris Jesseman & Evan King
CSX Technology
January 8, 2019
Real time business
discovery with Elastic

2. BUSINESS PERSPECTIVE

3. ABOUT CSX
▪ Over 21,000 route miles of track in 23 states, DC, and Canada
▪ Nearly two-thirds of Americans live within CSX’s service territory
▪ Operates an average of 1,300 trains per day
▪ Transports more than 6.5 million carloads of products and raw materials a year
▪ Maintains a fleet of more than 4,000 locomotives and approximately 70,000 freight
cars
▪ Safety is a way of life
3

4. BUSINESS NEED
▪ Ingest varying data at increasing speeds and density
▪ Support legacy and modern systems
▪ Real-time data discovery that enables development of business rules
▪ Improve analytical insight
▪ Provide an enterprise solution that empowers the business
4

5. DATA LANDSCAPE
▪ Mutable, logical entities compound the
complexity of capturing and acting on
data over satellite, cellular, 220mHz,
WiFi, and the track sensors
▪ Data from systems (what should
happen) and sensors (what is
happening)
▪Formats: EMP, XML, JSON, Protobuf
5

6. INITIAL BUSINESS PROBLEMS
▪ Improved visibility of PTC status and compliance
▪ Maintaining Locomotive to Train Association (LTA)
▪ Access to PTC telemetry
▪ Democratizing data
6

7. SEVERAL PROBLEMS…ONE STARTING PLACE

8. BUSINESS DISCOVERY IN ELASTIC
▪ No need for custom apps
▪ Data exploration does not impact running transaction systems
▪ Let the data be the analyst's tour guide
▪ Use real data to confirm, dispel anecdotes
▪ Synthesize data streams from various sources to 'enrich' the story
▪ Easy to jump right in
8

9. SHARED UNDERSTANDING WITH BUSINESS PARTNERS
9
Hey Matt. Here is a good example of Q02802
transitioning to PTC Active track. The algorithm
would find them out of compliance.
Perfect. That’s the information we need to know
when it’s happening.

10. DISCOVERING LTA
10

11. DEVELOPING PROOF OF CONCEPTS
11

12. DOES THAT ANECDOTE HOLD WATER?
12

13. BUSINESS VALUE
▪ Kibana can help to tell the story back to the business partner
▪ Visualizations and dashboard 'POCs' show the business partner what’s
possible
▪ More efficient testing…..compare the app with the ‘Kibana truth’
▪ Democratization of data makes us all better analysts
13

14. TECHNOLOGY PERSPECTIVE

15. ARCHITECTURE TENETS
▪ Capture events at their source, even the edge
▪ Humans can't log onto databases or servers in production
▪ Cloud Native
▪ Downstream entities can’t tell upstream what to do
▪ Separation of concerns
▪ Applies traditionally to software architecture
▪ Also apply it to data producers/consumers
▪ Don't query transaction systems for analytics data
15

16. LOGICAL ARCHITECTURE SUPPORTS FLOWING DATA
▪ Aqueduct
▪ Distillery
▪ Reservoir
▪ Tap
▪ Data Lake
16
Discovery

17. PHYSICAL IMPLEMENTATION
17

18. PROTOBUF
▪ Protobuf on the wire forces schema creation and enforcement as data enters
Aqueduct
▪ One Protobuf schema per topic
▪ One Elastic index per Protobuf schema
▪ Protobuf schemas and Elastic templates checked into Git happy as a clam
▪ Structured data is good for you
▪ Logstash pipelines use Protobuf codec and can be scaled like microservices
▪ Schemas and templates provide isolation for reliability and scaling
▪ Kafka is partitioned knowing that all data flows will also be going to Elastic
18

19. DASHBOARD EXAMPLES

20. HOW FAST IS THE NETWORK MOVING?
20

21. HOW IS PTC COMPLIANCE TRACKING?
21

22. WHAT LOCOMOTIVE FAULTS ARE OCCURRING?
22

23. MACHINE LEARNING AND PTC COMPLIANCE
23