1.
How a Time Series Database Contributes to a Decentralized
Cloud Object Storage Platform
InfluxData Time Series Webinar February, 25th 2020

2.
Our Presenters
John Gleeson | VP Operations @ Storj
Ben Sirb | Senior Data Scientist @ Storj

3.
Overview
A brief primer on Storj
Business need for time series data
We succeed or fail based on the data
What success looks like
Q&A
Our analytics approach

4.
A Primer on Storj

5.
Our Goal
To create the world’s largest and most secure, resilient,
performant, and economical cloud storage service -
without owning or operating a data center.
Our Goal

6.
Storj: A Decentralized Cloud Storage Network

7.
How it works System ViewHow it Works | System View

8.
Commitment to Open Source
If the dominant model for monetizing open source becomes
“using open source as a loss leader for infrastructure,” and…
Only a few large companies have the scale to operate
centralized clouds and traditional infrastructure, and...
Those same large companies also get a disproportionate
amount of traffic, data, talent, economic returns….
...how will we drive economic growth and innovation
from new open source projects?

9.
An Answer: Decentralized
Infrastructure
Fundamentally different economic model for delivering
infrastructure
Creates new model for monetizing open source

10.
The Business Need

11.
Need for time series data
Two-sided market: Supply + Demand
Durability: Function of Storage Node attrition over time
Retrievability: Function of SNO uptime at any point in time
Repair: Expensive business function to ensure Durability and Retrievability

12.
Our GoalMathematics behind functions is data intense
Storj V3 Whitepaper

13.
Our GoalIf You Can’t Measure It You Can’t Improve It
● We have a list of key metrics that directly impacts
roadmap progression - Qualification Gates.
● Exposing network behavior to the entire organization lets
us move faster
○ Everyone on the team needs access to quality data to
do their job successfully
● And some of these things can make or break the business

14.
Succeed or fail based on
the data

15.
Our GoalBoiled it All Down to a Simple ETL
Basic
Processing
and
Serialization
(We call it
Statreceiver)
InfluxDB Redash
Monkit Collection

16.
Bulk Data. Done!

17.
Our GoalFocus on Optimizing Design
● The faster we start generating key data the faster we
improve
● Don’t have resources for a lot of maintenance
overhead
● Needs flexibility as the network grows and data
needs change.
● Assume that non-Storj folks will use and run at some
point
● Last but not least: Needs to Scale with Network

18.
The Approach

19.
Our Goalmonkit: Metrics, Monitoring, and More
https://github.com/spacemonkeygo/monkit
● Captures event data
(meters, rates,
counts etc) right in
the code
● Collected directly on
network (satellites,
nodes, and uplinks).
Sent to separate
service for
processing.

20.
Our Goal
● With simple config, takes UDP
packets in and sends time
series data out to InfluxDB via
graphite protocol
● Some basic filtering and
processing in between
Statreceiver

21.
Our GoalInfluxDB Supports Graphite Protocol
Graphite Protocol
Time Series
Filter + template one
Filter + template two
Filter + template three
InfluxDB
Measurement
InfluxDB
Measurement
InfluxDB
Measurement
Allowed us to build useful additional structure.

22.
Our GoalInfluxDb - Redash BI
Super easy to learn if already familiar with SQL.

23.
The Challenge:
Managing Series Cardinality

24.
Framing the project
Save All
Data
Goal: Maximum Visibility
Result: Series Explosion
Database
Bloat

25.
Framing the project
● Launched with goal of maximum visibility
● Which meant saving all the data
● Which led to massive database bloat
● Series explosion
● Start filtering out certain measurements to prevent massive DB size
● ID’s as tags rather than fields

26.
Succeeding Based on the
Data

27.
What Influx lets us do
Enabling segment decay forecast essential to repair forecast
Monitoring growth rate
Control complexity - number of objects on the network
Tracking bandwidth usage for gets, puts, repairs audits - critical in bandwidth
constrained environment
Tracking vetting success rate over time - indication of health of network growth rate

28.
Our GoalState of the Network: Data Driven Guidance
● Internal dashboards for the benefit of
entire organization
● Root cause analysis of any anomalies
● Review current trajectory for key
metrics

29.
Our GoalMeasuring Model Parameters in Real Time

30.
The latest and greatest
With the release of monkit v3 we eliminated the need for the graphite layer
monkit v3 can natively output measurements with appropriate tags and names
No more need to “lock” measurements (before locking, code changes resulted in
broken measurements; not with monkit v3)

31.
Future Roadmap
● Full TICK Stack?
● Integrate statistical modeling
directly into architecture
● Decentralized network
eventually needs
decentralized analytics

32.
Open Source Partner Program
1.) The people who bring supply to the network (“SNOs”) should be fairly and
transparently compensated---otherwise they will leave the network
2.) The people who bring demand to the network should be fairly and transparently
compensated--so they help to build the network
...If open source is the biggest driver of cloud usage, why not let open source
projects get paid by the network for helping it to grow?

33.
How does the program work?
Delivering significant benefits for open source projects and their users.
Customers get secure, private, reliable cloud storage that is half the price of the big
cloud providers and drive revenue to open source projects they love.
Open source projects get a meaningful share of subscription revenue when their
end users store data on the Tardigrade platform.

34.
Network generates
revenue for OSS
Network benefits from
increased demand
OSS generates
demand for network
OSS innovates
Virtuous
Cycle

35.
Are you an OSS project that generates
demand for S3-compatible storage?
Build a connector that gives users
option to store on the network
Network tracks usage, and returns
meaningful portion of revenue that
your users generate to you
Sign up now (storj.io/partners)
Storj Partner Program

36.
Questions?

37.
Join our community at forum.storj.io

38.
Thank you!

39.
© 2019 InfluxData. All rights reserved.40
We are looking for great speakers for InfluxDays
London 2020. If you would like to speak, please
submit your speaker application.
8-10 June, 2020
etc.venues St. Paul’s / London, UK
Register now using code WEBINAR and save 20%
www.influxdays.com