CrowdStrike: Real World DTCS For Operators

Real World DTCS For Operators

An Introduction to
CrowdStrike
We Are CyberSecurity Technology Company
We Detect, Prevent And Respond To All Attack Types
In Real Time, Protecting OrganizationsFrom
Catastrophic Breaches
We Provide Next Generation Endpoint Protection,Threat
Intelligence & Pre &Post IR Services
NEXT-GEN
ENDPOINT
INCIDENT
RESPONSE
THREAT
INTEL

What Is Compaction?
• Cassandra write path:
– First the Commitlog
– Then the Memtable
– Eventually flushed to a SSTable
• Each SSTable is written exactly once
• Over time, Cassandra combines files
– Duplicate cells are merged
– Obsolete data is purged
• The algorithm Cassandra uses to determine when and how to combine
files is pluggable, and choosing the right strategy may be important at
scale
3© 2015. All Rights Reserved.

What Is Compaction?
• SizeTieredCompactionStrategy
– Each time min_threshold (4) files of the same size appear, combine them
into a new file
– Over time, you’ll naturally end up with a distribution of old data in large
files, new data in small files
– Deleted data in large files stays on disk longer than desired because
those files are very rarely compacted
4© 2015. All Rights Reserved.

SizeTieredCompactionStrategy
© 2015. All Rights Reserved. 5

SizeTieredCompactionStrategy
If each of the smallest blocks represent 1 day of data, and each write
had a 90 day TTL, when do you actually delete files and reclaim disk
space?

Why Compaction Strategy Matters
• We keep some data from sensors for a
fixed time period
• Processes
• DNS queries
• Files created
• It’s a LOT of data
• Talk tomorrow morning: One million
writes per second with 60 nodes
• We’re WELL past 60 nodes
• If we can’t delete it efficiently, costs go
way, way up

DateTieredCompactionStrategy
• Early tickets suggested creating a way to stop compacting cold
data
– CASSANDRA-5515 – track sstable coldness, stop compacting cold
sstables (measured by READ counts)
• CASSANDRA-6602 – optimize for time series specifically
– Solution provided by Björn Hegerfors from Spotify
– Use sstable’s min timestamp to find a target window
– Compact sstables within the same target
– Stop compacting sstables if max timestamp is older than a specified cutoff

DTCS In Pictures

DTCS Parameters
• max_sstable_age_days
• base_time_seconds
• timestamp_resolution
• Min_threshold
– Common to all compaction strategies
• Max Threshold
– Common to all compaction strategies

DTCS In Pictures

DTCS Benefits
In Theory…
• You can stop data compacting at a point you choose!
– max_sstable_age_days
• You can adjust the window size so that you can quickly expire
data when it’s approximately the size you want
– It’s not immediately intuitive, but you CAN calculate it (min_threshold and
base_time_seconds)
• We know cold data won’t be recompacted, so we can potentially
enable cold storage directories with cheaper disk
– CASSANDRA-8460 – patch available, I need to rebase

Do people consider DTCS Production Ready?
• It was added to 2.0 after 2.1 was out. Usually this means:
– Trivial and low risk, or
– Experimental and meant for advanced users only

– I challenge you to find documentation on which is true for DTCS

– I challenge you to find documentation on which is true for DTCS
• Spotify’s intro blog notes that they use it in production
• I’ve been told by a project committer that they feel DTCS is for
advanced users only, but I’ve never seen any public facing
messaging that normal users should avoid it
• It seems so easy, what could possibly go wrong…

DTCS Caveats
• The initial blogs give us some insight about what type of things
may not behave as intended
– “But something that works against the efforts of the strategy is writes with
highly out-of-order timestamps”
• How much is “highly out of order”?
– “Consider turning off read repairs. Anti-entropy repairs and hinted handoff
don’t incur as much additional work for DTCS and may be used like
usual.”

Out of order timestamps
• When an sstable gets flushed with an old timestamp in a new
table:
– The max timestamp is used to determine when to stop compacting, but
– The min timestamp is used to determine which other files will be
compacted with this sstable

• Windows are tiered, and they get bigger and bigger
• With default settings and 1 year of data, the largest window
covers 180 days
– This means even if most of the file is past max_sstable_age_days, you
can still end up compacting with a brand new sstable with read repaired
data
• “DTCS never stops compacting”
– Read repairs pull old data into new windows triggering
recompaction

• Windows are tiered, and they get bigger and bigger
• With default settings and 1 year of data, the largest window
covers 180 days
– This means even if most of the file is past max_sstable_age_days, you
can still end up compacting with a brand new sstable with read repaired
data
• “DTCS never stops compacting”
– Read repairs pull old data into new windows triggering recompaction
– Does that mean we better run repair?

Small SSTables from Repairs
(and other streaming operations)
• “If an SSTable contains timestamps that don’t match the time
when it was actually written to disk, it violates the size-to-age
correspondence that DTCS tries to maintain.”
• The suggestions on Spotify and Datastax blogs say run repair
more often than max_sstable_age_days, but that isn’t the only
cause of small sstables
– Bootstrap
– Decommission
– Bulk Loader

Real Pain:
If you can’t expand your cluster, what’s the point?
SSTable Count Per Node

Real Pain:
If you can’t expand your cluster, what’s the point?
Damn you, vnodes!

Well…

Small SSTables Shouldn’t Be Ignored
• If the small sstables are beyond max_sstable_age_days, they
won’t be compacted
– After all, that’s the point of max_sstable_age_days, right?
• If you raise max_sstable_age_days, the ever-growing DTCS
tiered windows will cause existing sstables to merge and get
much larger, negating one of the benefits of DTCS
• If you don’t raise max_sstable_age_days, you have to deal with
performance implications of ten thousand sstables
– Reduced somewhat by CASSANDRA-9882
– Before #9882, too many sstables could block flushing for a long time

Embarrassing Admission
• Our early bulk loading plan and bootstrapping procedure
acknowledged that sstables will be abandoned beyond
max_sstable_age_days
• We have python scripts that check the timestamps, and
manually submit compactions through JMX
forceUserDefinedCompaction()

Really Embarrassing Admission
acknowledged that sstables will be abandoned beyond
max_sstable_age_days
• Yes, really.

Really Embarrassing Admission
acknowledged and accepted that sstables will be abandoned
beyond max_sstable_age_days
• Yes, really.
• Does it actually scale?

When should you use DTCS?
• You TTL ALL of your data and writes come in order
• Fixed sized cluster and no plans for bulk loading, or rarely
changing cluster size and not using vnodes
– If you plan on growing, you better have a plan for small sstables
– If you do need to add/remove nodes, vnodes will cause far more small
sstables than single-token-per-node
• Extra space available for compaction
– You can’t rely on theoretical table sizes calculated with
max_sstable_age_days, because read repair, hints, etc, can force those
files to span much larger time ranges than you expect

Being Honest

What if?
• Do we really need max_sstable_age_days?
– The conventional logic is to use it to denote cold data, but we use it to
force window sizes
– If we give up tiering, and stick with fixed sized windows, do we need
max_sstable_age_days?
• Without tiering, can we swap base_time_seconds for more
intuitive configuration option?

TimeWindowCompactionStrategy
• Designed to be simple and efficient
– Group sstables into logical buckets
– STCS within each time window
– No more rolling re-compaction
– No more streaming leftovers
– No more confusing options, just Window Size + Window Unit
• “12 Hours”, “3 Days”, “6 Minutes”

• Submitted to Apache Cassandra as CASSANDRA-9666
• For now, we use it at Crowdstrike to clean up after streaming:
– echo "set -b
org.apache.cassandra.db:columnfamily=table,keyspace=keyspace,type=ColumnFamilies
CompactionStrategyClass
org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy" | java -jar
jmxterm.jar -l $IP:$PORT
– It’s not an accident that the TWCS defaults use 1 day windows with
microsecond timestamp resolution, that matches our sstable needs, but
we think it’s a good default
• Patches (and Tests) Available for 2.1, 2.2, 3.0

• No more continuous compaction
• No more tiny streaming leftovers
• No more confusing options
– Just Window Size, Window Unit
– “12 Hours”, “3 Days”, “6 Minutes”
• Work is ongoing for both DTCS and TWCS
– CASSANDRA-9645 to make DTCS easier to use
– CASSANDRA-10276 to make DTCS do STCS within each window (patch
available)
– CASSANDRA-10280 to make DTCS work well with old data

• There’s no guarantee that TWCS will make it into the project
– TWCS is certainly easier to reason about, but DTCS was there first and is
already deployed by real users
– Anecdotal evidence and preliminary benchmarks suggest TWCS comes out
ahead based on current state of both strategies (at the time of these slides)
– Formal benchmarking is needed
– DTCS probably wins for reads/SELECTS in SOME data models
• Even if TWCS doesn’t make it in, the source is available now on (see:
CASSANDRA-9666)
– It’s likely we’ll continue to maintain it, even if it’s not accepted upstream, so
pull requests are welcome

Q&A
• Talk to me about Cassandra or DTCS on twitter: @jjirsa
• Try to stop me from talking about DTCS on IRC: #cassandra
• Crowdstrike is awesome and hiring
– www.crowdstrike.com/careers/
• Jim Plush and Dennis Opacki, tomorrow morning
– “1 Million Writes Per Second on 60 Nodes with Cassandra and EBS”

CrowdStrike: Real World DTCS For Operators

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (18)

Destaque

Destaque (20)

Semelhante a CrowdStrike: Real World DTCS For Operators

Semelhante a CrowdStrike: Real World DTCS For Operators (20)

Mais de DataStax Academy

Mais de DataStax Academy (20)

Último

Último (20)

CrowdStrike: Real World DTCS For Operators