Sneak peak into the projects, challenges and overall culture of the Cloud Engineering team at Crowdstrike.

1. COSMIN BRATU, JUNE 2021
CLOUD ENGINEERING
@CRWD

2.  What We Do
 Cloud Engineering Culture
 Challenges
 Real World Use-Case
 Approach & Solution
 Q&A
AGENDA
©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

3. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
WHAT WE
DO

4. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
WESTOP
BREACHES
WHAT WE DO @CRWD
Buildasystemthatcan
 Stopfully-funded adversaries
 Innormalbusiness environments
 Andmakemoney doingit

5. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
10K FEET VIEW
Ingestion Microservices
5 TRILLION+
EVENTS/WEEK

6. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
IT HYGIENE
FALCON DISCOVER
AlwaysCurrent
ApplicationUsage
AssetandFirmware
Inventory
BUSINESSVALUE
Privileged
Account
Monitoring
Reducelicensingcosts
Minimizeriskassociatedwith rogue users,
apps,andsystems
Eliminateburdenof unmanaged
assets
Reducecostof endpointinspections

7. VULNERABILITY MANAGEMENT
FALCON SPOTLIGHT
Scanless
Technology
Integrated Threat
and Vulnerability
Workflows
Visibility from OS
to BIOS, On- and
Off-prem
BUSINESSVALUE
No Appliances,
No New Agents
Eliminate blind spots
Timely knowledge,
on-demand
Zero impact
Improve security posture
Eliminate the burden of
vulnerability scanning
2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

8. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
THREAT INTELLIGENCEAUTOMATION
FALCON X
EndpointIntegration
Adversary
Profiles
Indicatorsof
Compromise
BUSINESSVALUE
Malware
Sandbox
Expand IOCsand maximizeprotection
StreamlineSOC, IR andCTI
teams
Frictionlessdeployment
Assessalertsandrespondfaster
Accelerateand simplifythreat
investigations

9. DIGITAL RISK PROTECTION
FALCON X RECON
Deep, Dark
Web Visibility
Real Time
Monitoring
Prioritized
Alerts
Indicators of
Compromise
BUSINESSVALUE
Detect and stop data leaks and
impersonations
Accelerate Incident
Investigations
Visibility into restricted forums
and markets
Identifies risks to brands,
employees, and critical assets
Immediate time to value
2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

10. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
CLOUD
ENGINEERING
CULTURE

11. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
THE MISSION STATEMENT
 Favorasynchronous communication
 Partofadistributed team
 Continuous balanceofprocess andproductivity
 Nothingissetinstone
 Emphasisontesting atalllevels
 Never stoplearningandgrowing
 Aimforelegance butaccept thepractical
 Encourageandexpect theexchange ofideaswithintheteam

12. ROLE OF CLOUD ENGINEER
©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
Release– actslikea releaseengineerforownedservices
Ownerofproductiondeploymentforprojects
DevOpsphilosophy
Testing–unittests,functionaltests,stresstests,performancetestsetc.with
supportfromSDETs
Responsibleformonitoring–logs,alerts,debugging–andsolvingproduction
issues
Focusonsecurity
Writeandmaintaindocumentation
Lotsofautonomyintheday-to-day
Part of ONEteam – haveaccess,learn,contributeandhelpanyotherservice
Leadershipat alllevels Technicalambassadorforthecompany

13. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
CHALLENGES

14. IT’S ALL ABOUT THE SCALE
©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
 Closeto1Trillionevents perday
 250+Microservices
Oursystemprocessesasmanyeventsinonedayas
Twitterhastweetsinayear.

15. AND EFFICIENCY
©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
Quality
Large-scaledistributedsystemsin
productionrequiresignificantplanning
upfront
Speed
Expandingandreachingnewcustomers
requireearlyfeedbackandfastiteration
Cost
Wecannotsolveeveryproblemby
throwingmoremoneyatit

16. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
REAL
WORLD
USE-CASE

17. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
A REAL USE CASE
 Falconsensorscontinuouslyscanthecustomernetwork
 Theyreportbackallthedevices/hoststhatarereachablefromthatnodeinthenetwork
 Thisincludesdevicesthatdon’thaveaFalconsensorinstalled (unmanagedhosts)
 Weneedtobuildasystemthatstoresagraph-likestructureofthenetworktopology
 Andallowcustomerstoexplorethisgraph

18. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
REQUIREMENTS
REQUIREMENTS
MILLIONS
DISCOVERED
HOSTS PER
SECOND
• Presentthelatestknown stateof a host
• Searchbothmanagedandunmanagedhostsbya
substringof IP orMAC e.g. searchingfor“192.168”
shouldreturnall unmanagedhostswhich have anIP
thatmatches
• Searchunmanagedhostsby propertiesofmanaged
neighbourse.g. “showme all theunmanagedassets
which have beendiscoveredby a hostthathas
Linuxas anoperatingsystem”

19. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
UNMANAGED ASSETS SIMPLIFIED VIEW
Customer IP MAC Neighbours First Discovered At Last Discovered At
Customer 1 1.0.0.1 00-11-22-33-44-55 Host-1, Host-2 08.06.2021 09.06.2021
Customer 1 1.0.0.2 00-AA-BB-CC-DD-EE Host-3 01.05.2021 09.06.2021
Customer 1 1.0.0.3 00-11-22-33-44-AA Host-1, Host-3 01.05.2021 09.06.2021
IP Filter MAC Filter Neighbor Filter Timestamp filter

20. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
APPROACH &
SOLUTION

21. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
SO HOW DO WE SOLVE IT?
DATABASE
SEARCHING
EVENT PROCESSING
CACHING
• We could usea SQLdatabase,butcanit support 6M
writes/s? How do we partition thedata?
• Whatabouta NoSQLdatabase?Butwhatto choose –
Key-value,Documentetc. ?
• How do we quickly decide if a reported host alreadyhas a
Falcon sensorinstalled? Is a key-valuestore enoughor do
we need somethingmoreefficient?
• How do we implementthesearchinglayer? Do weuse a
dedicated technology?
• Canwecacheanyresults?What eviction policy?
• Whatis thedataretention?
• A host is reported by multiple sensors andcould have
conflicting info
• Eventsarereceived potentially outof order fromthesensors
(becauseof thedistributed messagequeue). Howdo we
handlethis?
• Do we process theevents as theycomeor in batches?

22.  Scaleindefinitely
 Servequeriesinmilliseconds
 Accommodate largespikes intraffic
 Supportdeletes i.e.anunmanaged hostcanbecome managed
 Kafkamessages haveatimestamp associated, butcanarriveoutoforder
 Eventsforthesamehostcanreachdifferent serversatthesametime -concurrent updates
 Cassandracanscalewellhorizontally,butweneedtoavoidcreatingaverylargecluster
 CassandraLightweight Transactions (CAS)aretooslowforourdatavolume
 Elasticsearch indices don’thandlelargevolumewritesverywell
©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
DESIGN CONSIDERATIONS

23. SOLUTION
©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
 Kafkakeyedpartitioning
 Localbatchingoneacheventprocessingnode
 Efficientmergingofgranularstateupdatesinto
thelateststate–readtimereconciliation
 Cacheformanagedhostlookup
 Elasticsearch“podding”
IngestSystem
UpdatesProcessor
ManagedHost
Lookup Service
SearchService
Local Database
SearchStorage
Updates Storage
LatestStateStorage
Managed Host
Cache
FALCON
AGENT

24.  Canwesomehow removetheUpdatesstorageandprocessor andavoidanextrastep?
 Theupdates storagesolves twoproblems:
 Concurrentupdates–theservicereadstheentity,appliestheupdatesinmemory,thenupdatestheentityifanythingchanges
 Outoforderupdateanddeleteevents–imaginewehaveadeleteeventremovingtherowfromtheLastStatestorageandanupdateeventarriving
afterwards
 Soisthereamoreefficient solution?
©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
CAN WE DO BETTER?

25.  Don’tdeletearowfromtheLastStatestorage,butmarkwitha
deletedflagandaTTLofdoubletheKafkaretentionperiod
 KafkamessagesarekeyedbyIDsoeventsforaparticularhost
willbeprocessedbythesameserver
 DuringKafkapartitionrebalanceswemightstillhave
concurrentupdates,souseZooKeepertolockapartition
 Thelockforapartitionwillbereleasedwhenalltheeventsin
memoryareprocessed
©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
OPTIMIZED SOLUTION
IngestSystem
ManagedHostLookup
Service
SearchService
Local Database
SearchStorage
LatestStateStorage
Managed Host
Cache
ZooKeeper
FALCON
AGENT

26. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
THE TEAM IS GROWING
crowdstrike.com/careers

27. ©2021CROWDSTRIKE,INC.ALLRIGHTS RESERVED.
Q&A