3. criteolabs.com/jobs
CRITEO ?
6 DATA CENTERS, 4 CONTINENTS.
120 BILLION REQUESTS/DAY*.
* EVERY DAY CRITEO IS CALLED MORE THAN 100 BILLION TIMES BY
ADVERTISERS AND PUBLISHERS
54 OPEN POSITIONS IN PARIS’ R&D
http://criteolabs.com/jobs
5. criteolabs.com/jobs
Usage of Hadoop is growing exponentially
• Learning curve is real
• Analysts discover interesting things with raw data
– Which causes them to ask more questions
• Increased insight leads to a better product
– Which leads to more data
• Data gains in value and more is kept (and studied!)
• YOU (the admin) are the bottleneck !
USAGE GROWTH
8. criteolabs.com/jobs
Rack and load!
• Machine is racked, cabled and provisionned for a role
• Chef is our one stop-shop for automation
• Diskless system install
AUTOMATING DEPLOYMENTS
INSTA-
CLUSTER!
9. criteolabs.com/jobs
• Learn from the past
• Previous cluster 1.5 years operation
• 78% failure rate on /dev/sda at restart
• Disk usage symmetry
• Garanteed statelessness
OS DISKLESS : WHY
10. criteolabs.com/jobs
• PXE Boot on custom CentOs image
• Automated Chef bootstrap
• Everything done by Chef
– Inventory
– Firmware updates
– OS / Service deployment
OS DISKLESS : HOW
11. criteolabs.com/jobs
• Evolutive maintenance (version bump)
• Not much to do on normal ops
• Most freq. issue is flacking / slow performing host
• Use Preprod / Prod for infra changes
• Progressive VS black out
MAINTENANCE
14. criteolabs.com/jobs
• Hadoop is a DDOS to your infrastructure
– Increase ARP retention (L2-specific)
– Use NSCD
• Increase Read ahead
• Disable THP compaction
• MTU jumbo frames
SYSTEM CONFIGS
18. criteolabs.com/jobs
• One datacenter topology will not fit all
• Web traffic VS Hadoop traffic
• Historical Fat-tree hierarchy with layer 2 routing
• Switched to meshed design (soon layer3)
NETWORK TOPOLOGY
22. criteolabs.com/jobs
• Dedicated kdc / realm
• Dedicated services principals
• Cross-realm trusts
• Delegate user management to your IT
KERBEROS SETUP
23. criteolabs.com/jobs
• Use multiple proxies
• Easy way to interconnect to the outside world
• Data injection / read with a simple curl
• High bandwidth transfers
HTTPFS PROXIES
24. criteolabs.com/jobs
• Multiple use cases (ML, BI analytics)
• Baseline Json (+gzip) is ok
• Don’t optimize too early
• We still use it(*) at Peta scale
(*) some teams also use Parquet and contributed to Hive integration
FILE FORMATS
26. criteolabs.com/jobs
Did I say we’re hiring!
We’re hiring lots of engineers in 2014. Come join us!
http://criteolabs.com/jobs
MY FELLOW CRITEOS WOULD KILL ME…
Notas do Editor
http://www.shutterstock.com/pic.mhtml?id=95662684Who are we ?* Serving the right ad….* Slide wasimposed by MarketingYouprobablywillencounter the cloud versus in-house dilemniaKey factor is the elastic aspect ;we use our cluster 100% of the time ;wealready have DCs ;in-house waslessexpensive
This is the story of a growing and successfull startup usingHadoop.Growingmeansincreased volume. Successfullmeansbuckloads of cash to grow the infrastructure. Startup meansverysmall teams to manage the wholething.PoCiseasyWhenyou gain traction, everythingwill go fastWentfrom 12 nodes to 150 2 yearsago, to 600 today, above 1000 by the end of the year.Whyisitgrowingthatfast ?Virtuouscircle :Variousteam aregatheringskillsBI analysts: the more theyget, the more theywantHadoop shows mutualizationbenefits, platform to consolidate ad-hoc data processingtoolsYou business will boom thanks to hadoop adoption
Becauseyouneed to scale infrastructure:Automate operations (prod VS devops)Tune hadoop system (Hardware, Linux, Hadoopitself)Specifically networkThis is about scaling the infrastructure. Withhundreds of clients usinghadoop as a service, youalsoneed to scale infrastructure usage. For instance: multi-tenancy.Managing ressource contention Mapreduce, storageMaintainsecurity (user sandboxingthroughauthorization & authentication)Allowhadoop to beused as a service
Don’tdo anything by hand ; youwillhurtyourselfmanagingthousands of serversBuildthings once, runforeverThe choice of freedom : don’tbebound to a specificvendor ; eg. We use CDH4.5.0 right now, but could, and probablywill switch to HDPFull stack automation : frombare-metal to live service
Our cluster are turnkeydeployed once hosting and network have finishedtheirworkWeassigneverynode a role, and the hosts will boot and setup themselvesaccordingly
Why a diskless system ?You want maximumstoragedensityThereforeyouwant to fillthose 14 slots per server with 3TB drivesThereforeyouwill break a nicesymmetry if yourun the OS fromdiskNot theoretical: hands-on experience on 150-node cluster operated for 1.5 yearsHard constraint; but veryworthwhile:RemoteloggingcompulsoryNothinghiddenfrom automation system2GB of RAM per node (2% of RAM)
How do weachievethisMinimize size of diskless imageBoot chef as soon as youcan, and let it flow fromthereInitial chef roleis an inventoryrole. Chef used for management of updates, OS, service deployment.
Maintenance : * EvolutiveUpgradingyour distribution regularly (don’twant to lagbehind)* CorrectiveHadoopworksbetterwhenyoujustdon’ttouchit* HowEverythingistested on a PREPROD environmentProgressive deployment (rolling-out node by node) maybedisrupting for long running jobs
http://blog.cloudera.com/blog/2014/03/a-guide-to-checkpointing-in-hadoop/Monitor user facing interfaces : usersfrequentlyassimilatecluster’s condition to the JT’s or NN’s GUIMonitor yourJobtracker (MRv1 willeventuallygetstuck)MOST IMPORTANT OF ALL: CHECKPOINTINGMonitor the checkpoints of yourfsimage or youwill end-up with a namenode in reallybadshapeAt one point wewerehavingnearly 6 months of edits ;) 12 hours to start a NN ; urbanlegend of NN beingunsafe to restartMonitor HDFS disk usage and local disk usage
10:00http://www.flickr.com/photos/76588645
In a realworld system most of yourtaskswillbe IO boundReadahead ! Very importantWhenyou hit a performance bottleneck, the first thing to watch for is *outside* hadoop, becauseHadoopis a DOS to yourwhole infrastructureUse infrastructure local caches as much as youcan
Default parameters are usable for small clusters / smallnodesThese are examples, wehad to tune a significant part of themDetaillist of significantones + explanations
Default parameters are usable for small clusters / smallnodesThese are examples, wehad to tune a significant part of themLog settings theywillkillyour JT / NNHandler countsSeparate the thread pool for internal / external clients. Alsoeasier for firewallingHA has somedownsides (checkpointing)
One of the first thingthatyouwillgetwhenyou move yourhadoop cluster past a rack isyour network engineersyelling at you.Plan aheadyour network topology !
Fat treesuited to North/South traffic.Hadoop uses the network as a bus: East-Westlayer2 FabricPath/TrillLayer3 BGP
Soundsobvious but impementing a correct definition of the rack topologyregarding network isvery importantlldp information flackingdepending on which interface withask (4 interfaces bonding)
20:00Hadoopis a shared ressource.Whenyour usagegrowsyouwill face ressource starvation and contention. This will lead to twoproblems:1) Accountability: You willneed to report ressource accountabilitynumbers to plan for growth and optimize2) Maintain a good user experienceHDFS quotas ; but has bugs in fsimagecheckpointingScheduling ; user facingproblem ; requireseducation to understand the time/spacefolding; achievewelldesigned jobs (mapper ~ 3 to 15 minutes)YARN solvesmap/reduce ratio (+20% computing power)
Once yourealizeyourcompany ’s mostcritical data has landed on HadoopAnd youronlysecurity model isobscurityYouwillwant to switch to somethingbuilt-in and robustsecurity model.
Very good documentation fromCloudera, HortonworksNot sufficientthough:ironing out the problems (SPENGO) needs close integrationwith IT
Hadoop limitation: POSIX-levelaccess to HDFS HTTPFS worksaround the absence of a scalable(and workswithKerberos, too !)In oursystems, HDFS replacedcompletelyIsilonStreaming a sustained 200-400MB/s of logs into the clusterDon’tcreatebottlenecks ; address the connectivitywith a many-to-many pattern