Lessons learnt on a 2000-core cluster

Running our software on a 2000-core cluster Lessons learnt

Structure For each problem Symptoms Method of investigation Cause Action taken Morale

Background Pretty simple: Distributing embarassingly parallel computations on a cluster Distribution fabric is RabbitMQ Publish tasks to queue Pull results from queue Computational listeners on cluster nodes Tasks are “fast” (~1s cpu time) or “slow” (~15min cpu time) Tasks are split into parts (usually 160) Also parts share the same data chunk – it’s stored in memcached and task input contains the “shared data id” Requirements: 95% utilization for slow tasks, “as much as we can” for fast ones.

RabbitMQ starts refusing connections to some clients when there are too many of them.

Investigation Eventually turned out RabbitMQ supports max ~400 connections per process on Windows.

Solution In RabbitMQ: Establish a cluster of RabbitMQs 2 “eternal” connections per client, 512 connections per instance, 1600 clients  ~16 instances suffice. Instances start on same IP, on subsequent ports (5672,5673..) In code: Make both submitter and consumer scan ports until success

Morale Capacity planning! If there’s a resource, plan how much of it you’ll need and with what pattern of usage. Otherwise you’ll exhaust it sooner or later. Network bandwidth Network latency Connections Threads Memory Whatever

RabbitMQConsumer uses a legacy component which can’t run concurrent instances in the same directory

Solution Create temporary directory. Directory.SetCurrentDirectory() at startup. The temp directories pile up.

Solution At startup, clean up unused temp directories. How to know if it is unused? Create a lock file in the directory At startup, try removing lock files and dirs Problem Races: several instances want to delete the same file All but one crash! Several solutions with various kinds of races, “fixed” by try/ignore band-aid… Just wrap the whole “clean-up” block in a try/ignore! That’s it.

Morale If it’s non-critical, wrap the whole thing with try/ignore Even if you think it will never fail It will (maybe in the future, after someone changes the code…) Thinking “it won’t” is unneeded complexity Low-probable errors will happen The chance is small but frequent 0.001 probability of error, 2000 occasions = 87% that at least 1 failure occurs

Then the thing started working. Kind of. We asked for 1000 tasks “in flight”, and got only about 125.

Gateway is highly CPU loaded (perhaps that’s the bottleneck?)

Solution Eliminate data compression It was unneeded – 160 compressions of <1kb-sized data per task (1 per subtask)! Eliminate unneeded deserialization Eliminate Guid.NewGuid() per subtask It’s not nearly as cheap as one might think Especially if there’s 160 of them per task Turn on server GC

Solution (ctd.) There was support for our own throttling and round-robining in code We didn’t actually need it! (needed before, but not now) Eliminated both Result Oops, RabbitMQ crashed!

Cause 3 queues per client Remember “Capacity planning”? A RabbitMQ queue is an exhaustable resource Didn’t even remove unneeded queues Long to explain, but Didn’t actually need them in this scenario RabbitMQ is not ok with several thousand queues rabbimqctl list_queues took an eternity

Solution Have 2 queues per JOB and no cancellation queues Just purge request queue OK unless several jobs share their request queue We don’t use this option.

And then it worked Compute nodes at 100% cpu Cluster quickly and sustainedly saturated Cluster fully loaded

Morale Eliminate bloat – Complexity kills Even if “We’ve got feature X” sounds cool Round-robining and throttling Cancellation queues Compression

Morale Rethink what is CPU-cheap O(1) is not enough You’re going to compete with 2000 cores You’re going to do this “cheap” stuff a zillion times

Morale Rethink what is CPU-cheap 1 task = avg. 600ms of computation for 2000 cores Split into 160 parts 160 Guid.NewGuid() 160 gzip compressions of 1kb data 160 publishes to RabbitMQ 160*N serializations/deserializations It’s not cheap at all, compared to 600ms Esp. compared to 30ms, if you’re aiming at 95% scalability

And then we tried short tasks ~1000x shorter

Oh well. The tasks are really short, after all…

And we started getting really a lot of memcached misses.

Investigation Have we put so much into memcached that it evicted the tasks? Log: Key XXX not found > echo “GET XXX” | telnet 123.45.76.89 11211 YYYYYYYY Nope, it’s still there.

Solution Retry until ok (with exponential back-off)

Desperately retrying Blue: Fetching from memcached Orange: Computing Oh.

Investigation Memcached can’t be down for that long, right? Right. Look into code… We cached the MemcachedClient objects to avoid creating them per each request because this is oh so slow

Investigation There was a bug in the memcached client library (Enyim) It took too long to discover that a server is back online Our “retries” were not actually retrying They were stumbling on Enyim’s cached “server is down”.

Solution Do not cache the MemcachedClient objects Result: That helped. No more misses.

Morale Eliminate bloat – Complexity kills I think we’ve already talked of this one. Smart code is bad because you don’t know what it’s actually doing

Then we saw that memcached gets take 200ms each

Investigation Memcached can’t be that slow, right? Right. Then who is slow? Who is between us and memcached? Right, Enyim. Creating those non-cached Client objects

Solution Write own fat-free “memcached client” Just a dozen lines of code The protocol is very simple. Nothing stands between us and memcached(well, except for the OS TCP stack) Result: That helped. Now gets took ~2ms.

Morale Eliminate bloat – Complexity kills Should I say more?

And this is how well we scaled these short tasks. About 5 1-second tasks/s. Terrific for a 2000-core cluster.

Investigation These stripes are almost parallel! Because tasks are round-robined to nodes in the same order. And this round-robiner’s not keeping up. Who’s that? RabbitMQ. We must have hit RabbitMQ limits ORLY? We push 160 messages per 1 task that takes 0.25ms on 2000 cores. Capacity planning?

Investigation And we also have 16 RabbitMQs. And there’s just 1 queue. Every queue lives on 1 node. 15/16 = 93.75% of pushes and pulls are indirect.

Solution Don’t split these short tasks into parts. Result: That helped. ~76 tasks/s submitted to RabbitMQ.

And then this An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.(during connection to the Gateway) Spurious program crashes in Enyim code under load

Solution Update Enyim to latest version. Result: Didn’t help.

Solution Get rid of Enyim completely. (also implement put() – another 10 LOC) Result: That helped No more crashes Postfactum: Actually I forgot to destroy the Enyim client objects 

Morale Third-party libraries can fail They’re written by humans Maybe by humans who didn’t test them under these conditions (i.e. a large number of connections occupied by the rest of the program) YOU can fail (for example, misuse a library) You’re a human Do not fear replacing a library with an easy piece of code Of course if it is easy (for memcached it, luckily, was) “Why did they write a complex library?” Because it does more, but maybe not what you need.

But we’re still there at 76 tasks/s.

Solution A thourough and blind CPU hunt in Client and Gateway. Didn’t want to launch a profiler on the cluster nodesbecause RDP was laggy and I was lazy (Most probably this was a mistake)

Solution Fix #1 Special-case optimization for TO-0 tasks: Unneeded deserialization and splitting in Gateway (don’t split them at all) Result Gateway CPU load drops 2x Scalability doesn’t improve

Solution Fix #2 Eliminate task GUID generation in Client Parallelize submission of requests To spread WCF serialization CPU overhead over cores Turn on Server GC Result Now it takes 14 instead of 20s to push 1900 tasks to Gateway (130/s). Still not quite there.

Look at the cluster load again Where do these pauses come from? They appear consistently on every run.

Where do these pauses come from? What can pause a .NET application? The Garbage Collector The OS (swap in/out) What’s common between these runs? ~Number of tasks in memory at pauses

Where did the memory go? Node with Client had 98-99% physical memory occupied. By whom? SQL Server: >4Gb MS HPC Server: Another few Gb No wonder.

Solution Turn off HPC Server on this node. Result: These pauses got much milder

Still don’t know what’s this. About 170 tasks/s. Only using 1248 cores. Why? We don’t know yet.

Morale Measure your application.Eliminate interference from others. The interference can be drastic. Do not place a latency-sensitive component together with anything heavy (throughput-sensitive) like SQL Server.

But scalability didn’t improve much.

How do we understand why it’s so bad? Eliminate interference.

What interference is there? “Normalizing” tasks Deserialize Extract data to memcached Serialize Let us remove it (prepare tasks, then shoot like a machinegun). Result: almost same – 172 tasks/s (Unrealistic but easier for further investigation)

So how long does it take to submit a task? (now that it’s the only thing we’re doing) Client: “Oh, quite a lot!” Gateway: “Not much.” 1 track = 1 thread. Before BeginExecute: start orange bar, after BeginExecute: end bar.

Duration of these bars Client: “Usually and consistently about 50ms.” Gateway: “Usually a couple ms.”

Very suspicious What are those 50ms? Too round of a number. Perhaps some protocol is enforcing it? What’s our protocol?

What’s our protocol? tcp, right? var client = new CloudGatewayClient( "BasicHttpBinding_ICloudGateway"); Oops.

Solution Change to NetTcpBinding Don’t remember which is which :(Still looks strange, but much better.

About 340 tasks/s. Only using 1083 of >1800 cores! Why? We don’t know yet.

Morale Double-check your configuration. Measure the “same” thing in several ways. Time to submit a task, from POV of client and gateway

Here comes the dessert. “Tools matter” Already shown how pictures (and drawing tools) matter. We have a logger. “Greg” = “Global Registrator”. Most of the pictures wouldn’t be possible without it. Distributed (client/server) Accounts for machine clock offset Output is sorted on “global time axis” Lots of smart “scalability” tricks inside

Tools matter And it didn’t work quite well, for quite a long time. Here’s how it failed: Ate 1-2Gb RAM Output was not sorted Logged events with a 4-5min lag

Tools matter Here’s how its failures mattered: Had to wait several minutes to gather all the events from a run. Sometimes not all of them were even gathered After the problems were fixed, “experiment roundtrip” (change, run, collect data, analyze) skyrocketed at least 2x-3x.

Tools matter Too bad it was on the last day of cluster availability.

Why was it so buggy? The problem ain’t that easy (as it seemed). Lots of clients (~2000) Lots of messages 1 RPC request per message = unacceptable Don’t log a message until clock synced with the client machine Resync clock periodically Log messages in order of global time, not order of arrival Anyone might (and does) fail or come back online at any moment Must not crash Must not overflow RAM Must be fast

How does it work? Client buffers messages and sends them to server in batches (client initiates). Messages marked with client’s local timestamp. Server buffers messages from each client. Periodically client and server calibrate clocks (server initiates).Once a client machine is calibrated  its messages go to global buffer with transformed timestamp. Messages stay in global buffer for 10s (“if a message is earliest for 10s, it will remain earliest”) Global buffer(windowSize): Add(time, event) PopEarliest() : (time,event)

So, the tricks were: Limit the global buffer (drop messages if it’s full) “Dropping message”…“Dropped 10000,20000… messages”…”Accepting again after dropping N” Limit the send buffer on client Same Use compression for batches (unused actually) Ignore (but log) errors like failed calibration, failed send, failed receive, failed connect etc Retry after a while Send records to server in bounded batches If I’ve got 1mln records to say, I shouldn’t keep the connection busy for a long time (num.concurrent connections is a resource!). Cut into batches of 10000. Prefer polling to blocking because it’s simpler

So, the tricks were: Prefer “negative feedback” style Wake up, see what’s wrong, fix Not: “react to every event with preserving invariants”Much harder, sometimes impossible. Network performance tricks: TCP NO_DELAY whenever possible Warm up the connection before calibrating Calibrate N times, average until confidence interval reached (actually precise calibration is theoretically impossible, only if network latencies are symmetric, which they aren’t…)

And the bugs were: Client called server even if it had nothing to say. Impact: *lots* of unneeded connections. Fix: Check, poll.

And the bugs were: “Pending records” per-client buffer was unbounded. Impact: Server ate memory if it couldn’t sync clock Reason: Code duplication. Should have abstracted away “Bounded buffer”. Fix: Bound.

And the bugs were: If couldn’t calibrate with client at 1st attempt, never calibrated. Impact: Well… Esp. given the previous bug. Reason: try{loop}/ignore instead of loop{try/ignore} Meta reason: too complex code, mixed levels of abstraction Mixed what’s being “tried” with how it’s being managed (failures handled) Fix: change to loop{try/ignore}. Meta fix: Go through all code, classify methods into “spaghetti” and “flat logic”. Extract logic from spaghetti.

And the bugs were: No calibration with a machine in scenario “Start client A, start client B, kill client A” Impact: Very bad. Reason: If client couldn’t establish a calibration TCP listener, it wouldn’t try again (“someone else’s listening, not my job”).Then that guy dies and whose job is it now? Meta reason: One-time global initialization for a globally periodic process (init; loop{action}).Global conditions change and initialization is needed again. Fix: Transform to loop{init; action} – periodically establish listener (ignore failure).

And the bugs were: Events were not coming out in order. Impact: Not critical by itself, but casts doubt on the correctness of everything.If this doesn’t work, how can we be sure that we even get all messages?All in all, very bad. Reason: ??? And they were also coming out with a huge lag. Impact: Dramatic (as already said).

The case of the lagging events There were many places where they could lag. That’s already very bad by itself… On client? (repeatedly failing to connect to server) On server? (repeatedly failing to read from client) In per-client buffer? (failing to calibrate / to notice that calibration is done) In global buffer?(failing to notice that this event has “expired” its 10s)

The case of the lagging events Meta fix: More internal logging Didn’t help. This logging was invisible because done with Trace.WriteLine and viewed with DbgView, which doesn’t work between sessions My fault – didn’t cope with this. Only failed under large load from many machines (the worst kind of error…) But could have helped. Log/assert everything If things were fine where you expect them to be, there’d be no bugs.But there are.

The case of the lagging events Investigation by sequential elimination of reasons. The most suspicious thing was “time-buffered queue”. A complex piece of mud. “Kind of” a priority queue with tracking times and sleeping/blocking on “pop” Looked right and passed tests, but felt uncomfortable Rewritten it.

The case of the lagging events Rewritten it. Polling instead of blocking: “What’s the earliest event? Has it been here for 10s yet?” A classic priority queue “from the book” Peek minimum, check expiry  pop or not. That’s it. Now the queue definitely worked correctly. But events still lagged.

The case of the lagging events What remained? Only a walk through the code.

The case of the lagging events A while later…

The case of the lagging events A client has 3 associated threads. (1 per batch of records) Thread that reads them to per-client buffer. (1 per client) Thread that pulls from per-client bufferand writes calibrated events to global buffer(after calibration is done) (1 per machine) Calibration thread

The case of the lagging events A client has 3 associated threads. And they were created in ThreadPool. And ThreadPool creates no more than 2 new threads/s.

The case of the lagging events So we have 2000 clients on 250 machines. A couple thousand threads. Not a big deal, OS can handle more. And they’re all doing IO. That’s what an OS is for. Created at a rate of 2 per second. 4-5 minutes pass before the calibration thread is created in pool for the last machine!

The case of the lagging events Fix: Start a new thread without ThreadPool. And suddenly everything worked.

The case of the lagging events Why did it take so long to find? Unreproducible on less than a dozen machines Bad internal debugging tools (Trace.WriteLine) And lack of understanding of their importance Too complex architecture Too many places can fail, need to debug all at once

The case of the lagging events Morale: Functional abstractions leak in non-functional ways. Thread pool functional abstraction = “Do something soon” Know how exactly they leak, or don’t use them. “Soon, but no sooner than 2/s”

Greg again Rewritten it nearly from scratch Calibration now also initiated by client Server only accepts client connections and moves messages around the queues Pattern “Move responsibility to client” – server now does a lot less calibration-related bookkeeping Pattern “Eliminate dependency cycles / feedback loops” Now server doesn’t care at all about failure of client Pattern “Do one thing and do it well” Just serve requests. Don’t manage workflow. It’s now easier for server to throttle the number of concurrent requests of any kind

The good partsOK, lots of things were broken. Which weren’t? Asynchronous processing We’d be screwed if not for the recent “fully asynchronous” rewrite “Concurrent synchronous calls” are a very scarce resource Reliance on a fault-tolerant abstraction: Messaging We’d be screwed if RabbitMQ didn’t handle the failures for us Good measurement tools We’d be blindfolded without the global clock-synced logging and drawing tools Good deployment scripts We’d be in a configuration hell if we did that manually Reasonably low coupling We’d have much longer experiment roundtrips if we ran tests on “the real thing” (Huge Legacy Program + HPC Server + everything) It was not hard to do independent performance optimizations of all the component layers involved (and there were not too many layers)

Morales Tools matter Would be helpless without the graphs Would have done much more if the logger was fixed earlier… Capacity planning How much of X will you need for 2000 cores? Complexity kills Problems are everywhere, and if they’re also complex, then you can’t fix them Rethink “CPU cheap” Is it cheap compared to what 2000 cores can do? Abstractions leak Do not rely on a functional abstraction when you have non-functional requirements Everything fails Especially you Planning to have failures is more robust than planning how exactly to fight them There are no “almost improbable errors”: probabilities accumulate Explicitly ignore failures in non-critical code Code that does this is larger but simpler to understand than code that doesn’t Think where to put responsibility for what Difference in ease of implementation may be dramatic

Lessons learnt on a 2000-core cluster

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Lessons learnt on a 2000-core cluster

Semelhante a Lessons learnt on a 2000-core cluster (20)

Último

Último (20)

Lessons learnt on a 2000-core cluster