1. TCP socket backlog
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü
Î÷åðåäü íà ñîêåòå
1K -> 20K rpm
wargaming.net
Ïàäàåò êëàñòåð - âñå èäóò íà ñàéò:
Èíâàëèäàöèÿ êýøà - âñå æäóò: 20K -> 1K rpm
Ðåøåíèå:
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_max_syn_backlog = 32768
ÍÅ UNIX ñîêåò (ECONNREFUSED)
502 Bad Gateway

2. conntrack
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü
wargaming.net
net.ipv4.netfilter.ip_conntrack_max = 294912
net.ipv4.netfilter.ip_conntrack_count = 294118
Ïåðåïîëíåíèå:
[8727241.821130] ip_conntrack: table full, dropping packet.
[8727243.790372] ip_conntrack: table full, dropping packet.
[8727248.908501] ip_conntrack: table full, dropping packet.
[8727253.859720] ip_conntrack: table full, dropping packet.
[8727258.846404] ip_conntrack: table full, dropping packet.
[8727264.276727] ip_conntrack: table full, dropping packet.
Áîëüøîé conntrack (over 300K): 97.5%si
Ðåøåíèå: dport 80 -> NOTRACK

3. TIME_WAIT
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü
frontend äëÿ îáùåíèÿ ñ âíåøíèì ìèðîì
wargaming.net
áûñòðûé backend (1..10ms)
frontend <-> backend HTTP/1.0 (conn per req)
$ netstat -napt | awk '{print $6}' | sort | uniq -c
...
63564 TIME_WAIT
Íåõâàòêà èñõîäÿùèõ ïîðòîâ!
Ðåøåíèå:
net.ipv4.ip_local_port_range = 2000 61000
net.ipv4.tcp_max_tw_buckets = 131072
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1

4. Out Of TCP Mem
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü
3494-[9404385.020387] TCP: too many of orphaned sockets
wargaming.net
3496:[9404390.463850] Out of socket memory
Äèàãíîñòèêà:
$ cat /proc/net/sockstat
sockets: used 7385
TCP: inuse 5236 orphan 98 tw 2696 alloc 7126 mem 25421
UDP: inuse 10 mem 0
RAW: inuse 0
FRAG: inuse 0 memory 0
$ /sbin/sysctl -a | grep tcp_mem
net.ipv4.tcp_mem = 196608 262144 393216
$ /sbin/sysctl -a | grep orphans
net.ipv4.tcp_max_orphans = 131072

5. memcached
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü
wargaming.net
ïëîõî ìàñøòàáèðóåòñÿ thread-àìè (-t 64 çëî)
8192 pthread mutex íà âñå êëþ÷è,
-t 6 - OK! (íå áîëüøå)
Nthreads == Ncpu
Ìàñøòàáèðîâàòü ïðîöåññàìè!
Êîíñèñòåíòíîå êîëüöî êåøèðîâàíèÿ - íåñêîëüêî èíñòàíñîâ

6. nscd (bonus)
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü Êýøèðîâàíèå îòâåòîâ DNS
wargaming.net
nscd must have!