DDOS

We sometimes experience high system loads on git.l.o. You'll see that system-load is really high, but the CPU is basically in waiting. Generally cause of this is that we are being [D]Dos'd (sometimes by an innocent internal user, sometimes a hostile external user).

The DDoS works by doing lots (~20) parallel fetches via the "git" protocol. git:// protocol is known to be a DoS loophole with its implementation (which does realtime object repacking for each client - which takes a bit too much of RAM).

Some folks just banned it (Google), other just discourage (github). We tried to ban it, but due to internal outcry (and well, due to issues with http:// access, which by now should be resolved btw) had to move to just "discouraged" policy. But we seemed to bend too much, as now we have each project on git.linaro.org list git:// as available access method. And "discouraging" means not listing it (at least github does just that).

Ways to determine the hostile user:

  netstat -npe | grep ESTA | grep git
  # gives a decent overview of who is hammering the server 
  # Killing a bunch of those process, especially if all coming from the same IP, 
  # should lower the load.

  ps ax | grep pack-objects | awk ... | kill
  # sometimes you have to resort to "kill -KILL"

  # killing a bunch of stuff from an ip/subnet:
  for p in `sudo netstat -np | grep ESTA | grep /git | grep 58.251.152 | cut -dD -f 2 | cut -d/ -f1`; do sudo kill $p ; done
  sudo iptables -A INPUT -s 58.251.152.228 -j DROP

Gerrit

An attack can also cause review.l.o to fail. It can be restarted using the /srv/gerrit/bin/gerrit.sh script

Platform/Systems/GitProblemResolution (last modified 2014-08-27 17:03:53)