This is geeky. Be warned.

There are some telltale signs when a server has been hacked – and nowadays I can smell them as clearly as I can smell bacon. Last night one of our servers exhibited the dead-cert-hack behaviour. To cut a long and tedious story short, I didn’t get to bed until around 3, and the problem was still not solved despite having access (albeit limited and a bit crap) to a KVM.
On reflection it’s hardly surprising the box got hacked – it was running Debian “Etch” which lost support in 2009. But what got me was how far wrong I was with my assumptions.

I’d assumed the rootkit was a kernel module and thus a right bugger to shift. It wasn’t. In fact it was an old fashioned SHV5 rootkit which nobbles programs like ifconfig, netstat, ps and friends. I didn’t think people still used them! Consequently they should be really easy to fix…but they weren’t. As root, trying to delete any of them failed with a “Permission Denied” error. This confirmed my theory that a kernel module was involved. The worst problem with this was that no matter how I tried to boot a clean kernel – these files were still immutable. It was really quite distressing.

After some Googling, I’d noticed a pattern in the clueless comments of users that asserted the protection of these binaries was down to filesystem attributes such as “immutable”. Obviously, in this day and age this was rubbish! If you 0wn the kernel, you can do what you like. But out of desperation, I tried using lsattr and chattr…and, fuck me, it worked!

The entire rootkit was removed without any special booting or rescue disk. Jesus guys – that is so 2000!

Meanwhile my colleague, and benefactor, on the other side of the globe, had been running some backups to protect against the worse case scenario. Our hacker friend was coming from a netblock in Poland, and no matter what I did, the fucker still managed to get back in. It was driving me mad!

I had blocked every single hole: pam libraries, sshd, etc etc yet the fucker was still able to glide in and do his bandwidth-destroying work!

It was only during an IM session with my colleague that a correlation became apparent: every time he started a backup job, it got killed. I checked his IP address and realised that for the past few hours I was battling against him! The crackers hadn’t tried anything else all day – I was tilting at Windmills (well, at my friend).

In defence, his netblock was very similar to our Polish friend’s. But it doesn’t stop me feeling like a bit of a prick.