Tag Archives: compromised

Why Heartbleed did more to increase security than any friendly advice you ever gave

Doing IT operations during these heartbleeding times isn’t easy. If you’ve been following even remotely the Heartbleed saga, you know it’s a critial design flaw in the most popular implementation of the most popular encryption protocol, OpenSSL. To be plain, most of the credentials you held in the recent past are potentially compromised, since even if eveyone are now updated and protected, your credentials might have been stolen beforehand.

If you indeed do operations, you know it gets painful logging into another web console of some service provider to see a warning banner with the latest security advisory about Heartbleed. It usually goes in the same spirit of “please reset all your passwords, please recycle all your keys, please reissue all your certificates”. And the more responsible of us do: We reset all our passwords, we annihilate the deployment keys we used for the last year, and certificates older than the buildings we work in are being reissued. We update and patch all our servers, and we have a lot of servers. We mass-email users to change their login credentials, and go on a frenzy of forcing everyone to enable 2-factor authentication and security questions. And we’re probably just getting started.

We press hard, and the users, for once as terrified as we are, are being cooperative. Hours are spent on memorizing, forgetting and resetting new passwords, patching servers, and fixing outages caused by revoked keys and certificates. Everyone gets a crash course in encryption. Security is on everyone’s agenda. Users who thought certificates were for gifts now ask about SSL. Naturally, educating, updating, revoking and reissuing is time consuming, and the regular workday goes down the drain. Yet another compromised vendor or service, yet another unpatched server.

And here you should stop and think about what you are going through: if you do value security with more than just your lips, you realize this is one of the best things that happened to your operation. You are basically forced to do the very same things you knew you should have been doing all along, but always left for later. Recycling my passwords? New ones are hard to remember! Reissuing certificates on a regular basis? What kind of a sadist would put that in OpenVPN’s Wiki! Changing deployment keys? Doesn’t look amazing on anyone’s resume! Just stop and try to remember how many times, before Heartbleed, have your users/coworkers/clients cared about the same thing you do? How many times have they came to you asking you about updating a server, instead of you suggesting to them in a friendly manner they should really apply that last year of updates on their corporate laptop?

Heartbleed is the best thing that happened to network security since firewalls: Very few credentials were likely compromised via Heartbleed, but everyone is rotating their credentials and protecting themselves from more common vulnerabilities and leaked passwords. It’s hard to appreciate it between all the password resets, but when you are done with it all, sit back and have a beer. To Heartbleed!




1 Comment

Filed under #!

Checking for Rootkits

What are rootkits? You can ask Wikipedia, or take this over-simplification:

The Windows flock has to worry about various security concerns – viruses, worms, trojans, bots. So may, in fact, they had been categorized into AdWare (unwanted commercial content), SpyWare (compromise of personal information) and Viruses (generally malicious code).

Linux folks, on the other hand, have to fear only one thing – a rootkit. Because Windows vulnerabilities allow exploiting a machine without administrative access for specific goals, a diversity of exploits was invented, while properly setup Linux can only be compromised with root (Administrator) access. However, once that had been achieved, the machine is fully compromised along with the network it’s on. Namely, there are very few things an attacker will not be able to do with a rooted Linux machine.

Periodic scans are a good measure to stay clean, similar to scheduled anti-virus scans in Windows PCs. chkrootkit is such an implementation, and appears to be the most popular today, for good reasons: it’s easy to build, easy to run, and easy to understand. A good idea might be to run it periodically and email the output to a system administrator.

Great, until you realize the catch22 of checking for root-kits: Once a machine had been infected with a rootkit (usually through an badly secured SSH account), there’s virtually no way to determine that is was compromised, unless a reference set of binaries can be used to compare the infected binaries to “clean” ones. This means that you may not use a machine to check itself for rootkits, because if it was already compromised at the time it was checked, the check would be useless.

There is, of course, a hack. Namely, Network Security Hack #99 from O’Reilly suggests we can run chkrootkit in Busybox! Right, Busybox has some 200 programs built into its binary, and we could use those instead of the host’s own suspected ones. Unfortunately, it does not show how.

Let’s think about this: We assume the machine might be compromised when we scan it. We would not want to use the machine’s own binaries (that could also be compromised). We would not want to use software that had been lying around on the machine while it was compromised, for the same reason, including Busybox or chkrootkit. Ideally, then, we would use a fresh copy of Busybox to download a fresh copy of chkrootkit, run one in the other and dump both (that way, one cannot compromise something that is not there).

I ended up with the (almost entirely self-sufficient) solution below. It will run on any machine that has internet access, GNU wget (to download Busybox and chkrootkit). Other than wget, the script makes use of Busybox applets instead of the system installed binaries, for extra safety 😉


# Temporary folder path. (It's a good idea to use a random name):

# A URL from which Busybox binaries can be downloaded

# A URL from which the chkrootkit source can be downloaded:

# Download and prepare the Busybox binary in the temporary folder:
/usr/bin/mkdir -p $TEMPORARY_FOLDER
/usr/bin/wget $BUSYBOX_URL/busybox-"`uname -m`"
/usr/bin/chmod +x $TEMPORARY_FOLDER/busybox-"`uname -m`"
BUSYBOX_BIN=$TEMPORARY_FOLDER/busybox-"`uname -m`"

# Download, confirm the MD5 sum, and extract the chkrootkit source using Busybox:
$BUSYBOX_BIN wget $CHKROOTKIT_URL/chkrootkit.tar.gz
$BUSYBOX_BIN wget $CHKROOTKIT_URL/chkrootkit.md5
if [ "`$BUSYBOX_BIN cat chkrootkit.md5`" != "`$BUSYBOX_BIN md5sum chkrootkit.tar.gz`" ]; then
  $BUSYBOX_BIN echo " !!! MD5 SUM check FAILED !!! "
  $BUSYBOX_BIN echo "MD5 checksum for chkrootkit source passed..."
$BUSYBOX_BIN tar vxf $TEMPORARY_FOLDER/chkrootkit.tar.gz

# Run the newly built chkrootkit in the just downloaded Busybox:
$BUSYBOX_BIN sh $TEMPORARY_FOLDER/chkrootkit*/chkrootkit

# Clean up:
cd /
$BUSYBOX_BIN echo "Done."


Filed under #!