We got hacked

March 22, 2013 by Nikita Salnikov-Tarnovski

Two weeks ago we got hacked. Which was something we have anticipated for the long time – after all, you cannot expect to run a reasonable business and not be a target of some malicious attacks. So as we finally reached our “goal” we had to face the consequences implied.

Full disclosure: no customer data was exposed during the attack and no Plumbr executables were harmed during the attack.

I discovered the problem during my AWS backup process. I created a new AMI by creating a clone from an existing machine and forgot to tick the checkbox “Do not reboot”. So I ended up rebooting the original machine. Not too bad I thought while waiting for the machine to revive. But 15 minutes later had to acknowledge that I do not have SSH access to the rebooted machine anymore. Machine was up and running as our services were available, but the SSH connections all got a timeout message.

I was nowhere near a panic mode yet, as I googled and found out that it can relate to magically failed boot procedures, so I did a few more restarts. No luck. Then I migrated the instance to a different availability zone. Unfortunately without any success to show for. But I was still under the impression that mr Jeff Bezos is to blame and started to figure out how can I access the box without a SSH access.

My first idea was about launching the new virtual machine and attach old EBS volume to it so I could check the file system in it. But being a lazy guy I postponed this one for a while. And then my inner genius woke up and reminded me of few things:

  1. We were running a Jenkins instance in this machine.
  2. Jenkins has a  built-in Groovy console
  3. This console can be used to launch OS level commands
  4. The user running Jenkins was a full passwordless sudo user.

So A & B & C & D = gigantic security loophole, but in this case it was beneficial for me as I now had the access to the machine.

And ten minutes later I was frightened out. First stop – somebody had deleted the SSH server configuration. Which I initially thought was the evil me doing things I should not be doing. But then the back hair started to raise as I expect most of you would if you find yourself staring at a C source named backdoor.h.

Now it was time to call the cavalry. Or namely – my fellow co-founder who just happens to have a lot more experience in the field of hacking and getting hacked. 30 minutes after Vladimir stepped in he had figured out that we were running on an OS not patched for six months containing a privilege escalation vulnerability. From there it was clear that the backdoor installed in our backyard was a well-known SSH exploit harvesting SSH passwords and periodically sending the harvested crop to an external IP. Luckily we were not using username/password to access this machine, but had set it up using a Public Key Infrastructure, so no passwords were neither gathered nor sent.

But ok, i needed to restore the SSH configuration to proceed. Via the very same Groovy console I managed to copy the configuration from a different AWS machine to which I luckily had SSH access using a public key. And sudo service sshd restart later I had the access restored.

So – to summarize – we were hit by a privilege escalation vulnerability used to upload C code compiled to a SSH hack. Which is one of the tools in script kid libraries. In this case it seems it was indeed a kid who did not understand how to install a proper rootkit and cover up his tracks.

We considered that there is a chance that all the existing instances could be compromised. So we spent the next couple of days migrating the whole infrastructure to new machines with a bit more focus on security this time.

But the saddest part for my inner genius happened during the aftermath. I was proudly presenting my clever thoughts on how to access the hacked machine via the Jenkins-Groovy hack, when I got a truly surprised stare. And a question – “You can launch any shell script through Jenkins, why bother with this Groovy madness?” And indeed, this time I had outsmarted myself.

In the case you followed us this far you might also be interested in our future posts, so go ahead and subscribe to our Twitter feed. Which I hope will not contain more security breaches anymore – we promise to stick to our bread and butter and write about Java performance optimization topics instead.

Can't figure out what causes your OutOfMemoryError? Read more

ADD COMMENT

COMMENTS

High security is a performance bottleneck ;)

Priit Liivak

Did you run any web application penetration test over your website in order to resuct break-ins?

Paolo Perego

Thank you, that was an interesting article.

iNikem

Thanks for posting openly. Did you have Jenkins configured with any authentication?

Nick

Sure. Wide-open Jenkins would be too much :)

iNikem

So, any ideas how the attacker got through the Jenkins auth? You identified other vulnerabilities (Jenkins running with sudo permissions, out-of-date kernel), but not how the attacker got in to the Jenkins Groovy console. Thanks

Nick

If the box was that far behind on patches, there is every chance the person got in through another vulnerability in a different package.

Twirrim

U so smart….

dod

Just a small note – those links to Wikipedia’s articles and similar for SEO and better page rankings are really annoying. I think the audience here are not morons.

Rob

I think any webapp server should run in a chrooted environment.

Regis

IMO, we have a very long road to go, before that be a case in Java ecosystem.

iNikem

I somehow doubt that you are using a PKI (i.e. certificates signed by a CA) to log into SSH. Did you mean public key based authentication?

nobody22

That is correct.

Vladimir Šor

Most freak-outs are people’s wild imagination. I wrote a C compiler. God is god, same as in the Bible.

Terrence Andrew Davis

Your ridiculous copy/paste script is still broken.

Cole Mickens

I’m fairly confused. Without the famous reboot you would have not noticed the hack, further more, the system had been out of date for 6 months.
So I wonder how you can you be so sure that the hacker have done nothing more — during an unknown time span — and that no user data was exposed ?
Could you elaborate more on this, because by looking at this article all we can see is amateurism.

Nevor

Analyzing log files gives us the confidence that besides the attempt to harvest SSH password nothing else draw the attention of the hacker. And we saw no evidence of those logs being tampered.

Ivo Mägi

Just because you saw no evidence of log tampering doesn’t mean they didn’t do it. Any self-respecting hacker would attempt to cover their tracks in the syslogs. On the other hand, you’re right that they probably would have tried to cover their entry method too.

Matt

Can't figure out what causes your OutOfMemoryError? Read more

Latest
Recommended
You cannot predict the way you die
When debugging a situation where systems are failing due to the lack of resources, you can no longer count on anything. Seemingly unrelated changes can trigger completely different messages and control flows within the JVM. Read more
Tuning GC - it does not have to be that hard
Solving GC pauses is a complex task. If you do not believe our words, check out the recent LinkedIn experience in garbage collection optimization. It is a complex and tedious task, so we are glad to report we have a whole lot simpler solution in mind Read more
Building a nirvana
We have invested a lot into our continuous integration / delivery infrastructure. As of now we can say that the Jenkins-orchestrated gang consisting of Ansible, Vagrant, Gradle, LiveRebel and TestNG is something an engineer can call a nirvana. Read more
Creative way to handle OutOfMemoryError
Wish to spend a day troubleshooting? Or make enemies among sysops? Registering pkill java to OutOfMemoryError events is one darn good way to achieve those goals. Read more