All I Wanted To Do Was Post Some Robocode

  • Posted on: 20 November 2014
  • By: davis

It started with a simple urge.

I just wanted to show some friends a simple ramming bot that I had worked on for Robocode. That's it.

So I started a post, and pasted my code in. Now to format it.. oh. It doesn't format by default with highlighting or any sort of markup. That's okay, I'll just hop over to Google and figure this out.

I had CKEditor and WYSIWYG installed on Drupal 7.32. I found this site which promised a seven step process to upload and activate the SyntaxHighlighter library. I just needed to install a few plugins, tweak a few options, and wham-o, blam-o, I'd be in business.

I dutifully installed the necessary files and diligently checked the correct options. I ran the scripts I was supposed to run. No success. I put the idea on a shelf and concentrated on work for a couple days. When I came back, I found that my site was down.

Not just down. Down implies that it's coming back up soon. 

Huge database issues and sorting through recovery options, failing one by one, all so I could now just salvage the articles I had on there. My site was down for three days - three days full of troubleshooting, learning, and frustration.

During this time, Drupal experienced a massive influx in exploitation of outdated installations. The exploit took advantage of un-updated Drupal cores to spread magnificently. Some authors have estimated that over 40% of Drupal sites on the web were compromised. Drupal composes 5.1% of known CMS systems in use on the Internet. That is an incredible market share. 

Drupal is used by 5.1% of all the websites whose content management system we know. This is 2.0% of all websites.

Drupal does not have auto-update for changes to the core. This seems like a stunning feature to omit in such a system. Hackers prey on information gaps - such is the nature of zero-day exploits, where hackers use the immediacy of a hack's release to maximize its effectiveness. This lack of auto-update (and the frequency with which Drupal upgrades go wrong, leading to issues) means that thousands of server admins across the world had to put their other projects on hold and deal with updating their live sites. The danger from the exploit was immediate and spreading quickly. Outdated systems were assuredly infected, according to Drupal itself. Thousands of prominent websites use Drupal. The White House uses Drupal. It is a BIG FUCKING DEAL.

If you did not update your site within < 7 hours of the bug being announced, we consider it likely your site was already compromised.

Here are some highlights of the extent of the intrusion:

  • Attackers may have created access points for themselves (sometimes called “backdoors”) in the database, code, files directory and other locations. Attackers could compromise other services on the server or escalate their access.

  • Notify the server’s administrator emphasizing that other sites or applications hosted on the same server might have been compromised via a backdoor installed by the initial attack

  • Consider obtaining a new server, or otherwise remove all the website’s files and database from the server. (Keep a copy safe for later analysis.)

  • While recovery without restoring from backup may be possible, this is not advised because backdoors can be extremely difficult to find. The recommendation is to restore from backup or rebuild from scratch.

  • Attackers may have copied all data out of your site and could use it maliciously. There may be no trace of the attack.

Amusingly (?), many of the hacked sites were also patched to the latest versions. The hackers had patched the installation on the way out, ensuring they were the only owners of the data.

Acquia has the probable source of the early attacks.

I've tracked down this attack to the IP address from Russia. Others on IRC confirmed being hit by the same IP and found their sites hacked. This IP hit our platform 36,786 times in the 10-16 03:00 to 10-17 13:00 time window. Those hits include:

  • the initial POST request to insert data in the database using the above query, taking advantage of the SQL injection on the user login form

  • several GET requests to the paths /dlov and /?q=dlov in order to install the backdoor in the docroot

  • more GET requests to access the backdoor PHP file (modules/aggregator/dlov.php).


Look at this incredible flowchart diagramming your chances of being infected. I'll skip to the end of the story - if you have a Drupal site, and you don't have 24/7 server admins, you were safe to assume that you had been compromised.

But let's bring this back to me. I just wanted to post some code, remember?

So I finally got my site back up and running. It was frustrating and made me consider just switching back to WordPress. I like WordPress, for all of its faults. It is easy to use, there are wide swathes of plugins avaliable, and tons of documentation. I complained to my girlfriend that Drupal was not intuitive, easy to use, or rewarding to work with. She cautioned against switching back to WordPress, which I am glad for. The challenge of getting Drupal working will be worth it in the future.

So here I am. Drupal 7.34 installed. Ready to jump back into the pit of installing the original SyntaxHighlighter libraries and double-checking my options. I found this guide, which works much better (and with less effort).

Installing it took about a half hour of messing around, and then updating permissions took a bit longer, as well as learning the classes needed for the SyntaxHighlighter, and what actions must take precedence in the filtered chain.

Anyways, I got it installed, played with the theme a bit, and I got it working. I'm proud of myself for even getting to this point - but my angst remains. Why is it so difficult, in this day and age, to accomplish somewhat trivial tasks? Not to underestimate the amount of work that has gone into developing this SyntaxHighlighter, but why is it so finicky?

Why is Drupal, a major platform used by international corporations, not equipped with a panic switch for emergency updates? Why, with all of the data stored on websites these days, does a CMS leave the task of staying up to date to lone server admins? 

I think the answer lays somewhere in  the realm of specialization increasing as people try to control more of their life, leading to specialization in virtually every field - which demands the frequent traversal of different knowledge sets and skills from someone trying to troubleshoot. We're building programs, protocols, and products based on 60+ years of layered advances in computing. Not every step is pretty, but some are ingrained for no good reasons. Any Linux user can tell you that resolving dependencies and troubleshooting new applications takes a huge amount of time. 

This is a new profession, computing. Agriculture, makeup, sports - these things all date back thousands of years. They have accepted traditions that are often borne out of thousands of years of experience. There is a great deal of institutional knowledge globally in these fields.

Meanwhile in computing, a new development environment is released every week. It's virtually impossible to keep up with the latest techniques without losing focus. Some developers are less focused on writing good code, and more interested in getting their product out to the public. Our modern day code is limited to primitive x86 architecture. We just finally moved past booting every machine into an outdated BIOS (thanks, UEFI). The pace of change is frenetic.

Takeaways: Drupal is not user or admin-friendly in its current iteration. Security issues happen to every platform, but not having an automatic update is foolish in this day and age. Companies cannot count on humans to update their vulnerable files. Doing the most mundane task on Drupal involves slogging through masses of documentation and trial and error. I am not against those things! But I do have my limits when I'm just trying to put up a goddamn post. I have stuck with Drupal for this site because I feel like it's building a valuable skillset. Sometimes, though, I wonder if it's worth the trouble.

If you'd actually like to see the Robot, follow this link or the .gif below.