Check Your Sites With Google

by Curtiss   |   last updated September 17, 2010   |   General Tutorials ·

Earlier this week, I received a report that something fishy was going on with one of my websites. The report indicated that some sort of spam had infiltrated the site, informing users about great deals on pharmaceuticals. Needless to say, since we had not recently gone into the business of selling drugs (legal or otherwise), this was a bit suspicious.

I headed to the page that was included in the report and checked it out in about 20 different ways. I opened it in each of the five browsers I have installed; I viewed the regular source of the page; I viewed the generated source (after the JavaScript has run and modified the source) of the page and couldn’t find anything about the pharmaceuticals reported in the message.

Then, I happened to notice that the user agent listed in the report was “googlebot”. On a whim, I decided to run a quick Google search of our website (use the “site:” prefix, followed by your site’s URL) to see what Google had to say about the situation. Sure enough, some of our Google listings had the spammy titles and meta descriptions shown in the report.

After a lot of digging, I finally discovered that there is a nasty hack out there that somehow tunnels into your website, watches what you do, then inserts itself into random places; essentially hijacking your site. It is virtually undetectable if you never look for your own website on search engines. It only affects the way your site is rendered when googlebot, msnbot or the Yahoo! slurp is used to view it. For all regular visitors, your website is still the same as it’s always been.

The first thing I found was a file called “indexd.php” stored in the root of our public HTML directory. I downloaded that file to my computer and removed it from the Web server. After taking a look in the file, I found some pretty frightening information. This was a script that essentially scans your entire Web server looking for any and all possible vulnerabilities. Among many other things, it searches for all common variations of “conf”, “config”, etc. files on your server; it searches for all .htaccess and .htpasswd files; it searches for all of the common apache configuration files (httpd.conf, vhosts.conf, etc.); it searches for all files owned by apache, all files owned by root and much, much more. It also attempts to bruteforce some of the passwords used on the system.

It then injects random strings that will present the spam information to search robots, and finds a way to completely take over your website through that.

At the top of the indexd.php file, I found a reference to something called the “WebShell by oRB”. After a Google search (through which I had to be extremely careful, as more then 3/4 of the results appeared to lead to hacker/cracker sites telling you how to use the script rather than telling you how to get rid of it), I found out that this is apparently a variation of an older hacking script known as c99Shell. After a bit more research, I found that this exploit can be used as a trojan, as a simple Web script (if they can somehow gain access to simply upload files to your server) and a few other variations.

I also found that a good way to try to locate the hacked files is to perform a search on your Web server (using grep or a similar tool) to look for any strings that start with eval(base64_decode(, as the hackers insert the spam messages as base64_encoded strings (so you can’t simply do a search looking for the specific pharmaceutical information, I suppose).

Whenever you find files that include that code, you need to determine whether they are files that belong on your system or not. If they are, then you should remove the malicious code and resave them (if at all possible, it’s also probably a good idea to rename them). If they do not belong on your system, remove them altogether.

Then, run a grep search looking for files that reference the file you just modified or removed. Modify those files appropriately, as well.

After I found and rectified the first batch, I proceeded to change my password, the password of the server’s root user, the MySQL username and password and more. I also set out changing the names of some of the files that were hacked or that might have been vulnerable based on the information in the indexd.php file. I changed our DirectoryIndex file from index.php to something completely unrelated (if you’re going to do that, make sure you update your .htaccess file(s) appropriately, too). I changed our configuration file (the filename initially included the word “config”) and a few others.

The next day, I found that the hack had re-presented itself on our server. Once again, I did a search for the hacked files. I found them and remediated the situation again. The next day, it was back again. This time, they had used a preg_replace() statement instead of an eval(base64_decode()) statement (so, make sure you look for those, too).

They had also modified our main .htaccess file so that it redirected whenever a search engine spider came along to a phantom file called “wp-status.php” (this particular website doesn’t even run on WordPress, it just uses it for one small portion of the site) that the hack had generated. I removed the wp-status.php file and corrected our .htaccess file again.

At some point throughout this entire process, I also locked down the permissions on our server quite a bit. I was surprised to find out that, since apache actually requests and serves the files and documents on a Web server, as long as apache (or whatever your apache user is called on your server) is either the owner of the file or is in the group that owns the file, you don’t need to grant “others” any permissions to those files or directories at all. I was able to run chmod -R o-rwx on our entire Web directory without any negative consequences (of course, depending on your server configuration, that may not be the case for you).

Finally, I decided to shut down and restart our entire Web server (not just apache, but the whole box). Once it started back up again, the hack was back. I had to search for and remove even more hacked files. This led me to believe that the issue wasn’t simply one of having uploaded a malicious file or two to our server, the server itself was infected.

I asked our host to install an anti-virus solution on the server for us. I then ran the scanner. Although it didn’t detect any threats or issues, the exploit does not seem to have returned since it was installed. I am keeping my fingers crossed.

The moral of the story is, make sure you’re checking your websites in Google (and other search engines). I found two tools that are helpful in this.

1. Google Webmaster Tools offers an experimental feature called “Fetch as Googlebot” that will send a request, as the googlebot, to the URL you provide. It will then return the full headers and source of the page it retrieved.
   However, I also found out that this tool apparently has a weekly request limit (not mentioned anywhere on the page on which the tool is located), so if you request too many pages throughout the week, they’ll cut you off and won’t let you use the service anymore for a few days.
2. There is a Firefox extension called User Agent Switcher that lets you change the user-agent used by Firefox. You can then use Firefox to see what GoogleBot, MSNBot and Yahoo! Slurp see when they visit your site.

Thankfully, in our case, we caught the issue before it wreaked major havoc on our search results. Yes, Google indexed and cached a handful of pages on our site with the spam information, but not nearly as many as it could have. Since we caught it when we did, Google had only caught the pages on our site that are crawled most often, which means that, having hopefully eliminated the issue, Google is already fixing our results in its index.

Also, in our case, it doesn’t look like the hack gained access to our database (I imagine that, on other sites, the hack does more than just hack specific files, I’m sure it injects all kinds of nasty code into the database, too), so we were extremely lucky on that account.

Have you run across this issue before? If so, do you have any advice other than what I’ve laid out here? I’m hoping that the issue is gone, but I’m also still extremely worried that it will come back. I can’t imagine that I somehow managed to eradicate it altogether, so I’m really just waiting for the other shoe to drop. If you have any thoughts or tips as to where this thing might be hiding itself on our server, please share.