Referers, downtime and caching

As you may have noticed the site has been down for a few weeks now. While it’s an eternity on the internet it flies by in realtime, especially when you’ve got so many other balls in the air. To make a long story shorter…

A few weeks ago I noticed the site was down with a “Forbidden” error. I thought the site might have been cracked so I created a trouble ticket with the host and made sure I had a recent file and database backup. As it turns out the site was disabled due to high CPU usage. Because referrer (or ‘referer’ as the standard spells it) spamming has been a growing problem I assumed the massive hits the site was getting was redlining the CPU and causing the problems. Because I was in the middle of CurrentJam I didn’t have time to deal with it so I setup a temporary page explaining the situation. Finally I got together witha few other people in the b2evolution forums. First, I did some rework on Ed’s referer recheck hack because I needed to do my development on 0.9.12 which was running on my dev server. After I got that working I whipped up the first version of the ReferThis! automatic htaccess generator. With some help we’re now up to build 3. Last night I turned the site back on with the htaccess file in place to limit hits from the bots but CPU usage was still too high for my host to be comfortable with. For the time being I’ve copied the site to another server and will be running it from there until I can bring down the CPU usage.

Thoughts on this are now revolving around some sort of caching system so I can serve semi-static pages to save CPU load. So far I’ve found this 2003 thought by the original b2evolution developer to be of some interest and the WP-Cache-2 project to be another good example of such a system. There’s also this 2002 early thought on the subject. Hopefully some other users will join me and help make this project a reality.

Comments are closed.