Given all that I've previously written about what interesting entries show up in my log files (all of which you should now be able to find tagged in the logged section), you might be a little surprised that I haven't said anything about that after having moved to the VPS. The reason is because I wanted to give it a little time to settle down, so now that a couple weeks have passed I finally have something to say.
A big part of the reason to take some extra time is because the exposure you have to worry about for a VPS is much greater than for a web host. On a shared host, everything is essentially managed for you. If there is any log file you can even access, it's usually just the web server's HTTP logs. For a VPS, on the other hand, you can run any service you'd like, but then you have to manage it yourself and there are any number of log files that you'd do well to keep an eye on. The scale of the potential abuse is just that much greater.
Out with the old: .htaccess
The main tool I used to have to combat impossibly stupid spammers was to make
deny from entries in the Apache config file. I would scan my log files for abusers on a roughly weekly basis, and I'd drop the IP blocks of the worst offenders into a list, which would then be added to the .htaccess file for each major site I ran. Here are the 1800+ items that have accumulated in that list over time.
Once updated, that config file would then be uploaded to the server at some later date, at which point the abusers would start getting 403 errors from the web server. It was all fairly automated and hassle free, but there were some annoying delays between when the abuse started and when it was taken care of. Still, it was about the best I could hope for from a shared hosting environment.
In with the new: iptables + fail2ban
And while the down side of running a VPS is that there is a greater attack vector, the up side is that you have more tools in your arsenal to fight back with. The heavy hitter here is the
iptables firewall software that should come installed with any Linux VPS. I'm not going to bother to go into the details of how it works here, other than to say it nicely takes care of the same sorts of banning duties that the .htaccess
deny from entries did. The big bonus here being that
iptables works for any service, and it doesn't even bother to bother Apache; packets from spammers just get dumped immediately. I've only added about 100 IP blocks so far, and I'm actually going to be more forgiving than I was before and not permanently save the list, giving spammers a chance to clean up their act and (possibly) be unbanned when they come back in the future.
The second major component of my abuse toolbox is
fail2ban. It functions like the log file scanning software I used before, with the big plus being that it does its scanning in real time, and automatically takes care of adding the worst offenders to
iptables for you! It, too, prefers temporary bans (called “jails”), which is great for many things, but there are still an awful lot of bad people out there who will keep coming back and and messing with your server at any chance they get.
Handling repeat offenders
The basic scenario/question that gets asked is something like this:
OK, so I can set up a 10 minute long (or whatever) ban for someone scanning me for vulnerabilities 5 times in 5 minutes, but if they keep doing that again and again, how do I escalate the ban to lock them out for, say, 1 day if they've hit the shorter ban 3 times in an hour?
If you do a search for a solution, you'll find various suggestions (including having
fail2ban monitor it's own log files!) but I didn't like any of them. They all seemed far more complicated than what struck me as the easiest solution, but one I didn't see mentioned anywhere. Behold, my Impossibly Stupid approach:
Mathematics to the rescue!
You can roughly approximate the result you want by doing some basic multiplication/addition of the desired values for the “lesser” jail and use that in the settings for the “greater” jail. That is to say, if you're calling 5 attempts a minor offense and 3 minor offenses a major offense, the simple thing to do is say 15 attempts is a major offense. The only real tricky part is making sure that the duration of the jail time allows for a major offense to be committed. So, in the above example, the sequence for the shortest 3-strikes jail is:
abuse followed by 10m ban + abuse followed by 10m ban + abuse followed by 10m ban = 30m
So the longer 1 hour search window leaves plenty of wiggle room.
Practical fail2ban escalation example
What I want is simple. There are certain key words that come up over and over again when my site is scanned for vulnerabilities. For example, anyone looking for files related to WordPress or PHP on my server is up to no good. So I created a filter that matches the major attack vectors I've seen before (let me know if it interests anyone and I can post it) and I want to apply it with increasing harshness.
Here's a set of jails taken from my own config file:
[apache-short-da] enabled = true port = http,https filter = apache-dictionary-attack logpath = /var/log/apache*/*error.log findtime = 60 maxretry = 3 bantime = 300 [apache-medium-da] enabled = true port = http,https filter = apache-dictionary-attack logpath = /var/log/apache*/*error.log findtime = 600 maxretry = 5 bantime = 3600 [apache-long-da] enabled = true port = http,https filter = apache-dictionary-attack logpath = /var/log/apache*/*error.log findtime = 7200 maxretry = 6 bantime = 50400
This gives me a short-term jail, so that anyone who tries a quick scan that results in 3 errors in less than a minute gets banned for 5m. Then I have my medium-term jail, which imposes a 1 hour ban for 5 errors in 10m. Put another way (aka, doing the math), what that means is that someone who was sent to the short-term jail and released after 5m is now on a kind of parole, so if they make 2 more errors in the next 5m, they're going right back inside with the longer 1h sentence.
And finally we have the long-term jail; I generously didn't make it a life sentence. If the above offender was released from the medium-term jail, any error in the next (roughly) 55m results in a ban of a healthy 14h. Adjust all times to be as cruel or as kind as you'd like.
The one cavet to note is that not all log file monitoring by
fail2ban is absolutely instantaneous. There may be sufficient delays such that some very fast spammer will hit your site so many times so quickly that they trigger the longer jail right off the bat. Some might consider that a bug, some might consider that a feature.