Fail2ban for 404

Today we had an issue were a server was continuously getting requests from IP addresses in a particular country. The requests were obviously automated and searching for PHP pages that didn’t exist within the application we were running.

However this did mean that apache was continually trying to process and log these requests, causing the server to slow.

We knew fail2ban should do what we wanted but weren’t quite sure how to implement it. So here is a quick run through of how to implement a fail2ban policy for continuous requests for pages that don’t exist.

First install fail2ban. Our setup is Debian so we just used apt-get

sudo apt-get update
sudo apt-get install fail2ban

In the current version of fail2ban, filters are added to the filter.d folder. The default install directory is /etc/fail2ban. We used the following commands to create a new filter

cd /etc/fail2ban/filter.d/
sudo nano apache-404.conf

The configuration of the filter requires a regex that will cause a fail and an optional regex that can be used to ignore requests. We just wanted a regex to block requests so we only needed the fail regex.

[Definition]
failregex = ^ - .* "(GET|POST|HEAD).*HTTP.*" 404 .*$
ignoreregex =

This regex does a match on any get, post or head request that returns a 404 e.g.

XX.XXX.XXX.XXX - - [1/Jan/2020:12:30:00 +0000] "GET /url/path/file.php HTTP/1.1" 404 465 "-"

We also eventually refined it to only fail on requests for unknown PHP pages by adding it to the regex

failregex = ^ - .* "(GET|POST|HEAD).*php.*HTTP.*" 404 .*$

Save the file and then we can add it into the fail2ban configuration.

Now create a file to append configuration to the default configuration. The current configuration is stored in the jail.conf file and is automatically overridden or appended by jail.local

cd /etc/fail2ban
sudo nano jail.local

Add the following lines:

[apache-404]                           # label
enabled = true                         # enable the filter
port = http,https                      # ports to enable filter on
filter = apache-404                    # name of the filter to use
logpath = /var/log/apache2/*access.log # the log file to do the regex check on
maxretry = 3                           # number of failed before banning
bantime = 600                          # how long to band for in seconds
ignoreip =                             # exempt IP addresses

We are getting the filter to check the apache access log but you could just as easily get to check the error log or any other log for a match. You can also easily update the number of allowed fails and how long to ban for.

Restart fail2ban and you should be ready to go. You can check the status of the filter using the following command:

sudo fail2ban-client status apache-404

If you called your filter something different then you exchange the name in the above command. This should show you a count of bans and currently banned IP addresses.

MySQL causing hard disk to running out of space

Recently we have had a couple of server run out of hard disk space causing inaccessibility and crashing. On further investigation the hard drive was full so many operations were complaining. The hard drive was being consumed by a very large file within the /var/lib/mysql directory called ibtmp1. So MySQL is causing the server to run out of disk space.

It turns out that this is the temporary working file for MySQL. When you turn the MySQL server off and on again the file is deleted and the server runs again without issue until the file becomes too large again.

After some quick research we found that adding or editing a few settings in the MySQL configuration allows you to limit the size of this file and prevent this issue from reoccurring.

Open the mysqld.cnf file. The location of our mysqld.cnf was in /etc/mysql/mysql.conf.d/mysqld.conf. According to the file you can also copy this configuration to /etc/mysql/my.cnf. Our my.cnf just had an include to the mysqld.cnf file.

The file should have some sections like:

[mysqld_safe]
[mysqld]

There should be a section for InnoDB settings. If not, it is fine to add these configuration settings at the bottom of the file.

Here is the configuration variable:

innodb_temp_data_file_path = ibtmp1:12M:autoextend:max:1G

This value can be tweaked based on your server setup and needs but these are the settings we used for our server to set the file to start at 12 MB and limit it to 1 GB.

While setting this we also set a few other variables for optimisation:

innodb_buffer_pool_size=2048M
innodb_log_file_size=25M
innodb_log_buffer_size=80M

If you have had any other experiences on optimising MySQL, please let us know.

Free SSL using Let’s Encrypt

Update – May 2017

After adding a ruby on rails application to my website I realised let’s encrypt wasn’t renewing. I had been using unicorn for my application and had done a proxy forward for incoming traffic. This had caused the .well-known path to not be recognised. I had to add in a path to the public folder in the application to get this working again.

Replaced

location ~ /.well-known {
allow all;
}

with

location ~ /.well-known {
allow all;
root /path/to/rails/app/public;
}

 

I have just recently set up a couple of sites with free SSL from Let’s Encrypt. Here is a summary of the commands I used for Nginx from this nice tutorial on Digital Ocean. There is also an Apache version.

sudo apt-get update
sudo apt-get -y install git bc
sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
sudo nano /etc/nginx/sites-available/default

Change SSL lines in nginx config
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

And add the following inside the server block
location ~ /.well-known {
allow all;
}

cd /opt/letsencrypt
./letsencrypt-auto certonly -a webroot --webroot-path=/usr/share/nginx/html -d example.com -d www.example.com

Add you email address and agree to T&Cs

Reload Nginx
sudo service nginx reload

Automatic renewal with crontab
sudo crontab -e
30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
35 2 * * 1 /etc/init.d/nginx reload

Mono server for ASP.NET

#install mono server
sudo apt-get install mono-fastcgi-server4

Add following to /etc/nginx/fastcgi_params
fastcgi_param PATH_INFO "";
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

Nginx configuration
server {
listen 80;
server_name [HOST_NAME];

location / {
root [APP_ROOT];
index index.html index.htm default.aspx Default.aspx;
fastcgi_index Default.aspx;
fastcgi_pass 127.0.0.1:9000;
include /etc/nginx/fastcgi_params;
}
}

Restart Nginx
sudo service nginx restart

Start mono server
fastcgi-mono-server4 /applications="[HOST_NAME]:/:[APP_ROOT]" /socket=tcp:127.0.0.1:9000
e.g. fastcgi-mono-server4 /applications="192.168.0.0:/:/var/www/net" /socket=tcp:127.0.0.1:9000

Mono Server initialization script
#!/bin/sh

### BEGIN INIT INFO
# Provides: monoserve.sh
# Required-Start: $local_fs $syslog $remote_fs
# Required-Stop: $local_fs $syslog $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start fastcgi mono server with hosts
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/mono
NAME=monoserver
DESC=monoserver

MONOSERVER=$(which fastcgi-mono-server4)
MONOSERVER_PID=$(ps auxf | grep fastcgi-mono-server4.exe | grep -v grep | awk '{print $2}')

WEBAPPS="/:[APP_ROOT]"

case "$1" in
start)
if [ -z "${MONOSERVER_PID}" ]; then
echo "starting mono server"
${MONOSERVER} /applications=${WEBAPPS} /socket=tcp:127.0.0.1:9000 &
echo "mono server started"
else
echo ${WEBAPPS}
echo "mono server is running"
fi
;;
stop)
if [ -n "${MONOSERVER_PID}" ]; then
kill ${MONOSERVER_PID}
echo "mono server stopped"
else
echo "mono server is not running"
fi
;;
esac

exit 0

Add appropriate rights:
chmod +x /etc/init.d/monoserve

And install the script:
update-rc.d monoserve defaults

Capistrano3-nginx permissions

Capistrano3 has a plugin for Nginx called capistrano3-nginx which allows you to manage it using the cap command or through your deployment scripts e.g. nginx:reload

However, if you try to run any of these command on some systems, it may start to complain about not having permissions or your deploy my hang while asking for a sudo password.

To get round this you need to give the deploying user the permissions to run a sudo command without needing a password. If you’re like me the, the first time you hear that you think “that can’t be very secure”. Correct! Giving any user full sudo access without a password isn’t recommended.

To improve things we can restrict what commands the deploying user can run as sudo without a password. To do this edit the sudoers file using the following command

sudo visudo

Adding the following line will then allow the deploying user to run the require Nginx commands without the need for a password.

deploy ALL=NOPASSWD:/usr/sbin/service nginx *

You may need to adjust the Nginx service path if your set up is different. The ‘*’ just allows all commands on the Nginx service. You could restrict this further by only allowing specific commands like ‘restart’.

Once you have saved the changes you should be able to run Nginx service commands without the need to enter a password and Capistrano should in turn be able to do the same.

EC2 Ubuntu unable to resolve host

During setup of an Ubuntu server on EC2 I was getting a warning like this:

sudo: unable to resolve host ip-xx-xx-xx-xx

To resolve this I used the following command to append a localhost entry for the hostname to the hosts file

echo "127.0.1.1 $HOSTNAME" | sudo tee -a /etc/hosts

locale settings warning on Ubuntu

Every time I set up a new Ubuntu instance on EC2 I get locale warnings during updates. The problem is that locale settings are not configured by default. To set them do the following:

List your locale details
locale

Generate the required locale settings
sudo locale-gen en_US.UTF-8

Reconfigure the locales package
sudo dpkg-reconfigure locales

Some installations may also require you to output the locale settings to the environment variables by appending the following to the /etc/environment file:
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8

Postgresql. Failed to build gem native extension. pg gem.

I have been trying to deploy a rails app with capistrano and bundler and kept getting the “Failed to build gem native extension” error. After installing, probably, many unnecessary packages, I finally came across a post which pointed me in the right direction.

It turns out that the default pg_config location did not match the path that capistrano was using. I found this by doing
which pg_config
on the server and comparing it to the command which was being executed in the trace.

Since they didn’t match I found a command to set the global local variable for pg_config.
bundle config build.pg --with-pg-config=/usr/bin/pg_config

Once run, bundler executed successfully while deploying.

Keep ssh connect alive

When programming my Raspberry Pi I often go and research something while connected to the Pi via ssh. Sometimes if left for too long the ssh will disconnect and i will have to restart to get a new connection

Here is a setting I have used to keep the ssh connection live while not being used.

Add this line to the end of /etc/ssh/ssh_config
ServerAliveInterval 60

Python script to test IO with WiringPi GPIO on Raspberry Pi

The last week or so I have been playing about with the Raspberry Pi and Picoborg. After running the Picoborg install script I created the following script to test that it was working.

import RPi.GPIO as GPIO # import GPIO library
import time # import time library
GPIO.setmode(GPIO.BCM)
GPIO.setup(4, GPIO.OUT) # 4 is the pin reference for first IO
GPIO.output(4, GPIO.LOW) # LOW turns IO off so this makes sure it's off to start
GPIO.output(4, GPIO.HIGH) # turn IO on
time.sleep(1) # wait 1 second
GPIO.output(4, GPIO.LOW) # turn IO off
GPIO.cleanup() # tidy up

This can be run by:
sudo chmod +x filename.py
sudo python file_name.py