Its installed along with rsync, and resides in /usr/bin/rrsync (on Ubuntu at least). Its basically a wrapper that limits access for a remote rsync server to named directories, and can additionally specify that they're read only.

/usr/bin/rrsync -ro /backups/

You've probably already added the ssh key to the user's authorized_keys file on your production server, so …

# Change this
ssh-rsa AAAAAAgasaofasdfndsfasdfablahblah
# To this
command="/usr/bin/rrsync -ro /backups",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAAAgasaofasdfndsfasdfablahblah

On your backup server you might already be running rsync to pull a directory over from your production server. So,

# Change this
rsync -avz -e ssh user@production.com:/backups/ /local/backups/production/
# To this
rsync -avz -e ssh user@production.com: /local/backups/production/

It looks wrong, like it will backup the whole server, but basically on the production server end it will only let the backup server 'see' the single directory you specified in authorized_keys. Any other command you try to run from the backup server on the production server will fail.

Limitation: As far as I can see, you can only specify a single directory. To do two directories, you'd need to connect with two separate ssh keys and do one directory each time.

I've been running Linux servers for 30 years or so, and have a muscle memory associated with changing sshd ports. Edit sshd_config to change Port=123 , update firewall to allow port 123, restart ssh, test config from another box to port 123, done.

Imagine my horror when that didn't work any more. On occasion, having done this hundreds or thousands of times, I might skip the 'test from another box' stage, but I did so this time and was unable to connect.

After a lot of searching, I found the culprit was systemd, or rather Ubuntu's decision to move control of the sshd process to there, so that edits to /etc/ssh/sshd_config were ignored. So when you search for things these days on the internet, you get some AI busting in telling you how to fix it. In this case it was Google Gemini, who confidently told me:

To change the port or address while keeping socket activation, you must modify the sshd.socket unit's configuration. The best practice is to create a "drop-in" file to override the default settings without directly editing the main unit file.

• Create a directory: sudo mkdir -p /etc/systemd/system/ssh.socket.d
• Create a new drop-in file: sudo nano /etc/systemd/system/ssh.socket.d/listen.conf
• Add the following content, replacing 1234 with your desired port. The empty ListenStream= line is crucial as it clears the default port 22.[Socket] ListenStream= ListenStream=1234
• Reload the systemd daemon: sudo systemctl daemon-reload
• Restart the socket: sudo systemctl restart ssh.socket
• Update your firewall rules to allow the new port.

Well the only trouble is, that didn't work. It left my system's sshd listening on ipv6 ONLY. And although the server had ipv6, my connection did not.

Luckily I was able to access the server via the host's console and fix it, but this is a massive gotcha for those VM hosts that don't facilitate that.

So how to fix it, really Google? Well, if you want to stick with ssh.socket:

sudo systemctl edit ssh.socket
# Add
[Socket]
ListenStream=
ListenStream=0.0.0.0:123
ListenStream=[::]:123

# Restart 
sudo systemctl daemon-reload
sudo systemctl restart ssh.socket

# Verify that sshd is now listening on your new port for both IPv4 and IPv6:
sudo ss -tulpn | grep 123

OR, if you want to go back to the way things were, so you don't get confused ….

# Disable and stop the socket unit: 
sudo systemctl disable --now ssh.socket

#Enable and start the service unit: 
sudo systemctl enable --now ssh.service

Now you can edit /etc/ssh/sshd_config and simply run sudo systemctl restart ssh.service for changes to take effect.

The crazy thing is, if you tell Google that their Recommended method doesn't work, it cheerfully acknowledges it and tells you the correct way to do it.

Be careful out there with AI.

#!/bin/bash

# Configuration
HTACCESS_FILE_DEFAULT="htpass_test"

# Function to display a list of users
display_users() {
  echo "--- Existing Users ---"
  # Grep for lines that don't start with # and aren't empty
  # Then use cut to show only the username before the colon, and number the lines
  grep -vE '^(#|$)' "$HTACCESS_FILE" | cut -d':' -f1 | cat -n
  echo "----------------------"
}

# Function to perform a backup
backup_file() {
  if [ -f "$HTACCESS_FILE" ]; then
    cp "$HTACCESS_FILE" "$HTACCESS_FILE.bak"
    echo "Backup created at $HTACCESS_FILE.bak"
  else
    echo "No .htpasswd file to back up."
  fi
}

# Function to generate a random password
generate_password() {
    tr -cd '[:alnum:]' < /dev/urandom | head -c 12
}

# --- Main Script ---

# Select the .htpasswd file
read -p "Enter .htpasswd file (default: $HTACCESS_FILE_DEFAULT): " HTACCESS_FILE
HTACCESS_FILE=${HTACCESS_FILE:-$HTACCESS_FILE_DEFAULT}

# Ensure the file exists, exit if not
if [ ! -f "$HTACCESS_FILE" ]; then
  echo "Password file doesn't exist"
  exit 1
fi

# Main menu loop
while true; do
  echo "Do you want to:"
  echo "a) Add a user"
  echo "r) Remove a user"
  echo "u) Update a user's password"
  echo "l) List users"
  echo "q) Quit"
  read -p "Enter your choice: " choice

  case "$choice" in
    a)
      read -p "Enter username to add: " username
      # Check if the username already exists
      if grep -q "^$username:" "$HTACCESS_FILE"; then
        echo "Error: User '$username' already exists. Use the 'u' option to update their password."
      else
        suggested_password=$(generate_password)
        read -p "Enter password for '$username' (or press Enter to use suggested: $suggested_password): " password
        password=${password:-$suggested_password}
        
        backup_file
        htpasswd -b "$HTACCESS_FILE" "$username" "$password"
        echo "User '$username' added."
      fi
      ;;
    r)
      display_users
      read -p "Enter reference number of user to remove: " ref
      # Use grep and sed to find the line number and get the username
      username_to_remove=$(grep -vE '^(#|$)' "$HTACCESS_FILE" | sed -n "${ref}p" | cut -d':' -f1)

      if [ -z "$username_to_remove" ]; then
        echo "Invalid reference number."
      else
        backup_file
        # Create a temp file without the user and then replace the original
        grep -v "^$username_to_remove:" "$HTACCESS_FILE" > "$HTACCESS_FILE.tmp" && mv "$HTACCESS_FILE.tmp" "$HTACCESS_FILE"
        echo "User '$username_to_remove' removed."
      fi
      ;;
    u)
      display_users
      read -p "Enter reference number of user to update: " ref
      username_to_update=$(grep -vE '^(#|$)' "$HTACCESS_FILE" | sed -n "${ref}p" | cut -d':' -f1)

      if [ -z "$username_to_update" ]; then
        echo "Invalid reference number."
      else
        suggested_password=$(generate_password)
        read -p "Enter new password for '$username_to_update' (or press Enter to use suggested: $suggested_password): " password
        password=${password:-$suggested_password}

        backup_file
        htpasswd -b "$HTACCESS_FILE" "$username_to_update" "$password"
        echo "Password for user '$username_to_update' updated."
      fi
      ;;
    l)
      display_users
      ;;
    q)
      echo "Exiting."
      exit 0
      ;;
    *)
      echo "Invalid option. Please try again."
      ;;
  esac

  echo # Add a newline for spacing
done

;;; php settings
php_flag[display_errors] = off
php_admin_value[error_log] = /home/bobthebuilder/logs/phpfpm.log
php_admin_flag[log_errors] = on

These all work fine. But if you look anywhere on the internet, it tells you you can also set error_logging there. It confidently tells you to use

php_admin_value[error_reporting] = E_ALL & ~E_WARNING & ~E_NOTICE & ~E_DEPRECATED & ~E_USER_DEPRECATED

…for example. But this doesn't work! You check and re-check the log files which are scrolling past at nausea-inducing rates. You try altering the values in php.ini, wordpress config files, anywhere you can try. But when you load up phpinfo, it just tells you you're wasting your time. You shout at google gemini a bit, but it tells you the same thing. Maybe you have some formatting errors, it suggests?

The trick, it seems is to use the numeric values. This works in your phpfpm pool file instead of the command above.

php_admin_value[error_reporting] = 8181

That's it. And there's a pretty handy page for calculating the masks here.

Now wouldn't it be nice if PHP had pre-set log levels so all you have to do is put error_log_level = 3 instead of invoking a bitmask calculator …

• In Updraft, chose S3-Compatible (generic) storage option
• S3 Access key and S3 Secret Key are obvious enough and are as given by Hetzner. Access Key is the smaller of the two.
• In S3 Location, you just need the bucket name, so that it reads s3generic://mybucketname (i.e. just type mybucketname in the box)
• In the S3 endpoint box, you want just the domain name of the storage server, eg fsn1.your-objectstorage.com
  

That's it. Hit Test, and away you go.

Backing up from US to Hetzner Germany took about 20 mins for 1.2Gb backup. Slow, but it completed without error.

Check out your current systemctl log usage with

journalctl --disk-usage

Wait, whaaaat? Its using up 4Gb? By default journalctl will take up 15% of your disk, which seems a little greedy. You can immediately prune some of the logs with the following, which will limit to the size specified 200M, 1G, 500k etc.

journalctl --vacuum-size=500M

That buys you some space immediately on a system which is running out of room. If you want a longer term solution you can edit /etc/systemd/journald.conf and try setting eg.

SystemMaxUse=500M
SystemKeepFree=1G

First thing to note is that not all clients support the new v2 API which is required for wildcard certs. I looked at the list of v2 supporting clients on the Letsencrypt site, and chose the acme.sh bash script. Not sure if I'm going to stick with it at this point but it got me going.

First thing you need to do is to run it with the –issue flag. You'll need to run it with DNS authentication, as that's the supported method for wildcard certs. You'll also need to run it with both the root domain AND the wildcard.

./acme.sh --log --issue --dns -d mydomain.com -d *.mydomain.com
[Tue Mar 13 23:42:54 MDT 2018] Multi domain='DNS:mydomain.com,DNS:*.mydomain.com'
[Tue Mar 13 23:42:54 MDT 2018] Getting domain auth token for each domain
[Tue Mar 13 23:42:55 MDT 2018] Getting webroot for domain='mydomain.com'
[Tue Mar 13 23:42:55 MDT 2018] Getting webroot for domain='*.mydomain.com'
[Tue Mar 13 23:42:55 MDT 2018] Add the following TXT record:
[Tue Mar 13 23:42:55 MDT 2018] Domain: '_acme-challenge.mydomain.com'
[Tue Mar 13 23:42:55 MDT 2018] TXT value: '123XuRD_z9FHfdGIIQR5HIoNY1kCn7WjqqND2s1Nxyz'
[Tue Mar 13 23:42:55 MDT 2018] Please be aware that you prepend _acme-challenge. before your domain
[Tue Mar 13 23:42:55 MDT 2018] so the resulting subdomain will be: _acme-challenge.mydomain.com
[Tue Mar 13 23:42:55 MDT 2018] Add the following TXT record:
[Tue Mar 13 23:42:55 MDT 2018] Domain: '_acme-challenge.mydomain.com'
[Tue Mar 13 23:42:55 MDT 2018] TXT value: '1233qm_dTj_tHeDljeZOgNywPCVKMrTxWcMHTYkrxyz'
[Tue Mar 13 23:42:55 MDT 2018] Please be aware that you prepend _acme-challenge. before your domain
[Tue Mar 13 23:42:55 MDT 2018] so the resulting subdomain will be: _acme-challenge.mydomain.com
[Tue Mar 13 23:42:55 MDT 2018] Please add the TXT records to the domains, and retry again.
[Tue Mar 13 23:42:55 MDT 2018] Please check log file for more details: /root/.acme.sh/acme.sh.log

You'll get a load of output which tells you it failed but gives you the entries you need to put into your DNS Zone editor. So now you need to add TWO text records to your domain:
_acme-challenge.mydomain.com    123XuRD_z9FHfdGIIQR5HIoNY1kCn7WjqqND2s1Nxyz
_acme-challenge.mydomain.com    1233qm_dTj_tHeDljeZOgNywPCVKMrTxWcMHTYkrxyz

You can verify these with dig:

dig txt _acme-challenge.mydomain.com
;; ANSWER SECTION:
_acme-challenge.mydomain.com. 60 IN TXT	"123XuRD_z9FHfdGIIQR5HIoNY1kCn7WjqqND2s1Nxyz"
_acme-challenge.mydomain.com. 60 IN TXT	"1233qm_dTj_tHeDljeZOgNywPCVKMrTxWcMHTYkrxyz"

Now you run the acme client again with the –renew flag.

./acme.sh --renew -d mydomain.com -d *.mydomain.com
[Tue Mar 13 23:51:49 MDT 2018] Renew: 'mydomain.com'
[Tue Mar 13 23:51:50 MDT 2018] Multi domain='DNS:mydomain.com,DNS:*.mydomain.com'
[Tue Mar 13 23:51:50 MDT 2018] Getting domain auth token for each domain
[Tue Mar 13 23:51:50 MDT 2018] Verifying:mydomain.com
[Tue Mar 13 23:51:53 MDT 2018] Success
[Tue Mar 13 23:51:53 MDT 2018] Verifying:*.mydomain.com
[Tue Mar 13 23:51:55 MDT 2018] Success
[Tue Mar 13 23:51:55 MDT 2018] Verify finished, start to sign.
[Tue Mar 13 23:51:56 MDT 2018] Cert success.

[Tue Mar 13 23:51:56 MDT 2018] Your cert is in  /root/.acme.sh/mydomain.com/mydomain.com.cer 
[Tue Mar 13 23:51:56 MDT 2018] Your cert key is in  /root/.acme.sh/mydomain.com/mydomain.com.key 
[Tue Mar 13 23:51:56 MDT 2018] The intermediate CA cert is in  /root/.acme.sh/mydomain.com/ca.cer 
[Tue Mar 13 23:51:56 MDT 2018] And the full chain certs is there:  /root/.acme.sh/mydomain.com/fullchain.cer

And that's about it. The client has more options to copy the cert into your apache/nginx config, but I had to add it by hand to the load balancer. Maybe I'll investigate automating this, and the cert renewal.

1. Download and install CSF
   cd code
   wget https://download.configserver.com/csf.tgz
   tar -xzf csf.tgz
   cd csf
   install.sh
2. Edit Open ports in /etc/csf/csf.conf to reflect  your environment. csf install will automatically detect ssh running on non-standard ports and add those. It will also tell you during install which ports are listening. Review:
   TCP_OUT = "20,21,22,25,53,80,110,113,443,587,993,995"
   TCP_IN = "22,25,80,110,143,443,465"
   Also TCPV6_OUT and TCPV6_IN.
3. Set the following values
   TESTING = "1"
   RESTRICT_SYSLOG = "3"
   RESTRICT_SYSLOG_GROUP = "sysloggers"
   LF_ALERT_TO = "x@domain.com"
   LF_ALERT_FROM = "csf@domain.com"
   LF_DISTATTACK = "1"
   PT_USERTIME = "1"
4. Review log settings from HTACCESS_LOG onwards. Specifically on Ubuntu, you need to set
   SSHD_LOG = "/var/log/auth.log"
   SU_LOG = "/var/log/auth.log"
   FTPD_LOG = "/var/log/syslog"
   SMTPAUTH_LOG = "/var/log/secure"
   POP3D_LOG = "/var/log/mail.log"
   IMAPD_LOG = "/var/log/mail.log"
   IPTABLES_LOG = "/var/log/syslog"
   SUHOSIN_LOG = "/var/log/syslog"
   BIND_LOG = "/var/log/syslog"
   SYSLOG_LOG = "/var/log/syslog"
   WEBMIN_LOG = "/var/log/auth.log"
5. You can now start csf. It will replace all the UFW rules with its own.
   ufw disable
   systemctl disable ufw
   systemctl disable fail2ban
   csf -ra
6. Archive off fail2ban and remove logrotate config
   tar -cvf /etc/fail2ban.tar /etc/fail2ban/
   apt remove fail2ban ufw
   rm /etc/logrotate.d/fail2ban
7. Extra steps for VestaCP
   In /usr/local/vesta/conf/vesta.conf file.
   FIREWALL_SYSTEM="
   FIREWALL_EXTENSION="
   Install the vesta UI and v-csf script from https://github.com/haipham/csf-vestacp/blob/master/install.sh
   (prefer to do this manually)
8. Final hacking. Over the next few days you'll need to pay attention to other files in /etc/csf/
   csf.ignore
   csf.pignore
   csf.blocklists
   csf.allow
   csf.deny
9. Extra aggressive settings for those email bruteforcers.
   LF_POP3D = 5
   LF_POP3D_PERM = 86400
   LF_IMAPD = 5
   LF_POP3D_PERM = 86400
10. Adjust Logwatch as necessary.

find /path/to/dir -regex ".*\.\(jpg\|png\|gif\)" -exec file {} \; | grep -i -v "image data"

If all is good, you won't get any output. If your server is seriously borked, then you might see things like this …

./wp-content/uploads/2011/01/22.jpg: HTML document, ASCII text

This is a flag that the image is in fact a PHP file. Investigate!

If you get this kind of thing,
./wp-content/uploads/2011/01/221.jpg: Minix filesystem, V2, 46909 zones

its probably a bug in an old version of file, so check your OS version, copy the file to a more recent OS and try again.

The script below helps with both these problems. It takes stats of a logfile (apache, but should also work on nginx), makes a backup, counts the number of lines it removes and each kind of bot, and then repeats the new stats at the end. Copy the following into a file eg. log-squish.sh and run with the name of the logfile as an argument. eg cleanlog.sh logfile.log
You'll definitely want to edit the LOCALTRAFFIC bit to fit your needs. You may also want to add bots to the BOTLIST. Run the script once on a sample logfile and then view it to see what bots are left …

#!/bin/bash

# Pass input file as a commandline argument, or set it here
INFILE=$1
OUTFILE=./$1.squish
TMPFILE=./squish.tmp

if [ -f $TMPFILE ] ; then 
    rm $TMPFILE
fi

# Check before we go ... 
read -p "Will copy $INFILE to $OUTFILE and perform all operations on the file copy. Press ENTER to proceed ..."

cp $INFILE $OUTFILE

# List of installation-specific patterns to delete from logfiles (this example for WP. Also excluding local IPaddress)
# Edit to suit your environment.
LOCALTRAFFIC=" wp-cron.php 10.10.0.2 wp-login.php \/wp-admin\/ "
echo
echo "-------- Removing local traffic ---------"
for TERM in $LOCALTRAFFIC; do
    TERMCOUNT=$( grep "$TERM" $OUTFILE | wc -l )
    echo $TERMCOUNT instances of $TERM removed >> $TMPFILE
    sed -i  "/$TERM/d" $OUTFILE
done
sort -nr $TMPFILE
rm $TMPFILE

# List of patterns to delete from logfiles, space separated
BOTLIST="ahrefs Baiduspider bingbot Cliqzbot cs.daum.net DomainCrawler DuckDuckGo Exabot Googlebot linkdexbot magpie-crawler MJ12bot msnbot OpenLinkProfiler.org opensiteexplorer pingdom rogerbot SemrushBot SeznamBot sogou.com\/docs tt-rss Wotbox YandexBot YandexImages ysearch\/slurp BLEXBot Flamingo_SearchEngine okhttp scalaj-http UptimeRobot YisouSpider proximic.com\/info\/spider "
echo
echo "------- Removing Bots ---------"
for TERM in $BOTLIST; do
    TERMCOUNT=$( grep "$TERM" $OUTFILE | wc -l )
    echo $TERMCOUNT instances of $TERM removed >> $TMPFILE
    sed -i  "/$TERM/d" $OUTFILE
done
sort -nr $TMPFILE
rm $TMPFILE

echo
echo "======Summary======="

#filestats before
PRELINES=$(cat $INFILE | wc -l )
PRESIZE=$( stat -c %s $INFILE )

#filestats after
POSTLINES=$(cat $OUTFILE | wc -l )
POSTSIZE=$( stat -c %s $OUTFILE )
PERCENT=$(awk "BEGIN { pc=100*${POSTLINES}/${PRELINES}; i=int(pc); print (pc-i<0.5)?i:i+1 }")

echo Original file $INFILE is $PRESIZE bytes and contains $PRELINES lines
echo Processed file $OUTFILE is $POSTSIZE bytes and contains $POSTLINES lines
echo Log reduced to $PERCENT percent of its original size.
echo Original file was untouched.

And here is a sample output.

~/temp $ ./log-squish.sh access.log.2017-09-03
Will copy access.log.2017-09-03 to ./access.log.2017-09-03.squish and perform all operations on the file copy. Press ENTER to proceed

-------- Removing local traffic ---------
5536 instances of wp-login.php removed
507 instances of \/wp-admin\/ removed
84 instances of wp-cron.php removed
0 instances of 10.10.0.2 removed

------- Removing Bots ---------
2769 instances of bingbot removed
2342 instances of Googlebot removed
2177 instances of sogou.com\/docs removed
1815 instances of MJ12bot removed
1651 instances of ahrefs removed
1016 instances of opensiteexplorer removed
578 instances of Baiduspider removed
447 instances of Flamingo_SearchEngine removed
357 instances of okhttp removed
295 instances of UptimeRobot removed
122 instances of scalaj-http removed
74 instances of YandexBot removed
60 instances of ysearch\/slurp removed
24 instances of YisouSpider removed
22 instances of magpie-crawler removed
9 instances of linkdexbot removed
7 instances of YandexImages removed
7 instances of SeznamBot removed
5 instances of rogerbot removed
2 instances of tt-rss removed
1 instances of SemrushBot removed
0 instances of Wotbox removed
0 instances of proximic.com\/info\/spider removed
0 instances of pingdom removed
0 instances of OpenLinkProfiler.org removed
0 instances of msnbot removed
0 instances of Exabot removed
0 instances of DuckDuckGo removed
0 instances of DomainCrawler removed
0 instances of cs.daum.net removed
0 instances of Cliqzbot removed
0 instances of BLEXBot removed

======Summary=======
Original file access.log.2017-09-03 is 19395785 bytes and contains 74872 lines
Processed file ./access.log.2017-09-03.squish is 15432796 bytes and contains 54965 lines
Log reduced to 73 percent of its original size.
Original file was untouched.

So you can see that around 20% of the traffic on here is crap. And now the log file is much easier to read.