# Current kernel
uname -r
6.8.0-101-generic
# What kernels do we have installed. 
# ii at the beginning of the line means installed, but rc entries might have some residual config files.
dpkg --list | grep 'linux-image-'

Now, if you're certain that you want to get rid of all the kernels beginning with 5.x, then go ahead and run the following. Or otherwise edit the commands so that you're disposing of the ones you want to get.

sudo apt remove $(dpkg --list | grep 'linux-image-5\.' | awk '{print $2}' | xargs)  
# And now we get rid of all the associated modules and headers
sudo apt remove $(dpkg --list | grep 'linux-headers-5\.' | awk '{print $2}' | xargs)
sudo apt remove $(dpkg --list | grep 'linux-modules-5\.' | awk '{print $2}' | xargs)   
sudo apt remove $(dpkg --list | grep 'linux-modules-extra-5\.' | awk '{print $2}' | xargs)
# And clean up
sudo apt autoremove --purge
sudo update-grub   

So that cleared over a Gigabyte. Ubuntu will generally (I think) keep the last 3 kernels, but sometimes it doesn't seem to clean up everything it can.

Well it turns out curl was available and curl can help! This command was able to send a test email without any of the tools I'd normally use on a full OS. Sweet.

curl --url 'smtp://103.125.202.12:25' \
 --mail-from no_reply@pretend.org \
 --mail-rcpt mailtest@datalude.com \
 --upload-file - <<EOF
From: Mr Roboto <no_reply@pretend.org>
To: Mailtest <mailtest@datalude.com>
Subject: Robot Mail Test
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

So is it working? Maybe.
EOF

Other useful looking options, not needed this time were:
–ssl-reqd \
–user 'username:password' \

I did find another even more rudimentary connectivity check, which I'll put here for kicks, which doesn't even need curl.

(echo >/dev/tcp/103.125.202.12/25) &>/dev/null && echo "open" || echo "closed"   

Its all about using what you've got. But then again if it wasn't for docker in the first place it would all have been a lot simpler …

You use it like this.

# Put the script in the parent directory of the one you want to clone and run it.
clone_directory_structure.sh targetdirectory > regenerate_directory_structure.sh

# Then move the script to where you want to clone the directory and run it. 
regenerate_directory_structure.sh 

Script listing

#!/bin/bash

# Check if directory is provided
if [ $# -ne 1 ]; then
    echo "Usage: $0 <directory>"
    echo "Example: $0 /path/to/directory"
    exit 1
fi

# Remove trailing slash if present
TARGET_DIR="${1%/}"
TARGET_BASENAME="$(basename "$TARGET_DIR")"
TARGET_PARENT="$(dirname "$TARGET_DIR")"

# Check if directory exists
if [ ! -d "$TARGET_DIR" ]; then
    echo "Error: Directory '$TARGET_DIR' does not exist"
    exit 1
fi

echo "#!/bin/bash"
echo "# Script generated on $(date)"
echo "# To recreate the directory structure, run this script as root or with sudo"
echo "set -e  # Exit on error"
echo

# Process the target directory itself
echo "# Creating target directory: $TARGET_BASENAME"
echo "mkdir -p \"$TARGET_BASENAME\""

# Get permissions, owner, and group of the target directory
TARGET_PERMS=$(stat -c "%a" "$TARGET_DIR")
TARGET_OWNER=$(stat -c "%U" "$TARGET_DIR")
TARGET_GROUP=$(stat -c "%G" "$TARGET_DIR")

echo "chmod $TARGET_PERMS \"$TARGET_BASENAME\""
echo "chown $TARGET_OWNER:$TARGET_GROUP \"$TARGET_BASENAME\""
echo

# Find all subdirectories and process them
find "$TARGET_DIR" -type d -not -path "$TARGET_DIR" -print0 | while IFS= read -r -d $'\0' dir; do
    # Quote the directory path to handle spaces
    dir="$dir"
    # Get relative path from target directory
    rel_path="${dir#$TARGET_DIR/}"
    
    # Quote the path to handle spaces
    rel_path_quoted="$(printf '%q' "$rel_path")"
    
    # Get permissions in octal
    perms=$(stat -c "%a" "$dir")
    
    # Get owner and group
    owner=$(stat -c "%U" "$dir")
    group=$(stat -c "%G" "$dir")
    
    # Output commands with proper quoting for paths with spaces
    echo "# Creating directory: $TARGET_BASENAME/$rel_path"
    echo "mkdir -p $(printf '%q' "$TARGET_BASENAME/$rel_path")"
    echo "chmod $perms $(printf '%q' "$TARGET_BASENAME/$rel_path")"
    echo "chown $owner:$group $(printf '%q' "$TARGET_BASENAME/$rel_path")"
    echo
done

echo "echo \"Directory structure recreation complete!\""
echo "echo \"Created structure under: $TARGET_BASENAME/\""

As always, don't trust everything you read on the internet, and test before you use in anger.

#!/bin/bash

# Configuration
HTACCESS_FILE_DEFAULT="/etc/nginx/htpasswd-developers

# Function to display a list of users
display_users() {
  echo "--- Existing Users ---"
  # Grep for lines that don't start with # and aren't empty
  # Then use cut to show only the username before the colon, and number the lines
  grep -vE '^(#|$)' "$HTACCESS_FILE" | cut -d':' -f1 | cat -n
  echo "----------------------"
}

# Function to perform a backup
backup_file() {
  if [ -f "$HTACCESS_FILE" ]; then
    cp "$HTACCESS_FILE" "$HTACCESS_FILE.bak"
    echo "Backup created at $HTACCESS_FILE.bak"
  else
    echo "No .htpasswd file to back up."
  fi
}

# Function to generate a random password
generate_password() {
    tr -cd '[:alnum:]' < /dev/urandom | head -c 12
}

# --- Main Script ---

# Select the .htpasswd file. Suggest the default value, but allow user to type another name
read -p "Enter .htpasswd file (default: $HTACCESS_FILE_DEFAULT): " HTACCESS_FILE
HTACCESS_FILE=${HTACCESS_FILE:-$HTACCESS_FILE_DEFAULT}

# Ensure the file exists, exit if not
if [ ! -f "$HTACCESS_FILE" ]; then
  echo "Password file doesn't exist"
  exit 1
fi

# Main menu loop
while true; do
  echo "Do you want to:"
  echo "a) Add a user"
  echo "r) Remove a user"
  echo "u) Update a user's password"
  echo "l) List users"
  echo "q) Quit"
  read -p "Enter your choice: " choice

  case "$choice" in
    a)
      read -p "Enter username to add: " username
      # Check if the username already exists
      if grep -q "^$username:" "$HTACCESS_FILE"; then
        echo "Error: User '$username' already exists. Use the 'u' option to update their password."
      else
        suggested_password=$(generate_password)
        read -p "Enter password for '$username' (or press Enter to use suggested: $suggested_password): " password
        password=${password:-$suggested_password}
        
        backup_file
        htpasswd -b "$HTACCESS_FILE" "$username" "$password"
        echo "User '$username' added."
      fi
      ;;
    r)
      display_users
      read -p "Enter reference number of user to remove: " ref
      # Use grep and sed to find the line number and get the username
      username_to_remove=$(grep -vE '^(#|$)' "$HTACCESS_FILE" | sed -n "${ref}p" | cut -d':' -f1)

      if [ -z "$username_to_remove" ]; then
        echo "Invalid reference number."
      else
        backup_file
        # Create a temp file without the user and then replace the original
        grep -v "^$username_to_remove:" "$HTACCESS_FILE" > "$HTACCESS_FILE.tmp" && mv "$HTACCESS_FILE.tmp" "$HTACCESS_FILE"
        echo "User '$username_to_remove' removed."
      fi
      ;;
    u)
      display_users
      read -p "Enter reference number of user to update: " ref
      username_to_update=$(grep -vE '^(#|$)' "$HTACCESS_FILE" | sed -n "${ref}p" | cut -d':' -f1)

      if [ -z "$username_to_update" ]; then
        echo "Invalid reference number."
      else
        suggested_password=$(generate_password)
        read -p "Enter new password for '$username_to_update' (or press Enter to use suggested: $suggested_password): " password
        password=${password:-$suggested_password}

        backup_file
        htpasswd -b "$HTACCESS_FILE" "$username_to_update" "$password"
        echo "Password for user '$username_to_update' updated."
      fi
      ;;
    l)
      display_users
      ;;
    q)
      echo "Exiting."
      exit 0
      ;;
    *)
      echo "Invalid option. Please try again."
      ;;
  esac

  echo ""
done

nginx: [warn] 4096 worker_connections exceed open file resource limit: 1024

There are a lot of places you can change this value so it gets a bit confusing. It could be the user that nginx is running under, (usually www-data) or the process, or it could be set in systemd init file or in security/limits.conf … OK, so we'll run a script to gather the info

#!/bin/bash

# --- Configuration ---
NGINX_CONF="/etc/nginx/nginx.conf"
NGINX_SERVICE="nginx.service"
LIMITS_CONF="/etc/security/limits.conf"
SYSCTL_CONF="/etc/sysctl.conf"
# --- End Configuration ---

echo "#################################################################"
echo "# NGINX ULIMIT (MAX OPEN FILES) CHECKER"
echo "#################################################################"

# 1. Find the NGINX User
NGINX_USER=$(grep -E '^\s*user\s+' $NGINX_CONF | awk '{print $2}' | sed 's/;//' | head -n 1)
if [[ -z "$NGINX_USER" ]]; then
    NGINX_USER="www-data" # Common default user if not explicitly set
fi
echo -e "\n[1] NGINX RUNNING USER: ${NGINX_USER}"

# 2. Get NGINX Process Limits
echo -e "\n[2] CURRENT NGINX WORKER PROCESS LIMITS (/proc/<pid>/limits)"
NGINX_PID=$(pgrep -u "$NGINX_USER" nginx | head -n 1)
if [[ -n "$NGINX_PID" ]]; then
    cat "/proc/$NGINX_PID/limits" | grep "Max open files"
else
    echo "NGINX worker process is not currently running under user '$NGINX_USER'."
fi

# 3. Check NGINX Configuration Directives
echo -e "\n[3] NGINX CONFIGURATION CHECK (${NGINX_CONF})"
NGINX_RBLIMIT=$(grep -E '^\s*worker_rlimit_nofile\s+' $NGINX_CONF | awk '{print $2}' | sed 's/;//' | head -n 1)
NGINX_CONNECTIONS=$(grep -E '^\s*worker_connections\s+' $NGINX_CONF | awk '{print $2}' | sed 's/;//' | head -n 1)

if [[ -n "$NGINX_RBLIMIT" ]]; then
    echo -e "   - worker_rlimit_nofile: \t**$NGINX_RBLIMIT**"
else
    echo -e "   - worker_rlimit_nofile: \tNot set (NGINX inherits OS limit)"
fi

if [[ -n "$NGINX_CONNECTIONS" ]]; then
    echo -e "   - worker_connections: \t**$NGINX_CONNECTIONS**"
else
    echo -e "   - worker_connections: \tNot set (Defaults to 512 or 1024)"
fi

# 4. Check systemd Service Unit Limit (Most modern Linux systems)
echo -e "\n[4] SYSTEMD SERVICE UNIT LIMIT (${NGINX_SERVICE})"
SYSTEMD_LIMIT=$(systemctl show $NGINX_SERVICE --property LimitNOFILE | awk -F'=' '{print $2}')
if [[ -n "$SYSTEMD_LIMIT" ]]; then
    echo -e "   - LimitNOFILE (systemd): \t**$SYSTEMD_LIMIT**"
else
    echo "   - LimitNOFILE (systemd): \tNot explicitly set (Inherits from system defaults/limits.conf)"
fi

# 5. Check User Limits (limits.conf)
echo -e "\n[5] USER LIMITS CONFIGURATION (${LIMITS_CONF})"
USER_LIMITS=$(grep -E "^$NGINX_USER\s+(soft|hard)\s+nofile" $LIMITS_CONF)
ALL_USER_LIMITS=$(grep -E "^\*\s+(soft|hard)\s+nofile" $LIMITS_CONF)

if [[ -n "$USER_LIMITS" ]]; then
    echo -e "   - Limits for $NGINX_USER:\n$USER_LIMITS"
elif [[ -n "$ALL_USER_LIMITS" ]]; then
    echo -e "   - General limits (*):\n$ALL_USER_LIMITS"
else
    echo "   - No explicit nofile limits found for '$NGINX_USER' or '*'."
fi

# 6. Check System-Wide Kernel Limit
echo -e "\n[6] SYSTEM-WIDE KERNEL LIMIT (fs.file-max)"
KERNEL_MAX=$(cat /proc/sys/fs/file-max)
echo -e "   - fs.file-max (Kernel): \t**$KERNEL_MAX**"

echo "#################################################################"

OK, so that gives us the output

#################################################################
# NGINX ULIMIT (MAX OPEN FILES) CHECKER
#################################################################

[1] NGINX RUNNING USER: www-data

[2] CURRENT NGINX WORKER PROCESS LIMITS (/proc/<pid>/limits)
Max open files            1024                 524288               files     

[3] NGINX CONFIGURATION CHECK (/etc/nginx/nginx.conf)
   - worker_rlimit_nofile: 	Not set (NGINX inherits OS limit)
   - worker_connections: 	**4096**

[4] SYSTEMD SERVICE UNIT LIMIT (nginx.service)
   - LimitNOFILE (systemd): 	**524288**

[5] USER LIMITS CONFIGURATION (/etc/security/limits.conf)
   - Limits for www-data:
www-data         hard    nofile          65536
www-data         soft    nofile          65536

[6] SYSTEM-WIDE KERNEL LIMIT (fs.file-max)
   - fs.file-max (Kernel): 	**9223372036854775807**
#################################################################

So we need to alter the low value for the nginx worker process, currently at 1024

Insert this line near the top of /etc/nginx/nginx.conf file, before the events {} section.

worker_rlimit_nofile 65536; 

And restart nginx. Now we see

[2] CURRENT NGINX WORKER PROCESS LIMITS (/proc/<pid>/limits)
Max open files            65536                65536                files     

[3] NGINX CONFIGURATION CHECK (/etc/nginx/nginx.conf)
   - worker_rlimit_nofile: 	**65536**
   - worker_connections: 	**4096**

But as a dyed-in-the-wool bash user, I've kinda got used to being able to scroll up and down my command history and copy items from there. But I always have to search the internet the exact commands I need in tmux. Here's an easy way to dump the whole tmux history buffer to a text file without complicated edit/scroll/copy start/copy end combinations. We'll assume your tmux prefix key is the default CTRL-B

CTRL-B  :                       # default prefix key and colon. opens the command pane, at the bottom
capture-pane -S -       # send whole history to tmux buffer

CTRL-B   :                                              # get ready for another command, tmux
save-buffer ~/tmux_output.txt         # self explanatory 

Now you can use your text editor to look at the command history, alerts, and outputs. I had to use this recently when I piped a long running script through tee -a but it failed to send any of the screen output to the specified text file. This was a lifesaver.

VARIABLE="variable"

# command format
cat << CONTENTS > /path/to/output.conf
Normal text here 
The second line contains a $VARIABLE
        These are Tabbed
        Indented
CONTENTS

# Result, cat /path/to/output.conf
Normal text here
The second line contains a variable
        These are Tabbed
        Indented

Output alternatives: Can also use >> to append to output file, or | tee to output to screen as it runs.

Quoting preserves content

If you're outputting a script you might need to preserve the variable notation instead of evaluating it. You can use either single or double quotes to do this.

VARIABLE="variable"

# command format
cat << 'CONTENTS' > /path/to/output.conf
Normal text here 
The second line contains a $VARIABLE
        These are Tabbed
        Indented
CONTENTS

# Result, cat /path/to/output.conf
Normal text here 
The second line contains a $VARIABLE
        These are Tabbed
        Indented

Zapping Tabs

The <<- construct will remove tabs from your input, which means you can format your code nicely. Doesn't work at the same time as "CONTENTS"

VARIABLE="variable"

# command format
cat <<- CONTENTS > /path/to/output.conf
Normal text here 
The second line contains a $VARIABLE
	These are Tabbed
		Indented
CONTENTS

# Result, cat /path/to/output.conf
Normal text here 
The second line contains a variable
These are Tabbed
Indented

Its installed along with rsync, and resides in /usr/bin/rrsync (on Ubuntu at least). Its basically a wrapper that limits access for a remote rsync server to named directories, and can additionally specify that they're read only.

/usr/bin/rrsync -ro /backups/

You've probably already added the ssh key to the user's authorized_keys file on your production server, so …

# Change this
ssh-rsa AAAAAAgasaofasdfndsfasdfablahblah
# To this
command="/usr/bin/rrsync -ro /backups",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAAAgasaofasdfndsfasdfablahblah

On your backup server you might already be running rsync to pull a directory over from your production server. So,

# Change this
rsync -avz -e ssh user@production.com:/backups/ /local/backups/production/
# To this
rsync -avz -e ssh user@production.com: /local/backups/production/

It looks wrong, like it will backup the whole server, but basically on the production server end it will only let the backup server 'see' the single directory you specified in authorized_keys. Any other command you try to run from the backup server on the production server will fail.

Limitation: As far as I can see, you can only specify a single directory. To do two directories, you'd need to connect with two separate ssh keys and do one directory each time.

LOGFILE="logs/$(date +%F)_backup.log"
# Append to the file
restic backup /etc/ >> $LOGFILE
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print >> $LOGFILE

That's great for small scripts. But sometimes you find that its not logging errors, which are also helpful, and you need to add those:

LOGFILE="logs/$(date +%F)_backup.log"
# Append to the file
restic backup /etc/ 2>&1 >> $LOGFILE
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print 2>&1 >> $LOGFILE

And different commands behave differently, so it all starts to become complicated. But there's an easier way. Two actually. The first is to use curly brackets to contain all the commands you want to log, and then pipe it through tee at the end. Commands after that block won't be logged.

LOGFILE="logs/$(date +%F)_backup.log"
{
echo "Starting Restic backup at $(date)"
restic backup /etc/
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print
echo "Restic backup and maintenance completed"
} 2>&1 | tee -a "${LOGFILE}"

And then there's the slightly less readable command substitution method.

exec > >(tee -a "${LOGFILE}") 2>&1
echo "Starting Restic backup at $(date)"
$RESTIC backup /etc/
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print
echo "Restic backup and maintenance completed"

Comparison

Both methods will log all output of all commands, including errors. I find the curly brackets way slightly easier to read. It also allows you to select which commands are logged, but putting them inside the brackets or not. On the other hand, the exec method will log everything until the end of the script, which may or may not be desirable. Curly braces also seems to be more widely compatible.

There may be other considerations, different performance considerations and buffering for heavy duty scripts, but that's beyond my regular use cases.

Will just add a poor-man's log rotation trick for compleness, and so I can copy and paste it.

# Start logging to rolling logfile
LOGFILE=/path/to.log
KEEPLINES=5000
{
            ### commands
} 2>&1 | tee -a "${LOGFILE}"

# Keep last XXXX lines of rolling logfile, defined above in KEEPLINES
tail -n $KEEPLINES "$LOGFILE" > "$LOGFILE".tmp
mv -f "$LOGFILE".tmp "$LOGFILE"

I've been running Linux servers for 30 years or so, and have a muscle memory associated with changing sshd ports. Edit sshd_config to change Port=123 , update firewall to allow port 123, restart ssh, test config from another box to port 123, done.

Imagine my horror when that didn't work any more. On occasion, having done this hundreds or thousands of times, I might skip the 'test from another box' stage, but I did so this time and was unable to connect.

After a lot of searching, I found the culprit was systemd, or rather Ubuntu's decision to move control of the sshd process to there, so that edits to /etc/ssh/sshd_config were ignored. So when you search for things these days on the internet, you get some AI busting in telling you how to fix it. In this case it was Google Gemini, who confidently told me:

To change the port or address while keeping socket activation, you must modify the sshd.socket unit's configuration. The best practice is to create a "drop-in" file to override the default settings without directly editing the main unit file.

• Create a directory: sudo mkdir -p /etc/systemd/system/ssh.socket.d
• Create a new drop-in file: sudo nano /etc/systemd/system/ssh.socket.d/listen.conf
• Add the following content, replacing 1234 with your desired port. The empty ListenStream= line is crucial as it clears the default port 22.[Socket] ListenStream= ListenStream=1234
• Reload the systemd daemon: sudo systemctl daemon-reload
• Restart the socket: sudo systemctl restart ssh.socket
• Update your firewall rules to allow the new port.

Well the only trouble is, that didn't work. It left my system's sshd listening on ipv6 ONLY. And although the server had ipv6, my connection did not.

Luckily I was able to access the server via the host's console and fix it, but this is a massive gotcha for those VM hosts that don't facilitate that.

So how to fix it, really Google? Well, if you want to stick with ssh.socket:

sudo systemctl edit ssh.socket
# Add
[Socket]
ListenStream=
ListenStream=0.0.0.0:123
ListenStream=[::]:123

# Restart 
sudo systemctl daemon-reload
sudo systemctl restart ssh.socket

# Verify that sshd is now listening on your new port for both IPv4 and IPv6:
sudo ss -tulpn | grep 123

OR, if you want to go back to the way things were, so you don't get confused ….

# Disable and stop the socket unit: 
sudo systemctl disable --now ssh.socket

#Enable and start the service unit: 
sudo systemctl enable --now ssh.service

Now you can edit /etc/ssh/sshd_config and simply run sudo systemctl restart ssh.service for changes to take effect.

The crazy thing is, if you tell Google that their Recommended method doesn't work, it cheerfully acknowledges it and tells you the correct way to do it.

Be careful out there with AI.