But as a dyed-in-the-wool bash user, I've kinda got used to being able to scroll up and down my command history and copy items from there. But I always have to search the internet the exact commands I need in tmux. Here's an easy way to dump the whole tmux history buffer to a text file without complicated edit/scroll/copy start/copy end combinations. We'll assume your tmux prefix key is the default CTRL-B

# default prefix key and colon. opens the command pane, at the bottom
CTRL-B  : 
# send whole history to tmux buffer
capture-pane -S -       
# get ready for another command, tmux
CTRL-B   : 
# self explanatorysave-buffer ~/tmux_output.txt        

Now you can use your text editor to look at the command history, alerts, and outputs. I had to use this recently when I piped a long running script through tee -a but it failed to send any of the screen output to the specified text file. This was a lifesaver.

LOGFILE="logs/$(date +%F)_backup.log"
# Append to the file
restic backup /etc/ >> $LOGFILE
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print >> $LOGFILE

That's great for small scripts. But sometimes you find that its not logging errors, which are also helpful, and you need to add those:

LOGFILE="logs/$(date +%F)_backup.log"
# Append to the file
restic backup /etc/ 2>&1 >> $LOGFILE
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print 2>&1 >> $LOGFILE

And different commands behave differently, so it all starts to become complicated. But there's an easier way. Two actually. The first is to use curly brackets to contain all the commands you want to log, and then pipe it through tee at the end. Commands after that block won't be logged.

LOGFILE="logs/$(date +%F)_backup.log"
{
echo "Starting Restic backup at $(date)"
restic backup /etc/
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print
echo "Restic backup and maintenance completed"
} 2>&1 | tee -a "${LOGFILE}"

And then there's the slightly less readable command substitution method.

exec > >(tee -a "${LOGFILE}") 2>&1
echo "Starting Restic backup at $(date)"
$RESTIC backup /etc/
find /root/scripts/ -type f -name "restic*.log" -mtime +14 -delete -print
echo "Restic backup and maintenance completed"

Comparison

Both methods will log all output of all commands, including errors. I find the curly brackets way slightly easier to read. It also allows you to select which commands are logged, but putting them inside the brackets or not. On the other hand, the exec method will log everything until the end of the script, which may or may not be desirable. Curly braces also seems to be more widely compatible.

There may be other considerations, different performance considerations and buffering for heavy duty scripts, but that's beyond my regular use cases.

Will just add a poor-man's log rotation trick for compleness, and so I can copy and paste it.

# Start logging to rolling logfile
LOGFILE=/path/to.log
KEEPLINES=5000
{
            ### commands
} 2>&1 | tee -a "${LOGFILE}"

# Keep last XXXX lines of rolling logfile, defined above in KEEPLINES
tail -n $KEEPLINES "$LOGFILE" > "$LOGFILE".tmp
mv -f "$LOGFILE".tmp "$LOGFILE"

I had a request to run qdrant database, and I didn't want to do it under docker, due to all the extra nonsense that entails. So having installed it, basically just dropping a binary on the filesystem, I wanted to run it as a systemd service, under a specific user, qdrant.

On the qdrant site, they don't give an example, so I thought it would be quickest to ask Google Gemini, which quickly squirted out a reply. So I activated it and was confused when it wouldn't start. About an hour later, I figured out that the inline comments which Gemini had put in the systemd service files are not allowed. And when telling Gemini about this it cheerfully admitted that it knew about this all along:

Even though I 'shared' the finding, I'm skeptical that Gemini will learn from it, so here is the working qdrant systemd file for anyone interested.

[Unit]
Description=Qdrant Vector Database
After=network.target

[Service]
# sudo adduser --system --no-create-home --group qdrant
User=qdrant
Group=qdrant

ExecStart=/usr/bin/qdrant --config-path /etc/qdrant/config.yaml

# Put all data in here, please
WorkingDirectory=/qdrant

# Increase open file limit for better performance
LimitNOFILE=65536

Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

[Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'

But not just one of them. Lots and lots of them, one a second, for the last few months. To be sure it didn't seem critical, but having lots of crap in your logs means its obscuring the really important bits, and making it hard to detect real problems.

So how to get rid of them? Well not too hard as it happens. First option is to suppress them in your mysql config. I didn't know this was an option, but yes, you can add the following line to your mysqld config file and they don't get logged any more.

log_error_suppression_list='MY-013360'

And restart. This can also be set without restarting the server using SET GLOBAL:

mysql> set global log_error_suppression_list='MY-013360' ;
Query OK, 0 rows affected (0.00 sec)

Very useful if you just want to get rid of it while you're troubleshooting something. But now to get down to the root cause. How do we fix the error completely. Again not too hard. First of all we can take a look and see which user is the offender, and the default server policy.

mysql -e " show variables like 'default_authentication_plugin';"
  default_authentication_plugin = caching_sha2_password

mysql -e "select Host,User,plugin FROM  mysql.user;"
| Host      | User               | plugin                |
| localhost | guilty-user          | mysql_native_password |
| localhost | mysql.session    | caching_sha2_password |
| localhost | mysql.sys         | caching_sha2_password |
| localhost | root               | auth_socket             |

Well it was guilty-user, what a surprise. So lets fix that. The current password for guilty-user will be known to you — it'll be in the WP config file, or database connection string of whatever app is using it. Lets say its "F$ntasticPW", so we just ALTER the user using the same password:

mysql -e "ALTER USER 'guilty-user'@'localhost' IDENTIFIED WITH caching_sha2_password BY 'F$ntasticPW';"

And there we have it. The app shouldn't even blink. And your monster db logs are gone. But as always, take a db backup before you try.

# Arrgh!
mysql -e "show variables" | grep innodb

# Much better
mysql -e "show variables" | awk '{print $1 " = " $2}' | grep innodb

That's all.

pm.status_path = /status

in your fpm pool file, then add a block to your nginx config so that it can only be accessed by certain IP addresses.

        location ~ ^/status {
         access_log off;
         allow 127.0.0.1;    # localhost
         allow 12.23.34.45;   # my remote IP
         deny all;  
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
         fastcgi_index index.php;
         include fastcgi_params;
         fastcgi_pass unix:/run/php/php-fpm-mysite.sock;
        }

You access the URL mysite.com/status and all is good.

Things get more complicated when you have a server where each domain has its own fpm pool file. The /status line in this case is just referring to that domain. If you access the mydomain.com/status URL from an external browser, the domain name resolves and nginx picks the correct php-fpm pool file, and you see the stats.

Now consider using curl from localhost.

 # On a server where there's only one pool file, this works. But when there are multiple pool files it doesn't know which one to use
curl http://127.0.0.1/status
# Can we resolve it with a Host header? No
 curl -H "Host: mysite.com" http://127.0.0.1/status
# Resolve looks hopeful, but actually it doesn't work with SSL
 curl --resolve 'mysite.com:443:127.0.0.1' https://127.0.0.1/status
# Maybe --insecure will help? No
 curl --resolve 'mysite.com:443:127.0.0.1' https://127.0.0.1/status --insecure
# Wait, maybe we can connect directly to the socket, and target the website that way! Nope.
curl --unix-socket /var/run/php/php-fpm-mysite.sock http://status/

After exhausting the Internet's wisdom on this, there didn't seem to be a way to get curl to fetch the information. It was needed locally so that it could be incorporated into a monitoring script, so allowing a remote IP to gather the info was not a desirable solution, although rapidly looking like the only option.

Salvation came in the form of a program called cgi-fcgi which talks pure cgi-ese to the fpm socket. Or something like that. (apt install libfcgi-bin or yum install fcgi). Once installed, we can ask it for the information like this:

SCRIPT_NAME=/status SCRIPT_FILENAME=/status REQUEST_METHOD=GET cgi-fcgi -bind -connect /var/run/php/php-fpm-mysite.sock

So, incorporating that into a script we can monitor our website's fpm usage. I guess this could also be used in telegraf, zabbix and other monitoring solutions. And you can have a different status URL for each site/pool on the server.

CREATE USER 'telegraf-client'@'localhost' IDENTIFIED BY 'xxxxxxx';
GRANT REPLICATION CLIENT ON *.* TO 'telegraf-client'@'localhost' WITH MAX_USER_CONNECTIONS 5;
GRANT PROCESS ON *.* TO 'telegraf-client'@'localhost';
GRANT SELECT ON performance_schema.* TO 'telegraf-client'@'localhost';
FLUSH PRIVILEGES;

And the in /etc/telegraf/telegraf.conf I added:

[[inputs.mysql]]
  servers = ["telegraf-client:xxxxxxxxxxx@tcp(127.0.0.1:3306)/"]

All good. Restart telegraf with systemctl and off we go: data flowed into grafana in a veritable torrent.

On the second server, I did EXACTLY THE SAME THING, but all I could see in the mysql error log was [Warning] Access denied for user ' telegraf-client'@'127.0.0.1' (using password: YES)

I tried everything I could think of over several days. I tried localhost instead of the IP. I tried different quotes, I tried spaces in the brackets, I tried specifying information_schema database. I trawled forums. I tried deleting the user and recreating it as something else (maybe it didn't like the hyphen? getting desperate here). Nope.
I tried using the root user and deb-sys-maint. Nope. From the command line it would connect with 'localhost' but not 127.0.0.1. I checked the hosts file, the mysql config. Many many hours wasted.

Here is what worked. I have no idea why. The servers and setups were identical. This is the answer to one of the Mysteries of Linux:

servers = ["telegraf-client:xxxxxxxxxx@unix(/var/run/mysqld/mysqld.sock)/"]

May it save you many hours.

• Searching the site for malware pointing to a blocked URL (no)
• Checking the firewall logs (nothing)
• Checking the db config, running performance tools on it to see that it wasn't exceeding max connections. (no)
• Running tcpdump to see where wp-cli was trying to connect to (it wasn't)
• Updating all plugins, themes, (good to do anyway)
• Checking db.php, which I found in wp-includes … which, hmmm, belongs to W3 Total Cache.

So on a hunch I tried

> wp-cli plugin deactivate w3-total-cache 
Connection refused
Connection refused

... plus 200 more lines 

Connection refused
Connection refused
Connection refused
Plugin 'w3-total-cache' deactivated.

… after which all wp-cli operations worked again.

After some further digging, I enabled w3 Total Cache again, and found that it was configured to use redis, which wasn't running on the server. So the connection attempts were failing to connect to that. Pretty easy to sort out after that …

The IP address you've 550-5.7.1 registered in your G Suite SMTP Relay service doesn't match domain of 550-5.7.1 the account this email is being sent from. If you are trying to relay 550-5.7.1 mail from a domain that isn't registered under your G Suite account 550-5.7.1 or has empty envelope-from, you must configure your mail server 550-5.7.1 either to use SMTP AUTH to identify the sending domain or to present 550-5.7.1 one of your domain names in the HELO or EHLO command. For more 550-5.7.1 information, please visit 550 5.7.1 https://support.google.com/a/answer/6140680

Pretty helpful as messages go. Less helpful was the fact that there were several websites on the server and a couple of other apps, and I didn't have access to the admin panels of any of them. The mails, once bounced, were removed from postfix's queue, never to be seen again. Time for some detective work.

I found that the mails were heading for zzzz@domain.com so I figured if I could make those appear locally I'd be able to look at the source and see what had created them. This is not so hard to do in postfix. First of all you need a line in your main.cf

# Added to troubleshoot rejection by gmail smtp           
# Need to add addresses to the file and run postmap on it 
recipient_bcc_maps = regexp:/etc/postfix/recipient_bcc_maps

Then you need to add entries in your /etc/postfix/recipient_bcc_maps

# run postmap recipient_bcc_maps after editing
/zzzz\@domain.com/  ubuntu

After that run postmap recipient_bcc_maps and restart postfix for good measure. So now any mail headed to zzzz@domain.com will also be bcced to the local ubuntu user, were we can pick it up in /var/spool/mail/ubuntu

All good. I waited. And waited. Then got bored of waiting and made a quick bash one liner to alert me when the ubuntu user's mail file changed.

while true; do if ls -hal "/var/spool/mail/ubuntu" | grep -q "Nov 15 07:14" ; then sleep 5 ; else mutt -s "changed" me@mydomain.com </dev/null ; break ; fi ; done

So this is pretty crude but it works. Every 5 seconds it checks the date and time of the mail spool file (you'd need to change to the actual date of the file, of course), and if it changes it emails to me@mydomain.com. I ran that under 'screen' and two days later I got a notification, from which I was able to figure out which script on which domain had fired off the email.

Maybe that one liner makes more sense as a script:

#!/bin/bash

FILE=/var/spool/mail/ubuntu
PATTERN="Oct 17 10:44"

while true;
do 
	if ls -hal "$FILE" | grep -q "$PATTERN" ; then 
		# Do nothing. 
		sleep 5
	else
		# echo "changed"
                # play fanfare.mp3
		mutt -s "changed" me@mydomain.com </dev/null
		break;
	fi 
done

CherryTree claims to be able to import from Notecase. However I tried various experiments, exporting to HTML, to Notecase HTML, plain text, MD, etc, but I couldn't get it to import any of the notes nested as originally intended. I could have sat there and dragged them around with the mouse until they were in the right place, and that would have been do-able, but a boring couple of hours. So I looked further.

I found mention in the Github Issues page of CherryTree that the previous version of CherryTree was able to correctly import the notes structure from NoteCase. But that since the developer is in the process of migrating the code from python to c++ (I think), then that functionality had been abandoned until he can re-implement it. I tried to find old versions of the app as flatpak, snap, and AppImage, but there weren't any. So here's the convoluted story of how I got it to work.

• Download Ubuntu 16.04 Desktop and install it as a virtual machine in VirtualBox
• Install the relevant CherryTree 0.34 version from here.
• Export from the current NoteCase file as a notecase ncd. (ncdb and ncz formats don't work). In order to get Notecase to start in 30 day trial mode I needed to first remove the ~/.notecase/ directory contents.
• Open the ncd file in CherryTree 0.34 on Ubuntu 16. Save as CherryTree .ctb format.
• Copy files back to main machine where they could be opened with a recent version of CherryTree.

OK, so now that's looking good, except all the notes are imported as rtf instead of plain text. Probably not a problem, but in my case it was, as I relied on monospace font for formatting in the originals, so I needed text format. The .ctb file format is in sqlite3, so I used dbeaver to examine it, and came up with the following SQL code to change all the rtf notes into txt notes.

# Strip rtf header and footer from note
UPDATE node SET txt = REPLACE(txt,'<?xml version="1.0" ?><node><rich_text>','') WHERE is_richtxt = "1";
UPDATE node SET txt = REPLACE(txt,'</rich_text></node>','') WHERE is_richtxt = "1";
# Set flags
UPDATE node SET syntax = "plain-text" WHERE is_richtxt = "1";
UPDATE node SET is_richtxt = "0" WHERE is_richtxt = "1";

And that was that. Probably more work than I initially intended, but the journey is part of the fun.