;;; php settings
php_flag[display_errors] = off
php_admin_value[error_log] = /home/bobthebuilder/logs/phpfpm.log
php_admin_flag[log_errors] = on

These all work fine. But if you look anywhere on the internet, it tells you you can also set error_logging there. It confidently tells you to use

php_admin_value[error_reporting] = E_ALL & ~E_WARNING & ~E_NOTICE & ~E_DEPRECATED & ~E_USER_DEPRECATED

…for example. But this doesn't work! You check and re-check the log files which are scrolling past at nausea-inducing rates. You try altering the values in php.ini, wordpress config files, anywhere you can try. But when you load up phpinfo, it just tells you you're wasting your time. You shout at google gemini a bit, but it tells you the same thing. Maybe you have some formatting errors, it suggests?

The trick, it seems is to use the numeric values. This works in your phpfpm pool file instead of the command above.

php_admin_value[error_reporting] = 8181

That's it. And there's a pretty handy page for calculating the masks here.

Now wouldn't it be nice if PHP had pre-set log levels so all you have to do is put error_log_level = 3 instead of invoking a bitmask calculator …

• In Updraft, chose S3-Compatible (generic) storage option
• S3 Access key and S3 Secret Key are obvious enough and are as given by Hetzner. Access Key is the smaller of the two.
• In S3 Location, you just need the bucket name, so that it reads s3generic://mybucketname (i.e. just type mybucketname in the box)
• In the S3 endpoint box, you want just the domain name of the storage server, eg fsn1.your-objectstorage.com
  

That's it. Hit Test, and away you go.

Backing up from US to Hetzner Germany took about 20 mins for 1.2Gb backup. Slow, but it completed without error.

pm.status_path = /status

in your fpm pool file, then add a block to your nginx config so that it can only be accessed by certain IP addresses.

        location ~ ^/status {
         access_log off;
         allow 127.0.0.1;    # localhost
         allow 12.23.34.45;   # my remote IP
         deny all;  
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
         fastcgi_index index.php;
         include fastcgi_params;
         fastcgi_pass unix:/run/php/php-fpm-mysite.sock;
        }

You access the URL mysite.com/status and all is good.

Things get more complicated when you have a server where each domain has its own fpm pool file. The /status line in this case is just referring to that domain. If you access the mydomain.com/status URL from an external browser, the domain name resolves and nginx picks the correct php-fpm pool file, and you see the stats.

Now consider using curl from localhost.

 # On a server where there's only one pool file, this works. But when there are multiple pool files it doesn't know which one to use
curl http://127.0.0.1/status
# Can we resolve it with a Host header? No
 curl -H "Host: mysite.com" http://127.0.0.1/status
# Resolve looks hopeful, but actually it doesn't work with SSL
 curl --resolve 'mysite.com:443:127.0.0.1' https://127.0.0.1/status
# Maybe --insecure will help? No
 curl --resolve 'mysite.com:443:127.0.0.1' https://127.0.0.1/status --insecure
# Wait, maybe we can connect directly to the socket, and target the website that way! Nope.
curl --unix-socket /var/run/php/php-fpm-mysite.sock http://status/

After exhausting the Internet's wisdom on this, there didn't seem to be a way to get curl to fetch the information. It was needed locally so that it could be incorporated into a monitoring script, so allowing a remote IP to gather the info was not a desirable solution, although rapidly looking like the only option.

Salvation came in the form of a program called cgi-fcgi which talks pure cgi-ese to the fpm socket. Or something like that. (apt install libfcgi-bin or yum install fcgi). Once installed, we can ask it for the information like this:

SCRIPT_NAME=/status SCRIPT_FILENAME=/status REQUEST_METHOD=GET cgi-fcgi -bind -connect /var/run/php/php-fpm-mysite.sock

So, incorporating that into a script we can monitor our website's fpm usage. I guess this could also be used in telegraf, zabbix and other monitoring solutions. And you can have a different status URL for each site/pool on the server.

Luckily its pretty easy to fix. Check if its enabled with

mysql -e "show variables like 'log_bin';"

Or, just look in your data directory and see a hoarde of binlog files, of course. You can disable it altogether with

skip-log-bin

in your mysql ini file (which one you set it in depends on your server version and OS). Or you can just limit the number of seconds it keeps with the following, which will let it keep one day's worth of binlogs, down from the default 2592000 seconds which is 30 days, and, in my opinion a little high for a default value!

binlog_expire_logs_seconds = 86400

If you're not sure whether or not you need binlogs, you probably don't need them. They are used it you're replicating the database to another one, and need to hold enough data so they can sync up in the event of a disconnection. You can also use them to roll the db back or forward to a specific point in time, but that's the province of database ninjas.

• Searching the site for malware pointing to a blocked URL (no)
• Checking the firewall logs (nothing)
• Checking the db config, running performance tools on it to see that it wasn't exceeding max connections. (no)
• Running tcpdump to see where wp-cli was trying to connect to (it wasn't)
• Updating all plugins, themes, (good to do anyway)
• Checking db.php, which I found in wp-includes … which, hmmm, belongs to W3 Total Cache.

So on a hunch I tried

> wp-cli plugin deactivate w3-total-cache 
Connection refused
Connection refused

... plus 200 more lines 

Connection refused
Connection refused
Connection refused
Plugin 'w3-total-cache' deactivated.

… after which all wp-cli operations worked again.

After some further digging, I enabled w3 Total Cache again, and found that it was configured to use redis, which wasn't running on the server. So the connection attempts were failing to connect to that. Pretty easy to sort out after that …

The IP address you've 550-5.7.1 registered in your G Suite SMTP Relay service doesn't match domain of 550-5.7.1 the account this email is being sent from. If you are trying to relay 550-5.7.1 mail from a domain that isn't registered under your G Suite account 550-5.7.1 or has empty envelope-from, you must configure your mail server 550-5.7.1 either to use SMTP AUTH to identify the sending domain or to present 550-5.7.1 one of your domain names in the HELO or EHLO command. For more 550-5.7.1 information, please visit 550 5.7.1 https://support.google.com/a/answer/6140680

Pretty helpful as messages go. Less helpful was the fact that there were several websites on the server and a couple of other apps, and I didn't have access to the admin panels of any of them. The mails, once bounced, were removed from postfix's queue, never to be seen again. Time for some detective work.

I found that the mails were heading for zzzz@domain.com so I figured if I could make those appear locally I'd be able to look at the source and see what had created them. This is not so hard to do in postfix. First of all you need a line in your main.cf

# Added to troubleshoot rejection by gmail smtp           
# Need to add addresses to the file and run postmap on it 
recipient_bcc_maps = regexp:/etc/postfix/recipient_bcc_maps

Then you need to add entries in your /etc/postfix/recipient_bcc_maps

# run postmap recipient_bcc_maps after editing
/zzzz\@domain.com/  ubuntu

After that run postmap recipient_bcc_maps and restart postfix for good measure. So now any mail headed to zzzz@domain.com will also be bcced to the local ubuntu user, were we can pick it up in /var/spool/mail/ubuntu

All good. I waited. And waited. Then got bored of waiting and made a quick bash one liner to alert me when the ubuntu user's mail file changed.

while true; do if ls -hal "/var/spool/mail/ubuntu" | grep -q "Nov 15 07:14" ; then sleep 5 ; else mutt -s "changed" me@mydomain.com </dev/null ; break ; fi ; done

So this is pretty crude but it works. Every 5 seconds it checks the date and time of the mail spool file (you'd need to change to the actual date of the file, of course), and if it changes it emails to me@mydomain.com. I ran that under 'screen' and two days later I got a notification, from which I was able to figure out which script on which domain had fired off the email.

Maybe that one liner makes more sense as a script:

#!/bin/bash

FILE=/var/spool/mail/ubuntu
PATTERN="Oct 17 10:44"

while true;
do 
	if ls -hal "$FILE" | grep -q "$PATTERN" ; then 
		# Do nothing. 
		sleep 5
	else
		# echo "changed"
                # play fanfare.mp3
		mutt -s "changed" me@mydomain.com </dev/null
		break;
	fi 
done

I'll assume you can figure out how to download and install wp-cli yourselves. On this occasion I was trying to get rid of the annoying pop-up screen of Really Simple SSL plugin. It should disappear, never to return, when you click the x in the corner of the message, but in my case it was coming back every time I loaded every single page. Which was irritating, to say the least, and it takes up a fair chunk of space at the top of the screen. So I decided to nail it for once and for all. I found out on Google that the key was altering the rlrsssl_options => ssl_success_message_shown option.

So if you use wp-cli to get the option value it looks like this:

wp-cli option get rlrsssl_options
array (
  'site_has_ssl' => true,
  'hsts' => false,
  'htaccess_warning_shown' => false,
  'review_notice_shown' => false,
  'ssl_success_message_shown' => false,
  'autoreplace_insecure_links' => true,
  'plugin_db_version' => '3.3.5',
  'debug' => false,
  'do_not_edit_htaccess' => false,
  'htaccess_redirect' => true,
  'ssl_enabled' => true,
  'javascript_redirect' => false,
  'wp_redirect' => true,
  'switch_mixed_content_fixer_hook' => false,
  'dismiss_all_notices' => false,
  'dismiss_review_notice' => false,
)

There is also the option to display that as json, which is what we need, so lets do that, and pipe it out to a text file.

wp-cli option get rlrsssl_options --format=json > ssl.txt

cat ssl.txt 
{"site_has_ssl":true,"hsts":false,"htaccess_warning_shown":false,"review_notice_shown":false,"ssl_success_message_shown":false,"autoreplace_insecure_links":true,"plugin_db_version":"3.3.5","debug":false,"do_not_edit_htaccess":false,"htaccess_redirect":true,"ssl_enabled":true,"javascript_redirect":false,"wp_redirect":true,"switch_mixed_content_fixer_hook":false,"dismiss_all_notices":false,"dismiss_review_notice":false}

So now we edit the ssl.txt file and change this bit from false to true.

"ssl_success_message_shown":true

Then we can just send that json file back into the database with the following:

wp-cli option update rlrsssl_options --format=json < ssl.txt

find /path/to/dir -regex ".*\.\(jpg\|png\|gif\)" -exec file {} \; | grep -i -v "image data"

If all is good, you won't get any output. If your server is seriously borked, then you might see things like this …

./wp-content/uploads/2011/01/22.jpg: HTML document, ASCII text

This is a flag that the image is in fact a PHP file. Investigate!

If you get this kind of thing,
./wp-content/uploads/2011/01/221.jpg: Minix filesystem, V2, 46909 zones

its probably a bug in an old version of file, so check your OS version, copy the file to a more recent OS and try again.

https://yourdomain.com/wp-admin/options.php

Obviously you have to be logged in as admin. This basically gives you access to the wp_options table without having to go to a separate database management app, or do some mysql command line ninja.

Who knew? Its not on a menu, probably to hide it from fat fingers, but hey, I wish I'd known about this earlier. Would have saved me a few hours over the course of my work life.