• Searching the site for malware pointing to a blocked URL (no)
• Checking the firewall logs (nothing)
• Checking the db config, running performance tools on it to see that it wasn't exceeding max connections. (no)
• Running tcpdump to see where wp-cli was trying to connect to (it wasn't)
• Updating all plugins, themes, (good to do anyway)
• Checking db.php, which I found in wp-includes … which, hmmm, belongs to W3 Total Cache.

So on a hunch I tried

> wp-cli plugin deactivate w3-total-cache 
Connection refused
Connection refused

... plus 200 more lines 

Connection refused
Connection refused
Connection refused
Plugin 'w3-total-cache' deactivated.

… after which all wp-cli operations worked again.

After some further digging, I enabled w3 Total Cache again, and found that it was configured to use redis, which wasn't running on the server. So the connection attempts were failing to connect to that. Pretty easy to sort out after that …

OK, time to roll my sleeves up …

I got into the admin interface and tried a few different themes. Now it was getting interesting … it was present on all themes, which means that an upgrade must have altered something in the main code. Some searching around on the WordPress forums didn't reveal the clammour of angry WP users I was expecting, so I was reduced to searching for generic solutions.

In one post in the darkest corners of the internet I found someone suggesting that a <div> tag wasn't closed properly. That made sense, so I downloaded the source code for the page and edited that. It seemed that adding a </div> tag immediately before the code which calls the sidebar does the trick. Like this:

<?php get_sidebar(); ?>

Becomes

 </div>
<?php get_sidebar(); ?>

Now the challenge was finding that code. I went into the theme editor and tried a few files (header.php, footer.php and page.php)  before I found what I wanted in the Main Index (index.php). Making the change was the work of seconds and reloading the page meant I was back to normal. You may find this code in different places depending on which template you're using. In another template I found the code in the footer.php file, and in this case the closing </div> tag was present, but just appeared before the get_sidebar code. The fix for this was just to reverse the two tags.

OK all good. But what actually caused it in the first place? I haven't altered anything in the templates for several months; after the last upgrade I did, a month or so ago, everything seemed to be OK; after the last post a week ago, everthing seemed to be OK. Strange. And Broken.