Joomla and mysterious memory usage

I've been running a server which has a fairly busy Joomla site on it. The server has 2Gb RAM, and is running nginx, php5-fpm and mysql, and not much else. However it would run for a while and then the disk would start swapping out. Not a lot, but enough to cause a few issues. If I restarted the server, memory usage would start at something like this

>$ free
 total       used       free
 Mem:       2048036    1024048     1023988
 Swap:      4192960          0    4192960

After about a day it would look like this

>$ free
             total       used       free    
Mem:       2048036    1924048      73988   
Swap:      4192960          0    4192960

And eventually it would have a flurry of activity which would make it look like this

>$ free
             total       used       free    
Mem:       2048036    1924048      23988   
Swap:      4192960       4567     4188393

Read more