I play guitar, and I’d previously written a small script to act as a guitar tuner. I generated some .ogg files of the correct pitch and then wrote a script to loop through them in sequence. “Pretty good”, I thought, “Clever Me.”

Then a few days ago I came across this little gem of a script.

for n in E2 A2 D3 G3 B3 E4;do play -n synth 4 pluck $n repeat 2;done

That’s it. Sheer brilliance. Of course you’ll need to install the sox package first.

There are a number of tools which are useful in finding out things about your filesystem. ls, du, df are three of them, but sometimes they just don’t give you the information you need. In my case I’m backing up a server to a remote location. The script was timing out becase I was trying to back up too many files at once, so I needed to find the number of files in each subdirectory.

Sounds easy at first, and there are numerous attempts at finding that information around the internet. But none of them did exactly what I wanted.

There were PERL scripts and python scripts, and the minimal

ls -alR | wc -l

which gives the total files under a directory, but not quite. Anyway, after experimenting for a long time, I finally put together the following command in all its glory.

find . -type f | awk -F/ '{ print $2 }' | sort | uniq -c

Lets just go through this, so you can tailor it to  your own needs. Each part of the command passes its output to the next part of the command through the pipe sign ( | ), so we can consider each part separately.

find . -type f . This provides a listing of all the files (ie not directories and symlinks etc) underneath the current directory. You can modify the options to the find command to find other items as well, but I wanted files. This pumps out a list like this.

./dir1/file1

./dir1/file2

./dir2/file1

./dir2/subdir1/file1

./dir2/subdir1/file2

awk -F/ ‘{ print $2 }’ This takes the previous list and splits it into fields using the / sign (-F./). It will then output a list of these. Continuing the example above:

dir1

dir1

dir2

dir2

dir2

sort Sorts the list, as you might expect. There are two reasons for this. First is that find doesn’t always find files in alphabetical order. Second is that the next command needs like terms to be grouped together to count them properly. Anyway, this leaves the list above unchanged in this case.

uniq -c Takes the list above and returns only unique values, with a count in the left margin ie.

2   dir1

3 dir2

… which is exactly what I wanted.

I also used the

du -h --max-depth=1

to find out the sizes of these directories.

Anyway, on getting it back up, I poked around in the logs and found that it was running out of memory, which it really shouldn’t do given that it was a fairly low load on the webserver, and only apache, mysql and php were running on it. I looked around the Interwebs and found an excellent resource called Troubleshooting Memory Usage, which gave me some pointers about how to rein in apache and stop it from eating up memory so quickly. Thanks!

Anyway on the page was a script for summarising memory usage, which I thought was a good idea, so i grabbed that one and adapted it to my own purposes. I thought I’d make the results available here, so feel free to grab it and adapt it to your own needs.

#!/bin/bash
# Mem-info.sh. Version 1.5 2008-12-20
# Adapted from various sources around the Internet.
# Requires mutt to be installed to use the email function. 

#################### Functions ####################
function print_help {
	echo "  Usage:"
	echo "    $0 [-a full|summary] -f /tmp/memoryreport.txt"
	echo "    Parameters:"
	echo "    -a              Specify full or summary report"
	echo "    -f              Specify alternate filename "
	echo "    -e <adddr>      Send report to specified email address"
	echo "    -h | --help     This usage information"
	echo "    Running the script with no parameters writes a summary report to ./output.txt"
}

function summary_report {
	echo "=========================== SUMMARY ============================" > $mem_outputfile
	date >> $mem_outputfile
	echo  >> $mem_outputfile
	echo "=========================== uptime ==============================" >> $mem_outputfile
	uptime >> $mem_outputfile
	echo  >> $mem_outputfile
	echo "========================== free -m ==============================" >> $mem_outputfile
	free -m >> $mem_outputfile
	echo  >> $mem_outputfile
	echo "========================= vmstat 1.5 ============================" >> $mem_outputfile
	vmstat 1 5 >> $mem_outputfile
	echo  >> $mem_outputfile
	echo "================== ps top 20 Processes by CPU ===================" >> $mem_outputfile
	ps -eo user,%mem,%cpu,pid,cmd --sort -%cpu | head -n 20 >> $mem_outputfile
	echo  >> $mem_outputfile

}

function full_report {
	summary_report
	echo "========================= FULL DETAIL ===========================" >> $mem_outputfile
	echo "======================== top raw output =========================" >> $mem_outputfile
	top -b -n 1 >> $mem_outputfile
	echo " " >> $mem_outputfile
	echo "======================= ps auxf raw output ======================" >> $mem_outputfile
	ps auxf --width=200 >> $mem_outputfile
	echo  >> $mem_outputfile
	echo "=========================== end ================================ " >> $mem_outputfile
	echo  >> $mem_outputfile
	echo  >> $mem_outputfile
}

#################### Process Command line ####################

while [ "$1" != "" ]; do
    case $1 in
        -f )           	shift
                                mem_outputfile=$1
                                ;;
        -a )    	shift
				mem_action=$1
                                ;;
        -e )            shift
				email_to=$1
				;;
        -h | --help )           print_help
                                exit
                                ;;
        * )                     print_help
                                exit
				;;
    esac
    shift
done

######################### Set Defaults #########################
if [ -z "${mem_action}" ]; then
    echo "    No action Specified. Defaulting to summary report"
    mem_action="summary"
fi
if [ -z "${mem_outputfile}" ]; then
    echo "    No output file specified. Writing to `pwd`/output.txt"
    mem_outputfile="output.txt"
fi

######################### Do Stuff #########################

if [ "$mem_action" = "full" ]; then
		full_report
	else
		summary_report
fi

if [ "$email_to" = "" ]; then
		echo "    No email address specified. Not emailing report."
	else
		mutt -s "Memory Usage Report" $email_to < $mem_outputfile
fi

Or you can download the file mem-info1 if the formatting above is all messed up. Make sure you go to the page I linked to above for excellent advice on interpreting the report.

There are a number of factors which are conspiring to make Linux a viable alternative:

• The fact that Linux, via distributions such as Ubuntu, Mandriva, Suse et al, are now easy enough to install and intuitive enough for the Everyday User. I set my girlfriend up with an account on a spare laptop and just let her play. Soon enough she was asking for it on her laptop as well.
• OpenOffice. Enough said. Hardly anyone uses the full feature set of Word, so why do we slavishly upgrade every time a new version comes out? Well actually we don’t …
• The nightmare of Vista. Hardware incompatibilities galore, a confusing new interface with more eyecandy than improved usability, increased hardware requirements. The security enhancements which, ahem, aren’t really all that more secure, are they?
• Open source = no license fees. Can’t argue with that.
• Reduced downtime due to viruses.

All of these are great reasons, as well as the more emotive reasons for doing it. (Hate Microsoft!; Can’t afford a Mac! Irritated with bloatware!). However while Linux is great for me, I still spend more time fixing it than most people are prepared to spend.

Take last week for example. A standard Update to my OS suddenly starts installing a different kernel – it decided all of a sudden that I’d like a laptop kernel. This would have been alright in itself, but then suddenly

• my display driver doesn’t work. I install it again.
• My microphone stops working in Skype. I try for a few hours to get it going again, but can’t. I use my spare laptop to do Skype calls.
• My VMware server doesn’t work. I try to re-compile it, but in the end can’t get it to work,

So I roll back, eventually to the desktop kernel. I reinstall the display driver. I manage to get VMware back up. The Skype Mic problem is still with me.

I’ve still no idea why I was given a different kernel. I’m experienced enough after 4 years with Linux to get it back working again, but that really shouldn’t have happened in the first place. And its things like that which I think is hampering the progress of Linux.

I plugged the thing in (its a USB scanner, which pulls all its power from the USB bus) and the light went on. However the scanner application which comes with both PCLinuxOS and Ubuntu wouldn’t start up. Time for some investigation. I got a root terminal shell and tried poking around

lsusb showed that the scanner was recognised.

Bus 002 Device 005: ID 05d8:4002 Ultima Electronics Corp. Artec Ultima 2000 (GT6801 based)/Lifetec LT9385/ScanMagic 1200 UB Plus Scanner

And the Sane website told me it was supported.

The first piece of research suggested that I needed to edit a configuration file, /etc/sane.d/gt68xx.conf

I opened this up, and found the following four lines, all commented, so I uncommented the last three.

# Trust Flat Scan USB 19200:
override “artec-ultima-2000″
vendor “Trust”
model “Flat Scan USB 19200″

The xsane application still didn’t want to work though.

Now I tried the command line application scanimage. scanimage -L correctly listed the scanner

device `gt68xx:libusb:002:005′ is a Trust Flat Scan USB 19200 flatbed scanner

But scanimage on its own gave me the following message

[gt68xx] Couldn’t open firmware file (`/usr/share/sane/gt68xx/Gt680xfw.usb’): No such file or directory

OK, so we need to find some firmware. A google search on Gt680xfw.usb revealed two candidates. I made a directory in /usr/share/sane/gt68xx/ and dropped the first firmware in there, taking care to rename it with the correct capitalization. Bingo. Now Xsane works. You can get the file here …

That was easy. And yes, I know I could have downloaded the windows drivers from here, but that wasn’t the point!

I excused myself, went back home and after some more fiddling, decided to backup and re-install. Having made this decision I was looking through my pile of install CDs, and I came across PCLinuxOS 2008, which I’d downloaded a few weeks previously, and I’d been meaning to try out. “So why not try it out on this laptop?” said the evil part of my brain — the same part which forces me to spend time on Facebook instead of working.

I decided to go ahead with this monstrous plan, partly due to the inexplicable freezing above, and partly because there were a couple of issues I’d never managed to resolve with Ubuntu 7.10, namely:

• Truecrypt wouldn’t work
• Wireless didn’t work very well. It had taken me a couple of days to get it working initially, and it still doesn’t fully work – sometimes dropping the connection.
• One part of my filesystem was mounted twice. I forget which part now, and it was not causing any problems, but it was irritating me.
• Probably some more things as well …

OK, so time to try something new then. I’m not going to dwell on the details, but here’s the shortened version of the process.

1. Backup existing partitions to an external USB hard disk, using Gparted Live CD. These were a two ext3 partitions, mounted on / and /home, and a VFAT32 partition containing some bits of data, and storage.
2. Run the PCLinuxOS installer. What I was trying to do was to leave the /home and VFAT partitions intact, and replace the / partition. However the installer was somewhat clumsy in this respect, and actually reformatted the whole disk. I wasn’t sure whether or not this was due to me misunderstanding the installer or due to a bug. Anyway, good job I backed up.
3. Restore the /home and VFAT partitions from the backup using Gparted CD again. Awesome tool.
4. Replace the existing /home directory with the newly restored partition. eg.
   • Booting in single mode, rename /home to /oldhome.
   • edit fstab to mount the new partition as /home
   • Restart
   • Check permissions. Need to change ownership of files within my user directory.
   • Some applications don’t work. Remove preference directories from /home/user directory and replace them a few at a time. Replace the .gnome directory with the corresponding one from /oldhome.

               /dev/hda6       /home   ext3    user,exec,rw,auto       0 0

That was actually pretty easy. Now the good stuff.

• Wireless worked immediately with WPA2, something I’d never been able to get running on Ubuntu.
• The fonts on the screen were a good size. I’d always found Ubuntu fonts a bit large, but the ones on PClinux OS were much better suited to my 1024×768 display.
• Sound and video worked fine out of the box. I’d had to fiddle with Ubuntu to get them going. I still needed to use the noapic boot trick though, which I was anticipating anyway.

Now to install some software. PClinuxOS doesn’t come with as much pre-installed as Ubuntu, so I added the following applications. Oddly, PClinuxOS is an rpm based distro, but it still uses apt-get.

• su -l  (N.B. PClinuxOS uses a root login rather than sudo)
• apt-get install nano, openoffice, filezilla, keepass, wine, skype, kopete, k3b, bind-utils
• apt-get remove ephiphany, gnumeric, abiword

Then I installed my favourite windows apps under wine: treepad and notepad++

I wasn’t able to install vmware server from apt-get, so I installed it by downloading from the vmware site, which entailed also getting a new serial number.

Truecrypt wasn’t in the original list of available applications in Synaptic. However I updated the repositories to a different one, and it appeared. Great! I installed it, and it worked perfectly immediately. This was a major coup, as it wouldn’t work under Ubuntu 7.10.

That was pretty much it. The initial setup had taken me around 4 hours, which I could have easily spent tinkering with config files, trying to get the old, troubled install of Ubuntu to work. I installed more applications as I needed them over the next few weeks, but the initial install was good enough to get me up and running again.

After having used it for a month or so, one frustrating element of PClinuxOS is that there is a limited selection of software available for it. There are dire warnings and threats in the user forum about compiling your own software, for which you forego any chance of support. They have a point of course, but the’re definitely a bit less friendly than Ubuntu.

By the time I got around to writing this, of course Ubuntu 8.10 is out. I may switch back to it next time I have to re-install, but the point that this exercise has demonstrated to me, is that switching OSes, linux to linux is a lot easier than going cross platform, and can easily be done inside a day. Food for thought.

cat /proc/acpi/thermal_zone/THM0/temperature

which seems a bit unhealthy. My motherboard fried itself twice last year, and I figure that might have had something to do with it.

Anyway, I started casting around for things to reduce the power consumption, and found this powertop utility, apparently developed by Intel. (http://www.lesswatts.org/projects/powertop/) You can download it and install it from the site, but I just did a quick

sudo apt-get install powertop

and that seemed to work pretty well. When you run it (with sudo powertop), it reports all the things that are keeping your CPU awake and suggests a few things you can do to consume less power. Even better it will make these changes for you, if you press the appropriate letter on the keyboard.

1. Make a windows install CD from the install files on the windows partition (Thinkpads don’t have an install CD, they use an Install partition)
2. Install vmware on Ubuntu and make a Windows Virtual Machine.
3. Delete the old Windows and IBM Install partitions
4. Re-arrange the remaining partitions to suit the new arrangement.

Making the Windows Install CD.

This is a little bit complex because I have an IBM Thinkpad. The OS on these comes as a 4Gb rescue partition which you can boot into and restore your Thinkpad to factory condition. I made the DVDs and CDs to complete this, but actually I didn’t want to use these, as it a) reformats the hard disk into a single partition + recovery partition, and b) installs a whole raft of IBM drivers widgets and bloatware that I don’t want. Nevertheless I have an OEM license for Windows XP home which I’m entitled to use on this machine, so the question is how to get it installed in a Virtual machine.

The answer I found in several blog posts and howto articles, which I’ve subsequently lost. You can get most of the info here http://www.howtohaven.com/system/createwindowssetupdisk.shtml

Here is how I modified that info and the main steps I took:.

1. Start with the files installed in c:\i386. Copy these to another location so you can work on them without breaking anything (despite the fact you’re going to vape your windows install in a couple of hours anyway)
2. With the files in the new location (say d:\xpinstall) you need to add some plain text files to tell it that the install is an OEM install (see reference article above)
3. Slipstream in Service Pack 2.
4. Use nLite to slim down the installation to the essentials, (we don’t need no stinkin’ helpfiles!) and then build you a boot CD iso image.

VMware Install and setup

Meanwhile, back in Linux Mint / Ubuntu, we have to install vmware server. This is done pretty easily through Synaptic. If you don’t see it in there, you’ll have to enable the third party software repositories. I chose VMware server as opposed to Workstation or Player because it was freer than Workstation, and it had more functionality than the Player.

Once vmware Server is set up, you need to start it from the item it put in your Start Menu, and connect to localhost. From here you can create a new machine. I opted to use a dynamic container with a max size of 8Gb, although I could have got away with less. I set the CD Drive to point to the .iso image I created above (brilliant feature!), and then booted it up. Windows installed like a charm, and using the OEM licence key on the bottom of my Thinkpad, it was all legal. I also set up the Network connection to NAT to my network so that the Windows machine is never directly exposed to the Network / Internet and the Ubuntu install effectively acts as a router. VMware takes care of all this for you, so you don’t have to think about it too much. However if you want to run a server in VMware, you’d probably use the direct, non-NAT connection.

The final setting you may want to play with is the memory allocation. I have 1Gb of RAM, and I allocated half of that to the virtual machine. Windows runs fine in that, and this is a very low spec machine.

Once Windows is installed, obviously you have to do all that updating, and installing of AntiVirus. I also installed a couple of tools I use and my accounting software. Within the virtual machine, I set up a Truecrypt container for my accounts data, and then wrote a quick script to sync this back up to my Ubuntu installation using scp. This means once I’ve finished doing my monthly accounts I can just hit the button and it backs it up for me.

Having set all this up, I took a snapshot, and backed up the entire VM to an external hard disk, so if I ever need to recover it, I can do so in minutes. Very cool.

Deleting the old partitions

Once again, I left this for a week or so until I was sure things were working smoothly. During that time I also experimented with a couple of other VM images – I built one of Centos 5 server so I could prototype some development, and tried out Zimbra mail server. Unfortunately the Zimbra server comes as an rPath device, so you don’t really get to play with it properlly. Anyway, this virtualization stuff is great. I wish I’d done it properly sooner.

The trouble with having all these virtual images is they take up a lot of space, so I was now beginning to run out of disk space. Time to get rid of those unused partitions.

The tool for the job is the gparted live CD, which I hadn’t used for about a year. I was pleased to see that the latest version, despite a modest increment in its version number, is streets ahead of the last one I used. Great job guys!

So here’s what we do.

1. Plug in an external USB drive to back up onto.
2. Boot from the gparted Live CD, and make sure the USB drive is mounted. eg.
   • mount -t vfat /mnt/sda1 /mount/usb
3. Start partimage from the menu, and use it to back up the partitions we’re going to delete. Hack them into 2Gb chunks as suggested by partimage, as we’re backing up to a VFAT drive. A 20Gb partition took about 20 mins. That’s fast if you’ve ever used something like Encase …
4. Back up any live data partitions you’ve got as well, just for good measure. You can delete them if all goes well
5. Use Gparted to delete the partitions on the disk that you don’t want.
6. Reboot the computer and check you got rid of the correct ones …
7. OK now to organise them

Rearranging the partitions on the disk

Reboot into the Gparted CD. In my case what I now have is

• a 10 Gb root Linux partition (/) on /dev/hda5
• a 1 Gb swap partition on /dev/hda6
• a 100Mb /boot partition on /dev/hda7
• a 15Gb vfat data partition on /dev/hda3 which is in an extended partition, /dev/hda4
• Two chunks of free space.

Moving partitions around in gparted is just a case of copying and pasting. Really. The potential for disaster is high, but if you’re all backed up, then there’s nothing to lose right? I cut and pasted, and re-sized and re-organized, until I was left with this:

• /dev/hda1 – 20Gb – Linux main system (moved and enlarged)
• /dev/hda2 – 100Mb – /boot partition (moved)
• /dev/hda4 – 1 Gb – swap partition (moved)
• /dev/hda3 – Extended partition containing
  • /dev/hda5 – 17 Gb Data partition (enlarged)
  • /dev/hda6 – 17Gb /home partition (newly created to house all the VMs)

Having done such major reconstructive surgery I knew I was in for a few glitches on boot up, but I knew I could get around them eventually.

This post has got a bit long … its continued here.