Ubuntu hard drive upgrade

I just finished upgrading the hard drive on my Ubuntu machine, and it wasn’t as easy or straightforward as I was expecting.  So I figured I’d write up some notes for the next time I do it.

First I backed everything up. Then I shut down the computer, put the new drive in, and booted up with a copy of Knoppix I had lying around. Under Knoppix, I opened up a shell and mounted my old root filesystem:

# mount /dev/hda1 /mnt

I then mounted the new root filesystem on /mnt2, and copied all the files over:

# mount /dev/hdb1 /mnt2
# cd /mnt
# tar cvpf - . | (cd /mnt2; tar xpf -)

Then, I installed the grub boot loader in the MBR of the new drive:

# grub
# root (hd1,0)
# setup (hd1)

At that point, I shut down the computer, removed the old drive, installed the new one in its place, and attempted to boot back up. Happily, it found the grub boot loader and proceeded to load the kernel. But then it hung trying to mount the root filesystem.

It turns out that a couple releases ago, Ubuntu started referring to disk partitions by UUID rather than using a specific device name such as /dev/sda1 or /dev/hda1. Both /boot/grub/menu.lst and /etc/fstab still contained UUID references for the old hard drive, so I had to go through and painstakingly replace all the old UUID references with the updated UUID for the new disk. I just used vi and did a search-and-replace, although there’s probably an easier way. Once I did this, everything booted up just fine.

I can see the advantages to using UUIDs, but it does add an extra layer of complexity when doing something like this. At least I know what to expect the next time around.

Coming soon: my adventures upgrading from Ubuntu 7.04 (Feisty Fawn) to 7.10 (Gutsy Gibbon).  There were a couple of snags, but it was mostly painless.

Server shuffle again

My server shuffling is finished, and went off (almost) without a hitch. Slight hiccup installing my new Zonet USB 2.0 PCI card. When I first popped it in, it worked fine. But for some reason, after I finished all the hardware shuffling and booted back up, the Linux kernel no longer recognized the USB 2.0 EHCI host controller on the card.  I ran ‘lspci’ and the card was there, but it showed only the UHCI (USB 1.1) controller.  And of course, I had just cut the UPC off the box and mailed in the rebate submission, thus rendering the card un-returnable.  Funny how those things work.

Anyhow, before I tried anything drastic, I upgraded my Linux kernel from 2.6.20.1 to 2.6.23.14 (the latest revision as of this writing), and damned if that didn’t fix it. Not sure if this was a one-time glitch, or if the newer kernel actually fixed something related to this. But in any case, I’ll keep an eye on it. I can’t complain too much.. the thing only set me back 8 bucks.

The server shuffle

I’ve decided to do a bit of server shuffling this weekend. I’m basically going to do a case/motherboard swap of concerto, my 700mhz server at the office, with my 450mhz server at home. That will give me a little bit more CPU at home to run stuff like GnuCash and OpenOffice inside my VNC desktop. The motherboards in the two boxes are very similar, so this should be a really easy swap… no new kernels needed, etc. Ironically, this will put concerto back on the original hardware that ran it, which should more than suffice for what it runs now, namely Apache, MySQL and Samba. One difference between the two motherboards is that [I believe] the 700mhz board does not have an ISA slot. That means I won’t be able to use my really-old ISA 56K modem card at home any more. I don’t think I’ll miss it, though, and if I do, PCI modem cards are cheap.

Yesterday I ordered a Western Digital “My Book” 750G external USB hard drive from newegg.com. I need something portable to use for backups of important documents, digital photos, videos, music, etc. The sale price was $175, including free shipping. That works out to just over 23 cents per gigabyte… amazing. And a few years from now, that’s probably going to seem expensive.

Of course, to get any kind of transfer speed out of a USB hard drive, USB 2.0 is a necessity. My old machines only support USB 1.1 on-board, so I also needed to buy a USB 2.0 PCI card. These are amazingly cheap now too. Grand total of $9.99 – $7.00 mail-in rebate, or $2.99. Technology is a funny thing. Compared to 10 years ago, the price of milk and gas seems sky-high nowadays.  But that same 10 years ago, I paid $3000+ for a 300mhz Pentium-II with an 8 gig hard drive, which seemed unthinkably cutting-edge at the time.  Computers (and electronics for that matter) are cheap, cheap, cheap now by comparison.

Saturday update

Got a start on winterizing the pool today, with occasional breaks to shoo Andrew off the pool cover.  I drained the water down below the tile line and added chlorine and algaecide.  The water was nice and clean even after a month of neglect.  Wonder if the algaecide I added last month helped.  Anyways, tomorrow I hope to get out earlier and get the bulk of the work done.  Not sure if I’ll get to blowing out the return lines.  We’ll see.

On the calendar front…  turns out Sunbird is not buggy after all as I had assumed yesterday.  Apple’s iCal exhibits similar behavior.  It appears that if I have events with RECURRENCE-ID properties, somewhere there needs to be an event that “defines” the recurrence with an RRULE or RDATE property.  Oracle Calendar’s output is missing this “defining” event.  I thought briefly about trying to “fix” the recurrences by adding RDATEs, etc. to the iCalendar output, but I think that’s more trouble than it’s worth.  I’m just going to try rewriting the recurring events as separate events, giving them unique IDs based on the start date of the event.  I’ll try it out Monday and see how it goes.

Calendaring revisited

It’s been a year or so since I gave up on my home-grown calendar sync setup.  It was nice for awhile, then we upgraded our Oracle Calendar server, it broke, I tried to fix it and didn’t get very far, and that was the end of that.  Well, as it happens, there’s been some recent interest in an Oracle-calendar-to-iCalendar gateway at work, so I decided to drag my old stuff out and try again.  And it turns out, things have improved in a year’s time.  First off, the Oracle Calendar SDK seems to be more reliable.  I used to get lots of internal library errors, particularly when trying to download large chunks of calendar data.  But that doesn’t seem to be happening now (I know, famous last words).  And on top of that, the iCalendar output is much cleaner.  For example, recurring events are now properly tagged with RECURRENCE-ID properties, so recurrences “just work” now without any extra work on my part.  There are still a few little quirky things, but by and large, it’s a huge improvement.

Also improved is Sunbird, Mozilla’s standalone calendar app.  It’s still a little rough around the edges, but it seems much more robust than previous versions.  I’d eventually like to use Sunbird as my main calendaring app everywhere, because it’s cross-platform and it allows interactive editing of subscribed WebDAV calendars (unlike Apple’s iCal).  The only stumbling block is my old, crusty Palm PDA, which only syncs with iCal.  Much as I’ve liked the Palm PDAs I’ve used over the years, I’m wondering if it isn’t time to start thinking about something different.  It’d be great to have something with functionality similar to Sunbird’s, in a PDA form factor.  Never going to get that with something that relies on desktop sync.

Fun with ssh tunnels

A couple months back, UMBC decided to block off-campus access to most of its internal hosts. Included in this bunch was concerto, which houses this blog as well as my house wiki. Although I could probably apply for and get a hole punched in the firewall for http and ssh, I decided not to bother. I’m not the most proactive guy in the world when it comes to keeping up with security patches, so the firewall thing is probably for the best. Of course, the down side to this is that I can’t use concerto as a free web hosting environment any more, which again, is probably mostly a blessing in disguise. However, there was one big thing I didn’t want to give up: ssh and web access to concerto from our home LAN. After all, there’s not much point in having a house wiki if I can’t get to it from my house! So the challenge became, how do I get this back, and make it as seamless as possible?

Continue reading

Fixing openoffice.org fonts..

I use openoffice.org all the time on my Debian server box, most of the time through a VNC connection. Problem is, the fonts have always looked horrible. ISTR that they weren’t always bad, but they’ve certainly been bad for awhile. Well, today I finally sat down and fixed it.

It all started out when I upgraded the system (which I hadn’t done in forever) to get some updated packages. I was hoping that would fix my OO.o font problems, but it didn’t, so I dug a little deeper…

Basically, there are two issues I was seeing:

  1. The menu font was cartoonishly large in proportion to all of my other apps; and
  2. It appeared that anti-aliasing was not working or something, because all of the fonts had a very crude, blocky scaled look to them.

None of this prevented me from using the system, but it sure didn’t make it enjoyable. Anyhow, first I tackled the ugly-scaled-fonts problem. I noticed that I didn’t have the problem when I started OO.o directly on an X.org display. The problem was limited to the VNC server. The solution turned out to be starting Xvnc with a depth of 24 bits instead of 16. Don’t ask me why it works, but it does. It remains to be seen if the increased resolution will cause any performance degradation over my slow DSL uplink. I’ve not noticed any difference over our 100mbps LAN at home.

I solved the second issue by adding the following line to my $HOME/.Xresources file:

Xft.dpi: 85

So there you have it. As always, Google was extremely helpful in tracking down this info. References here and here.

Let’s go paperless

I’m on a paperless kick. I’ve decided that I have too much paperwork cluttering up my file cabinets, desk drawers, etc., so I’m getting rid of as much of it as I can. My goal is to shrink my paperwork collection down so that it only takes up one file cabinet (I currently have three). It’s one part of an overall downsizing theme that’s pervading our household lately, the idea being that if we get rid of as much stuff as possible now, it’ll be easier to move into a smaller, lower-maintenance house down the road.

It’s also getting easier to go paperless. More and more billers, financial institutions, etc. are offering electronic (usually PDF) statements with the option of turning off paper mailings. It took me a little while to warm up to this technology, but now I’ve accepted it wholeheartedly (the key was deciding that I trust the online delivery mechanism more than I trust our mailman).

The centerpiece of the paperless scheme is what I call a “virtual file cabinet”, which is just a fancy name for a directory hierarchy to hold PDF documents. I use ‘unison’ to keep an exact copy of the hierarchy on a different machine, which serves as an effective backup scheme.

I’ve centered on PDF as my document format of choice, because it works well and is used by the majority of the e-document providers I deal with. And that means that everything that is not PDF, has to be converted to PDF. The best way I’ve found to do this is to set up a virtual “PDF Printer”, which creates a PDF file in lieu of actually sending the document to the printer. Then, just send the document to the virtual queue to create a PDF. This saves a step over printing the document to a file (which creates a PostScript file which then must be converted with ps2pdf). And some apps, like H&R Block’s TaxCut, don’t allow printing to a file, but they’ll happily send stuff to the PDF queue.

Setting up the PDF printer on my Ubuntu machine was a piece of cake, following these instructions. Condensed version:

  1. sudo apt-get install cups-pdf
  2. sudo chmod 4755 /usr/lib/cups/backend/cups-pdf
  3. Go to System -> Administration -> Printing -> New Printer
  4. Select ‘PDF Printer’
  5. Select ‘Generic’, ‘postscript color printer’ driver

It was a little more difficult, but not overly so, to set this up on my home server box (and also configure our Windows box to print to it). See the Wiki for details.

XP and QEMU

Last week I decided to try building a virtual machine to run Windows XP. I figured I’d install XP, install all the recent patches, then make a snapshot of the virtual disk. Then I’d have a clean XP install that I could use to run various apps as needed. Several years back I purchased a copy of VMWare for this purpose, and it worked great. This time around I figured I’d try QEMU, an alternative emulator that’s free. Now, I’ll prefix this by saying that my host machine (a Debian Linux box) is somewhat underpowered to run an XP guest OS. It’s a 700mhz PIII with 512 megs of RAM. It works, and it’s usable, but it’s quite comically slow. In particular, I’d estimate that the installation process took about 12 hours (this was spread out over several days of occasional attention, so I don’t have an exact figure). The basic XP install goes something like:

  1. Install XP from CD.
  2. Install Service Pack 2.
  3. Download and install recent updates from Microsoft to bring the installation completely up to date.

And that’s essentially what I did. The process was marked by long periods of waiting and wondering if it was hung, followed by entering some info in a dialog box, follow by more long periods of waiting and wondering if it was hung, etc. etc. etc. I ran into a couple of known QEMU/XP issues:

  • After the initial install, the reboot hangs on the “Windows XP” splash screen with the message “Please Wait”. Resetting the machine at this point seems to get it moving along again.
  • After initially logging in as “Administrator”, I got a dialog box that said “A problem is preventing windows from accurately checking the license for this computer. Error Code: 0x800703e6.”. Following some advice on the qemu-devel mailing list, I booted into safe mode and installed Service Pack 2, and that fixed this problem.

After that, there was lots more waiting while it downloaded updates, installed IE7, etc. etc. etc., but it eventually finished and left me with a working system. How much use I get out of it remains to be seen, but at least all the effort wasn’t for naught. Recommendation: if you’re going to do this, use a beefier machine than I did 🙂 Also, I highly recommend building and using the kqemu kernel accelerator module. For me, this sped things up from “super mind-numbingly comically slow” to just “comically slow.” And the final tip: If you think it’s hung, go to the host OS and check for activity on the virtual disk image. I lived by this during the install.

And in other news… it’s tax time again! And, this year I’ve decided that I’m tired of getting refunds. Even a “nominal” refund (say $100-$200) is still an interest-free loan to Uncle Sam. So it occurred to me, why don’t I just estimate our 2007 taxes and adjust my withholding appropriately so that I end up owing a nominal amount, instead of getting a nominal refund. Then, with each paycheck, I’ll set the extra amount aside in an interest-bearing account. When next year’s tax season rolls around, I’ll pay the amount owed, and the remainder becomes my “refund”. Seems like a no-brainer, and I wonder why I didn’t come up with this sooner. The complicated part, of course, is estimating our 2007 taxes and avoiding the whole underpayment/penalty/interest thing. But I’m going to give it a go.

mp3act hacking

9 months or so ago, I installed mp3act, a PHP-based online MP3 streaming server. I’ve found that I really like it, but like everything else, it would be oh-so-much cooler with a few little tweaks. Development on it seems to have stagnated (no releases since mid 2005), so I’ve decided to hack on it a bit to make it do my bidding. So far, I’m really happy with the results.

Here are some of the issues I’m trying to address:

  • mp3act doesn’t deal well with albums containing multiple artists. It breaks each unique artist out into its own separate album. So for example, if you have an album called ‘great songs’ containing songs from artist a, artist b, and artist c, mp3act will create 3 albums called ‘great songs’, each containing songs from one of the three distinct artists. This is not what I want.
  • mp3act doesn’t handle artist sorting very well. If you have an album by Tom Petty and the Heartbreakers, mp3act displays it under ‘T’. If you want it correctly filed under ‘P’, the mp3 would need to be tagged ‘Petty, Tom and the Heartbreakers’, which again, is not what I want.
  • mp3act’s search and download features are a little lacking. The search results page allows you to play and/or add songs to playlists one by one, but there’s no ‘play all’ or ‘add all to playlist’ feature. And, it only allows downloads of albums. A ‘download playlist’ feature would be really useful.
  • With multiple-CD sets, if the songs on multiple discs are grouped under the same album name, mp3act will interleave the tracks together (disc 1 track 1, disc 2 track 1, disc 1 track 2, disc 2 track 2, …).

There are a few other little things, but those are the biggies. I’m taking a two-fold approach to dealing with these: tweaking the mp3act database tables for some, and tweaking the mp3act PHP code for others.

mp3act’s database schema is simple and easy to understand. It has a few quirks (biggest quirk: the year field is associated with albums, when in fact, it should be associated with songs), but I’ve found that I can get a lot done just by importing music into the database, and then going through manually and tidying things up. Specifically, I’ve corrected the multi-artist album problem and the multi-disc track interleaving problem this way.

To reconstruct a multi-artist album, first find all of its parts:

select * from mp3act_albums where album_name like '%name here%'

Note album IDs, which are typically consecutive. Then create a new artist, if necessary, for the album:

insert into mp3act_artists values(null, 'Various Artists', '')

Now create a new album:

insert into mp3act_albums values(null, 'name here', last_insert_id(), 'genre here', year, '')

Then, using the album IDs from the first query, re-associate songs with the new album:

update mp3act_songs set album_id = last_insert_id() where album_id between low id and high id

And finally, delete the old, no-longer-used albums.

delete from mp3act_albums where album_id between low id and high id

That’s it. Now, this works great, but it does introduce a couple of problems. First off, when you browse to one of these albums, it doesn’t show the artist name in the track listing. And secondly, there’s now no way to browse to the individual artists that make up the album (unless you have other albums by those artists). The first issue is a quick, easy fix to the PHP code. The second is more complicated. For this one, I added a separate ‘browse’ option that browses songs by artists, instead of albums. It was a bit of work, but not too bad.

And so it goes. Here’s a summary of all the changes I’ve made so far.

  • Fix database to deal better with multi-artist albums
  • Fix database to deal better with multi-cd track ordering
  • Show artist name in track listing for album/song browse
  • Add ‘browse for songs by artist’ feature
  • Create custom genres for some albums. This is easy to do and you are not limited to valid ID3 genres, which is nice, because they suck.
  • Modify ‘browse for albums by artist’ to not show artists with no corresponding albums
  • Fix artist sorting issue by adding ‘sort key’ field to albums table, and modifying artist entries where the sort key is different from the artist name

mp3act works very well for me with these changes, and the framework has the potential to do a lot more. Over time I’ll keep hacking on it to add features and improve its operation.

Followup… My next big “to do” is to make the album downloads more flexible. mp3act handles downloads by building a zip file on the fly, then sending this to the browser as an attachment. But the zipfile library it uses (which is included with the app) is kind of bare-bones, and it requires the mp3 library to have a certain directory structure. Looking at this library, it’s easy to see why downloads are not as flexible as I would like. Fortunately, PHP 5.2 now includes a much nicer built-in zipfile library, so the first step is to modify mp3act to use this. At that point, I should be able to greatly improve the download functionality.