It turns out, from my readings on the net, that iCal is a little limited with regards to publishing remote calendars and subscribing to them. You can either publish, or subscribe, but not both. That knocks iCal down a few pegs in my book, but it still seems to be the best option going for Palm synchronization, so I’m still going to build my initial calendaring solution around it. I won’t have all the functionality that I want, but it will be a good start.

Incidentally, after my initial experimentation with Sunbird, I went back and read more about it. It turns out the version I was using, version 0.3alpha, is based on a total code rewrite and is actually less stable than its predecessor, version 0.2. I tried 0.2 out, and it doesn’t seem to suffer from the crashing problems I had with 0.3alpha. In fact, I can subscribe to and edit a remote WebDAV calendar just fine. iCal cannot do this (presumably because Apple wants to sell me a .mac membership to get this functionality). This, coupled with the fact that Sunbird works on Mac, Windows and Linux, makes me lean towards Sunbird as my ultimate Calendaring solution. However, as Sunbird does not have Palm sync support yet, I’m still stuck with iCal for the time being. So here’s what I’m going to do:

1. Shell out the $40 for the Missing Sync product, to get tightly integrated Palm sync support for iCal. Hope the product works as well as advertised.
2. Load all of my calendar data into iCal (export data out of Oracle Calendar and merge any local changes that I put into Palm Desktop)
3. Publish all of my iCal calendars to my WebDAV server so I can view them with PHP iCalendar. Set up autopublish so they stay up to date.
4. Try out the Oracle Calendar SDK. It would be nice to have a command line tool to export the data out of Oracle Calendar and automatically generate a .ics file that I can subscribe to with iCal.

This will give me most of the functionality I’m after. I’ll be able to view, but not edit, the published calendars with PHP iCalendar or Sunbird. To edit them I’ll either need to use iCal on my laptop, or my Palm. Not perfect, but livable.

Just spent a bit of time playing around with Sunbird. I would call it “not quite ready for prime time.” It’s got a nice interface. I was able to subscribe to my iCal-published calendar without a hitch. It can auto-reload remote calendars on startup. I can also create local calendars and publish them successfully. However, the publishing support seems to be kind of a work-in-progress. When I publish a new calendar, it doesn’t remember its URL. I have to re-type it every time I publish. When I try to modify a remote calendar that I subscribed to, it crashes. There appears to be no support for auto-publishing as entries are made.

I’ll keep an eye on Sunbird, but it looks like it’s not quite ready for day-in day-out use. I expect it’ll get there eventually though.

I did some more research on calendar sharing over the long weekend. The first thing I tried was using iCal to publish a calendar to my new WebDAV server. It worked without a hitch. However, all this does is upload the .ics file to the web directory. I can then subscribe to the calendar with another iCalendar-capable client, but there’s no built-in facility to view the calendar with a web browser, as I had originally thought (apparently this is a feature of Apple’s .mac subscription service).

A little bit more searching turned up a nifty app called PHP iCalendar. This does exactly what I was looking for: install it, point it at a directory with iCalendar files, and it displays everything on my web browser. Installation was as easy as putting mod_php4 on my web server, untarring the distribution, and editing the config file. It works great and the interface is nice. So, with that I now have read-only access to my calendar from anywhere on the net.

I also wondered if there was a full-featured iCalendar-compatible calendar app out there for Linux. I found the Mozilla Calendar and Sunbird Project. Mozilla Calendar is a plugin for Firefox or Thunderbird, while Sunbird is a standalone app. Both share the same codebase, features and bugs. This is alpha pre-release software, but it’s being actively developed and appears to be the most promising solution. It works with Windows, Mac and Linux, meaning I could hypothetically use it on all three platforms (although on the Mac, I currently need iCal for Palm synchronization). I’ve downloaded this and verified that it seems to work, but I haven’t played with it extensively.

All this gives rise to the question: Could I keep the “official” version of my calendar on my web server, access it with Sunbird, iCal and/or PHP iCalendar, and have all my changes automatically sync up with the web copy? That is definitely the question, and if it’s doable, I think it’s the optimum solution. If I can track down a working version of the Oracle Calendar SDK for Linux, I could even automate the integration with Oracle Calendar.

I know I can configure iCal to automatically publish to the web whenever I make a change to it (although I’ve never tried it so I’m not sure how it works in practice). I think Sunbird has a similar capability. Again, it remains to be seen how well this will actually work. I don’t have the Mac with me today, so I’ll play around with Sunbird on my Linux box, and see what it can do.

The final piece of the puzzle is the Palm. I need to get the Palm syncing up with iCal. Apple provides an app called iSync which can do this. However, I tried it and was not impressed. iSync is an intriguing application, but it seems mainly geared towards cell phone type devices. Its support for the Palm seems to be kind of an afterthought. It works, but it doesn’t support a lot of the features on the Palm, particularly with the address book

A more promising looking solution is a commercial product called The Missing Sync. This is a total replacement for Palm Desktop/HotSync that is specifically geared to the Mac. It reportedly integrates very well with iCal and the Mac’s address book. It also includes a lot of nifty extras (internet connection sharing, expansion card mounting, etc).

Missing Sync costs $40, but based on what I read, it’s worth the price. There are two things about it that I would consider negatives:

1. No trial/demo version available I’m willing to look beyond this because of the overwhelmingly positive reviews I’ve read about the product. A lot of people, myself included, like try-before-you-buy, particularly with software. They could probably sell more copies of this if they had a demo version. Let people try it, wonder how they got along without it, and they’ll buy it.
2. Can only be installed on 2 macs Not an issue for me now, as I have only 1 mac, but who knows how many I’ll accumulate over the years? I really hate it when companies cripple their products like this. Still, I’ll live with this despite my philosophical problem with it. If it ever becomes an issue for me, I’ll complain loudly.

Issues aside, it looks like a high quality product, and unless I change my mind about using iCal, I’ll probably give this a go.

No secret now that I’m looking to iCal as my calendaring app of choice, so I’m continuing to put it through its paces. Next up is to try out its publishing feature. Supposedly, I can publish the calendar to any WebDAV server and then access it using a web browser (or subscribe to it using another iCal client).

I didn’t have a WebDAV server sitting around anywhere, so I went ahead and set one up using Apache 2.0.54 with mod_dav/mod_dav_fs. It was pretty easy, basically just load the DAV modules and enable DAV in the appropriate directory (with authentication of course). I can access it with a command-line tool called cadaver, which provides FTP-like access to the WebDAV space. So it looks like everything is good to go on the server side.

I don’t have my Mac with me today, so I’ll try uploading a calendar tonight after I get home.

Playing around some more with Oracle Calendar’s export feature, and the resulting data file, to try to get my all-day events to show up properly in iCal. First off, Oracle Calendar can actually export into two different formats, vCal and iCalendar. iCalendar is actually a newer spec that grew out of vCal. Apple’s iCal can import either format. This is significant, because when I export in iCalendar format, the all-day events get imported into iCal properly. When I use vCal, they show up as one-minute meetings starting at midnight.

So… let’s just use iCalendar instead of vCal, right? Wrong. First off, I have no problem exporting a small date range (say, a month) in iCalendar format. When I try to export a 3-year date range, Oracle Calendar goes off into la-la land and never comes back. OK, well maybe it eventually comes back, but I gave up after about 20 minutes and killed it. This happened with multiple clients on multiple machines, so it’s not just the Mac. It’s either having a problem with one of my entries (which would be a bug), or it just plain takes forever. Either way, this makes it unusable for my purposes.

Not only that, when I managed to successfully export an iCalendar file, iCal wouldn’t import it. It imports the first entry and then seems to quit. Not sure which app is at fault here, but don’t really care.

That leaves me with vCal. Oracle Calendar will export three years of data in vCal format in about a minute or so. Not exactly lightning fast, but usable. Now, how to get iCal to treat my events as events and not 1-minute meetings?

Here’s the deal: the vCal spec uses the DTSTART and DTEND fields to identify meeting start and end times, respectively. If you leave out DTEND, it treats the entry as an untimed event. And it works. If I edit the vCal file and remove DTEND from an untimed event, iCal imports it properly. Here’s a Perl script to do it:

#!/usr/bin/perl
#
while (<>) {
    print;
    if (/^BEGIN:VEVENT/) {
        printEvent();
    }
}
exit 0;


sub printEvent() {
    my @evData = ();
    my $miscFlag = undef;
    while (<>) {
        push(@evData, $_);
        if (/^CATEGORIES:MISCELLANEOUS/) {
           nbsp;$miscFlag = 1;
        }
        elsif (/^END:VEVENT/) {
            foreach (@evData) {
                print unless (/^DTEND:/ && $miscFlag);
            }
            last;
        }
    }
  }

iCal is the calendaring app that Apple includes with MacOS X. It comes bundled with another app, iSync, which handles synchronization with the Palm. On MacOS 10.4 (Tiger), iCal’s app data lives under $HOME/Library/Application Support/iCal. It appears that the calendar data itself is stored in iCal format (.ics), while the metadata is stored in XML format. Separate calendar categories are stored in separate files, unlike the single monolithic binary file Palm Desktop uses. All data files appear to be flat ASCII. That means that, if I wanted to, I could easily maniuplate the files using Perl or somesuch. However, not knowing the interrelationships between the various files, I feel like I should probably avoid this. So, let’s look at the app and see what kind of import features it offers.

iCal calls calendar categories “Calendars”. When I go to import a vCal file, it asks me to select a destination calendar. I can pick an existing calendar, or tell it to create a new one. In the latter case, iCal creates a calendar with the same name as the vCal file (minus extension). It appears to ignore the category data in the vCal file, which for my purposes is good.

To delete a calendar, I just select it in the left pane and press the “Delete” key. It warns me that all entries under that calendar will be deleted, then does it. Everything is undoable.

This seems to work great. The import process is smooth. It creates a new calendar category for me automatically. Deleting the data (prior to a subsequent import) requires only one keystroke. It appears to import locations and attendees too. The only drawback is the handling of “daily notes” and “day events” from Oracle Calendar. However, this problem is on the export side. Ideally I’d like these to come in as “all day” events rather than 12:00am-12:01am meetings. If I can’t configure things this way on the export side, I should be able to write a perl script to modify the vCal import file.

iCal also includes sharing features. I can subscribe to a calendar on the internet, and also publish my calendar to a WebDAV server. I haven’t researched this much, but I hopefully I can use this to access my calendar from anywhere on the net (read: my desktop machine at work).

Palm Desktop is, of course, the app that Palm produces to do desktop integration with its handhelds. All in all it’s not a bad little app suite, and you certainly can’t beat it for integration with the Palm. It supports all of the Palm’s features and synchronization (via the bundled HotSync manager) is flawless.

Palm Desktop is not server based, so all of your data lives on the machine where you have it installed (which, by extension, is the same place you synchronize the handheld). There doesn’t appear to be a way to share the calendar data on the internet. That means I’ll need either my laptop or my Palm handy to view my calendar. A drawback, but not a show stopper.

On the Mac, the Palm Desktop calendar data appears to live in a single file, $HOME/Documents/Palm/Users/Paul Riddle/User Data. This file contains binary data presumably in some database format that Palm Desktop can read. The ‘file’ utility reports it as a “DBase 3 data file”. If I run ‘strings’ on it I can see entries from both my calendar and my address book. This doesn’t look like something I want to muck with outside of the Palm Desktop app. So, we need to see if Palm Desktop can do bulk deletion of imported entries.

Looks like we’re out of luck with the bulk deletion. In an older version of Palm Desktop (on my Windows machine), it works. I can create a custom category, create some entries within that category, then delete the category. The software gives me the option to delete all entries within the category, or move them to the “Unfiled” category. However, this doesn’t work with the newer version of Palm Desktop on the Mac. It looks like they’ve added the ability to have multiple categories assigned to an event. If I delete a category, the events in that category stick around, even if no other categories are assigned to the event in question. Kind of sucks that they would cripple that functionality. But it pretty much means that I can’t use Palm Desktop for what I want to do. Nevertheless, I’ll continue to evaluate it a bit and see what I find.

Palm Desktop also seems to have problems with “day events” and “daily notes” under Oracle Calendar. These events are actually untimed, so they should be imported as “untimed” (or “all day”) events. However, Palm Desktop imports them as timed events and sets the start/end times to 12:00am and 12:01am. I’m not sure if this is the fault of Palm Desktop or the Oracle Calendar export.

OK, the “day event” handling appears to be a “feature” of the Oracle Calendar export. The 12:00am-12:01am time range is in the vCal import file, so Palm Desktop is just recording it as it sees it. The vCal file also includes Category information, and Palm Desktop is reading that and creating new categories as it sees them. The Oracle Calendar entries show up with categories of “APPOINTMENT”, “HOLIDAY”, “MISCELLANEOUS”, etc. If a bulk delete were possible, I’d need to delete all of these categories before re-importing; a bit of a pain. Alternatively, I could massage the vCal import file to put everything into a single category.

To summarize: Palm Desktop uses a (possibly proprietary) binary file format to store entries on the host computer. There’s no out-of-box way to share calendars among multiple computers (although apparently there are third-party products that will do this). Bulk deletion of entries does not appear to be easily possible. Even if it was possible, the import process would require multiple steps to recategorize imported entries into a single category.

Given all this, it looks like Palm Desktop is not going to be the answer for me. A shame, because I’m sure I won’t find anything else as tightly integrated with the Palm. Next up, I’ll check out iCal.

Earlier I mentioned iCal and Palm Desktop as two possible solutions for my desktop calendar. To integrate with Oracle Calendar, both of these apps would require me to manually import the Oracle Calendar data in vCal format. Oracle Calendar’s export function gives me several options for choosing what data gets exported. I can export entries within a specific date range (from 1/31/2004 through 12/31/2006 for example), I can export a relative date range (1 year in the past through 2 years in the future), or I can export the entire enchilada (data that dates back to early 1999, in my case). In the past, I’ve gone with a relative date range starting 1 year in the past and going 2 years in the future. Reason being, exporting the whole thing is kind of slow and there’s no bounds on the size of the data set being exported, so it will keep getting slower and slower the more data I accumulate in Oracle Calendar.

Now, with a 3-year date window, it’s a given that I’ll be exporting a lot of the same data over and over, and importing the duplicate data into iCal or Palm Desktop. Neither of these apps has what I would call an “intelligent” import feature. That is, if I import data more than once, it will show up multiple times in the application. This isn’t a show stopper, but duplicates are undesirable and we need a way to prevent this from happening.

One way to do this would be to just import the data, then cobble something up (in Perl, perhaps) to strip out the duplicate entries. This would require the ability to manipulate the calendar app’s data outside of the calendar app itself, so presumably it would rely on the app using some kind of standard, documented format to store its data. My surfing reveals that someone has done this with iCal, so this may be possible if iCal is my solution. Dunno about Palm Desktop, but intend to find out.

Another way to do it would be to simply delete all of the previously imported data out of the calendar, than re-import. If I went this route, I would need a way to differentiate imported data from other data inside the calendar, so I can delete just the imported data and nothing else. Then, I’d need the ability to do a bulk delete, ether from within the app’s GUI, or (once again) by manipulating the app’s data with an external script.

Both of these methods will require a relatively robust Palm sync app, so that deleted entries are picked up and properly deleted off the Palm. Otherwise we’ll still end up with duplicate entries on the Palm.

Next I’ll look at Palm Desktop and iCal more closely, to see if either will work as part of my evil plan.