Phase One Calendar Plan

It turns out, from my readings on the net, that iCal is a little limited with regards to publishing remote calendars and subscribing to them. You can either publish, or subscribe, but not both. That knocks iCal down a few pegs in my book, but it still seems to be the best option going for Palm synchronization, so I’m still going to build my initial calendaring solution around it. I won’t have all the functionality that I want, but it will be a good start.

Incidentally, after my initial experimentation with Sunbird, I went back and read more about it. It turns out the version I was using, version 0.3alpha, is based on a total code rewrite and is actually less stable than its predecessor, version 0.2. I tried 0.2 out, and it doesn’t seem to suffer from the crashing problems I had with 0.3alpha. In fact, I can subscribe to and edit a remote WebDAV calendar just fine. iCal cannot do this (presumably because Apple wants to sell me a .mac membership to get this functionality). This, coupled with the fact that Sunbird works on Mac, Windows and Linux, makes me lean towards Sunbird as my ultimate Calendaring solution. However, as Sunbird does not have Palm sync support yet, I’m still stuck with iCal for the time being. So here’s what I’m going to do:

  1. Shell out the $40 for the Missing Sync product, to get tightly integrated Palm sync support for iCal. Hope the product works as well as advertised.
  2. Load all of my calendar data into iCal (export data out of Oracle Calendar and merge any local changes that I put into Palm Desktop)
  3. Publish all of my iCal calendars to my WebDAV server so I can view them with PHP iCalendar. Set up autopublish so they stay up to date.
  4. Try out the Oracle Calendar SDK. It would be nice to have a command line tool to export the data out of Oracle Calendar and automatically generate a .ics file that I can subscribe to with iCal.

This will give me most of the functionality I’m after. I’ll be able to view, but not edit, the published calendars with PHP iCalendar or Sunbird. To edit them I’ll either need to use iCal on my laptop, or my Palm. Not perfect, but livable.

Fun with the National Student Clearinghouse

Can I just say that I hate configuring passthrough authentication to outsourced sites on remote web servers?

Rule 1: The company’s step-by-step setup instructions will not apply to your particular system configuration.
Rule 2: It never works the first time.
Rule 3: See rule 2.
Rule 4: Debugging is impossible, because the error logs are on the remote site.

It’s kind of like playing pin-the-tail-on-the-donkey: Put on blindfold, spin around a few times, try some stuff, and maybe you can get it to work. If not, call the rep. The rep is invariably non-technical and will have to pass the request on to a developer. The developer may or may not get back to you with useful info. Or maybe an error log. Or something. Then, you try again.

Oh, and I forgot….
Rule 5: Once you eventually do get it working, it will work for awhile, then break at the most inconvenient time possible, when the vendor decides to make some change to the remote site without telling anyone. Thus, it becomes a perpetual maintenance-hassle-waiting-to-happen that hangs over your head for, well, forever.

I just, finally, got the passthrough authentication to work with the National Student Clearinghouse. Basically, their stuff works like this: Post several magic variables to their web server. Server sends back a form that includes an encrypted token. Form uses javascript to auto-post itself back to NSC web server, which brings up the student’s clearinghouse view.

To make a long story short: It’s very picky about the HTTP_REFERER. We gave them the URL of our development server ahead of time. In the initial request, the referrer string that the script sends must begin with the URL we provided to them. Fine, I can use LWP::UserAgent to post the initial request, and set the referrer to whatever I want. I was still missing one piece though: In the second request (the auto-post form which is sent to the browser), the referrer must exactly match the referrer that I sent in the first request. Of course, NSC’s documentation conveniently doesn’t mention this.

Example: If I give them ‘http://devel.umbc.edu’ as the development URL, then my initial referrer must begin with that. I can send ‘http://devel.umbc.edu/cgi-bin/blah.pl’ and it will accept that. On the second request, I have to send it exact the same referrer string as I did in the first request, or it won’t accept it, even if it matches the development URL. Got that?

I probably made this harder on myself than it needed to be, by trying to get cute and fudge the referrer header from a URL other than the development URL I gave them. But it’s a bit annoying to have to jump through these hoops to make the thing happy about the referrer, given that it’s a really bad idea to rely on the referrer for any kind of security purposes in the first place.

Oh well, let’s hope I don’t have to do this again for awhile.

Initial Impressions of Sunbird

Just spent a bit of time playing around with Sunbird. I would call it “not quite ready for prime time.” It’s got a nice interface. I was able to subscribe to my iCal-published calendar without a hitch. It can auto-reload remote calendars on startup. I can also create local calendars and publish them successfully. However, the publishing support seems to be kind of a work-in-progress. When I publish a new calendar, it doesn’t remember its URL. I have to re-type it every time I publish. When I try to modify a remote calendar that I subscribed to, it crashes. There appears to be no support for auto-publishing as entries are made.

I’ll keep an eye on Sunbird, but it looks like it’s not quite ready for day-in day-out use. I expect it’ll get there eventually though.

More on publishing and sharing calendars

I did some more research on calendar sharing over the long weekend. The first thing I tried was using iCal to publish a calendar to my new WebDAV server. It worked without a hitch. However, all this does is upload the .ics file to the web directory. I can then subscribe to the calendar with another iCalendar-capable client, but there’s no built-in facility to view the calendar with a web browser, as I had originally thought (apparently this is a feature of Apple’s .mac subscription service).

A little bit more searching turned up a nifty app called PHP iCalendar. This does exactly what I was looking for: install it, point it at a directory with iCalendar files, and it displays everything on my web browser. Installation was as easy as putting mod_php4 on my web server, untarring the distribution, and editing the config file. It works great and the interface is nice. So, with that I now have read-only access to my calendar from anywhere on the net.

I also wondered if there was a full-featured iCalendar-compatible calendar app out there for Linux. I found the Mozilla Calendar and Sunbird Project. Mozilla Calendar is a plugin for Firefox or Thunderbird, while Sunbird is a standalone app. Both share the same codebase, features and bugs. This is alpha pre-release software, but it’s being actively developed and appears to be the most promising solution. It works with Windows, Mac and Linux, meaning I could hypothetically use it on all three platforms (although on the Mac, I currently need iCal for Palm synchronization). I’ve downloaded this and verified that it seems to work, but I haven’t played with it extensively.

All this gives rise to the question: Could I keep the “official” version of my calendar on my web server, access it with Sunbird, iCal and/or PHP iCalendar, and have all my changes automatically sync up with the web copy? That is definitely the question, and if it’s doable, I think it’s the optimum solution. If I can track down a working version of the Oracle Calendar SDK for Linux, I could even automate the integration with Oracle Calendar.

I know I can configure iCal to automatically publish to the web whenever I make a change to it (although I’ve never tried it so I’m not sure how it works in practice). I think Sunbird has a similar capability. Again, it remains to be seen how well this will actually work. I don’t have the Mac with me today, so I’ll play around with Sunbird on my Linux box, and see what it can do.

Can’t forget the Palm

The final piece of the puzzle is the Palm. I need to get the Palm syncing up with iCal. Apple provides an app called iSync which can do this. However, I tried it and was not impressed. iSync is an intriguing application, but it seems mainly geared towards cell phone type devices. Its support for the Palm seems to be kind of an afterthought. It works, but it doesn’t support a lot of the features on the Palm, particularly with the address book

A more promising looking solution is a commercial product called The Missing Sync. This is a total replacement for Palm Desktop/HotSync that is specifically geared to the Mac. It reportedly integrates very well with iCal and the Mac’s address book. It also includes a lot of nifty extras (internet connection sharing, expansion card mounting, etc).

Missing Sync costs $40, but based on what I read, it’s worth the price. There are two things about it that I would consider negatives:

  1. No trial/demo version available I’m willing to look beyond this because of the overwhelmingly positive reviews I’ve read about the product. A lot of people, myself included, like try-before-you-buy, particularly with software. They could probably sell more copies of this if they had a demo version. Let people try it, wonder how they got along without it, and they’ll buy it.
  2. Can only be installed on 2 macs Not an issue for me now, as I have only 1 mac, but who knows how many I’ll accumulate over the years? I really hate it when companies cripple their products like this. Still, I’ll live with this despite my philosophical problem with it. If it ever becomes an issue for me, I’ll complain loudly.

Issues aside, it looks like a high quality product, and unless I change my mind about using iCal, I’ll probably give this a go.

WebDAV Server

No secret now that I’m looking to iCal as my calendaring app of choice, so I’m continuing to put it through its paces. Next up is to try out its publishing feature. Supposedly, I can publish the calendar to any WebDAV server and then access it using a web browser (or subscribe to it using another iCal client).

I didn’t have a WebDAV server sitting around anywhere, so I went ahead and set one up using Apache 2.0.54 with mod_dav/mod_dav_fs. It was pretty easy, basically just load the DAV modules and enable DAV in the appropriate directory (with authentication of course). I can access it with a command-line tool called cadaver, which provides FTP-like access to the WebDAV space. So it looks like everything is good to go on the server side.

I don’t have my Mac with me today, so I’ll try uploading a calendar tonight after I get home.

More on exporting and importing

Playing around some more with Oracle Calendar’s export feature, and the resulting data file, to try to get my all-day events to show up properly in iCal. First off, Oracle Calendar can actually export into two different formats, vCal and iCalendar. iCalendar is actually a newer spec that grew out of vCal. Apple’s iCal can import either format. This is significant, because when I export in iCalendar format, the all-day events get imported into iCal properly. When I use vCal, they show up as one-minute meetings starting at midnight.

So… let’s just use iCalendar instead of vCal, right? Wrong. First off, I have no problem exporting a small date range (say, a month) in iCalendar format. When I try to export a 3-year date range, Oracle Calendar goes off into la-la land and never comes back. OK, well maybe it eventually comes back, but I gave up after about 20 minutes and killed it. This happened with multiple clients on multiple machines, so it’s not just the Mac. It’s either having a problem with one of my entries (which would be a bug), or it just plain takes forever. Either way, this makes it unusable for my purposes.

Not only that, when I managed to successfully export an iCalendar file, iCal wouldn’t import it. It imports the first entry and then seems to quit. Not sure which app is at fault here, but don’t really care.

That leaves me with vCal. Oracle Calendar will export three years of data in vCal format in about a minute or so. Not exactly lightning fast, but usable. Now, how to get iCal to treat my events as events and not 1-minute meetings?

Here’s the deal: the vCal spec uses the DTSTART and DTEND fields to identify meeting start and end times, respectively. If you leave out DTEND, it treats the entry as an untimed event. And it works. If I edit the vCal file and remove DTEND from an untimed event, iCal imports it properly. Here’s a Perl script to do it:

#!/usr/bin/perl
#
while (<>) {
    print;
    if (/^BEGIN:VEVENT/) {
        printEvent();
    }
}
exit 0;


sub printEvent() {
    my @evData = ();
    my $miscFlag = undef;
    while (<>) {
        push(@evData, $_);
        if (/^CATEGORIES:MISCELLANEOUS/) {
           nbsp;$miscFlag = 1;
        }
        elsif (/^END:VEVENT/) {
            foreach (@evData) {
                print unless (/^DTEND:/ && $miscFlag);
            }
            last;
        }
    }
  }

iCal seems promising

iCal is the calendaring app that Apple includes with MacOS X. It comes bundled with another app, iSync, which handles synchronization with the Palm. On MacOS 10.4 (Tiger), iCal’s app data lives under $HOME/Library/Application Support/iCal. It appears that the calendar data itself is stored in iCal format (.ics), while the metadata is stored in XML format. Separate calendar categories are stored in separate files, unlike the single monolithic binary file Palm Desktop uses. All data files appear to be flat ASCII. That means that, if I wanted to, I could easily maniuplate the files using Perl or somesuch. However, not knowing the interrelationships between the various files, I feel like I should probably avoid this. So, let’s look at the app and see what kind of import features it offers.

iCal calls calendar categories “Calendars”. When I go to import a vCal file, it asks me to select a destination calendar. I can pick an existing calendar, or tell it to create a new one. In the latter case, iCal creates a calendar with the same name as the vCal file (minus extension). It appears to ignore the category data in the vCal file, which for my purposes is good.

To delete a calendar, I just select it in the left pane and press the “Delete” key. It warns me that all entries under that calendar will be deleted, then does it. Everything is undoable.

This seems to work great. The import process is smooth. It creates a new calendar category for me automatically. Deleting the data (prior to a subsequent import) requires only one keystroke. It appears to import locations and attendees too. The only drawback is the handling of “daily notes” and “day events” from Oracle Calendar. However, this problem is on the export side. Ideally I’d like these to come in as “all day” events rather than 12:00am-12:01am meetings. If I can’t configure things this way on the export side, I should be able to write a perl script to modify the vCal import file.

iCal also includes sharing features. I can subscribe to a calendar on the internet, and also publish my calendar to a WebDAV server. I haven’t researched this much, but I hopefully I can use this to access my calendar from anywhere on the net (read: my desktop machine at work).

Palm Desktop – The simple solution?

Palm Desktop is, of course, the app that Palm produces to do desktop integration with its handhelds. All in all it’s not a bad little app suite, and you certainly can’t beat it for integration with the Palm. It supports all of the Palm’s features and synchronization (via the bundled HotSync manager) is flawless.

Palm Desktop is not server based, so all of your data lives on the machine where you have it installed (which, by extension, is the same place you synchronize the handheld). There doesn’t appear to be a way to share the calendar data on the internet. That means I’ll need either my laptop or my Palm handy to view my calendar. A drawback, but not a show stopper.

On the Mac, the Palm Desktop calendar data appears to live in a single file, $HOME/Documents/Palm/Users/Paul Riddle/User Data. This file contains binary data presumably in some database format that Palm Desktop can read. The ‘file’ utility reports it as a “DBase 3 data file”. If I run ‘strings’ on it I can see entries from both my calendar and my address book. This doesn’t look like something I want to muck with outside of the Palm Desktop app. So, we need to see if Palm Desktop can do bulk deletion of imported entries.

Looks like we’re out of luck with the bulk deletion. In an older version of Palm Desktop (on my Windows machine), it works. I can create a custom category, create some entries within that category, then delete the category. The software gives me the option to delete all entries within the category, or move them to the “Unfiled” category. However, this doesn’t work with the newer version of Palm Desktop on the Mac. It looks like they’ve added the ability to have multiple categories assigned to an event. If I delete a category, the events in that category stick around, even if no other categories are assigned to the event in question. Kind of sucks that they would cripple that functionality. But it pretty much means that I can’t use Palm Desktop for what I want to do. Nevertheless, I’ll continue to evaluate it a bit and see what I find.

Palm Desktop also seems to have problems with “day events” and “daily notes” under Oracle Calendar. These events are actually untimed, so they should be imported as “untimed” (or “all day”) events. However, Palm Desktop imports them as timed events and sets the start/end times to 12:00am and 12:01am. I’m not sure if this is the fault of Palm Desktop or the Oracle Calendar export.

OK, the “day event” handling appears to be a “feature” of the Oracle Calendar export. The 12:00am-12:01am time range is in the vCal import file, so Palm Desktop is just recording it as it sees it. The vCal file also includes Category information, and Palm Desktop is reading that and creating new categories as it sees them. The Oracle Calendar entries show up with categories of “APPOINTMENT”, “HOLIDAY”, “MISCELLANEOUS”, etc. If a bulk delete were possible, I’d need to delete all of these categories before re-importing; a bit of a pain. Alternatively, I could massage the vCal import file to put everything into a single category.

To summarize: Palm Desktop uses a (possibly proprietary) binary file format to store entries on the host computer. There’s no out-of-box way to share calendars among multiple computers (although apparently there are third-party products that will do this). Bulk deletion of entries does not appear to be easily possible. Even if it was possible, the import process would require multiple steps to recategorize imported entries into a single category.

Given all this, it looks like Palm Desktop is not going to be the answer for me. A shame, because I’m sure I won’t find anything else as tightly integrated with the Palm. Next up, I’ll check out iCal.

Duplicates duplicates duplicates…

Earlier I mentioned iCal and Palm Desktop as two possible solutions for my desktop calendar. To integrate with Oracle Calendar, both of these apps would require me to manually import the Oracle Calendar data in vCal format. Oracle Calendar’s export function gives me several options for choosing what data gets exported. I can export entries within a specific date range (from 1/31/2004 through 12/31/2006 for example), I can export a relative date range (1 year in the past through 2 years in the future), or I can export the entire enchilada (data that dates back to early 1999, in my case). In the past, I’ve gone with a relative date range starting 1 year in the past and going 2 years in the future. Reason being, exporting the whole thing is kind of slow and there’s no bounds on the size of the data set being exported, so it will keep getting slower and slower the more data I accumulate in Oracle Calendar.

Now, with a 3-year date window, it’s a given that I’ll be exporting a lot of the same data over and over, and importing the duplicate data into iCal or Palm Desktop. Neither of these apps has what I would call an “intelligent” import feature. That is, if I import data more than once, it will show up multiple times in the application. This isn’t a show stopper, but duplicates are undesirable and we need a way to prevent this from happening.

One way to do this would be to just import the data, then cobble something up (in Perl, perhaps) to strip out the duplicate entries. This would require the ability to manipulate the calendar app’s data outside of the calendar app itself, so presumably it would rely on the app using some kind of standard, documented format to store its data. My surfing reveals that someone has done this with iCal, so this may be possible if iCal is my solution. Dunno about Palm Desktop, but intend to find out.

Another way to do it would be to simply delete all of the previously imported data out of the calendar, than re-import. If I went this route, I would need a way to differentiate imported data from other data inside the calendar, so I can delete just the imported data and nothing else. Then, I’d need the ability to do a bulk delete, ether from within the app’s GUI, or (once again) by manipulating the app’s data with an external script.

Both of these methods will require a relatively robust Palm sync app, so that deleted entries are picked up and properly deleted off the Palm. Otherwise we’ll still end up with duplicate entries on the Palm.

Next I’ll look at Palm Desktop and iCal more closely, to see if either will work as part of my evil plan.