Well, it appears I spoke too soon about backwards compatibility with the new Oracle Collaboration Suite and the old Calendar API that I’m using for my iCal downloader stuff. The first bad sign happened a couple weeks ago, when I noticed that a bunch of my Oracle Calendar entries had mysteriously disappeared from my iCal subscription. After investigating, it turned out that an entry in September 2007 was screwing it up. If I downloaded a date range up to but not including that date, it worked fine, but as soon as I included that date, about 50% of my entries disappeared. Hmmmm, not good. Then, today, I noticed that my iCal subscription was about a week out of date. When I went to run the download job manually, it bombed out with the API error code CAPI_STAT_DATA_ICAL_COMPVALUE. The docs have the following description for this code: “There was a problem with what a component contained.” Thanks guys, that’s really helpful.

So anyhow, it looks like I’m back to square one with the calendar stuff. When I get the time, I’ll rewrite it to use the newer API. Alternatively, maybe OCS has a way to do this without having to write custom code. That would sure be nice. Unfortunately though, until I get around to this, I guess I’m stuck with no Oracle Calendar data on my Palm. Bummer.

Well, we finally upgraded our calendar server last night, from old-and-crusty Steltor CorporateTime to its successor, new-and-shiny Oracle Collaboration Suite. And, kudos to Oracle, as it looks like they’ve kept it backwards-compatible with the old CorporateTime API. That is, my homegrown OracleCalendar-to-iCalendar exporter thingy is still working. That’s nice, because I’ve come to depend on it, and this means that I won’t need to spend lots of time fixing it. Things are a little busy here right now, so if it had broken, it probably would have stayed broken for awhile.

Once things slow down, and I can revisit this, I bet I can make it work even better with the newer APIs now. Actually I may not need to use the APIs at all any more, as OCS supposedly supports CalDAV. In the meantime though, I’ll very happily continue to use the stuff I already have.

Oh, and a few days ago, I installed MediaWiki. I think I’m going to get a lot of use out of it. Read about my recent trials and tribulations with PHP iCalendar.

Party on…

I took another look at time zone data in iCalendar, to see if I could include time zone info in my downloaded Oracle Calendar data. This isn’t a super high priority thing, as the only thing that it affects is how my Palm displays my meetings. The meetings are displayed correctly, but they include a UTC time identifier at the end. Not a huge deal, but if I can get rid of it easily, it’d be a win.

A quick Google search on “vtimezone” turns this helpful page up. Based on the info therein, I was able to craft a working VTIMEZONE section for US-Eastern, that includes daylight savings rules for both pre-2006 and post-2006. To wit:

BEGIN:VTIMEZONE
TZID:US-Eastern
LAST-MODIFIED:20060101T000000Z
BEGIN:STANDARD
DTSTART:19671029T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10;UNTIL=20061029T070000Z
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
END:STANDARD
BEGIN:STANDARD
DTSTART:20071104T020000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=11
TZOFFSETFROM:-0400 TZOFFSETTO:-0500
TZNAME:EST
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:19870405T020000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=4;UNTIL=20060402T070000Z
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:20070311T020000
RRULE:FREQ=YEARLY;BYDAY=2SU;BYMONTH=3
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
END:VTIMEZONE

Now, this is great, but the problem remains that all of the times I pull out of Oracle Calendar are in UTC. So I guess if I want to take advantage of this, I’ll need to manually convert all of these times from UTC to US-Eastern. This seems like too much work for too little gain, so I’m just going to leave it the way it is.

I haven’t done much with my calendar stuff recently, for a couple of reasons; one being that work has been crazy and I haven’t had much time to hack on it, and another being that it’s all working without a hitch. I don’t even have to think about the Oracle Calendar stuff; my cron job auto-downloads it several times a day, and all I need to do is fire up iCal, refresh all calendars, and sync to the Palm. I see that Missing Sync version 5.0.3 final has been released, but beta6 is working fine for me so I’m not in a fired-up hurry to upgrade. I’m starting to make more and more use of iCal’s concept of multiple calendars; I already have a calendar with various family birthdays and anniversaries, and I’m going to do one with holidays. Holidays will be a good test of how the Palm handles repeating events, and how well Missing Sync translates repeating events from the iCalendar files to the Palm. I’ve also found myself using PHP iCalendar quite a bit when I don’t have my Mac handy. All in all, I’m really happy with how this whole thing has turned out.

What’s next? More features, of course. I’d really like to have attendee data for some of my meetings. Because of the performance issues involved with downloading attendees, I’ll have to do two separate downloads (a large range without attendee data, and a small range with attendees) and merge the two together. This could create some problems because of the way I’m munging UIDs for repeating events. Since I’m just appending ascending numbers to create unique UIDs, there’s the chance that the same event could get a different UID depending on the date range that’s being downloaded. I think before I try to do anything fancy, I need to rethink the way I’m assigning UIDs to events. This could present an interesting challenge. Anyhow, more on that later.

The Holy Grail of this project, of course, would be to have two-way syncing with Oracle Calendar, where changes I make on the Palm or in iCal will get back-propagated to Oracle Calendar. However, this increases the complexity of the project quite a bit, and I doubt I’ll ever go there. I don’t really miss the functionality all that much anyhow.

Today I gave the Mark/Space iCal sync conduit a really good workout, and it came through it without missing a beat. First off, I added the LOCATION field to the Oracle Calendar download. I’m not sure why I wasn’t pulling it down originally, I guess it was just an oversight. As a result, I now have location data for my meetings, and a lot of existing entries now have the additional field.

Next, I created a new calendar that has a bunch of repeating events (various birthdays and anniversary dates). Then I deleted all of the birthdays and anniversarys that I had stored in Oracle Calendar. Finally, I added a couple new events via the Palm.

In short: Add a new iCalendar field to several dozen existing events, add a new calendar consisting of repeating events, delete a dozen or so existing events, and copy a couple events from the Palm to the Mac, all in one sync. The Mark/Space conduit handled everything without a hitch. The deletions propagated, the new fields were added, and the new calendar was added with the proper recurrence rules.

This is really the way things are supposed to work when you pay for software: It should work, work well, and work reliably, and if it doesn’t, the company should stand behind the product and either make it work, or refund your money. So far, it looks like the $40 for Missing Sync was money well spent.

This also makes me curious about event recurrences. The iCalendar spec allows for some pretty fancy recurrence rules. I’m not sure if the Palm Date Book allows that much flexibility. If I were to define a really fancy recurrence (say, an event that happens the second tuesday of every month except March, or whatever) I wonder how that would propagate to the Palm. One of these days I’ll have to try it.

Incidentally, I created my birthday/anniversary calendar from scratch with a text editor. Then I copied it to my web server and subscribed to it with iCal. It worked fine, and seems to be a good way to handle relatively static calendars like this. The only requirement is that each event needs to have a unique UID. I used UIDs of the form:

YYYYMMDDTHHMMSSZ-#@concerto.ucs.umbc.edu

Where # is an ascending number. For the date stamp, I used the approximate time that I created the file.

I’m now up and running with the automatic Oracle Calendar export stuff. The other day I tested the Palm sync stuff out, to make sure event deletions were propagating properly to the Palm. Initially, they weren’t. However, after I followed the instructions that I got from Mark/Space support, everything worked fine. These instructions are worth repeating here, as they may come in handy down the road:

Go to Missing Sync and hold the ‘option’ key down while you double click on the ‘Mark/Space Events’ conduit. When the settings window appears, click on the ‘Advanced Options’ button and then click on the ‘Unregister Sync Client’ button.

Then sync, when you do, you will get a dialog box with an orange iSync icon on it. Check the box to erase the device, then click the ‘Allow’ button.

This appears to erase all the existing calendar data on the Palm, and download totally fresh data from iCal. In any event, upgrading to 5.0.3b6 and then following these instructions solved my sync issues.

The next step was to automate the export of the Oracle Calendar data. I did this with a cron script. Right now I have the script run every day at 10am, 1pm, 4pm, and 7pm. It connects to the calendar server, downloads updated data, and updates the iCalendar file on the web server. (This runs on my Linux box at work, concerto.ucs.umbc.edu, as user www-data.)

So far, this appears to be working fine. I plan on keeping an eye on it for awhile to see if there are any issues. With this piece working, I can start to focus on improvements. Here’s the current wish list:

• Add timezone data to the iCalendar file
• Download attendee data for a small window of time (maybe 1 month or so starting from current date). Don’t want attendee data for everything, as it makes the iCalendar download take too long.
• Do I want to treat “declined” events any differently from “accepted”? Right now, there’s no differentiation between the two in the downloaded calendar. Addressing this would mean keying off the STATUS field somehow.
• Split different types of events into separate calendars. Various event types include “Meetings”, “Daily Notes”, “Day Events”, and “Holidays”. I might also want to separate out “Daily Notes” and “Day Events” that are created by users other than myself; this might be a way of handling the “accepted/declined” issue.
• Revisit handling of alarms, if necessary.

A few of these will require tweaks to the actual download process, and the others involve rewriting the iCalendar output in various ways.

I haven’t had much time to work on the calendar thing lately due to the holidays. That will probably continue into the early part of January, with work shaping up to be pretty busy during this time. However, a week or so ago I went ahead and sync’d my published Oracle Calendar data to my Palm. I encountered two issues: #1, The times displayed on the Palm are kinda wonky due to the iCalendar file having times specified in UTC instead of US/Eastern. iCal shows the events OK. The Palm shows the events at the correct times, but the events are displayed with the UTC times appended. Example: “Big meeting (2:30pm GMT)”. This isn’t the end of the world, but I guess if I want to fix it I’ll need to add timezone data to the iCalendar file. I was hoping to avoid this step as it entails computing start and end times for Standard and Daylight time. Oh well.

The second problem is a bit more troubling, and I’ve actually contacted Mark/Space support about it. When I delete an event from the published calendar and then re-sync, the event is not deleted from the Palm. I can duplicate the problem with a very simple published calendar with only a couple events, so this is not a problem with the specific calendar I’m using.. it appears to be more general. I tried hard-resetting the Palm and starting over, thinking maybe it had gotten confused after all my previous mucking, but that didn’t accomplish anything (well, OK, it did accomplish something… it happily blew away my address book and all of my TODO events on both the Palm and the Mac. Joy).

I did hear back from Mark/Space regarding this issue. They suggested trying the latest beta release, and if that didn’t work they gave me a list of steps to follow that will hopefully fix it. I went to download the beta, and unfortunately, it looks like it does not work on my Tungsten E2 (there’s a message there to that effect). I haven’t yet gotten around to trying their other suggested fix. I plan on trying that within the next couple of days, so we’ll see how it goes.

Update 29 Dec 2005: I went back to the Mark/Space testing site today and found that they had posted a new beta, 5.0.3b6. I downloaded and installed it, and it seems to fix the problem. I haven’t pounded on it yet, but it worked for my simple test calendar. I’m optimistic that it’ll work with my Oracle Calendar data. Will test that out shortly.

It turns out that Daily Notes, Day Events, and Holidays all get the same treatment from the CAPI export process, so I need to rewrite the iCalendar output for all of them. Instead of using a DURATION to these events, I ended up just removing DTEND. Thus we end up with an event with DTSTART but no DTEND, which iCalendar defines as an event that takes up no time. That’s pretty much accurate, except in the case of Day Events, which technically take up all day. Unfortunately, in Oracle Calendar, some people put entries in as Day Events when they really should be Daily Notes. For that reason I’m not quite decided yet as to whether I should put DURATION in for Day Events. In any case, I’ve fixed the problem, and everything shows up in PHP iCalendar. For now I’ll just leave DURATION out, until I change my mind.

One thing I might consider, is splitting the four Oracle Calendar categories (appointment, daily note, day event, holiday) into separate calendars, so I can differentiate the various events more easily in iCal and on the Palm. It seems like a good idea, but will require some extra work.