My Fun Day.

Today was lotsa fun. It started out with the National Student Clearinghouse. I decided to get a “real” development instance going where I could connect to them as a student, demo it to Academic Services, etc. I ended up wrestling with their stupid referrer-based security scheme again. I took my existing clearinghouse script, which was working fine, and added Webauth authentication to it. I figure I’ll have the script verify the user’s Webauth credentials, then do an LDAP query to get the student ID, then pass that to the remote site. That way, my local script will have some authentication built in. Well, that broke it. On the initial authentication attempt, Webauth adds a query parameter called WebAuthExtAction (which the client is supposed to decode, and use the result to set a cookie). Great, but that changes the HTTP Referrer string, which breaks the clearinghouse crap. Hey, but they changed their site so it actually tells you what’s going on now, rather than just booting you out. Have to at least give them props for that, it saved me some head-scratching. OK, first attempt at fixing this: I’ll check for a WebAuthExtAction parameter, and if it exists, I’ll append it to the initial referrer string that I send them. Nope, that makes the referrer string too long, and the clearinghouse code can’t deal with it. Second attempt: look for the WebAuthExtAction parameter, and if it’s there, redirect the browser back to the same script, omitting the parameter. Bloody convoluted, but it works. Fortunately, in production, we won’t have to deal with this, because the prod code will run from the same web server as the portal, and the user will always have valid creds when they come to the site. Aargh.

Then there was fun with myUMBC itself. In an attempt to speed things up on the myUMBC web server, I decided to redo the Webauth ticket-logout script that it was using, and make it part of the myUMBC app itself. That way, logouts will go to the FastCGI processes, reducing overhead (the script needs to connect to the database, among other things) and hopefully speeding the machine up. This actually worked OK eventually, but of course, it broke things at first. Turns out I was short-circuiting the FastCGI loop without resetting certain global variables, which of course is a big no-no. But, that was good for a few choice expletives.

When does Christmas break start again?


Attendee field definitely the culprit

See subject. I checked the API documentation, and it has a complete list of iCalendar attributes that the server returns. So, in my downloader code I just listed out each attribute except ATTENDEE. With that list of attributes, it took about 1 minute to download a year’s worth of data. When I added ATTENDEE in, the download pretty much ground to a halt.

So at any rate, it looks like I want to leave ATTENDEE out when doing my bulk downloads. That’s a bit of a bummer, though, because it means I won’t get attendee lists for meetings etc. It’d be useful to have that. What about this compromise: for a small window, say today through two weeks from today, I’ll export events with attendees. Then, for events outside that window, I’ll export events without attendees. I think that gives me the best compromise between performance and function.

The only other possible issue I can think of is, the UID field. Each event has a UID field that uniquely identifies that event. I’m hoping the UID stays consistent across multiple exports of the same data. I’m pretty sure the Palm sync stuff keys off the UID, so keeping the UID consistent should make the Palm sync process faster and more reliable. I did try exporting the same date range (all of 2005) twice, saving the results to a file, and comparing the files. The UIDs were the same both times, so that’s a good (although not conclusive) sign.

Next up: I’ll massage the data as necessary and try turning it into an iCal subscription.


More on iCalendar vs. vCalendar export

I used the Oracle Calendar GUI client to create iCalendar and vCalendar exports for the same date range. The formats are obviously different, but the data extracted is similar. Exceptions: The vCalendar export includes “TODO” items from Oracle Calendar, which I’ve never used and don’t care about. And, the iCalendar export includes email addresses of attendees. I wonder if that’s what’s slowing it down. If the email lookup process is time consuming, it will REALLY drag things down, because our calendar server includes a LOT of entries with many attendees.

Also, earlier I tried importing one of these iCalendar files into Apple’s iCal app, and had problems. It would import one entry and then stop. Upon closer examination, I think I see why this is happening. The exported iCalendar file encloses every single event inside a separate BEGIN:VCALENDAR / END:VCALENDAR block, complete with all the vCalendar headers, etc.! In other words, it’s not a valid iCalendar-format file at all. iCal probably aborts when it sees the first END:VCALENDAR, thinking it’s done.

Anyhow, back to my earlier point, I decided to test my theory that the email lookups are dragging down the iCalendar export. The API allows me to explicitly specify which iCalendar properties I want included in the exported file. So, I told it to include just a small set of properties (start date, end date, summary), to see how it would do. Yep, the export was much faster. Now, I just need to build up the full set of properties I need, omit the attendee info, and see how it does then. Unfortunately, there’s no way to say “I want every property except attendee info.” I either have to request every property, or request a specific list. So, I’ll need to look at the iCalendar spec, get a full list of properties, and explicitly list out the ones I want. Kind of an annoying limitation if you ask me.

The other question is, is there a way to get the vCalendar format through the API? Or, assuming the email lookup is the culprit, can I pull attendee info without doing the email lookup? Dunno, but I’m getting a little ahead of myself here.


Shedding the Schedule of Classes Page Albatross…

I’m working on redoing UMBC’s Online Schedule of Classes page. The current version is generated by a big, messy Perl script that reads the raw data uploaded from the HP3000, formats it, generates pages for all the individual disciplines, and then generates the top-level page (which contains links for each semester along with links to various informational pages). What’s the problem, you ask? Well, all of the HTML (except for a few PHP includes for headers, footers, and style info) is hardcoded into the Perl script. You can make modifications to the generated HTML pages, but they get immediately overwritten the next morning when the script runs to regenerate them. Any time someone needs a permanent change (which happens frequently, because the Academic Services folks are always tweaking their informational links, particularly during advance registration), it has to go through me. This is a hassle both for me, and for the folks who need the changes made.

The Perl script is old. It dates to 1996 or so. The model it uses is outdated. It needs to be rewritten so that the Academic Services folks can manage the content themselves. The whole thing is just begging to be rewritten in PHP or some other embedded language, but unfortunately I don’t have time (or staff) to sit down and make a major project out of this right now. So, for now I’ll settle for slow, incremental improvements.

My first tweak was to rework the script so that it uses PHP includes to read the auxiliary links for the top-level page. That way, the AS folks can manipulate auxiliary links themselves, and they will show up instantly (rather than having to wait until the script runs overnight). This is a quick win, and should eliminate the bulk of the busy-work I have been doing to support this. However, I’m not sure it’ll work perfectly, because the AS staff are using Dreamweaver to edit all of their HTML content, and I’m not sure if Dreamweaver can edit partial HTML files, or if it’ll try to add its own tags all over the place. As with everything around here, time will tell, and we’ll tweak things down the road as necessary.


Wrote an iCalendar downloader

Tonight I wrote my first app with the Calendar API. Basically I used the calendar.c demo code as a reference, and pulled out the pieces I needed to do a quick iCalendar download app. It worked without a hitch. With just a little more tweaking, I’ll have a command-line app I can use to fetch arbitrary date ranges in iCalendar format.

The caveat: It’s very slow with date ranges of more than just a few days. I guess that’s to be expected, given the performance I’ve seen with iCalendar downloads from the GUI client. It does eventually come back, though. A one-month range took a few minutes to complete; I hate to think how long three years would take!

Of course, I plan to run this unattended from cron, so in theory I don’t care how long it takes, even if it’s a couple hours. The question is, if I try to suck down a three-year range in one big chunk, will it fubar the calendar server? Obviously that would not be good. The other option would be to pick a manageable chunk, say a few months, and only download that much at a time. I could just run it in a loop until I get the full 3 years. Will have to play with this.

The other puzzler here is, the Oracle Calendar GUI calendar lets me download in either of 2 formats: iCalendar or something it calls “vCalendar”. I’m not sure of the exact difference, but vCalendar is a lot faster to download. What’s the difference between the two, and is there a way of doing the vCalendar download from the API? I wonder if there are certain iCalendar fields that take significantly longer to process. If the vCalendar download omits those, that could speed things up significantly.

I suppose the thing to do is generate both iCalendar and vCalendar dumps for a specific date range, and compare the differences between the two. That may shed some light on things.


Drilled my holes

I got kinda lazy yesterday, and didn’t get too much done on the bedroom fan wiring project. But, I did get one step closer to running a new wire from the basement to the attic. I got holes drilled in the top and bottom plates, and verified that both holes hit the same stud cavity. Fishing the wire should be pretty easy at this point.

Rules of thumb for anyone who wants to attempt this…

  1. Ensure that you own a ranch house. 🙂
  2. Tools of the trade: tape measure, stud finder, drill, 1/8″ x 12″ feeler bit, 5/8″ spade bit, flashlight.
  3. Find reference points in the basement and attic so that you can (somewhat accurately) pinpoint where to drill. Examples are: pipe penetrations, wire penetrations, ducts, etc.
  4. Locate the wall studs, and pick your drilling spots so that you don’t hit the top (or bottom) of a vertical wall stud.
  5. Measure, measure, measure! Can’t stress this enough.
  6. Drill a pilot hole first to make sure you hit an open stud cavity. I use a 1/8″ x 12″ bit for this. After drilling, leave the bit in the hole. Then go upstairs/downstairs and make sure the bit is not sticking through the ceiling/floor!
  7. Assuming everything looks good, use a 5/8″ spade bit to enlarge the hole. When drilling downwards from the attic, make sure the bit is TIGHT in the chuck.
  8. Once you have drilled both holes, put a flashlight over the hole in the attic. Adjust the light for a focused (not diffuse) beam. Go down to the basement. Turn the lights off, look up into the hole, and make sure you can see the beam.

Next: Let’s fish some wire..


Wiring fun

So, I’m doing a bit of rewiring in our spare bedroom, soon to become my 3-year-old’s new room. The previous owners put up a ceiling fan. Problem is, they didn’t bother to install an approved box. It’s one of those lovely nail-to-the-underside-of-the-studs metal bar jobbies. Rather than sit around twiddling my thumbs until the fan comes crashing down, I’m putting a new box in with a proper brace. Along the way, I’m replacing the wall switch, a fan/light control that has (putting it nicely) seen better days.

Well, nothing is ever that easy. The new fan control is slightly deeper than the old one. The power feeds through the wall box. There are 3 cables coming into the box: A 14-2 BX feed, a 14-3 BX carrying switched, unswitched and neutral to the ceiling box, and a 14-2 romex that someone added at a later date, to carry switched power to the fan (separate from the lights). So, we’ve got 3 cables and a grand total of 5 wirenuts (3 for the switch, 1 for neutral, and 1 to cap off the unused wire in the romex cable). Short story: I can’t fit everything in the box with the deeper switch.

I want to use this switch. I’ve got 2 other identical ones elsewhere in the house. So, looks like I need to do some rewiring.

I suppose I could replace the box. Nothing I haven’t done before. But, these boxes are holy hell to get out of the wall, without destroying the wall (plaster over gypsum board, or “rock lath” as it was called back in the day). No, I think I’ll rewire things and take the wall box out of the circuit, and make it a simple switch loop. Then I can cut it down to a single 14/3 cable going to the box, with a hot and two switched lines, 1 for the fan and 1 for the lights. Then it’ll fit.

So, I’ll just find a wire in the attic that’s ahead of the switch box in the circuit, cut it, and route it directly to the ceiling box, right? Sure, in theory. Problem is, there are no wires in the attic that are ahead of the switch box. The switch box IS the feed from the basement to the attic. Time for plan B. Looks like I’ll need to run a new wire from the basement to the attic. This will require me to break out my wire fishing skills. Now, having wired my own security system, I’ve done my share of fishing. However I’ve never run a wire straight from the basement, through a stud cavity, and into the attic (this is a rancher, BTW. Unfinished attic on top, drop-ceiling basement on bottom. If you’re into wiring, ranchers rock). Seems like it should be easy in theory. I guess I’ll find out. This should be fun! More later.


Legacy CorporateTime API

It appears that Oracle has totally dropped support for legacy CorporateTime servers in its recent SDK releases. I found some API documentation for the old CAPI_ functions, and as I mentioned before, some of them do appear to show up in the shared libraries. However, try as I might, I couldn’t reference any of the functions using the 10.1.1 libraries. The fact that none of them are prototyped in ctapi.h is also kind of telling. Looks like I won’t be getting any joy out of the 10.1.1 SDK. Could it be that Oracle dropped support so that more people would buy their new, expensive Collaboration Suite? Naaaah.

However, I won’t give up that easily. I have Google. I plugged CAPI_Connect into Google, sifted through a few pages of results, and lo and behold, I found a site that had the entire 2.0 CorporateTime SDK, circa 2002, for Linux/i686!!! After glancing over my shoulder to make sure Oracle’s lawyers weren’t looking, I grabbed it.

The SDK includes some example code, calendar.c, which connects to the server and allows interaction through a text-driven menu system. It built painlessly using the supplied Makefile. The next challenge was figuring out how to use it to log in. To make a long story short:

  1. First prompt: Hit return (no config file)
  2. Second prompt (name of calendar server): Enter
  3. Third prompt (ACE mechanism): Enter 2. It seems to work fine without specifying an ACE mechanism.
  4. Fourth prompt (user or Sysop): Enter 1 to authenticate as a user.
  5. Fifth prompt (username): This was the tricky part to figure out. Enter ?/S=lastname/G=firstname/. In my case I used ?/S=riddle/G=paul/. Obviously patterned after the LDAP attributes, surname and givenName.
  6. Sixth prompt: Enter password.

Once I figured out the proper incantations, it logged me into the calendar server! I tried out a few menu options. Data seems to come through in iCalendar format, which is what I want.

This is great, it looks like I’ll be able to automate the export/massage/publish process as I had originally hoped.

More later…


The ongoing Oracle Calendar saga

Well, my initial efforts to use the Oracle Calendar SDK have been a resounding failure. I got everything installed, and was able to compile a “hello world” type program. However it appears that our calendar server is too old and crusty to work with the newer set of API functions, all of which have the prefix CSDK_. Apparently I need to use the old CorporateTime API, with functions that begin with CAPI_. Now, when I run nm * in the SDK library directory, it turns up a bunch of these functions, but I’m not sure they’re actually what I want, and I don’t have any documentation or sample code. I’ll search around, and give this one more shot, but I’m beginning to resign myself to the fact that this is going to have to be a partially-manual process, at least until we upgrade our calendar server.

On the same front, I started looking at what it would take to create my own .ics file which I could subscribe to with iCal. I started by creating a dummy calendar in iCal, publishing it, copying the published .ics file into /tmp on my linux box, and then unpublishing the file in iCal (which deletes it off the WebDAV server). I then edited the .ics file in /tmp, changed the name, copied it file back to the web server, and successfully subscribed to it with iCal. Apple sticks a GUID field, among other things, into its published .ics files, and I’m not sure how I’d generate one from scratch, so I’ll just use this file as a template and I’ll be able to build a .ics file that iCal will be happy with. So, making progress here.


Oracle Calendar API Revisited

Well.. I’ve managed to track down the Oracle Calendar SDK libraries for Linux. They’re included with the Oracle Collaboration Suite distribution, which I had originally thought only included libraries for the Mac. I did a little more digging and found this post on Oracle’s Calendar SDK message board. It turns out I need to run the installer and let it do an “official” install of the stuff. A bit convoluted given that all I need is a couple libraries and include files, but if it works…

Here’s what I did:

  1. Downloaded the Oracle Collaboration Suite distribution (version 10g) from Oracle’s web site. It came as a 3.8-gig multipart compressed tar archive (thank God for Internet2)! After running the script to assemble the parts, I ended up with a file called OCS_101200.tar.gz.
  2. I’m not interested in everything in the tar file, just the calendar stuff. This is all in a directory called calendar_standalone. So, I untarred just that directory: tar xzvf OCS_101200.tar.gz calendar_standalone.
  3. I ran the installation script, runInstaller -ignoreSysPrereqs. The option tells the installer to keep running even though I’m not running one of Oracle’s pre-approved flavors of Linux.
  4. Walk through the menus, select the SDK, and follow the directions.

After doing this, I ended up with the SDK (along with 300 megs of other kruft) installed in my home directory under a directory called product.

Haven’t played around with it yet, but hopefully I can write something to do a simple event extract. If that works, I can write an automated script that extracts the data, massages it, and posts it as an iCal subscription. Stick it in cron to run nightly, and I’ve got my ultimate solution for Oracle Calendar integration. Wish me luck!