First the good news… we’ve decided to go to uPortal 2.5 for the fall, and switch from the buggy Aggregated Layout Manager (ALM) to the purportedly-much-better Distributed Layout Manager (DLM). I’ll be counting the days until ALM is history.

With that in mind, I pulled down a fresh copy of uPortal 2.5.2 and created a new CVS repository for it. Except CVS didn’t import the whole thing. I tried again: same results. It was omitting two directories from the distribution, both named core.

RTFM RTFM… Turns out that by default, CVS excludes certain filenames that it thinks don’t belong in repositories. ‘Core’ is among the blacklisted filenames, presumably because CVS is assuming that anything named ‘core’ is a UNIX core dump file. Except that these aren’t files, they’re directories. Hel-LO…!

Anyhow… here’s the command I ended up using to create the repository:

cvs import -m 'imported 2.5.2 source' -I '\!' portal/framework uPortal uPortal_rel-2-5-2

The -I ! option tells it not to ignore any filenames.

With that out of the way, onward we march to the brave new world of 2.5.x and DLM.

I’ve decided to work a little harder to make myUMBC work for more users. Of course, the ulterior motive is to save work for myself and the help desk. Actually, sometimes my entire job seems geared towards doing things to save myself work. I guess once I’m completely successful, I’ll be able to just sit around my office and surf porn all day long. But anyhow..

We really need a way to make myUMBC work for people without a SSN (soon to be HP-ID) attribute in LDAP. Background on this problem is here. These people are able to log into the portal, but since we can’t look up an identifier for them, we can’t authenticate them to the legacy portal code. As a result, they can’t properly render any of the web proxy channels served by the legacy portal server. These channels handle all of myUMBC’s SIS related functionality (IOW, most of what you currently would want to do in myUMBC), so in effect, this leaves these users dead in the water.

[More:]

Now, unless we know the user’s HP-ID, the user still won’t be able to do anything really useful in the portal. However, we’d still like for them to be able to navigate through the portal and see the channels. To fix the user “for real”, we need to do one of the following:

1. Manually query SIS to see if the user has an HP-ID. If so, check LDAP to see if the user has multiple entries, and merge the entries if necessary. Otherwise, manually add the user’s HP-ID to LDAP, so we can map the username to the HP-ID. If no HP-ID in SIS, instruct the user to contact the registrar’s office and request one.
2. Do nothing, and hope the problem resolves itself (might just have to wait for the user’s info to percolate through the various systems of record).

What we’d ideally like to do, is get the portal to do the work for us. If a user logs in and we can’t find an HP-ID for them, we need to get in their face and tell them to contact the registrar to correct the problem (duplicate LDAP entries are a difficult issue, though.. I’m hoping that some of our problems with them will go away after we finish with SSN remediation). But in the meantime, we should make portal navgation work for the user, so they can at least get to functionality that doesn’t require a “real” HP-ID.

The solution I came up with, was to assign the user a “fake” HP-ID on the fly when they initially log in. We can then use that as a primary key for the legacy session management table (auxil.www_sessions), and the user will be able to render the legacy web proxy channels.

The basic logic for getting a session table key now looks like this:

1. Query auxil.myumbc_user_preferences table for an SSN (HP-ID) that matches the user’s UMBC username. If successful, use that. This allows us to map the user’s LDAP SSN to the SIS SSN when the two differ, which happens a lot when the user’s SSN changes and the change hasn’t percolated to one or the other system. Hopefully, we won’t need this step after SSN remediation, because the HP-ID is never supposed to change.
2. Query LDAP for the user’s SSN (HP-ID). If successful, use that.
3. If we couldn’t find the SSN that way, query the legacy session table for a temp SSN matching the username (indicating that we’ve already created them a temp identifier). If we find one, use it.
4. If all else fails, generate the user a temp identifier within a specified range (currently 995000000-995999999) and use that.

The user is thus able to render legacy channels right away. And the nice thing here, is that once the user gets a “real” SSN (HP-ID) in our system, it’ll automatically stop using the temporary one without any intervention on our part.

All of this magic happens in edu.umbc.uportal.UmbcPersonFactory, which currently lives in the main uPortal source tree.

I have this in production now, and I’ll be keeping an eye on it to see what problems develop. If it seems to be working, we’ll need to see about somehow hassling the user to get the problem rectified. This might be something to do via the big, fancy, vaporware alerts system we’re supposedly launching this fall.

The fun never ends..

We’re one step closer to the happy world of SSN Remediation. Our development SIS database is now using the HP-ID, which is the new primary key we’ve designated to replace the SSN in SIS. HP-IDs look distinctly familiar: They’re 9-digit numbers just like SSNs. That way, we cleverly avoid having to rewrite most of the SIS code to deal with the new IDs.

On the myUMBC side of things, I just need to change the code to look for the HP-ID LDAP attribute, instead of socialSecurityNumber. Which brings up a temporary problem: HP-IDs aren’t in LDAP yet. So how do I look them up? Well, right now, I’m still querying the SSN in LDAP, then bouncing that against a super-top-secret SSN-to-HP-ID translation table in SIS. That does the job nicely for testing purposes, but I must stress that it’s only a temporary measure.

Next up, I’m doing some hacking on the class list code in the Perl myUMBC codebase, to add in some new features (part of the big drive to improve student retention rates). I’m looking at the class list code, and I’m thinking I could really speed it up if I changed it to use prepared statements with bind variables, but the code is oh-so not set up for that. I may take a closer look at it tomorrow to see how hard it would be to do.

[More:]

The class list code, as it currently is, works like this:

foreach ( student in the class )
   create perl object with student info
   tell perl object to fetch info from DB
   display student info in table
end

The student info object it creates, has all of the SQL necessary to pull the student’s data out of SIS. It’s doing this stuff over and over, which is where we could benefit from prepared statements. To do it, though, we’d need to create the prepared statements ahead of time (outside the loop), then pass them to each student object that we create. I don’t think it’d be really hard to do, just need to figure out the best way to do it.

Followup… tried this out, seems to work. The challenge is going to be figuring out where to create all the prepared statements, and how to manage them, without obfuscating the code even more than it already is. Seems like it might make sense to do it as part of the db_connect and db_disconnect routines… will mess with it more tomorrow.

Actually, I think I’ll go with a slight variation on that theme. Rather than creating all the prepared statements when I connect, I’ll modify my db_prepare routine to maintain a hash of statement handles. The first time it handles a particular statement, it’ll prepare it and cache the statement handle. Then on subsequent calls, it’ll just retrieve the previously cached handle. When I disconnect, I’ll just walk through the hash and call finish on all of them. Should work great, with a minimum amount of surgery on the code.

I just wanted to take a minute to plug Standardista Table Sorting, a nifty set of Javascript routines that automagically sorts rows in HTML tables. I was recently asked to add row sorting to myUMBC’s class list function, and my first thought was to do it in Javascript, which would give the end user instant results as well as reducing server and database overhead — a double win. I started out writing something myself, then decided to poke around and see if someone had already written something. I found the Standardista stuff, and it works great.

[More:]

Getting this working was fairly straightforward. I had to modify myUMBC’s Perl HTML ‘table’ object so that it sets the headings apart from the body with <thead> and <tbody> tags. Then, I just added <script> blocks to pull in the Standardista Javascript code, and gave my tables the sortable class.

At first, it didn’t work. It turns out the Standardista stuff doesn’t like any additional markup (text decoration, font tags, etc) within the heading tags. Once I got rid of the markup, it worked.

I also had to fiddle a bit to get the sorting to play nicely with my striped rows (alternating white and light gray). The Standardista code includes support for striping, but I had to change the way I was applying CSS to my table (move the class attribute from the <td> elements to the <tr> elements), and I also tweaked the Standardista code to use my CSS attribute (myumbcLightGray) instead of its own (odd). With uPortal, I have to be sensitive to CSS namespace collision issues, and odd was a bit too generic. Once I made these adjustments, my striping was preserved after sorting.

This is a really cool package and I highly recommend it for any table-sorting application.

This June, we’re finally, finally going to stop using SSN as a primary key at UMBC, and frankly, the big day can’t come soon enough. And it’s not for the reasons you’d think. Yeah, there are a lot of privacy and liability issues at stake, but putting that aside, it causes us nothing but headaches. For starters, a primary key is supposed to be a unique, non-changing entity. SSN is neither of these. This is particularly true for international students. Many of these students arrive at UMBC without a permanent, government-issued SSN. We issue these students a temporary, bogus SSN, so we can get them into our various systems (SIS, HR, etc). Then, sometime during their stay at UMBC, they get a permanent SSN. And, depending on how the change percolates through our various systems of record, this wreaks unholy havoc on the student’s myUMBC account.

Nothing will illustrate this better than a real-life example. Read on for the gory details.

[More:]

Today I looked at a grad student, who we’ll call Larry (names changed to protect the innocent). When Larry logged into myUMBC, he was only getting the Personal and Support tabs. Well, it turns out that Larry is an international student. He’s also a graduate assistant, which means he has an HR entry as well as a SIS entry. He’s in SIS under his temporary, bogus SSN, and he’s in HR under his real, permanent SSN. As a result, he has two entries in our LDAP directory, one for each data source. When I do an LDAP lookup on his account, it brings up the entry from HR, which is lacking a student affiliation. Thus, Larry doesn’t get any student content in myUMBC.

Now, the first thing to do here is to get Jason to merge the two LDAP entries together. In the meantime, I can grant Larry a temporary student affiliation, which will give him the tabs he needs to see. But, that’s still not enough. SIS still has his old, temporary SSN. When the portal goes to look up Larry’s class schedule, it’s going to use the new, correct SSN, and it’s not going to find any info there. So Larry gets an Academics tab full of blank channels.

The permanent solution here is to get the registrar to update Larry’s SIS records to reflect the new SSN. However, for reasons beyond my comprehension, there always seems to be an interminable amount of time between requesting this and actually seeing the change reflected in SIS. In the meantime, we have to do something so that Larry can access myUMBC. Which brings us to…. the SSN translation table.

For students like this, I have an SIS table (AUXIL.MYUMBC_USER_PREFERENCES) where I store the student’s username, and the student’s SSN as shown in SIS. When the student logs into myUMBC, the portal first checks this table. If it finds an entry matching the student’s username, it uses that SSN to override the one it gets from LDAP. So for Larry, I just plug his bogus SIS SSN into AUXIL.MYUMBC_USER_PREFERENCES, and presto, myUMBC magically starts working for him.

But, we’re still not done. In effect, we’ve created a time bomb. Because eventually, SIS is going to get updated to reflect the student’s real SSN. It could be a week from now, it could be a year from now. But when that happens, the overridden entry will still be out there, and it will continue to map the student to the old SSN. So at some point, Larry’s myUMBC account is going to break again. At that point, I’ll need to delete his entry from AUXIL.MYUMBC_USER_PREFERENCES, and then he’ll finally be fixed for good.

I’ve been dealing with this (ahem) CRAP (I’ll keep this PG-rated) for 6 years now, and once we finally stop using SSNs as primary keys, it’s finally going to end. Students will get an SIS ID, it’ll be unique, it’ll never change, these problems will be gone, and the world will be happier. At least in theory.

Just to document this for future reference.. In uPortal, the AUXIL.MYUMBC_USER_PREFERENCES lookup happens in the UMBC Person Factory (edu.umbc.uportal.UmbcPersonFactory), via the UMBC Local Connection Context (org.jasig.portal.security.UMBCLegacyLocalConnectionContext).

Followup.. in a sudden stroke of brilliance, I added a join against the SIS BADDR.MNAME table to my overridden SSN query. If there’s no BADDR.MNAME data for the SSN, it won’t use it. That way we automatically stop using overridden entries once they become obsolete. That should be a big help.

Urrgh. I added new functionality to the Perl myUMBC codebase today, and I had forgotten what a pain it was. You know things are bad when you wrote the code, and you have to spend an hour trying to recall how it works. Since it looks like the Perl stuff is going to be around for awhile, I figured I might as well document this process while it’s fresh on my mind. Maybe it’ll save me some time down the road.

Just to clarify… when I speak of myUMBC here, I’m just talking about the Perl stuff, not uPortal. With that in mind, here goes.

Background: Everything in myUMBC is database driven. Every link, table, dropdown menu, heading, etc. that you can see, click on, or otherwise manipulate, has a corresponding entry in a database table. All internal URLs in myUMBC are of the form

http://blah.blah/myumbc?function=foo

The function code tells myUMBC what the end user is currently doing. Most of myUMBC’s core functions (registration, etc.) actually have two function codes, one for the initial screen, and one for the results page. Every function code needs to have an entry in the database. Otherwise the user will get a “not authorized” type error. This is to keep users from entering arbitrary function codes and producing undefined results. The database table that contains all of this stuff is called BRCTL.PROD_PROG_DESC (don’t ask about the name).

To add to the confusion, myUMBC also has a built-in function mapping table, which maps shorter function names to possibly-longer Perl subroutine names. For example, the Financial Aid Inquiry function I added today has a short name faidinq which maps to a subroutine finaid_inquiry. There’s no real reason for doing this, other than shortening the URL.

To make a long story short, first, you need a PROD_PROG_DESC entry with PPD_LINK_TYPE set to “function” and PPD_URL set to the name of the Perl subroutine that builds the initial HTML (which could be a table, dropdown etc). Then, you need a second PROD_PROG_DESC entry for the results page. This one should have PPD_URL set to NULL, and PPD_FUNCTION_NAME should match the Perl subroutine that generates the results page.

That’s basically it… unfortunately, that probably didn’t make much sense unless you’re already intimately familiar with the myUMBC Perl codebase. One of these days I’ll improve and expand on this, to make it fit for general consumption. That’s why it’s in my blog and not the Syscore Wiki..

[More:]

6/16: Here’s how I made our new Student Parking Registration form come up:

INSERT INTO prod_prog_desc

    (ppd_prog_id, ppd_url, ppd_desc, ppd_add_date_time,

     ppd_add_user_id, ppd_link_type, ppd_restrict_access,

     ppd_auth_level, ppd_function_name, ppd_portal_disposition)

VALUES

    ('NEWPARK', '%2bParkingRegistrationForm',

     'Student Parking Registration (new)', sysdate, 'paulr',

     'myumbc', 'N', 'password', '+ParkingRegistrationForm', 'maximized')

Great to have notes to refer back to..

The new era has begun…

Today I hacked our legacy myUMBC portal code to generate XML output
for the registration eligibility function. The goal is to eventually
get away from the big, monolithic app and separate the business logic
from the presentation. Registration eligibility was a good place to
start, because (1) it’s a simple function with simple output; (2)
we’ve gotten a request to display registration eligibility within MAP,
so we needed to figure out a way to share the code; and (3)
registration eligibility still talks directly to the HP3000 mainframe,
which makes it impractical to do code sharing via Perl modules,
libraries, etc. So, this is what we came up with. It’s the first
step in coming up with a web-services type API for our back-end SIS
business logic.

The first thing I needed to deal with was the Perl code itself. The
legacy myUMBC code, while nice and modular, is very HTML-centric.
The business logic stuff is all mixed up with the code for HTML
generation. The code works by building data structures to represent
HTML objects, and the business logic bits are all built into those
data structures. Separating this stuff out is not really major
surgery, but it does require some work. I’ve managed to do this for
the registration eligibility code, but I still need to put a run-time
flag into the code to toggle XML vs. HTML output. Also, there will be
the ongoing challenge of dealing with error conditions and generating
valid XML in these cases, but we’re getting there.

On the actual XML front, the main challenge is defining what exactly
you want to output, coming up with a spec (DTD), and sticking to it.
This can require quite a bit of thought, and needs to be done before
any actual code is written. For registration eligibility, the DTD I
came up with is pretty simple:


<?xml version="1.0"?>
<!ELEMENT registration (eligibility)>
<!ELEMENT eligibility (status*)>
<!ATTLIST eligibility status (eligible|ineligible|registered|error) #REQUIRED>
<!ELEMENT status (description,extra?>
<!ELEMENT description (#CDATA)>
<!ELEMENT extra (#CDATA)>

And some sample XML…

<registration>
 <eligibility status="ineligible">
  <status type="academic">
   You are academically ineligible to register.
  </status>
  <extra>
   Your GPA is too sucky for you to continue
   attending the university.
   Please go away.
  </extra>
  <status type="financial">
   You are financially ineligibile to register.
  </status>
  <extra>
   Your parents didn't send us any money this
   semester. Please go away.
  </extra>
 </eligibility>
</registration>


Coupla notes here.. I used the generic tag name “registration” with
the idea that down the road, the object will contain other
registration related stuff besides eligibility. We’ll see if that
actually happens. And, although I do include text-based descriptions
of eligibility status, these are only intended to be guidelines.
Applications can use the descriptions as-is, or key off the “type”
attribute in the “status” tag, and display their own messages.

The code generates this XML using the same Perl objects I wrote to
construct HTML. It seems to work OK for generalized markup, and
guarantees that the output will be “clean”.

Next up is to see how this works when we pull it in as a uPortal
channel. To do this, we’ll need an XSLT stylesheet and we’ll need to
set up a “Generic XML/XSLT Transformation” type channel (rather than a
web proxy). We’ll need to use our custom local connection context
code to connect, and it remains to see how that will work with a
non-web proxy channel. Should be fun..