Enterprise

(LinuxWorld) -- Editor's note: The "I" in this scenario is that of a hapless systems consultant who didn't do his homework before setting off to meet the client. The Happy Valley Tax Authority, its staff and mandates, are fabrications but the situation presented, and the remedies offered, reflect the author's recent experience with real-world clients facing similar problems. This tax authority is imaginary, but the conditions, decisions, and outcomes described are broadly based on real events.

The Happy Valley Tax Authority was set up as a regional cooperative to administer tax programs for local governments along an 80-mile stretch of highway. At the time of incorporation none of the players would agree to use the largest municipality's name for the joint effort and so the tax co-operative was named for a local tourist attraction: the Happy Valley Ranch. Although now rebuilt as a federally funded national heritage site, the ranch house had been built in the 1890s as a second-generation cattle baron's imitation of an English country manor, acquired a rather different cultural status during prohibition, and been razed to the ground in an uprising of the local moral majority in 1957.

In the 12 years since its 1988 start-up, the tax authority has acquired duties that go beyond simple property tax assessment and collection. One town has a hotel room tax, another provides school tax credits for couples with two or more children, while a third has an industrial land development program with both rebate and tax relief schemes to attract tenants. Today, the authority collects 32 different levies from about 45,000 taxpayers. It administers eight rebate, direct support, or tax relief programs, and collects tolls on one road bridge and two park entrances.

About eight months ago a town councilman with strong connections to the firm I usually work with got his council to hire the firm to do an operational audit of the authority's effectiveness and assess what value the town was getting for its continued support of the authority's mandate. That report was dully produced with the usual platitudinous result and recommendations for minor change.

The councilman had lunch sometime last week with the managing partner, and now Barb Rush and I have been dispatched with specific orders to "review and evaluate the use of information systems technology with a view to making recommendations to improving the efficiency and effectiveness of operations." She works in the insolvency practice, but apparently has audit experience with the client so we're to spend a day there and a day working out our report.

The tone for the day gets set at the 8:30 AM meeting. Not only am I the only male not strapped into a suit and tie, but also Barb switches from normal to distant as soon as we enter the building. In the boardroom to which we're conducted to wait, the informal chatter preceding the executive director's arrival amounts to a secret handshake among initiates to a cult. I'm not good at this but Barb and the systems director deftly negotiate the protocol by talking about upgrading their home systems from ME to Windows 2000 Professional while others listen and nod approvingly.

Their behavior reminds me uncomfortably of ants rubbing antennae together to establish their mutual allegiance to the same queen. According to their words, neither one can get the thing to work, but what they're actually saying to each other is: "I'm no threat" and "me neither". There's nothing I can contribute to this, but when eventually asked for an opinion there's a sudden hush and I am marked as an outsider by not playing.

When the Executive Director arrives a precise 11 minutes late, Barb smoothly slides the knife in as she introduces me: "Paul," she tells the group, "is a Unix/Macintosh expert who sometimes works with us."

Findings

I'd have been better off, I think, if she'd introduced me as a child pornographer out on day parole. Their board has told them to answer questions, including mine, but everyone else at the meeting, including Barb, make it clear that they constitute a group -- to whose membership I need not aspire.

The systems director, who attaches himself to us for the day, tells us that his department has a staff of nine including himself but that the budget is both too complex and too sensitive to share. Barb promptly agrees we don't need it, leaving me floundering -- and unwilling to ask for the service level agreement because I have no idea what she'll say.

The authority employs 68 people of whom 61 have PC desktops. All of these have Windows NT 4.0 Workstation with SP1 on 550-MHz P3 chips, 15-inch screens, and 64 megabytes of RAM. The NT Servers are in four rackmounts and there's one real surprise: an HP K220 that turns out to have 24 gigabytes of Oracle 7.31 dataspace on two refrigerator-sized external disk packs that each have 32 650-megabyte drives.

Given the patent hostility in the place, finding a K220 is a bit like unexpectedly running into an old friend. I tell my two tour guides what great machines these were, but there isn't even a pretence of interest.

The primary property tax application was custom-written for one of the municipalities involved, originally for a McDonnell Douglas Microdata running PICK. As part of the authority's start-up, this package had been ported, mostly by outside consultants, to Oracle on HP-UX with Windows 3.11 clients. They're in the process now of porting it to NT with SQL-Server. That's why there are nine staff, four are client-server developers on this project. When I ask whether they're working with SQL-Server 7.0 it turns out they started over a year ago -- but they're sure the upgrade will be no problem at all.

Overall, the production software doesn't look too bad. There are individually licensed copies of Microsoft Office on all the PCs -- but all of them, including the rackmounts, look like they came from a basement assembler. When I ask Barb if the auditors had verified their license status she doesn't know -- confirming my earlier guess that whatever she's here for, it's not her systems audit expertise.

There's no formal workflow or document management program in place, but most of the smaller tax programs have some useful form of automated support. In many cases, this is minimal; but the volume is so small I don't see cause for concern. The hotel room tax, for example, applies to a total of six establishments and 82 rooms. So what if it's managed from an old Lotus 123 spreadsheet that's been ported to Excel? You could pretty much do this kind of thing on paper without missing a beat.

The missing workflow and tracking applications raise intriguing audit issues but Barb's not interested in that either. On balance, I'm starting wonder what she plans to do all day and why we're even here when we get our next hint of serious trouble. A couple of assessment clerks take our visit as an opportunity to harass the systems director about database crashes. This happens all the time, they say; and often means doing work over. Right now, they say, they're re-entering data from yesterday.

To me, this doesn't make sense: Oracle's 7.31 was one of those stopped clock moments in product development (stopped clocks are right twice a day). Particularly on an HP K-class, it should take serious effort -- something like high explosives or an idiot with the root password -- to cause it to fail. I ask the director about the PC client and the network as more probable suspects -- and trigger an off-the-scale defensive reaction.

Networking, it turns out, is incredibly complicated stuff for them. Despite using NT Workstation, they rely on Windows for Work Groups for PC networking. The TCP/IP access needed for Oracle and SQL-Net is layered on using a dedicated DHCP server accessed after someone booting a PC logs into the appropriate departmental file and print server. Not only are they running half a dozen LANs on the same wiring, but they're using 124.0.0.x for the DHCP server and cycling through 200 or so IPs to allocate new addresses for every PC reboot -- triggering new rounds of cleanup and resource authentication effort each time.

There's an external Web server and a firewall too. Both are NT machines but there's no actual content on the Web site yet. It has Crystal Reports and IIS hooked up, but the application they're developing for it isn't ready. The director explains that the building permits system will allow people to access and pay for local building permits on the Internet. Right now, he says, the municipalities do this manually and the property assessments often don't get updated. By putting this on his Web servers, he'll integrate the databases and save the local governments money while raising tax revenues. It's an experimental program, he says, that they've had to put on hold for a few months because they have had so much trouble with Oracle that they decided to expedite the conversion to SQL-Server before proceeding with the Web site services mandate.

Although Barb worked on the audit team, she doesn't admit to knowing anything about project authorizations and I don't want to ask the director for his files because I've been trying to reduce his hostility and suspicion. Instead, I offer to check out the problem with the K220 for him.

He does not want me to look at the machine, but I do my best humble cowboy shuffle and, mainly because he's deeply conflicted between wanting to throw me out and not being sure what credibility his bosses will attach to my report, he decides to accept my offer to "just check it over and see if it's something obvious."

There are many obvious somethings, but nothing bad enough to cause frequent failure. It's still running 10.20 and has no patches more recent than December 1998 but it's got four 120 MHz CPUs, 768-megabytes of RAM, and separate narrow-SCSI controllers that go out to each of the disk packs. Swap and system disks are on an internally mirrored volume consisting of two 650-megabyte disks on each side and there's a DDS3 tape drive that they tell me is used for backups.

The Oracle installation isn't what I'd do either. Cooked files, no obvious attempt to balance dataspaces across controllers, and no log mirroring at all. Inefficient, poorly structured and unmaintained, but none of this is remotely sufficient to explain their problems. Why does it fail so often?

The answer is that it doesn't. What's happening is that years of disk fragmentation, inappropriate kernel parameter settings (25 percent system buffers), the use of cooked files, and very long record formats, derived from the PICK system the software was first written for, sometimes combine to force Oracle to issue thousands of sequential page reads to satisfy relatively simple lookup requests. That causes long delays during which the system seems to lock up. When the DBA reboots [!] the machine to clear it, Oracle data written to system buffers but not yet flushed to disk is lost -- causing long recovery times and work rollbacks.

The DBA, who acts as the Unix admin, thinks the IPL stuff he sees on screen at boot time is Unix and tries to run Oracle via a socket connection from his NT workstation. When he can't get Oracle's attention this way, he just power cycles the server. This, he reports cheerily, always works.

"You need to get some help here," I tell the systems director, "get Oracle set up right; use raw devices -- or, if you must have cooked files, use direct I/O -- balance the load across some bigger disks. You could probably pay for some 9-gigabyte disks just on power savings. You could contract-out for a part-time Unix administrator. This thing should never fail, there are K220s out there with the same Oracle release that haven't been restarted literally in years, get it set up right and it will have years of life left in it. Get some 10,000 RPM disks instead of the 3300 RPM units in the machine and it will easily out perform SQL- server too."

While I was working on the machine the director's natural friendliness had started to override his fears but all the faces around me close up tightly when I say these things. I'm contradicting absolute and revealed truths here: Unix is obsolete, disks are many times more expensive than for Windows, the technology is hopelessly unreliable and hard to manage -- just look at their experience with it for proof -- no, no, no, NT will fix everything, that K220 is the enemy incarnate. By association, so am I.

On my better days, I'm smart enough to know that there's no point in arguing with a client, but this wasn't a good day. I point out that the table definitions used in the Oracle implementation will be impractical in SQL-Server, they're going to have to redo all the data structures, rebuild the application as a set of stored procedures, and rebuild the client. All that will take time and testing, so why not buy a whole bunch of time by fixing the K220 setup now? Then, if the conversion is going to take more time, they could do the building permits service with Apache running directly on the K220 using a Perl or PHP application.

This is emphatically not going to happen. They have an all NT strategy, the director says with finality, and that's where they're going. To even suggest anything else is just unacceptable.

I tell them Microsoft is making their hard work obsolete. They'll have to do it all again for Win2K and SQL-Server 7.0, and then again for whatever the next generation is -- just to keep up with Windows product obsolescence. This, of course, is my worst blasphemy yet -- they know their code will work perfectly with future Windows releases. In response, I make things worse by asking them if this was true for their upgrades from Windows 3.11 to Windows 95? From NT 3.51 to NT 4 Server? From NT Workstation to Windows 2000 desktop?

The result is a further change in attitude. People literally turn away as the hostility and suspicion visible before get submerged into the kind of bemused condescension usually reserved for other people's obnoxious children or the very old. After some verbal fencing, the director asks, do I have any other recommendations?

I do, but they aren't anatomically possible -- and would contravene his NT licenses -- so, instead, I ask about his schedule for meeting the public information access mandate. This is a bluff, I'd never heard of it until he used the magic word earlier, but he doesn't know that.

The deadline set by the board wasn't realistic, he explains. It's all the fault of Oracle running on that old HP box. They have installed an NT server outside their firewall with a IIS front end that lets users run Crystal Reports but Unix doesn't work across the firewall, so they put the project on hold until they get the SQL-Server conversion done.

I'm dumbfounded; and finally silenced. He fills the silence by talking about having had to sign a non-disclosure agreement to get early access to Microsoft E commerce Server for his building permits initiative. Microsoft's sales guy is even talking, he boasts, about partnering with the tax authority on this. "We'll be helping them sell their package to other municipalities," Barb chimes in, helpfully.

It takes a minute or so to absorb all of that. A couple of hours ago, while looking at the K220 I'd seen the support modem with its dedicated phone line properly in place and asked what help they were getting from HP on their Oracle problem. They'd cancelled support right after starting the conversion project -- about a year and something ago -- expensive, and unnecessary because they have the NT server already, he said. When I call the number helpfully listed on the back of the K220's cabinet, the modem answers. This is more of the same, but worse.

By now even I know it's hopeless, but I don't want to think about what he just said because tempers are already frayed and asking how his NT server would keep production running if the K220 failed just wouldn't help. Instead, I seize on a related issue that's been niggling at me most of the day: if I have the chronology right, they started the permits application about 18 months ago, and there are four people. This just seems way off scale for something that any competent Linux developer could whack together with a mysql/php combo in a matter, probably, of a few weeks to prototype and maybe two months to final product. I ask for the original and current project plans. Alas, the person who has the licensed copy of Microsoft Project isn't in right now. Of course, the director says, they're using object techniques, but their expert on using UML within Visual Studio is the same missing person, so they'll send me the information later.

At this point Barb finally proves useful, she saves me from a possible assault charge by pulling me away. We're out of time, she says, and the executive director wants to see us before we leave for the day. When we get into his office, it's clear that he's been briefed on my discreditable views but has a message of his own to convey. The Web issue isn't important, he says. The board caved to one town's demand for public access and didn't really mean it, so the delay doesn't matter. Furthermore, they're going to love the building permits initiative, he says, and accept any delay as the cost of getting it right.

Message delivered, he and the systems director rub antennae with Barb again, and we're ushered out of the building.

Next day, I do what I should have done first -- find and read the file from the previous review. It is, of course, signed by one of the insolvency partners Barb reports to -- and he thought the systems operation was "forward looking," "innovative," "on track," and "meeting best practices standards for highly effective computing."

Recommendations made

The report was, if I do say so myself, a masterpiece in the art of weasel wording. "The problems and delays encountered by the authority appear," I wrote, "to be due to the inadequacy of systems resources in the face of rapid and unexpected change in the external technical environment."

Then I pointed out that the need to replace or substantially upgrade all of the Authority's computers and applications as Windows 2000 replaces NT "creates a significant and timely opportunity for the Authority to formally consider its longer term systems infrastructure and staffing ratios." I wrote, "Such a review would focus on accountability, cost, and performance as the Authority's senior management adapts its systems deployments to changes in its mandate, the emerging public awareness of information security issues, and the availability of inexpensive outsourcing or other third party solutions."

Then I washed my hands, cleaned the keyboard, changed the printer toner, and resolved never to darken the authority's doorways again.