Designing for Decades
I originally wrote this up as a reply to this excellent comment on the MUMPS thread. Since that thread is getting exceedingly long, I decided to start with a new post here--and anyway, I'm basically changing the subject away from MUMPS in specific and more towards some of the other painful realities (and ironies) illustrated in the aforementioned comment. This excerpt tells the tale:
MUMPS applications are likely to remain in place simply because it would take a very long time to create solutions that replicate the 30 years worth of features. At best, it would probably take 5-10 years of *transitioning* between MUMPS, a MUMPS/SQL hybrid and then SQL-only to achieve a pure SQL solution. During that time, few new features or extensions would be able to be added to the application since the developers focus would be on transitioning. The end benefit would be a system that new programmers could immediately start working on and in a language that offers modern features. Future development from that point would be faster and more efficient. Still, the administration is unlikely to back this scheme since they won't want to pay a team of programmers 5-10 years to "run the red queen's race" and the programmers won't want to do it since they'll see it as an effort to make them replaceable.
Ouch. There are real challenges like this one facing the software world related to long-life applications, both the ones we're stuck with (or, to put it another way, the ones that have been serving us well all these years) and the ones we've yet to build. I see it as two sides of the same coin, in a way, the first side being, How we are going to effectively manage an aging fleet of systems and a retiring workforce of programmers? The second side is, How can we do better at building systems from scratch that are likely to have long lifetimes? How can we *plan* for decades of use?
When software applications have a life span measured in decades, the dynamics change. Looking back to 60's, it's easy to be forgiving of the people who developed the solutions that have stayed in production since then because the idea of decades-long use might not have been conceivable. I wasn't there, and who am I to second-guess. But now that we know that decades-long application lifetimes are a reality, how are we as a profession addressing this need? How are we advising our clients?
What are the favored techniques for designing for decades? Is this an explicit topic on which people have researched and/or published? Language, tool, and platform choices seem like key factors. Is "roll your own" from top to bottom the only way to ensure longevity? Or is the adoption of "open" standards and technologies protection enough from market-driven innovation cycles?
One recent place I recall this topic coming up is Pete McBreen's manifesto Software Craftsmanship, which, while flawed in some ways, is a worthy read. McBreen tackles explicitly the topic of taking a perspective as software craftspeople on how we can build systems that will really last for decades.
And of course I've so far ignored the question of whether companies and governments would be willing to pay up front to design and build a new system with decades of use in mind (not to mention the complications raised in Saintly's comment for situations in which one decades-old system must be replaced with a new one that will live just as long!).
Didn't Frederick Brooks say "Plan to throw one away."?
Thanks for reading,
Dan
The technical is the political
Dan, I think the Social Security Administration example is appropriate here. SSA data processing started with the punched card model and evolved successfully not for technical reasons but becaue fat cats in government and in industry could not figure out how to raid "the third rail of American politics" for money.
They could shoot at Bonus Marchers in the 1930s because dese guys were bums.
But the "abstract" beneficiary of Social Security is your sweet old Granny who used to work at Woolworth's and who had FICA taken out of her pay for years. In more "serious" terms, precisely because the beneficiary is so broadly defined, it is difficult to complicate the "requirements document" with demands for controls (over and above what is necessary).
Contrast IRS data processing. Because of a famous court decision, establishing "no legal responsibility" to pay "more" taxes than lawful, and because no similar legal clarity exists as to the difference between gross and net income, each tax return is almost undecidable, and IRS data processing is mess.
IRS data processing is further complicated by something noted by Ralph Nader. The US, owing to Supreme Court decisions of the late 19th century, is too generous by world standards in endowing the limited liability corporation with person-like rights, and it did so in order to withhold rights from African-American freedmen. Thus one physical person (or group of persons) is able to file as several entities.
Conservatives, who demand a flat tax, get close (but no cigar). They want a just definition of what's mine and what's Uncle Sam's. They get no Cuban cigar from me, because justice would demand a PROGRESSIVE tax...but one without exceptions.
The abstract constituent of data processing at the VA is the military veteran who is in our society somewhat of a suspicious character unworthy of the SSA's level of diligence, and I am hypothesizing that the use of MUMPS and its flaws as listed in the original thread, the VA's failure to evolve (as the SSA evolved) is due to the fact that owing to a lower committment to veterans than to sweet grannies, it was never thought important enough to change Mumps when there was time...say when it was realized in 1976 that the relational model was more powerful than the b tree.
How much "software complexity" is social oppression and ambiguity?
Veterans made homeless by Katrina are today fighting battles to get checks they need to survive. OK, does Mumps have the ability to "sense" that "two" guys, one living in New Orleans and the other in Utah, are the same?
Sure, it's going to match on military ID. But are there "controls" that prevent the match? And is the software being changed to reflect the increase in veteran mobility as a result of Katrina?
Note that the tail may wag the dog, because on the agenda of a hypothetical meeting may not only be the needs of the Katrina veterans, it will also be the fact that the "only" people able to change the software and remove the controls are approaching retirement.
In a system with an "abstract" constituent imaged as a person with unquestioned rights, such as at the SSA, the issue of personnel may be dismissed but it may SILENTLY weigh the agenda when the constituent is a concrete and (eek!) African American person.
During the run-up to the 2000 election, data processors at the firm responsible for purging felons from Florida voter rolls asked each other on the job whether or not the "selection criteria" (probably, a regular expression) being used to purge was not overgeneral, but according to the record, they reassured each other that the people being processed were "probably" scum bags.
Note that they were involved in the manufacture of the "knowledge" of who was scum and who not but they used the image of the knowledge to predetermine the result, when in an ideal world, they would have descended upon Katherine Harris' office in a group and invited her to shove her voter rolls up her bony ass.
The data processors in Florida, or ay my hypothetical meeting at the VA may differentiate themselves and not have empathy with the beneficiary, for the same reason Barbara Bush thought that the New Orleans refugees would have fun at the Super Bowl, they being "different" from predominantly white data processors with health insurance.
The underlying structural problem in the US is that except for Social Security, and there only with respect to older people, do we have an abstract conception of a person with economic in addition to political rights.
Ironically, the careerist, Michael Brown, who ran FEMA at the time of Katrina, had a good idea. He thought to send or provide through central points a simple debit card preloaded with 2500.00 dollars to each person proving residence in the affected areas, and my own experience is that when you have squat, 2500.00 can carry your for long enough to find a job.
The simplicity of the data processing emanates here from the abstract humanity of the business plan. He defines the beneficiary not as the end product of a gameable process but as someone with an ID. The computer system to implement Brown's idea could be developed by a high school kid.
Likewise, now that I am in Hong Kong on a work visa, I pay 100.00 HKDs if I go to the emergency room. There is no complex exchange of unpayable bills from the emergency room itself, from the doctor as an independent contractor, from the radiologist, from the audiologist, and from the voodoo priestess for all one knows. There's no need to reconcile multiple bills and multiple sources of coverage.
Software is hard to do in and of itself; but it's not THAT hard; it's not rocket science. But if you're in the server room at 3:00 in the morning trying to figure out why addresses originating in Addis Abbaba appear on outgoing property tax bills for property in North Carolina, something other than if and then and do while is probably the culprit.
The user told you that the bills were always to be sent to the property taxed without forwarding. Her boss "forgot" to tell her that he had made an exception for a CIA agent resident part of the year in beautiful downtown Addis Abbaba.
The complexity is political.