Software Infrastructure - Do you know what you have? Do you have what you need?
I've been thinking this week about software plumbing (though infrastructure might be better word than plumbing). I'm referring to those parts of our systems that don't directly fulfill business requirements (though sometimes they do), but that we still can't do without.
When someone (particularly someone who is not a fellow programmer) asks us to describe a system, we seldom begin by describing the really cool logging system, or the elegant base classes and interfaces that we designed for collections, or the centralized exception handler. However, this infrastructure is as necessary for a system as the things we might normally talk about: business objects, user interfaces, reports, and database tables. No one talks about the system's plumbing during requirements meetings, but if the plumbing were missing it would stand out like a new house without a flush toilet.
Often, designing and developing the infrastructure takes more time than the it does for the "first class" parts of the system. Where this really gets us into trouble is in planning and estimation. We forget, when telling our project manager, "That should take about two weeks." (the default estimate for all tasks), that we don't yet have the infrastructure for logging, caching, or exception handling. Often it's not until we're in front of the keyboard that we realize (or remember) that we lack certain key infrastructure elements. This can happen, especially, after joining/forming a new team and when using new tools or languages.
The evil in this is not so much that we gave a faulty estimate to the project manager (which is bad enough), but rather that we're saddled with the temptation to hack our way out of the situation. When struck with the realization that we're missing critical plumbing, we have at least three options: revise the estimate (often not an option), hack in some shortcuts, or work extra hours to properly code the parts of the infrastructure we need. In practice, the solution is probably some combination of these.
Personally, hacking in shortcuts is so distasteful to me that when caught in this dilemma I'll work the extra hours or revise the estimate (the latter might especially be a good option if I was pressured to pull the estimate out of thin air without being allowed time for proper analysis--which is so often the case). In those situations where a shortcut is necessary (the quest for the ideal must always be balanced by the business realities), I at least want to take a little extra time to think about how I would design it given the time, then hide my shortcut behind an interface or other indirection layer to allow for easier refactoring later.
It is worthwhile, I think, to train ourselves to consider the state of our infrastructure (and our programming tool box) when estimating the duration and effort for a given task. It's definitely something I'm working on for myself, as estimating continues to be the single most challenging part of my job.
Good managers are aware of the estimation risks stemming from unknown requirements. However, as programmers we may have some work to do in educating managers on the infrastructure issues that can have significant effect on the quality of an estimate. This is especially true when combining the risk of unknown requirements with the risk of unknown infrastructure-related tasks, for often the former leads directly to the latter.
Some questions we can ask before giving an estimate:
- What is the state of my toolbox (common utility functions, database access, code templates, etc.)? If my toolbox is lacking, then every time I need a tool, I will need to stop and invent it, borrow it, buy it, or steal it.
- Have I done work for this client before?
- Have I done work on this project before?
- Have I done work on this part of the system before?
- Have I programmed in this language and/or paradigm before?
- Will the task require the use of any tools or technologies that are new to me and/or the organization?
- Do I already have a workable model of the system in my head?
- Do any of the requirements introduce a new "class" of problem that has never been solved before within the system? For example, if a new requirement says that the system needs to send an email, does the general capability to send emails already exist in the system? And, as a follow-on questions, has our team/organization solved this class of problem before--and is there anything re-usable there, even if it's just the design pattern?
- Are there bureaucratic obstacles that will slow me down?
- And of course, the all important question, how much do I know about the requirements at this point?
So what are these infrastructure elements that so often bite us this way? Here are some off the top of my head:
- Logging
- Exception handling and exception post-processing (logging, emailing, etc.)
- Test harnesses and unit testing frameworks
- Build scripts
- Database connectivity
- Configuration files and global data storage/retrieval
- Caching
- Object/Relational Mapping and Persistence Frameworks
- Session management (in a web context)
- Debugging infrastructure (particularly in distributed architectures)
- Distribution (install scripts, mechanisms for performing patches, etc.)
- Plumbing for searching, sorting, paging, and presentation of large data sets (common in a web context)
- Encryption
- Load and stress testing
- Code templates and code generators
- Transaction design strategy
- Faxing, emailing, etc.
- Report/document generation and distribution
- Installation programs
- Bases classes, interfaces, namespaces, etc.
If we're lucky, our out-of-the-box toolset (for example, libraries, runtimes, frameworks, IDEs, operating systems, etc.) will provide some of the tools we need. Even in those cases, however, we need to be wary of excessive optimism, for at least a few reasons:
- Just because I have an existing tool does not mean I've used it before; there will likely be a learning curve.
- The out-of-the-box tool may have limitations I did not consider, making the tool unusable for my particular purpose.
- The out-of-the-box tool may not work exactly the way I expect, requiring me to augment it or write glue code to supplement it with some other tool.
The problem of unexpected limitations bit me when I started my first Microsoft .NET DLL project. I started the task assuming that my need for data-driven settings would be served by the CLR's built-in "AppSettings" capability, which I had read about and used in other .NET projects. However, I soon found out that those *.config files are only supported for executable assemblies (not DLL libraries). So I was left to roll my own solution, which of course took time.
My original version of this essay failed to include any mention of configuration management-related issues. It's easy to forget this because it is not "writing code." It's also easy to forget because often there is such good tool support. Nonetheless, configuration management definitely adds overhead to the coding process and must be taken into account. Some items to consider (not all apply to all situations):
- Creating new build scripts
- Updating existing build- and configuration-management related scripts and configuration files when adding, changing, or removing components in an existing system
- Figuring out and controlling who can do builds, on which machines (a big factor in environments that use a lot of shared libraries and dynamic linking)
- Deciding how and whether builds are automated
- Integrating builds with automated tests
- Checking in and labeling builds in the source code control system
I think there are also other kinds of tasks that we forget when estimating:
- Configuring servers and middleware
- Preparing release notes, installation instructions, and other must-have documentation
- Preparing test environments
- Creating test data and/or cleaning up old test data
- Code reviews
- Unit testing
Can you think of additions to these lists? Please post your suggestions using the Comments form.
Dan
A key issue! Thanks, Dan
Dan, this has long been a concern of mine.
Managers do not seem to realize that infrastructure takes time. Even if there is a tool for part of infrastructure, two issues arise:
(1) Researching the availability and functionality of the tool takes time especially on Microsoft platforms because documentation on Microsoft platforms is often incomplete and marketing-driven, concealing significant problems as a result.
(2) It is often necessary to build glue to adapt a tool to specific environments.
For example, the tool may report errors in a way unacceptable to your user.
In the Apollo 13 launch, ground engineers directed the crew to create a channel for oxygen recycling out of duct tape which happened to be available on the Apollo 13.
But the costs of using the duct tape were not only the cost of the tape but also of adaptation.
Unfortunately, since programmers are insufficiently organized as a profession and have no political or legal power, infrastructure will never be properly authorized: for if you read the MIS literature you learn that "programmers" are subprofessionals, almost subhuman, creatures who haven't yet learned that "the business of America is business".
What this means is that you won't get credit, in many environments, for creating great infrastructure because this isn't "focusing" on the "needs" of the "business".
Nonetheless, I see countless business failures due precisely to lack of infrastructure quality. This is why in my own practice, whenever I am waiting for requirements, I spend hours on adding to the utilities.DLL and other tools that form the basis of the compiler in Build Your Own .Net Language and Compiler.
Having said all this, I will confess that there is a real danger in "infrastructure".
You see, the goal of structured design is to minimize preconditions. "Infrastructure" adds preconditions in such a way that from a strictly managerial standpoint, it adds blockages and dependency without authorization or control.
Consider your desire to write config information from DLLs. The problem is I think that *.config files presume at least one hard drive and knowing Microsoft it's probably the C drive. If the software is a Web service, it will be key that the config information is stored on the server (and on only one server), therefore if it includes specific user preferences, these preferences will need to be stored with a userid.
A parallel situation would be the need of infrastructure to write config information to the Registry where the config information assumes that the user, and not a network server, owns the Registry.
I admit the need for infrastructure, of course. Build Your Own is based on string-handling extensions in utilities.dll and Windows extensions in windowsUtilities.DLL.
At the same time, you have to be real careful about factoring. My older COM version of utilities.DLL mixed generic string and math utilities with things like 'center the form' and 'systematically write all the form settings on the Registry'.
It was in factoring the compiler for a Web service client that I realized that in .Net, Windows is a separate and optional level.
I mentioned to you that I am writing a presentation for OOPSLA. I have submitted it for review. It is tentatively, subject to review, going to be Deconstruction for Programmers.
In it I show that programming is ineradicable writing and as such fated to always end up with a combination of power and a completely subaltern weakness, because as the necessary inferior term in all MIS relationships, programming has lots of what Derrida speaks of as "death, absence, transgression", which nonetheless is a complete precondition to the bright shining lie in the first place.
[I call it a "lie" because of a Platonism that sees that the ultimate pretense to full presence can never be fulfilled by media, by definition: media after all mediates terms by whose grace and favor it exists: there can be an abstract art but not an abstract TV program, that is NOT a representation of the "art" by necessity.]
[And even the most random code is trying to mediate something outside code. "Hello world" is a simulacrum of a talking human subject.]
The problem in talking to managers about "infrastructure" is you're talking about "the inferior term to the inferior term", writing squared.
The PARADOX is that infrastructure is KEY.
I can well imagine a manager telling Edsger Dijsktra in 1968 that "your 'structured' nonsense is ivory tower theorizing at its worst and by your next performance review I need you to demonstrate more attention to the needs of the User. I need you to stop hiding behind your computer and *writing* code and I need you to *speak* to the user and above all *listen* to him. Just code what they tell you to code, exactly and in a single main procedure."
What I hope to show in the article I am working on for you for delivery 2nd or 3rd May is a return to the way in which structured programming, while it didn't increase "programmer productivity" for the very good reason that the LACK of "productivity" is a disciplinary urban legend, was a Copernican revolution which helped us to understand our job better: I was there.
My oopsla paper shall include material by means of which I hope to show that WRITING is the key to good object implementation: that in fact the design of many objects is the design of a language, a notation.
I only understood the Quick Basic variable model when I developed a notation for writing down, without loss of signal, the object's state in ASCII, which could then be used by a constructor to create the objects.
Good objects, that is, seem to have a "calculus", their own notation. For example, a simple QuickBasic integer can be fully reconstructed from the calculus expression integer(255). An array of integers is array(integer, 0, 10) where the 0 and 10 are its bounds. An array of structures is array(structure(string, integer), 0, 10).
Note that this replaces the "diagrams" which noncoders seem to love by a return to symbolic computation...what Willy Loman, in Death of a Salesman, calls with dismay "math, math, math" when Willy realizes that Biff has failed the Regents exam: what George Bush called "fuzzy math" with a smirk because he didn't understand Al Gore.
But the real key is that symbolic math, more than pretty diagrams and authoritarian games in endless meetings, is writing in relation to "speech".
I will blog the progress of the OOPSLA presentation. It hasn't been approved, but I hope that it will. I first spoke on "Derrida and Dijkstra" in 1992 at a computers for education conference and since that time I have noticed an increasing number of papers in this area.
The trouble is that this isn't "value-free" science. The barbarism of much computing is that people take as a given the oral structures on which it is based, and that necessarily the constructors of this grand illusion will be less credited than scorned as reminders of its constructed nature.
Don't Forget Building
Somehow it seems to me (in my limited experience) that the issue of building is not typically given as much attention in the Microsoft world as it is elsewhere. Perhaps this is because the Visual Studio tools tend to hide the concept of building from developers by making the process deceptively simple and "automatic." This worldview might be changing with the advent of NAnt (which my team used with great success to build a VB.NET project last year).
In my experience, the build process is one of the biggest plumbing operations out there. It's not something that people (well, non-developers, anyway) usually think of in advance, but if you want to have a system that you can easily build and install, well, you've got to pay very close attention to your build process. Tools like Ant and NAnt make it possible to have nice build scripts, but they don't build your application for you. You've still got to figure out exactly how you want this to occur, and ensuring that your builds are always reliable and reflect the latest source code changes, or the changes as of Label XYZ is no mean feat. It takes a great deal of planning and organization to pull this off.
So add "Build scripts" to your list, Dan.
Builds and Configuration Management
Excellent catch, Rob, and you also make an excellent point that Microsoft-oriented developers (such as myself) tend to think less of builds as a major task since the tools make it so easy. What's not always so easy, though, is configuration management in general. I'm going to take your advice to add "Build scripts" to the list even further and add a new section to the original post for build- and configuration-management overhead. What's missing from the new list?
Dan
The Daily Build
Building is a frustrating issue because it's so seldom planned and it happens at the last minute. One way to address this problem is to build SOMETHING on the first day of the development project at quitting time, and every day add to this build.
The "power" of build packages like Installshield is that they support a full-scale programming language and this "power" becomes paradoxically a weakness by multiplying the time it takes to deliver a product.
The transition from COM to .Net solved a lot of build problems and added few but left the "nature of the beast" unchanged of necessity. Furthermore, some ways in which .Net addresses build issues (by, for example, creating duplicate DLLs by default) have that sort of inelegance which is in fact related to practical difficulties.
Problem breakdown is supposed to save us but it doesn't when it is made along the fault lines between persons, as where the least dominant male in a development team is named the build master because nobody else wants to do it. This is because the "real" developers then assume that the problem is solved and code without consideration of build problems.
. . . from the ground up
On a new project we are about to begin at work, we just had a developers meeting where we planned how the work would be divided up and what steps needed to be taken at the beginning of the project to make sure we didn't get tripped up later on. One of the big discussions was on the build process. I've learned that building should never be an afterthought. That's a sure-fire way to get into trouble.
Whether the lack of a defined build processes in an MS project is, in fact, easier than what goes on in the build script world of Ant is surely a subject worthy of debate. I suspect that there are advantages to configurable build scripts created for tools like Ant and nAnt that are unknown to the many thousands (millions?) of MS developers who have never used these fine (and free) products. For example, they make it fairly easy to automate the kind of daily builds that Edward speaks of.
Infrastructure and Estimation
Dan, you have a fine list here. When developing non-commercial software (i.e. for internal organizations) two steps that are often overlooked are Data Archiving/Retention planning, and Decommissioning a legacy system.
Developers are fairly guilty of taking the Scarlett O'Hara approach to archiving...we'll think about it tomorrow or a year or two from now when it becomes an issue and then we'll deal with it.
Decommissioning a legacy system isn't always applicable, but it is a discussion that should happen up-front. In my organization, we're still floating a legacy system over two years after its replacement was implemented...simply because the clients refuse to allow the apron-strings to be cut. (They're not entering new data, but looking up old data that was agreed not to be migrated due to the vastly different file structure. We've offered to offload it for viewing.) At some point, we'll have to insist, but if an understanding had been agreed upon up-front, it would have been much easier to manage.
Also..about estimation. I manage 10 developers and we have at least a dozen projects going on simultaneously. Every one of us is guilty of estimating perfect-case scenarios. Yes, you probably could create a form in two hours, but by the time you wrestle with the devil (unexpected technical difficulties) and talk about it and review it, well, next thing you know, you've done nothing else for two days. Even knowing this...knowing we are bad under-estimaters, the practice continues. I think it is due in part to ego: developers like to think they can jump through hoops quickly and maybe fear that admitting to "slow" productivity would be viewed poorly.
On the other hand, the practice of blindly doubling or quadrupling estimates randomly can hurt those developers whose personality demands a challenge. They may spread out and take all the time estimated, whether necessary or not. (I don't think that is true of everyone, and it often isn't due to laziness, but to perfectionism..."I've got more time, I think I can do it better...")
I find some comfort in knowing that we're all pretty bad at estimating. (Isn't that true?) The thing is, if we ever did the same sort of thing twice in a row, we could improve. In reality, I almost never have. The technology/tools change so often, we're almost never using tools that we're entirely comfortable with in every aspect...there is always some learning curve anticipated.
We do try to compare estimates to actuals so that at least you can look back and see how long it took you last time, but again, I think there is some fudge factor happening in the actuals. Developers don't tend to be like attornies...billing for the time spent *thinking about* a case. We dismiss that as though it doesn't count, when in essence, thinking about a project is a part *the central part* of the design process, whether we're drawing a diagram at the time or not.