Software Design: Top-Down or Bottom-Up?

Recently, I was working on the design of a new software system. As the lone designer and programmer, I had the requirements in my head, and had made lists on my white board of some constraints and pitfalls that I knew I had to design around. But I was having trouble. Over a few days of working on the design on and off, I kept getting stuck. No fun at all.

I'm a top-down, big picture person. I like to work in the abstract, and I work best when I can form a high level structural picture of something in my head and then drill further down into details from that high level. For example, when I design a code module, such as a class, I like to build the structure first, making frequent use of stub procedures, holding off on committing to too many details until I am confident in how the overall structure is going to come out.

(Not all modules are complex enough to be so interesting, of course. I probably picked up this habit from Code Complete; in fact, one reason that Code Complete resonated so strongly with me (damn) almost ten years ago is that I get the sense that Steve is this kind of top-down thinker (not that I'm trying to compare myself), and his thinking about programming is very rooted in the the structured design and programming tradition, which I've always thought of as tending to be top-down in nature. Somehow it seems hard to visualize an elegant modularization of a system-level design problem--maximizing strong cohesion and loose coupling, with good data structures--without taking a high-level view. Ah, but can one evolve into an elegant system design (I'm talking *system* design, with multiple layers and components) by developing only one module at time? And would good refactoring and automated unit testing be essential tools in attempting such a feat? Man, I've digressed big time here.)

I think my affinity for top-down design has something to do with a pattern matching, generalization, and taxonimization (is that a word?) process that goes on in my head constantly. As I'm processing smaller bits of information or ideas, I generalize a level beyond the factotum itself and determine where it fits. Which category? Where in the big picture? Do I need a new category? Does this information invalidate current assumptions?...You get the idea. I'm not trying to say that this is better than someone else's way of thinking; it's just the way my brain works.

Because of this, when designing software I tend to start with the big picture, forming a mental framework into which I can build smaller mental models, and then I test these subsystem mental models against the higher level mental model of the system. If I encounter something at the lower levels that runs against the grain of the higher level, then I have to adjust the higher level model to fit. Through an iterative process of this I can end up with a system design that is self consistent and relatively complete. As I complete the design during the coding phase, I can refine things further--but I almost never find that I have to rethink the whole design once I've mentally gone through this exercise.

So last week I was having trouble. I couldn't come up with a high level design to accomodate my lower level concerns, so each iteration of the process described above would short-circuit. I was pacing in front of the white board, trying out various designs, working through the angles. Sometimes I would switch to researching some aspect of the requirements to settle a question or validate an assumption. After a few days the frustration and lack of progress was really bothering me, like some kind of personal failing, and it starting creating stress in other non-work aspects of my life.

As Robert Glass points out in his essay "The Cognitive View: A Different Look at Software Design," this type of process is normal for software designers (and probably other kinds of designers as well--a friend of mine wrote to me to say the essay really resonated with him as a mechanical engineer). Glass writes, "failure is an essential part of successful design." Having this knowledge in the back of my mind was some comfort, but only some. So I decided to try a change in technique.

I told myself to forget the high level, to abandon my comfortable and familiar top-down design process, and focus instead on a smaller part of the problem--namely one of the subsystems that I understood pretty well. I decided to jump right into building this lower level component, hoping that on the other side I'd not only have the component that I need, but also have a clearer idea of the overall system design.

It worked! Taking a bottom-up approach, moving from the abstract on the white board into some concrete subsystem coding, made the top-down view of things become clear, almost magically. By the time I had finished coding this one DLL, even though I didn't touch the white board or consciously run through any additional high-level-design attempts, the system design was clear to me. When I was done with the DLL, I walked to the white board and sketched out the whole system design. I'm still not sure how it happened.

This delayed epiphany was no doubt influenced by simply setting the problem aside for some time, letting my subconscious mind work on it a bit. But I have the feeling it was more than that. My subconscious had already had three nights to chew on this one, after all.

I think somehow my bottom-up view of things was more revealing than the top-down. It might have also been that working at the bottom levels of my problem for awhile allowed me to clear some clutter, to get rid of questions that were distracting my from the questions I really needed to focus on.

I'll be paying attention to this phenomenon going forward to see if I can find ways to improve my personal design process. What is your process?

Dan