Open Comments Thread for "Code As Design: Three Essays by Jack W. Reeves"

I wouldn't have expected so much resistance to these ideas, since they ring so true to me.

If one attends a few status meetings with hardware engineers, it seems clear that their designing and prototyping is somewhat equivalent to our software design, prototyping, programming, and unit testing efforts combined. And their working with Manufacturing so that Manufacturing can learn to build the product is equivalent to our working with the CM group to build the product.

What about the less-able programmer: Higher-quality programmers that seem expensive are really less costly, and we shouldn't hire less-able programmers for the same reason that we shouldn't hire a doctor or a lawyer using cost as the only consideration.

This is bang on but there are few in this industry who will get it. Hopefully the failures of those who refuse to acknowledge reality, and the successes of those who do understand this, will act as a Darwinian Gauntlet; survival of the fittest.

Perfect.

This is great stuff, Jack, and thank you for making these points so clearly. As a developer with over 20 years in this biz, I concur with you on every point -- and I'd like to add two of my own.

To continue your parallel with physical engineering, the most expensive part of the build process in the physical world is the cheapest part in the software world. Every developer has at his fingertips the most powerful machines on the planet for building software. In fact, these machines sit idle 90% of the time, just waiting for him to issue commands.

In the physical world, big machines with which to build infrastructure and transport goods are the most expensive and rarest machines of all and the capital they require prevents everyone but the largest corporations from playing in the big leagues. In the software business, any teenager can buy a PC on eBay for next to nothing and can fully justify that cost with other uses, like word processing or internet access. That same machine with its 90% free time is capable of building software that competes on an even keel with Microsoft. Our industry is unlike any other because our designs are so expensive while our tools are incredibly cheap. Our products themselves cost nothing and can be replicated infinitely for free.

If software for the developer is only design, then software for the user is only interface. Every part of the software, including those things developers call internals, data, design, features, bugs, installation, settings, operating system issues, hardware limits, etc. are all, to the user, just interface.

From the user's point of view, the key failure of software is that it simply works poorly. The traditional process of building software detaches programmers from the problem space, adding an impassable layer of abstraction. In order to create the next generation of better software for the future, programmers must become more involved in the business processes their code represents and regular folks must take a more active role in creating and coding programs.

The key to achieving both of these goals, better designs and better interfaces, lies in the construction of new programming languages and tools which have these ends in mind. In my view, this is the most important work that can be done in the software field today because it has the biggest potential to improve the software experience for the largest number of users, which in turn drives revenue in our industry and benefits us as developers.

-- D

David A. Bethune
david@trelliscorp.com

Thanks for writing this very accurate article, I totally agree, I'd like managers to be aware of this... but the (wrong) manufacturing metaphor is still in everybody's mind !

Cyrille

I agree with the underlying point being made in the article that there is no substitute for expertise.

Nor is there a substitute for having people with the word "Engineer" on their college degree(s) when you assign those people to "Engineer" something; software or otherwise.

How many Software "Engineers" really exist? Likely 2 orders of magnitude less than those that have it on their biz card I suspect.

Being an Engineer is not like choosing a nickname for yourself because you like the sound of it.

If you are really an Engineer you have been trained to think and behave like an engineer in an accredited Engineering program. To prove it you better have a college degree with the word "Engineer" on it.

If not...just as you are not a medical doctor, you are *not* an Engineer; software or otherwise. You may be a SW coder, a SW programmer, SW technician, etc. You may indeed be fantastically talented at those roles, a great asset to your employer
and even have many natural tendencies to think and behave like an Engineer.

But you are not an Engineer.

Engineer is a title reserved for those who have earned it suffering through 2 years of University Physics, 2 years *minimum* of Calc and Diffy Q and another 2 years of multiple death march weedout classes
in their engineering focus area. It's a 4 year, if you are lucky, mental bootcamp that changes the way you think about everything. Every *real* Engineer will tell you the same thing.

If the SW industry ever figures out that they need to require that architecture and design is done by an elite class of tested and accredited professionals with the same "esprit de corps" that exists
with *real* Engineers we will start to head in the right direction.

You can't just let anybody with a case of Jolt, a bag of doritos, a sunless complexion and a keyboard cook up mission critical SW that someone's life may be depending on.

Until then the SW industry will remain the laughingstock it has been for quite a while now.

My favorite soapbox since 1995. :-)

Roger T

I've been programming for over eight years and I have read many articles and books about software design. In all that time and all those words, I've never read a more concise and accurate explanation of the reason that software development processes, when put into practice, bother me so much.

Mentally, I've always approached software development as a sort of perpetual design process. That's why the so called "spiral" method of development made some academic sense to me. My frustration lies in the reality that, (in my experience), software managers have an apparent need to schedule development around very distinct "phases" of development, usually "design", "implementation", and "test", along with associated "code freezes", which they think will "stabilize" the bits. I've never been able to articulate why this bothered me so much, until now. So, "THANK YOU!" Jack, for articulating it so well.

One of the dangers of not considering implementation, testing and debugging as part of software DESIGN is usually manifest when the budget gets tight. In my experience, the first thing that takes a hit is testing. Dedicated software testers, (whom we all know do a better job of testing than we developers), tend to get laid off. Then, developers end up testing their own code, which means we have less time to fix bugs, which means less time to "fix" our design, which means we compromise our end product, which means we generate more support calls, and on and on and on.... (There is much more to talk about here, but I'll spare everyone).

Everything Jack said in his article(s) rings completely true to me. They are things that I now realize I have subconciously internalized, but have never been able to explain clearly. I'm not sure why, except that maybe my indoctrination into the world of software development as a junior programmer included not rocking the boat. After all, I was just a newbie, - what could I possibly have understood about software development, right?.

Anyway, I now feel as though my past rants about software development were not merely born of frustrating corporate environments. I only wish I had run across the original article years ago. The article(s) should be required reading for anyone interested in software development, from "mere" programmer, (no, we are not just assembly line droids), to project managers.

John Davies
jcdavies@pobox.com

IMO, design is about the separation of "what" from the "how", as far as possible. It's a valuable process, because it clarifies thought and intent. It facilitates communication, and removes hidden pitfalls. Eric Evans has suggested that a crucial part of the design process is establishing a common language (i.e. set of jargon) that all the participants in a project, from non-technical users through to developers, can use to describe the problem domain. If you can't do this, the chances that you are all talking about the same thing are slight.

So code that reveals intent is valuable, because it relates the code to the design process. If code were sufficiently well-written, it *might* completely reveal the design, as some have suggested. That's true. Then again, it may not. Either way, there has to be a design to be revealed.

The waterfall process might work, if it were not for two almighty problems.

First, design is *very hard* to communicate. The only place that "design" really exists (if anywhere) is in our minds. UML diagrams are not design, code is not design, documentation is not design. But they are all useful tools for communicating it, and we need all the help we can get.

Second, it's hard to know when you have successfully specified all th "what" without impinging on any of the "how". IMO, this is crucial. In mature fields like architecture and engineering, there are conventions on where you stop. In software, a world of constant change, it's essentially impossible, at the current time.

So in practice, an iterative process is much better. The bidirectional communication constantly refines mutual understanding, and the boundary between "what" and "how" can be discovered rather than guessed at.

Well, that's my 2c. All comments welcome.

Jack,

Congratulations, and thanks, for writing another brilliant article on this topic! (Will we have to wait another 13 years for the next one? ;-)

I particularly liked the section on "What do we do with the less able physician?". I think the desire to substitute process for talent (or at least to use process as insurance against lack of talent) is a huge problem in the industry. Those who believe it blame their failures on the process, so they try harder to use an even better process next time, and the vicious circle continues. We're left with big development processes and small recruitment processes -- recruitment processes which are underdeveloped, largely ignored, and exist only to poke "resources" into the development process.

John

This was the same view I had before being rudely shocked by people who had learnt software engineering.

Jack, I'm glad I had the opportunity to read through your views on software design. Your insights have helped me to crystalize my own view points on the nature of my profession.

I'd like to relate my experiences to you, hoping to add another voice to chorus seconding your views.

About five years ago, I joined a rapid application development team that was floundering. We were a loose affiliation of contractors and junior programmers without a guiding force. After I had been with the team for about six months, I suggested that we start creating formal designs for our projects. At the time, I thought I was doing it to help ensure that the code we generated was more correct and so our development times would be shorter.

It turned out that even though we generated the documents and went through the design cycle, we still weren't able to produce software as reliably and quickly as I thought we could. Shortly after this round of design documents were produced, we went through the industry-wide IT purge and lost roughly half of our developers. After the purge happened, we stopped doing the formal design documents, but the tools we built were built faster and with fewer bugs - even though we didn't have time to go though the formal process.

I think the real key is in the "Less-Able" Programmer concept. Looking back now, I think the reason I pushed so hard for design documents is not that I felt like I needed to have a design document, but that I wanted the less experienced programmers or the weaker developers to have to go through the process so that their designs could be reviewed and modified as early as possible. Perhaps I even thought that I could teach these people how to improve their software designs.

In the end, though, even though design documents were written and reviewed and approved, the people responsible for building the software couldn't do it. They wrote the documents, but still couldn't write the code. And this was for a design that they had written. I find it hard to imagine that any large software projects ever succeed that have different teams doing design and implementation.

I've been thinking alot about the Less-Able progremmer concept in relation to CMM levels as well, because there is a push at the company I work for to attain CMM level 3. I believe that you need the overhead of CMM level 3, 4, or 5 class processes to ensure that the producitivity and quality you get out of each developer is the same. That's not to say that you raise the productivity and quality of all your developers - just that all your developers look alike. This helps to build schedules, timelines and budgets, but doesn't do a whole lot to build better software faster. I'm not saying that having a process is bad, just that large, cumbersome processes do not accentuate the abilities of great developers - it mitigates the failures of "Less-Able" developers.

Eric

I enjoyed the article because it is an exploration of questions and issues that I have pondered over myself. The comment that I want to raise is more of a spin-off from your article than a direct comment. There's a lot of debate about how to develop software by people who are actually doing the work. What I find difficult and frustrating is the fact that the work we do is often ultimately commissioned by people who have no programming background. Because they don't understand the nature of the process which delivers them a piece of software they are often the people who drive unreasonable expectations for projects.
I like the article because it's written (I think) in pretty simple terms and could be understood by people who aren't programmers. (You don't need a degree in Computer Science to understand the 'less able physician' analogy, for example.) Maybe articles like this and debate about articles like this should be part of the curriculum for people who study law, accountancy, etc - basically anyone who might end up as a project sponsor or a decision-maker for an IT project.

The article says how C++ is a very expressive design language. That may be true, but I don't understand why the world doesn't give Smalltalk the recognition it deserves. There is no more expressive and source-code-as-documentation language in existence that surpasses Smalltalk. It's object-oriented and has been around since the 70s, so C++ is not exactly a breakthrough in comparison. Not to mention it's infinitely easier to read and debug and making something useful out of it can be done very quickly. Many claim that Smalltalk just isn't fast enough to compete with C++, but there have been virtual machines made for Smalltalk that make it just as fast, suitable for embedded systems. But somehow this beautiful language never really caught on, and that is real shame, because this language can really fulfil the criteria for advances to which Mr. Reeves refers.

I'll buck the trend here. Most of both (very nicely written) articles are based on a basic assumption: source code listings (in your choice of programming language) are the only things that can be debugged, hence the only things we can have confidence in.

The world of domain-specific languages and transformations allows us to express (solutions to) problems as models (text or graphical, but not what you would normally consider a 'source code listing') expressed in the language of the problem domain, and to test and debug at that level.

Somewhat separately we describe the mappings to other (implementation) languages. These mappings can also be more incomplete, customizable, or requiring explicit guidance than the command-line switches of your typical programming language compiler. And they can cover much more than c++ to machine code: they can target languages for include platform configuration, deployment, security, and more.

All creative parts of "design" are then located either in the choice and design of the languages themselves, in the models that describe the (solutions to) problems, and in the mappings to other languages.

I think the 'Source Code Listing' view will need to be stretched quite a bit to accomodate this.

I'm not sure I agree that the definition of source code needs to be overly stretched to incoroprate the artifacts you mention. I'm not 100% sure that I understand what it is that you're talking about re: "the world of domain-specific languages and transformations", but I believe that the definition of source code should be basically anything that is used to build your final product. This obviously includes such things as configuration files and build scripts. And I think it should also include any documents or artifacts that are used to generate other source code.

For example, if you create UML diagrams and then use these to generate basic source code files and subsequently modify the UML and regen the code, then your UML and your generated code are both source code. Further, the argument could be made that if you simply compile and use the generated source code files as-is, without modifying them, then the only source code is the UML. The generated "code" files are more along the lines of compiled class files--the results of your source code, and not the code itself.

However, if you simply create UML diagrams to guide you in the design and writing of your code (in other words, nothing automatic happens) then your UML diagrams are documentation and not source code.

This may sound like so much semantic tap dancing, but I think there's an important distinction to be made. The term "source code" should not be limited to simply programming language code. A broader understanding of what source code is helpful in understanding the ideas behind agile programming and xp.

In the following code fragment, what is the design?

int multiply(int a, int b)
{
  /* Returns the product of a and b */
  return a + b;
}

Surely a typo does not count as a design, as much as a missed brush stroke by a master painter. Source code is the outcome of the design phase of a project, plus errors, omissions, and workarounds. As computer power increases, level of abstraction is raised, allowing the programmer to "design" at a different level, while the design decisions that used to have to be made at a lower level are done with prebuilt software libraries or optimized by specialized software such as compilers, runtime optimizers and query plans.

Brilliant...yet it's so simple I find it difficult to understand how there has been so much opposition to the concept. It is fundamentally in line with concepts developed and used in all other engineering disciplines.
As Software Engineers we have a distinct advantage and distinct disadvantage over our engineering counterparts. The distinct advantage is that the computers we use are powerful enough to "build" our product very quickly and cheaply. This makes concepts such as TDD (Test Driven Development...which should be more appropriately known as Test Driven Design) so powerful because we can immediately see the impact and validity of our design. The distinct disadvantage we have is that we have to make our design readable to a machine...and currently that machine has a limited vocabulary and cannot make decisions on there own. We have to fill in most of the blanks. Imagine the size of construction design documents if it was necessary to tell the manufacturers how the wood was to be made...or the steel.

IMO,the development process goes through follwoing phases.

problem specification
[there may be many ways to solve a problem ]
|
^
we select one way to solve the problem
[generally accepted as the top level design.
but the top level needs refinements..for example
at algorithmic level]
|
^
We accept ceratin solutions and stick to them
[generally accepted as the detailed design.From this
point or roughly at this point we will be
concerned about the implementation
(analogous to which materials to use
in construction), we might choose some
technology, language ,platform or what
ever which is feasible and acceptable.]

|
^
we implement the detailed design.some
refinemnts may be made here as well(some thing
specific to the technology used).
[please note that from the top level we may have
a handful of solutions for the detailed design,
or from the detailed design a few other ways
to implement the design available but as we get
more and more specific we get a ready to build stuff]
|
^
Just produce the output!

*) Now its up to us to pick what is the design. I would say the detailed design docs are the actual design statements.From this point onwards we worried only about a specific product [ie, we have already made the choice for the materials that we are going to use in the construction process].

*) I can point to the implementation output[the code] and say that this is the design for this particular build.Thats it.

*) the implementation is no mean job, sheer choices and complexity associated with this in itself is 'the' challenge.

*) this is pretty anaogous to a motor car design.Whats the design here. Is it the final spec that says -
a)use 20mm nuts here,use aluminium alloy to coat the surface at .2 mm thickness?
OR
b) say the detailed design saying the nut size can be 2mm-4mm[essentilly use 2.5% of the length of the bolt]

I would say option b is simple and every body understands.we maintain sufficient abstraction but we have a certain level of specification as well.

#1)Jack is right if we say that this is the design for version 2.1.1 of SOFTWARE.

Its a matter of choice.who ever we are either belonging to group1,2or group 3 our aim, as well as the end users aim, is to make say release 10.0.0 a complete one.

many thanks,
+ashly

One of 'THE' best article about software design. For a person who wants to be a good programmer, he should be able to read others WELL written programs.

Which is the good code base that you guys can think of (ofcourse, open source), please share your thoughts.

As computer power increases, level of abstraction is raised, allowing the programmer to "design" at a different level, while the design decisions that used to have to be made at a lower level.