developer.* Blogs - Does Software Need an Apgar Score? - Comments

working software <-> good software

Sven Busse — Sat, 02 Dec 2006 00:47:48 -0800

Hi,

reading your article, Dan and then the comments, i feel, two things here might be mixed up. For me, the original apgar score sounds like a good way to see, if the baby is actually well and alive, to see, if the baby is 'working'. The apgar score doesn't tell you, if the brain has any damage, if the kidney is working and many more things. So i think, the apgar score is good in telling, if the baby will survive the next days but it cannot tell, if the baby will develop to a healthy grown-up, and i guess, it wasn't meant to.

The same is for software. We already have instruments for knowing, if a program 'works'. Unit Testing, for example. Unit tests have the binary system, that you mentioned. It tells you, if that thing is running or not. But of course, it doesn't tell you, if the design of the software is good, if the architecture is good.

It is the same, as the apgar score doesn't tell you, if the baby has a high potential of getting cancer in the future.

To get this information, the baby would have to be checked very intensively and the resulting report would surely consist of many subjective opinions.

The same goes for analyzing a programm in terms of desgin and architecture.

My 2 cents :o)

Cheers

Sven

CVS - the Vessel of The Awful Truth

Steve Benz — Tue, 14 Nov 2006 14:01:37 -0800

I tend to agree with Chris that there's perhaps more value in repository changes than many other metrics. My personal take on it is that it might be a good idea to look at who writes code that gets delta'ed alot. Check out my blog entry on the subject for a complete musing on the subject.

Balanced scorecard

chrishmorris — Sat, 04 Nov 2006 22:16:25 -0800

At a recent management review, I was asked a great question: "Name a feature of the recent release which was directly requested by a customer". A software Apgar should include some indicators of whether the right software was built.

It's true that all metrics can be gamed. This is a direct consequence of the fact that what we do is hard. But it doesn't follow that they are useless.

I use number of commits per month as an indicator of team morale. It's a very crude one, but it has some utility. Either it confirms what I already knew - but in an objective way, that I can quote, my bosses don't want to rely on my gut feelings. Or it contains a surprise, in which case I have to do some listening and thinking. Sometimes the conclusion is that the result is an artefact, e.g. due to holidays. But any reality check is good, even an imperfect one.

On important guideline is not to reward the team for improving the metrics. That is asking for trouble. A objective measure of the fitness for purpose of the product is not possible.

Incidentally, I'm sceptical about the idea that doctors can't game the Apgar score. We'd have to ask an obstretrician, but babies are more complicated than software products. In the UK we have free health care, which also means rationed health care, and we have lots of experience of hospital administrators gaming targets.

Getting Us Started

Daniel Read — Fri, 03 Nov 2006 15:28:12 -0800

Thanks, Dave! I hope to post a new blog post soon to summarize some of the things that have been said in these discussions, and to start exploring some actual "score-able traits" (like those in your list), program classifications, and systems for scoring.

Best,
Dan

Other categories

Dave Aaron Smith — Fri, 03 Nov 2006 14:37:24 -0800

I bet it's best to pick the priorities for your project, and then devise a unique set of tests before you get going. That being said, here are some guidelines that I wish were in place for every project I've worked on.

1. No stupidly big code files, say over 1000 lines.
2. No commented out code.
3. No unused files, functions, or lines of code (this one could be tricky to enforce)

The Joel Test is Process-Oriented

Daniel Read — Fri, 03 Nov 2006 08:35:43 -0800

Mark, thank you for pointing out The Joel Test. I had read this essay by Joel a couple times (once online, and once in the Joel on Software book), and it didn't even come to mind in my thinking about this Apgar score idea. Now, thanks to your link, I've read it a third time, and I can see why it did not come to mind: Joel's score is process-oriented. It measures, "how do you go about building software?" as opposed to, "how does this particular piece of software measure up to some idea of minimum quality?"

I don't mean this as a criticism of Joel's Test, nor as a criticism of your suggestion. I do, however, see the software Apgar as more about the *what* than the *how*, if that makes sense. There is a relationship, of course, between the what and the how. As I understand it, in obstetrics the Apgar does not judge hospital procedures, doctor and nurse training, sanitary conditions, equipment quality, etc. But of course, the how influences the what. If a maternity ward is doing a poor job with procedures, training, cleanliness, and equipment, then the babies it brings into the world might have lower Apgar scores. (Though of course it's more complicated than that, with things like the diet and health of the mother for the previous nine months having a huge influence.)

I really like this statement from your comment, Mark:

But the Apgar doesn't tell you if the baby has a tumor or genetic heart problem. It isn't trying to replace MRI machines.

Thanks again for the contribution.

Dan

The Joel Score

Mark — Thu, 02 Nov 2006 13:44:25 -0800

The Joel Test is probably at a higher level than you were intending, but I think it is pretty quick and pretty telling.

Yes you can game it and the code can still be horrible, same for ISO-9000 and CMM. But the Apgar doesn't tell you if the baby has a tumor or genetic heart problem. It isn't trying to replace MRI machines.

I think we should improve on The Joel Test (e.g. automated unit tests, quality/quantity of documentation, etc.).

Score candidates

Ron Porter — Thu, 02 Nov 2006 10:35:47 -0800

Here are some candidates. Assume everything is worth one point, for now.

Error Handling
Error Logging
Comments that make explicit reference to the requirement supported by the code in question
Descriptive names
Internally consistent naming convention

I picked these because I can imagine taking any non-expert and bringing them up to speed on the scoring system in a couple of hours. Thus, I left out things like global variables, because that might require a level of expertise that may not be available. I'm thinking in terms of a system that the client can use to score the program. I'm not sure about 'descriptive names' because there is a subjectivity about it that may be at odds with objective scoring.

Another reason for picking these was because it isolates the 'code score' from the 'program score' in the same way that Apgar separates physical health from mental and emotional health. Not that we should be ignoring other factors, but other issues probably require different scoring systems.

See my blog if you like on this matter

Edward G Nilges — Thu, 02 Nov 2006 09:00:21 -0800

See my blog post "Apgar score: less than zero"

[Editor's note: for those interested in following the discussion, an interesting thread has also sprung up on Edward's blog post, starting with this comment by regular developer.* reader Ron Porter.]

Scorable Traits

Daniel Read — Thu, 02 Nov 2006 07:01:02 -0800

Since the topic of my post falls squarely into his area of expertise, I emailed a link to Robert L. Glass. He was kind enough to read it and reply with some comments, which he agreed to let me share:

A thought-provoking post. Your original idea is a good one, but/and your readers have done a good job of identifying the problems.

I think the key to making this work is easily-scored traits. And that's a HUGE problem. As you point out, metrics are different from scores, and I think they're a wrong path to go down. (Metrics are also hugely controversial--far too many are more a gleam in an academic's eye than anything meaningful to practice). Perhaps someone should hold a contest to identify software Apgar candidates. I could do something in The Software Practitioner. Want to put together an article?

You glossed over something else that I think is important, the idea that "nothing ought to be introduced into practice unless it has been properly tested and proved efficient by research centers." Implementing that idea alone would make for dramatic changes in how we do computing research; it's an idea I've been arguing for (unsuccessfully!) for far too long. That may not result in an Apgar either, but it would sure improve both how practice functions and how research and practice interact.

I agree with Mr. Glass that coming up with the list of "easily-scored traits" is key, and appears to be the next step in taking this idea further--and should indicate whether there's anywhere further to go. I might try getting a list going with developer.* readers first, and then perhaps we can open it up to Bob's Software Practitioner readers if we have something going.

The issue raised in the last paragraph of Bob's comments is interesting in that Bob and I seemed to draw opposite conclusions. Here was my comment back to Bob:

One of the reasons I included that quote from the article about "nothing ought to be introduced into practice unless it has been properly tested and proved efficient by research centers" is that the article seemed to be crediting obstetrics with being successful in spite of (or even because of) this lack of pre-confirmation. This reminded me of the general disconnect between software academe and practice, and also of the "practice often precedes theory" idea that you have discussed in your writing.

Thanks for reading,
Dan

P.S.
I can't resist getting in a plug for the two books by Robert L. Glass published by developer.* Books, Software Conflict 2.0 and Software Creativity 2.0.

The baby and the bathwater

Edward G Nilges — Wed, 01 Nov 2006 18:24:06 -0800

The problem with an Apgar score is that the Apgar score can be checked against and supplements the physical baby as the poor little tyke squirms in the nurse's hands, poor little perisher that he is.

You can't "game" the score.

Wherease software is a pure or *reinen* Idea on which we must focus, and gaming our Apgar score is a distraction.

Ultimately, a work of software must be judged as we judge a literary work, with the added constraint that it must be mathematically correct. This is the hard fact which "software engineers" try to avoid, because it means that the only qualified assessors of software are either collectives of users, or individuals who have been spared educational moronization and tracking into "SAT Verbal" or "SAT Math".

Balance in Scoring

Daniel Read — Wed, 01 Nov 2006 14:28:44 -0800

Brad writes:

For any metric you can design, I can write crappy software that meets the metric perfectly.

and Andy writes:

I'm with you on a generalized distaste for metrics (lies and damn lies, if I may paraphrase Twain). And so this notion kinda falls into that hole for me. BUT... if an effective rubric could be developed, one that subtlety enforced overall quality via the total impact of each scored component, you might have something.

Fair points. I think the key to making something like this work is balancing objectivity, subjectivity, and observability. Brad has a good point that, "we know how a baby is supposed to be." But, allowing for variations among different classifications of software programs, I don't know if I'm ready to concede that we can't identify some fundamental qualities or inclusions that a software program should have. Once we have that, there may also be a way to allow for a degree of subjectivity in the evaluation.

I also think it's useful to disconnect the concept of "score" from the concept of "metrics." I think people associate "metrics" with things like "how many bugs did the program have", "what percentage over budget are you on that task," and "how many bugs did you open and close today in the bug tracker?" and also with measurements of size and complexity. Counting function points and person-months interest me not at all. It may be important to disconnect the "score" from anything related to process, schedule, size, and complexity.

Perhaps the theme here is clearly defining the limits of the score, since "overall quality" type issues are so large, subjective, and hard to evaluate. On the other end of the scale, perhaps we should also avoid scoring elements that involve counting or "measuring." What I like about all of the Apgar score criteria is that they are all essentially binary, and easy to observe.

I do see a challenge in the relationship between items that are scored and requirements. For example, should we withhold extra points if the programmer did not implement logging as part of the exception handling scheme? Maybe the requirements didn't call for that, or maybe the programmer was expressly told not to add logging. Perhaps, though, if logging were seen as something important by virtue of the fact that it had been included in the scoring, then maybe stakeholders would empower programmers to make sure the system has logging, lest the score suffer.

My immediate reaction to Brad's statement, "For any metric you can design, I can write crappy software that meets the metric perfectly." was that if you included an error handling scheme in your crappy program just to get your score up, then that doesn't seem like such a bad thing. I suspect that the sentiment here is linked to that other association with "metrics" as a means of control in an organization to get people to work harder, produce more, and make fewer mistakes. My thinking about this score concept is, I think, separate from that--more of a bottom-up approach, in which programmers have a way of evaluating whether a piece of software meets some basic criteria; management can catch up with us later.

But I'm ready to let this idea go if it indeed does not have legs...

Dan

For any metric you can

Brad — Wed, 01 Nov 2006 13:48:39 -0800

For any metric you can design, I can write crappy software that meets the metric perfectly. Perhaps a score where software cannot be good without meeting the metric is possible and useful (two points for on time, two points for making into actual use, two for users NOT crying and having a heart rate over 100).

The problem is we know how a baby is supposed to be, and variation from the end result differs only by a few factors, all of which it is obvious if they are important or not.

Interesting, yes. Practical? Well....

Andy Tegethoff — Wed, 01 Nov 2006 13:05:59 -0800

This is definitely thought-provoking, esp. considered in light of an "After the Gold Rush"-flavored push towards more organization and standards in the field. But as you say at the end, the devil is in the details.

Would you need a basic Apgar score for pretty much all software, and then subsequent industry/context specific additions to it? Would there be a side-effect where more software gets written to meet a pre-determined score level, but still largely misses the wider notion of "quality"? I'm with you on a generalized distaste for metrics (lies and damn lies, if I may paraphrase Twain). And so this notion kinda falls into that hole for me. BUT... if an effective rubric could be developed, one that subtlety enforced overall quality via the total impact of each scored component, you might have something.

I guess I think overall, my doubt springs from the fact that very little in software is as simple as "is the baby moving all four limbs". Sure, there are some very basic elements that are pretty much true/false. But so many are contextually variable.

Nonetheless, thought-provoking as always, D!

Does Software Need an Apgar Score?

Daniel Read — Wed, 01 Nov 2006 11:31:15 -0800

This morning I was reading the Oct 9, 2006 issue of The New Yorker magazine, which contains an interesting article called "The Score: How childbirth went industrial." The article is about the process of childbirth, and more specifically, the medical techniques and industry, for lack of a better word, that have developed around childbirth. When I started reading the article, I was not expecting to encounter an intriguing idea related to software development--but ideas often spring from unexpected sources.