developer.* Blogs - Professional and Personal Development - Comments

working software <-> good software

Sven Busse — Sat, 02 Dec 2006 00:47:48 -0800

Hi,

reading your article, Dan and then the comments, i feel, two things here might be mixed up. For me, the original apgar score sounds like a good way to see, if the baby is actually well and alive, to see, if the baby is 'working'. The apgar score doesn't tell you, if the brain has any damage, if the kidney is working and many more things. So i think, the apgar score is good in telling, if the baby will survive the next days but it cannot tell, if the baby will develop to a healthy grown-up, and i guess, it wasn't meant to.

The same is for software. We already have instruments for knowing, if a program 'works'. Unit Testing, for example. Unit tests have the binary system, that you mentioned. It tells you, if that thing is running or not. But of course, it doesn't tell you, if the design of the software is good, if the architecture is good.

It is the same, as the apgar score doesn't tell you, if the baby has a high potential of getting cancer in the future.

To get this information, the baby would have to be checked very intensively and the resulting report would surely consist of many subjective opinions.

The same goes for analyzing a programm in terms of desgin and architecture.

My 2 cents :o)

Cheers

Sven

Google Video

Steve Benz — Fri, 17 Nov 2006 19:05:57 -0800

That was quite an interesting talk! At one point the speaker calls out data that seems to conclusively show that, almost all of the time, the committer turns out to be a useless predictor of whether a commit will be buggy or not.

Honestly, I'm not entirely surprised. I think good programmers can be as error prone as bad ones. What I think distinguishes a good programmer from a bad one is the survivability of their code. That is, the good coder will check in reusable components that last in the product, unchanged, for many years. The bad programmer will treat each new challenge as a separate entity and never write much code that is reused. When their code does persist, it's usually because it becomes dead code.

Bugs via SCM

Siddharta Govindaraj — Tue, 14 Nov 2006 19:48:13 -0800

Hi Pat, there was a video on Google video about predicting bugs using source control information. Not sure if its the same one. The video is here.

I detect a self-contradiction

Edward G Nilges — Tue, 14 Nov 2006 17:40:11 -0800

See my extended post "Squaring the Circle"

similar ideas

Pat Eyler — Tue, 14 Nov 2006 16:34:57 -0800

In his OSCon 2006 presentation 10 Tools for Open Source Projects, Karl Fogel presented some similar ideas about using tools to evaluate bug reports and repository commits to screen potential commit-bit cendidates or other folks who would be good to groom for a bigger role in an open source project. I can't find any links to his talk right now, but I think it was released as an mp3 somewhere.

CVS - the Vessel of The Awful Truth

Steve Benz — Tue, 14 Nov 2006 14:01:37 -0800

I tend to agree with Chris that there's perhaps more value in repository changes than many other metrics. My personal take on it is that it might be a good idea to look at who writes code that gets delta'ed alot. Check out my blog entry on the subject for a complete musing on the subject.

Balanced scorecard

chrishmorris — Sat, 04 Nov 2006 22:16:25 -0800

At a recent management review, I was asked a great question: "Name a feature of the recent release which was directly requested by a customer". A software Apgar should include some indicators of whether the right software was built.

It's true that all metrics can be gamed. This is a direct consequence of the fact that what we do is hard. But it doesn't follow that they are useless.

I use number of commits per month as an indicator of team morale. It's a very crude one, but it has some utility. Either it confirms what I already knew - but in an objective way, that I can quote, my bosses don't want to rely on my gut feelings. Or it contains a surprise, in which case I have to do some listening and thinking. Sometimes the conclusion is that the result is an artefact, e.g. due to holidays. But any reality check is good, even an imperfect one.

On important guideline is not to reward the team for improving the metrics. That is asking for trouble. A objective measure of the fitness for purpose of the product is not possible.

Incidentally, I'm sceptical about the idea that doctors can't game the Apgar score. We'd have to ask an obstretrician, but babies are more complicated than software products. In the UK we have free health care, which also means rationed health care, and we have lots of experience of hospital administrators gaming targets.

Getting Us Started

Daniel Read — Fri, 03 Nov 2006 15:28:12 -0800

Thanks, Dave! I hope to post a new blog post soon to summarize some of the things that have been said in these discussions, and to start exploring some actual "score-able traits" (like those in your list), program classifications, and systems for scoring.

Best,
Dan

Other categories

Dave Aaron Smith — Fri, 03 Nov 2006 14:37:24 -0800

I bet it's best to pick the priorities for your project, and then devise a unique set of tests before you get going. That being said, here are some guidelines that I wish were in place for every project I've worked on.

1. No stupidly big code files, say over 1000 lines.
2. No commented out code.
3. No unused files, functions, or lines of code (this one could be tricky to enforce)

The Joel Test is Process-Oriented

Daniel Read — Fri, 03 Nov 2006 08:35:43 -0800

Mark, thank you for pointing out The Joel Test. I had read this essay by Joel a couple times (once online, and once in the Joel on Software book), and it didn't even come to mind in my thinking about this Apgar score idea. Now, thanks to your link, I've read it a third time, and I can see why it did not come to mind: Joel's score is process-oriented. It measures, "how do you go about building software?" as opposed to, "how does this particular piece of software measure up to some idea of minimum quality?"

I don't mean this as a criticism of Joel's Test, nor as a criticism of your suggestion. I do, however, see the software Apgar as more about the *what* than the *how*, if that makes sense. There is a relationship, of course, between the what and the how. As I understand it, in obstetrics the Apgar does not judge hospital procedures, doctor and nurse training, sanitary conditions, equipment quality, etc. But of course, the how influences the what. If a maternity ward is doing a poor job with procedures, training, cleanliness, and equipment, then the babies it brings into the world might have lower Apgar scores. (Though of course it's more complicated than that, with things like the diet and health of the mother for the previous nine months having a huge influence.)

I really like this statement from your comment, Mark:

But the Apgar doesn't tell you if the baby has a tumor or genetic heart problem. It isn't trying to replace MRI machines.

Thanks again for the contribution.

Dan

The Joel Score

Mark — Thu, 02 Nov 2006 13:44:25 -0800

The Joel Test is probably at a higher level than you were intending, but I think it is pretty quick and pretty telling.

Yes you can game it and the code can still be horrible, same for ISO-9000 and CMM. But the Apgar doesn't tell you if the baby has a tumor or genetic heart problem. It isn't trying to replace MRI machines.

I think we should improve on The Joel Test (e.g. automated unit tests, quality/quantity of documentation, etc.).

Score candidates

Ron Porter — Thu, 02 Nov 2006 10:35:47 -0800

Here are some candidates. Assume everything is worth one point, for now.

Error Handling
Error Logging
Comments that make explicit reference to the requirement supported by the code in question
Descriptive names
Internally consistent naming convention

I picked these because I can imagine taking any non-expert and bringing them up to speed on the scoring system in a couple of hours. Thus, I left out things like global variables, because that might require a level of expertise that may not be available. I'm thinking in terms of a system that the client can use to score the program. I'm not sure about 'descriptive names' because there is a subjectivity about it that may be at odds with objective scoring.

Another reason for picking these was because it isolates the 'code score' from the 'program score' in the same way that Apgar separates physical health from mental and emotional health. Not that we should be ignoring other factors, but other issues probably require different scoring systems.

See my blog if you like on this matter

Edward G Nilges — Thu, 02 Nov 2006 09:00:21 -0800

See my blog post "Apgar score: less than zero"

[Editor's note: for those interested in following the discussion, an interesting thread has also sprung up on Edward's blog post, starting with this comment by regular developer.* reader Ron Porter.]

Scorable Traits

Daniel Read — Thu, 02 Nov 2006 07:01:02 -0800

Since the topic of my post falls squarely into his area of expertise, I emailed a link to Robert L. Glass. He was kind enough to read it and reply with some comments, which he agreed to let me share:

A thought-provoking post. Your original idea is a good one, but/and your readers have done a good job of identifying the problems.

I think the key to making this work is easily-scored traits. And that's a HUGE problem. As you point out, metrics are different from scores, and I think they're a wrong path to go down. (Metrics are also hugely controversial--far too many are more a gleam in an academic's eye than anything meaningful to practice). Perhaps someone should hold a contest to identify software Apgar candidates. I could do something in The Software Practitioner. Want to put together an article?

You glossed over something else that I think is important, the idea that "nothing ought to be introduced into practice unless it has been properly tested and proved efficient by research centers." Implementing that idea alone would make for dramatic changes in how we do computing research; it's an idea I've been arguing for (unsuccessfully!) for far too long. That may not result in an Apgar either, but it would sure improve both how practice functions and how research and practice interact.

I agree with Mr. Glass that coming up with the list of "easily-scored traits" is key, and appears to be the next step in taking this idea further--and should indicate whether there's anywhere further to go. I might try getting a list going with developer.* readers first, and then perhaps we can open it up to Bob's Software Practitioner readers if we have something going.

The issue raised in the last paragraph of Bob's comments is interesting in that Bob and I seemed to draw opposite conclusions. Here was my comment back to Bob:

One of the reasons I included that quote from the article about "nothing ought to be introduced into practice unless it has been properly tested and proved efficient by research centers" is that the article seemed to be crediting obstetrics with being successful in spite of (or even because of) this lack of pre-confirmation. This reminded me of the general disconnect between software academe and practice, and also of the "practice often precedes theory" idea that you have discussed in your writing.

Thanks for reading,
Dan

P.S.
I can't resist getting in a plug for the two books by Robert L. Glass published by developer.* Books, Software Conflict 2.0 and Software Creativity 2.0.

The baby and the bathwater

Edward G Nilges — Wed, 01 Nov 2006 18:24:06 -0800

The problem with an Apgar score is that the Apgar score can be checked against and supplements the physical baby as the poor little tyke squirms in the nurse's hands, poor little perisher that he is.

You can't "game" the score.

Wherease software is a pure or *reinen* Idea on which we must focus, and gaming our Apgar score is a distraction.

Ultimately, a work of software must be judged as we judge a literary work, with the added constraint that it must be mathematically correct. This is the hard fact which "software engineers" try to avoid, because it means that the only qualified assessors of software are either collectives of users, or individuals who have been spared educational moronization and tracking into "SAT Verbal" or "SAT Math".