Live! from Orlando: 42 Million Lines of Code

42 million lines of code.

200 million installations.

2200 engineers.

10 hours to build clean.

Microsoft’s Russ Ryan opened his "What Goes in the Sausage: Lessons from the Factory Floor on How Microsoft Does Software Engineering" session with some impressive stats relating to Visual Studio and the .NET framework. With the sheer volume and complexity of the projects Microsoft tackles, it would seem the software giant would have some interesting words of wisdom (or horror stories) to share and Ryan was kind enough to do it. As customers we know that Microsoft's target dates have been fluid a time or two and some tolerance exists for shipping product with bugs. Yet, Microsoft knows that we know that and was still willing to invite guests into the house with dishes in the sink.

First, more on code names. Visual Studio 2002 was code named Las Vegas; VS 2003 : Everett, and VS 2005: Whidbey. Maybe everyone else had heard the story behind the code names, but alas, I don’t get out enough, so I had not. Apparently, originally Visual Studio 2003 was code-named Whidbey (an island 30 miles north of Seattle). When Microsoft realized they would not be able to deliver all the planned functionality in one product release, they decided to relabel the project Everett, a locale half-way to Whidbey. Incidentally, Orcas is another island further north than Whidbey.

Next, a few inside details about Microsoft’s development environment:

C++ and C# are the two common languages used in Microsoft.

Each product unit has 30-200 staff members.

Code that is shipped is supported for at least 10 years by company policy.

A central build lab is staffed 24 hours. The build process happens every day: it’s started about 4:00am and finishes about 1:00pm. (There are 40,339,207 functional test cases for Visual Studio 2005.)

All code check-ins (no matter how senior or junior the developer) must be code-reviewed by another developer prior to submitting any changes to the source tree.

If a developer gets to a specified number of bugs, he is in "bug hell" and is not allowed to work on anything else until they’re resolved.

The "war team" consists of senior members of the team. If any of the developers wants to fix a bug or make a change they tell the war team. When it's getting close to a release, they have to ask the war team if the change can be done and it has to be very critical and directly affect customers.

"Feature crews" are small, interdisciplinary teams who work together to design, code, test, and deliver specific product features.

Microsoft uses "scenario-driven" design.

Testing is very "intentional," starting with a test case.

So what are some of the lessons learned?

Shipping is a feature too. When deliberating which features to include in a release, they have to add the reality-check that meeting a reasonable ship date is as important a feature as the others.

The recognition that "cool technology" is not the same as "solutions to real customer problems."

Superman just doesn't scale. In the past one hero could pull a project out, but today's projects are too large and complex for that.

Load balance across units. (If one unit finishes their product early at Microsoft, they are shifted to another product unit; something that was unheard of in the past.)

Manual testing is expensive. You have to invest in more automation.

Once you check in the code, you are a hostage to it.

How has Microsoft changed, internally, as a result of lessons learned?

With Visual Studio 2005, Microsoft created:

Project "Ladybug" (code name): the product feedback center.

Self-sustaining communities (blogs, newsletters, forums)
Community Technology Previews (CTPs)

Technical Assistance Programs driven by (get this) the "evangelism organization." (It seems I wasn't too far off in my characterization of the Church of Microsoft in an earlier blog.)

Summary: While Jay Schmelzer might have won my arbitrary award for the most personality, Russ Ryan won the prize for sincerity. Even though I walked into the conference room with cold cynicism, thinking that having Microsoft lecture on software engineering would be akin to Jimmy Swaggart leading the sinner's prayer, I bought what he was selling. I came away with the impression that managing a software project is tricky, and managing countless numbers of projects with dependencies is crazy. Microsoft isn't perfect, but they are refining the process as they go. I hope I can say as much for my organization.