Deployment Fun (or How I Learned to Stop Worrying and Love Manual Processes)
So, yet another Release Day has come and gone here at Business X, with the usual fallout of slightly askew server files, missed scripts, improperly set-up DB tables, and the like.
It's all in a day's work, here at Business X: Home of the WHOPR (a little WarGames joke, for the kids). Actually, we don't have a WHOPR or anything like that, all vintage and AI-looking. That would be cool.... but only our codebase is "vintage". If you can call 1998 vintage, which you can't.
Nor is it a day's work, given that our release cycle is more like 2 months. But it sounded like a fun name, and I got to write "all in a day's work". But as usual, I digress...
The point isn't the errors resulting from the release--the team is good enough to have all those handled before any sort of customer exposure or backlash could come into the picture. The point, instead, is that it pains me greatly that any release errors should happen at all, and that the level of manual effort required to perform a release remains pretty constant after leveling off last year.
See, I'm the guy who inherited the job of "InstallShield Chump" here at B-X. I got it because the person who used to do it had amassed enough political capital (thanks to W for that phrase, which I looooovvve to overuse) to legitimately complain about it, and even to be acknowledged and listened to. Whoa. So, since I was the last person to squawk about how bloody awful our release process was, I pulled the duty without meaning to and definitely without wanting to. Therefore: I, InstallShield Chump.
And when I say "squawk[ed] about the release process," it should be clarified that "squawking" merely means providing my solicited opinions/observations about sitting here in my cube for 12 hours on a Saturday to help perform a task that should be largely automated to begin with. This happened less than 2 months after I joined the team here at B-X, when I was eager to bring my outside experience to bear on whatever problems I encountered. Well, you should have witnessed the flaming hatred of our config manager when I expressed how ridiculous a waste of person-hours I thought our release process was. You'd have thought I recommended keying her car twice daily until things improved. And so it came to pass I was tasked with improving the process, and so it was thusly improved. The longest anybody has spent here for any subsequent release is roughly half that time, and that was due to a server failure during the release itself. But then, after making the improvements, I became the I.S.C. For improving the process my reward was to be permanently shackled to it.
Still, even with my magic handcuffs, I'm pretty powerless to make the changes that need to be made. Part of it is the sheer number of moving parts in our system and the oddball architecture under which they're deployed. Part of it is organizational sluggardry and political geechiness (yes, I totally made those words up. I have a cold). But in our "post-mortem" meeting this morning I actually heard myself pointing out, in so many words, that tilting at this windmill has been done, that significant effort had been made but that a brick wall had been hit in terms of process improvement on our (the Dev team's) end.
So apparently, I've drunk the kool-aid without even knowing it. Or maybe I'm just learning to put my effort where it counts, and not hurtle headlong into that same old brick wall.
Or maybe it's just the cold talking...
Sincerely,
Your Working Boy
I sympathize
Thanks for the post.
But what about doing a DAILY build? Is that possible?
This is too my knowledge the only way around this misery.
Basically, in too many builds, we sit shiva. They are not fun like Irish wakes, instead the build becomes an all night festival of wailing and saying Kaddish over the corpse of a flawed system.
They are a perverted vigil, in which fathers and mothers neglect their family responsibilities because this is the oh-so-important Build on which the company's future rests.
I still dream about one firm in which I wrestled with a giant VB system. In my dream I am surrounded by every multiplying and interlocking pipes as in the Windows screen saver, in the bowels of a black freighter.
The irony is that all this modularity was SUPPOSED to be liberating, all these little handy dandy tools that we done be gots for free (since they were ripped off under eval agreements) that turn upon each other, fighting for space in the system, making the installation a dwarf toss.
The ONLY solution that I know is the daily build that starts on day one. I don't care if you build Hello World, JUST DO IT.
Oh, how I wish
"Continuous Integration" is a phrase I've heard and read.... somewhere in a far-away land of rainbows and lollipops.
I fear we are thwarted here by the proverbial "deficit of imagination". This is not only on the part of certain KDM's involved in config mgmt here at B-X, but also on the part of those people's managers. Strategy is not this company's strong point, and to see the benefit of daily builds would involve strategic thinking, b/c it would require resources to accomplish. And of course those resources are much better used in maintaining the break-fix cycle and for pushing fictitious deadlines...
There really are at least 2-3 other developers who feel the way I do, that there are easier ways to run a railroad. But, for reasons too long-winded and frnakly too depressing to go into, those people will always be shouted down.
Donna's appreciation of/sympathy with my shop's "issues" is noted. I share a conspiratorial and resigned sigh.
BTW - thanks for reading, guys. I'm no writer, so it's nice for you to even take the time....
Continuous
The company I work for is a pretty small consulting shop, and we've been able to get away with moderately well automated, semi manual build processes for a long time. Really, we've got most everything automated with Ant or Nant scripts, and it's all pretty easy to deal with.
But as I said, we're a small shop. Never more that 3 or 4 developers on any one project.
We've recently starting using a very different process for a particular project, however. The client is a fairly large luxury business with a big and technically advanced tech team. Funny thing is that they brought us in because we were percieved by the business end to be more nimble. This is funny because the tech team has slowly but surely brought us into their fold so that now we are certainly no more nimble than internal IT.
The processes this company's internal IT uses are actually very good, state-of-the-art processes. I find it all a massive pain in the neck. I start to feel that I must fill out a form in triplicate and have it stamped by six departments before I can check in a 2 line code change, but that's the rebel in me. I can see the benefit of all of this infrastructure and process when you absolutely must have 0 downtime in a large and complex system that is constantly expanding and changing.
The point here is that one thing these guys do is continuous integration and continuous builds. Not daily, but hourly. If something ain't gonna build, we'll know about it right away. That's cool.
So the build process is a snap. Once you have everything set up according to an elaborate set of rules, that is. After that, everything builds the same way. Not that there are no headaches. As it happens, today is release day for them, and everyone over there works on Sunday on release weekend. Lucky for me, I only have to be on call. And I was required to create a bug fix related to a fringe scenario that was not tested that turned out to be not such a fringe scenario after all.
So there's always scrambling to do on release day. But at least we know the build process itself is solid. That's something I guess, though still my day of rest was interrupted.
Blog-Reading Fun
(or How I enjoy coming to d.* and seeing new posts.)
Keep 'em coming. Some days I am just too tired, stressed, or discouraged to contribute one fresh word myself, so it's a pleasure to read someone else's. There's an odd sort of satisfaction to be found in the discovery that yours isn't the only flawed workplace with imperfect people and processes.