The best way I have found to fix this is to delete and re-add each and every Union All Transformation that is downstream from the place where you added the new column(s). Alternately, you can edit each Union All and manually add the missing columns, but since this usually happens to me with multiple columns, I don't prefer this solution most of the time.
Has anyone else found a way to prevent and/or fix this pesky bug? Or have I mis-characterized the problem?
Dan
]]>If you want to retain your original surrogate primary key values from your source data when inserting into a target table that has an Identity-based primary key, use the OLE DB Destination and choose the Table or view - fast load option for the "Data access mode" option. This will make visible a "Keep identity" checkbox. After checking that all you need to do is map your source key column to the target Identity key column, and it should work.
In my situation, this worked with a local source Access MDB file and a remote target SQL Server 2005 database connected through SQL authentication; in both cases the Connection Managers used OLE DB connections, and used the OLE DB source and target in the data flow.
Here is the original post:
After several happy months of *not* working with SSIS, I returned to the tool today to help me with a medium-complexity data migration. It hasn't even been one day, and already I'm facing a near-show-stopping limitation that's going to force me to either abandon SSIS or implement some kind of ugly hack. I'm starting remember why I was so happy to put SSIS in my rear view mirror after spending several months fighting with it on a previous project.
The problem I have today is transferring a table from a legacy database where I want to keep the original numeric PK values when I insert into the new destination table, which uses an IDENTITY column for the PK. The only way to do this is to use the SQL Server Destination, which only works in a bulk insert fashion (which is just so stupid--why can't we have a SQL Server-aware destination that is not limited to bulk insert?); bulk insert is no good to me because I'm not inserting into a local database connected via Windows authentication (another truly stupid limitation). The OLE DB destination sees an Identity column as "read only."
I found a blog post elsewhere lamenting this sorry state of affairs, with the suggested workaround of inserting first into a holding table and then calling a stored procedure that moves the rows from the holding table to the real table. This might be the way I end up going--IF I stay with SSIS for this project. I might have to sleep on that. I hate to walk away from SSIS, though, because I've already invested a day getting this far into my project, and if I go to a custom-coded solution I'm going to lose all of the convenient bootstrapping features of SSIS.
Aaarrgghhhh! I hope by now people at Microsoft have realized how poorly SSIS has been received (from my reading of the blogs and forums, anyway). It's not that it doesn't do some things pretty well, but the flaws it does have (don't get me started) are *really* annoying. Maybe my problem is that I just keep trying to use SSIS for things just outside of it's envisioned sweet spot... I hope someone at Microsoft has a master plan to stop SSIS from sucking. We need a tool like this that doesn't suck and that doesn't cost huge amounts of money in licensing fees.
I feel better now. I guess I better get to hacking...then I can take a shower.
Dan
UPDATE:
In case you were wondering whether you could use a pair of Execute SQL tasks to call SET IDENTITY_INSERT before and after the data flow task, it does not appear so. I tried this and received the error "User does not have permission to write to this column" on the first attempted insert.
To my way of thinking, a "real developer" is someone who consistently produces quality software systems that delight customers. "Real developers" do this in spite of whatever handicaps happen to exist in their world. They know that no technology, platform, API or software ecosystem is perfect, but they know how to leverage the strengths of what they are given to work with to produce excellent results. And if they don't know, they can admit it to themselves, and learn quickly.
I also like this from Bob's bio:
I’ve been an independent software designer / developer for 24 years. In this business, that makes me a Really Old Fart. In the past I’ve done everything from Z-80 assembler to Visual FoxPro (in the early 90's I wrote two books on that product, and edited FoxTalk for a couple of years). These days I mostly concentrate on the .NET platform, architecting and building line-of-business applications, usually involving large databases.
Leave it to an old Xbase guy to know what it's like to work in what some programmers who consider themselves to be more "elite" like to think of as a technology ghetto! I can relate.
After a childhood spent two versions of the Tandy TRS-80 Color Computer, I started out hacking DOS and Clipper in my first IT job; after that I spent several years in the worlds of VB 4-6 and ASP (with a good measure of Oracle and Java thrown in), and since then have been using .NET. To make me even more unfashionable, I am the co-author of two editions of a VBScript book (with a third edition in the works); few would consider VBScript a "cool" technology, but it is nonetheless a powerful and enduring (though not always elegant) one.
All this time, my primary job has been to design, build, and support business- and database-oriented solutions for clients who just want something to work. In the big picture, all this puts me squarely in the programming mainstream, but there's no getting around the fact that some people persist in seeing my world as less worthy of respect. I've learned to live with it, even embrace it, though early on in my programming career I was somewhat insecure about it.
(Just when I thought we had made some progress, an otherwise excellent article in a recent Dr. Dobb's Journal categorized Visual Basic as a "scripting language," despite the fact that there are at least four flavors of "Visual Basic," only one of which can be considered a scripting language. I've lost count of the number of times I've seen this kind of thing, and the Dr. Dobb's editors should know better. I don't mean to suggest that the author or publication has the kind of active condescension we've been discussing here, but this example does underline the impression that people who operate outside of the Microsoft development stack have that "Visual Basic" is some kind of toy.)
I'm adding "Bob on Development" to my RSS reader.
Dan
]]>At first I thought that the meaning of "lookup table" might be obvious enough not to require much explanation, but later I changed my mind. I started where lots of people start these days and looked up lookup table in Wikipedia; I realized that the term does not necessarily have a universal meaning. The Wikipedia entry (at the time I am writing this) defines the term in more general terms:
In computer science, a lookup table is a data structure, usually an array or associative array, used to replace a runtime computation with a simpler lookup operation. The speed gain can be significant, since retrieving a value from memory is often faster than undergoing an expensive computation.
This definition hints at my usage of the term, but Wikipedia's context assumption is different, whereas every time I've used this term it has been to describe a table in a relational database that is designed to provide a set of possible values for a column, which serves to limit the possible values for a column, standardize input, and reduce the amount of data stored in a table.
Another term I have encountered for this same concept is reference table (not to be confused with cross-reference table, which is used to define a many-to-many relationship between two other tables; lookup/reference tables are typically used to define one-to-many relationships).
Lookup tables are often used to populate dropdown lists and are also often used as a sorting/grouping mechanism for a query or report.
Just be clear about what I mean, this table structure...
Customer Name
Customer Type Name
ACME Products
Retail
Some Store, Inc.
Retail
Pity the Fool Enterprises
Wholesale
Crazy Time Industries
Wholesale
...becomes this when we add a lookup table to the design:
Customer Name
Customer Type ID
ACME Products
1001
Some Store, Inc.
1001
Pity the Fool Enterprises
1002
Crazy Time Industries
1002
Customer Type ID
Customer Type Name
1001
Retail
1002
Wholesale
With the above definition in mind, I have described below my (current) preferred design for a lookup table.
Column
Surrogate Primary Key
Short Name
Description
Active Flag
Display Order
Created Stamp
Last Modified Stamp
A description of each column follows:
Surrogate Primary Key
Depending on your stance on the endless debate over surrogate vs. natural keys, you may prefer a natural key, but I'm a surrogate key person myself. Naturally, I'm sure you'll agree with me and design your database using surrogate keys. :-) Regardless of what kind of key you choose, though, if you're using anything like the lookup table pattern I'm describing here, your lookup table will need a primary key, and that primary key will be used as a foreign key in other tables, along the lines of the "Customer Type ID" example above.
Surrogate primary keys have different implementation mechanisms in different database engines. Oracle has its SEQUENCE object, SQL Server has IDENTITY columns, and Access has the AutoNumber. Usually these are numeric keys, but I have started to notice a trend in the SQL Server world towards using GUIDs for all keys; often the explanation for this is that database replication in SQL Server requires GUID-based keys, though I'm not so sure that's as much of a concern for lookup tables.
Short Name
The "Short Name" column might also be labeled just "Name." If you imagine your lookup table showing up in a dropdown list or in a column on a report or query output, you can probably imagine as well that a longer name could make things awkward; you dropdown list or report column might need to be exceedingly wide in order to accommodate longer names. My preferred method is to use a "Name" column in conjunction with a "Description" column, which can be used to help users and administrators understand the meaning conveyed by the Name. I can imagine in some circumstances where instead of, or in addition to, a Description column a "Long Name" column might make sense. Along the same lines, it is sometimes also appropriate to have more than one "Name" column for different purposes--perhaps one for display, and one for sorting (see also "Display Order," below).
Description
As described above under "Short Name," the "Description" column is intended to convey additional meaning. For a simple lookup table, a "Description" column may be overkill. However, I have found this column very handy in the presentation tier; most commonly, this has come up when there is doubt about whether the user will be able to figure out the meaning of a lookup table entry based on just the "Name." If there is a possibility that the Name could be confusing, or that entries might have similar names that are difficult to distinguish, then you can display the Description in the user interface to help the user out, perhaps in a "tooltip" that floats over the field in the UI or some similar mechanism.
Active Flag
This is one of those I had to learn about the hard way. Here's what happens: when you design and build your application for your client, they have exactly six Customer Types. So you design a lookup table, put those six entries in it, and proceed to design the rest of the database, using the CustomerType lookup table key as a foreign key in various tables. Most likely, you are using referential integrity (RI), so once you start adding "parent" rows that utilize your "child" lookup table rows, the database engine will prevent you from removing any of your lookup table rows because they are locked in by the RI. You also write code that retrieves the rows for the lookup table for dropdown lists, reports, etc.
Six months later, your client wants to drop one of those Customer Types because it's not relevant anymore. You can't just remove it from the table because of the RI, and you wouldn't want to do that anyway because all that old data is still valid. But you have a Catch-22 because the now defunct Customer Type is still showing up in various dropdown lists, report parameter prompts, etc.
If you had designed your lookup table with an "Active" flag column *and* written all of your queries to filter on the Active flag, then "deactivating" the defunct Customer Type would be as simple as flipping the Active flag. You get the best of both worlds: the old Customer Type will still show up as a label when querying old data, but it won't be available as an option any more for the creation of new data.
Display Order
You may have noticed that people (especially your client) do not think about the order in which they want lookup table values to be sorted. It's almost always after the fact that this comes up (unless you're savvy enough to ask the question in advance and your client client is forward thinking enough to have the right answer). Regardless, it sure is nice to be able to manipulate the sort order of a dropdown list or other query/report without having to write new code or (as is often the case) implement some kind of hack in the presentation tier. Save yourself the trouble and build a Display Order column into all of your lookup tables. I generally use a numeric type for this column that is the same width as the numeric surrogate primary key. As you write any queries that select from this table, include an ORDER BY clause that uses the Display Order column. When requirements change relative to the sorting of your lookup table, all you have to do is manipulate this value. (In a SQL Server environment, you might also consider using the DisplayOrder column for a, clustered index.)
There is one exception to this rule, I think: some lookup tables might so obviously lend themselves to a pure alphanumeric sort on the "Name" column that a Display Order column seems silly.
Created Stamp
This column is a simple date/time column that reflects when the lookup table row was first created. Some might argue that Created and Last Modified audit columns are overkill for lookup tables, but I have found them useful and almost always use them. They are especially handy in an application that includes an administrative interface that enables users (not just database administrators and developers) to manage the data in lookup tables.
Last Modified Stamp
This column is a simple date/time column that reflects when the lookup table was last modified. In some applications it might also make sense to have "Created By" and "Last Modified By" columns to reflect which user made a change.
* * *
A lookup table is also much like a dimension table in a dimensional data warehouse design, a context in which the fascinating topic of the slowly changing dimension also comes into play. The connections between lookup tables in a transactional database design and corresponding slowly changing dimensions in a downstream data warehouse is an area for additional interesting discussion, I think. For example, typically a lookup table in an OLTP system does not support changes over time (though with the suggested Active Flag, I believe this could be done), but Kimball Group consultant Joy Mundy advocates tracking "attribute changes" over time:
The key flaw Kimball Group typically sees in existing enterprise data warehouses is a lack of historically accurate attributes. Almost all data warehouses track transaction history--what was sold, to whom and for how much--but many do a poor job of associating the transaction with the attributes as they were at the time of the transaction. This attribute history is very important for data mining and other smart enterprise applications. Suppose, for example, that you want to build an application that will present product choices to a potential customer based on where he lives. If the data warehouse only keeps the customer's current address, then for the purposes of queries, analysis and prediction, it looks as if the customer has always lived where he does now. We lose the vitally important information about how the customer's behavior changed when he moved from a cold climate to a warm climate. And once that history is gone from the data warehouse, it's difficult or impossible to reconstruct it.
Another thing I've wondered is whether OLTP database designers could do certain things with their table designs to make it easier for OLAP database designers and ETL coders to come along later and do their thing.
I'm sure my quest for the "ideal" lookup table design will continue. Your thoughts are most welcome.
Thanks for reading,
Dan
P.S.
A quick Google search turned up an opinion piece by Don Peterson called Lookup Table Madness. It appears that the author makes a case against the whole practice of using lookup tables, which sounds very interesting to me--but unfortunately that particular site locks their content behind passwords, so I have not read it.
(Irrelevant rant: Granted, the account I would need to read that article is free, but I always find myself annoyed when a page comes up at the top of the search results and when I click through I find that the real content is hidden. When that happens, I make a mental note to avoid that site in the future. If people want to hide their content, that's fine by me, but I think search engines should reciprocate by penalizing these pages rather than keeping them at the top of the results).
]]>In my colleague's view, setting aside the question of whether or not this legislation was a good idea, the blame for these problems rests squarely on the shoulders of the software and hardware vendors whose products require patching, support, etc.--they should have seen this coming, they should have designed their products to be more flexible, etc. This is their fault, plain and simple. After all, he says, DST is not a force of nature; it is not embedded in reality at such a fundamental level as something like the Gregorian calendar or the metric system.
On the other hand, commenter Kevin Dean posted recently on a ZDNet thread a view that we should give these vendors a break:
Why is this Microsoft's fault?
Microsoft isn't the only one having problems with this. I opened a case with Sun regarding the availability of their DST-patched Java runtimes and got a resolution to the problem only last week. Now I'm scrambling to get all my client workstations, my application server, and my database properly patched.
The blame for this fiasco lies squarely with Fred Upton, a Michigan Republican, and Edward Markey, a Massachusetts Democrat, for sponsoring the amendment to the Energy Policy Act of 2005. The amount of energy saved is miniscule at this time of year (unlike in summer with its longer days) and the disruption they have caused to computer systems and transportation schedules (especially airlines) is phenomenal.
I, like everyone else, would have loved it if Microsoft could have made things smoother, but this is a massively complex undertaking that every computer software provider is having trouble with and the fault lies squarely with a political system that rewards style over substance.
Again, setting aside the question of the wisdom of the legislation (which is certainly also a legitimate topic of debate), Mr. Dean's position is that the complexity of this issue dictates that we should cut software vendors and hardware manufacturers a break on this.
Is there a way that vendors and designers could have avoided this? Or is this an inescapable problem once you decide to cross the boundary into making your software "smart" about changing the time automatically relative to DST, rather than just keeping your software "dumb," like my microwave oven, which I have to change manually twice a year.
In a DST software bugs post I wrote in April of 2006, before I had heard anything about this impending DST start date change, I took the position that we have ourselves to blame for difficulties with software bugs and system synchronization problems related to DST, because after all, the time changes happens twice a year, every year, and we should not be caught with our pants down, beepers going off, support phones ringing, every time it happens. However, I think the context of that discussion was a little different (but maybe not); Scott Rosenberg posted similar sentiments recently, more from a support/logistics standpoint.:
Back when my job as Salon managing editor involved overseeing our daily production, I noticed that, every spring and fall, almost without fail, our publishing system would experience a glitch of some kind on the weekend that the clocks got moved forward or back — nothing serious, mind you, but enough to throw a wrench in the works of our site updates. It wasn't a single bug, but some sequence of related bugs, so we'd fix one and then six months later something else would happen. Eventually we got in the habit of just making sure that one of the developers kept a close eye on things when that weekend rolled around. It was prudent.
In the same post, Rosenberg speculates that perhaps these kinds of challenges are related to something deeper:
It seems that this is one of the unexpected consequences of living in a world operated by software: new danger zones lie where human abstractions — borders, measurements, languages — change or conflict or fail to behave as expected. Clocks and calendars and maps are no longer just assists for human understanding; they are symbols at the heart of systems upon whose performance lives depend. I suppose this started with the first railway schedule, but with the dateline-addled F-22 it has entered a whole new realm of disconcert.
What do you think? Are there fundamental lessons here that software designers and companies should be heeding? Are there some products or manufacturers that got this *right*, who have not experienced difficulties with the great 2007 DST transition? If so, what can we learn from their example?
Thanks for reading,
Dan
The scenario I was working in was an import scenario--that is, reading a provided Excel spreadsheet file and importing into a database. Some of my tips below may be tied to this assumption, but I think my advice will apply to most Excel automation scenarios. (Also, I'll be a lot of this applies to automating other Office applications such as Word.)
First, a bit of background: keep in mind that the Excel API is implemented in COM, which is a precursor to .NET. Microsoft has gone to great lengths to make .NET backwards compatible with COM (and vice versa), but the fact remains that when you are communicating with Excel from your .NET code you are not communicating with something that is native to .NET.
One more thing: my experiences as documented here are based on using Excel 9.0 (Office 2000) and Excel 11.0 (Office 2003) from .NET 2.0 running on Windows XP SP 2. I can't say for sure whether anything would be different relative to newer versions of Excel, .NET, or Windows. Feel free to add comments to this post to provide more up-to-date information.
The links at the end of this post point to some related information that you may find useful.
On to the tips...
Tip #1: Seriously Consider NOT Automating Excel from .NET
I don't mean to be flippant: the most significant conclusion I have reached from my experience is that if I ever have to do this again, I will twist whatever arms I have to in order to purchase some kind of Excel component so that I do not have to program against the Excel API at all. There are many .NET component libraries on the market that can read from and write to Excel spreadsheet files without Excel even needing to be installed on the machine. I recommend strongly that you seek one of these out. As you read the rest of my tips, you will have a better idea why. A quick Google search should turn up several options.
(Component vendors, please refrain from adding comments to this post to advertise your Excel library; requests to advertise, review, or plug your component will be politely declined. Thanks for understanding.)
Tip #2: Use VB.NET instead of C#
Even if your preferred .NET language is C# (or some other .NET language), even if you absolutely hate VB.NET (it's really not so bad), I strongly recommend that you use VB.NET instead of C#. If the rest of your solution is written in C#, break out your Excel-related functionality into a separate VB.NET DLL. I got about 10% into the initial attempt at my solution before I said "Screw this!", threw out my C# code, and started over again in VB.NET.
Why? In short, two reasons:
First, Office applications make prodigious use of optional arguments--in many cases a single function can take a dozen optional arguments. C# does not know how to handle optional arguments, so you end up passing a ton of useless null or Object values. This might not be so bad if there weren't so many, and if the lack of useful Intellisense in the IDE didn't make it even more difficult to tell which argument is which.
Second, C# is not really hip to the Variant data type, which is used often in Excel's VBA-centric API.
Trust me: just use VB.NET. If you're still not convinced, check out the link to the article "Programming Office Applications Using Visual C#" at the end of this post.
Tip #3: Communicate with Excel in a late bound manner
Late bound? That's not a term you hear much these days since .NET has overtaken COM. This term refers to a technique of not compiling a reference to the Excel library into your assembly. Late binding is otherwise known as IDispatch binding, with early binding otherwise known as vtable binding. Since Excel is still a COM application, these terms still apply in a .NET interop context.
I recommend NOT adding an early bound reference to Excel in your project. Microsoft has not done a good job over the years in managing the compatibility between the various versions of the Excel library; depending on which version of Office you have installed, you will have different version-specific Excel reference options in your COM references list. Don't be tempted by these early bound reference opportunities. Late bound all the way, baby. Protect yourself from version incompatibilities when you find out that you compiled against Excel 11.0 and your client is using 9.0. Similarly, you can protect your solution from breaking when your user or administrator upgrades Office.
In .NET, going late bound requires some syntax and techniques that may be unfamiliar to you, but it's not that bad.
To instantiate an Excel Application object, use the Interaction.CreateObject() method, which is much like the old VB6/VBA/VBScript CreateObject() function. It is part of the Microsoft.VisualBasic namespace (see Tip #4 if you decide to go with C# instead of VB.NET). Here is my LoadExcel() method:
Private Sub loadExcel()
m_excelApp = Interaction.CreateObject("Excel.Application", "")
If m_excelApp Is Nothing Then
Throw New Exception("The Excel libarary could not be loaded. " & _
"You may an incorrect version of Excel installed. " & _
"Please consult your system administrator. The instantiation " & _
"failed without an exception.")
End If
End Sub
Similarly, in order to invoke any properties or methods on the Excel.Application object, you want to use InvokeMember, like so:
workbooks = m_excelApp.GetType().InvokeMember("Workbooks", _
BindingFlags.GetProperty Or BindingFlags.Public, _
Nothing, m_excelApp, Nothing)If that code looks a little cumbersome to you, see Tip #5. If you wrap your Excel calls, your main application logic won't be burdened with the more verbose code required for late bound automation.
Check the .NET documentation on InvokeMember(). You may also prefer using Interaction.CallByName(), which offers a kind of shorthand version of InvokeMember(), but I prefer the more explicit control offered by InvokeMember().
The original version of my solution used an early bound reference. When I refactored my code to be late bound, I did not notice any significant performance difference. This jibes with my experiences with early vs. late binding performance when I used to write a ton of COM code in VB 6. If performance is a real concern, I return your attention to Tip #1; I think there is a pretty low performance ceiling you'll hit when automating Excel no matter what you do; theoretically, a good Excel component would be faster.
(You may also need an Imports System.Reflection statement at the top of your class file to do late binding. I have this at the top of my class, but as I write this I can't remember why.)
Tip #4: If you use C#, go through VB to get to Excel
While researching my Excel automation solution, I came across a great Dan Appleman article that is linked at the end of this post. It shows how you can reference the VB.NET runtime library from your C# project to make use of the Microsoft.VisualBasic.Runtime namespace, which is where you will find the aforementioned Interaction.CreateObject(). I think it's safe to say that you can't do late bound COM programming from C# without going through the Visual Basic library.
Tip #5: Wrap your Excel communication points
I strongly recommend that you use good old information hiding to protect the main logic of your code from the particulars of communicating with Excel. Here are a few of the functions I developed:
Private Function getRange(ByRef rangeString As String, ByVal worksheet As Object) As Object
Dim range As Object = Nothing
range = worksheet.GetType().InvokeMember("Range", _
BindingFlags.GetProperty Or BindingFlags.Public, Nothing, _
worksheet, New Object() {rangeString})
Return range
End Function
Private Function getCellValString(ByRef columnLetter As String, ByRef rowNum As Integer, _
ByVal workSheet As Object) As String
Dim expectedCell As Object = Nothing
expectedCell = getRange(columnLetter & rowNum.ToString(), workSheet)
Return cellToString(expectedCell).Trim()
End Function
Private Function cellToString(ByVal singleCell As Object) As String
'The Value property is a VBA Variant, which does not
'translate well to .NET. When the cell is blank, the
'Value property will be Nothing
Dim singleCellValue As Object = Nothing
singleCellValue = singleCell.GetType().InvokeMember("Value", _
BindingFlags.GetProperty Or BindingFlags.Public, _
Nothing, singleCell, Nothing)
If singleCellValue Is Nothing Then
Return String.Empty
Else
Return singleCellValue.ToString()
End If
End Function
Note: I have omitted by retry wrapper code from these methods for the sake of clarity; see Tip #7 for an explanation. If you reuse these methods, you may want to add in the retry wrapper--though in my testing so far, it looks like when the "application is busy" error happens, you may be irrevocably screwed.
See Tip #6 for an explanation for why I use cellToString().
Tip #6: Watch out for empty cells
In testing I ran into some annoyances related to empty cells, so I developed the cellToString() wrapper function you see above. The issue is that Range.Value is a Variant, not a String or numeric type. So when a cell is empty you can (but not necessarily always) get Nothing back instead of an empty string.
Tip #7: Watch out interruptions and disconnects
I don't have a lot of information to provide for what causes this problem, but in testing I have run into problems with lost communication between the .NET code and the running instance of Excel. What I mean is, if your Excel automation code is running and the user makes use of the Excel application at the same time, instability may occur. The main scenario we've been able to reproduce is, oddly enough, if the user opens a .XLS file by double clicking on it (as opposed to using the File->Open menu) then your .NET code will suddenly lose its ability to connect with Excel. This is the error we saw in this scenario:
System.Exception: There was an error during the main import process: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Runtime.InteropServices.COMException (0x8001010A): The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))
It's possible that other Excel usage scenarios could cause this same problem.
If this scares the hell out of you (and it should) see Tip #1.
One solution I tried is wrapping all of my Excel communication calls in some trap-and-retry logic. So a statement like this:
range = worksheet.GetType().InvokeMember("Range", _
BindingFlags.GetProperty Or BindingFlags.Public, Nothing, _
worksheet, New Object() {rangeString})becomes this:
Do
Try
range = worksheet.GetType().InvokeMember("Range", _
BindingFlags.GetProperty Or BindingFlags.Public, Nothing, _
worksheet, New Object() {rangeString})
Catch ex As Exception
If ex.Message.Contains("application is busy") Then
If m_excelRetryCount <= MAX_EXCEL_RETRIES Then
m_retryExcel = True
m_excelRetryCount += 1
System.Threading.Thread.Sleep(EXCEL_RETRY_DELAY)
Else
m_retryExcel = False
m_excelRetryCount = 0
Throw New Exception("The Excel application is apparently busy, which is " & _
"interfering with the import process. We have retried communication with Excel " & _
MAX_EXCEL_RETRIES & " times with a delay between each retry, but we are not " & _
"able to establish communicatation with Excel.", ex)
End If
Else
m_retryExcel = False
m_excelRetryCount = 0
Throw ex
End If
End Try
Loop While m_retryExcel
m_retryExcel = False
m_excelRetryCount = 0
Note that I declare EXCEL_RETRY_DELAY, m_retryExcel, and m_excelRetryCount at the class level since I had to repeat this retry logic in several of my Excel wrapper functions. That may or may not have been the best technique, but there it is.
Your mileage may vary. So far from testing it appears that the retries cannot ever reestablish the communication link.
Some Potentially Useful Links
Good luck!
Thanks for reading.
Dan
P.S.
If you thought this was helpful and feel compelled to do something nice in return, considering buying one of our high quality software development books, published by developer.* Books. You will be doubly enriched. :-)
This recent interview with digital publishing advocate Morris Rosenthal is also revealing.
You may also recall the Shaking up Tech Publishing post on David Heinemeier's Loud Thinking blog that generated some buzz last year. Reading through the original post and all of the comments (particularly Tim O'Reilly's) is enlightening.
Some people have asked us, "You've started a book publishing company? Huh? Why would you do that? I thought all the book publishers were going out of business." The above linked articles do a great job of describing why we think there is a future for new publishers to succeed by embracing the change happening right now.
As an aside, Mr. Rosenthal is part of a groundswell that sees the new digital printing and distribution methods as a tool to empower self publishers. I do agree that it this is important, and a good thing. However, more along the lines of Mr. Page, I do not believe that the future will be changed in a fundamental way in the balance of publishing houses and self publishing. There will always be books that will succeed quite well as self published books (as 37signals found with Getting Real), largely owing to their particular subject matter (how-to books do quite well, traditionally), timing, and the ingenuity, perseverance, and audience reach of their authors; digital, on-demand printing, ebooks, and online distribution definitely make things easier, but the dynamics of self publishing are essentially the same as they always have been, IMO. And there will always be a role for publishing houses to "curate" and add value to books through editing, packaging, and promotion.
We hope that people who read our first two books, Software Conflict 2.0 and Software Creativity 2.0 agree that the books are worth the effort and reflect the care and design that went into them after the author finished the writing.
Happy reading!
Dan
]]>It's emblematic, I think, of how insanely busy 2006 was for me that the five year anniversary of developer.* passed in the middle of last Summer without any comment or celebration here on the site. Five years! It's hard for me to believe it was June 2001 that I posted two of the first essays on the site, "Principled Programming" and "The Human Impact of Software."
Those early pieces published on the site were some of my first forays into writing about software development, and they reflect not only where I was at the time as a person, as a writer, and as a software developer. The story of DeveloperDotStar's changes and growth reflects my own story over the same period. As such, there are both successes to relish and disappointments to reconcile. I'm proud of the many accomplishments, and of all the great software writing that has been published here (mostly by contributors other than myself), but lately, because last year I tried to do too much, I have also been feeling the sting of loose ends left frayed and people let down.
The story of DeveloperDotStar can also be seen as one of increasing scope (and it is my hope that this expansive trend can continue into the future). Originally the site started out as a place to publish my own essays about software development, like the two linked above. At the time, early 2001, the blogging thing had not really taken off yet. At the time, as I recall, Joel on Software was the closest analog on my radar screen, though I'm sure others were doing similar things at the time. (FWIW, according to Alexa, Joel On Software went online July 29, 2000, just under a year before developer.*.)
I had some things I wanted to say, but I did not see any publication or web site where they would seem to fit, and I also knew that my own writing voice needed some time to develop, so I started my own publication-of-one in June 2001 (according to Alexa, it was actually in March of that year, but I think it took me a few months to get things together). The first version of the site was hosted on an early server-side web hosting product called Internet on a Stick, which was owned by my friends Bryan and Stan Sedwick. Looking back on it now, what I really wanted was a blog, but I didn't know to call it that; so I started what amounted to a blog, but I called it a "magazine," replete with its own old fashioned Letters to the Editor page.
Over the next couple years I continued to publish occasional essays, and the site built up a small following. The next phase of developer.* started when I decided that my own voice was not enough; if I'm honest with myself, the expansion of the scope of the site was also related to my own growing dissatisfaction with my writing, both in terms of content and style. Through my own growth process I was realizing that I was not satisfied with the certainty in my writing, nor what I felt was a tone that was too didactic. My essay "Software Design and Programmers," published on this site in March of 2003, was the significant piece of polished writing that I wrote for developer.* (not counting the many blog posts, editorials, and book reviews I have written since then).
One memorable event around this time was my attendance at one of Gerald Weinberg's last Problem Solving Leadership workshops (where, incidentally, I was lucky enough to be in attendance at the same workshop as Agile/XP people such as Martin Fowler, Andrew Hunt, Alistair Cockburn, Ron Jeffries, and John Brewer). This coincided with my increasing admiration of the writing styles of authors such as Gerald Weinberg and Martin Fowler (and probably also Alistair Cockburn) who always manage a fine balance between a confident and humble tone, speaking with authority, but also empathy, with an appreciation for complexity (and a corresponding avoidance of ideology), and with an attitude that says that the author is ready to be proven wrong.
I continue to strive for these ideals; I have found that the best writers in our field (Brooks, McConnell, DeMarco, Weinberg, Glass, Parnas, Yourdon, Spolsky, Wiegers, and too many more to mention) exhibit these qualities. I think what these writers have is more than a highly developed writing style; I believe these people have also been in the software game long enough to have been humbled by software, by computing machines, and by the enormous difficulty of pulling together teams of people to make all that software stuff happen. These words of Gerald Weinberg, quoted from his personal biography on his web site, have always been a great inspiration to me:
After almost 40 years working with computers, I've learned a couple of things, but I still can't make sense out of most of it. Most of all, I've discovered that people are at the bottom of just about every problem - but I think I knew that when I was little, then got talked out of it somewhere along the way. I've worked hard at relearning this lesson, and learning how to do something about it. While educating myself, I learned a second principle: I'm the "people" at the bottom of most of my problems.
When I first read that it about knocked me over, and went and re-read Understanding the Professional Programmer in a whole new light. The move from being DeveloperDotStar's only author into the role of editor and curator allowed me to gain some distance from my own writing, to process some of these insights, and to focus more on my own growth as a software professional and a person. In the meantime, I was able to work with some great people who contributed so much to this site from the Summer of 2003 until now.
The first article published the "new" developer.* was by Danny Faught, another attendee of the aforementioned PSL workshop and a talented writer (mostly on QA-related topics). It was called "Diving in Test-First." We published it on July 26, 2003, just over two years after "Principled Programming." I will be eternally grateful to Danny for contributing it, and to every other writer who contributed writing to this site in the three years since. (You can check out the full list of edited articles published on this here.)
I posted the first official blog post to developer.* on Halloween (October 31st) of 2004. It was titled, predictably, "Welcome." Since then, the blogging platform of developer.* has seen its share of periods of high activity as well as slow times. The original idea was called the "developer.* Cooperative Digest," and it was to be a "community blogging platform." Partly this was driven by the limited individual blog personalization capabilities of the open source CMS framework we adopted, Drupal (an unfortunate flaw to this day in this otherwise best-in-class tool), but the "Cooperative Digest" idea was also based on the seeming promise of community-based blogging.
We've had surges of blogging over the years, followed always by lulls whenever the inspiration for the surge wears off. Partly I know that my own inconsistency as a contributor to the blogs and discussion have been a large factor in the failure to sustain a thriving community of bloggers. But I try not to feel too bad about it since none of early attempts at creating a community blogging space has really survived. Instead, the blogosphere has evolved into a true distributed network, with each blogger controlling his or her own node, connected but still apart. Things like social bookmarking and wikis have instead provided the glue. My deepest thanks go out to the many people who have blogged or commented here--most especially the indefatigable Edward Nilges, who was one of the first bloggers to join the "Cooperative Digest"; Edward has never needed my prodding or guidance to share his unique viewpoint on software, philosophy, culture, and politics. (There are too many other names to mention, and I fear if I start I'll miss someone important.)
The future of blogging on the developer.* web site remains uncertain. I can say for sure, however, that I have no regrets about our experimentation thus far with various blogging ideas for one reason: some really great writing has been posted here, all without me standing in the middle as editor. It has been a great complement to the edited articles side of the site, and has filled in during many quiet times between edited articles.
Over the years I've stubbornly kept the developer.* web site free of a commercial focus. This comes from my vision of developer.* as an "indie" enterprise, inspired partly by BBS sysops (in the days before the web hit big I spent many late night hours in on independently operated BBS systems and reading Jack Rickard's wonderful Boardwatch Magazine), partly by the many small press zines I used to read in college by the stack (anyone else remember Factsheet Five?), and partly by the example of great indie record labels like SST, Touch and Go, and Dischord.
I was also deeply affected by what happened to software publishing, especially on the web, during and after the great gold rush circa 1997-2001: many of the best sites on the web, which had been created by people just like me, people who might have been BBS sysops or zine editors a decade before, were bought up by exceedingly lame big media companies who promptly covered the sites with ads, popups, and tracking cookies, locked the good content behind passwords, and started spamming their mailing lists with "newsletters" and "special offers from partners" every two hours or so. (The new site InfoQ gives me hope that the future of commercially oriented software development publishing will be brighter. I could give other examples of excellent sites that are operating now in the post-media-consolidation era, but I'll resist further distraction from the present topic...)
Our independent posture will hopefully continue, but the nature of the game has changed somewhat with the most recent step in the evolution of developer.*: the new developer.* Books publishing enterprise, in which I am joined by my business and life partner, Gayle Devereaux. By necessity of survival, and in order to do the right thing by the authors who sign up to have their books published by us (nothing is more disheartening to an author than to have his or her book disappear when the publisher goes under), developer.* Books must operate as a business, independent spirit or not. It may be a number of years before it's a business that will pay anyone a full salary, but it is much more a business than a hobby, as the web site has always been.
As such, developer.* Books requires a more serious commitment of time and energy. This commitment to book publishing must be balanced with the fact that I still have a challenging day job designing and building software for my employer's clients, with the demands of keeping up a web site, with promises already made to writers with articles in the works, and with all of the other important aspects of life.
Books have always been a passion of mine, and the chance to be involved in their development and production was something I could not resist. 2006 saw the publication of our first two books, Software Conflict 2.0 and Software Creativity 2.0, both by software luminary Robert L. Glass. We are immensely proud of these two books, not only because they are fine books, but also because they ask critical questions and offer important insights for the field of software. Most importantly, they are excellent examples of what we have always searched for as a publisher of software writing: a timeless quality. We have more books in the works (stay tuned for announcements in the near future).
An early version of draft of this long post you are reading about the past and future of developer.* (which I have been trying to finish writing for two months) started off with this hook sentence: developer.* has a big problem: me. The current draft you are reading (for the few who have made it this far) does not start off this way, but the sentiment still holds because it reflects perfectly what happened in 2006: I tried to take on way too much, said "Yes!" too often, indulged too many of my ambitious ideas for the Great Web and Book Publishing Company In the Sky, and made promises I couldn't keep. I don't like letting people down, and I did that a lot last year--so much so that here it is the end of February 2007, and I have yet to finish picking up the pieces from what fell off of my overfull plate in 2006.
(If you're one of the people who are wondering if I fell off the face of the earth, please accept for now my sincere apology in this public forum; I hope to catch up person-to-person soon. I was voted Most Dependable by my 1987 high school class, believe it or not--a date which puts me squarely in mid life crisis territory. My dad bought a 1985 Mustang GT 5.0 for his mid life crisis; I started a book publishing company...there's something backwards about that. Interestingly, another thing that happened to me at the end of 2006 is that I realized that I was devoting too much energy to my online life, to "virtual" things; I think my dad had the right idea with that Mustang; I think it made him feel more connected to something real, something connected to physical things like gravity and torque, something directly under his control.)
Which brings us to now, looking ahead. What is the future of developer.*?
I can't say for sure, other than to say that as a general matter developer.* will hopefully be around for at least another five years. I can also say for sure that the book publishing side of developer.* must remain my primary focus. That ship has left the port, and I have to steer it. Not to mention, I'd like to repeat as often as possible that wonderful feeling that comes from holding in your hand the first copy of a new book, hot off the press, as they say. (In the case of Software Conflict 2.0, I go to repeat the experience again when the new Korean edition arrived in my mailbox.)
However, I still feel strongly that the developer.* web site is an important outlet for diverse software writing, particularly by less experienced writers, whom I have always enjoyed working with, no matter how many drafts it took to get the article in first class shape. In some ways the blogosphere, which gives everyone his or her own publishing platform, supplants the need for a "magazine" of edited articles based on gift culture principles, but I believe there will always be room for edited articles. Writers want to write them, and readers want to read them. The problem is that I just can't keep up with both the articles and the books.
So what to do? If you have any ideas, I'd love to hear them. My best idea at this point is that developer.* needs one or more volunteer articles editors; this person or persons would work with writers to get articles ready for publication. I've got several excellent pieces in the hopper from last year (if the authors have not given up on me altogether), and I get emails every week or so from new and established authors seeking a home for their work. Ideally, this person or persons would also share with me a passion for our unique content niche: diverse, general interest, mostly non-technology-specific, articles, essays, and interviews with, as we like to say, "a timeless quality." The person or persons would also need to share the "indie" spirit of the web site, which pretty much means that it will remain for the foreseeable future a non-money-making enterprise. (I'd rather shift the web site into archive mode than give that up.) The job also requires (despite my own shortcomings in some or all of these areas) the basic traits of all editors: strong opinions; a passion for writing; some knowledge of grammar and language mechanics; a willingness to be frank; and perhaps most importantly generosity, empathy, and patience.
This would be a tall order even if I was offering money for the job. Unfortunately, there is no money to offer. The work is its own reward, as I can attest. I regret in some ways that my own journey has brought me to a place where I cannot make room for it any longer.
Maybe, here in the midst of Web 2.0, there is a technical solution to this problem: a web-based software platform that will "harness the crowd" to turn ideas and rough drafts into polished, edited, articles. Kind of like a Web 2.0 writing workshop, but with a very pragmatically focused end game: the edited and polished article can be published right here on the developer.* web site once the process and people have deemed it ready.
I welcome any thoughts, comments, questions, and/or ideas you may have related to anything in this post. Feel free to use the comments form to share them, or if you prefer email me privately. I am honored and humbled by your interest.
Thanks for reading,
Dan
P.S.
As for my own writing, I do plan to get back some day to doing more than writing the occasional blog post. I have two software books that have been in the works for some time (really, for as long as there has been a developer.*), and when the time is right I aim to finish them. I believe that both will have that elusive "timeless quality" and will be worthwhile contributions to the field. I also believe that while curating, editing, and publishing have in some sense been distractions from the writing, these activities have also made me a better writer and a better person (it's hard to achieve the former without the latter, I think), so no regrets here. If having a mid life crisis has taught me one thing, it's that there is little time in this life for regrets.
If you’re just embarking upon a death-march project, and you want to share the agony and the ecstasy with all of us in the blogosphere, get in touch with me. If you’ve been trapped in a death-march project for six months, and you desperately want to tell everyone what went wrong, and/or why nobody else should suffer a similar fate, send me an email. If you’ve finished an amazingly complex, high-pressure death-march project with triumphant success, and you want to share your secrets with everyone else, let me know. If it was a tool, or a technology, or a methodology, or an amazing management approach that turned out to be the difference between success and failure, track me down and tell me.
This is your chance to have your tale of woe enshrined in an important book, and maybe your story can help someone else down the road.
Speaking of Ed Yourdon and important books, Mr. Yourdon has also recently taken a very interesting publishing leap that should be getting more attention: he has converted his book Modern Structured Analysis into a free and open wiki. (As far as I can tell, Mr. Yourdon has now released the text under the GNU license.) The new version is called Just Enough Structured Analysis. He blogs about the project, and how you can participate, here. I've been digging into the wiki, and not only is the wiki itself extremely well done, but much of the material is still very relevant and readable.
All the best,
Dan
For example, in a winforms app I'm working on, with a couple thousand rows in it, the grid would pause for up two seconds after the user clicked a checkbox in a cell. Ouch. The grid was also very slow to load, even using some tricks with SuspendLayout() and ResumeLayout().
Turns out all this slowness is due to painting issues, not data issues. Before you give up on the DataGridView as hopelessly slow, try these things:
* Set both AutoSizeColumnsMode and AutoSizeRowsMode to "None." If you want to call the AutoSizeColumns() method one time after you load the grid, that works pretty well, especially if you auto size based on the headers. You can also resize certain columns using the AutoSizeColumn() method or by setting a literal value for the Width propery.
* Set EnableHeadersVisualStyles to False. You will lose a little bit of "slickness" in your grid's appearance, but it will speed up considerably. It seems that enabling this feature makes the grid "inherit" its visual style from the OS style. Check the documentation or intellisense for a more specific description.
* I saw a suggestion on this page to turn off cell borders (CellBorderStyle = DataGridViewCellBorderStyle.None), but I did not try this one myself.
That same page linked above also confirms that the .NET 2.0 DataGridView is slower than the .NET 1.1 DataGrid. They must not have done a lot of testing with the autosizing and visual styles turned on. Hopefully a future release will improve things, but I shall not be holding my breath since I half suspect Microsoft is going to start letting winforms languish in favor of WPF...
This page also points to a DataGridView FAQ document that has some useful information.
Hope that helps,
Dan
Update: I'm a little slow on the uptake, but a week after writing this post, I noticed that VB.NET now has intrinsic IsNumeric() and IsDate() functions; check the Microsoft.VisualBasic namespace. Keep reading, though, if you need these for C# or for VB.NET 1.x.
Your mileage may vary depending on your performance needs; if you're calling these thousands of times in a loop, then be sure to test them against alternate techniques. Here are C# and VB.NET versions of the functions I use:
C# IsNumeric() Function
Double.TryParse() is available in .NET 1.1 and 2.0, so this function works with either.
public static bool IsNumeric(string anyString)
{
if (anyString == null)
{
anyString = "";
}
if (anyString.Length > 0)
{
double dummyOut = new double();
System.Globalization.CultureInfo cultureInfo =
new System.Globalization.CultureInfo("en-US", true);
return Double.TryParse(anyString, System.Globalization.NumberStyles.Any,
cultureInfo.NumberFormat, out dummyOut);
}
else
{
return false;
}
}
C# 1.1 IsDate() Function
Since we're assuming a strongly typed language, if the value already exists in a DateTime variable, then we can assume that it is a valid value. So we start with the assumption that the user has the value in a string or something easily converted to a string. This routine is .NET 1.1 compatible since we're not using DateTime.TryParse().
public static bool IsDate(string anyString)
{
if (anyString == null)
{
anyString = "";
}
if (anyString.Length > 0)
{
DateTime dummyDate = null;
try
{
dummydate = DateTime.Parse(anyString);
}
catch
{
return false;
}
return true;
}
else
{
return false;
}
}
An alternate variation removes some of the cohesion from the routine, but you could also make resultDate variable in this an 'out' argument and make it that much easier for the caller to get a DateTime value once it's been confirmed that the value is indeed a date; (this is basically how the DateTime.TryParse() function added in .NET 2.0 works, anyway, so it's not inconsistent). Here is my version of this alternate syntax. In C# it's easy enough to offer overloaded versions of both (also .NET 1.1 compatible):
public static bool IsDate(string anyString, out DateTime resultDate)
{
bool isDate = true;
if (anyString == null)
{
anyString = "";
}
try
{
resultDate = DateTime.Parse(anyString);
}
catch
{
resultDate = DateTime.MinValue;
isDate = false;
}
return isDate;
}
VB.NET IsNumeric() Function
Double.TryParse() is available in .NET 1.1 and 2.0, so this function works with either.
Public Shared Function IsNumeric(ByVal anyString As String) As Boolean
If anyString Is Nothing Then
anyString = ""
End If
If anyString.Length > 0 Then
Dim dummyOut As Double = New Double()
Dim cultureInfo As System.Globalization.CultureInfo = _
New System.Globalization.CultureInfo("en-US", True)
Return Double.TryParse(anyString, System.Globalization.NumberStyles.Any, _
cultureInfo.NumberFormat, dummyOut)
Else
Return False
End If
End Function
VB.NET 2.0 IsDate() Function
This code will only work in .NET 2.0 and above, since it depends on DateTime.TryParse(), which was not added until 2.0. If you need a VB.NET 1.1 IsDate() function you can rewrite the C# 1.1 version above without too much trouble.
Public Shared Function IsDate(ByVal anyString As String) As Boolean
If anyString Is Nothing Then
anyString = ""
End If
If anyString.Length > 0 Then
Dim dummyOut As Date = Nothing
Return DateTime.TryParse(anyString, dummyOut)
Else
Return False
End If
End Function
I've found the BackgroundWorker to be very handy, but a little tricky to start using at first. I like that it has built-in support for reporting incremental progress for a long-running operation, and that it has a simple way of allowing a long-running operation to be canceled. And the big benefit, of course, is that your form remains responsive to the user and the operating system as it runs on the main UI thread. For a basic tutorial, MSDN has you pretty well covered. But there are some subtleties I had to overcome that are not covered in the documentation, especially in the area of exception handling.
This entry on MSDN about the BackgroundWorker.DoWork() event contains the following paragraph:
If the operation [that you call from your DoWork() handler] raises an exception that your code does not handle, the BackgroundWorker catches the exception and passes it into the RunWorkerCompleted event handler, where it is exposed as the Error property of RunWorkerCompletedEventArgs. If you are running under the Visual Studio debugger, the debugger will break at the point in the DoWork event handler where the unhandled exception was raised.
That's all good, but what MSDN fails to mention is that "unhandled" means that this DoWork() handler must not have Try-Catch block around any of its code. We need to let the error happen unhandled, and then the backgroundworker will take care of getting it to the RunWorkerCompleted() handler. I spent some frustrating time being stuck on this point, because it is my normal practice to put a Try-Catch block in *all* event handlers, because an event handler is effectively the top of the stack. The DoWork() handler is now my one exception to this rule. If you want the control to behave as advertised, then you need to let errors bubble up from your DoWork() handler as advertised.
There is another reason to make sure that you conform to the BackgroundWorker's exception handling model: if you want to manipulate any of your form's controls when an exception comes out of DoWork(), then you *cannot* do that in your DoWork() handler. It's very common to need to do something on your form after an exception, such as disabling a button or changing a label. This must be done in your RunWorkerCompleted() handler, as confirmed by MSDN:
You must be careful not to manipulate any user-interface objects in your DoWork event handler. Instead, communicate to the user interface through the BackgroundWorker events.
As long as you don't put a Try-Catch block in your DoWork() handler, you should fine on this point. However, as soon as you craft your code to work this way, you'll run into another effectively undocumented issue: in a debugging scenario, whether you launched the app from within the IDE or from a compiled debug assembly, whenever you have an exception in any of the code that is called from the DoWork() handler, the debugger will stop on what it of course sees as an unhandled exception--damned if you do, damned if you don't.
If you "handle" the exception with a Try-Catch block in the DoWork() handler, the BackgroundWorker won't work, but as soon as you leave it out, the debugger implements it well intentioned by mostly just annoying behavior of stopping on the line that produced the exception. The trick to get around this is to decorate your code with the DebuggerNonUserCodeAttribute attribute from the System.Diagnostics namespace. I discovered this trick in this thread from the MSDN forums. Here's what the attribute decoration of the DoWork() handler looks like in VB.NET:
<System.Diagnostics.DebuggerNonUserCodeAttribute()> _
Private Sub myBackgroundWorker_DoWork(ByVal sender As Object, _
ByVal e As System.ComponentModel.DoWorkEventArgs) Handles verifyFileBWWorker.DoWork
However, I had to take this a step further since in my case what I was calling from my DoWork() handler was a method on a class in a separate DLL assembly--in other words, I was calling code that did not reside within my form, but in another class. What I found was that the debugger still saw exceptions that happened in this child class's code as "unhandled." I solved this by adding the DebuggerNonUserCodeAttribute to the class decoration as well:
<System.Diagnostics.DebuggerNonUserCodeAttribute()> _
Public Class MyClass
Now my BackgroundWorker code behaves in a consistent way whether I'm debugging or not. (Update: an unfortunate side-effect of setting DebuggerNonUserCodeAttribute on the whole class is that the debugger won't visit that class at all, even when you set a breakpoint--so it looks like I'll have to put up with the debugger stopping on "unhandled" exceptions...)
Here's one more gem I was not able to find in the documentation: if, like me, you are implementing the code that you want to call with BackgroundWorker in a separate class and/or DLL (rather than keeping the code in the form, which the simplistic MSDN examples do--a practice I don't recommend if the code in question is not specifically presentation-oriented logic) then you will wonder how you are going to report progress back to the GUI, or check for a cancelation, down in that code that's outside the form. My solution to this was to provide a way for the form to pass the BackgroundWorker object down to the class method. However, part of my goal with putting my business logic is a separate DLL is to make it reusable in other contexts, and I don't want to tie in a hard assumption that the code is being called from a BackgroundWorker. So the way I did this is to make the BackgroundWorker optional in the class. This could be done with a Property, and then the code in the class method could check to see if it's been set:
if (m_myBW != null)
{
m_myBW.ReportProgress(50);
}
You could also accomplish this by making the BackgroundWorker object an argument, but I recommend making the argument optional (in VB.NET) or using overloaded methods (in C#).
Implementing my first BackgroundWorker was more work than it needed to be, but I'll be implementing dozens of these in this application, so I expect the effort to pay dividends. Hopefully this article helps someone else get up to speed more quickly that I was able.
Hope that helps,
Dan
Not that I'm resisting--merely lagging. One of the things I like about my job is that it won't be long before our clients expect us to use this stuff.
For those who are not lagging as far behind as I, Charles Petzold does us a service with this collection of links to get going with ".NET Framework 3.0". When it comes time to get down with some XAML, it looks like Petzold's book will be the one to get (though based on my recent experiences with giganitc the XML files used by SQL Server Integration Services, I'm skeptical about--though admittedly ignorant of--more giant XML files to contend with in my day-to-day development).
Thanks for reading,
Dan
The article narrates dramatic improvements that were achieved in childbirth and mortality rates in the 20th century. Author Atul Gawande describes a stark situation for childbirth in the U.S. in the 1930's:
But in 1933 the New York Academy of Medicine published a shocking study of 2,041 maternal deaths in childbirth. At least two-thirds, the investigators found, were preventable. There had been no improvement in death rates for mothers in the preceding two decades; newborn deaths from birth injuries had actually increased. Hospital care brought no advantages; mothers were better off delivering at home. The investigators were appalled to find that many physicians simply didn’t know what they were doing: they missed clear signs of hemorrhagic shock and other treatable conditions, violated basic antiseptic standards, tore and infected women with misapplied forceps. The White House followed with a similar national report. Doctors may have had the right tools, but midwives without them did better.
The author then describes how childbirth was improved through standardization of techniques, training, and regulation of who exactly was allowed to perform certain procedures (based on whether they had the training and experience):
These standards reduced the number of maternal deaths substantially. In the mid-thirties, delivering a child had been the single most dangerous event in a woman’s life: one in a hundred and fifty pregnancies ended in the death of the mother. By the fifties, owing in part to the tighter standards, and in part to the discovery of penicillin and other antibiotics, the risk of death for a mother had fallen more than ninety per cent, to just one in two thousand.
But the situation wasn’t so encouraging for newborns: one in thirty still died at birth—odds that were scarcely better than those of the century before—and it wasn’t clear how that could be changed.
Later in the article, the author describes huge improvements that came in the second half of the century:
In the United States today, a full-term baby dies in just one out of five hundred childbirths, and a mother dies in one in ten thousand. If the statistics of 1940 had persisted, fifteen thousand mothers would have died last year (instead of fewer than five hundred)—and a hundred and twenty thousand newborns (instead of one-sixth that number).
How did these huge improvements happen?
...a doctor named Virginia Apgar, who was working in New York, had an idea. It was a ridiculously simple idea, but it transformed obstetrics and the nature of childbirth. ... she took a less direct, but ultimately more powerful, approach: she devised a score.
The Apgar score, as it became known universally, allowed nurses to rate the condition of babies at birth on a scale from zero to ten. An infant got two points if it was pink all over, two for crying, two for taking good, vigorous breaths, two for moving all four limbs, and two if its heart rate was over a hundred. Ten points meant a child born in perfect condition. Four points or less meant a blue, limp baby.
The score was published in 1953, and it transformed child delivery. It turned an intangible and impressionistic clinical concept—the condition of a newly born baby—into a number that people could collect and compare. Using it required observation and documentation of the true condition of every baby. Moreover, even if only because doctors are competitive, it drove them to want to produce better scores—and therefore better outcomes—for the newborns they delivered.
Is anyone else seeing the parallels to software development here? Am I off base thinking that the idea of a simple "score" to assess the "health" or "quality" of a software system could be a powerful tool, as it was, apparently, for obstetrics? Such a score, I think, would have to be as simple as described above for the Apgar score, based on easily observable phenomena, and only marginally dependent on subjective factors.
Typical "ility" descriptors used often in software circles would, I think, be ineffective in this context. Is the software maintainable? Is the software secure? These are too fuzzy, too broad, too open to interpretation. We need things more like, is the baby pink or blue? Is the baby crying?
Does the application have an exception handling scheme, or doesn't it? Is there clean separation of presentation logic and business logic, or isn't there? These are things that can be assessed relatively objectively by someone with the proper knowledge and experience.
I've spent the last five months on my day job spelunking through our client's entire enterprise of back office and web software systems, reading tons of old code, combing through databases, and trying to make sense of arcane user interfaces. All this has resulted in about 500 pages of maintenance documentation for my client, who recently purchased a company and wanted to know whether the software it purchased would scale up for further growth and acquisition. I've done similar system assessments many times in the past (though not usually this large), and my gut tells me that assigning a score to each of the systems I've examined would be very useful to my client.
On the front side, with a known scoring system in mind, is the novice (or even "senior") developer more likely to produce software that at least attempts to implement the necessary elements to achieve a respectable score? Would developers working in team environments encounter the same peer-related effects as seen in obstetrics?
The Apgar score changed everything. It was practical and easy to calculate, and it gave clinicians at the bedside immediate information on how they were doing. In the rest of medicine, we measure dozens of specific things: blood counts, electrolyte levels, heart rates, viral titers. But we have no measure that puts them together to grade how the patient as a whole is faring. It’s like knowing, during a basketball game, how many blocked shots and assists and free throws you have had, but not whether you are actually winning. We have only an impression of how we’re performing—and sometimes not even that.
More parallels with software development abound in this article about birthing babies:
There’s a paradox here. Ask most research physicians how a profession can advance, and they will talk about the model of “evidence-based medicineâ€â€”the idea that nothing ought to be introduced into practice unless it has been properly tested and proved effective by research centers, preferably through a double-blind, randomized controlled trial. But, in a 1978 ranking of medical specialties according to their use of hard evidence from randomized clinical trials, obstetrics came in last. Obstetricians did few randomized trials, and when they did they ignored the results.
...
The question facing obstetrics was this: Is medicine a craft or an industry? If medicine is a craft, then you focus on teaching obstetricians to acquire a set of artisanal skills—the Woods corkscrew maneuver for the baby with a shoulder stuck, the Lovset maneuver for the breech baby, the feel of a forceps for a baby whose head is too big. You do research to find new techniques. You accept that things will not always work out in everyone’s hands.
But if medicine is an industry, responsible for the safest possible delivery of millions of babies each year, then the focus shifts. You seek reliability. You begin to wonder whether forty-two thousand obstetricians in the U.S. could really master all these techniques.
For the record, I've always been skeptical of software engineering metrics and their usefulness in the software shops where I've worked. (See my "Balance in Scoring" comment, below.) But something about the directness, simplicity, and apparent effectiveness of the Apgar score struck me. The author of this article, himself a physician at the Harvard School of Public Health, was it seems struck by it also:
In a sense, there is a tyranny to the score. Against the score for a newborn child, the mother’s pain and blood loss and length of recovery seem to count for little. We have no score for how the mother does, beyond asking whether she lived or not—no measure to prod us to improve results for her, too. Yet this imbalance, at least, can surely be righted. If the child’s well-being can be measured, why not the mother’s, too? Indeed, we need an Apgar score for everyone who encounters medicine: the psychiatry patient, the patient on the hospital ward, the person going through an operation, and the mother in childbirth. My research group recently came up with a surgical Apgar score—a ten-point surgical rating based on the amount of blood loss, the lowest heart rate, and the lowest blood pressure that a patient experiences during an operation. We still don’t know if it’s perfect. But all patients deserve a simple measure that indicates how well or badly they have come through—and that pushes the rest of us to innovate.
Even if this is a good idea for software, obviously many details would need to discussed and worked out. For example, would we really need multiple scoring systems for different kinds of software? However, I'll stop writing at this point and assess whether there is any further interest in this idea. What do you think?
Thanks for reading,
Dan
P.S.
I've quoted heavily from it here, but I recommend the full article as a worthwhile read.
I'v been a big fan of Scott Rosenberg's writing for several years now at Salon.com, which he co-founded. (I happen to mostly identify with the political slant at Salon.com, but even if you don't, in my opinion you have to admire the way they have survived and thrived as an online-only publication through many years that people said they'd never make it--Scott's role on the technology side of Salon, not to mention his autorial and editorial roles, were a big part of that success.)
Scott has a new software-related book coming out in January 2007, which sounds interesting. It's called Dreaming in Code, and it will be published by mainstream trade publisher Crown--a feat for which Scott deserves a special congratulations. We'll offer more coverage of the book here in the future.
Dan
]]>