End of day anti-pattern

I made a foolish mistake today. I should have known better... Now it is time to resolve to make new and different mistakes tomorrow, instead of making this same mistake yet again.

Wait For Green

Two of the programs used for our installation only make sense when run as an administrator. The Windows Vista and Windows Server 2008 user account control (UAC) code does not allow administrator privileged users to run programs as an administrator unless they explicitly select "Run as Administrator", or if the program has been marked with a manifest which tells the operating system to always run this program as an administrator.

Those two programs were using the default manifest, and one of the two was adding itself to the "Start Programs" menu. When a Windows Server 2008 user clicked the program with user account control enabled, it crashed painfully and offered to send a report of the problem to Microsoft. We didn't want that.

Visual Studio has an easy way to add manifests to applications, and it seemed simple enough that even a manager could do it...

I made the change, compiled the code, reviewed the code with a real programmer, and let it sit for a few days. This evening, just before leaving for home, I submitted it to the source master and left the building for the swim team dinner.

Don't Do That!

That was a mistake. The compilation failed on the production build machine, even though it had worked flawlessly on my machine. Apparently the production build machine has a different version of one of the compiler components in its path, and that different version did not recognize the options inserted by my version of the Visual Studio IDE. Ugh.

Check the Continuous Integration Server After You Commit!

I've reverted my change, but not after keeping someone else up late into the night because I was not paying attention to the results of my code change.

(There's another thing which is "have fast continuous integration", but we're still working on that...)

Personal Continuous Integration

I've been experimenting for some months with distributed version control system (Mercurial and Git) on various personal projects.

The most interesting of the personal projects was a branch from an active code base that I wanted to extend in a slightly different direction. Because my changes are going in a slightly different direction, they were not likely to be included in the original code base for a very long time. That meant I was going to be "on a branch" from that original code base for a very long time.

Past experiences with CVS and Perforce suggested that being on a branch for a very long time could be a problem. Perforce was not as bad as CVS, but still it was difficult to maintain a separate personal branch and "keep it healthy". Inevitably I would make changes that were harmful to the main code, and not realize I had harmed it because I was not getting all the feedback that is available to developers on the main code.

The main code developers have a continuous integration setup which runs automated tests in many different configurations and summarizes the results to a central location. The results are easy to browse and easy to watch as they evolve.

Since I'm not on the main code, I don't have that nice infrastructure to support my private branch. With distributed version control, I have all the power of a version control system (incremental checkin, revert to previous point in time, branching, merging, etc.). Why can't I have all the power of a continuous integration server for myself as well.

Why Not Use My Own Machine?

I'm sure others have already realized this, but with the ease of installing, configuring, and using the Hudson continuous integration server, I can run my own continuous integration server which compiles and tests code from my personal version control, before it is ever pushed to any other person (or system) in the organization.

Setup Idea

1. Install Sun JDK (needed by Hudson)
   
2. Install Hudson continuous integration server
   
3. Run the Hudson continuous integration server
   
4. Install distributed version control (Mercurial or Git or ...)
   
5. Install Hudson plugin for selected distributed version control
   
6. Clone the development repository from the company central location
   
7. Create a new Hudson job which monitors the local repository, checks out the source from the local repository when something changes, compiles it, tests it, and reports its results
   
8. Start making local changes, checking them in, and enjoying the benefits of continuous integration test runs while developing the code, without the danger of checking into a central repository before the code is "done done"
   

I think those setup steps could be reduced to less than a day, and that day would repay itself within the first 4 weeks of work as developers were more confident in their changes before they shared them with others. I don't yet know if the idea will work for "real" developers, since I'm the manager trying to provide them the tools and environment to be successful, rather than a full time developer. We'll see if the idea actually reduces the feedback loop, and if it actually is viewed positively by the developers. What Will They Value? I think developers will gain

1. Rapid feedback on their changes
   
2. Background work (compiling and testing while they work on other steps)
   
3. Isolation of their changes from others until they are "done done"
   
4. Visibility to the impact of changes from others in their test environment (integrating changes from others onto their branch is a "trigger" event, just as checking in their own changes is a trigger event)
   

Five Whys and Four Fingers Pointing Back At Me...

A bug report at work traveled a somewhat strange path as we tried to deduce the root cause of the problem. That strange path reminded me that very frequently when I follow the "Ask Five Whys" heuristic, I discover that there are things which I can change which will improve the situation.

In this particular case, a bug was found, fixed, and then reopened because it was apparently not fixed. There were then several e-mail exchanges between various people as they tried to deduce why the bug was still not fixed. The submitter was confident that the bug was still in the software, so it could not have been fixed. The fixer was confident the change had been submitted, so it must be a problem somewhere else. Others in the conversation wondered if there were additional complications which had not been considered. All of those ideas (and more) could have been correct.

In this specific case, a series of simple gaps were enough to mislead us all.

1. A translation mistake was discovered in late March
   
2. The bug report was assigned to the wrong person, but e-mail exchanges alerted the translation team that the bug existed and needed to be fixed
   
3. In early April the corrected translation was added to the source master
   
4. Just before the corrected translation was added, a new build was generated as part of our once a week schedule of builds
   
5. The submitter tested the fix with the build just prior to the fix
   
6. The e-mail discussion was then started trying to understand why the bug was not fixed
   

When I started asking "Five Whys", I thought it was obvious where the problem originated, and even how to fix it. The bug had been sent to the wrong person, and then when the bug was fixed the bug report was not updated to show which build included the fix.

However, as I stared at the problem further, I realized there was a more significant problem than I had seen initially, and that more significant problem has caused other issues as well.

Why did the fixer need to waste the time guessing which build would include the fix. Couldn't a system tell the submitter when their bug fix was in a build? For example, most bug fixes will reference the bug number in their submit message. Why not pass that information automatically to the submitter, or to the bug report so the fixer does not have to think about the number or name of the next build.

That would have helped, but it appeared that a bigger problem was that the tester did not have easy access to the list of changes which had been made in the build being tested. That list of changes was difficult to find, and difficult to read (I don't find CruiseControl output especially friendly) and probably not known to the submitter at all.

Make Available Information Reachable - Reduce Guessing

When information is not readily available to our very smart people, they will apply skill and judgment and make the best assumptions they can with the information they have. Making that information more readily available will allow them to do their jobs better and reduce wasted work.

The root problem seemed to be someone else's issue, until my thought processes came back to highlight that it was really my problem. I'm the manager, and ultimately it is my fault. Sometimes it becomes obvious more quickly, other times it takes a little more time...

I admit was well that making information easily reachable by those who need it, when they need it, is only part of the answer. That would have helped the tester, but did not help the submitter send the bug to the right person, nor did it help the fixer insert the right data in the bug report. There are so many "why" questions to ask, and so many ways to make small improvements that might help a little.