Team Centered Bug Finding

We've agreed to deliver a "release ready" version of our internal use components every three weeks. That is a nice step from the "deliver once a year" model we've used in the past, and we're trying to make the transition to the new release cycle successfully.

As part of that transition, we've returned to a practice we had before. We hold once a week test days for the entire development team. "Test Friday" is dedicated to testing.

Positives

• Test day happened! It feels good to dedicate one day per week to whole team testing
  
• Color printer sitting in the pairing area and pens for handwritten notes on the sheets was a great way to rapidly record bugs as they were detected, without too badly interrupting the flow of testing for entering bugs into a bug tracking database. It made a difference that the printer was within a few steps of the developer desks
  
• Bug triage at the end of the testing day with the papers on tables in a large conference room seemed to work reasonably well
  
• Bug pages with pictures were much quicker to process than bug pages which only contained words
  
• Chance to use products "end to end" increases our exposure to customer experiences, helps us understand how the products feel, and ultimately will help us make better products
  
• Native language speakers make localization testing much more effective (German and Russian native speakers in the team are a great help)
  
• Having a "reference machine" with a previous version of the products was a good idea
  

Negatives

• System setup too time consuming - some of us spent most of the test time configuring systems to meet the base requirements before we could install and test our own code
  
• Builds had to be taken from a "temporary branch" because one of the other teams in the company had provided code which was not ready to install. Building on that temporary branch meant using a temporary machine, and the temporary machine was not configured the same as the standard build machines, so it did not sign the executables
  
• Inexperienced with bug detection techniques to rapidly detect if there is a problem in "known hot spots". Log file analyzers and other bug detection oracles need to be added to the code and used in our investigations
  
• The "reference machine" did not have enough configuration to use as a reference in the most important areas