As part of that transition, we've returned to a practice we had before. We hold once a week test days for the entire development team. "Test Friday" is dedicated to testing.
Positives
- Test day happened! It feels good to dedicate one day per week to whole team testing
- Color printer sitting in the pairing area and pens for handwritten notes on the sheets was a great way to rapidly record bugs as they were detected, without too badly interrupting the flow of testing for entering bugs into a bug tracking database. It made a difference that the printer was within a few steps of the developer desks
- Bug triage at the end of the testing day with the papers on tables in a large conference room seemed to work reasonably well
- Bug pages with pictures were much quicker to process than bug pages which only contained words
- Chance to use products "end to end" increases our exposure to customer experiences, helps us understand how the products feel, and ultimately will help us make better products
- Native language speakers make localization testing much more effective (German and Russian native speakers in the team are a great help)
- Having a "reference machine" with a previous version of the products was a good idea
- System setup too time consuming - some of us spent most of the test time configuring systems to meet the base requirements before we could install and test our own code
- Builds had to be taken from a "temporary branch" because one of the other teams in the company had provided code which was not ready to install. Building on that temporary branch meant using a temporary machine, and the temporary machine was not configured the same as the standard build machines, so it did not sign the executables
- Inexperienced with bug detection techniques to rapidly detect if there is a problem in "known hot spots". Log file analyzers and other bug detection oracles need to be added to the code and used in our investigations
- The "reference machine" did not have enough configuration to use as a reference in the most important areas
2 comments:
Hey Mark,
Sounds like you are running something similar to what I know as a bug hunt, just a lot less structured.
I would recommend you read more about the technique since it may help you get other useful tips to enhance the results of your Testing Fridays.
In the past I wrote about what I do when running hunts - http://qablog.practitest.com/2008/05/what-do-you-pack-when-you-go-for-a-bug-hunts/ - again you should take only what suits your team.
Cheers!
-joel
Joel, thanks for the pointer to your blog entry on bug hunts. I'll certainly consider the ideas there for our future tests.
I was surprised that you said "a lot less structured" when describing the exercise we used.
It hints that I failed to describe all the prioritizing and preparing we did before that first test day. We reviewed the business priorities for our first internal release, identified risk areas with the developers, created virtual machine environments for their testing, and planned the infrastructure we'd need to run the tests.
The day of the test we started with a brief discussion in the team to understand who would be exploring which areas (reduce risk of duplicated effort).
At the end of the testing day, we reviewed the effort and tried to identify areas we can improve before the next test day.
As an indicator that we used some of the techniques you suggested, I paired with our native German speaker most of the day. I've liked paired testing in the past for some cases (typically where the two halves of the pair bring very different skills).
Since this was the first test day, I preferred to avoid soap opera style testing, instead trying to traverse the application end to end without the "high drama" of soap opera events. Soap opera will likely be a future technique (it was in the past with our team, typically when we felt our bug finding effectiveness was decreasing with our existing techniques).
I've tried tournament techniques in the past and found that they were a net negative in my team. Other teams probably have different results, but for my team (been together for 8+ years), the subjectivity of judging bug finding results by counts or weighted counts or severities or any other numeric measure has been unhelpful. They don't enjoy the results and they tend to be more demotivated by the comparisons than motivated by them.
You mentioned using a tracking system. Our tracking system was printed pieces of paper that we use to provide a "bookmark for a conversation" with the person who found the problem. Inside the team I want to fix bugs fast, so I don't want the overhead of a bug tracking database with forms and templates and fields. If bugs escape from the team, then I'm OK with using a bug tracking database.
I do appreciate your suggestion to publicize the test day. I used that technique to spread the word to others outside the immediate team so they were aware we were testing, and could choose to participate if they wanted. No one chose to participate from outside the team.
Post a Comment