Systematic Source Formatting

When my team switched to Extreme Programming in March 2003 (XP by 3/03), we decided to try all the practices and all the recommendations. There were plenty of bumps and bruises as we learned what worked for us and what didn't work for us. We learned that the original XP descriptions were too light on testing for our business needs, and we learned that integration tests worked better for us than pure unit tests with mock objects. Those can be topics for another time.

One of the surprisingly effective results of that adoption of all the practices and all the recommendations was using automated source code formatting to assure all our code looked the same. We had a diverse team of people developing the code, some in the U.S., some in Germany. That diverse team of people had different opinions and attitudes about the "one true way" to format source code. Those different attitudes and opinions lead to annoying little changes from one file version to another as the files changed hands from one person to another.

Since XP espouses "no code ownership" and allows anyone to be anywhere in the code, those formatting differences tended to worsen as people moved through the source tree.

We took a "top down edict" approach to implementing "no code ownership". I decided (as the manager) that we would use a program to format all our Java source code as part of each compile. The concept was that developers would no longer need to think about formatting their code, a program would do it for them. We inserted the open source version of the "jalopy" source code formatter into our standard build process and then added an additional step to systematically format the code once a day with the same process, in case someone forgot to format before they committed changes to the master.

The initial adoption of the change caused some angst, since the chosen format (Sun Java code formatting guidelines) disagreed with the "one true way" which some of the team expected. We pushed our way through that initial hurdle and found over the course of years that automated source formatting freed us from a number of problems.

Positive results of our switch to automated source code formatting:

• Code looks the same no matter who wrote it
  
• Changes in the source master are "real changes", not white space shuffling
  
• Developers don't need to think about source code format, they can insert code however they want and the formatter will fix it to meet the standard
  

Negative results of our switch to automated source code formatting:

• A "diff wall" was created at the point where we first ran the automated source code formatting tool. The automated formatting created a single massive change to the source master to convert from the old ad-hoc format to the new consistent format. I worried at the time that the "diff wall" was going to be a real problem, but it never appeared as a problem in our work
  
• Command and control style management is frequently a bad pattern and should be reserved for rare occasions (our major process change from waterfall to Extreme Programming was done as a manager dictate from above, and this was "hidden" in that process change)
  

There are newer tools to support automatic source code formatting (astyle seems to be popular) and I hope to persuade my new organization to adopt automated source code formatting. I'm not in a position to be the dictator in the new organization, so changes like that are more difficult to create.

Deliberate Practice - Mary Poppendieck Talk

As a manager at PTC, I need to write a performance appraisal for each of my direct reports. I will also receive a performance appraisal from my manager to help me identify strengths and weaknesses and set goals for the coming year. While preparing my own self-evaluation, I remembered reading about the concept of "deliberate practice". I'd already been trying to apply my understanding of the idea, but decided to listen to an original source before including more about it in my self appraisal.

Mary Poppendieck's Agile 2009 deliberate practice talk describes deliberate practice as practice which is intentionally focused on improving performance. I want to improve my performance. Mary highlights four key components that are required for a person to be using deliberate practice:

• Mentor - a high skills expert to review, critique, and highlight flaws


• Challenge - tasks that require greater skill than we currently possess


• Feedback - review and analysis of results used to improve future attempts


• Dedication - hard work, time and energy applied diligently

I don't have a mentor for the things I'm trying to improve! At least, I haven't identified someone as my "coach" and had them agree to review, critique, and highlight flaws in my performance. Flaw number 1 (and the most glaring flaw) in my recent improvement attempts.

Many of my tasks and assignments are the same assignments I've had before, coordinating, planning, discussing, and interacting with others. However, I always find those tasks challenging because I don't feel like a naturally social person. Negotiating, discussing, persuading, and debating do not come naturally to me. There is room to improve in the "challenge" area, although I don't see it as making as much an improvement as the absence of a coach in my improvement efforts.

Most of my activities have feedback built into the activity in one form or another. Meetings which start on time, stay on task, complete their objectives, and end on time leave me feeling refreshed, invigorated, and useful. Meetings which start late, wander aimlessly, don't have objectives, or extend beyond their scheduled end leave me frustrated and edgy. That's a form of feedback. However, there are other feedback forms (like team retrospectives) which I've not been using faithfully lately, and need to start using again.

Dedication may actually need some "negative attention", since lately I've been spending too many hours at work and not enough hours with my wife and children. Mary's talk notes that expert performers in other fields (like music) have discovered that they cannot apply more than 3 hours of deliberate practice at a time because it is too tiring. They stop, change tasks, take naps, or otherwise refresh themselves rather than continuing, and risking developing bad habits by practicing poorly.

With the identified gaps in my efforts to improve, now it is time to choose a mentor and start hearing the performance critiques, then acting on them. Finding a mentor seems like a difficult task for a manager. I want a mentor that is

• Regularly and naturally exposed to my performances, without requiring that I present them a summary of what I did. Swimming coaches do not ask the swimmer to describe the most recent swim, they watch the swimmer and then tell the swimmer directly and openly what could have been improved in that swim. My mentor needs to be someone who "watches my performances" on a regular basis as part of their normal work


• Able to spend time and energy critiquing my performances with focus on improving them


• Credible as an expert. My request for coaching is an act of trust and that extension of trust requires that I believe in the skills of the person providing the coaching. I may disagree with their perspectives, challenge their ideas, and still want their coaching


• Interested in my success. Without interest in my success, I doubt the coach can be trusted to provide excellent feedback

I'm sure there are other things I need in a mentor as well, but that list already worries me. Those who are regularly involved in my work tend to be my direct reports, my peers, and my manager. My direct reports aren't very interested in coaching their manager (other than possibly to smile with him about his many failings). My manager is already a mentor by being my manager. My peers and I frequently disagree on methods and techniques and so I'm not sure peers are the best source of mentors either.

All this musing might also be less useful if instead of a mentor, I need to become a "buccaneer scholar" as suggested by James Bach. James suggests that I should take responsibility for my own education and for my own learning. That may make seeking a single perfect mentor a waste of effort, rather I could accept that there are mentors all around me and those mentors can provide useful information at times, and information to be ignored at other times.

Another alternative is to consider Bob Sutton's questioning of the value of annual performance appraisals. So many ideas to consider, so many things to learn (the act of writing this has already taken me on a different path than I expected when I began...).

Was Our Switch to Git a Mistake?

My team is part of a larger, multi-team, multi-site organization in our company. The corporate choice for software configuration management is ClearCase. Unfortunately, my team is "remote", and "small". ClearCase does not handle remote teams well. I could find more polite ways to say it, but that is the simple result of our experiment. ClearCase was costing us far too much time due to its poor performance over a wide area network.

We solved the ClearCase problem by creating a bidirectional bridge between ClearCase and Subversion. My team's interface to the source master became Subversion. We saw great performance improvements, and were seeing the ClearCase updates within a few minutes of their arrival on the ClearCase master. The bridge was conceptually very simple, and it worked very, very well for our needs. Life was good.

We were then faced with a new, even more challenging problem. The new development work needed to be spread even more widely than the previous development work, with a larger number of teams involved and more dispersed geography. The new work would create multiple products, each with their own release cycle and their own development lifetime.

At the time we were making this transition, I'd been experimenting with the git version control system. Git is version control software created by Linus Torvalds when he needed to switch from BitKeeper for Linux kernel development. It runs very, very well on Linux. It is significantly faster than Subversion, which (in our environment) is significantly faster than ClearCase. Performance looked like a big winner.

Another challenge of the new environment was branch related. The new teams thought their development model would likely be "branch intensive". My prior experiences with branches have been with CVS and Perforce, where branches are globally visible and merging things between branches is a hassle. I hate branches. However, considering that the new world would be "branch intensive" and Subversion is generally not perceived favorably for branch management, we didn't want to use Subversion in a "branchy" environment.

With those two needs, distributed teams and branch intensive environment, we skipped Subversion and went looking. My recent git experience (and recent Mercurial experience) biased me in favor of a distributed version control system. A key opinion leader in the company had also been using git in a subteam of a very large project, pushing their results back to ClearCase. They reported positive results. My experience had also been positive while I was experimenting with taking a work project off on a "tangent". Git worked well for me, sitting on my underpowered Linux box doing my personal "skunk works" project.

We chose git as the team source control system.

Unfortunately, I had failed to detect my own biases, and the biases of the other early adopters of git. Those biases were very different than the biases of my co-workers.

I'm a command line fan. I'm old enough that my first high school experience programming computers was with the newly installed terminals to the school district mainframe (thanks Davis School District and Layton High for spending the money, the time, and the pain to install those machines!), then I moved to a University that required I submit programs on punch cards (makes me sound old). Before I left the University, they had upgraded to dumb terminals communicating with a DEC minicomputer.

As a command line fan, I found the git "user interface" perfectly comfortable and very similar to CVS, Subversion, and Perforce. There were a few surprises while I tried to understand distributed version control, but those surprises were related to version control concepts, not the specifics of git.

Unfortunately, many in my team and in other teams are not command line fans. They are accustomed to productivity accelerators like graphical user interfaces, integrated development environments, and mouse clicking to perform work much faster. The transition to git has been painful for them. In addition to my transition experience (centralized vs. distributed, new commands, new concepts), they've also had to deal with transitions from robust GUI tools (TortoiseSVN, Perforce Windows client, etc.) to weak and brittle GUI tools (GitSVN, gitk, git gui, etc.).

The challenge has been made worse by our decision as a management team to isolate teams on branches. Two of the managers in the team come from a large scale development organization (5-10x larger than our current organization) and they are accustomed to requiring branches as a way to isolate one team from the potentially breaking changes made by another team. The price of that branch isolation is that we now are required to perform more frequent merges of work, with the resulting complexity and frustrations which come from merging with conflicts. It gets worse when the files to be merged are coming from the Visual Studio IDE, and the meaning of the contents of the files is not always clear.

I think the branch configuration decision has done more damage than the choice of git, but that is probably biased (again) by my command line centric mindset. Unfortunately, we're far enough into the project that we aren't willing to switch SCM systems. We'll remain with git for at least the duration of this project, glad to have a source master, glad to have it connected to our continuous integration servers, and glad to not have the awful performance of remote ClearCase.

In all fairness to git, I still remember the growing pains when we switched from CVS to Perforce. I whined mightily at paying hundreds of dollars per developer for our corporate standard SCM system. Then I whined mightily at the tool changes and use model changes forced upon us by Perforce's way of thinking. After 6 months or a year, I discovered that I had changed my way of thinking, and was now very comfortable using Perforce, getting value from its way of branching, and being very grateful that it was so fast.

Maybe 6-12 months from now I'll say the same things about git. Maybe it is a part of "climbing the learning curve", and unfair to judge our experience this early. Or maybe not...

I still don't know what we should have chosen instead of git, since it is not clear to me that there were any better alternatives for my team at that time. The company was not willing to purchase another SCM system, since they were already paying for ClearCase. That excluded all the purchased SCM systems (Perforce, Microsoft Team System, Accurev, BitKeeper, etc.). The teams were known to be widely distributed, so that pushed us towards distributed SCM. The benchmark comparisons suggested that Git was faster than Mercurial in many operations, and the Bazaar people were still not settled on their final "on disc" format. Subversion was not well perceived for handling "branchy" development, and CVS was worse than Subversion.

The Linux kernel handles massive amounts of change (averaging 2-4 changes per hour continuously for the last 4 years) from many, many developers. It scales well for that widely distributed, branch intensive team, yet we're struggling with it. Of course, Linux kernel developers are even more likely to be command line biased than I am, and scaling the tool is not the issue that is getting in our way, it is more our choice to be "branchy" and the user interface weaknesses in git.

So many things to learn, so little time...

Dangerous Interruptions

Bob Sutton's blog post on reducing medical interruptions reminds me of Sunday mornings when I take my mother-in-law from her nursing home to church. I frequently interrupt or disrupt the nursing staff with my out of sequence, unpredictable arrival, and with my desire to get "Nana" to church on time.

Nana's trip from the nursing home to church starts from her room at the nursing home. I arrive at her room between 10:20 AM and 10:40 AM (depending on how late I arrive from home). Sunday is the only day we do this, so I tend to disrupt all sorts of people at the nursing home, including the nursing staff.

My mother-in-law is a diabetic, and church runs over the noon hour when she would normally receive her medications. The nurse would normally check her blood sugars right before lunch, then based on the results of that blood sugar test, she would select the proper dose of insulin, draw that dose, and administer the dose to Nana. Each of those steps has a potential for error, and each of those steps needs careful thought and attention to detail by the nurse.

Because I arrive as much as 90 minutes prior to lunch, and Nana will be gone for the three hours of church, the nurse is required to interrupt her current medication process, test Nana, medicate Nana, and then return to her previous task. The nurses are always very kind about handling the interruption, and they provide great care. I worry that my interruption may cause them to make unnecessary mistakes...

The study which Bob Sutton references was performed in the UCSF hospital system in San Francisco and is described in a San Francisco Chronicle article. The study was an attempt to reduce the frequency of medication errors at hospitals. They used both low-tech solutions and high tech solutions to reduce medication errors by nurses.

The low tech solutions described in the article focused on reducing nursing interruptions when administering medication. The article describes "do not interrupt" sashes and vests, closing blinds to prevent distractions, and other relatively simple techniques to reduce interruptions during the crucial activity of administering medication. The article noted that the nursing teams were encouraged to develop their own solutions, within their own working environment (own your process). It appears from the study that nurses administer medications (a detailed technical task) less accurately when they are interrupted than when they are undisturbed. It also appears that nurses allowed to explore improvement techniques tend to improve.

The study may not directly apply to my software development team, but I think there are several lessons I should take from the study. They are lessons others have noted, but the article serves as a good reminder.

• Interrupting technical work (pair programming, software design, software testing, etc.) increases the chances for error. I need to interrupt my people less
  
• Allowing and encouraging people to improve their own processes, their own ways of working is likely to generate improvement. I need to find ways to acknowledge my mistakes openly, learn from those mistakes, and encourage others to do the same. A software bug is a late manifestation of a mistake, mistakes will happen in human endeavors, and we want to learn from those mistakes, not hide them until later
  
• Fear of failure tends to hide those failures, particularly in organizations with a culture of fear. Sutton's posting notes that hospitals which acknowledge and seek to reduce their drug administration errors tend to report 10x more drug administration errors than units with a more punitive attitude towards errors. The failures will still occur, but they will be discovered later, and likely be discovered with more damage done, or higher costs incurred from the failure. It would be a gross mistake to declare the nursing unit which reports 10x the drug administration errors as a failed unit without further investigation. If the clinical results of the unit are better (fewer deaths, fewer injuries, lower costs, etc.), then the larger number is actually highlighting their good practice of learning, rather than the bad practice of medication errors. Don't worry about bug counts, worry about what bugs can tell us about how to be better
  
• "Best practices" at one location were not necessarily "best" at another location, although sharing practice based experiences seems like a good way to learn from mistakes and thus make fewer mistakes in the future.
  

Learning and Responding

A mistake was made today. Code was merged from one branch to another branch and the destination branch was broken in its intended use. The break was detected late in the day of the team that caused the break, and they had already mostly left for the day. The break highlighted all sorts of weaknesses in how I was handling things, including things like

• Why didn't I make it clear to everyone both the purpose and the target configuration of each branch? Poor communication I had not made it clear what the purposes and expected configuration was for each branch, and they assumed that since they could see the branch building in one case, that was sufficient
  
• Why didn't the person who performed the merge detect the broken build on the continuous integration server? Unclear information sources. We had configured 3 different continuous integration servers because we needed three different configurations. Unfortunately, I then "muddied the waters" by having one of the branches made compatible with all three configurations, and actively visible on all three configurations. When the developer performed the merge, they saw that it was "green" on the screen they were watching, and thought they were done. It had gone "red" on the other two servers, and those two were the most important to my team
  
• Why wasn't the team which performed the harmful merge able to repair their damage? Unavailable spare configuration They had no machine available to them which matched the problem configurations and could be used for diagnosis and development. Their machines were all configured for their needs, and the break was in an area needed by other teams
  
• Why did it take half a day to recover from the damage? Inexperience with our tools We recently switched from Perforce to Subversion to Git and the transition has left us less skilled in dealing with the complexity of this type of failure.
  

All told, the damage cost my team less than a day to recover, and because we're using a distributed version control system, they were able to continue their work locally, but they were not able to push to the central repository.

Moral of the story: Communicate clearly, listen carefully, and be willing to change as better ideas arrive

Ask More Questions - Get More Answers

I had been tolerating a nagging problem with my web browser on multiple machines and now have a solution, because I had the presence of mind to finally ask a question.

Firefox is my preferred web browser because it includes the Gmarks plugin. The Gmarks plugin brings my google bookmarks into the Firefox menu. That makes portable bookmark management easier (all my bookmarks are stored at Google, visible from any web browser on the internet).

I also prefer the Foxit PDF reader instead of the Adobe reader. It feels faster, cleaner, and seems less likely to be attacked by malware (smaller installed base, newer code base).

Unfortunately, Firefox would report "OCX failed to load" when I tried to open a PDF file. I had found all sorts of strange alternatives for opening PDF files in Firefox. For example, sometimes I would download the PDF file, then open it in Foxit from the local machine. Other times I would copy and paste the URL from FIrefox to Internet Explorer, then use Internet Explorer to open the PDF file.

All those strange alternatives (work arounds, fixes, etc.) have now stopped. I was weary of the alternatives, so I used Google to search for the error message. In classic Google search fashion, the first page had a perfect match for my needs, a post which described my problem and an easy solution to my problem.

The moral: Ask questions sooner, don't be afraid of questions or their answers. I'll need to think more about why I didn't ask the question sooner...

Feedback Junkie

I was reminded again this morning that I am a "feedback junkie". My team is on a new project and the backlog items are starting to appear as small working pieces of code. In addition, we're connecting with other groups in the company and connecting them to our continuous integration server, showing them how it works, showing them why it helps, and hoping that together we can arrive at working faster because we have better feedback systems.

We've already exceeded my 5 minute threshold on the feedback system. My arbitrarily chosen rule of thumb is that we need to know a checkin is "good" or "bad" within 5 minutes of that checkin. I know that is not always possible. I know there are many tests and sets of tests which will take more than 5 minutes. Still, that 5 minute barrier is the goal, and we'll keep splitting, partitioning, and refining to keep the feedback system within those 5 minutes.

Now, we continue fomenting revolution, getting other people just as addicted to feedback as I am...