One of my favorite (and most popular) Dilbert cartoons of all time is the one where the Pointy-haired Boss rolls out a new incentive program to the engineering team by offering $10 for each bug fix.
As funny as the unintended consequences this kind of incentive would have, the truth is creating meaningful performance metrics for a software engineering team is difficult. For a project management company like AtTask where a culture of using metrics to measure performance is part of who we are, this presents challenges.
One of the more popular ideas we’ve incorporated into our development process is something we call Bug Week.
During our 3-week sprints our QA team collects all of the internally and customer reported bugs and works with the engineering and support team to assign "Bounty Points" that would properly reward engineers based on complexity, impact, priority, etc. Beginning at midnight after our Friday sprint demos, Bug Week begins with the following rules of engagement:
- An engineer can only assign himself 1 bug at a time.
- No stealing bugs.
- Engineers are rewarded the Bounty Points for successfully fixing the bug.
- An additional point is earned if the bug required a test case.
- A penalty of 2 points is given if QA reopens the bug.
What makes this really work are the incentives.
For individuals, we have prizes (cash and gift cards) for the Top 3 point earners for the week. The top engineer adds his name to the Bug Week Champion plaque mounted in the office. A bit cheesy, yes, but also a fun way to earn some bragging rights over your cube mates for a few weeks.
The real kicker, however, is that once all bugs are fixed, the test suite passes, and the new build is ready to go, everyone is free to leave for the weekend. Congrats! Good Job! And, yes, even if it’s only Wednesday.
Since this program has begun, we’ve gained numerous benefits:
Increased focus factor in sprints. Fixing bugs during sprints often requires a lot of context switching between production and development branches of code. Reducing this kind of thrashing makes engineers more efficient.
Bug fix rate/Code quality has gone up. Not only are we fixing more customer reported bugs (30-40%) in any given 4 week period, but this has also led to a significantly lower new incident reporting rate.
Bug Week gives our product owners more time for sprint preparation. Running an effective Scrum organization requires a well-maintained product backlog. Allocating additional engineering time during bug week for story estimation and story requirement review allows for much more efficient sprint planning meetings the following Monday.
Individuals can shine, but teamwork creates additional benefit. As anyone who has been on a scrum team, you know that team morale and team productivity is more important than any individual. Creating a competition that allows for your stars to get some recognition while keeping it fun for the entire team has made it a great contrast from the repetitive nature of sprints.
We know if we’re getting better. From a work management perspective we gain some additional data that can be used to learn how we’re performing, both individually and collectively.
Bug Week may not work for every software development organization. However, I believe by adhering to some general principles, it can be adapted to work for most teams. In my next blog post, I’ll review some things we’ve learned along the way and outline some basic principles that should be remembered when trying this in your organization.
This sounds like a bad idea from Dr. Deming’s point of view. According to Deming incentives corrupt the communication system in the company; they focus the organization on individual performance instead of on the system within which people work; they optimize part of the overall system and sub-optimize the rest; they reward the lucky and punish the unlucky. Do you agree or disagree with Deming? How you make sure that you avoid the possible pitfalls? I have experience on continuously deploying consumer portal over past 3 years to production and we have had about 30 bugs in total (28 of those being layout issues with very old browsers), 2 3rd party integration bugs that have been fixed by stop-the-line practice. According to my experience (also in my customer’s development teams) you do not need suggested incentive-based arrangement to get high quality software – engineering discipline, practices (like ATDD) and process supporting the people doing the work will be enough.
Great question. The short answer is while I agree with many of Deming’s principles (those that align well with agile project management philososphies such as sustainable pace, continual improvement, breaking down barriers between departments, focus on quality, etc.), I don’t agree with the strictest interpretation of his principles that by recognizing any kind of individual accomplishment it will, in every circumstance, corrupt the department. Please keep in mind that the Bug Week program we have adopted addresses a common problem that vexes many software teams, namely, how to allocate time for bug fixes in sprints while continuing to add new capabilities to the product. It is a period of time between sprints for our team to address a critical business need while catching our breath. So far, it has proven quite effective. That said, and to your point, organizations should always be reviewing its programs and organization to learn what’s working and what should be improved. In my next post I will talk about reasons why I believe this has been a good cultural fit for our group and how we’ve tweaked the program to keep it effective.
Let me open up how we have found non incentive driven ways to solve those same common problem: Problem #1: “How to allocate time for bug fixes in sprints while continuing to add new capabilities to the product?” Solution #1: – If you want to sprint (we don’t any more), then just allocate time from your velocity (as story points if that’s what you measure) to bug fixing. Some teams have used 20% of their velocity for such tasks. The amount depends on your quality debt. Solution 2: – Move to non-timeboxed model and practice stop-and-fix. We do it, see: http://huitale.blogspot.com/2010/03/huitale-way-our-value-stream-map.html Solution 3: – Estimate biz value for fixing the bug (not all bugs are worthy, really) and handle them as usual backlog items (relative to other stories).
Love the idea of Bug Week. I might have to give that a try if we can find enough bugs.
Great first blog post Nate. Looking forward to reading your regularly. I’ve added you to my Google Reader.
I would like to extend Marko’s original comment on Dr. Deming’s approach. Dr. Deming’s concern was that individual contribution, in the vast majority of cases, can not be objectively determined. In any measurement scheme, there will be a first and a last, however, this does not imply that anyone measured did anything significantly different – either better or worse. Trying to identify “stars” can be very destructive to a team – it is better to spend time and effort to bring everyone up a level than to reward someone on a typically arbitrary measure. I’ve seen too many self-proclained “stars” destroy teammates to want to encourage this type of behavior.
Wayne, I don’t think Bug Week, for example, is an effort to promote stars within the organization, but rather an effort to pull the team together with the common goal of solving issues and making the software better. Both are objectives that I believe Deming would have approved of_��even though there might be a spiff or two attached to the effort.
I am not concerned about the concept of a “bug week”, this particular implementation states that its purpose is “Creating a competition that allows for your stars to get some recognition …” Consider that this approach is leaving the 3 week sprint work unrecognized, but then awarding points for 1 week of debugging work and determining the “top engineer.” It also does not seem to bode well for addressing any complex issues that might take more than one person. For a very close team, these things may do no harm, but if there are any emotional fault lines, I fear these things would exasperate them.
Thank you for the comments. I agree that the individual incentive is something you need to be careful about in order to maintain team morale. Lots of options exist for how to tweak a program like this while retaining the benefit. If bug sprints are something you are considering in your agile process, I go into more depth about what what led to our decision to incorporate Bug Week into our development process in my most recent post: http://blogs.attask.com/blog/work-management-3/0/0/are-bug-sprints-a-scrum-best-practice-or-a-scrum-anti-pattern
To me having 1 week of bug fixing is an anti-pattern. It is a mini-waterfall approach and it contradicts agile paradigm.
I like your article and it really gives an outstanding idea that is very helpful for all the people on web.