Transitioning to Agile development: What about quality assurance?

When facing an Agile transition, QA managers get worried. They don’t see a job description for a “QA manager” in the new Agile organization. The testers that worked for them are farmed out to cross-functional teams. Both the manager and the testers risk losing their identity and sense of value. Indeed, the development and product managers often don’t understand what testers contribute, nor do they value that contribution.

Too often, the resulting Scrum teams become silos, replacing the old development/testing /business analysis silos. They get focused on delivering code, and development degenerates quickly into mini-waterfalls where testing gets short shrift – just as it did in traditional waterfall!

Companies have been transitioning to Agile for more than ten years now. Why does this keep happening? What should and can we do about it?

The value of quality

Testers often end up reporting to development managers who see testers as “second class citizens.” Lack of attention to testers and testing activities is a symptom of the company undervaluing product quality. Many executives see “Agile” as a way to make teams “hyper-productive,” cranking new features out fast and furious. They don’t provide teams slack time to experiment with ways to improve their process or product.

When management doesn’t value quality, testers don’t stand a chance. As my colleague Janet Gregory notes, organizations who forget how to learn and grow will stagnate, and potential improvements that could come from moving to Agile development will be lost. Teams that don’t give testing the same attention as coding will pile up so much technical debt, they’ll slow down and possibly fail over the long term.

Let’s look at some reasons why companies shouldn’t eliminate the QA department without giving thought to how their development organization can maximize the value their testers can contribute.

Supporting testers

If your company is planning a transition to Agile development, get all the development and QA/Test managers together to decide what they will do in the new Agile organization. Having testers integrated into cross-functional teams which take a “whole-team approach” to quality, works best. Plan time and training to help the entire team take responsibility for quality and testing and to embrace the whole-team approach.

Some organizations, especially large ones, may still benefit from having a separate system test team that makes sure different parts of the product work well together. If your company produces embedded software, or has many large systems that must integrate, you might need a full-time team focused on testing the integration between systems or between software and hardware. They’ll still work closely with development teams, and in most cases, the development teams should still include professional testers.

Create a testing Community of Practice (CoP) to support everyone doing testing activities, across the organization. Practice leaders can help nurture a learning culture, making sure teams have time to share ideas and try new experiments to improve software quality. When I led a testing CoP in a sizeable development organization, I created a wiki space where different testers and teams could ask questions, have a dialog, share ideas, post links to blog posts and more information. We held lunchtime meetings twice a month during which someone demonstrated their automated tests, continuous integration, exploratory testing ideas, anything related to testing. Interestingly, the community attracted programmers, business analysts and product owners as well as testers. It helped break down the “Scrum team silos.”

The Testing Practice Manager can also make sure testers get the training they need. A transition to Agile development is hard, and on top of that, we have advances in testing and test automation every day. Though the whole team is responsible for these activities, testers need to be up to speed with the latest skills and technology in order to help ensure testing “keeps up” with coding on their teams.

Demonstrating the value of testers

It’s our responsibility to educate the rest of the company about how testers add value. And if we’re transitioning to Agile, we have the opportunity to make sure every team understands and adopts the “whole-team approach” to software development.

One skill that testers possess is exploratory testing. Experienced exploratory testers aim to learn more about desired and undesired behaviors of the system. They aren’t just looking for what doesn’t work, they’re alert for missing features, and they’re applying critical thinking to many aspects of quality, not only functionality. Educate your development organization on the benefits of exploratory testing. One good way to do this is to have programmers pair with testers to try out some exploratory testing.

Another area where testers excel is domain knowledge. Programmers have to maintain a narrow focus, so they can complete specific coding tasks. Testers maintain a “big picture” perspective. Given a particular user story, testers think about the business problem being solved, the potential technical implementation, and whether that implementation will really solve the problem. Teams won’t succeed in the long run without good domain knowledge. Try budgeting some time for the team to learn more about some area of the domain. The testers will get a chance to lead the way.

Raising consciousness about quality

Lack of attention to quality and undervaluing testing isn’t unique to Agile companies, of course. We’ve battled these problems for decades. If you’re willing to take on the role of change agent in your organization, I highly recommend trying the patterns in Fearless Change (Manns and Rising, 2003).

Back in the DotCom boom, I was the first tester hired by a fast-growing Internet startup. Nobody in the company really thought about quality. I decided to try to educate everyone in the company about why paying attention to quality would help the company. One way I did this was to get the company to buy pizza, and I gave a lunchtime introduction to quality and testing. People from both the engineering and business sides of the company showed up, and showed an interest. Though I didn’t know it then, this is the Brown Bag pattern in Fearless Change: “Use the time when people normally eat lunch to provide a convenient and relaxed setting for hearing about the new idea.”

I also simply tried to earn my credibility. I worked closely with the product managers and the developers. When I found issues, we discussed them together. I knew my efforts were paying off when the developers asked me to come in and help them one weekend when they wanted to put in extra time on a project.

It’s tough to change a culture, and if you can’t identify the right pain points, you might not be able to instigate much passion about quality. Still, it’s usually worth a try. Find some creative ways to demonstrate the contributions of testers to your company.

Take time to experiment

Approach your Agile transition as a series of small experiments. Try integrating all the testers into cross-functional teams, while maintaining a testing Community of Practice. Use retrospectives to see how that works. Are teams allowed to manage their workload so that they can finish all testing activities for all user stories during each iteration? Are different teams checking in code that breaks functionality used in code other teams are developing? Is everyone struggling with load or security testing? Perhaps more training is in order, or maybe you need to try setting up some specialist teams to take on activities such as performance testing or system integration testing.

QA managers are needed to help these testers get the time, training and support they need. Make sure the testing community is represented in discussions of how to maintain and improve software quality. We don’t get high-quality software without a team that is diverse in skills and viewpoint. Make sure each role in your cross-functional teams is represented and supported at a high level. Use the Agile Testing Quadrants as a tool to discuss all the different testing activities needed, and make sure the right people are available to ensure they get done.

Agile Testing Quadrants: Most experts agree that both automated and manual testing are important, but depending on the type of testing you’re doing, one type may be the better choice. In Agile Testing, written by SSQ’s Agile expert Lisa Crispin along with co-author Janet Gregory, four quadrants of testing are described.

Quadrant one encompasses the automated test efforts including unit tests and component tests. These are typically done by the developer, perhaps using test-driven development (TDD) even before the code is written.

Quadrant two is testing that may be automated or may be manually done by developers and testers. This includes functional tests, examples, story tests, prototypes and simulations.

Quadrant three contains manual tests including exploratory testing, usability testing, user acceptance testing and alpha/beta testing.

Quadrant four includes speciality testing that is typically done with tools such as performance testing and load testing.

Above all, realize that successful Agile transitions don’t happen overnight. It takes years for the new teams to jell, for communication and collaboration among different roles and specialties to develop. It takes practice to keep teams and roles from becoming silos.  Aim for baby steps, and celebrate every small success. Even if the old “QA Team” is history, you can still grow a vibrant testing community, which can help teams produce a high-quality product.

Agile problem areas that pain developers and testers

The long laundry list of software development pros’ problems with the Agile methodology includes inadequate training, poor leadership, rigid adherence to Agile principles that don’t fit the project, and more. That said, there are Agile problem areas that are slammed more often than these, including required meetings, inadequate documentation and issues related to short iterations.
Need for speed crimps Agile documentation
Some people think that Agile calls for doing hardly any documentation. They complain that they can’t give the kinds of reports they used to give, calling Agile reports “Cragile” -- or crappy Agile – or Scrumbutt.
 “That’s the phoniest belief about Agile I’ve ever heard,” said BigVisible Solutions Agile coach Mike Dwyer. Agile entails doing documentation that has value, he said, not “your earned-value estimated nonsense or spending six weeks writing a proposal to prove that you have done something, when all that you’ve done for the past six weeks is write a presentation.”
The problem for developers, however, is that there’s little time in Agile’s short-iteration scheme to write enough documentation, development pros told us. Iteration cycles are every three weeks, “boom, boom, boom,” and it’s hard to fit doing documentation into that cycle, said Huckleberry Carignan, lead QA engineer at Vistaprint, a print services provider.
Veteran software tester Chris York agreed, saying he has never seen managers call for adequate documentation in Agile projects. They ask for concise, but not complete documentation. That means testers are left  with short pieces of stories they have to try to put together, “It’s nearly impossible,” said York. “We always got concise, but we never got complete, documentation. There’s nothing that really tells you how things work.”
Developers grump about Agile meetings
Meetings are a cornerstone of the Agile methodologies and a thorn in the side of some developers and testers. Known as daily stand-ups, daily Scrums and other names, Agile meetings often draw the most heated criticism of all Agile practices.
Kern hears more complaints about Agile meetings being done poorly than anything else. “Usually, when you dig deeper, it’s because the meetings are like dumb status meetings; just like the same old same olds,” he said.
Some tools and/or practices used in Agile meetings seem demeaning and infantile to some participants. Complaints center on having to stand; huddles; the use of talking sticks or cards; and dollar penalties for infractions like sitting, drinking coffee, speaking out of turn, etc.
York recalled of being in one Agile manager’s meetings. “You held a Kush ball when you spoke. The manager wanted us to throw the ball to each other. Somebody threw the Kush ball and hit the new projector. Now suddenly the Kush ball was no longer being thrown. It felt like grade school.”
Distributed development teams are another challenge related to Agile’s focus on meetings. Making Agile meetings and Agile’s reliance on constant communication work is difficult for global development teams, said Carignan.
“Having the communication smoothly enough in a global environment is hard, especially with time zone differences,” Carignan explained. “If we’re in Lexington, Mass., and we’re trying to work together with people in Venlo, Netherlands and Switzerland and Australia, it is pretty difficult to get that communication smoothly to work in Agile method.”

On the other hand, veteran software tester Chris McMahon has worked in distributed Agile teams that succeeded in communicating well. Those teams mixed written and voice communication, using multi-channel real-time chat applications, voice-over-IP, wikis, IM and remote workflow tools like defect trackers.
Kern has been in many good meetings, ones that went beyond being simple status reports to identifying obstacles and ways to remove them. Those meeting leaders and participants succeeded in getting people on the same page and making plans; gaining consensus on each person’s role in the plan; creating contingency plans and so on.
Estimates for iterations not based on reality
Requirement goals or stories must be realistic and accurate, or required features don’t get produced in an iteration, said Scott Barber. The result is usually recriminations, project slowdowns and technical debt.
Typically, iterations are two-to-three weeks long. That quick delivery of features is very attractive to businesses and is seen as a main reason to use Agile instead of waterfall and other development methodologies. Focus too intently on that one aspect of Agile and forego others at your peril.
York has seem missed estimates kill projects. “You have to be dead on with your estimates otherwise the whole thing is screwed up,” he said. “There’s not enough time in the next iteration or the next to catch up when a goal is missed. He’s seen estimates being way off the mark in three Agile projects. In those cases, York said, at the very end, the project manager cut out what didn’t get done and then claimed it all got done. The result was debilitating technical debt.
If you don’t get everything done in one iteration, you have to change your plan and go back and fix things before technical debt builds up, wrote software test veteran David Christiansen in his article, How to deal with iteration issues in Agile.
An even better approach is not planning  too far in advance, said Crispin. Her Agile team avoids missed estimates by estimating only a sprint or two in advance. This gives them the ability to switch priorities very fast, to be very agile.
Agile’s short iterations lead to burn-out and mistakes
Business managers love short iterations, but they wear people out, developers told us.
Management is getting away with short iterations today because of the recession, said York. “People are going to burn out and start leaving. And as soon as the economy picks up, people are going to start switching jobs because they’re tired of being pushed to the limit.”
Dwyer sees testers feeling the pinch of short iterations the most, largely because they’re not allowed into a process earlier. Carignan agreed, noting that the expectation is often for test and QA teams to get their work done in the last eight days of a three-week iteration, a time when testing and production is happening almost simultaneously.

Agile development: What’s behind the backlash against Agile?

Try Googling “hate Agile” and “Agile sucks” and “Agile fails,” and hundreds of pages will show up. Some contain articles on how to avoid Agile software development failures and problems, and others are by development pros who have tried Agile and just don’t like it. Read the mostly anonymous comments on these articles, and you’ll find the hate being put on Agile right and left.
That’s the Agile backlash, another instance of any action causing an opposite reaction. Not everyone is on board the Agile train, despite the wave of almost-fanatical evangelism for and quick adoption of it. What’s written on the Web is the tip of the iceberg of dissatisfaction with the process, according to some of the users and experts cited in this series on the so-called backlash. Others think it’s the by-product of rapid and poorly-planned adoption.
In this article, we’ll explore the reasons why some software pros hate Agile, based on interviews with Agile coaches and consultants -- including Mike Dwyer, David Whalen, and Jon Kern – as well as software test and development veterans Scott Barber, Chris York and Lisa Crispin.
Too much Agile too soon?
Picture this: A CEO/CTO/CIO/CXO sitting in an airplane reading a magazine article that’s gung-ho about Agile. He or she goes back to the office and issues an order to adopt Agile development and, perhaps, to use the Agile process in application lifecycle management (ALM) and all business-side processes. Managers rush to obey.
This opening scene in the Agile backlash story was suggested without solicitation by almost all of this article’s sources. The chief executive’s decision sets the stage for mistakes to be made and has led to project failures and disgruntled development, systems architecture and project management pros. Frequently, the blame is place on Agile..and, of course, the media.
“Nobody has been writing about the problems with Agile,” said Chris York, a 15-year software test and development pro. “Every manager reads the same publications, and all the articles say Agile is good. So, the managers want to do it, even though a lot of them have no idea what it really is.”
Enthusiasm for Agile principles and real need, fueled by the economic downturn, has led to rushed adoption and project failures, according to software tester Scott Barber. “Folks got caught up in the idea or the ideal around Agile,” he said. “They decided, without knowing very much about it, that it was going to solve whatever problems they had. Barber is president of PerfTestPlus, a software testing services firm, and co-author of both Performance Testing Guidance for Web Application. and Beautiful Testing.
On various development projects, York has seen first-hand results of too much Agile too quickly. “No one at the top really knew in any depth what Agile is,” he said.  Barber adds that non-development executives often don’t understand all the implications of a methodology change. They require the switch, but say: “Do whatever you want, just don’t make things hard on us.”
Many of the pro Agile articles have been written by early adopters who have had success in Agile or Agile consultants promoting it as the answer to all problems, Barber said. “No one thing can be the answer to all problems.”
Too little analysis is being done of Agile’s appropriateness to the projects’ or organizations’ people, culture, management structure, sales cycle, business needs, development processes and tools and other key aspects, sources said. “Failures and problems with Agile can simply be because that it’s the wrong decision for that organization,” said Barber. “Is that because Agile is broken? Not fundamentally.”
Too strict with  Agile principles?
Leaders and organizations who insist that teams adhere too strictly to perceived Agile principles can create problems.
York has worked on Agile projects for more than one strictly Agile company. “Everything’s about the process and it’s not about the process being helpful. It’s about fitting into the process. That’s just backwards,” he said.
Development consultant Dave Whalen saw this same thing happen so often that he wrote the widely-read and controversial article, I Hate Agile.
Knowing the principles of Agile isn’t the whole ballgame, Whalen told SearchSoftwareQuality.com. Understanding the philosophy, which is still developing, is a must. Anyone who believes that all Agile practices must be done as written is missing the point. “Every environment is not best served by a completely Agile notion,” he said. “Probably only a few are.”
Mike Dwyer, a 30-year IT and development veteran, sees “a conservatism in some of the Scrum and Agile community that you have to do it a certain way, and if you’re not doing it that way, you’re not Agile.” Dwyer is principal Agile coach for BigVisible Solutions in Boston and a significant contributor to the Scrum, Agile, and Lean software community.
The Agile philosophy is about flexibility and using what works for particular projects, not implementing the exact same practices even when the project isn’t a fit for them, Whalen said.
Yearning for the silver bullet
Done well, Agile can help organizations streamline development, cut costs, improve software quality and more; but it’s not a shortcut or a universal panacea to all that ails Agile.
“People want the Agile Manifesto and the 12 principles to be more than just a bunch of principles,” Dwyer said. “They don’t understand in the simplicity of Agile. There’s a need for discipline, and they’re looking for something to take the place of discipline.”
Reading about Agile for the first time could be a “Eureka!” moment, but making Agile work well calls for leaders to do more than make top-down ultimatums for Agile adoption, these experts and developers said. Without discipline every step of the way, Agile can be yoke on the development teams’ neck and fuel the rebellion against it.

Nine Things Successful People Do Differently

8:58 AM Friday February 25, 2011
by Heidi Grant Halvorson | 

 

Why have you been so successful in reaching some of your goals, but not others? If you aren't sure, you are far from alone in your confusion. It turns out that even brilliant, highly accomplished people are pretty lousy when it comes to understanding why they succeed or fail. The intuitive answer — that you are born predisposed to certain talents and lacking in others — is really just one small piece of the puzzle. In fact, decades of research on achievement suggests that successful people reach their goals not simply because of who they are, but more often because of what they do.
1. Get specific. When you set yourself a goal, try to be as specific as possible. "Lose 5 pounds" is a better goal than "lose some weight," because it gives you a clear idea of what success looks like. Knowing exactly what you want to achieve keeps you motivated until you get there. Also, think about the specific actions that need to be taken to reach your goal. Just promising you'll "eat less" or "sleep more" is too vague — be clear and precise. "I'll be in bed by 10pm on weeknights" leaves no room for doubt about what you need to do, and whether or not you've actually done it.

2. Seize the moment to act on your goals. Given how busy most of us are, and how many goals we are juggling at once, it's not surprising that we routinely miss opportunities to act on a goal because we simply fail to notice them. Did you really have no time to work out today? No chance at any point to return that phone call? Achieving your goal means grabbing hold of these opportunities before they slip through your fingers.
To seize the moment, decide when and where you will take each action you want to take, in advance. Again, be as specific as possible (e.g., "If it's Monday, Wednesday, or Friday, I'll work out for 30 minutes before work.") Studies show that this kind of planning will help your brain to detect and seize the opportunity when it arises, increasing your chances of success by roughly 300%.
3. Know exactly how far you have left to go. Achieving any goal also requires honest and regular monitoring of your progress — if not by others, then by you yourself. If you don't know how well you are doing, you can't adjust your behavior or your strategies accordingly. Check your progress frequently — weekly, or even daily, depending on the goal.

4. Be a realistic optimist. When you are setting a goal, by all means engage in lots of positive thinking about how likely you are to achieve it. Believing in your ability to succeed is enormously helpful for creating and sustaining your motivation. But whatever you do, don't underestimate how difficult it will be to reach your goal. Most goals worth achieving require time, planning, effort, and persistence. Studies show that thinking things will come to you easily and effortlessly leaves you ill-prepared for the journey ahead, and significantly increases the odds of failure.

5. Focus on getting better, rather than being good. Believing you have the ability to reach your goals is important, but so is believing you can get the ability. Many of us believe that our intelligence, our personality, and our physical aptitudes are fixed — that no matter what we do, we won't improve. As a result, we focus on goals that are all about proving ourselves, rather than developing and acquiring new skills.
Fortunately, decades of research suggest that the belief in fixed ability is completely wrong — abilities of all kinds are profoundly malleable. Embracing the fact that you can change will allow you to make better choices, and reach your fullest potential. People whose goals are about getting better, rather than being good, take difficulty in stride, and appreciate the journey as much as the destination.

6. Have grit. Grit is a willingness to commit to long-term goals, and to persist in the face of difficulty. Studies show that gritty people obtain more education in their lifetime, and earn higher college GPAs. Grit predicts which cadets will stick out their first grueling year at West Point. In fact, grit even predicts which round contestants will make it to at the Scripps National Spelling Bee.
The good news is, if you aren't particularly gritty now, there is something you can do about it. People who lack grit more often than not believe that they just don't have the innate abilities successful people have. If that describes your own thinking .... well, there's no way to put this nicely: you are wrong. As I mentioned earlier, effort, planning, persistence, and good strategies are what it really takes to succeed. Embracing this knowledge will not only help you see yourself and your goals more accurately, but also do wonders for your grit.
7. Build your willpower muscle. Your self-control "muscle" is just like the other muscles in your body — when it doesn't get much exercise, it becomes weaker over time. But when you give it regular workouts by putting it to good use, it will grow stronger and stronger, and better able to help you successfully reach your goals.
To build willpower, take on a challenge that requires you to do something you'd honestly rather not do. Give up high-fat snacks, do 100 sit-ups a day, stand up straight when you catch yourself slouching, try to learn a new skill. When you find yourself wanting to give in, give up, or just not bother — don't. Start with just one activity, and make a plan for how you will deal with troubles when they occur ("If I have a craving for a snack, I will eat one piece of fresh or three pieces of dried fruit.") It will be hard in the beginning, but it will get easier, and that's the whole point. As your strength grows, you can take on more challenges and step-up your self-control workout.
8. Don't tempt fate. No matter how strong your willpower muscle becomes, it's important to always respect the fact that it is limited, and if you overtax it you will temporarily run out of steam. Don't try to take on two challenging tasks at once, if you can help it (like quitting smoking and dieting at the same time). And don't put yourself in harm's way — many people are overly-confident in their ability to resist temptation, and as a result they put themselves in situations where temptations abound. Successful people know not to make reaching a goal harder than it already is.

9. Focus on what you will do, not what you won't do. Do you want to successfully lose weight, quit smoking, or put a lid on your bad temper? Then plan how you will replace bad habits with good ones, rather than focusing only on the bad habits themselves. Research on thought suppression (e.g., "Don't think about white bears!") has shown that trying to avoid a thought makes it even more active in your mind. The same holds true when it comes to behavior — by trying not to engage in a bad habit, our habits get strengthened rather than broken.
If you want change your ways, ask yourself, What will I do instead? For example, if you are trying to gain control of your temper and stop flying off the handle, you might make a plan like "If I am starting to feel angry, then I will take three deep breaths to calm down." By using deep breathing as a replacement for giving in to your anger, your bad habit will get worn away over time until it disappears completely.
It is my hope that, after reading about the nine things successful people do differently, you have gained some insight into all the things you have been doing right all along. Even more important, I hope are able to identify the mistakes that have derailed you, and use that knowledge to your advantage from now on. Remember, you don't need to become a different person to become a more successful one. It's never what you are, but what you do.
Heidi Grant Halvorson, Ph.D. is a motivational psychologist, and author of the new book Succeed: How We Can Reach Our Goals (Hudson Street Press, 2011). She is also an expert blogger on motivation and leadership for Fast Company and Psychology Today. Her personal blog, The Science of Success, can be found at http://heidigranthalvorson.com/. Follow her on Twitter @hghalvorson

The secrets to getting started in your software testing career

The secrets to getting started in your software testing career

Reasons to Repeat Tests

Reasons to Repeat Tests

by James Bach
(with help from colleagues Doug Hoffman, Michael Bolton, Ken Pugh, Cem Kaner, Bret Pettichord, Jim Batterson, Geoff Sutton, plus numerous students who have participated in the "Minefield Debate" as part of my testing class. The minefield analogy as I talk about it was inspired by Brian Marick's talk Classic Testing Mistakes.)

Testing to find bugs is like searching a minefield for mines. If you just travel the same path through the field again and again, you won't find a lot of mines. Actually, that's a great way to avoid mines. The space represented by a modern software product is hugely more complex than a minefield, so it's even more of a problem to assume that some small number of "paths", say, a hundred, thousand, or million, when endlessly repeated, will find every important bug. As many tests as a team of testers can physically perform in a few weeks or months is still not that many tests compared to all the things that can happen to a product in the field.

The minefield analogy is really just another way of saying that testing is a sampling process, and we probably want a larger sample, rather than a tiny sample repeated over and over again. Hence the minefield heuristic is do different tests instead of repeating the same tests.

But what do I mean by repeat the same test? It's easy to see that no test can be repeated exactly, any more than you can exactly retrace your footsteps. You can get close, but you will always be a tiny bit off. Does repeating a test mean that the second time you run the test you have to make sure that sunlight is shining at the same angle onto your mousepad? Maybe. Don't laugh. I did experience a bug, once, that was triggered by sunlight hitting an optical sensor inside a mouse. You just can't say for sure what factors are going to affect a test. However, when you test you have a certain goal and a certain theory of the system. You may very well be able to repeat a test with respect to that goal and theory in every respect that A) you know about and B) you care about and C) isn't too expensive to repeat. Nothing is necessarily intractable about that.

Therefore, by a repeated test, I mean a test that includes elements already known to be covered in other tests. To repeat a test is to repeat some aspect of a previous test. The minefield heuristic is saying that it's better to try to do something you haven't yet done, then to do something you already have done.

If you disagree with this idea, or if you agree with it, please read further. Because...

...this analysis is too simplistic! In fact, even though diversity in testing is important and powerful, and even though the argument against repetition is generally valid, I do know of ten exceptions. There are ten specific reasons why, in some particular situation, it is not unreasonable to repeat tests. It may even be important to repeat some tests.

For technical reasons you might rationally repeat tests...

1. Recharge: if there is a substantial probability of a new problem or a recurring old problem that would be caught by a particular existing test, or if an old test is applied to a new code base. This includes re-running a test to verify a fix, or repeating a test on successively earlier builds as you try to discover when a particular problem or behavior was introduced. This also includes running an old test on the same software that is running on a new O/S. In other words, a tired old test can be "recharged" by changes to the technology under test. Note that the recharge effect doesn't necessarily mean you should run the same old tests, only that it isn't necessarily irrational to do so.
2. Intermittence: if you suspect that the discovery of a bug is not guaranteed by one correct run of a test, perhaps due to important variables involved that you can't control in your tests. Performing a test that is, to you, exactly the same as a test you've performed before, may result in discovery of a bug that was always there but not revealed until the uncontrolled variables line up in a certain way. This is the same reason that a gambler at a slot machine plays again after losing the first time.
3. Retry: if you aren't sure that the test was run correctly the other time(s) it was performed. A variant of this is having several testers follow the same instructions and check to see that they all get the same result.
4. Mutation: if you are changing an important part of the test while keeping another part constant. Even though you are repeating some elements of the test, the test as a whole is new, and may reveal new behavior. I mutate a test because although I have covered something before, I haven't yet covered it well enough. A common form of mutation is to operate the product the same way while using different data. The key difference between mutating a test and intermittence or retry is that with mutation the change is directly under your control. Mutation is intentional, intermittence results from incidental factors, and you retry a test because of accidental factors.
5. Benchmark: if the repeated tests comprise a performance standard that gets its value by comparison with previous executions of the same exact tests. When historical test data is used as an oracle, then you must take care that the tests you perform are comparable to the historical data. Holding tests constant may not be the only way to make results comparable, but it might be the best choice available.

For business reasons you might rationally repeat tests...

6. Inexpensive: if they have some value and are sufficiently inexpensive compared to the cost of new and different tests. These tests may not be enough to justify confidence in the product, however.
7. Importance: if a problem that could be discovered by those tests is likely to have substantially more importance than problems detectable by other tests. The distribution of the importance of product behavior is not necessarily uniform. Sometimes a particular problem may be considered intolerable just because it's already impacted an important user once (a "never let it happen again" situation). This doesn't necessarily mean that you must run the same exact test, just something that is sufficiently similar to catch the problem (see Mutation). Be careful not to confuse the importance of a problem with the importance of a test. A test might be important for many reasons, even if the problems it detects are not critical ones. Also, don't make the mistake of spending so much effort on one test that looks for an important bug that you neglect other tests that might be just as good or better at finding that kind of problem.
8. Enough: if the tests you repeat represent the only tests that seem worth doing. This is the virus scanner argument: maybe a repeated virus scan is okay for an ordinary user, instead of constantly changing virus tests. However, we may introduce variation because we don't know which tests truly are worth doing, or we are unable to achieve enoughness via repeated tests.
9. Mandated: if, due to contract, management edict, or regulation, you are forced to run the same exact tests. However, even in these situations, it is often not necessary that the mandated tests be the only tests you perform. You may be able to run new tests without violating the mandate.
10. Indifference/Avoidance: if the "tests" are being run for some reason other than finding bugs, such as for training purposes, demo purposes (such as an acceptance test that you desperately hope will pass when the customer is watching), or to put the system into a certain state. If one of your goals in running a test is to avoid bugs, then the principal argument for variation disappears.

I have collected these reasons in the course of probably a hundred hours of debate with testing students and colleagues. Many of my colleagues prefer different words or a different breakdown of reasons. There's nothing particularly sacred about my way of doing it (except that some breakdowns would lead to long lists of very similar items). The important thing is that when I hear a reason that seems not to fit within the ones I already have, I add that reason to this list. I started with two reasons, in 1997. I added the tenth one in late 2004.

Applying the Minefield: An Example

Ward Cunningham wrote "I believe the automation required of TDD [Test Driven Design] (and Fit) is exempt from the analogy because the searching we are doing is for the best expression of a program in the presence of tests, not the best tests."

Here's how I think it applies:

Your units tests might pass or they might fail. You write them so that they will fail in the event that some interesting expectation is violated. So, you call them tests and they seem to be tests.

We introduce the minefield criticism the first time you run any given test in your unit test suite. The first time you run it, it fails, right? Of course, since it wouldn't be TDD, otherwise. The questions below are inspired by the Minefield heuristic "vary your tests instead of repeating them."

Question: Why run it again?

Answer: Exception #1, "recharge." You run it again because you have added code to make the test pass, therefore running the test again is not merely redundant, the value of the test has been recharged by the code changing around it.

Question: During the course of development, but after the first time the test passes, why not delete it? Why bother to run it again?

Answer: Several reasons. Recharge still applies a little bit, since you may accidentally break the product during development, but it could be argued that most of those unit tests most of the time don't fail, and some of them are extremely unlikely to fail even if you change the code quite a bit. But here you have the second reason: exception #6, "inexpensive." It's so cheap to create these tests and to run them and to keep them running, while at the same time they do have some value, even if not a lot. And you have a third reason for some of the tests: exception #7, "importance." For a good many of the unit tests, failure would indicate a very serious problem, were it to occur. If you are testing something that is particularly complex, or involves many interacting sub-systems, you may also want to epeat because of exception #2, "intermittence". Perhaps something will fail after the forty-third run because of probabilistic factors in he test. Finally, there's #3, the "retry" exception, which reminds us that we might not have run the test correctly, before. As you once said, Ward, something might give off a bad smell only after you've seen the test run a hundred times or so. In other words, as a result of running a test many times, you might come to an insight about the product that reveals a failure that was there all along, but never noticed.

Question: Let's say that I'm a really good developer and though I write good tests, they just don't fail because I just don't put bugs into my code. I have a whole lot of tests and they don't fail. What was the sense in investing in such tests?

Answer: Two potential reasons. Exception #10, "avoidance/indifference." You may create the tests as a form of documentation for future developers and you like them to be exactly the same in order to minimize the chance that they will fail (and thus be less useful as documentation). Or maybe you want to impress a customer with your great software, and they won't be as impressed if the tests don't pass. A second reason is exception #9, "mandated." you may work this way because your peer group or your manager requires you to. This is a little like avoidance except that with a mandate you do, in fact, want to find bugs. You are searching for them, you just are required to use a certain technique to do so.

We therefore see that the fairly simple, often repeated unit tests of TDD may indeed be exempt from the minefield-based argument in favor of varying tests, inasmuch as the reasons I cited apply. But TDD is not exempt from this kind of heuristic analysis. It is always reasonable to question the value of repeated tests, and that's what the minefield invites us to do.

Estimation approaches in Agile development

Estimation approaches in Agile development