Transitioning to Agile development: What about quality assurance?

When facing an Agile transition, QA managers get worried. They don’t see a job description for a “QA manager” in the new Agile organization. The testers that worked for them are farmed out to cross-functional teams. Both the manager and the testers risk losing their identity and sense of value. Indeed, the development and product managers often don’t understand what testers contribute, nor do they value that contribution.

Too often, the resulting Scrum teams become silos, replacing the old development/testing /business analysis silos. They get focused on delivering code, and development degenerates quickly into mini-waterfalls where testing gets short shrift – just as it did in traditional waterfall!

Companies have been transitioning to Agile for more than ten years now. Why does this keep happening? What should and can we do about it?

The value of quality

Testers often end up reporting to development managers who see testers as “second class citizens.” Lack of attention to testers and testing activities is a symptom of the company undervaluing product quality. Many executives see “Agile” as a way to make teams “hyper-productive,” cranking new features out fast and furious. They don’t provide teams slack time to experiment with ways to improve their process or product.

When management doesn’t value quality, testers don’t stand a chance. As my colleague Janet Gregory notes, organizations who forget how to learn and grow will stagnate, and potential improvements that could come from moving to Agile development will be lost. Teams that don’t give testing the same attention as coding will pile up so much technical debt, they’ll slow down and possibly fail over the long term.

Let’s look at some reasons why companies shouldn’t eliminate the QA department without giving thought to how their development organization can maximize the value their testers can contribute.

Supporting testers

If your company is planning a transition to Agile development, get all the development and QA/Test managers together to decide what they will do in the new Agile organization. Having testers integrated into cross-functional teams which take a “whole-team approach” to quality, works best. Plan time and training to help the entire team take responsibility for quality and testing and to embrace the whole-team approach.

Some organizations, especially large ones, may still benefit from having a separate system test team that makes sure different parts of the product work well together. If your company produces embedded software, or has many large systems that must integrate, you might need a full-time team focused on testing the integration between systems or between software and hardware. They’ll still work closely with development teams, and in most cases, the development teams should still include professional testers.

Create a testing Community of Practice (CoP) to support everyone doing testing activities, across the organization. Practice leaders can help nurture a learning culture, making sure teams have time to share ideas and try new experiments to improve software quality. When I led a testing CoP in a sizeable development organization, I created a wiki space where different testers and teams could ask questions, have a dialog, share ideas, post links to blog posts and more information. We held lunchtime meetings twice a month during which someone demonstrated their automated tests, continuous integration, exploratory testing ideas, anything related to testing. Interestingly, the community attracted programmers, business analysts and product owners as well as testers. It helped break down the “Scrum team silos.”

The Testing Practice Manager can also make sure testers get the training they need. A transition to Agile development is hard, and on top of that, we have advances in testing and test automation every day. Though the whole team is responsible for these activities, testers need to be up to speed with the latest skills and technology in order to help ensure testing “keeps up” with coding on their teams.

Demonstrating the value of testers

It’s our responsibility to educate the rest of the company about how testers add value. And if we’re transitioning to Agile, we have the opportunity to make sure every team understands and adopts the “whole-team approach” to software development.

One skill that testers possess is exploratory testing. Experienced exploratory testers aim to learn more about desired and undesired behaviors of the system. They aren’t just looking for what doesn’t work, they’re alert for missing features, and they’re applying critical thinking to many aspects of quality, not only functionality. Educate your development organization on the benefits of exploratory testing. One good way to do this is to have programmers pair with testers to try out some exploratory testing.

Another area where testers excel is domain knowledge. Programmers have to maintain a narrow focus, so they can complete specific coding tasks. Testers maintain a “big picture” perspective. Given a particular user story, testers think about the business problem being solved, the potential technical implementation, and whether that implementation will really solve the problem. Teams won’t succeed in the long run without good domain knowledge. Try budgeting some time for the team to learn more about some area of the domain. The testers will get a chance to lead the way.

Raising consciousness about quality

Lack of attention to quality and undervaluing testing isn’t unique to Agile companies, of course. We’ve battled these problems for decades. If you’re willing to take on the role of change agent in your organization, I highly recommend trying the patterns in Fearless Change (Manns and Rising, 2003).

Back in the DotCom boom, I was the first tester hired by a fast-growing Internet startup. Nobody in the company really thought about quality. I decided to try to educate everyone in the company about why paying attention to quality would help the company. One way I did this was to get the company to buy pizza, and I gave a lunchtime introduction to quality and testing. People from both the engineering and business sides of the company showed up, and showed an interest. Though I didn’t know it then, this is the Brown Bag pattern in Fearless Change: “Use the time when people normally eat lunch to provide a convenient and relaxed setting for hearing about the new idea.”

I also simply tried to earn my credibility. I worked closely with the product managers and the developers. When I found issues, we discussed them together. I knew my efforts were paying off when the developers asked me to come in and help them one weekend when they wanted to put in extra time on a project.

It’s tough to change a culture, and if you can’t identify the right pain points, you might not be able to instigate much passion about quality. Still, it’s usually worth a try. Find some creative ways to demonstrate the contributions of testers to your company.

Take time to experiment

Approach your Agile transition as a series of small experiments. Try integrating all the testers into cross-functional teams, while maintaining a testing Community of Practice. Use retrospectives to see how that works. Are teams allowed to manage their workload so that they can finish all testing activities for all user stories during each iteration? Are different teams checking in code that breaks functionality used in code other teams are developing? Is everyone struggling with load or security testing? Perhaps more training is in order, or maybe you need to try setting up some specialist teams to take on activities such as performance testing or system integration testing.

QA managers are needed to help these testers get the time, training and support they need. Make sure the testing community is represented in discussions of how to maintain and improve software quality. We don’t get high-quality software without a team that is diverse in skills and viewpoint. Make sure each role in your cross-functional teams is represented and supported at a high level. Use the Agile Testing Quadrants as a tool to discuss all the different testing activities needed, and make sure the right people are available to ensure they get done.

Agile Testing Quadrants: Most experts agree that both automated and manual testing are important, but depending on the type of testing you’re doing, one type may be the better choice. In Agile Testing, written by SSQ’s Agile expert Lisa Crispin along with co-author Janet Gregory, four quadrants of testing are described.

Quadrant one encompasses the automated test efforts including unit tests and component tests. These are typically done by the developer, perhaps using test-driven development (TDD) even before the code is written.

Quadrant two is testing that may be automated or may be manually done by developers and testers. This includes functional tests, examples, story tests, prototypes and simulations.

Quadrant three contains manual tests including exploratory testing, usability testing, user acceptance testing and alpha/beta testing.

Quadrant four includes speciality testing that is typically done with tools such as performance testing and load testing.

Above all, realize that successful Agile transitions don’t happen overnight. It takes years for the new teams to jell, for communication and collaboration among different roles and specialties to develop. It takes practice to keep teams and roles from becoming silos.  Aim for baby steps, and celebrate every small success. Even if the old “QA Team” is history, you can still grow a vibrant testing community, which can help teams produce a high-quality product.

Agile problem areas that pain developers and testers

The long laundry list of software development pros’ problems with the Agile methodology includes inadequate training, poor leadership, rigid adherence to Agile principles that don’t fit the project, and more. That said, there are Agile problem areas that are slammed more often than these, including required meetings, inadequate documentation and issues related to short iterations.
Need for speed crimps Agile documentation
Some people think that Agile calls for doing hardly any documentation. They complain that they can’t give the kinds of reports they used to give, calling Agile reports “Cragile” -- or crappy Agile – or Scrumbutt.
 “That’s the phoniest belief about Agile I’ve ever heard,” said BigVisible Solutions Agile coach Mike Dwyer. Agile entails doing documentation that has value, he said, not “your earned-value estimated nonsense or spending six weeks writing a proposal to prove that you have done something, when all that you’ve done for the past six weeks is write a presentation.”
The problem for developers, however, is that there’s little time in Agile’s short-iteration scheme to write enough documentation, development pros told us. Iteration cycles are every three weeks, “boom, boom, boom,” and it’s hard to fit doing documentation into that cycle, said Huckleberry Carignan, lead QA engineer at Vistaprint, a print services provider.
Veteran software tester Chris York agreed, saying he has never seen managers call for adequate documentation in Agile projects. They ask for concise, but not complete documentation. That means testers are left  with short pieces of stories they have to try to put together, “It’s nearly impossible,” said York. “We always got concise, but we never got complete, documentation. There’s nothing that really tells you how things work.”
Developers grump about Agile meetings
Meetings are a cornerstone of the Agile methodologies and a thorn in the side of some developers and testers. Known as daily stand-ups, daily Scrums and other names, Agile meetings often draw the most heated criticism of all Agile practices.
Kern hears more complaints about Agile meetings being done poorly than anything else. “Usually, when you dig deeper, it’s because the meetings are like dumb status meetings; just like the same old same olds,” he said.
Some tools and/or practices used in Agile meetings seem demeaning and infantile to some participants. Complaints center on having to stand; huddles; the use of talking sticks or cards; and dollar penalties for infractions like sitting, drinking coffee, speaking out of turn, etc.
York recalled of being in one Agile manager’s meetings. “You held a Kush ball when you spoke. The manager wanted us to throw the ball to each other. Somebody threw the Kush ball and hit the new projector. Now suddenly the Kush ball was no longer being thrown. It felt like grade school.”
Distributed development teams are another challenge related to Agile’s focus on meetings. Making Agile meetings and Agile’s reliance on constant communication work is difficult for global development teams, said Carignan.
“Having the communication smoothly enough in a global environment is hard, especially with time zone differences,” Carignan explained. “If we’re in Lexington, Mass., and we’re trying to work together with people in Venlo, Netherlands and Switzerland and Australia, it is pretty difficult to get that communication smoothly to work in Agile method.”

On the other hand, veteran software tester Chris McMahon has worked in distributed Agile teams that succeeded in communicating well. Those teams mixed written and voice communication, using multi-channel real-time chat applications, voice-over-IP, wikis, IM and remote workflow tools like defect trackers.
Kern has been in many good meetings, ones that went beyond being simple status reports to identifying obstacles and ways to remove them. Those meeting leaders and participants succeeded in getting people on the same page and making plans; gaining consensus on each person’s role in the plan; creating contingency plans and so on.
Estimates for iterations not based on reality
Requirement goals or stories must be realistic and accurate, or required features don’t get produced in an iteration, said Scott Barber. The result is usually recriminations, project slowdowns and technical debt.
Typically, iterations are two-to-three weeks long. That quick delivery of features is very attractive to businesses and is seen as a main reason to use Agile instead of waterfall and other development methodologies. Focus too intently on that one aspect of Agile and forego others at your peril.
York has seem missed estimates kill projects. “You have to be dead on with your estimates otherwise the whole thing is screwed up,” he said. “There’s not enough time in the next iteration or the next to catch up when a goal is missed. He’s seen estimates being way off the mark in three Agile projects. In those cases, York said, at the very end, the project manager cut out what didn’t get done and then claimed it all got done. The result was debilitating technical debt.
If you don’t get everything done in one iteration, you have to change your plan and go back and fix things before technical debt builds up, wrote software test veteran David Christiansen in his article, How to deal with iteration issues in Agile.
An even better approach is not planning  too far in advance, said Crispin. Her Agile team avoids missed estimates by estimating only a sprint or two in advance. This gives them the ability to switch priorities very fast, to be very agile.
Agile’s short iterations lead to burn-out and mistakes
Business managers love short iterations, but they wear people out, developers told us.
Management is getting away with short iterations today because of the recession, said York. “People are going to burn out and start leaving. And as soon as the economy picks up, people are going to start switching jobs because they’re tired of being pushed to the limit.”
Dwyer sees testers feeling the pinch of short iterations the most, largely because they’re not allowed into a process earlier. Carignan agreed, noting that the expectation is often for test and QA teams to get their work done in the last eight days of a three-week iteration, a time when testing and production is happening almost simultaneously.

Agile development: What’s behind the backlash against Agile?

Try Googling “hate Agile” and “Agile sucks” and “Agile fails,” and hundreds of pages will show up. Some contain articles on how to avoid Agile software development failures and problems, and others are by development pros who have tried Agile and just don’t like it. Read the mostly anonymous comments on these articles, and you’ll find the hate being put on Agile right and left.
That’s the Agile backlash, another instance of any action causing an opposite reaction. Not everyone is on board the Agile train, despite the wave of almost-fanatical evangelism for and quick adoption of it. What’s written on the Web is the tip of the iceberg of dissatisfaction with the process, according to some of the users and experts cited in this series on the so-called backlash. Others think it’s the by-product of rapid and poorly-planned adoption.
In this article, we’ll explore the reasons why some software pros hate Agile, based on interviews with Agile coaches and consultants -- including Mike Dwyer, David Whalen, and Jon Kern – as well as software test and development veterans Scott Barber, Chris York and Lisa Crispin.
Too much Agile too soon?
Picture this: A CEO/CTO/CIO/CXO sitting in an airplane reading a magazine article that’s gung-ho about Agile. He or she goes back to the office and issues an order to adopt Agile development and, perhaps, to use the Agile process in application lifecycle management (ALM) and all business-side processes. Managers rush to obey.
This opening scene in the Agile backlash story was suggested without solicitation by almost all of this article’s sources. The chief executive’s decision sets the stage for mistakes to be made and has led to project failures and disgruntled development, systems architecture and project management pros. Frequently, the blame is place on Agile..and, of course, the media.
“Nobody has been writing about the problems with Agile,” said Chris York, a 15-year software test and development pro. “Every manager reads the same publications, and all the articles say Agile is good. So, the managers want to do it, even though a lot of them have no idea what it really is.”
Enthusiasm for Agile principles and real need, fueled by the economic downturn, has led to rushed adoption and project failures, according to software tester Scott Barber. “Folks got caught up in the idea or the ideal around Agile,” he said. “They decided, without knowing very much about it, that it was going to solve whatever problems they had. Barber is president of PerfTestPlus, a software testing services firm, and co-author of both Performance Testing Guidance for Web Application. and Beautiful Testing.
On various development projects, York has seen first-hand results of too much Agile too quickly. “No one at the top really knew in any depth what Agile is,” he said.  Barber adds that non-development executives often don’t understand all the implications of a methodology change. They require the switch, but say: “Do whatever you want, just don’t make things hard on us.”
Many of the pro Agile articles have been written by early adopters who have had success in Agile or Agile consultants promoting it as the answer to all problems, Barber said. “No one thing can be the answer to all problems.”
Too little analysis is being done of Agile’s appropriateness to the projects’ or organizations’ people, culture, management structure, sales cycle, business needs, development processes and tools and other key aspects, sources said. “Failures and problems with Agile can simply be because that it’s the wrong decision for that organization,” said Barber. “Is that because Agile is broken? Not fundamentally.”
Too strict with  Agile principles?
Leaders and organizations who insist that teams adhere too strictly to perceived Agile principles can create problems.
York has worked on Agile projects for more than one strictly Agile company. “Everything’s about the process and it’s not about the process being helpful. It’s about fitting into the process. That’s just backwards,” he said.
Development consultant Dave Whalen saw this same thing happen so often that he wrote the widely-read and controversial article, I Hate Agile.
Knowing the principles of Agile isn’t the whole ballgame, Whalen told SearchSoftwareQuality.com. Understanding the philosophy, which is still developing, is a must. Anyone who believes that all Agile practices must be done as written is missing the point. “Every environment is not best served by a completely Agile notion,” he said. “Probably only a few are.”
Mike Dwyer, a 30-year IT and development veteran, sees “a conservatism in some of the Scrum and Agile community that you have to do it a certain way, and if you’re not doing it that way, you’re not Agile.” Dwyer is principal Agile coach for BigVisible Solutions in Boston and a significant contributor to the Scrum, Agile, and Lean software community.
The Agile philosophy is about flexibility and using what works for particular projects, not implementing the exact same practices even when the project isn’t a fit for them, Whalen said.
Yearning for the silver bullet
Done well, Agile can help organizations streamline development, cut costs, improve software quality and more; but it’s not a shortcut or a universal panacea to all that ails Agile.
“People want the Agile Manifesto and the 12 principles to be more than just a bunch of principles,” Dwyer said. “They don’t understand in the simplicity of Agile. There’s a need for discipline, and they’re looking for something to take the place of discipline.”
Reading about Agile for the first time could be a “Eureka!” moment, but making Agile work well calls for leaders to do more than make top-down ultimatums for Agile adoption, these experts and developers said. Without discipline every step of the way, Agile can be yoke on the development teams’ neck and fuel the rebellion against it.

Nine Things Successful People Do Differently

8:58 AM Friday February 25, 2011
by Heidi Grant Halvorson | 

 

Why have you been so successful in reaching some of your goals, but not others? If you aren't sure, you are far from alone in your confusion. It turns out that even brilliant, highly accomplished people are pretty lousy when it comes to understanding why they succeed or fail. The intuitive answer — that you are born predisposed to certain talents and lacking in others — is really just one small piece of the puzzle. In fact, decades of research on achievement suggests that successful people reach their goals not simply because of who they are, but more often because of what they do.
1. Get specific. When you set yourself a goal, try to be as specific as possible. "Lose 5 pounds" is a better goal than "lose some weight," because it gives you a clear idea of what success looks like. Knowing exactly what you want to achieve keeps you motivated until you get there. Also, think about the specific actions that need to be taken to reach your goal. Just promising you'll "eat less" or "sleep more" is too vague — be clear and precise. "I'll be in bed by 10pm on weeknights" leaves no room for doubt about what you need to do, and whether or not you've actually done it.

2. Seize the moment to act on your goals. Given how busy most of us are, and how many goals we are juggling at once, it's not surprising that we routinely miss opportunities to act on a goal because we simply fail to notice them. Did you really have no time to work out today? No chance at any point to return that phone call? Achieving your goal means grabbing hold of these opportunities before they slip through your fingers.
To seize the moment, decide when and where you will take each action you want to take, in advance. Again, be as specific as possible (e.g., "If it's Monday, Wednesday, or Friday, I'll work out for 30 minutes before work.") Studies show that this kind of planning will help your brain to detect and seize the opportunity when it arises, increasing your chances of success by roughly 300%.
3. Know exactly how far you have left to go. Achieving any goal also requires honest and regular monitoring of your progress — if not by others, then by you yourself. If you don't know how well you are doing, you can't adjust your behavior or your strategies accordingly. Check your progress frequently — weekly, or even daily, depending on the goal.

4. Be a realistic optimist. When you are setting a goal, by all means engage in lots of positive thinking about how likely you are to achieve it. Believing in your ability to succeed is enormously helpful for creating and sustaining your motivation. But whatever you do, don't underestimate how difficult it will be to reach your goal. Most goals worth achieving require time, planning, effort, and persistence. Studies show that thinking things will come to you easily and effortlessly leaves you ill-prepared for the journey ahead, and significantly increases the odds of failure.

5. Focus on getting better, rather than being good. Believing you have the ability to reach your goals is important, but so is believing you can get the ability. Many of us believe that our intelligence, our personality, and our physical aptitudes are fixed — that no matter what we do, we won't improve. As a result, we focus on goals that are all about proving ourselves, rather than developing and acquiring new skills.
Fortunately, decades of research suggest that the belief in fixed ability is completely wrong — abilities of all kinds are profoundly malleable. Embracing the fact that you can change will allow you to make better choices, and reach your fullest potential. People whose goals are about getting better, rather than being good, take difficulty in stride, and appreciate the journey as much as the destination.

6. Have grit. Grit is a willingness to commit to long-term goals, and to persist in the face of difficulty. Studies show that gritty people obtain more education in their lifetime, and earn higher college GPAs. Grit predicts which cadets will stick out their first grueling year at West Point. In fact, grit even predicts which round contestants will make it to at the Scripps National Spelling Bee.
The good news is, if you aren't particularly gritty now, there is something you can do about it. People who lack grit more often than not believe that they just don't have the innate abilities successful people have. If that describes your own thinking .... well, there's no way to put this nicely: you are wrong. As I mentioned earlier, effort, planning, persistence, and good strategies are what it really takes to succeed. Embracing this knowledge will not only help you see yourself and your goals more accurately, but also do wonders for your grit.
7. Build your willpower muscle. Your self-control "muscle" is just like the other muscles in your body — when it doesn't get much exercise, it becomes weaker over time. But when you give it regular workouts by putting it to good use, it will grow stronger and stronger, and better able to help you successfully reach your goals.
To build willpower, take on a challenge that requires you to do something you'd honestly rather not do. Give up high-fat snacks, do 100 sit-ups a day, stand up straight when you catch yourself slouching, try to learn a new skill. When you find yourself wanting to give in, give up, or just not bother — don't. Start with just one activity, and make a plan for how you will deal with troubles when they occur ("If I have a craving for a snack, I will eat one piece of fresh or three pieces of dried fruit.") It will be hard in the beginning, but it will get easier, and that's the whole point. As your strength grows, you can take on more challenges and step-up your self-control workout.
8. Don't tempt fate. No matter how strong your willpower muscle becomes, it's important to always respect the fact that it is limited, and if you overtax it you will temporarily run out of steam. Don't try to take on two challenging tasks at once, if you can help it (like quitting smoking and dieting at the same time). And don't put yourself in harm's way — many people are overly-confident in their ability to resist temptation, and as a result they put themselves in situations where temptations abound. Successful people know not to make reaching a goal harder than it already is.

9. Focus on what you will do, not what you won't do. Do you want to successfully lose weight, quit smoking, or put a lid on your bad temper? Then plan how you will replace bad habits with good ones, rather than focusing only on the bad habits themselves. Research on thought suppression (e.g., "Don't think about white bears!") has shown that trying to avoid a thought makes it even more active in your mind. The same holds true when it comes to behavior — by trying not to engage in a bad habit, our habits get strengthened rather than broken.
If you want change your ways, ask yourself, What will I do instead? For example, if you are trying to gain control of your temper and stop flying off the handle, you might make a plan like "If I am starting to feel angry, then I will take three deep breaths to calm down." By using deep breathing as a replacement for giving in to your anger, your bad habit will get worn away over time until it disappears completely.
It is my hope that, after reading about the nine things successful people do differently, you have gained some insight into all the things you have been doing right all along. Even more important, I hope are able to identify the mistakes that have derailed you, and use that knowledge to your advantage from now on. Remember, you don't need to become a different person to become a more successful one. It's never what you are, but what you do.
Heidi Grant Halvorson, Ph.D. is a motivational psychologist, and author of the new book Succeed: How We Can Reach Our Goals (Hudson Street Press, 2011). She is also an expert blogger on motivation and leadership for Fast Company and Psychology Today. Her personal blog, The Science of Success, can be found at http://heidigranthalvorson.com/. Follow her on Twitter @hghalvorson

The secrets to getting started in your software testing career

The secrets to getting started in your software testing career

Reasons to Repeat Tests

Reasons to Repeat Tests

by James Bach
(with help from colleagues Doug Hoffman, Michael Bolton, Ken Pugh, Cem Kaner, Bret Pettichord, Jim Batterson, Geoff Sutton, plus numerous students who have participated in the "Minefield Debate" as part of my testing class. The minefield analogy as I talk about it was inspired by Brian Marick's talk Classic Testing Mistakes.)

Testing to find bugs is like searching a minefield for mines. If you just travel the same path through the field again and again, you won't find a lot of mines. Actually, that's a great way to avoid mines. The space represented by a modern software product is hugely more complex than a minefield, so it's even more of a problem to assume that some small number of "paths", say, a hundred, thousand, or million, when endlessly repeated, will find every important bug. As many tests as a team of testers can physically perform in a few weeks or months is still not that many tests compared to all the things that can happen to a product in the field.

The minefield analogy is really just another way of saying that testing is a sampling process, and we probably want a larger sample, rather than a tiny sample repeated over and over again. Hence the minefield heuristic is do different tests instead of repeating the same tests.

But what do I mean by repeat the same test? It's easy to see that no test can be repeated exactly, any more than you can exactly retrace your footsteps. You can get close, but you will always be a tiny bit off. Does repeating a test mean that the second time you run the test you have to make sure that sunlight is shining at the same angle onto your mousepad? Maybe. Don't laugh. I did experience a bug, once, that was triggered by sunlight hitting an optical sensor inside a mouse. You just can't say for sure what factors are going to affect a test. However, when you test you have a certain goal and a certain theory of the system. You may very well be able to repeat a test with respect to that goal and theory in every respect that A) you know about and B) you care about and C) isn't too expensive to repeat. Nothing is necessarily intractable about that.

Therefore, by a repeated test, I mean a test that includes elements already known to be covered in other tests. To repeat a test is to repeat some aspect of a previous test. The minefield heuristic is saying that it's better to try to do something you haven't yet done, then to do something you already have done.

If you disagree with this idea, or if you agree with it, please read further. Because...

...this analysis is too simplistic! In fact, even though diversity in testing is important and powerful, and even though the argument against repetition is generally valid, I do know of ten exceptions. There are ten specific reasons why, in some particular situation, it is not unreasonable to repeat tests. It may even be important to repeat some tests.

For technical reasons you might rationally repeat tests...

1. Recharge: if there is a substantial probability of a new problem or a recurring old problem that would be caught by a particular existing test, or if an old test is applied to a new code base. This includes re-running a test to verify a fix, or repeating a test on successively earlier builds as you try to discover when a particular problem or behavior was introduced. This also includes running an old test on the same software that is running on a new O/S. In other words, a tired old test can be "recharged" by changes to the technology under test. Note that the recharge effect doesn't necessarily mean you should run the same old tests, only that it isn't necessarily irrational to do so.
2. Intermittence: if you suspect that the discovery of a bug is not guaranteed by one correct run of a test, perhaps due to important variables involved that you can't control in your tests. Performing a test that is, to you, exactly the same as a test you've performed before, may result in discovery of a bug that was always there but not revealed until the uncontrolled variables line up in a certain way. This is the same reason that a gambler at a slot machine plays again after losing the first time.
3. Retry: if you aren't sure that the test was run correctly the other time(s) it was performed. A variant of this is having several testers follow the same instructions and check to see that they all get the same result.
4. Mutation: if you are changing an important part of the test while keeping another part constant. Even though you are repeating some elements of the test, the test as a whole is new, and may reveal new behavior. I mutate a test because although I have covered something before, I haven't yet covered it well enough. A common form of mutation is to operate the product the same way while using different data. The key difference between mutating a test and intermittence or retry is that with mutation the change is directly under your control. Mutation is intentional, intermittence results from incidental factors, and you retry a test because of accidental factors.
5. Benchmark: if the repeated tests comprise a performance standard that gets its value by comparison with previous executions of the same exact tests. When historical test data is used as an oracle, then you must take care that the tests you perform are comparable to the historical data. Holding tests constant may not be the only way to make results comparable, but it might be the best choice available.

For business reasons you might rationally repeat tests...

6. Inexpensive: if they have some value and are sufficiently inexpensive compared to the cost of new and different tests. These tests may not be enough to justify confidence in the product, however.
7. Importance: if a problem that could be discovered by those tests is likely to have substantially more importance than problems detectable by other tests. The distribution of the importance of product behavior is not necessarily uniform. Sometimes a particular problem may be considered intolerable just because it's already impacted an important user once (a "never let it happen again" situation). This doesn't necessarily mean that you must run the same exact test, just something that is sufficiently similar to catch the problem (see Mutation). Be careful not to confuse the importance of a problem with the importance of a test. A test might be important for many reasons, even if the problems it detects are not critical ones. Also, don't make the mistake of spending so much effort on one test that looks for an important bug that you neglect other tests that might be just as good or better at finding that kind of problem.
8. Enough: if the tests you repeat represent the only tests that seem worth doing. This is the virus scanner argument: maybe a repeated virus scan is okay for an ordinary user, instead of constantly changing virus tests. However, we may introduce variation because we don't know which tests truly are worth doing, or we are unable to achieve enoughness via repeated tests.
9. Mandated: if, due to contract, management edict, or regulation, you are forced to run the same exact tests. However, even in these situations, it is often not necessary that the mandated tests be the only tests you perform. You may be able to run new tests without violating the mandate.
10. Indifference/Avoidance: if the "tests" are being run for some reason other than finding bugs, such as for training purposes, demo purposes (such as an acceptance test that you desperately hope will pass when the customer is watching), or to put the system into a certain state. If one of your goals in running a test is to avoid bugs, then the principal argument for variation disappears.

I have collected these reasons in the course of probably a hundred hours of debate with testing students and colleagues. Many of my colleagues prefer different words or a different breakdown of reasons. There's nothing particularly sacred about my way of doing it (except that some breakdowns would lead to long lists of very similar items). The important thing is that when I hear a reason that seems not to fit within the ones I already have, I add that reason to this list. I started with two reasons, in 1997. I added the tenth one in late 2004.

Applying the Minefield: An Example

Ward Cunningham wrote "I believe the automation required of TDD [Test Driven Design] (and Fit) is exempt from the analogy because the searching we are doing is for the best expression of a program in the presence of tests, not the best tests."

Here's how I think it applies:

Your units tests might pass or they might fail. You write them so that they will fail in the event that some interesting expectation is violated. So, you call them tests and they seem to be tests.

We introduce the minefield criticism the first time you run any given test in your unit test suite. The first time you run it, it fails, right? Of course, since it wouldn't be TDD, otherwise. The questions below are inspired by the Minefield heuristic "vary your tests instead of repeating them."

Question: Why run it again?

Answer: Exception #1, "recharge." You run it again because you have added code to make the test pass, therefore running the test again is not merely redundant, the value of the test has been recharged by the code changing around it.

Question: During the course of development, but after the first time the test passes, why not delete it? Why bother to run it again?

Answer: Several reasons. Recharge still applies a little bit, since you may accidentally break the product during development, but it could be argued that most of those unit tests most of the time don't fail, and some of them are extremely unlikely to fail even if you change the code quite a bit. But here you have the second reason: exception #6, "inexpensive." It's so cheap to create these tests and to run them and to keep them running, while at the same time they do have some value, even if not a lot. And you have a third reason for some of the tests: exception #7, "importance." For a good many of the unit tests, failure would indicate a very serious problem, were it to occur. If you are testing something that is particularly complex, or involves many interacting sub-systems, you may also want to epeat because of exception #2, "intermittence". Perhaps something will fail after the forty-third run because of probabilistic factors in he test. Finally, there's #3, the "retry" exception, which reminds us that we might not have run the test correctly, before. As you once said, Ward, something might give off a bad smell only after you've seen the test run a hundred times or so. In other words, as a result of running a test many times, you might come to an insight about the product that reveals a failure that was there all along, but never noticed.

Question: Let's say that I'm a really good developer and though I write good tests, they just don't fail because I just don't put bugs into my code. I have a whole lot of tests and they don't fail. What was the sense in investing in such tests?

Answer: Two potential reasons. Exception #10, "avoidance/indifference." You may create the tests as a form of documentation for future developers and you like them to be exactly the same in order to minimize the chance that they will fail (and thus be less useful as documentation). Or maybe you want to impress a customer with your great software, and they won't be as impressed if the tests don't pass. A second reason is exception #9, "mandated." you may work this way because your peer group or your manager requires you to. This is a little like avoidance except that with a mandate you do, in fact, want to find bugs. You are searching for them, you just are required to use a certain technique to do so.

We therefore see that the fairly simple, often repeated unit tests of TDD may indeed be exempt from the minefield-based argument in favor of varying tests, inasmuch as the reasons I cited apply. But TDD is not exempt from this kind of heuristic analysis. It is always reasonable to question the value of repeated tests, and that's what the minefield invites us to do.

Estimation approaches in Agile development

Estimation approaches in Agile development

Managing the Testing Process

Planning for testing on a software project is often challenging for program managers. Test progress is frequently unpredictable, and during software testing painful schedule and feature "surprises" typically occur. Software testing is often viewed as an obstacle—more as a problem and less as a vital step in the process. For this reason, testing is treated as a "black box" and addressed at the end of the schedule. While budget and time may be allocated for it, testing is not really managed in the same way as development. Typically, software development is measured in terms of overall progress in meeting functional and business goals. Software testing needs to be measured in similar terms to understand its true progress and make informed decisions. By considering testing dimensions other than cost and schedule, managers and other team members can better understand and optimize the testing process, in effect opening the black box and managing testing more effectively. In this way they can avoid costly and painful "surprises" late in the project.

Years ago I worked on a new "next-generation" project at a software company. Everything seemed to be going well until the time came to do quality testing. As the software approached the test phase, the schedule looked pretty good. There had been some slips, but the project seemed on target for the scheduled release date. That is, until it actually went into testing.
On this project, testing had been treated as a black box. Software went into the lab, the testers did whatever it is that testers do, and bug reports came out. Since it was a black box, it was difficult for management to understand or justify time, resources, or equipment for testing. Also, nobody outside of the testing group, including project management, really understood what the process was or how to determine whether or not testing was complete. Several weeks were allocated to do a single test run, after which it was anticipated that the software would be ready to release.
Unfortunately, that was not the way the project went. Instead of a single pass through testing, ten different versions of the software required complete testing runs. The schedule was hopelessly overrun, and instead of having a new release to announce at an important show, all the company had were some "closed-door" demos for selected customers.
Testing does not have to happen this way. Looking beyond simple cost and schedule considerations opens up the black box, helping managers make better decisions about testing. When the testing black box is opened, and enough detail is provided to managers, it becomes much easier to understand the tradeoffs and to make informed decisions about resources and schedules needed to test a product. Managers are able to plan for and guide testing from the start, rather than ignoring it and hoping for the best. When the entire project team is kept up to date on testing, understands the approach used, and can observe the progress of testing while it is being done, it is easier for the team to function effectively as problems are found and as new versions go through the testing phases of the project. There are other advantages to opening up testing as well.
Often, there are early warning signs that testing is going to have problems. These show up in the details of the analysis and design phases of the tests themselves. They appear in the form of incomplete or deferred work due to missing information, improperly managed problems recorded against key functionality, and other "small" indicators accumulating over time. If these indicators are spotted far enough ahead of time by managers, developers, and the testers themselves, work can be done to head problems off while they are still small. This in turn ensures that the testing group is better prepared for the software and that the software is better prepared for testing.
Organizations can avoid last-minute quality issues by addressing testing problems earlier in the process, when they are still small. Doing this requires better insight into a project than what can be gotten from a Gantt chart. During test development, management needs to know the status of test planning and preparation to properly gauge the readiness of the test team to test the software. This knowledge comes from understanding

• the relationship between the software functionality and the testing to be done on that functionality
• the relationship between the software design and the tests expected to verify that software
• specific problems the tests are intended to address

It is not just management that needs to know this information. Software developers and testers also need to be able to see and work with this information, and to understand the parts that are relevant to their own work. With adequate information, testers are better able to plan and perform their tasks more efficiently, and to get an accurate picture of the system. For example, it is not uncommon for a set of unmeasured functional tests to repeatedly test one part of the code, while partially or completely missing other parts. In addition, feedback and questions from testers can help to prevent problems later as inconsistencies and inaccuracies are found. Likewise, the developers are able to anticipate potential problems with either the software or the tests, and work with the testers to address these before they have a significant impact.
While testing is in progress, results should be measured in terms of software functionality tested, degree of code coverage, progress against schedule, and problems found. Usually these factors provide a much clearer picture of the status of the testing (and the overall quality of the software) than how long the testing has been running.
Compiling and communicating complete information about test activities provides a common way for everyone on the project to track test activity progress. Testing status needs to be presented to each member of the team in a way that is understandable and that lets them make the best decisions possible. The project manager needs to have a comprehensive view of the progress of test preparation and execution, understanding at a global level what tests are being designed, what tests were run against different versions of the software, what areas of functionality are affected by significant bugs, and what parts of the code will be affected by the bugs that have been found. Developers need to have information specific to their part of the software process, knowing what tests will touch on their area, what problems have been found in their area, how significant those problems are, and what testing has actually been done on their software. Testers and QA staff need to have a different, but similarly detailed view of the system. In short, each person on the project has a different role, and so each needs information that is appropriate to their function.
To support these different needs, different levels of detail in each of the following categories must be provided to each group.

1. Schedule: What tests will be run? When will the tests be ready? How much effort will it take? When will it be complete?
2. Functionality: What requirements will be tested and where? How will tests divide up application requirements? How much of the functionality has been tested for a given version of the software?
3. Code: What parts of the code are exercised by the tests? What problems have been found? How much of the code in a given version has been executed during testing?
4. Problems: What problems are tested for? What problems have been found? How significant are the problems? What parts of the software are affected by the problems? What versions are affected by the problems? What requirements are impacted by these problems? What is the impact of these problems on the testing?

The questions posed for each of these areas must be carefully examined in order to properly understand and track the status of project test activities. In addition, having a solid understanding of the planning and preparation requirements for each testing phase is critical to making correct decisions about project schedule, status, and release.
Table 1 below summarizes commonly used testing metrics and where in the testing process they apply.

Metric Type	Test Development Metrics	Test Execution Metrics
Functional Metric	Number of requirements allocated by test % of requirements by test development phase	Number of requirements verified % of requirements tested by version % of requirements tested by major software component Stability of server/platform per user
Code Metric	% of code covered per test % of code coverage per major software component	Code coverage of tests completed for each version under test
Problem Metrics	Problems tested for in regression tests Extreme conditions tested for in functional tests	Problems found per version tested Problems found per software component Number of critical/high problems found per version
Schedule Metrics	% completion of functional test requirements by testing phase Weighted functional requirement completion	Tests completed per version Estimated number of days to complete Test cycle completion time Time to complete testing per functional area

TABLE 1: Commonly Used Testing Metrics
Managing a successful testing effort requires knowing all about project schedules. The process begins during test planning and involves not only understanding how long it will take to prepare tests, but also estimating how long it will take to run the tests and what resources will be required. During planning, preemptive decisions about required resources and the scope of testing can be made before testing actually begins. Tradeoffs between resources and time are possible at this stage. Also, decisions about what tests to automate and how much to automate can be made and tracked. At the same time, estimates of costs and numbers of test runs for different versions can be made and refined as test time approaches. This allows estimates of how much testing needs to be done concurrently, and how to plan for it. As testing is performed, the number of times the software needs to be tested and the amount of time each version takes to test can be factored in to decision making. However, it is not possible to make scheduling decisions without knowing the functionality, code, and problem aspects of the testing to be done.
Functionality is a critical measure of testing completeness. Decisions need to be made up front about the degree to which functionality will be tested, and which parts of the functionality will be emphasized during testing. Functionality is usually described in terms of requirements, which are divided among different functional and systems tests according to the nature of the requirements and the needs of the system. Often, for reasons of efficiency and to get a better understanding of how software will perform after release, test requirements are allocated very differently from the allocation of requirements among different components of the software. For this reason, it may be necessary to run multiple tests from several areas to completely test software components. Following this process can reveal problems in areas of the software that have been passed over in earlier tests. This is also why it is important to see testing from the code perspective.
When considering functionality in testing, it is important to think also about how the functionality is being tested. It is common to develop a relatively "quick-and-dirty" set of tests that exercise the functionality of the code in a "nominal" way, to make sure that all of the pieces are in place and a given version of the software is worth the added effort of doing a complete set of tests on. The complete testing suite should be targeted at testing the functionality to a more exacting standard, in essence deliberately trying to push the software to its limits. This type of functional testing includes load, performance, and endurance testing where appropriate.
Understanding how tests and problems map to the code is necessary to understand how far along testing has progressed, what remains to be tested, and how complete the testing has been. A detailed knowledge of the code is not required for the entire project team. However, management does need to understand how much of the code has been tested, how much remains to be tested, and how many modules are affected by problems discovered during testing. With this information, informed decisions can be made about the consequences of fixing or leaving problems alone. Armed with such information about test coverage, it is possible to understand how thoroughly the software is being tested, and how much of the software is being exercised by the tests. Often this comes as an unpleasant but enlightening surprise the first time it is measured.
When considering code coverage as a measure of the completeness of testing, it is often important to decide what a reasonable level of coverage is for the software being tested. There are many different ways to measure completeness of code coverage, some of which are extremely difficult to achieve for a typical piece of software. These higher levels of coverage are commonly used for software with a correspondingly high level of criticality—either software that is widely used, such as operating systems or Web servers, or software that is used in very critical functions, such as medical devices or aircraft controls. Regardless of the project, however, teams need to decide on a level of coverage that is most appropriate to the particular software under test.
Measuring the impact and consequences of problems that arise during testing is a critical step in the process. This should include how much of the software is affected by a given problem, at what point during testing a problem was found, and what kinds of problems regression tests are attempting to uncover. This information is needed to monitor the overall progress of the software through testing and to make informed decisions about software release. So by combining all of the different perspectives of schedule, functionality, code, and problem resolution, it is possible to understand and manage software testing, rather than treating it as a black box.
The catch to this is that while these elements of information are usually present in a software project, it is often difficult to find them. Also, typically such information is presented at a level of detail and terminology understandable only by testers and developers. Recently, test management tools have been developed to collect and present this information in the different forms that are better targeted to different members of the software development team. For example, higher level progress and statistical information is presented to management in a simple graphical form, while more detailed technical information is expanded out in drill-down views for users who need that level of detail. These tools can be set up to provide a common basis for discussion and decision making during the testing cycle. They can also help management to avoid treating testing like a black box.
As we have seen, treating testing as a black box at the end of the project schedule invites failure. Testing and test preparation need to be open, managed, and manageable processes that everyone on the project team can understand and work with. Having insight into all testing requirements and desired outcomes is an absolute must for effectively managing a successful testing effort.
Finally, having current information available for testing is important to prepare and manage the project for testing. Testing progress needs to be measured in terms of schedule, functionality, code coverage, and problems. While the exact way in which this information is presented will vary according to the needs of the organization and the project, these four measures of progress need to be available in a form that allows each member of the team to understand the current status of testing, what needs to be done, and how that will affect their part of the project. Using that knowledge, the team can ensure the success of testing and remove one of the major stumbling blocks to a successful project and a successful product.