Ad-hoc Testing: An important Process of Software Testing

Most of the software professional doesn’t like the term called “Ad-hoc Testing” because it implies a lack of testing process or purpose, but this really plays a very important role in Software Testing Life Cycle. “Ad- hoc testing” is a process which comes under Black Box testing and is the least formal method of testing. During Ad-hoc testing, testers doesn’t need to execute test cases, not bounded to test the functionality assigned to him/her, here they just use their intuition or experience. The tester has to find bugs without any proper planning and documentation solely based on just his intuition. If carried out by a skilled tester, it can often find problems that are not caught in regular testing cycle. Sometimes, if testing occurs very late in the development cycle, this will be the only kind of testing that can be performed.

When I was a tester, I was told to focus on the functionality assigned to me and make sure that customer should not find any issues while using the functionality, the term called “FUNCTIONALITY OWNER”. But I wanted to do something different as I got bore to perform the same activities for every product testing along with their release cycle, so I started doing testing of the entire product instead of testing only those functionality assigned to me just after 2 rounds of System Integration Testing using my knowledge and without using any test cases. The result was excellent. I found good issues which were with the product from the very beginning. After analysing those issues it was found that those information was missing in the requirement document.

Based upon my findings, I approached to my Test Manager along with my test lead and try to convince him that if the entire testing team will perform one round of Ad-hoc testing just after finishing the System Integration Testing cycle then there is a chance to get more issue which we might miss during the regular testing cycle. It was very hard to convince him as in the Testing Approach / Strategy document; no information about the Ad-hoc testing cycle was mentioned. We were already completed 2 rounds of System Integration Testing cycle and waiting for the new release / build to perform regression testing followed by Acceptance Testing by Product Management team. Test Manager also wanted to have one round of Ad-hoc testing based upon the issue found by me so he has taken those issues to the Steering Committee meeting and tried to convince the entire stake holders and requested to include one week on Ad-hoc testing. Somehow we got the approval from Steering Committee to include one round of Ad-hoc testing and in this way, we the entire testing team has been asked to perform one round of Ad-hoc testing based upon our testing skills, knowledge of the product and intuition only. After one week of Ad-hoc testing, the result was:

1.       20% Defects (Showstoppers and High priority) of SIT Cycle 1 and 2 has been found.

2.       10% Defects (Medium priority) of SIT Cycle 1 and 2 has been found.

One of the best uses of ad-hoc testing is for discovery. Reading the requirements / specifications Use cases rarely gives you a good sense of how a program actually behaves. Even the user documentation may not capture the “look and feel” of a program. Ad hoc testing can find holes in your test strategy, and can expose relationships between subsystems that would otherwise not be apparent. In this way, it serves as a tool for checking the completeness of your testing. Missing cases can be found and added to your testing arsenal. Finding new tests in this way can also be a sign that you should perform root cause analysis.

Finding new tests in this way can also be a sign that you should perform root cause analysis. Ask yourself or your test team, “What other tests of this class should we be running?” Defects found while doing ad-hoc testing are often examples of entire classes of forgotten test cases. Another use for ad-hoc testing is to determine the priorities for your other testing activities.

It has also been found that ad-hoc testing can also be used effectively to increase your code coverage. Adding new tests to formal test designs often requires a lot of effort in producing the designs, implementing the tests, and finally determining the improved coverage. A more streamlined approach involves using iterative ad-hoc tests to determine quickly if you are adding to your coverage. If it adds the coverage you’re seeking, then you’ll probably want to add these scenarios to your existing test cases.

A good ad-hoc tester (Ad hoc testing can be performed effectively and efficiently by the experienced testers) needs to understand the design goals and requirements for these low-level functions. What choices did the development team make, and what were the weaknesses of those choices? As a tester, we are less concerned with the correct choices made during development. Ad-hoc testing can be done as black box testing. However, this means you must check for all the major design patterns that might have been used. This allows you to narrow the testing to many fewer cases.

An important element of any IT strategy is to ensure deployment of defect free systems. Among other benefits, ad-hoc testing helps minimize significantly the total cost of ownership (TCO) of applications. However, organizations quickly discover that despite their best intentions and efforts, their QA team has hit a ceiling from a defect leakage standpoint. It seems as if an invisible hurdle is preventing the QA team from achieving its true potential - deploying defect free systems.

Now a day’s ad-hoc testing finds a place during the entire software testing cycle. Early in the project, ad-hoc testing provides breadth to testers’ understanding of the application and helping them more effective and quality test cases, thus aiding in discovery. In the middle of a project, the data obtained helps set priorities and release schedules of the software program / Application. As a project nears the ship date / acceptance testing by business users / product management team, just after completion all of your testing cycles as mentioned in your Test Approach / Strategy document, ad-hoc testing can be used to examine the quality of the application.

A primary goal of ad-hoc testing is to uncover new defects in the product / specification. In the hands of a skilled tester, it can be highly effective at discovering such problems. Regression tests and ad-hoc tests complement each other in remarkable ways. When you find a defect with a regression test, you'll often need to use ad hoc methods to analyze and isolate the defect, and to narrow down the steps to reproduce it. You may also want to explore “around” the defect to determine if there are related problems. On the other hand, when you find a defect with an ad-hoc test, you'll probably document it in your defect tracking system, thereby turning it into a regression test, for verification after the defect is fixed.

The major benefits of ad-hoc testing are:

1. No planning and documentation needed. It can be added successfully during beginning of the project, mid or before release of the product.
2. The important bugs are found which helps you in discovering missing test cases - can find holes in your original testing strategy that can be incorporated later.
3. Gives more understanding of how a software application or feature behaves which may not be known by reviewing specification documents or use cases.
4. Gives better understanding of the testing priorities, for example if an ad hoc test runs very well, you can decide testing in that area can be deferred to the later stage.
5. Easy to start and Implement.
6. It helps to save a lot of precious time. Sometimes it’s after spending valuable time on preparing and explaining tests that the program’s design is changed. With Ad-hoc testing precious time is not wasted on planning and documents.

      The success of Ad-hoc testing depends upon the capability of the tester who carries out the test. The tester has to find bugs without any proper planning and documentation solely based on just his / her intuition.

In my testing carrier, wherever I work, I have always given importance of ad-hoc always put a round of Ad-hoc testing just after completion of System Integration Testing cycle if the new version of product is getting tested. I have also used the same before the regular testing cycle for a brand new product (first version of product) followed by performing the same before sending to product to either release or sending for the acceptance testing by product management team just after finishing the regular testing cycle. I always mentioned 2 type of testing in my Test Strategy / Approach document which are Formal and Less Formal Test like:

Formal Test		Less Formal Test
Testing Type	Duration	Testing Type	Duration
Build Acceptance Test	1 day
System Integration Testing Cycle 1	2 weeks*
Initial Regression Testing	2 days*
System Integration Testing Cycle 2	2 weeks*	Ad-hoc Testing	1 week
Final Regression Testing	1 week	Bug Hunts	1 day
Performance Testing	8 weeks*	Alpha testing	1 week
Acceptance Testing by Users	2 weeks*
Regression Testing	1 week
Smoke / Release Testing	1 day
* duration depends upon the size of the project and may varies. The standard duration is mentioned hers.

      

      Last but not the least, if you are planning to establish TMMi framework within your organization, then you must implement a round of Ad-hoc testing during PA (process area) 2.1 of Level 2 which is “Test Policy and Strategy” and PA 2.2 which is “Test Planning”. I am sure if you implement the ad-hoc testing during Level 2 you will easily achieve Level 3 – “Defined” to Level – 5 “Optimization”. 

      

      The same article has been published at: 

      http://qa.siliconindia.com/qa-expert/Adhoc-Testing-An-important-Process-of-Software-Testing-eid-134.html

Defect Driven Development Continue......

Defect Driven Development (DDD) sounds similar to TDD (Test Driven Development) which is a major function of Agile. But in fact DDD is not fall under Agile Rule. As per my experience I can say that the main principal of DDD falls under Level 5 of TMMi, which is “Defect Prevention” and “Quality Control”. DD actually comes in the picture when the entire functionality as per requirement has been implemented with 2 rounds of SIT (system Integration testing) and you are in the process of creating Final Candidate Build / Release followed by inviting business users for their Acceptance testing. Just 3-4 weeks before testing a FC Build / Release, you do “Triage” of open defects with the help of Business, Development and Test Managers and prioritise the defects which needs to be fixed before FC Build / Release gets prepared.

Even if TDD eliminates 75% of the defects – and how likely is that – the situation is qualitatively the same. You still need to expose the product to testers and field users – “Defect-Driven Development.”

Defect Driven Development (DDD), is also a software-development methodology based around using a project's defect tracking system to track not just defects, but also software features. The central concept in DDD is that from a user's perspective, the biggest defect in a piece of software is that a “necessary feature is missing”. Whether those feature works as it should is critically important, but actually is a secondary concern in the project lifecycle.

The primary advantage of DDD is that it addresses the need for both project governance and transparency, which are generally quite difficult to achieve without rigid project lifecycle methodologies and considerable overhead, usually at the cost of productivity. Because DDD uses a project's ubiquitous defect tracking system that developers already use, but in a novel way, it is a lightweight methodology that can be easily adopted without the need to introduce significant new processes or tools.

I have personally used this methodology in my previous company Cincom and found that this really helps in fastening the Final Release Cycle with Quality and meeting the 100% customer’s expectation. I formulated this approach because, the engineering teams work under very demanding circumstances including wide geographic distribution (UK, US France) and native language differences, and we needed better ways to track project progress and feature implementation before the final release cycle. My product management team has created a virtual image of Mrs. Smith (who was doing manual work before started working on software product) while writing the use cases and the same was used during the analysis of Defects Review also like How Mrs. Smith will use the functionality....what will be there in her mind before using the same.....etc.

We should think of defects as good things that help us achieve what users really want and need.

Assumptions and Prerequisites

There are a few assumptions and prerequisites needed to make DDD work. On the whole, the list is quite short, and covered by most defect tracking systems in widespread use:

• Project uses a defect-tracking system.
• Developers are familiar with and committed to using the defect-tracking system.
• The defect-tracking system allows for assignment of defect to an individual, who can then track his assigned defect easily.
• The defect-tracking system allows defects to be designated as dependent on or blocking another defect, and can automatically help users traverse dependency graphs of defects.
• Pre-emptive notification (for example, email) of changes to defect status or addition of new defects.

How It Works

1. Setup a Defect Triage Team. The member should be Development Manager, Test Manager and Product Manager / Business Development Manager with BA’s (If any).
2. Create a list of all Open Defects with Priority and Defect ageing by running a query.
3. Circulate the defect list to the Defect Triage Team.
4. Setup a meeting. If the entire team is at one location then books a meeting room with System connected to the internet and having Defect Tracking Tool installed else setup a conference call with the Defect Triage Team.
5. Start reviewing the defect which has priority as “Critical”.
6. Discuss every defects with customer prospective, analyze the same that what happen if Mrs. Smith will come across with those issues and then take a call   

            a.  Whether the defect is a real defect or an enhancement.

b. In case of Defect, is Priority Ok and customer will really suffer or     the same should be of lower priority and can be defer for the next release.

c. In case of defer, update the defect with the status a “Defer for next release”.

d. In case the defect is really a critical, assign the defect to the Development Manager. Once the meeting will be over Development Manager will discuss those defects with development engineer and identify the ETA followed by assigning the defect to the development engineer and update the ETA.

e.  In case of Enhancement, assign the defect to the Product Manager / Business Development Manager for further action. If Product Manager / Business Development Manager wants those enhancements should be fixed then create a document for the same enhancement which has estimation from Development Manager and Test Manager. If those enhancements can be implemented without impacting the release date then start auctioning on the same and put the information for those enhancements with the Steering Committee pack / deck and discuss the same in details during the next Steering Committee meeting.

                       i.   After estimation, if you find that the same is impacting the release date then take the same to the CCB (Change Control Board) for further action followed by discussing the same in Steering Committee meeting. If stake holders are agree with changed release date then take an action else defer that defect for the next release.

f.  Do the same activity for High, Medium and Low priority defects also and follow the steps 6.

g. At the end of Defect Triage meeting, you will see a major change with the status of Open Defect.

Make sure while Triaging a defect the TEAM is aware about the Product Release date and all the action should be taken accordingly. It will be to keep a release date a week earlier as a contingency during the defect triage meeting to mitigate any last moment change.

Effects

Overall defect counts start high and go down

Summary

At this point, the general concept of DDD should be clear. In summary:

• DDD is not really very different than the process most projects already follow. The difference is primarily in timing of and perspective in using the defect tracking system: The defect tracking system is used from the very beginning of the development process to track features.
• Clear owners are defined for each deliverable, whether that's an overall feature, code, designs, icons, docs, or something else relating to the project.
• Because defects are dependent on other defects, owners have a natural reason to talk to colleagues and resolve issues rather than go off on their own.
• Defects can be refactored into sub-defects as needed in a right-sized manner for the project. Sometimes, a feature can be wholly implemented by one person, so only one defect is needed. But in the cases where multiple people are involved, that can be expressed with clear ownership.
• The information in the defect tracking system can be mined to create a dashboard of overall project status.
• Defects filed by customers, QA, and others can be related back to the parent feature. This means that both unimplemented features and defects appear in the same dashboard. This is critical for getting an overall picture of the completeness of the project.
• Defects can be assigned and reassigned as needed to gather more information, or to transition work from one person to another, even across widely separated time zones and organizational boundaries. The overall project status remains the same, however.

Consistency of implementation

Because features and defects are in the same place, the developer's primary task list is the set of defects assigned to him. This puts him on the same pages as his manager, architect, and tech lead; there is no sudden changeover between feature implementation and defect fixing, as they are aspects of the same thing. And because the features in the tracking system really describe constraints and deliverables, engineers are still free to innovate like before. The defect tracking system gives him a place to capture his thinking, ask questions, and communicate about what he's doing beyond a very loose notion of "done" or "not done".

Better engineering and QA relationship

In traditional engineering organizations, the engineering/QA relationship is adversarial in nature, or tends to become that way, because the roles are distinct and responsibility is tightly bounded. Part of this comes from the fact that developers may feel caged by having to work on defects instead of new features, and QA engineers may feel disenfranchised when voicing user concerns that are ignored.

DDD has the potential to remove the adversarial component from this relationship in two ways. First, defects are part of the standard development process from the beginning, rather than coming in near the end. Developers have less reason to react negatively to issues filed there because they have been socialized to it from the beginning of the project, and they are accustomed to the idea of both missing features and defects as contributing to user dissatisfaction.

Second, QA engineers can be much more involved in the ongoing feature work because it's well-documented and there is a means by which to be heard. They can comment, add new features and requirements to the set, and even work on developing features rather than feeling powerless until a feature has been implemented. They can feel that their voice has been heard throughout the process, and so there is less reason for them to have to say, "I told you so" when something is implemented or designed poorly.

By making features transparent to developers, QA, and management, and futhermore by giving a concrete means to discuss features and design openly before completion, DDD can offer a formal channel for user advocacy to flow into the design process, not only from QA but from anyone with access to the defect tracking system.

Advantages

There are numerous advantages to the DDD approach:

• Development transparency from the beginning of the development process. As features are completed, the defects are closed. As defects are filed, they can be related back to the feature using the dependency mechanism, and the feature can be reopened as a visible indicator of status.
• Issue and task tracking are automated. The defect tracking systems have excellent methods for assigning defects to developers in an easy-to-track way and notifying interested parties of changes in state.
• Full organizational involvement. Engineering managers are good at making sure defects get closed; the key to this is giving them something they can track. Furthermore, docs and QA need a way to track their deliverables as they relate to project features. Because anyone can either add themselves as interested parties to a defect, or can file their own defects with dependencies, this becomes much easier.
• Cross-organizational coordination. When working across teams and geos with large numbers of features and designs, it becomes imperative to track design decisions and status in a more formal way. Emails are easily lost, and if the right aliases are not used (and archived), then design decisions are all too easy to remain detached from other organizations.
• Better design tracking. All features have a unique defect number, and all discussion (or summary of discussion) can be added to one place for all to see. Design discussion will still occur face-to-face and over email, but relevant decisions can be recorded and then easily tracked in the defect.
• Easy prioritization during traige. All defect tracking systems have the concept of a priority which can be used to indicate the importance of a feature in the final product.
• DDD is very effective when used in iterative development. For example, the umbrella defect for each iteration can contain all the tasks which need to be done before that iteration is complete. When a feature task is complete, QA can add blocking defects on the iteration, so that the release iteration will be stable and closed, thus providing a usable set of functionality that has been tested.

X                                                       

                                                           XXXXXXXX

Defect Driven Development

I am getting questions / query about "Defect Driven Development", some of the testing professionals are showing their interests towards this topic, hence I have decided to to write down a document for the same.

But for the time being here is some information.......

"Defect Driven Development (DDD) sounds similar to TDD (Test Driven Development) which is a major function of Agile. But in fact DDD is not fall under Agile Rule. As per my experience I can say that the main principal of DDD falls under Level 5 of TMMi, which is “Defect Prevention” and “Quality Control”. DD actually comes in the picture when the entire functionality as per requirement has been implemented with 2 rounds of SIT (system Integration testing) and you are in the process of creating Final Candidate Build / Release followed by inviting business users for their Acceptance testing. Just 3-4 weeks before testing a FC Build / Release, you do “Triage” of open defects with the help of Business, Development and Test Managers and prioritise the defects which needs to be fixed before FC Build / Release gets prepared."

Guys....wait for some time...I will publish the complete document on the same topic very soon.....

Test automation strategy: Getting started

You’re the QA director or CIO at a growing software organization. You’re sold on the idea that test automation is necessary in order to keep your team on track, provide quick feedback and manage technical debt. But like many software organizations, you have none at all. Where do you get started?

The whole-team approach

The history of test automation is littered with failed “test automation projects.” Test automation sounded like a good idea, so the company bought a vendor GUI test automation tool, gave it to the test team, and waited for the magic to happen. Here’s the news: automating tests is software development. It must be done with the same care and thought that goes into writing production code. And it needs a diversity of people and skills: testers to know the right things to test, customers to provide examples of desired behaviour, programmers to design maintainable tests and more.

Test automation succeeds when the whole development team treats it as an integral part of software development. It’s not any harder or easier than writing production code. Learning to do it well takes time, effort and lots of small experiments.

Change is hard, and people are motivated to change when they feel pain. Start by sharing the pain of manual regression testing. Your product can’t be released to production without regression testing, right? Ask the testers to work with customers to identify the most critical areas of the application. They can then write manual regression testing scripts to ensure those areas work.

Automating the manual regression test scripts provide major motivation for automating tests!

Overcome barriers to automation

Your cross-functional team has all the skills needed to overcome barriers to test automation. Get together for some time-boxed brainstorming meetings to identify the impediments standing in the way of automating tests. In my experience, the best approach is to identify the biggest obstacle, then try a small experiment to get around it.

For example, when my team wanted to try test-driven development (TDD), which involves automating unit-level tests, we couldn’t figure out how to do that with our nasty legacy code. First we tried brown bag sessions at lunch where the whole team tried writing unit tests. Then we brought in an expert to do training on it. That helped, but finally we just had to budget time to get traction on it.

Should you tackle automating unit tests with test-driven development? Would GUI tests be the easy win? It’s helpful to outline a strategy for automating tests.

Plan your automation strategy

There are so many tests to automate and so little time; it pays to focus on the automation that will provide you with the best Return on Investment (ROI). Mike Cohn’s Test Automation Pyramid provides a good guideline.

Unit and component tests verify our code design and architecture, and provide the best return on investment. They are quick to write, quick to run and provide a good safety net as part of a continuous integration process. They form the solid base of just about any test automation strategy. If your team has no test automation, learning to automate unit tests is the obvious place to start.

The next-best return on investment is the middle layer of the triangle, at the API or service level. These tests generally take the place of the user interface. They pass test inputs to production code, obtain results, and compare actual and expected results. Writing and maintaining them takes more time than unit tests, and they generally run more slowly, but they provide crucial feedback.

Today’s test libraries and frameworks allow us to automate and maintain GUI tests more cheaply than the old-school record/playback tools. Even so, they run more slowly than unit or API-level tests, and may require more frequent updating.

In my experience, a two-pronged approach to start automating tests on a legacy system works well. While the team masters TDD and grows its unit test library, you can get some quick wins with simple GUI tests covering the most critical functionality.

The test automation pyramid represents our long-term goal, but you won’t get there overnight. Most teams start with an “upside-down” pyramid, more GUI tests than anything else. You can “flip” the pyramid over time.

Choosing tools

We don’t have a wide array of tools at the unit test level; you just use the flavor of xUnit that goes with your production programming language, such as JUnit for Java or NUnit for .Net.

For API and GUI level tests, start by experimenting with how you’d like to specify your tests. Does a given/when/then format work well for your business experts? Perhaps your business domain lends itself to a tabular test format. Or, you might have a domain where specifying inputs and expected outputs in a spreadsheet work better. Some businesses prefer working with a time-based scenario.

Once you’ve found a format to try, look for a test library or framework that supports that. If your team has enough bandwidth, you might be able to “grow your own.” Use retrospectives to evaluate whether your experiment is having good results. If not, start a different experiment. This is a big investment. The right choices now will mean big returns in the long term.

Collaborating

Automating tests is coding. Automated test code deserves the same respect, care and feeding as production code. It makes sense for the people writing production code to also write the test automation code.

Testers are expert at knowing the right tests to automate. Other team members contribute expertise that helps us get timely feedback, with data and scenarios that represent production. It just makes sense to have everyone on the team work together to automate tests.

Get started

Pick one problem to solve, and do a small experiment to overcome it. If an automating unit test seems too daunting, try “defect-driven development”. For each defect that your team fixes, first write a unit test to reproduce the problem, and then correct the code. Once the test passes, check in the test and the code.

This is a journey, not a destination. Just take that first step.

What are some of the obstacles to beginning test automation in your organization, and what approaches have you tried to address them? Email comments to sanjay_kumar@hotmail.com . 

The end of an iteration: When is testing in Agile complete?

How do you know when you're done testing a feature? In a time-boxed environment, such as Scrum, are you "done" when the iteration is over, even if you don't feel you've had time to fully test?
In Agile development, testing isn’t a separate phase, it’s tightly integrated with coding. In fact, we start development on each user story by writing business-facing tests that will tell us what to code and help us know when we’re done. Testers, business analysts and developers work with the business stakeholders to elicit examples of desired and undesired behavior for each user story and feature, and turn these into executable tests. This is called Acceptance Test-Driven Development (ATDD) or Specification by Example. The development team collaborates with the customers to decide which tests will prove a particular user story delivers what the customer expects. This will include automated functional tests, manual exploratory tests and non-functional testing such as performance or security testing. When those tests are all passing, you’re done with that user story.
User story estimates must include time for all testing activities, including test automation and manual exploratory testing. When we plan our iteration, we only plan the user stories which can be completed, including all testing activities. New Scrum teams often over-commit, planning more work than they can possibly finish. They end up in a mini-waterfall, with testing pushed to the end, and their features are not done just because the last day of the sprint arrived. This leads to a death spiral of stories dragging from one iteration to the next, with the testers unable to ever catch up.
Agile teams have an advantage – they include all the roles necessary to understand the customer needs and deliver high quality software. Diversity of skills and experiences allows Agile teams to find ways to help business stakeholders define their needs with concrete examples, and translate those examples into the tests which define “done” for each user story and each feature.
One way to avoid the “mini-waterfall” syndrome is to focus on finishing one story at a time. On my team, when testing tasks fall behind, programmers, DBAs and other team members pitch in to finish up an almost-done story rather than work on different ones. That’s one way to spread out the testing work over the whole iteration. The whole team takes responsibility for automating all regression tests, so that testers can focus on manual exploratory testing and other important activities.
We testers can always find more testing to do; we never feel “done.” However, working closely with business stakeholders, and planning adequate time for proving features are “done,” allows us to delight our customers. New Agile teams usually can’t deliver much new functionality in each iteration. But if they invest time to find ways to understand customer requirements and translate those into tests which guide development, they’ll bake in quality that allows them to do more and go faster later on.