Software Test Planning

Course Title: Software Test Planning

Course #: STR13

Duration: 2 or 3 days

Overview:

Effective testing is vital to organizations, and developing workable test plans is an essential survival skill for testing & QA professionals and their managers.

Testing often consumes 25% to 35% of the entire software development and maintenance budget, but typically only catches 80% to 90% of the bugs. Conclusion: testing is expensive (and time consuming), and despite the high cost is not very reliable. An implication: many testing projects probably could be managed better. Another implication: since managing depends on effective planning, it is fruitful for us to examine how we can plan our testing better.

Organizations often resist adequate testing, perhaps out of denial (“we can’t have many bugs”). Perceptions that testing delays the software delivery –and that deadlines are paramount – are other major reasons. In addition, testers are often unhappy with the intense work pressures, lack of appreciation and what they see as a Acan=t win@ atmosphere. By providing a sense of the test goals (where we want to be) and direction (how to get there), a good test plan alleviates the undesirable aspects of test projects.

Dwight Eisenhower was the chief planner of the Normandy invasion, D-Day in World War II. He said that the plan itself was not significant, it was the act of planning which really counted. We agree with this idea: an artifact like a test plan accomplishes nothing by itself. Nevertheless, the test plan is the physical manifestation of the process of getting ready for testing. And the test plan, if it is well done, can be a vital instrument for managing the day-by-day activities of the on-going test project.

This workshop addresses the challenges of test planning and their solutions, the test planning process, the outcome of that process (the test plan), and how the test plan will be used to run the test project.

Learning Objective:

The objective of this workshop is to develop the participants’ skills in developing effective test strategies and writing system test plans, including how to get organized and ready for testing projects.

Audience:

Software testers & quality assurance professionals, systems analysts and designers, software engineers and programmers, project leaders, auditors, and users who are involved in systems testing.

Books and Software Tools that Accompany this Workshop:

Participants receive software tools, and four books in printed form or on a CD:

• “Software Test Planning”, by Ross Collard, published 2007 (approximately 400 pages).
• “Software Testing & QA: Glossary of Terms and Concepts” by Ross Collard (approx. 75 pages).

The Case Study:

The case study in this workshop begins:

Congratulations! (Or condolences.) You have just been appointed the test team leader for a new, mission-critical project for your organization. The application uses client/server, web, database, network and wireless technologies. It contains a mix of application software and utilities like middleware. Some software is newly developed in-house, some was acquired from third parties and modified to fit.”

“The senior managers are confident you will do a good testing job, and have asked you to give them a presentation on how you will proceed. Their expectations are known to be high – a well reasoned, fairly thorough test plan that you are able to defend under rapid–fire questioning. They expect your resource, timeline and budget estimates to be credible. Your presentation is scheduled for next week. What will you tell them???

Topics:

You will learn how to:

1. Develop the overall test strategy and approach.

• Review the business context and assess its consequences for testing.
• Review the technical infrastructure and assess its testing implications.
• Gauge the capabilities of the available test resources – people, equipment and facilities, tools, etc.
• Obtain or develop testable quality criteria (e.g., from quality gateways).
• Justify why your test planning is important, worth the attention and effort.
• Examine the major issues in developing a test plan.
• For each major challenge in test planning, assess what solutions are available and how to apply them.
• Make trade-offs (e.g., how much testing is enough?) and set priorities in testing.

2. Prepare test plans to fit the needs.

• Define the test project objectives, focus, scope and approach.
• Use checklists and templates for test planning.
• Perform risk assessments and use them in test planning.
• Identify feasible alternatives in testing, and their consequences, e.g.:
  
  • Exploratory vs. scripted testing.
  • Agile testing vs. waterfall.
  • Automated vs. manual, etc., etc.
• Formulate a test automation strategy: the automation framework, tools, facilities, skills, test case repositories, etc.
• Follow a methodology (a step-by-step process) to develop a test plan.
• Since methodologies are necessarily generic, tune the methodology to fit
  
  • the particular project’s context.
• Develop a test project work plan, budget and schedule, including contingencies.
• Select the structure and format of the test planning documents.
• Document the test plan.
• Review and critique test plans.

3. Prepare the transition from planning to action (testing).

• Organize and orient the test team.
• Prepare the testware:
• The test environment.
• Automated and manual test case libraries.
• Test processes.
• Test artifacts (e.g., a problem reporting form).
• Prepare the test environment – trial run or dress rehearsal.
• Set up the test facilities.
• Check them out, part 1 (e.g., are networks configured correctly?).
• Load application software, test tools and libraries.
• Check them out, part 2 (e.g., are the right versions loaded of the software being tested; does it run well enough to test?)
• Select or develop the test runs.
• Determine if test cases need to be grouped into test suites and test runs,

And if so how.

• Review test cases for possible misleading or false results.
• Determine test case run sequences, interdependencies, deliberate overlap to test interference among subsystems, etc.
• Prepare instructions for the testing sessions, as necessary.
• Decide how to evaluate test results and follow-up.
• Develop or choose test oracles.
• Review the adequacy of support processes:
• Problem reporting, defect tracking.
• Version control, software builds, etc.
• Maintain the test plan.
• Plan for the re-use of test plans.

Approach:

This workshop shows the participants how to:

• Analyze test situations.
• Identify and understand the key issues the test planner needs to master in developing the test plan.
• Develop workable test plans with a reasonable assurance of test coverage and reliability.

In small groups, participants will work on a series of real-life test planning scenarios. For each scenario, we will go through four steps: (a) the instructor explains the techniques that are most likely to apply and introduces the exercise, (b) each group develops its solution, (c) the group reviews and critiques a suggested model answer, and (d) the entire class compares results and discusses the issues from the exercise.

This workshop is very hands-on and practical, and gives the participants test planning skills, which they can apply immediately.

Exercises:

The core of the workshop is writing a draft of a test plan in a series of exercises.

Because of time limitations, not all these exercises and topics listed below may be addressed in a particular class or, if addressed, they may not all be examined in depth.

The titles of these exercises (in a typical workshop, not all may be included in yours), are as follows:

1. Exercise. Understanding the Situation and Context

• (1) What are the major business issues?
• (2) What are the technical issues?
• (3) What are the testing issues?

2. Exercise. Constituencies, Test Objectives and Approach

• (1) Who are the constituencies or groups who have a vested interest in the success of the testing project?
• (2) What are the success factors for each major constituency, i.e., what is needed to satisfy them?
• (3) Based on the success factors which you have identified, what should be the objectives for this testing project?
• (4) What overall approach will be utilized on this testing project, and what are the guiding principles and underlying assumptions for this approach?

3. Exercise. Test Scope and Priorities

• (1) What types of testing should be included within the scope of the project?
• (2) What are the quality priorities for this system?
• (3) If you cannot answer the above questions, what information and decision-making do you need in order to answer the questions?

4. Exercise. Risk Strategy

• (1) What are the major risks, if any, which may be encountered with this new system?
• (2) What should be the focus of the testing efforts, based on the perceived risks?
• (3) Which areas, if any, do not need to be tested or need only minimal testing?

5. Exercise. Test Entry & Exit Criteria

• (1) What criteria should be met in order to begin testing?
• (2) What criteria should be met in order to finish testing, i.e., to move the system from the testing process to live operation?

6. Exercise. Test Coverage, Test Case Design and Results Evaluation Techniques

• (1) How much testing is enough? What should be the targets for test coverage? How should these be measured and assessed?
• (2) What techniques will be used, to
  
  • i. Identify the conditions to be tested?
  • ii. Develop the test cases?
  • iii. Review and validate the test cases?
• (3) What methods will be used to evaluate the test results (test case pass/fail criteria)?
• (4) What methods will be used to avoid false test results (i.e., false positive results and false negative results)?

7. Exercise. Test Automation Strategy

• (1) What parts of the testing should be automated, using which tools, and what is involved in the automation effort?

8. Exercise. Test Resources

• (1) What kinds of resources are needed for this testing project?
  
  • i. People.
  • ii. Test facilities, equipment and tools.
  • iii. Processes.
• (2) What should be the roles and responsibilities of the various people who will be involved in this test project?

9. Exercise. The Test Project Work Plan

• (1) What are the major work activities that must be undertaken in order to test the system?
  
  • i. Prior to the test execution (i.e., in the test planning and preparation).
  • ii. In the test execution and follow-up to the test execution.
• (2) What are the interdependencies among the work tasks?

10. Exercise. Contingencies

• (1) What major contingencies might occur, which and thus should be planned for, specifically in regard to how they affect the testing project?
• (2) What actions should be taken, if any, to minimize that impact, if one or more of these contingencies do in fact occur?

11. Exercise. Planning for Maintenance Changes and Regression Testing

• (1) How can flexibility be designed into the test plan to accommodate possible future changes?

12. Exercise. Estimating the Test Effort and Schedule

• (1) What resources and time frame are needed to test the system?

13. Exercise. Test Plan Format and Organization

• (1) What is a suitable format and organization for the test plan?

14. Exercise. Draft of the Test Plan

• (1) How do we develop a first draft of the test plan?
• (2) How are the executive summary and presentation prepared?

15. Exercise: Updating the Test Plan

• (1) How do we adapt the test plan as conditions change?

16. Exercise 15: Reviewing the Test Plan

• (1) How do we review the test plan and ensure it is workable?

Books and Tools that Accompany this Workshop:

Participants receive software tools, and two or more books in printed form or on a CD:

• “Software Test Planning” by Ross Collard, published 2007, approximately 300 pages.
• “Software Test Estimating” by Ross Collard, published 2007, approximately 300 pages.
• “Software Testing & Quality Assurance Fundamentals” by Ross Collard, published 2007, approximately 400 pages.
• "Software Testing & QA: Glossary of Terms and Concepts” by Ross Collard, published 2007, approximately 200 pages.