Software Test Estimating
Course Title: Software Test Estimating
Course#: STR12
Duration: 2 or 3 days, depending on the topic coverage and the number of exercises.
Overview:
This workshop is for you -- if you are interested in estimating the testing efforts earlier in projects, despite limited information of unknown quality, and in delivering more accurate and more credible estimates with less anxiety and heartburn.
Testing is a major part of almost all software projects – it often consumes 25% to 40% of the software development and maintenance budgets, but only catches about 80% to 90% of the bugs. In other words, most testing is far from perfect.
We could argue that effective testing is vital to organizations, and developing effective estimates is an essential survival skill for testing & QA professionals and their managers. Nevertheless, most authoritative surveys of testing practices find that test estimation is poor.
Poor estimates can lead to last-minute “work crunches” of testing, debugging, defect fixing, and re-testing. These crunches encourage skimpy test coverage, sloppy testing by exhausted testers, compromised quality checkpoints, increased tensions, and poor communications between testers and others. The true cost of poor estimates is surprisingly high.
The test and quality compromises needed to meet a tight deadline, as decreed by unrealistic estimating, can introduce hidden new risks. Though intangible, these risks are still real. Since intangibles by definition are difficult to quantify, the risks and costs of poor testing tend to be denied or under-estimated. Characteristics like morale, enthusiasm and working conditions affect productivity and thus estimates, but are impractical to quantify and measure. These intangibles certainly add to the uncertainties, and may contribute to a “can’t win” atmosphere.
In this seminar, we will examine examples of what is wrong with most test estimating today. (There’s plenty.) We will also review success stories of good estimating in some depth, absorb the lessons learned and discuss how we can apply them in our own work.
Learning Objectives:
We expect the workshop to significantly sharpen your estimates, deepen your understanding of testing and provide useful insights about testers. You will recognize behaviors that tend to lead to poor estimates, and know how to control them. Through extensive exercises and peer critiques in class, you will master practical estimating techniques, applying them in a variety of context.
Specifically, you will:
Because of time limitations, the workshop normally focuses on feature testing in software applications. The workshop does address specialized types of software testing such as load testing, nor hardware, network, database testing. Testing both new software and modifications to software are discussed, both manual and automated testing, and both traditional and iterative RAD development methodologies.
No one particular application domain (e.g., developing bus routes and schedules), test environment (e.g., Unix), tool set (e.g., WinRunner), or technical background (e.g., database administrator or network engineer), is assumed.
Testing occurs in many contexts, and the estimates are highly context-dependant. We encourage the participants to bring their relevant artifacts (i.e., list of current estimating needs, methods and styles, estimate calculations and work papers) to class.
Audience:
Project leaders, system test leads & quality assurance professionals, and users who are involved in managing systems testing projects.
Participants include non-technical professionals in fields like health care, telecom, banking, government and retailing, to name a few, who are system users involved in testing systems, and marketing people in the vendors of software and software dependent products. (A little knowledge may be dangerous, of course. If a person never has been involved in a test project, the rules of thumb in this book are better than nothing. We encourage that person to attend class and also seek the counsel of experienced testers.
Books and Software Tools that Accompany this Workshop:
Participants receive software tools, and two books in printed form or on a CD:
Combined Test Planning and Estimating Class:
Many participants in the estimating seminars opt to attend our test planning class also, usually before they attend the estimating class. Depending on the version, the test planning class runs from 2 to 4 days. Some prefer to attend the two classes back to back in the same week. Others prefer to spread them over a period of a few weeks or a few months.
Prerequisites:
This workshop does not assume any specific technical background, such as knowledge of Unix or TCP/IP. It does assume some familiarity with information systems generally, such as a basic idea of what a system, system development project, network and database are, and a basic understanding of software feature testing.
Topics:
This workshop provides a practical, hands-on approach to improve your estimating and negotiating skills. Testing both new software and modifications to software will be discussed, both manual and automated testing, and both traditional and iterative RAD development methodologies.
This dialog concerns a time-honored estimation technique (adding a huge factor of safety to a dubious estimate):
Let’s face it: developing realistic and credible estimates is a critical survival skill for test professionals and managers.
This workshop provides a practical, hands-on approach to improve your estimating and negotiating skills.
Because of time limitations, not all these topics listed below may be addressed in a particular class or, if addressed, they may not all be examined in depth.
Books and Tools that Accompany this Workshop:
Participants receive software tools, and two or more books in printed form or on a CD:
Course#: STR12
Duration: 2 or 3 days, depending on the topic coverage and the number of exercises.
Overview:
This workshop is for you -- if you are interested in estimating the testing efforts earlier in projects, despite limited information of unknown quality, and in delivering more accurate and more credible estimates with less anxiety and heartburn.
Testing is a major part of almost all software projects – it often consumes 25% to 40% of the software development and maintenance budgets, but only catches about 80% to 90% of the bugs. In other words, most testing is far from perfect.
We could argue that effective testing is vital to organizations, and developing effective estimates is an essential survival skill for testing & QA professionals and their managers. Nevertheless, most authoritative surveys of testing practices find that test estimation is poor.
Poor estimates can lead to last-minute “work crunches” of testing, debugging, defect fixing, and re-testing. These crunches encourage skimpy test coverage, sloppy testing by exhausted testers, compromised quality checkpoints, increased tensions, and poor communications between testers and others. The true cost of poor estimates is surprisingly high.
The test and quality compromises needed to meet a tight deadline, as decreed by unrealistic estimating, can introduce hidden new risks. Though intangible, these risks are still real. Since intangibles by definition are difficult to quantify, the risks and costs of poor testing tend to be denied or under-estimated. Characteristics like morale, enthusiasm and working conditions affect productivity and thus estimates, but are impractical to quantify and measure. These intangibles certainly add to the uncertainties, and may contribute to a “can’t win” atmosphere.
In this seminar, we will examine examples of what is wrong with most test estimating today. (There’s plenty.) We will also review success stories of good estimating in some depth, absorb the lessons learned and discuss how we can apply them in our own work.
Learning Objectives:
We expect the workshop to significantly sharpen your estimates, deepen your understanding of testing and provide useful insights about testers. You will recognize behaviors that tend to lead to poor estimates, and know how to control them. Through extensive exercises and peer critiques in class, you will master practical estimating techniques, applying them in a variety of context.
Specifically, you will:
- Learn how to develop realistic, justifiable estimates for software testing projects.
- Review methods and rules of thumb for estimating the time, resources and budgets needed for testing projects.
- Adapt methods and rules of thumb to fit your own context.
- Explore how to develop new rules of thumb for your own testing projects.
- In exercises, work through practical examples of how to develop test project estimates, and critique and validate your own results.
- In small teams, compare and critique others’ estimating approaches and results.
- After the class, transfer and use this knowledge productively back on the job.
Because of time limitations, the workshop normally focuses on feature testing in software applications. The workshop does address specialized types of software testing such as load testing, nor hardware, network, database testing. Testing both new software and modifications to software are discussed, both manual and automated testing, and both traditional and iterative RAD development methodologies.
No one particular application domain (e.g., developing bus routes and schedules), test environment (e.g., Unix), tool set (e.g., WinRunner), or technical background (e.g., database administrator or network engineer), is assumed.
Testing occurs in many contexts, and the estimates are highly context-dependant. We encourage the participants to bring their relevant artifacts (i.e., list of current estimating needs, methods and styles, estimate calculations and work papers) to class.
Audience:
Project leaders, system test leads & quality assurance professionals, and users who are involved in managing systems testing projects.
Participants include non-technical professionals in fields like health care, telecom, banking, government and retailing, to name a few, who are system users involved in testing systems, and marketing people in the vendors of software and software dependent products. (A little knowledge may be dangerous, of course. If a person never has been involved in a test project, the rules of thumb in this book are better than nothing. We encourage that person to attend class and also seek the counsel of experienced testers.
Books and Software Tools that Accompany this Workshop:
Participants receive software tools, and two books in printed form or on a CD:
- “Reliably Estimating Software Testing”, by Ross Collard, published 2007 (approximately 400 pages).
- “Software Testing & QA: Glossary of Terms and Concepts” by Ross Collard (approx. 75 pages).
Combined Test Planning and Estimating Class:
Many participants in the estimating seminars opt to attend our test planning class also, usually before they attend the estimating class. Depending on the version, the test planning class runs from 2 to 4 days. Some prefer to attend the two classes back to back in the same week. Others prefer to spread them over a period of a few weeks or a few months.
Prerequisites:
This workshop does not assume any specific technical background, such as knowledge of Unix or TCP/IP. It does assume some familiarity with information systems generally, such as a basic idea of what a system, system development project, network and database are, and a basic understanding of software feature testing.
Topics:
This workshop provides a practical, hands-on approach to improve your estimating and negotiating skills. Testing both new software and modifications to software will be discussed, both manual and automated testing, and both traditional and iterative RAD development methodologies.
This dialog concerns a time-honored estimation technique (adding a huge factor of safety to a dubious estimate):
- Question: When will the system testing be completed???
(Note #1: At the time the question is asked, you do not know (a) the final scope of the functionality, (b) when the developers will be done, and (c) what test resources you will have available.)
(Note #2: The boss wants a definitive answer within two minutes anyway.) - Answer: Take a wild guess and multiply by two.
- Question: What do you do when the boss cuts your agreed-on test duration by 85%??
- Answer: Whine pathetically that you thought he or she really understood that “quality is important”.
Let’s face it: developing realistic and credible estimates is a critical survival skill for test professionals and managers.
This workshop provides a practical, hands-on approach to improve your estimating and negotiating skills.
Because of time limitations, not all these topics listed below may be addressed in a particular class or, if addressed, they may not all be examined in depth.
- Introduction
Welcoming Comments and Introductions
Seminar Objectives and Style
Brief Overview of Topics
Class Logistics and Schedule
Self-Assessment Quiz - The Nature of Test Estimation
The Nature of Testing
The Nature of Estimation
Combining Testing and Estimation - Estimating the Test Effort
Overview of Estimating Techniques
Estimating -- Art or Science?
Which Estimating Technique is Best for Me?
Attainable Levels of Accuracy
The Importance of the First Estimate
Agile or Rapid Application Development (RAD) and Test Estimation - What you Need to Know to Estimate
Data Sources
Data Validation
Evolving Data
Guideline for the Estimating Process
Components of a Complete Estimate - Assumptions and Biases
Recognizing Assumptions
Recognizing Biases - Making and Checking Assumptions
Communicating Assumptions & Uncertainties
Time Needed for Developing Estimates
Getting Better Data Earlier - Twenty-Plus Estimating Techniques
Bottom-Up or Micro-Estimating
Top-Down or Decomposition Estimating
Ratios
Largest Components First
Adjustment Factors
Simulation Models
Comparisons to History
Formulae
- Application
- Calibration
- Validation
SWAGs - Examples
- Estimating Heuristics
Rules of Thumb
Adapting the Rules of Thumb
Estimating Myths and Voodoo Science - The Relationship of Estimating and Project Work Planning
Project Checklists and Estimating Worksheets
Allowing for Test Overheads
The Mythical Person-Month
Relative Productivity of Professionals
Adjusting for the Learning Curve - The Basics of a Test Estimating Formula
Estimating Adjustment Factors
Setting the Factors of Safety
Common Reasons for Test Slippage
Exercise: Developing the Ball Park Estimate - Estimating Formulae
Estimating the Number of Test Cycles
Estimating the Number of Test Cases - Estimating the Number of Problems
Estimating Rules for –
- Software Development
- Software Maintenance
- Software Package Installation
- Software Package Upgrade
- Hardware / Software Co-Development
- Etc.
- Estimating by Project Phase and Activity
Test Planning
Test Case and Test Data Development
Test Automation
Test Prep. and Set-Up
Test Execution
Test Results Evaluation and Problem Reporting
Debugging & Fixing
Re-Testing after Fixes
Test Project Management
The Ratio Approach
Software Packages
Resource Allocation
Specialized Types of Testing - Test Project Staffing
Estimating the Headcounts
Estimating Peaks and Flow of Demand
Varying Rates of Productivity
Test Cases Developed and Run
Problems Reported - Estimating the Test Automation Effort
Test Automation Frameworks
Tool Taxonomy
Quick Review of the Main Types of Tools
Tool Selection and Acquisition
Tool Installation and Training
Tool Upgrades
Tool Adaptation or Building
Test Case Automation
Maintenance of Automated Tests
Test Case Re-Use - Managing the Estimating Process
How the Estimating Software Tools Work
Using Early Feedback to Adjust Estimates
Building Your Own History Database
The Dangers of Anecdotal Memory
Checking the Reasonableness of Assumptions
Deciding when to Re-Estimate
The Basic Problems with Forecasting - Regression Testing
Estimating the Modification & Regression Test Effort
Cost/Benefit Analysis for Regression Testing - Checking Estimates
How Human Behavior affects Estimates
Behaviors that Lead to Poor Estimates
The Introduction of Biases
Democratization of Estimates
Estimating and Competitive Bidding
Test Estimate Review Checklist - Quality Metrics
The Goal of Metrics
Hard Data Vs. Educated Guesses
Payoffs from Metrics
Prescriptive Vs. Descriptive Metrics
Predictive Metrics
Determining What to Measure - Software Sizing
The Nature of Sizing
- Why Sizing is Difficult and Imperfect
- Why it is Still Important to Try
The Function Point Technique
Adjustments for Complexity
Complexity Measures
Other Sizing Methods and Measures - Types of Quality Metrics
Test Process Metrics
Product Quality Metrics
Customer Service Metrics
Operational Metrics
Goal-Oriented Metrics - Data Gathering Considerations
Precision Needed in Metrics
Statistical Methods
Sample Sizes and Sampling Methods
Apples and Oranges
Data Cleansing
Curve Fitting
Why Six Sigma does Not Work for Software - Metrics Starter Kit
Metrics Examples
Cost of Quality (CoQ)
Quality Norms & Statistics
Customer Service Norms
Ranking & Use of Quality Metrics
Limitations of Metrics
Practices Needed for Effective Measurements
The Quality Payback Report
An Example: Assessing the Effectiveness of Test Techniques
Etc.
Comparison of Techniques
Applying Techniques
Books and Tools that Accompany this Workshop:
Participants receive software tools, and two or more books in printed form or on a CD:
- “Software Test Planning” by Ross Collard, published 2007, approximately 300 pages.
- “Software Test Estimating” by Ross Collard, published 2007, approximately 300 pages.
- “Software Testing & Quality Assurance Fundamentals” by Ross Collard, published 2007, approximately 400 pages.
- “Software Testing & QA: Glossary of Terms and Concepts” by Ross Collard, published 2007, approximately 200 pages.
Clients
