 |
|
Browse a few books on software testing and you'll get the message that software testing is impossible. It is impossible to test completely. Changes are inevitable, and changes add bugs. Software is complex and its complexity is an essential quality, "not an accidental one," in Fred Brooks's phrase. When someone says their software is tested, you wonder what the meaning of "is" is.
Testing Computer Software documents the development and testing of commercial software in the real world of Silicon Valley. If Silicon Valley is the only real world you've ever known, you won't perceive this focus as the book's main weakness.
Many development books assume you have a set of requirements your customer has signed off on, followed by a specification, followed by some species of design and coding. In Silicon Valley, you don't have a list of requirements to sign off on, or a customer to do the signing. Software is created by small teams, all of whom are deeply committed to the product and whose every move is designed to improve it (see excerpt from the Preface).
This book began (in 1983) as a series of notes made by the authors to help with training new testers. These beginnings have led to a familiar, clear approach and to a density of material that is indicative of wide experience. There are sections on many topics in testing and development: testing user manuals, how changes in the software affect the writer and the tester, which regression tests to run and when, localization of software, and writing test planning documents that are readable by experienced and inexperienced testers.
The early chapters discuss basic black box testing and test execution and later chapters add information on test planning -- the later chapters would be useful to people planning and directing testing, especially chapter 13, "Tying it together," and chapter 15 "Managing a testing group."
The authors see testing partly as the creation of documents -- lists of features to test, breakdowns of software functionality, tables of devices (mouses, printers) versus device modes that need testing. Since you can't count on a specification (or help at all) from the developers, creating these lists and tables resembles an archaeological expedition. A black box approach is assumed throughout.
As a practical book, the focus is on answering questions such as: How can you remain productive when there is no code to test yet? What tests do you perform right before release, and how do you negotiate with management when management wants to ship the product with serious bugs remaining? (At release, it is assumed parts of the software are still sick; you apply a triage to the parts that are dying.)
The practical focus of the book leads to two weaknesses, however. Testing Computer Software accepts the status quo uncritically, condoning practices like design by prototype and dropping random changes into software without warning. In addition, the frequent asides offering advice to the individual tester invariably show the tester as powerless. These sections can be summarized as follows: "Do a good job in a difficult situation but be careful about questioning the 'way things are.' Don't try to change anything. Especially, although programmers can be difficult, even patently wrong or vindictive, never do battle with one because you will lose."
Mr. Kaner, in addition to his years testing, has a Ph.D. in psychology and a law degree, which gives credence to chapter 14 on the legal ramifications of bad software. Mr. Falk and Mr. Nguyen are experienced testers and managers. As is usual in a book with more than one author, it's occasionally unclear who is doing the talking.
Grady Booch, in Object Magazine (October 1995), describes a conference he attended in which a speaker described certain aspects of Microsoft's software process; their testing buddy system and automated regression testing, for example. The speaker noted that Microsoft makes a ton of money, and concluded that Microsoft's development paradigm is the best. Mr. Booch and other attendees found this logic to be bogus.
Similarly, not all companies will want to adopt the development methods implicit in this book. If you can overcome your disappointment that real world software isn't always developed the way it should be, the book provides much broad and useful information.
-- Doug Nickerson
"In the real world, product changes are made late in development. When you're developing a product for public sale, your customers -- your potential customers -- did not agree to any specification. If a competitor creates something more appealing, you have to respond quickly. Desirable features are frequently dropped because there is not time to implement them. Code rewrites are also set aside, even if they're needed to bring a fragile first working version to a level the programmer considers professional. A conscientious programmer might take the initiative and do the job anyway, 'on the sly.' Late in the project he may drop a significant change into the package without warning. These efforts go beyond the call of duty and the 40-hour week. They may improve the product tremendously or destabilize it badly, probably both for a time. Whatever the result, people take personal initiative to improve the product.
"There are always late changes. The goal is to cope with them -- they are needed... If you can't keep up with them as a tester, you have a problem, But the solution won't come from complaining about them or in trying to stop them."
"Evolutionary Development of Test Materials
"Traditional software development books say that ‘real development teams’ follow the waterfall method. Under the waterfall, one works in phases, from requirements analysis to various types of design and specification, to coding, final testing, and release.
"In software design and development as a whole, there are very serious problems with the waterfall method. For details, see Tom Gilb's excellent book (Principles of Software Engineering Management, Addison-Wesley, 1988) and his references. (See also Gould & Lewis, 1985, and Chapter 11 of Baecker & Buxton, 1987.)
"As an alternative, Gilb says to deliver a small piece, test it, fix it, get to like it eventually, then add another small piece that adds significant functionality. Test that as a system. Then add the next piece and see what it does to the system... Over time, the product evolves into a rich, reliable, useful product. This is the evolutionary method."
"Regression testing
"After you've thoroughly tested an area of the program, you must retest it regularly. There will be new problems and old ones will reappear. The goal of regression testing is to provide coverage comparable to the focused work but without the cost in time.
"Your regression test series always includes tests of recent bug fixes. However, these particular regression tests come and go. You'll use most only once or twice throughout testing. Along with these retests is a core regression test suite.... [Your regression suite] should include the most interesting or useful retests of fixed bugs and the best other tests run so far. Add them to your test notes during mainstream and guerrilla testing. Add more tests while doing the more detailed test planning. Spend up to half of that time creating tests that you'll want to use again."
"Rating the Reliability of the Product
"Once you finish pre-final testing, the product will either by mastered and shipped or it will leave your hands for final acceptance testing by someone else, perhaps by some other group, such as customer support.
"Before the product leaves, you will be asked to evaluate the quality of the program. Is it ready for release? Your opinion may be ignored, but it will be solicited.
"The quality of a product is its fitness for use. The product's design, functional capabilities, usability, and reliability all contribute to the product quality. Don't get caught up in this when management asks you for a rating of the program's quality at the end of the project. They don't want a rehash of the design issues-all they want (probably) is information about the program's reliability. When asked for pre-release quality rating, provide pre-release reliability ratings, possibly supplemented by some design comments."
1. An example test series
2. The objectives and limits of testing
3. Test types and their place in the software development process
4. Software errors
5. Reporting and analyzing bugs
6. The problem tracking system
7. Test case design
8. Testing printers (and other devices)
9. Localization testing
10. Testing user manuals
11. Testing tools
12. Test planning and test documentation
13. Tying it together
14. Legal consequences of defective software
15. Managing a testing group
Appendix: Common software errors
References
Index
| Readability |
|
| Originality |
|
| Organization |
|
| Accuracy |
|
| Consistency |
|
| Depth |
|
| Timeliness |
|
| Editing |
|
| Design |
|
| Overall Value |
|
Explanation of ERCB rating scale: No stars = unacceptable, 1 Star = marginal, 2 Stars = average, 3 Stars = above average, 4 Stars = exceptional.