Guidelines for Smoke Testing
In software, the term smoke testing describes the process of validating code changes before the changes are checked into the product’s source tree. After code reviews, smoke testing is the most cost effective method for identifying and fixing defects in software. Smoke tests are designed to confirm that changes in the code function as expected and do not destabilize an entire build.
Note
The term smoke testing originated in the hardware industry. The term derived from this practice: After a piece of hardware or a hardware component was changed or repaired, the equipment was simply powered up. If there was no smoke, the component passed the test.
The following guidelines describe best practices for smoke testing. The effects of following the guidelines will vary widely, ranging from improving communication among team members to developing specific ways to use testing and debugging tools.
Work with the Developer
Because smoke testing focuses on changed code, you must work with the developer who wrote the code. You will have to understand:
What changed in the code. To understand the change, you will also have to understand the technology used; the developer can help explain it.
How the change affects the functionality.
How the change affects the interdependencies of various components.
Conduct a Code Review Before Smoke Testing
Before you run a smoke test, conduct a code review that focuses on any changes in the code. Code reviews are the most effective and efficient method to validate code quality and ensure against code defects and faults of commission. Smoke tests ensure that the primary critical or weak area identified either by code review or risk assessment is primarily validated, because if it fails the testing cannot continue.
Install Private Binaries on a Clean Debug Build
Because a smoke test must focus on validating only the functional changes in updated binaries, the test must run on a clean test environment by using the debug binaries for the files being tested.
Note
Testing with mismatched binaries is a common mistake in smoke testing. To avoid this mistake, when there is a dependency between two or more updated binaries, include all the updated binaries in the test build. Otherwise, the results of the test might not be valid.
Create Daily Builds
Daily builds require the team members to work together and encourage the developers to stay in sync. If iterations of new builds are delayed, the delay can easily cause products with multiple dependencies to get out of sync. Following a process of building daily and smoke testing any changed or new binaries ensures high quality.
For more information about setting up recurring builds, see Running Builds in Team Foundation Build. For more information about verifying your product’s builds, see How to: Configure and Run Build Verification Tests (BVTs).
Note
Make a quality daily build the team’s top priority. If the build is broken because a code check-in has not been smoke-tested, require the developer and tester to stop all other work until the problem is resolved. The penalty for breaking the build should not be harsh, but the penalty should emphasize that a proper daily build is the team’s top priority.
You need not perform exhaustive tests. The purpose of smoke testing is not to ensure that the binary is 100% error-free. This would require too much time. You perform smoke tests to validate the build at a high level. You want to ensure that changes in a binary do not destabilize the general build or cause catastrophic errors in functionality.
Web and Load Testing
When you build your Web tests and load tests, it is a good practice to run a smoke test before running any long, heavy test. In Web and in load testing, smoke testing is short, light test. You use a smoke test to validate that everything is correctly configured and running as expected before running your tests for performance or stress testing.
See Also
Tasks
How to: Configure and Run Build Verification Tests (BVTs)