Test Dashboard (CMMI)
By using the Test dashboard, you can monitor test activities, report on progress, find gaps in test coverage, and identify test areas that may require additional investigation. This dashboard displays five reports that provide information about testing that has occurred in the most recent four weeks.
You access dashboards through your team project portal. You can access the Test dashboard only if that portal has been enabled and is provisioned to use Microsoft Office SharePoint Server 2007. For more information, see Dashboards (Agile) or Access a Team Project Portal and Process Guidance.
In this topic
You can use this dashboard to answer the following questions:
To view the dashboard, you must be assigned or belong to a group that has been assigned the Read permission in SharePoint Products for the team project. To modify, copy, or customize a dashboard, you must be assigned or belong to a group that has been assigned the Members permission in SharePoint Products for the team project. For more information, see Add Users to Team Projects.
To modify a report in Office Excel, you must be a member of the TfsWarehouseDataReaders security role in SQL Server Analysis Services and you must be assigned or belong to a group that has been assigned the Members permission in SharePoint Products for the team project. For more information, see Grant Access to the Databases of the Data Warehouse for Visual Studio ALM.
To view a work item, you must be a member of the Readers group or your View work items in this node permission must be set to Allow. To create or modify a work item, you must be a member of the Contributors group or your Edit work items in this node permission must be set to Allow. For more information, see Managing Permissions.
You can use the Test dashboard to understand how well the team is progressing in testing the requirements. Specifically, this dashboard displays the Web parts that the following illustration shows and the following table describes.
The Test Plan Progress, Test Case Readiness, Requirement Test Status, and Test Activity reports are available only when the team creates test plans and runs tests by using Test Runner and Microsoft Test Manager. For information about how to define test suites and test plans, see Organizing Test Cases Using Test Suites.
Burndown, progress, trend charts, and reports through do not appear when the server that hosts Analysis Services for the team project is not available.
Stacked area graph of the test results of all test grouped into their most recent recorded outcome during the past four weeks. Outcomes include Never Run, Blocked, Failed, and Passed.
Stacked area graph that shows how many test cases have been in the Design or Ready state for the most recent four weeks.
Horizontal bar chart that shows the count of test results for each combination of test case and test configuration that is defined for each requirement. The chart groups the test results according to their most recent test run, where the options are Passed (green), Failed (red), Blocked (purple), or Not Run (gray).
Line chart that shows the cumulative count of all results run for all manual test cases during the most recent four weeks.
Stacked area graph that shows the cumulative count of all failed outcome results for tests, sorted by failure type, during the most recent four weeks. Failure types include Regression, New Issue, and Known Issue.
List of upcoming events. This list is derived from a SharePoint Web part.
Count of active, resolved, and closed work items. You can open the list of work items by clicking each number. This list is derived from a Team Web Access Web part.
List of recent builds and their build status. You can view more details by clicking a specific build. This list is derived from a Team Web Access Web part.
: Build in Progress
: Build Not Started
: Build Succeeded
: Build Failed
: Build Stopped
: Build Partially Succeeded
List of the most recent check-ins. You can view more details by clicking a specific check-in. This list is derived from a Team Web Access Web part.
For the reports in the Test dashboard to be useful and accurate, the team must perform the following activities:
Define test cases and requirements, and create Tested By links from test cases to requirements.
Define test plans, and assign test cases to test plans.
For more information, see Defining Your Testing Effort Using Test Plans.
For manual tests, mark the results of each validation step in the test case as passed or failed.
Testers must mark a test step with a status if it is a validation test step. The overall result for a test reflects the status of all the test steps that the tester marked. Therefore, the test will have a status of failed if the tester marked any test step as failed or did not mark it.
For automated tests, each test is automatically marked as passed or failed.
(Optional) To support filtering, assign Iteration and Area paths to each test case.
You can use the first three reports in the Test Dashboard to monitor test progress and answer the questions in the following table:
Test Case Readiness
Test Plan Progress
Requirement Test Status
You can use the Requirement Test Status report to determine whether tests are covering all the code and to answer the following questions:
Which requirements have a low overall count of test cases?
Which requirements have a high overall count of test cases that are blocked or have never been run?
Does the test case coverage for each requirement meet expectations?
Which requirements have a high rate of test failures?
What is the average number of test cases that are defined for each requirement?
By monitoring test failures, you can identify and address problems in the code early. You can use the last two reports in the Test dashboard to gain better insight into the number of tests that are failing.
Manual Test Activity
The Manual Test Activity report indicates the results for each Test Case run for each test configuration and for all test plans. Spikes that might occur may be early indicators of problems in either the test activity or the quality of code that the team is checking in.
You might want to check the metrics for recent builds, bug status, and code churn to determine whether any of them can help explain the changes.
Test Failure Analysis
A healthy Test Failure Analysis report shows moderate numbers of new issues, known issues, and regressions. If any spikes occur in these areas, the team might need to investigate further. Spikes may indicate problems in either the test activity or the quality of code that the team is checking in.
Also, you might want to check the metrics for recent builds, bug status, and code churn to determine whether any of them can help explain the changes.
You can customize the Test dashboard in the following ways:
Change the filters of each report in Office Excel to focus on specific product areas or iterations.
Filter the Manual Test Activity report in Office Excel for specific test plans or on Test Cases that are either manual or automated.
Add existing Excel reports such as Bug Status, Code Churn, and Code Coverage to the dashboard.
Create and add reports in Office Excel that show progress by specific members of the team. For an example, see Bugs by Assignment Excel Report.
For more information about how to work with and customize reports in Office Excel, see the following pages on the Microsoft Web site: