Share via


Testing Windows Store apps Running on a Device Using the Exploratory Test Window

You can do exploratory testing on your Windows Store apps that are running on a Windows 8 remote device such as a Windows 8 tablet.

Exploratory testing on a remote device

Requirements

  • Visual Studio Ultimate, Visual Studio Premium, Visual Studio Test Professional

Why should I do this?

By connecting to your Windows 8 remote device to test your Windows Store app, you can identify problems that users will run into when they use their devices in real-world situations. The problems that are typically identified when you run exploratory tests include incorrect functionality or bugs, poor performance, and security and connectivity issues. The outcome, resolutions, and comments that you can enter during an exploratory test are saved as test results. To test your other apps, you do not need to connect to a Windows 8 remote device. You can test the application on the same machine running Microsoft Test Manager. For more information about performing exploratory testing applications that are not Windows Store apps, see the topic: Performing Exploratory Testing Using Microsoft Test Manager.

How do I do this?

Step 1: Prepare the Windows 8 remote device for testing

Before you can run exploratory tests on your Windows 8 remote device, you must install the Remote Debugger on the device that you want to test. The Remote Debugger includes the Microsoft Test Tools Adapter, which is automatically installed and runs as a service on the remote device to enable testing. The remote debugger setup programs are available on the Visual Studio installation media in subfolders of the vs\Remote Debugger folder. If you want to see more information about the remote debugger before you try to install it on your Windows 8 remote device, see the topic: Installing the Remote Debugger.

Step 2: Connect to your remote Windows 8 device from Microsoft Test Manager

In this step, use the following illustrations as a guide for the steps that are required in order to connect to your remote Windows 8 device.

Connect to your remote Windows 8 device:

Connecting to remote device

You will need to enter your credentials when prompted:

Enter credentials

Step 3: Install you Windows Store app onto the remote device

Now that you’re connected to the remote device, you can install your Windows Store app onto your remote device for testing. Follow the steps in the illustrations below.

Installing your Windows Store app on the remote device:

Install Metro style app on remote device

Install your Windows developer license when prompted:

Install license

Install a testing certificate when prompted:

Install certificate

Step 4: Test your Windows Store app!

You should now be able to test your Windows Store app on your remote Windows 8 device. When you run exploratory tests on a Windows 8 remote device, you perform the actual exploratory testing on the remote device and add comments, screenshots, file attachments on the machine running the Exploratory Test window.

Now, test your Windows Store app:

Test on device, comment in Exploratory Test window

Capturing issues during an exploratory test session

While you implement your exploratory test session, you can add notes, screenshots, and any applicable file attachments as you discover errors, requirements violations, or other discrepancies.

After you add your comments, screenshots, and file attachments, you can create a new bug. The new bug will automatically include these items. When you create a new bug, you can specify which action steps to include in the bug. For more information, see Exploring the application in the topic Performing Exploratory Testing Using Microsoft Test Manager.

You can also create a new manual test case that is based on the action steps that you have performed against the application you are exploring. Just like the bugs, the new manual test cases are listed within your session and can also be opened and edited as you continue to explore the application. For more information, see Make re-testing easy in the topic Performing Exploratory Testing Using Microsoft Test Manager.

Step 5: Submitting a bug

When you submit a bug, fill out the fields and save the bug. When you submit bugs for your Windows Store apps, you can select the specific steps to include in your bug and you can remove steps that are not necessary. The next illustration shows how to select the steps to include in the bug.

Selecting the steps to include in the bug

Change steps

What else should I know?

Prerequisites

  • Prerequsite Make sure you have the required version of Visual Studio installed. To use the exploratory testing window from Microsoft Test Manager, you must have Microsoft Visual Studio 2012 Testing Essentials, Premium or Ultimate installed.

  • Prerequsite The remote Windows 8 device and the machine running Microsoft Test Manager must be connected over a network or connected directly through an Ethernet cable. Testing over the internet is not supported.

  • Prerequsite To display the Microsoft Test Manager window, choose Start, and then choose All Programs. Point to Microsoft Visual Studio 2012 and then choose Microsoft Test Manager.

  • Prerequsite When you start Microsoft Test Manager for the first time, you are not connected to any team project. You must first connect to Team Foundation Server, select your project collection, and your team project. The team project is the same one that is used by other users on your team to create requirements or user stories, maintain source code, and build the apps that you want to test. Your testing artifacts are created and managed in this same team project. For more information about connecting to a team project, see How to: Connect to a Team Project For the First Time from Microsoft Test Manager.

Important issues

  • CautionSecurity warning: After you have started an exploratory test session, the action steps that you perform on the application being explored are recorded automatically. Make sure that you protect any sensitive data. For example, avoid any unnecessary sensitive content in your screenshots and recordings.

  • CautionSecurity warning: You can optionally choose to run the Microsoft Test Tools Adapter (remote debugger) in No Authentication mode, but this mode is strongly discouraged. There is no network security when you run in this mode. Choose the No Authentication mode only if you are sure that the network is not at risk of from malicious or hostile traffic.

  • Caution iconWarning: Remember, when you run exploratory tests on a Windows 8 remote device, the test steps must be executed on the remote device. The test steps are marked as passed or failed on the machine runs Test Runner.

  • Caution iconCaution: Both the machine that is running Microsoft Test Manager for Visual Studio 2012 and the Windows 8 remote device should be running as the same user. If the user is different, a credentials dialog box will display when you try to connect. The dialog box will require you to provide the same credentials to connect to the remote device.

    You can also log on using the same Microsoft Live ID on both machines if your logon credentials differ.

  • Caution iconCaution: Testing Windows Store apps on Windows 8 remote devices that use the ARM architecture is not supported in Visual Studio 2012.

  • Caution iconCaution: To record audio, you must have an audio recording device configured on your computer.

Tips

  • Tip To connect directly to a remote device, connect the two machines with a standard Ethernet cable. If the device does not have an Ethernet port, you can use a USB to Ethernet adapter to connect to the cable.

  • Tip When testing a Windows Store app, an enhanced html action log is created. You can view it in your web browser. The action log includes screenshots that help identify the UI elements being used when a problem is found.

  • Tip By default, the port 6905 is used by Microsoft Test Manager to communicate with remote devices. You can override this port if necessary. If you use a different port, you must enter the IP address in the Device name/IP address field. For example, enter testDevice1:nnnn.

    Tip You can view or change the port being used by the Microsoft Test Tools Adapter by opening the service configuration file mttaservice.exe.config in the Visual Studio installation folder, on the remote device.

  • Tip If something else requires your attention while the exploratory test session is running, you can pause the recording and use your computer for other activities. Those activities won’t be included in the action step recording. For example, you can switch to your e-mail application to respond to something critical and nothing will be recorded.

What else should I read?

Related tasks

Topics

Tutorial on performing an exploratory test session: Get an overview about using the Exploratory Testing window feature of Microsoft Test Manager to perform acceptance testing on a product backlog item.

Working requirements work item types for exploratory testing: From Microsoft Test Manager, you can start an exploratory test session that is based on a product backlog item, user story, or requirement work item type. Alternatively, you can implement a nonspecific exploratory session that is performed without any specific planning or documentation.