July 2012

Volume27 Number 07

Touch and Go - Windows Phone Motion and 3D Views

By Charles Petzold | July 2012

The new Motion class introduced with Windows Phone 7.1 is a wonderful tool for programmers who need to know how the phone is oriented in three-dimensional space. The Motion class combines information from the sensors (accelerometer, magnetometer and gyroscope), and also smooths the data and makes it available in convenient forms.

When a Windows Phone program knows the orientation of the phone in 3D space, the phone can provide a portal into a 3D world. This facility has applications in mapping, virtual reality, augmented reality and, of course, games, but probably also some applications that have not yet been conceived. This is definitely an area where you’ll want to exercise your imagination as much as your coding skills.

Motion Data

I want to focus solely on obtaining the orientation of the phone rather than velocity or acceleration, which are also available from the Motion class. The sensor provides a MotionReading structure, and the orientation information is available in the Attitude property. The word “attitude” comes from flight dynamics, where it indicates the orientation of a plane in 3D space—distinct from altitude, which is the height above the Earth. The word is also sometimes used in vector geometry.

This Attitude property is of type AttitudeReading, another structure that defines five properties describing three-dimensional orientation:

• Yaw of type float, an angle in radians
• Pitch of type float, an angle in radians
• Roll of type float, an angle in radians
• RotationMatrix of type Matrix, a 4 × 4 matrix type defined in XNA
• Quaternion of type Quaternion, a type defined in XNA

Yaw, Pitch and Roll are also terms used in flight dynamics, and they’re often referred to as Euler angles. Yaw indicates the compass direction that the nose of the plane is facing, so as the plane turns right or left, the yaw changes. Pitch changes as the nose goes up for a climb or down for a dive. Roll changes as the plane banks left and right. To visualize these relative to the phone, it helps to imagine “flying” on your phone like a magic carpet by sitting on the phone’s screen with the top of the phone to your front and the three standard buttons to your rear.

A little program called YawPitchRoll included with the downloadable code for this article might also help visualize these angles. (Like all programs in this article, it requires references to the Microsoft.Devices.Sensors and Microsoft.Xna.Framework assemblies.) As shown in Figure 1, the program displays the values of these three angles converted to degrees, and it also symbolizes their values graphically.

Figure 1 The YawPitchRoll Display

Yaw is displayed with a simple line pointing north like a compass, whereas Pitch and Roll are displayed as solid balls that seem to roll toward the Earth. When the phone is sitting on a level table with the screen up and the top of the phone pointing north, all three angles have zero values.

As you tilt the top of the phone up and down, you can change Pitch from 90° when the phone is upright to -90° when the top of the phone points down. Similarly, you can change Roll from 90° to -90° by tipping the phone right and left.

When the phone’s display faces down, the Yaw angle points south. Pitch takes on values ranging from 90° to 180°, and from -90° to -180°. In the YawPitchRoll program, these values are symbolized by a hollow, red-outlined ball. The values of Roll continue to take on values ranging from 90° to -90°.

Rotational Perspectives

Although we spend our entire lives in a three-dimensional universe, many of us have a very poor intuitive grasp of three-dimensional rotation. Consequently, visualizing and working with rotations in 3D space can be challenging. Yaw, Pitch and Roll seem to adequately describe rotation in 3D space, but they turn out to be rather awkward in practical programming.

Programmers much prefer working with alternative ways to describe 3D rotation:

• A single angle describing rotation around a 3D vector (axis/angle rotation)
• A rotation matrix (a subset of a full 3D matrix transform)
• The quaternion, a 3D analogue of 2D rotation in the complex plane

These forms can all be converted between each other, as I demonstrate in Chapters 7 and 8 of my book “3D Programming for Windows” (Microsoft Press, 2008). The quaternion is particularly useful for smooth movement from one orientation to another because it lends itself to linear interpolation.

The Motion class provides 3D rotation with a rotation matrix and quaternion, but I’ll focus exclusively on the RotationMatrix property of the AttitudeReading structure. This is an XNA Matrix value, which is a standard 4 × 4 transform matrix, but the matrix represents only rotation. It has no scaling and no translation. The M14, M24, M34, M41, M42 and M43 fields are all 0, and the M44 field is 1.

When working with this rotation matrix, a change in perspective will be useful. In the previous installment of this column, I described how the three-dimensional vector available from the Accelerometer and Compass sensors is relative to a 3D coordinate system imposed on the geography of the phone. However, when working with the rotation matrix, you really need to visualize two different 3D coordinate systems, one for the phone and the other for the Earth:

• In the phone’s 3D coordinate system, positive Y points to the top of the phone (in portrait mode), positive X points right and positive Z comes out of the screen.
• In the Earth’s 3D coordinate system, positive Y points north, positive X points east and positive Z comes out of the ground.

When the phone sits on a level surface with the screen up and the top pointing north, the RotationMatrix value is the identity matrix. Otherwise, it describes how the Earth is rotated relative to the phone, which is opposite of the rotation described by the Euler angles.

To illustrate this, I wrote a little program called MotionMatrix. The display (shown in Figure 2) consists only of numeric values: the Yaw, Pitch and Roll values, the 3 × 3 rotation matrix subset of the full matrix, and rotation expressed in the axis/angle form.

Figure 2 The MotionMatrix Display

When you first start playing with this program, your immediate instinct is probably to orient the phone so that all the Euler angles are zero. However, the axis vector down at the bottom goes crazy because there’s almost no rotation, so the axis value can be almost anything. Simple rotations calm the display. The screenshot in Figure 2 shows the top of the phone pointing north but elevated about 45°. That’s what the Pitch value reports, and the axis/angle rotation shows a very close value of 46°.

But notice the axis of rotation: It’s approximately (–1, 0, 0), which is the coordinate axis that points to the left of the phone. XNA axis/angle rotations abide by the right-hand rule: Point the thumb of your right hand in the direction of the axis (that is, to the left of the phone in this case). The curve of your fingers then shows the direction of positive rotation. This means the rotation is opposite the Pitch angle. It tells us that the phone must be rotated -45° to be aligned with the Earth.

The MotionMatrix and XNA

It seems reasonable to use the rotation matrix from the Motion class for viewing 3D objects on the phone. But rather than the normal approach—where a 3D object is rotated relative to the display, perhaps with a mouse or finger—here the display would be rotated relative to the 3D object (conceptually, anyway).

To experiment with this, I downloaded a 3D model of an antique telephone from the archive3d.net Web site (specifically bit.ly/GSNTi3) and then converted the 3DS format to the XNA-friendly FBX format using a converter tool from Autodesk. I created four Visual Studio projects that show progressively better ways of displaying this object. (Actually, I floundered around quite a bit and only later created these four projects to disguise that fact!) To reduce your source code downloading time, all four projects reference the same 3D model file.

The first of the four projects is called View3DPhonePlain, which is just a static view of the object; Figure 3 shows the relevant code. This program loads in the model, defines a world transform that scales the model down quite a lot, and defines a simple camera. (The apparent superfluity of field variables will make sense shortly.)

Figure 3 The View3DPhonePlain Program

public class Game1 : Microsoft.Xna.Framework.Game
{
    GraphicsDeviceManager graphics;
    Model model;
    Matrix scaleMatrix, worldMatrix, lookatMatrix,
           viewMatrix, projectionMatrix;
    ...
    protected override void LoadContent()
    {
        model = this.Content.Load<Model>(“Phone Retro Caesar N170910”);
        foreach (ModelMesh mesh in model.Meshes)
            foreach (BasicEffect effect in mesh.Effects)
                effect.EnableDefaultLighting();
        // World matrix
        scaleMatrix = Matrix.CreateScale(0.0001f);
        worldMatrix = scaleMatrix;
        // View matrix
        lookatMatrix = Matrix.CreateLookAt(new Vector3(0, 0, 10f),
                                                       Vector3.Zero,
                                                       Vector3.Up);
        viewMatrix = lookatMatrix;
        // Projection matrix
        float aspectRatio = 
            graphics.GraphicsDevice.Viewport.AspectRatio;
        projectionMatrix =
            Matrix.CreatePerspectiveFieldOfView(MathHelper.PiOver4,
                                                aspectRatio,
                                                1f, 100.0f);
    }
    ...
    protected override void Draw(GameTime gameTime)
    {
        GraphicsDevice.Clear(Color.CornflowerBlue);
        model.Draw(worldMatrix, viewMatrix, projectionMatrix);
        base.Draw(gameTime);
    }
}

In a 3D graphical system like XNA, models are subjected to a series of matrix transforms on their way to the display. The world transform is applied first, and this situates the model in 3D space. The camera is a combination of two transforms applied in succession: The view transform accounts for the location and orientation of the camera, while the projection transform handles perspective effects and also normalizes all the coordinates for clipping.

Figure 4 shows how the model looks in XNA’s default landscape mode, and it’s pretty much as we might expect.

Figure 4 The View3DPhonePlain Display

The next step is the project named View3DPhoneReoriented. I wanted to switch to portrait mode so that the 3D coordinates used within XNA were the same as the 3D coordinate system used by the sensors. This is a simple matter of adding a few statements to the constructor of the Game derivative:

graphics.IsFullScreen = true;
   graphics.PreferredBackBufferWidth = 480;
   graphics.PreferredBackBufferHeight = 800;

I also wanted to reorient the phone itself. As I mentioned, the 3D rotation matrix supplied by the Motion class is the identity matrix when the phone is sitting on a table with the screen up and the top pointing north. How did I want the 3D telephone to appear when the phone was in that position? I wanted to be looking at the top of the phone as if viewed from above, so I simply added a rotation around the X axis to the world transform:

worldMatrix *= Matrix.CreateRotationX(MathHelper.PiOver2);

The result is shown in Figure 5.

Figure 5 The View3DPhone­Reoriented Display

The camera looks a little too close to the object at this point, but this didn’t bother me because I knew that when I incorporated the rotation matrix, I’d be able to rotate the phone a bit to get it all in view. (A sharp-eyed viewer of 3D light and shading might notice a conceptual flaw in the approach I’m using, but let me naively traipse down this path first and then I’ll fix things up.)

Now let’s incorporate the rotation matrix from the Motion class.

Conceptually, the 3D object should seem to remain fixed in space, and moving the phone should allow you to view the object from different directions. Of course, the experience is not like “real life” because you can’t ever move the phone to look away from the object. The object is always there on the display, but the orientation of the display in 3D space gives you different views of it.

To achieve this effect, we need to apply a matrix to the object that achieves rotation from the phone’s coordinate system to the Earth’s coordinate system, where this object conceptually exists.

We should be able to achieve this effect simply by including the RotationMatrix object from the Motion class in the calculation of the world transform. The View3DPhoneWorldMotion project does this. It includes code to create a Motion sensor in its constructor, start it in the OnActivated override and stop it in the OnDeactivated override. I then moved the calculation of the world transform to the Update override, as shown in Figure 6.

Figure 6 The Update Override in View3DPhoneWorldMotion

protected override void Update(GameTime gameTime)
{
    if (GamePad.GetState(PlayerIndex.One).Buttons.Back == 
      ButtonState.Pressed)
        this.Exit();
    worldMatrix = scaleMatrix * 
      Matrix.CreateRotationX(MathHelper.PiOver2);
    if (motion != null && motion.IsDataValid)
        worldMatrix *= motion.CurrentValue.Attitude.RotationMatrix;
    base.Update(gameTime);
}

In this program, as you move the phone’s orientation in 3D space, you view the 3D object from all different directions. The effect is very smooth and—I must admit—very cool.

Shifting the Rotation

And yet, the more I played around with this version of the program, the more I felt that something was wrong.

By applying the rotation matrix to the world transform, the 3D object is effectively rotated not only relative to the camera, but also relative to the light source. In some scenarios, this would be correct. For example, if you implement a touch interface so you can turn the 3D image around with your fingers, applying the rotation matrix to the world transform would be appropriate.

But for this program, the paradigm is quite different. I wanted the phone’s display to be like a mobile camera that moves relative to the 3D object, and that means this object should remain fixed relative to the light source. The rotation matrix from the Motion sensor needs to be applied to the view transform rather than the world transform.

For the final version of the program—called View3DPhone­CameraMotion—I restored the calculation of the world transform to the original version:

worldMatrix = scaleMatrix;

Even applying the initial rotation was a mistake because it implied that the light was coming from behind the phone rather than from above. The new Update method is shown in Figure 7.

Figure 7 The Update Override in View3DPhoneCameraMotion

protected override void Update(GameTime gameTime)
{
    if (GamePad.GetState(PlayerIndex.One).Buttons.Back == 
      ButtonState.Pressed)
        this.Exit();
    if (motion != null && motion.IsDataValid)
        viewMatrix = Matrix.CreateRotationX(MathHelper.PiOver2) *
                        motion.CurrentValue.Attitude.RotationMatrix *
                        lookatMatrix;
    else
        viewMatrix = Matrix.CreateRotationX(MathHelper.PiOver2) *
                        lookatMatrix;
    base.Update(gameTime);
}

You really need to try out this program on a phone to get the full effect, but the image in Figure 8 shows one possible view.

Figure 8 One View in View3DPhoneCameraMotion

Charles Petzold is a longtime contributor to MSDN Magazine and is currently updating his classic book “Programming Windows” (Microsoft Press, 1998) for Windows 8. His Web site is charlespetzold.com.

Thanks to the following technical expert for reviewing this article: Donn Morse