September 2009

Volume 24 Number 09

Windows Mobile and Accelerometers - Shake and Skip to Interact with Your Music

By Chris Mitchell | September 2009

The G-Sensor, or accelerometer, is fast becoming a staple in the mobile development arsenal. Windows Mobile devices have joined the array of gadgets offering capabilities that let developers do everything from determining how often someone drops his phone to allowing users to control applications and games by tilting, shaking, hitting, or throwing their phones about—the last action perhaps needs a healthy disclaimer attached. This article looks at how to access and use accelerometer data with an HTC Diamond to control music playback, so the phone skips tracks in Windows Media Player Mobile when a user shakes it.

Because the use of accelerometers is a new area of development for Windows Mobile, and to keep with the general purpose of the article, I’ve written the application Shake ’n Skip (shown in Figure 1) to make it easy to determine how to start, get data from, and close the sensor, as well as change the sample application’s function with a couple of lines of code. The application discussed in this article can easily be recoded to open an Internet browser, lock the device, or do anything else you want to try, all with minimal coding effort. In addition, the application has a logger that enables raw output from the sensor to be viewed so that other detection techniques beyond the simple shake-detection algorithm described here can be developed. Such algorithms might encompass the detection of other phone gestures to control your applications and games.

You can read more about developing with the HTC Diamond accelerometer at blog.enterprisemobile.com/2008/07/using-htc-diamonds-sensor-sdk-from-managed-code/. Note that the code in this article works with the HTC Diamond and other HTC devices, but will not work with other phones.

Figure 1 Screenshots of the Shake 'n Skip application (left) and controlled Windows
Media Player Mobile (right).

Application Discussion

The Shake‘n Skip application needs to perform several tasks. First, it needs to periodically get the most up-to-date information from the sensor. Second, it needs to encompass a metric that can detect whether the phone is being shaken. Third, if the application determines that the phone is being shaken, it needs to take the relevant action, which in this example is to cause a Windows Mobile media player to skip to the next track. In the sample application, I’ll simulate key presses to launch and control Windows Media Player. This method enables you to control much of the phone’s functionality, return to the home screen, lock the device, or launch any application.

Many of the applications that use accelerometers are giving a new lease on life to older applications and games. For example, Labyrinth, a game in which you move a ball around on a mazelike board, trying to get through the maze without rolling the ball into a hole, has been around for a long time. A Labyrinth toy was patented as long ago as 1891 by S. D. Nix. Does anyone think he worried about having enough spare MIPS to run his application, or that he investigated using vibration control to simulate a ball hitting the side of the maze at some speed? Many of the applications use the accelerometer in different ways to create an engaging new interface for the phone and can also combine the accelerometer with other outputs to create some very compelling user experiences.

Accelerometers

A quick definition to get us started: an accelerometer is a sensor that measures acceleration, the rate of change in velocity across time. Velocity is described using a vector, which, unlike a scalar, has a magnitude and direction—something I’m sure you remember from Physics 101.

Th ere are many diff erent types of accelerometers, from a water tube with an air bubble that shows the direction of acceleration to the integrated circuit types we work with in mobile phones. The HTC Diamond uses a MEMS (Micro-Electro Mechanical System)- based accelerometer, a type that forms small structures with dimensions in the micrometer scale (one millionth of a meter). Most other phones on the market do as well. In the phone, a three-axis accelerometer provides orientation, indicating whether the phone is facing up or down, whether the phone is standing up with the screen toward you or is upside down with the screen away, and all combinations in-between. Th e orientation of the phone at any point can be described with the following three values:

1. TiltX or Roll; 0 is fl at. Measures rotation around the center of the phone running lengthwise from the mouth piece to the earphone.
2. TiltY or Pitch; 0 is fl at. Measures rotation around the center of the phone running widthwise across the screen.
3. TiltZ (not yaw); 0 is straight up, the minus sign (-) is flat, and the plus sign (+) is upside down. These measures can be seen more clearly in Figure 2, which shows raw output together with the axis. Keep in mind that these three values together form a vector that shows the direction of gravity.

Figure 2 A Graph Showing Output from the Accelerometer for a Phone Position

Now we need to detect when the phone is being shaken and, in that case, to issue a command to skip a track in Windows Media Player.

System Components

Th e Shake‘n Skip application is not hosting the Windows Mobile media player, nor is the application hosted in some type of plug-in, despite a plug-in being the obvious choice for redistribution. The application is written as a simple mobile Win form, so you can reuse the application for many diff erent purposes and test your own ideas for how you might use the accelerometer. The application allows you to output the sensor’s data to a log file that you can then test offline to see how well your algorithm ideas work. Figure 3 shows the logical flow of the application.

Figure 3 System Flow for Monitoring and Alerting of Location-Aware Tasks

The application’s sensor component is made up of a single function that takes a reference to a structure and returns the values in that structure. Th is sensor component is called on a periodic timer. In the application, the timer is set to call this function every 100 milliseconds. Be careful if you change this value because you could end up with aliasing eff ects if the value is set too low. The shake algorithm detection is also made up of a single function that takes the most up-to-date data returned from the sensor component. Th e shake algorithm compares the Euclidean distance (described later) between the last sensor data and the most recent sensor data. It then determines, using a threshold, whether a shake has been detected over a set number of comparisons. The number of successful comparisons needed to detect a shake is determined by two inputs: detect window and reset window. After a shake has been detected, the command component conditionally calls two functions, depending on the mode of operation selected on the interface: KeyCommand for Application or Logger. KeyCommand for Application takes an application named Wmplayer.exe and a phone key to simulate being pressed. The logger logs the data from the accelerometer to Gsensorlog.txt, a file placed in the root of your phone’s storage. For simplicity, I added the class for the application to the same namespace created by Visual Studio. The namespace holds the default class inherited from the form class that holds the UI.

System Components—G-Sensor

Accessing the accelerometer’s data requires calling from a managed C# environment to the unmanaged DLL provided on HTC phones that queries the sensor and provides the data. This sensor call requires three parts to be set up.The first part is a struct that is marshaled and stores the information needed from the main routine that calls the unmanaged DLL. The variables AngleY and AngleX are not used in the application but return the number of degrees to their respective planes.

//Data structure passed to sensor query api

public struct SensorData

{

public short TiltX;

public short TiltY;

public short TiltZ;

public short Unknown1;

public int AngleY;

public int AngleX;

public int Unknown2;

};

Several Pinvoke calls are needed as well, as you can see in the following code. It is worth noting that if you want to access an external DLL function such as CreateEvent but the function is listed as using User32 as the DLL, you can replace User32 with Coredll on Windows Mobile in a large number of situations and the call will still work.

[DllImport("HTCSensorSDK")]

extern static IntPtr HTCSensorGetDataOutput(IntPtr handle,

out SensorData sensorData);

[DllImport("HTCSensorSDK")]

extern static IntPtr HTCSensorOpen(int sensor);

[DllImport("HTCSensorSDK")]

extern static void HTCSensorClose(IntPtr handle);

[DllImport("coredll", SetLastError = true)]

extern static IntPtr CreateEvent(IntPtr eventAttributes, bool

manualReset, bool intialState, string name);

[DllImport("coredll", SetLastError = true)]

extern static bool EventModify(IntPtr handle, uint func);

[DllImport("coredll")]

extern static bool CloseHandle(IntPtr handle);

Figure 4. A Graph Showing Euclidean Distance Change over Two Shakes

 
Figure 5. A Graph Showing Changes to TiltX (Blue), TiltY (Red) and TiltZ (Green) Over Two Shakes

The final part of getting the data is to call the functions shown in the following code and pass in the SensorData struct to handle the data. The CreateEvent call synchronization object tells the OS that HTC_ GSENSOR_SERVICESTART must occur before the thread can resume running. When that event occurs, the thread can again be scheduled for CPU time. After it is scheduled, the thread continues running. The application thread is now synchronized with the sensor event.

public void GetSensorData(ref SensorData data)

{

//Initialise and start sensor

IntPtr Handle = HTCSensorOpen(1);

IntPtr hEvent = CreateEvent

(IntPtr.Zero, true, false, "HTC_GSENSOR_SERVICESTART");

EventModify(hEvent, 3);

CloseHandle(hEvent);

HTCSensorGetDataOutput(Handle, out data);

return;

}

System Components—Shake Algorithm

Now that the sensor information can be obtained, we need to fi nd an algorithm that allows us to detect a shake using this data.

The thinking behind the development of the algorithm goes like this: shaking a phone creates rapid change in the direction of gravity relative to the phone’s axis. This means we should see rapid change in the TiltX, TiltY and TiltZ values, and if we set a threshold value to determine when rapid-enough change is occurring, we should be able to fi lter out slow-moving change—for example, answering or picking up the phone rather than shaking.

However, we still need a measure to detect the rapid change. The most obvious way is to look at distance measured between two sets of data. If the distance is greater than a certain value, we can say the change is happening rapidly.

Given the distance is computed as

Euclidean distance is a simple distance metric I’ve used to measure rapid change. Th e calculation takes the past three axis accelerometer outputs and compares the result with the current output . However, dropping the phone sharply or hitting the phone could also generate a rapid change, so we need to also insert a time feature into the algorithm to ensure that only a protracted shake will trigger the shake-detection component.

Figure 6 Main Shake-Detection Function

public bool DetectShake(SensorDataOld dataold, SensorData datanew,

int threshold, int detectwindow, int resetwindow)

{

if (FirstTimeEntryFlag != 0)

{

//Convert values to use inbuilt Math library

double Xold = Convert.ToDouble(dataold.TiltX);

double Yold = Convert.ToDouble(dataold.TiltY);

double Zold = Convert.ToDouble(dataold.TiltZ);

double X = Convert.ToDouble(datanew.TiltX);

double Y = Convert.ToDouble(datanew.TiltY);

double Z = Convert.ToDouble(datanew.TiltZ);

//Set new values to old

dataold.TiltX = datanew.TiltX;

dataold.TiltY = datanew.TiltY;

dataold.TiltZ = datanew.TiltZ;

//Calculate Euclidean distance between old and data points

double EuclideanDistance = Math.Sqrt(Math.Pow(X - Xold, 2)

+ Math.Pow(Y - Xold, 2)

+ Math.Pow(Y - Yold, 2));

//Set shake to true if distance between data points is

//greater than the defined threshold

if (EuclideanDistance > threshold)

{

DetectWindowCount++;

if (DetectWindowCount > detectwindow)

{DetectWindowCount = 0; ResetWindowCount = 0; return true;}

}

else

{

ResetWindowCount++;

if (ResetWindowCount > resetwindow)

{DetectWindowCount = 0; ResetWindowCount = 0; return false;}

}

}

//No longer the first run.

FirstTimeEntryFlag = 1;

return false;

}

Keys and Key Codes for Windows Mobile

There are many fairly sophisticated actions that can be performed on Windows Mobile devices with minimal programming by simulating key presses. Want to show the today screen? Want to answer a call or lock your phone? The same command can be used for all three. Windows Mobile maintains a device– independent keyboard model that enables it to support a variety of keyboards. At the lowest level, each key on the keyboard generates a scan code when the key is pressed and released. The scan code is a hardware–dependent number that identifi es the key. The keyboard driver translates or maps each scan code to a virtual key code. The virtual key code is a hardware–independent hexadecimal number that identifi es the key.

const int KEYEVENTF_KEYUP = 0x02;

const int KEYEVENTF_KEYDOWN = 0x00;

keybd_event(VK, 0, KEYEVENTF_KEYDOWN, 0);

keybd_event(VK, 0, KEYEVENTF_KEYUP, 0);

Answer a call:

byte VK = 0x72; // Simulate answer phone call being pushed.

Lock device:

byte VK = 0x85; // Simulate phone being locked.

Go to homepage:

byte VK = 0x5B; // Simulate home key being pressed.

For more information, see msdn.microsoft.com/en-us/library/bb431750.aspx

As you can see from Figure 4, a large and (most important) sustained distance value diff erentiates the phone being shaken from its normal resting position. If you look at Figure 5, you see the corresponding sensor values. You should be able to see that the phone was shaken with the screen facing upward. The change in TiltX is caused by my wrist twisting to shake the phone, whereas the TiltY value changes very little because the phone was held mostly flat while resting and when being shaken. The code in Figure 6 shows the calculation of the Euclidean distance and the strategy for determining when the phone has been shaken.

One other feature of the algorithm worth mentioning is the reset count. I found that it was useful not only to require multiple consecutive values but to also allow two high values followed by one small followed by another large to trigger the shake detection—three consecutive high values separated by one low value. To deal with scenarios similar to these, I used two counters instead of one. The first counts up to the required number of high Euclidean values detected above the threshold. The second keeps a reset count zeroed at the point of one large change detection, and can reset the entire count if a given number of large changes are not detected consecutively after the last large change. And now that we have a metric that can be used to determine whether a phone is being shaken, we need to know what to do with this information and what it can control.

Watch the Shake ‘n Skip Video

To see how the Shake ‘n Skip application works, you can watch a video that is available with the online version of this article at msdn.microsoft.com/magazine/.

System Components—Skip Track

As I mentioned earlier, one of the reasons to construct the application as a Win Form application is to be sure the application can be reused as much as possible. To continue this strand of simplicity, I used a specific technique to control Windows Mobile media player that involved simulating keypad controls that can be used to achieve a number of activities. (See the sidebar, Keys and Keycodes for Windows Mobile.) This technique is joined with the use of the System Diagnostic namespace to make sure that the application is started and at the forefront of the view on the Windows Mobile device. Here’s the code that starts an application and simulates a key press on the device.

public void KeyCommandForApplication(string FileName, byte VK)

{

System.Diagnostics.Process p = new System.Diagnostics.Process();

p.StartInfo.FileName = FileName;

p.Start();

const int KEYEVENTF_KEYUP = 0x02;

const int KEYEVENTF_KEYDOWN = 0x00;

keybd_event(VK, 0, KEYEVENTF_KEYDOWN, 0);

keybd_event(VK, 0, KEYEVENTF_KEYUP, 0);

}

Wrapping Up

I am sure that the algorithm used here is a little simplistic and can probably be improved, but under limited testing the application functions well. Th e use of accelerometers in mobile application development is likely to increase. When this is combined with mobile GPS (see msdn.microsoft.com/en-us/magazine/dd315419.aspx), light sensors and other inputs, new and exciting interactions with applications become possible.

Many thanks to Scott Seligman and Koushik Dutta for their great pages and code on working with HTC’s accelerometer. Also many thanks to Craig of bexmedia.net for making the video look good for me.

Chris Mitchell finished his Ph.D. in machine learning and music/sound signal process and was then a Kauffman/NCGE fellow. He is currently involved in a setting up a startup company in Cambridge, UK. You can contact him at chris.mitchell@anglia.ac.uk.