April 2011

Volume 26 Number 04

UI Frontiers - Lissajous Animations in Silverlight

By Charles Petzold | April 2011

We commonly think of software as more flexible and versatile than hardware. It’s certainly true in many cases, because hardware is often stuck in one configuration while software can be reprogrammed to perform completely different tasks.

Yet some rather prosaic pieces of hardware are actually quite versatile. Consider the common—or these days, not so common—cathode ray tube (CRT). This is a device that shoots a stream of electrons on the inside of a glass screen. The screen is coated with a fluorescent material that reacts to these electrons by briefly glowing.

In old-time TV sets and computer monitors, the electron gun moves in a steady pattern, repetitively sweeping horizontally across the screen while traveling more slowly from top to bottom. The intensity of electrons at any time determines the brightness of a dot at that point. For color displays, separate electron guns are used to create the primary red, green and blue colors.

The direction of the electron gun is controlled by electromagnets, and it can actually be aimed at any arbitrary location on the two-dimensional surface of the glass. This is how the CRT is used in an oscilloscope. Most commonly, the beam sweeps horizontally across the screen at a constant rate, usually in synchronization with a particular input waveform. The vertical deflection shows the amplitude of that waveform at that point. The fairly long persistence of the fluorescent material used in oscilloscopes allows the entire waveform to be displayed—in effect “freezing” the waveform to be visually examined.

Oscilloscopes also have an X-Y mode that allows the horizontal and vertical deflection of the electron gun to be controlled by two independent inputs, usually waveforms such as sine curves. With two sine curves as input, at any point in time the point (x, y) is illuminated, where x and y are given by the parametric equations:

The A values are amplitudes, the ω values are frequencies and the k values are phase offsets.

The pattern resulting from the interaction of these two sine waves is a Lissajous curve, named after French mathematician Jules Antoine Lissajous (1822 - 1880) who first visually created these curves by bouncing light between a pair of mirrors attached to vibrating tuning forks.

You can experiment with a Silverlight program that generates Lissajous curves on my Web site (charlespetzold.com/silverlight/LissajousCurves/LissajousCurves.html). Figure 1 shows a typical display.

Figure 1 The Web Version of the LissajousCurves Program

Although it’s not quite obvious in a static screenshot, a green point is moving around the dark gray screen and leaving behind a trail that fades out over four seconds. The horizontal position of this point is governed by one sine curve, and the vertical position by another. Repetitive patterns result when the two frequencies are simple integral ratios.

It’s now a truth universally acknowledged that a Silverlight program of any good fortune must be ported to Windows Phone 7, and subsequently reveal any performance problems previously masked by high-powered desktop computers. That was certainly the case with this program, and I’ll be discussing these performance issues later in this article. Figure 2 shows the program running on the Windows Phone 7 emulator.

Figure 2 The LissajousCurves Program for Windows Phone 7

The downloadable code consists of a single Visual Studio solution named LissajousCurves. The Web application consists of the projects LissajousCurves and LissajousCurves.Web. The Windows Phone 7 application has the project name LissajousCurves.Phone. The solution also contains two library projects: Petzold.Oscilloscope.Silverlight and Petzold.Oscilloscope.Phone, but these two projects share all the same code files.

Push or Pull?

Aside from the TextBlock and Slider controls, the only other visual element in this program is a class named Oscilloscope that derives from UserControl. Providing the data for Oscilloscope are two instances of a class named SineCurve.

SineCurve has no visuals itself, but I derived the class from FrameworkElement so I could put the two instances in the visual tree and define bindings on them. In fact, everything in the program is connected with bindings—from the Slider controls to the SineCurve elements and from SineCurve to Oscilloscope. The MainPage.xaml.cs file for the Web version of the program has no code beyond what’s provided by default, and the equivalent file in the phone application only implements Tombstoning logic.

SineCurve defines two properties (backed by dependency properties) named Frequency and Amplitude. One SineCurve instance provides the horizontal values for Oscilloscope, and the other the vertical values.

The SineCurve class also implements an interface I called IProvideAxisValue:

public interface IProvideAxisValue {
  double GetAxisValue(DateTime dateTime);
}

SineCurve implements this interface with a rather simple method that references two fields as well as the two properties:

public double GetAxisValue(DateTime dateTime) {
  phaseAngle += 2 * Math.PI * this.Frequency * 
    (dateTime - lastDateTime).TotalSeconds;
  phaseAngle %= 2 * Math.PI;
  lastDateTime = dateTime;

  return this.Amplitude * Math.Sin(phaseAngle);
}

The Oscilloscope class defines two properties (also backed by dependency properties) named XProvider and YProvider of type IProvideAxisValue. To get everything moving, Oscilloscope installs a handler for the CompositionTarget.Rendering event. This event is fired in synchronization with the refresh rate of the video display and thus serves as a convenient tool for performing animations. On each call to the CompositionTarget.Rendering handler, Oscilloscope calls GetAxisValue on the two SineCurve objects set to its XProvider and YProvider properties.

In other words, the program implements a pull model. The Oscilloscope object determines when it needs data and then pulls the data from the two data providers. (How it displays that data is something I’ll discuss soon.)

As I began to add more features to the program—in particular, two instances of an additional control that displayed the sine curves, but that I eventually removed as an unenlightening distraction—I began to doubt the wisdom of this model. I had three objects pulling the same data from two providers, and I thought maybe a push model would be better.

I restructured the program so that the SineCurve class installed a handler for CompositionTarget.Rendering and pushed data to the Oscilloscope control through properties now named simply X and Y of type double.

I probably should’ve anticipated the fundamental flaw in this particular push model: The Oscilloscope was now receiving two separate changes in X and Y and constructing not a smooth curve but a series of stair steps, as shown in Figure 3.

Figure 3 The Disastrous Result of a Push-Model Experiment

Making the decision to go back to the pull model was easy!

Rendering with WriteableBitmap

From the moment I conceived this program, there was absolutely no question in my mind that using WriteableBitmap was the best solution to implement the actual Oscilloscope screen.

WriteableBitmap is a Silverlight bitmap that supports pixel addressing. All the pixels of the bitmap are exposed as an array of 32-bit integers. Programs can obtain and set these pixels in an arbitrary manner. WriteableBitmap also has a Render method that allows rendering the visuals of any object of type FrameworkElement onto the bitmap.

If Oscilloscope just needed to display a simple static curve, I’d use Polyline or Path and wouldn’t even consider WriteableBitmap. Even if that curve needed to change shape, Polyline or Path would still be preferable. But the curve displayed by Oscilloscope needs to grow in size, and it needs to be colored oddly. The line needs to fade out progressively: Recently displayed parts of the line are brighter than older parts of the line. If I used a single curve, it would need various colors along its length. This is not a concept 
supported under Silverlight!

Without WriteableBitmap, the program would need to create several hundred different Polyline elements, all colored differently and juggled around, and triggering layout passes after each CompositionTarget.Rendering event. Everything I knew about Silverlight programming indicated that WriteableBitmap would definitely offer much better performance.

An early version of the Oscilloscope class processed the CompositionTarget.Rendering event by obtaining new values from the two SineCurve providers, scaling those to the size of the WriteableBitmap, and then constructing a Line object from the previous point to the current point. That was simply passed to the Render method of WriteableBitmap:

writeableBitmap.Render(line, null);

The Oscilloscope class defines a Persistence property that indicates the number of seconds for any color or alpha component of a pixel to decrease from 255 to 0. Making those pixels fade out involved direct pixel addressing. The code is shown in Figure 4.

Figure 4 Code to Fade out Pixel Values

accumulatedDecrease += 256 * 
  (dateTime - lastDateTime).TotalSeconds / Persistence;
int decrease = (int)accumulatedDecrease;

// If integral decrease, sweep through the pixels
if (decrease > 0) {
  accumulatedDecrease -= decrease;

  for (int index = 0; index < 
    writeableBitmap.Pixels.Length; index++) {

    int pixel = writeableBitmap.Pixels[index];

    if (pixel != 0) {
      int a = pixel >> 24 & 0xFF;
      int r = pixel >> 16 & 0xFF;
      int g = pixel >> 8 & 0xFF;
      int b = pixel & 0xFF;

      a = Math.Max(0, a - decrease);
      r = Math.Max(0, r - decrease);
      g = Math.Max(0, g - decrease);
      b = Math.Max(0, b - decrease);

      writeableBitmap.Pixels[index] = a << 24 | r << 16 | g << 8 | b;
    }
  }
}

At this point in the development of the program, I took the steps necessary to also get it running on the phone. On both the Web and the phone, the program seemed to run well, but I knew it was not quite finished. I wasn’t seeing curves on the Oscilloscope screen: I was seeing a bunch of connected straight lines. And nothing destroys the illusion of digitally simulated analog faster than a bunch of extremely straight lines!

Interpolation

The CompositionTarget.Rendering handler is called in synchronization with the video display refresh. For most video displays—including the display on Windows Phone 7—this is usually in the region of 60 frames per second. In other words, the CompositionTarget.Rendering event handler is called approximately every 16 or 17 ms. (Actually, as you’ll see, that’s only the optimum situation.) Even if the sine waves are a leisurely one cycle per second, for a 480-pixel-wide oscilloscope, two adjacent samples might have pixel coordinates some 35 pixels apart.

The Oscilloscope needed to interpolate between consecutive samples with a curve. But what kind of curve?

My first choice was a canonical spline (also known as a cardinal spline). For a sequence of control points p1, p2, p3 and p4, the canonical spline provides a cubic interpolation between p2 and p3 with a degree of curviness based on a “tension” factor. It’s a general-purpose solution.

The canonical spline was supported in Windows Forms, but never made it into Windows Presentation Foundation (WPF) or Silverlight. Fortunately, I had some WPF and Silverlight code for the canonical spline that I developed for a 2009 blog entry called, appropriately enough, “Canonical Splines in WPF and Silverlight” (bit.ly/bDaWgt).

After generating a Polyline with interpolation, the CompositionTarget.Rendering processing now concluded with a call like this:

writeableBitmap.Render(polyline, null);

The canonical spline worked, but it was not quite right. When the frequencies of the two sine curves are simple integral multiples, the curve should stabilize into a fixed pattern. But that wasn’t happening, and I realized that the interpolated curve was slightly different depending on the actual sampled points. 

This problem was exacerbated on the phone, mostly due to the tiny phone processor having trouble keeping up with all the demands I was putting on it. At higher frequencies, the Lissajous curves on the phone looked smooth and curvy, but seemingly moving in almost random patterns!

Only slowly did I realize that I could interpolate based on time. Two consecutive calls to the CompositionTarget.Rendering event handler are about 17 ms apart. I could simply loop through all these intermediate millisecond values and call the GetAxisValue method in the two SineCurve providers to construct a smoother polyline.

That approach worked much better.

Improving Performance

One piece of essential reading for all Windows Phone 7 programmers is the documentation page “Performance Considerations in Applications for Windows Phone” at bit.ly/fdvh7Z. Aside from many helpful hints about improving performance in your phone applications, it will also tell you the meaning of those numbers that are displayed at the side of the screen when you run the program under Visual Studio, as shown in Figure 5.

Figure 5 Performance Indicators in Windows Phone 7

This row of numbers is enabled by setting the Application.Current.Host.Settings.EnableFrameRateCounter property to true, which the standard App.xaml.cs file does if the program is running under the Visual Studio debugger.

The first two numbers are the most significant: Sometimes, if nothing is going on, these two numbers display zero, but they’re both intended to display frame rates—which means they display a number of frames per second. I mentioned that most video displays are refreshed at the rate of 60 times per second. However, an application program might attempt to perform animations where each new frame requires more than 16 or 17 ms of processing time.

For example, suppose a CompositionTarget.Rendering handler requires 50 ms to do whatever job it’s doing. In that case, the program will be updating the video display at the rate of 20 times per second. That’s the program’s frame rate.

Now 20 frames per second is not a terrible frame rate. Keep in mind that movies run at 24 frames per second, and standard television has an effective frame rate (taking interlacing into account) of 30 frames per second in the United States, and 25 in Europe. But once the frame rate drops to 15 or 10, it’s going to start being noticeable.

Silverlight for Windows Phone is able to offload some animations to the Graphics Processing Unit (GPU), so it has a secondary thread (sometimes referred to as the composition or GPU thread) that interacts with the GPU. The first number is the frame rate associated with that thread. The second number is the UI frame rate, which refers to the application’s primary thread. That’s the thread that any CompositionTarget.Rendering handlers run in.

Running the LissajousCurves program on my phone, I saw numbers of 22 and 11, respectively, for the GPU and UI threads, and they dropped down a bit when I increased the frequency of the sine curves. Could I do better?

I began to wonder how much time this crucial statement in my CompositionTarget.Rendering method required:

writeableBitmap.Render(polyline, null);

This statement should have been called 60 times per second with a polyline consisting of 16 or 17 lines, but it was actually being called more like 11 times per second with 90-segment polylines.

For my book, “Programming Windows Phone 7” (Microsoft Press, 2010), I wrote some line-rendering logic for XNA, and I was able to adapt that for Silverlight for this Oscilloscope class. Now I wasn’t calling the Render method of WriteableBitmap at all, but instead directly altering pixels in the bitmap to draw the polylines.

Unfortuntely, both frame rates plunged to zero! This suggested to me that Silverlight knew how to render lines on a bitmap much faster than I did. (I should also note that my code was not optimized for polylines.)

At this point, I began wondering if an approach other than WriteableBitmap might be reasonable. I substituted a Canvas for the WriteableBitmap and Image element, and as each Polyline was constructed, I simply added that the Canvas.

Of course, you can’t do this indefinitely. You don’t want a Canvas with hundreds of thousands of children. And besides, these Polyline children needed to fade out. I tried two approaches: The first involved attaching a ColorAnimation to each Polyline to decrease the alpha channel of the color, and then removing the Polyline from the Canvas when the animation completed. The second was a more manual approach of enumerating through the Polyline children, decreasing the alpha channel of the color manually, and removing the child when the alpha channel got down to zero.

These four methods still exist in the Oscilloscope class, and they’re enabled with four #define statements at the top of the C# file. Figure 6 shows the frame rates with each approach.

Figure 6 Frame Rates for the Four Oscilloscope Updating Methods

Figure 6 tells me that my original instinct about WriteableBitmap was wrong. In this case, it’s really better to put a bunch of Polyline elements in a Canvas. The two fade-out techniques are interesting: When performed by an animation, the fading out occurs in the composition thread at 20 frames per second. When performed manually, it’s in the UI thread at 15 frames per second. However, adding new Polyline elements always occurs in the UI thread, and that frame rate is 20 when fading-out logic is off-loaded to the GPU.

In conclusion, the third method has the best overall performance.

So what have we learned today? Clearly, to eke out the best performance, it’s necessary to experiment. Try different approaches, and never, ever trust your initial instincts.

Charles Petzold is a longtime contributing editor to MSDN Magazine*. His new book, “Programming Windows Phone 7” (Microsoft Press, 2010), is available as a free download at bit.ly/cpebookpdf.*

Thanks to the following technical expert for reviewing this article: Jesse Liberty