Composing Music for Interactive Titles: An Overview of DirectMusic Producer

 

Microsoft Corporation

August 1998

Introduction

The quality of sound in game titles has improved significantly over the last few years. Taking their cue from movies and television, game producers have learned that a compelling soundtrack can mean a compelling gaming experience. So why do large numbers of users continue to turn off the audio while they play? Perhaps because the game music of the past has suffered from inconsistent music delivery, repetitious musical phrases, and noninteractive approaches. But innovations such as Microsoft® DirectMusic® are making it easier than ever to create multimedia applications that use sound to their best advantage.

This document will explore the use of sound in game titles and explain how developers can use the DirectMusic API to create interactive sound scores that have the quality of CD audio for games.

Sound Card Roulette

The General MIDI (GM) standard authored in 1982 offered some newfound consistency to game developers, and some excellent GM cards were built. But the GM standard was limited by:

  • The difficulty of creating 128+ high-quality GM instruments on a single card.
  • The inability of General MIDI instruments to provide the variety and richness required for professional audio.
  • The uneven quality of GM soundsets.

The quality of a MIDI score depended on hardware that was out of the composer's control. Because of this, game developers typically sacrificed the flexibility of MIDI in favor of digital streaming audio solutions that could provide both consistency and superior sound quality.

Repetition Is Repetitious

Successful games can entertain the same user again and again. This means that users will probably tire of hearing the same sound track over and over. But if the musical scoring was "dynamic" so that it changed with the events of each game, the user experience, and the shelf life of the game, would be improved. There is no reason why the music in games cannot be as dynamic as the visual experience.

The concept of a dynamically changing musical score is not new. Jazz music, for example, was built on the premise that a musical composition should change and be improvisational. Numerous artists have recently released interactive remixes. In the future we will probably see more of this interactive approach to composition.

Interactive Music

The term "interactive music" has been bandied about for years. But the reality has not lived up to the hype. This is obvious when we compare the use of sound in film and games. Consider the transition from a shot of a tense battlefield scene to a shot of an airplane in descent over a deserted island. In the cinematic world, music helps to create a seamless transition from scene to scene. In games, however, the most likely scenario is that the battle music comes to an abrupt halt when the next sequence is triggered, with no transition whatsoever. In the best case, the game might include music for both the battle and the airplane landing, but there would be a loss of continuity because of a poor musical transition.

Why is the use of sound in games inferior to the use of sound in film? Because game developers have lacked the technology to create interactive scores with CD-quality audio.

DirectMusic's Brave New World

Enter DirectMusic. This new DirectX® technology makes it possible for interactive media developers to create dynamic and interactive musical worlds.

DirectMusic is based on these principles:

  • Music should be of the highest sound quality, delivering the same aural experience created by the composer directly to the end user's machine.
  • Music should be able to dynamically change over time with creative and dramatic situations.
  • Music should be interactive, responding in real time to the actions of the user and to the scene.
  • Music transitions from scene to scene and sound to sound should be seamless.
  • Music should be synchronized to the visuals, or the visuals synchronized to the music, depending on the dramatic situation.
  • Music should be rendered within variables defined by the composer, even in interactive scenarios.

DirectMusic Producer provides authoring tools for the composer wishing to use DirectMusic to create music that embodies these principles. Composers can use Producer to abstract parameters of musical performance such as harmonic and melodic patterns, musical intensity, tone color, or harmonic progressions. These parameters are expressed as file types that can be integrated into real-time performance.

The file formats supported by DirectMusic provide critical links between developers and designers, sound designers, and composers. Producer's composer-friendly environment and DirectMusic's developer-friendly environment work together to simplify and enhance the creation of interactive media. With the DirectMusic file formats, integrating audio into games is quicker than ever.

Creating a Consistent Sound Experience: DLS Designer

The downloadable sounds (DLS) standard, based on the high-quality wave format, provides new possibilities for scoring game titles and Internet applications. It works like this: The Microsoft synthesizer gives you the ability to synthesize audio in software. Using the Microsoft synthesizer with a DLS-compatible sound card can make the user's computer function like a high-quality sampler. In this way, DLS allows the intent of the composer to be realized on the desktop. DLS makes for the best of all worlds by utilizing the excellent fidelity of the wave format and taking advantage of the flexibility and interactivity of MIDI.

DLS collections can be created in DLS Designer, a DirectMusic Producer module. To create a DLS collection, you import wave files into a folder called a wave pool. DLS Designer features a basic Wave Editor in the DLS Designer to "audition" wave files for your application. You can also use the wave editor to cut-and-paste selected samples from a wave file, and to set loop points. You can designate up to 16 regions on the MIDI keyboard for each instrument. Drum instruments can have a different region for each of the 128 MIDI notes.

Figure 1. DLS Designer

Each region is assigned a single wave file. Articulations are set for each instrument (on drum instruments, for each region). Articulations consist of volume and pitch envelopes with standard attack, decay, sustain, and release (ADSR) parameters and low frequency oscillator (LFO) settings.

You can use two basic types of wave files in a DLS collection:

  • traditional synthesizer-style sound samples (usually with loop points for held notes)
  • more complete musical performances such as guitar riffs or drum grooves

DirectMusic allows you to synchronize both types effectively.

Bands consist of any number of program changes, with pan and octave settings. They can be stand-alone files or elements of DirectMusic files known as styles and segments. Bands provide the link between custom DLS instruments and the note data in styles or segments. You can designate for each DirectMusic performance channel a GM or GS instrument (courtesy of the Roland General MIDI sound set), or custom DLS instruments from multiple DLS collections. This format is very interactive; you can easily swap band data into a musical performance in progress.

Figure 2. DirectMusic band: program changes (including DLS instruments) are assigned in the list on the left; the grid on the right is an editor for volume (up and down) and pan (left and right) settings for each performance channel.

Creating Variable Forms: Style Designer and Chordmap Designer

DirectMusic Producer includes a number of resources to create variable musical forms. Two of these resources are styles and chordmaps.

Styles

Styles are a powerful component of the DirectMusic arsenal. You can use them to create nonrepetitive musical forms. Styles consist of patterns that have particular uses. Each pattern is made up of a number of parts (for example, piano, bass, and drum parts). Parts are melodic and harmonic figurations that can be edited by a standard piano roll editor and assigned to a particular performance channel. You can also import pattern data from standard MIDI files.

Styles are very flexible because they carry the "functionality" of a note as well as the note itself. This means that styles make notes "intelligent" by giving them information about how to respond to a given harmonic situation. A style pattern can contain chord movement, or it can be authored without it so that chord progressions can be independently applied.

Variations

Variations are the key to creating dynamic musical performances in styles. Each part can have up to 32 independent variations. For example, the creation of 32 bass variations and 32 drum variations provides 1,024 combinations for the DirectMusic Performance Engine to randomly select. You can use variations to create variable soundscapes in applications where you might have been forced to use looping wave files in the past.

Figure 3. Variations: This style pattern features piano, pad, bass, and drum parts. The piano part (in maximized view, displaying only variation 1) features variations 1 through 9 with note data and variations 10 through 20 without note data. Variations 21 through 32 are disabled.

Part variations can be locked with other part variations to layer instruments to create rich timbres or to provide rhythmic synchronization, all in the context of variability.

Segments

The basic unit of measure in DirectMusic is the segment; all files interact with segments. For example, the most common way to integrate style data with chord progressions is to create a segment with style and chord tracks.

Figure 4. Segment with chord, chordmap, groove, style, and band tracks.

Chordmaps

Style patterns can perform in numerous ways and the chords you can apply to those patterns can vary as well. You can use the Chordmap Designer in Producer to construct chord progressions that can be independently applied to the note functionality of styles. For example, in combination with templates, a type of DirectMusic segment defining overall form, you can use chordmaps to create chord progressions that vary depending upon the parameters you specify.

To compose a chordmap, you must first create signpost chords. Signpost chords form the basis of the chordmap structure and can be flagged with signpost markers, including 1 through 7 and A through F. Signpost chords can have multiple markers, and markers can have multiple signpost chords.

Figure 5. Signpost list

The existence of a signpost track in a template indicates signpost markers. The DirectMusic Performance Engine randomly chooses any available signpost chord for that particular marker. In support of cyclical forms, the Engine will consistently choose the same chord for a particular marker until the segment is recomposed.

Figure 6. Template

Connecting Chords

You can also create variability using connecting chords. Signpost chords can be inserted into the Chordmap Editor in the Chordmap Designer, with beginning and ending signpost chords on the first and last measures of the chordmap. Variable connecting chord paths between the signpost chords can then be created on any beat in the timeline.

For any two signpost chords indicated in a template, the Performance Engine will randomly select a valid chord path.

Figure 7. This chordmap features two different paths to connect the beginning signpost chord with the ending signpost chord.

You can combine style variations with chordmap signpost and connecting chord paths to create and control a high degree of musical variability in your applications.

Creating Interactivity: Groove Levels and Secondary Segments

You can create interactive musical scores in DirectProducer using groove levels and secondary segments. Groove levels can shift patterns to enhance a dramatic situation. Secondary segments make it possible to layer an appropriate sound on top of a musical performance in progress.

Groove Levels

Each style pattern is assigned a groove range between 1 and 100. When a segment referencing a style is performed, the style's groove track indicates which groove level and, therefore, which pattern, is to be performed at any given bar.

While these groove ranges can be configured in many ways, composers commonly group patterns according to their intensity. As game play progresses to a different level, a new segment that features the next groove level can be swapped into the performance. New segments can be started on boundaries such as beat or bar to create a seamless performance.

Figure 8. Two segments with chord, groove, tempo, style, and band tracks: They are identical, except for their respective groove levels. You can create transitions between the two segments to interactively change the groove level and the intensity of the music. You can also create dynamically changing groove levels through code.

Secondary Segments

Secondary segments provide a powerful means to create an immediate connection between drama and music. Any segment can serve as a secondary segment to be layered on top of the primary segments already in progress.

Motifs

Motifs are special patterns you can create in the Pattern Editor of Style Designer. They are also considered secondary segments. Motifs can be any length, and can be a single note or a fully orchestrated passage featuring many parts. Motifs can be designed to play instantaneously (especially useful for sound effects), on the beat, or on measure boundaries. You can also design motifs to fit harmonically with a performance in progress by tracking chord progressions.

A very effective use of motifs is to associate the presence of a particular character in the drama with a motif, playing harmonically and rhythmically in sync with the underlying music. Wagner's use of leitmotifs in his operas is a good example of this type of usage.

Figure 9. The Style Window: While the snaregroove1 pattern is playing, the fiddle motif is selected to interactively layer it on top of the music in progress.

Creating Continuity: Transitions

DirectMusic makes it possible for you to create instant transitions on musical boundaries to new segments and new pattern groove levels. You can also use transitions to enhance continuity from one musical passage to another. A special AutoTransition call in the DirectMusic Engine uses special patterns—Intros, Breaks, Fills, and Ends—to provide continuity between musical passages. These special patterns have defined groove ranges so that you can associate patterns at any groove range with them. For example, a pattern with a groove range of 20 through 30 might have an associated Fill with a groove range of 20 through 30. AutoTransitions can occur instantaneously, or on musically appropriate boundaries.

Transitions are particularly effective when used between patterns performing at different groove levels. You can use a transition to create a graceful fade to a more serene passage, or an uplifting bridge into a more intense one.

Synchronizing Audio and Visuals

DirectMusic provides powerful tools for the coordination of audio and graphics. Musical events such as changing groove levels or performing a motif can be synchronized with graphics. You can also synchronize graphics with musical events using a notification mechanism. The DirectMusic Performance Engine can communicate music time to the graphics engine in terms of bar, beat, tempo, chord changes, groove level changes, and control change events. You can use this capability to create scenarios such as changing a scene on a certain musical event, or associating the movements of a character to a beat.

DirectMusic: Revolutionizing Music for the Digital Age

DirectMusic provides new tools and new ways to think about interactive media. DirectMusic Producer combines an interactive interface with the power to build musical interactivity into your games and put it into the hands of your users. This groundbreaking technology makes it easy for composers, producers, designers, and developers to work as a team to provide compelling multimedia experiences.

------------------------------------------------------------------------------------------------

© 1998 Microsoft Corporation. All rights reserved.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

The BackOffice logo, Microsoft, Windows, and Windows NT are registered trademarks of Microsoft Corporation.

Other product or company names mentioned herein may be the trademarks of their respective owners.

Show: