This documentation is archived and is not being maintained.

Managed Babel System Essentials

[This topic is pre-release documentation and is subject to change in future releases. Blank topics are included as placeholders.]

The Managed Babel System is a new component of the Managed Language Services integrated into Visual Studio. The Managed Babel System provides colorization of tokens, triggering of token actions, and IntelliSense functionality. 

The Managed Babel System generates only C# files and operates without the use of third-party tools such as lex and yacc. Lex and yacc are general-purpose tools designed to process and parse text and are required downloads in the C++ Language Services counterpart of Babel. They are not required with the Managed Babel System because they have been replaced with entirely new tools that are integrated into the Visual Studio Extensibility framework.

One real-world application of the Managed Babel System would be to integrate a new programming language into Visual Studio to take advantage of the token colorization, token-triggering action, and IntelliSense facilities available to built-in Visual Studio languages, such as Visual C# and Visual Basic.

Composition of the Managed Babel System

The functionality provided by the Managed Babel System is performed by three tools:

  • The Managed Package Lex (MPLEX) Scanner Generator

  • The Managed Package Parser Generator (MPPG)

  • Babel

These tools replace lex and yacc, although the tools use similar syntax and grammar, which is composed in external configuration files. This design enables developers familiar with lex and yacc to more easily write scanners and parsers within the Managed Babel System.

How the Managed Babel System Works

The developer first writes two files, a scanner specification file and a parser specification file, both of which follow lex and yacc conventions, as described below. The scanner specification is processed by MPLEX, which generates a scanner file in the form of a C# class. The parser specification file is processed by MPPG, which generates a parser file in the form of a C# class. Babel extracts information from these scanner and parser files (or their compiled versions), and generates a configuration file that the developer can modify as necessary.

After these preliminary tasks are accomplished, the developer does the following:

  1. Invokes the Project Wizard.

  2. Creates a new VSPackage.

  3. Creates a directory structure in Solution Explorer according to the requirements expected by the Managed Babel System.

  4. Edits user-supplied files, kept in the directories created in Solution Explorer, that enable the language-service tasks required to integrate the new language service into Visual Studio. 

The Managed Package Lex (MPLEX) Scanner Generator

MPLEX is a scanner generator that accepts a lex-like specification file and produces a C# scanner class. A scanner is a software mechanism for processing textual or binary input and providing output in the form of tokens. The tokens provided by the MPLEX-generated scanner class can then be processed by a parser generated by MPPG.

For example, MPLEX could be used to generate a scanner based on a specification file for a new programming language. The scanner then would be available to process files containing code written in the new language for further processing by a related parser created using MPPG.

Noteworthy, MPLEX generates colorizing scanners that can be called on a line-by-line basis in arbitrary line order.

The Managed Package Parser Generator (MPPG)

MPPG is a parser generator that accepts a yacc-like specification file and generates a C# parser class. The input grammar for MPPG is based on the yacc language.

Triggering of Token Actions

The user-supplied Configuration class is where token triggers are specified. It has two parts: an invariant part, which means it remains the same from project to project, and a user-specified part. The user-specified part declares the colors and triggers associated with various tokens of the language.

Colors, tokens, and triggers are closely associated. There are a number of standard colors defined in the template. When a language requires custom colors, they are stored in the configuration file. The developer then declares the color to be associated with each colored token. For each such case, the developer also declares a trigger action. For most tokens, the trigger action will be TokenTriggers.None. However, special trigger actions must be specifically declared. For example, left and right braces could declare TokenTriggers.MatchBraces.

IntelliSense Functionality

The Managed Babel System divides the programmatic labor of generating new IntelliSense functionality between the components of the scanner and the parser.

A fundamental unit of interchange between the components is the TextSpan structure. A text span specifies a starting and ending point within a document. Visual Studio does not attempt to interpret the semantics of every target language. However, it does understand how to color or highlight all of the text in a given text span.

Importantly, even though the Managed Babel System functions independently of yacc, it still uses yacc syntax and grammar in the creation of a parser configuration file.Therefore, the development of a new Visual Studio language service with Babel requires that the user understand how to program with yacc. The MPPG specification file, essentially equivalent to a yacc input file, must specify the language grammar, with the semantic actions resulting from processing the grammar invoking generic facilities of Visual Studio to achieve their purpose. Then, as further capabilities are added, the MPPG specification file must be enhanced to create objects that are the result of parsing with more information.

Babel

Babel's main task is to integrate the results of MPLEX-generated scanners and MPPG-generated parsers into Visual Studio language services. Babel does this by subclassing a Visual Studio shell class and extensively using the Microsoft.VisualStudio.Package namespace. Here is what Babel does:

  • It generates an abstract subclass of LanguageService for integration into Visual Studio language services. The subclass is responsible for creating parser, scanner, and error handler objects.

  • It generates a class called AuthoringSink that holds data about text that is common to almost all possible language services. It does not need to be specialized for particular languages.

  • It generates an abstract class in the Package namespace called AuthoringScope. Objects of this class hold information about the parse of a text.

  • It generates LineScanner, a Babel-generated class that implements the Colorizer interface. This extracts colorization information from the source text. This class mediates between Visual Studio requests and the user-generated scanner class.

The user does not have to modify every file generated by Babel, only those that connect customized scanners and parsers to the Babel-to-Visual Studio plumbing.

Note

Interfaces to Babel, MPPG, and MPLex are available by calling them from the .NET executable.

See Also

Show: