Shawn Wildermuth
MCW Technologies
First Published: April 2009
Updated June 2009 for May CTP
In the first article in this series, you were introduced to
the textual domain specific languages and why they can help you build solutions
that match the intent of project’s stakeholders. The second article walked you
through how to build your own textual domain specific language. But now you
should have a parse-able textual DSL. What is next? You are ready to consuming
your textual DSL. In this article I will explain how to consume instances of
your textual DSL and use the data to feed an application.
Consuming a Textual Domain Specific Language
You have now created a grammar for a domain specific
language. An example of the textual DSL looks like this:
ReportBundle "Daily Reports"
// Reports for the Board
Group "Status Reports"
Reports
"http://reportserver/reports/dailystatus.aspx"
"http://reportserver/reports/checklist.aspx"
End Reports
Recipients
"management@litware.org"
End Recipients
End Group
Group "Problem Reports"
Reports
- "http://reportserver/reports/problemreports.aspx"
"http://reportserver/reports/systemstatus.aspx"
End Reports
Recipients
"itstaff@litware.org"
"devmanagers@litware.org"
End Recipients
End Group
End ReportBundle
This textual DSL is now ready to power an application. In this case, an application that can take
this data and actually process the reports and e-mail them to the recipients
would be the logical choice. But how is that done?
The goal of most textual DSLs is to aid writing and
comprehension. The MGrammar is there to validate and enable parsing of the data
in the DSL. Ultimately you will need a way of parsing the DSL into something
that is easier to code against. There are two strategies for using the data
that the DSL describes: runtime parsing and using the toolchain.
At runtime, Oslo supports a number of classes that allow you
to parse the DSL and manipulate the results natively as shown in Figure 1:
.jpg)
Figure 1: Runtime Parsing
With Runtime Parsing, you will use the managed APIs to load a
grammar and the DSL and interrogate the resulting tree of information. Alternatively you can use the command-line
tools to turn the textual DSL information into data stored in a database as
shown in Figure 2:
.jpg)
Figure 2: Using the Tool Chain
In Oslo, there are number of tools that will allow you to
take your grammar and textual DSL and compile and store them into a standard
SQL Server database. In either case, the data structure of your DSL will be
stored in the database in a simple form (usually Table Per Type (TPT)). Let’s look further into the details of both
these techniques.
Using Your DSL
To consume the textual DSL there is a simple example called
“BundleRunner” that is included with this article. The sample will process the
report bundles using each of these techniques. In the BundleRunner, there is a simple
mechanism that reads a ReportBundle, retrieves the reports, and emails them to
the recipients. The BundleRunner application is a simple command-line program
(though could be refactored to a Windows Service or GUI tool).
The example uses a simple processing class that does the work
of retrieval of reports and sending of the emails. This processor takes an
interface (IBundleProvider) that will return a data structure containing the
details of the reports to process. This allows the BundleRunner to supply different
implementations of the provider for each of the example’s scenarios:
public interface IBundleProvider
{
ReportBundle Bundle { get; }
}
The data that the provider returns is a simple structure of
the data inside the DSL as shown in Figure 3:
.jpg)
Figure 3: Data Structures
The details of how the processing is done are trivial. What
matters most is how you are creating the data structures. The example will
create the data with two providers:
- FileBundleProvider: Uses
Runtime Parsing of the DSL to produce the data.
- DbBundleProvider: Reads
the data from a database.
First up is using the tool chain to insert your data into a
database so you can use the DbBundleProvider to build your bundle data. After
that you will see how to parse the textual DSL at runtime using the TextBundleProvider.
Using the Tool Chain
Oslo ships with a number of command-line tools that are used
to process and validate the “M” family of languages. In the case of building
your own grammar, you will use a chain of tools that can take your grammar and
textual DSL and turn them into real data to consume in your applications. This chain includes the M Compiler (m.exe)
and the M Executor (mx.exe). By using these tools to take your grammar and
textual DSL, you can automate the storage of information contained in the
textual DSLs.
The first of these is the M Compiler (m.exe). This tool
allows you to take your grammar and build a compiled grammar that can be used
to parse your textual DSL. You can think of a compiled grammar file like an
assembly in .NET. It’s a compiled version of the instructions that your Grammar
file contained.
The m.exe tool simply takes the Grammar file (and any other
grammar files that are required):
Using the M Compiler builds a compiled M file (.mx) file that
is then used to parse your DSL files with the M Executor.
The M Executor takes the compiled grammar file and the
textual DSL and exports either “M” or XAML (though more formats are likely to
appear in the future builds). To use the
M Executor, simply run the mx.exe tool
specifying the source file and a reference (-r) to the compiled grammar that
was created with the M Compiler (m.exe):
mx.exe DailyReports.bundle -r:ReportBundle.mx
This command takes the textual DSL (“DailyReports.bundle”)
and processes it with the compiled grammar (“ReportBundle.mx”). By default the M
Executor produces an M file containing MGraph. The MGraph is an approximate
representation of the Abstract Syntax Tree (AST) but since the AST and MGraph are
different structurally, there are differences. For example, here is the AST of your
DailyReports.bundle file:
ReportBundles{
{
Name{
"Daily Reports"
},
Groups{
{
{
GroupName{
"Status Reports"
},
Reports{
{
{
Url{
"http://reportserver/reports/dailystatus.aspx"
}
},
{
Url{
"http://reportserver/reports/checklist.aspx"
}
}
}
},
Recipients{
{
{
EMail{
"management@litware.org"
}
}
}
}
},
{
GroupName{
"Problem Reports"
},
Reports{
{
{
Url{
"http://reportserver/reports/problemreports.aspx"
}
},
{
Url{
"http://reportserver/reports/systemstatus.aspx"
}
}
}
},
Recipients{
{
{
EMail{
"itstaff@litware.org"
}
},
{
EMail{
"devmanagers@litware.org"
}
}
}
}
}
}
}
}
}
When the M Executor creates
the MGraph file, it conforms to the “M” language spec:
module DailyReports
{
ReportBundles
{
{
Name = "Daily Reports",
Groups
{
{
{
GroupName = "Status Reports",
Reports
{
{
{
Url = "http://reportserver/reports/dailystatus.aspx"
},
{
Url = "http://reportserver/reports/checklist.aspx"
}
}
},
Recipients
{
{
{
EMail = "management@litware.org"
}
}
}
},
{
GroupName = "Problem Reports",
Reports
{
{
{
Url = "http://reportserver/reports/problemreports.aspx"
},
{
Url = "http://reportserver/reports/systemstatus.aspx"
}
}
},
Recipients
{
{
{
EMail = "itstaff@litware.org"
},
{
EMail = "devmanagers@litware.org"
}
}
}
}
}
}
}
}
}
The MGraph representation is a well-formed MGraph version of
the AST. There are some subtle (but important) differences. For example, notice that the MGraph
representation has a module name and that simple structures are converted to
property assignment (e.g. EMail { “foo@bar.com” } becomes EMail = “foo@bar.com”).
In the MGraph that was created, you should notice that the
Module matches the file name of your textual DSL. Usually you will want to
override this by specifying the module name so that all textual DSL files will
end up in the same module. You can do
this by using the “–MModuleName” flag and specifying the name of the module.
You can use the short form of this flag (-m) to specify the module name as
shown below:
mx dailyreports.bundle -r:reportbundle.mx -m:Litware.Data.Reporting
At this point if your MGraph were very simple, you could
insert this directly into the database using the tool chain, but in most cases
you will want to create an MSchema file to define the structure of your
MGrammar projections. This will allow you to compile your M file for
preparation to storing them in a database. For example, your MSchema file
defines the types that are used in your MGraph production:
module Litware.Data.Reporting {
type ReportBundle
{
ReportBundleID : Integer32 => AutoNumber();
Name : Text#255;
Groups : ReportGroup* where value <= ReportGroups;
} where identity ReportBundleID;
ReportBundles : ReportBundle*;
type ReportGroup
{
ReportGroupID : Integer32 => AutoNumber();
GroupName : Text#255;
Reports : Report* where value <= Litware.Data.Reporting.Reports ;
Recipients : Recipient* where value <= Litware.Data.Reporting.Recipients;
} where identity ReportGroupID;
ReportGroups : ReportGroup*;
type Report
{
ReportID : Integer32 => AutoNumber();
Url : Text;
} where identity ReportID;
Reports : Report*;
type Recipient
{
RecipientID : Integer32 => AutoNumber();
EMail : Text;
} where identity RecipientID;
Recipients : Recipient*;
}
This MSchema file defines the four types and extents that your
M Graph uses: ReportBundles, ReportGroups, Reports and Recipients. The M Schema
file tells Oslo how to store the M Graph instances of the data in a data store.
With your M Schema defined, you can use both the M Graph created by the M
Executor and the M Schema you’ve defined to compile your “M” files for use in a
data store by using the “M” Compiler:
m.exe DailyReports.m ReportBundleSchema.m /t:TSql10
Notice you’re passing in an M Schema file (ReportBundleSchema.m)
and the daily reports M Graph file created by the M Grammer Executor (DailyReports.m).
The target flag (/t) specifies that the
output will be used. “TSql10” represents
a SQL Server 2008 compatible T-SQL.
By default this compiler will produce an image file (DailyReports.mx)
that can be used by the M Executor tool. Alternatively you can specify a
package flag (/p) to specify that instead of an image file, you want a SQL
Script:
m.exe DailyReports.m ReportBundleSchema.m /t:TSql10 /p:Script
This will produce a T-SQL Script that contains the databases
schema creation as well as the inserting of the data into the database. In most
cases, an image file is what you want if you are going to use the M Executor to
insert your data. Using the SQL Script
is useful if you are already handing database scripts in your projects in
specific ways. Otherwise, use the M
Executor to insert the data (and optionally the schema) directly into the
database:
mx.exe /i:dailyreports.mx /d:Reports /s:localhost /t:Integrated
Using the M Executor (mx.exe) allows you to take the image
file that was created with the M Compiler and insert into a database. The flags
used here are:
- /i: The input file
(usually an image file).
- /d: The name of the
database to insert into.
- /s: The SQL Server
name/address to use.
- /t: The Trusted
Connection Type. Alternatively you
can use /u and /p to pass in the username and password to the database.
Additionally, you can specify that the M Executor create the
database using additional flags:
mx.exe /i:dailyreports.mx /d:Reports /s:localhost /t:Integrated /c /f /ig
All the new flags are not necessary but make creating the
database much easier. The flags are:
- /c: Create the database
- /f: Forced install which
means that schema elements (extents, types and constraints) that exist in
the database will be overwritten.
- /ig: Ignore dependencies
when creating the database.
Once created, the database objects all use the module name as
the schema in the data. For example, the
database tables created map to each of the types in your schema (as well as
mapping tables) as seen in figure 4:
.jpg)
Figure 4: Database Schema
Created by M Executor
If you compare Figure 3 and 4 you will notice that the
structure of the data in the database is different than the object model you’re
mapping to, so the example will need to use standard data access techniques to
get your data from the database and use it at runtime to create the structure the
example needs.
The BundleRunner has an Entity Framework model based on the
database:
.jpg)
Figure 5: BundleRunner Entity
Model
Taking this entity model and using it to create your runtime
data is relatively straightforward.
public class DbBundleProvider : IBundleProvider
{
ReportsEntities ctx = new ReportsEntities();
string theBundleName;
ReportBundle theBundle;
// Accept the Bundle Name to find in the database
public DbBundleProvider(string bundleName)
{
theBundleName = bundleName;
}
// Process the bundle once its asked for
public ReportBundle Bundle
{
get
{
if (theBundle == null)
{
ProcessBundle();
}
return theBundle;
}
}
void ProcessBundle()
{
string reportPath =
"ReportBundles_Groups.ReportGroup.ReportGroups_Reports.Report";
string recipPath =
"ReportBundles_Groups.ReportGroup.ReportGroups_Recipients.Recipient";
// Find the bundles and eager load whole model
var qry = from b in ctx.ReportBundles
.Include(reportPath)
.Include(recipPath)
where b.Name == theBundleName
select b;
// Find first bundle (if any)
var bundle = qry.FirstOrDefault();
if (bundle != null)
{
// Build Structure
theBundle = new ReportBundle();
theBundle.Name = bundle.Name;
// Build the Groups
foreach (var bundleGroup in bundle.ReportBundles_Groups)
{
var newGroup = new ReportGroup()
{
Name = bundleGroup.ReportGroup.GroupName
};
theBundle.Groups.Add(newGroup);
// Add Reports
foreach (var groupReport in bundleGroup.ReportGroup.ReportGroups_Reports)
{
newGroup.Reports.Add(
new Report()
{
Url = groupReport.Report.Url
});
}
// Add Recipients
foreach (var groupRecipient in
bundleGroup.ReportGroup.ReportGroups_Recipients)
{
newGroup.Recipients.Add(
new Recipient()
{
EMail = groupRecipient.Recipient.EMail
});
}
}
}
}
}
In this case, the code simply
returning the bundle as an object graph of managed objects so that the
processor can get the reports and send them off to the recipients.
In order to get your DSL
data into the database, you would use the tool chain as shown in Figure 2. This
means compiling your grammar, processing your DSL into M utilizing the grammar,
then compiling schema with the M to store it into the database. This tool chain
represents a way of taking textual DSLs and turning them into useful data.
Storing this data in the
database can be very useful in managing the longer-lived functionality of the
processing of reports. But having to
store the data in the database is simply not necessary. There are times when
using the data purely at runtime is more valuable. That’s where runtime parsing
of your DSLs becomes the tool of choice.
Runtime Parsing
Everything that the tool chain can do you can accomplish with
code. In fact, Oslo exposes all those services to developers. To use runtime parsing, you will need to
reference the Microsoft.M.dll assembly, the System.DataFlow.dll and the
Xaml.dll assembly from the Oslo SDK directory. Once you do that you can compile
the M Grammar in your own code like so:
using Microsoft.M;
Stream mgStream = File.OpenRead("ReportBundle.mg");
CompilationResults results = Compiler.Compile(
new CompilerOptions
{
Sources =
{
new TextItem
{
Reader = new StreamReader(mgStream),
ContentType = TextItemType.MGrammar
}
}
});
When you use the M Executor to produce M, the code inside the
mx.exe tool simply reads the AST and produces valid MGraph statements that are
similar to the AST. When you are building runtimes or tools that parse textual
DSLs you can do the same. Again, the same code is available to you as the code
that the tool chain authors used.
The functionality for parsing your textual DSL starts with
the DynamicParser class (which is contained in the System.Dataflow.dll
assembly). To start out, you need to load up the compiled grammar by using the MImage
class. Then you can use the new grammar to create a parser (for your specific
language). Finally, specify the type of GraphBuilder to use by using the NodeGraphBuilder
class. The NodeGraphBuilder exposes a simple navigation through the resulting
parsed DSL to simplify our code (this was introduced in the Oslo May 09
CTP). Creating the parser is shown
below:
var grammar = new MImage("ReportBundle.mx");
theParser = grammar.ParserFactories["Litware.Data.Reporting.ReportBundle"].Create();
theParser.GraphBuilder = new NodeGraphBuilder();
Once loaded, the DynamicParser can parse your textual DSL. To
parse the textual DSL, the DynamicParser’s Parse method is used as shown below:
// Retrieve the root of the AST by performing Parsing
Node root = (Node)Parser.Parse(sourceFileName, null);
Because we specified the NodeGraphBuilder, the Parse method
returns a Node that represents the root of the parsed DSL file. The Node class
exposes a simple API for walking the AST.
To iterate over the AST, the framework uses the Node class.
This class contains methods and properties for walking the tree.
From the root node, you can get a list of descendants by
calling the ViewAllNodes method. This method returns a NodeCollection object. You
can retrieve this collection like so:
// Get list of Bundles
NodeCollection bundles = root.ViewAllNodes();
The NodeCollection supports the IEnumerable<Node>
interface so we can use LINQ against the collection. Because the example assumes
only a single bundle in your BundleRunner, you can retrieve the single node
like so:
// Get the node that represents the single Bundle
var bundleNode = bundles.First();
Because the bundle’s node contains properties (Name and Groups),
could get each of these by retrieving the element by ordinal like so:
var nameNode = bundleNode.ViewAllNodes().ElementAt(0);
But instead, you should find it easier (and more
refactorable) to use a property called Brand to determine the node type. The Brand is the name of the item in the AST
that you gave it in your grammar. For
example, the bundle’s name was specified in your grammar as “Name”. So by checking the Brand.Text value for that
name, you can be sure that the item in the AST is the right one. Here is an example of using a switch
statement to handle each of the nodes as they show up:
// Go through Children
foreach (var bundleChild in bundleNode.ViewAllNodes())
{
switch (bundleChild.Brand.Text)
{
case "Name":
{
// Get the first element to retrieve the name
theBundle.Name = (string)bundleChild.ViewAllNodes().ElementAt(0).AtomicValue;
break;
}
case "Groups":
{
// Get the first element to retrieve the Group
CreateGroup(bundleChild.ViewAllNodes().ElementAt(0));
break;
}
default:
{
break;
}
}
}
You should notice that because the AST is fairly nested that
the code uses the Name node and retrieves the first element (using ElementAt)
then casts its AtomicValue to a string. Only leaf nodes will have a valid
AtomicValue (whereas it will be null in all other cases). The AtomicValue
property therefore is used only when retrieving the final leaf of a node.
You can see the type of named objects to return by looking at
the resulting AST. In this case your example must get the named member called
“Groups” which happens to be a collection of your Group nodes. You can see this by looking at the AST in
figure 8:
.jpg)
Figure 6: AST Node Types
By using the Node class, you can iterate through an entire
AST. The entire parsing of your textual
DSL can be seen here:
// Perform the parsing of the AST
void Parse()
{
// Retrieve the root of the AST by performing Parsing
Node root = (Node)Parser.Parse(sourceFileName, null);
// Get list of Bundles
var bundles = root.ViewAllNodes();
// We expect a single bundle so throw an exception if necessary
if (bundles.Count == 0)
{
throw new InvalidOperationException(
"Expecting a single Bundle, found none.");
}
if (bundles.Count > 1)
{
throw new InvalidOperationException(
"Expecting a single Bundle, found more.");
}
// Get the node that represents the single Bundle
var bundleNode = bundles.First();
// Go through Children
foreach (var bundleChild in bundleNode.ViewAllNodes())
{
switch (bundleChild.Brand.Text)
{
case "Name":
{
// Get the first element to retrieve the name
theBundle.Name = (string)bundleChild.ViewAllNodes().ElementAt(0).AtomicValue;
break;
}
case "Groups":
{
// Get the first element to retrieve the Group
CreateGroup(bundleChild.ViewAllNodes().ElementAt(0));
break;
}
default:
{
break;
}
}
}
}
void CreateGroup(Node groupNode)
{
foreach (var info in groupNode.ViewAllNodes())
{
// Create a new Group to fill in
ReportGroup group = new ReportGroup();
// Navigate to the actual Group Contents
foreach (var groupPart in info.ViewAllNodes())
{
// Look at each part of the group
switch (groupPart.Brand.Text)
{
case "GroupName":
{
group.Name = (string)groupPart.ViewAllNodes().ElementAt(0).AtomicValue;
break;
}
case "Reports":
{
var contents = groupPart.ViewAllNodes().ElementAt(0);
foreach (var rptNode in contents.ViewAllNodes())
{
Report rpt = new Report();
rpt.Url = (string) rptNode.ViewAllNodes().ElementAt(0).AtomicValue;
group.Reports.Add(rpt);
}
break;
}
case "Recipients":
{
var contents = groupPart.ViewAllNodes().ElementAt(0);
foreach (var rcptNode in contents.ViewAllNodes())
{
Recipient rcpt = new Recipient();
rcpt.EMail = (string)rcptNode.ViewAllNodes().ElementAt(0).AtomicValue;
group.Recipients.Add(rcpt);
}
break;
}
default:
{
break;
}
}
}
Bundle.Groups.Add(group);
}
}
Where we are…
As you should know by now, building the textual DSL is only
half the task. Being able to work with the data that your textual DSL
exemplifies is the other half. The Oslo toolset provides a number of ways to
work with that data. While the Oslo tool chain allows you to process your
textual DSL into a data store for use in the rest of the system, this doesn’t
represent the only way. Being able to write your own code against the raw AST
provides a lower level of power to turn the textual DSL back into the intent of
the requirements.
By understanding the entire chain from why textual DSLs are
so powerful, how to build the grammars that represent the rules for those
textual DSLs and finally being able to use the data in the textual DSLs; you
should be able to build powerful, data-driven systems without relying on
technical and terse ways of capturing that data. Involving those important
users and stakeholders will help you as a developer keep the business of
writing compelling solutions without having to worry about as much of the
minutiae of the system requirements. You can write your applications to be
powered by the rules instead of encoding the rules.
Resources
·
Microsoft’s Oslo DevCenter
·
My Blog