Best Practices for Regular Expressions in the .NET Framework
Updated: March 2011
The regular expression engine in the .NET Framework is a powerful, full-featured tool that processes text based on pattern matches rather than on comparing and matching literal text. In most cases, it performs pattern matching rapidly and efficiently. However, in some cases, the regular expression engine can appear to be very slow. In extreme cases, it can even appear to stop responding as it processes a relatively small input over the course of hours or even days.
This topic outlines some of the best practices that developers can adopt to ensure that their regular expressions achieve optimal performance. It contains the following sections:
Consider the Input Source
Handle Object Instantiation Appropriately
Take Charge of Backtracking
Capture Only When Necessary
Related Topics
Consider the Input Source
In general, regular expressions can accept two types of input: constrained or unconstrained. Constrained input is text that originates from a known or reliable source and follows a predefined format. Unconstrained input is text that originates from an unreliable source, such as a web user, and may not follow a predefined or expected format.
Regular expression patterns are typically written to match valid input. That is, developers examine the text that they want to match and then write a regular expression pattern that matches it. Developers then determine whether this pattern requires correction or further elaboration by testing it with multiple valid input items. When the pattern matches all presumed valid inputs, it is declared to be production-ready and can be included in a released application. This makes a regular expression pattern suitable for matching constrained input. However, it does not make it suitable for matching unconstrained input.
To match unconstrained input, a regular expression must be able to efficiently handle three kinds of text:
Text that matches the regular expression pattern.
Text that does not match the regular expression pattern.
Text that nearly matches the regular expression pattern.
The last text type is especially problematic for a regular expression that has been written to handle constrained input. If that regular expression also relies on extensive backtracking, the regular expression engine can spend an inordinate amount of time (in some cases, many hours or days) processing seemingly innocuous text.
.gif "Caution note") Caution |
|---|
The following example uses a regular expression that is prone to excessive backtracking and that is likely to reject valid email addresses. You should not use it in an email validation routine. If you would like a regular expression that validates email addresses, see How to: Verify That Strings Are in Valid E-Mail Format. |
For example, consider a very commonly used but extremely problematic regular expression for validating the alias of an email address. The regular expression ^[0-9A-Z]([-.\w]*[0-9A-Z])*$ is written to process what is considered to be a valid email address, which consists of an alphanumeric character or underscore, followed by zero or more characters that can be alphanumeric, periods, underscores, or hyphens. The regular expression must end with an alphanumeric character or underscore. However, as the following example shows, although this regular expression handles valid input easily, its performance is very inefficient when it is processing nearly valid input.
Imports System.Diagnostics
Imports System.Text.RegularExpressions
Module Example
Public Sub Main()
Dim sw As Stopwatch
Dim addresses() As String = { "AAAAAAAAAAA@anyco.com",
"AAAAAAAAAAaaaaaaaaaa!@anyco.com" }
' The following regular expression should not actually be used to
' validate an email address.
Dim pattern As String = "^[0-9A-Z]([-.\w]*[0-9A-Z])*$"
Dim input As String
For Each address In addresses
Dim mailBox As String = address.Substring(0, address.IndexOf("@"))
Dim index As Integer = 0
For ctr As Integer = mailBox.Length - 1 To 0 Step -1
index += 1
input = mailBox.Substring(ctr, index)
sw = Stopwatch.StartNew()
Dim m As Match = Regex.Match(input, pattern, RegexOptions.IgnoreCase)
sw.Stop()
if m.Success Then
Console.WriteLine("{0,2}. Matched '{1,25}' in {2}",
index, m.Value, sw.Elapsed)
Else
Console.WriteLine("{0,2}. Failed '{1,25}' in {2}",
index, input, sw.Elapsed)
End If
Next
Console.WriteLine()
Next
End Sub
End Module
' The example displays output similar to the following:
' 1. Matched ' A' in 00:00:00.0007122
' 2. Matched ' AA' in 00:00:00.0000282
' 3. Matched ' AAA' in 00:00:00.0000042
' 4. Matched ' AAAA' in 00:00:00.0000038
' 5. Matched ' AAAAA' in 00:00:00.0000042
' 6. Matched ' AAAAAA' in 00:00:00.0000042
' 7. Matched ' AAAAAAA' in 00:00:00.0000042
' 8. Matched ' AAAAAAAA' in 00:00:00.0000087
' 9. Matched ' AAAAAAAAA' in 00:00:00.0000045
' 10. Matched ' AAAAAAAAAA' in 00:00:00.0000045
' 11. Matched ' AAAAAAAAAAA' in 00:00:00.0000045
'
' 1. Failed ' !' in 00:00:00.0000447
' 2. Failed ' a!' in 00:00:00.0000071
' 3. Failed ' aa!' in 00:00:00.0000071
' 4. Failed ' aaa!' in 00:00:00.0000061
' 5. Failed ' aaaa!' in 00:00:00.0000081
' 6. Failed ' aaaaa!' in 00:00:00.0000126
' 7. Failed ' aaaaaa!' in 00:00:00.0000359
' 8. Failed ' aaaaaaa!' in 00:00:00.0000414
' 9. Failed ' aaaaaaaa!' in 00:00:00.0000758
' 10. Failed ' aaaaaaaaa!' in 00:00:00.0001462
' 11. Failed ' aaaaaaaaaa!' in 00:00:00.0002885
' 12. Failed ' Aaaaaaaaaaa!' in 00:00:00.0005780
' 13. Failed ' AAaaaaaaaaaa!' in 00:00:00.0011628
' 14. Failed ' AAAaaaaaaaaaa!' in 00:00:00.0022851
' 15. Failed ' AAAAaaaaaaaaaa!' in 00:00:00.0045864
' 16. Failed ' AAAAAaaaaaaaaaa!' in 00:00:00.0093168
' 17. Failed ' AAAAAAaaaaaaaaaa!' in 00:00:00.0185993
' 18. Failed ' AAAAAAAaaaaaaaaaa!' in 00:00:00.0366723
' 19. Failed ' AAAAAAAAaaaaaaaaaa!' in 00:00:00.1370108
' 20. Failed ' AAAAAAAAAaaaaaaaaaa!' in 00:00:00.1553966
' 21. Failed ' AAAAAAAAAAaaaaaaaaaa!' in 00:00:00.3223372
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;
public class Example
{
public static void Main()
{
Stopwatch sw;
string[] addresses = { "AAAAAAAAAAA@anyco.com",
"AAAAAAAAAAaaaaaaaaaa!@anyco.com" };
// The following regular expression should not actually be used to
// validate an email address.
string pattern = @"^[0-9A-Z]([-.\w]*[0-9A-Z])*$";
string input;
foreach (var address in addresses) {
string mailBox = address.Substring(0, address.IndexOf("@"));
int index = 0;
for (int ctr = mailBox.Length - 1; ctr >= 0; ctr--) {
index++;
input = mailBox.Substring(ctr, index);
sw = Stopwatch.StartNew();
Match m = Regex.Match(input, pattern, RegexOptions.IgnoreCase);
sw.Stop();
if (m.Success)
Console.WriteLine("{0,2}. Matched '{1,25}' in {2}",
index, m.Value, sw.Elapsed);
else
Console.WriteLine("{0,2}. Failed '{1,25}' in {2}",
index, input, sw.Elapsed);
}
Console.WriteLine();
}
}
}
// The example displays output similar to the following:
// 1. Matched ' A' in 00:00:00.0007122
// 2. Matched ' AA' in 00:00:00.0000282
// 3. Matched ' AAA' in 00:00:00.0000042
// 4. Matched ' AAAA' in 00:00:00.0000038
// 5. Matched ' AAAAA' in 00:00:00.0000042
// 6. Matched ' AAAAAA' in 00:00:00.0000042
// 7. Matched ' AAAAAAA' in 00:00:00.0000042
// 8. Matched ' AAAAAAAA' in 00:00:00.0000087
// 9. Matched ' AAAAAAAAA' in 00:00:00.0000045
// 10. Matched ' AAAAAAAAAA' in 00:00:00.0000045
// 11. Matched ' AAAAAAAAAAA' in 00:00:00.0000045
//
// 1. Failed ' !' in 00:00:00.0000447
// 2. Failed ' a!' in 00:00:00.0000071
// 3. Failed ' aa!' in 00:00:00.0000071
// 4. Failed ' aaa!' in 00:00:00.0000061
// 5. Failed ' aaaa!' in 00:00:00.0000081
// 6. Failed ' aaaaa!' in 00:00:00.0000126
// 7. Failed ' aaaaaa!' in 00:00:00.0000359
// 8. Failed ' aaaaaaa!' in 00:00:00.0000414
// 9. Failed ' aaaaaaaa!' in 00:00:00.0000758
// 10. Failed ' aaaaaaaaa!' in 00:00:00.0001462
// 11. Failed ' aaaaaaaaaa!' in 00:00:00.0002885
// 12. Failed ' Aaaaaaaaaaa!' in 00:00:00.0005780
// 13. Failed ' AAaaaaaaaaaa!' in 00:00:00.0011628
// 14. Failed ' AAAaaaaaaaaaa!' in 00:00:00.0022851
// 15. Failed ' AAAAaaaaaaaaaa!' in 00:00:00.0045864
// 16. Failed ' AAAAAaaaaaaaaaa!' in 00:00:00.0093168
// 17. Failed ' AAAAAAaaaaaaaaaa!' in 00:00:00.0185993
// 18. Failed ' AAAAAAAaaaaaaaaaa!' in 00:00:00.0366723
// 19. Failed ' AAAAAAAAaaaaaaaaaa!' in 00:00:00.1370108
// 20. Failed ' AAAAAAAAAaaaaaaaaaa!' in 00:00:00.1553966
// 21. Failed ' AAAAAAAAAAaaaaaaaaaa!' in 00:00:00.3223372
As the output from the example shows, the regular expression engine processes the valid email alias in about the same time interval regardless of its length. On the other hand, when the nearly valid email address has more than five characters, processing time approximately doubles for each additional character in the string. This means that a nearly valid 28-character string would take over an hour to process, and a nearly valid 33-character string would take nearly a day to process.
Because this regular expression was developed solely by considering the format of input to be matched, it fails to take account of input that does not match the pattern. This, in turn, can allow unconstrained input that nearly matches the regular expression pattern to significantly degrade performance.
To solve this problem, you can do the following:
When developing a pattern, you should consider how backtracking might affect the performance of the regular expression engine, particularly if your regular expression is designed to process unconstrained input. For more information, see the Take Charge of Backtracking section.
Thoroughly test your regular expression using invalid and near-valid input as well as valid input. To generate input for a particular regular expression randomly, you can use Rex, which is a regular expression exploration tool from Microsoft Research.
Handle Object Instantiation Appropriately
At the heart of the .NET Framework’s regular expression object model is the System.Text.RegularExpressions.Regex class, which represents the regular expression engine. Often, the single greatest factor that affects regular expression performance is the way in which the Regex engine is used. Defining a regular expression involves tightly coupling the regular expression engine with a regular expression pattern. That coupling process, whether it involves instantiating a Regex object by passing its constructor a regular expression pattern or calling a static method by passing it the regular expression pattern along with the string to be analyzed, is by necessity an expensive one.
Note
For a more detailed discussion of the performance implications of using interpreted and compiled regular expressions, see Optimizing Regular Expression Performance, Part II: Taking Charge of Backtracking in the BCL Team blog.
You can couple the regular expression engine with a particular regular expression pattern and then use the engine to match text in several ways:
You can call a static pattern-matching method, such as Regex.Match(String, String). This does not require instantiation of a regular expression object.
You can instantiate a Regex object and call an instance pattern-matching method of an interpreted regular expression. This is the default method for binding the regular expression engine to a regular expression pattern. It results when a Regex object is instantiated without an options argument that includes the Compiled flag.
You can instantiate a Regex object and call an instance pattern-matching method of a compiled regular expression. Regular expression objects represent compiled patterns when a Regex object is instantiated with an options argument that includes the Compiled flag.
You can create a special-purpose Regex object that is tightly coupled with a particular regular expression pattern, compile it, and save it to a standalone assembly. You do this by calling the Regex.CompileToAssembly method.
The particular way in which you call regular expression matching methods can have a significant impact on your application. The following sections discuss when to use static method calls, interpreted regular expressions, and compiled regular expressions to improve your application's performance.
| Important |
|---|
The form of the method call (static, interpreted, compiled) affects performance if the same regular expression is used repeatedly in method calls, or if an application makes extensive use of regular expression objects. |
Static Regular Expressions
Static regular expression methods are recommended as an alternative to repeatedly instantiating a regular expression object with the same regular expression. Unlike regular expression patterns used by regular expression objects, either the operation codes or the compiled Microsoft intermediate language (MSIL) from patterns used in instance method calls is cached internally by the regular expression engine.
For example, an event handler frequently calls another method to validate user input. This is reflected in the following code, in which a Button control's Click event is used to call a method named IsValidCurrency, which checks whether the user has entered a currency symbol followed by at least one decimal digit.
Public Sub OKButton_Click(sender As Object, e As EventArgs) _
Handles OKButton.Click
If Not String.IsNullOrEmpty(sourceCurrency.Text) Then
If RegexLib.IsValidCurrency(sourceCurrency.Text) Then
PerformConversion()
Else
status.Text = "The source currency value is invalid."
End If
End If
End Sub
public void OKButton_Click(object sender, EventArgs e)
{
if (! String.IsNullOrEmpty(sourceCurrency.Text))
if (RegexLib.IsValidCurrency(sourceCurrency.Text))
PerformConversion();
else
status.Text = "The source currency value is invalid.";
}
A very inefficient implementation of the IsValidCurrency method is shown in the following example. Note that each method call reinstantiates a Regex object with the same pattern. This, in turn, means that the regular expression pattern must be recompiled each time the method is called.
Imports System.Text.RegularExpressions
Public Module RegexLib
Public Function IsValidCurrency(currencyValue As String) As Boolean
Dim pattern As String = "\p{Sc}+\s*\d+"
Dim currencyRegex As New Regex(pattern)
Return currencyRegex.IsMatch(currencyValue)
End Function
End Module
using System;
using System.Text.RegularExpressions;
public class RegexLib
{
public static bool IsValidCurrency(string currencyValue)
{
string pattern = @"\p{Sc}+\s*\d+";
Regex currencyRegex = new Regex(pattern);
return currencyRegex.IsMatch(currencyValue);
}
}
You should replace this inefficient code with a call to the static Regex.IsMatch(String, String) method. This eliminates the need to instantiate a Regex object each time you want to call a pattern-matching method, and enables the regular expression engine to retrieve a compiled version of the regular expression from its cache.
Imports System.Text.RegularExpressions
Public Module RegexLib
Public Function IsValidCurrency(currencyValue As String) As Boolean
Dim pattern As String = "\p{Sc}+\s*\d+"
Return Regex.IsMatch(currencyValue, pattern)
End Function
End Module
using System;
using System.Text.RegularExpressions;
public class RegexLib
{
public static bool IsValidCurrency(string currencyValue)
{
string pattern = @"\p{Sc}+\s*\d+";
return Regex.IsMatch(currencyValue, pattern);
}
}
By default, the last 15 most recently used static regular expression patterns are cached. For applications that require a larger number of cached static regular expressions, the size of the cache can be adjusted by setting the Regex.CacheSize property.
The regular expression \p{Sc}+\s*\d+ that is used in this example verifies that the input string consists of a currency symbol and at least one decimal digit. The pattern is defined as shown in the following table.
Pattern |
Description |
|---|---|
\p{Sc}+ |
Match one or more characters in the Unicode Symbol, Currency category. |
\s* |
Match zero or more white-space characters. |
\d+ |
Match one or more decimal digits. |
Interpreted vs. Compiled Regular Expressions
Regular expression patterns that are not bound to the regular expression engine through the specification of the Compiled option are interpreted. When a regular expression object is instantiated, the regular expression engine converts the regular expression to a set of operation codes. When an instance method is called, the operation codes are converted to MSIL and executed by the JIT compiler. Similarly, when a static regular expression method is called and the regular expression cannot be found in the cache, the regular expression engine converts the regular expression to a set of operation codes and stores them in the cache. It then converts these operation codes to MSIL so that the JIT compiler can execute them. Interpreted regular expressions reduce startup time at the cost of slower execution time. Because of this, they are best used when the regular expression is used in a small number of method calls, or if the exact number of calls to regular expression methods is unknown but is expected to be small. As the number of method calls increases, the performance gain from reduced startup time is outstripped by the slower execution speed.
Regular expression patterns that are bound to the regular expression engine through the specification of the Compiled option are compiled. This means that, when a regular expression object is instantiated, or when a static regular expression method is called and the regular expression cannot be found in the cache, the regular expression engine converts the regular expression to an intermediary set of operation codes, which it then converts to MSIL. When a method is called, the JIT compiler executes the MSIL. In contrast to interpreted regular expressions, compiled regular expressions increase startup time but execute individual pattern-matching methods faster. As a result, the performance benefit that results from compiling the regular expression increases in proportion to the number of regular expression methods called.
To summarize, we recommend that you use interpreted regular expressions when you call regular expression methods with a specific regular expression relatively infrequently. You should use compiled regular expressions when you call regular expression methods with a specific regular expression relatively frequently. The exact threshold at which the slower execution speeds of interpreted regular expressions outweigh gains from their reduced startup time, or the threshold at which the slower startup times of compiled regular expressions outweigh gains from their faster execution speeds, is difficult to determine. It depends on a variety of factors, including the complexity of the regular expression and the specific data that it processes. To determine whether interpreted or compiled regular expressions offer the best performance for your particular application scenario, you can use the Stopwatch class to compare their execution times.
The following example compares the performance of compiled and interpreted regular expressions when reading the first ten sentences and when reading all the sentences in the text of Theodore Dreiser's The Financier. As the output from the example shows, when only ten calls are made to regular expression matching methods, an interpreted regular expression offers better performance than a compiled regular expression. However, a compiled regular expression offers better performance when a large number of calls (in this case, over 13,000) are made.
Imports System.Diagnostics
Imports System.IO
Imports System.Text.RegularExpressions
Module Example
Public Sub Main()
Dim pattern As String = "\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]"
Dim sw As Stopwatch
Dim match As Match
Dim ctr As Integer
Dim inFile As New StreamReader(".\Dreiser_TheFinancier.txt")
Dim input As String = inFile.ReadToEnd()
inFile.Close()
' Read first ten sentences with interpreted regex.
Console.WriteLine("10 Sentences with Interpreted Regex:")
sw = Stopwatch.StartNew()
Dim int10 As New Regex(pattern, RegexOptions.SingleLine)
match = int10.Match(input)
For ctr = 0 To 9
If match.Success Then
' Do nothing with the match except get the next match.
match = match.NextMatch()
Else
Exit For
End If
Next
sw.Stop()
Console.WriteLine(" {0} matches in {1}", ctr, sw.Elapsed)
' Read first ten sentences with compiled regex.
Console.WriteLine("10 Sentences with Compiled Regex:")
sw = Stopwatch.StartNew()
Dim comp10 As New Regex(pattern,
RegexOptions.SingleLine Or RegexOptions.Compiled)
match = comp10.Match(input)
For ctr = 0 To 9
If match.Success Then
' Do nothing with the match except get the next match.
match = match.NextMatch()
Else
Exit For
End If
Next
sw.Stop()
Console.WriteLine(" {0} matches in {1}", ctr, sw.Elapsed)
' Read all sentences with interpreted regex.
Console.WriteLine("All Sentences with Interpreted Regex:")
sw = Stopwatch.StartNew()
Dim intAll As New Regex(pattern, RegexOptions.SingleLine)
match = intAll.Match(input)
Dim matches As Integer = 0
Do While match.Success
matches += 1
' Do nothing with the match except get the next match.
match = match.NextMatch()
Loop
sw.Stop()
Console.WriteLine(" {0:N0} matches in {1}", matches, sw.Elapsed)
' Read all sentnces with compiled regex.
Console.WriteLine("All Sentences with Compiled Regex:")
sw = Stopwatch.StartNew()
Dim compAll As New Regex(pattern,
RegexOptions.SingleLine Or RegexOptions.Compiled)
match = compAll.Match(input)
matches = 0
Do While match.Success
matches += 1
' Do nothing with the match except get the next match.
match = match.NextMatch()
Loop
sw.Stop()
Console.WriteLine(" {0:N0} matches in {1}", matches, sw.Elapsed)
End Sub
End Module
' The example displays output like the following:
' 10 Sentences with Interpreted Regex:
' 10 matches in 00:00:00.0047491
' 10 Sentences with Compiled Regex:
' 10 matches in 00:00:00.0141872
' All Sentences with Interpreted Regex:
' 13,443 matches in 00:00:01.1929928
' All Sentences with Compiled Regex:
' 13,443 matches in 00:00:00.7635869
'
' >compare1
' 10 Sentences with Interpreted Regex:
' 10 matches in 00:00:00.0046914
' 10 Sentences with Compiled Regex:
' 10 matches in 00:00:00.0143727
' All Sentences with Interpreted Regex:
' 13,443 matches in 00:00:01.1514100
' All Sentences with Compiled Regex:
' 13,443 matches in 00:00:00.7432921
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;
public class Example
{
public static void Main()
{
string pattern = @"\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]";
Stopwatch sw;
Match match;
int ctr;
StreamReader inFile = new StreamReader(@".\Dreiser_TheFinancier.txt");
string input = inFile.ReadToEnd();
inFile.Close();
// Read first ten sentences with interpreted regex.
Console.WriteLine("10 Sentences with Interpreted Regex:");
sw = Stopwatch.StartNew();
Regex int10 = new Regex(pattern, RegexOptions.Singleline);
match = int10.Match(input);
for (ctr = 0; ctr <= 9; ctr++) {
if (match.Success)
// Do nothing with the match except get the next match.
match = match.NextMatch();
else
break;
}
sw.Stop();
Console.WriteLine(" {0} matches in {1}", ctr, sw.Elapsed);
// Read first ten sentences with compiled regex.
Console.WriteLine("10 Sentences with Compiled Regex:");
sw = Stopwatch.StartNew();
Regex comp10 = new Regex(pattern,
RegexOptions.Singleline | RegexOptions.Compiled);
match = comp10.Match(input);
for (ctr = 0; ctr <= 9; ctr++) {
if (match.Success)
// Do nothing with the match except get the next match.
match = match.NextMatch();
else
break;
}
sw.Stop();
Console.WriteLine(" {0} matches in {1}", ctr, sw.Elapsed);
// Read all sentences with interpreted regex.
Console.WriteLine("All Sentences with Interpreted Regex:");
sw = Stopwatch.StartNew();
Regex intAll = new Regex(pattern, RegexOptions.Singleline);
match = intAll.Match(input);
int matches = 0;
while (match.Success) {
matches++;
// Do nothing with the match except get the next match.
match = match.NextMatch();
}
sw.Stop();
Console.WriteLine(" {0:N0} matches in {1}", matches, sw.Elapsed);
// Read all sentnces with compiled regex.
Console.WriteLine("All Sentences with Compiled Regex:");
sw = Stopwatch.StartNew();
Regex compAll = new Regex(pattern,
RegexOptions.Singleline | RegexOptions.Compiled);
match = compAll.Match(input);
matches = 0;
while (match.Success) {
matches++;
// Do nothing with the match except get the next match.
match = match.NextMatch();
}
sw.Stop();
Console.WriteLine(" {0:N0} matches in {1}", matches, sw.Elapsed);
}
}
// The example displays the following output:
// 10 Sentences with Interpreted Regex:
// 10 matches in 00:00:00.0047491
// 10 Sentences with Compiled Regex:
// 10 matches in 00:00:00.0141872
// All Sentences with Interpreted Regex:
// 13,443 matches in 00:00:01.1929928
// All Sentences with Compiled Regex:
// 13,443 matches in 00:00:00.7635869
//
// >compare1
// 10 Sentences with Interpreted Regex:
// 10 matches in 00:00:00.0046914
// 10 Sentences with Compiled Regex:
// 10 matches in 00:00:00.0143727
// All Sentences with Interpreted Regex:
// 13,443 matches in 00:00:01.1514100
// All Sentences with Compiled Regex:
// 13,443 matches in 00:00:00.7432921
The regular expression pattern used in the example, \b(\w+((\r?\n)|,?\s))*\w+[.?:;!], is defined as shown in the following table.
Pattern |
Description |
|---|---|
\b |
Begin the match at a word boundary. |
\w+ |
Match one or more word characters. |
(\r?\n)|,?\s) |
Match either zero or one carriage return followed by a newline character, or zero or one comma followed by a white-space character. |
(\w+((\r?\n)|,?\s))* |
Match zero or more occurrences of one or more word characters that are followed either by zero or one carriage return and a newline character, or by zero or one comma followed by a white-space character. |
\w+ |
Match one or more word characters. |
[.?:;!] |
Match a period, question mark, colon, semicolon, or exclamation point. |
Regular Expressions: Compiled to an Assembly
The .NET Framework also enables you to create an assembly that contains compiled regular expressions. This moves the performance hit of regular expression compilation from run time to design time. However, it also involves some additional work: You must define the regular expressions in advance and compile them to an assembly. The compiler can then reference this assembly when compiling source code that uses the assembly’s regular expressions. Each compiled regular expression in the assembly is represented by a class that derives from Regex.
To compile regular expressions to an assembly, you call the Regex.CompileToAssembly(RegexCompilationInfo[], AssemblyName) method and pass it an array of RegexCompilationInfo objects that represent the regular expressions to be compiled, and an AssemblyName object that contains information about the assembly to be created.
We recommend that you compile regular expressions to an assembly in the following situations:
If you are a component developer who wants to create a library of reusable regular expressions.
If you expect your regular expression's pattern-matching methods to be called an indeterminate number of times -- anywhere from once or twice to thousands or tens of thousands of times. Unlike compiled or interpreted regular expressions, regular expressions that are compiled to separate assemblies offer performance that is consistent regardless of the number of method calls.
If you are using compiled regular expressions to optimize performance, you should not use reflection to create the assembly, load the regular expression engine, and execute its pattern-matching methods. This requires that you avoid building regular expression patterns dynamically, and that you specify any pattern-matching options (such as case-insensitive pattern matching) at the time the assembly is created. It also requires that you separate the code that creates the assembly from the code that uses the regular expression.
The following example shows how to create an assembly that contains a compiled regular expression. It creates an assembly named RegexLib.dll with a single regular expression class, SentencePattern, that contains the sentence-matching regular expression pattern used in the Interpreted vs. Compiled Regular Expressions section.
Imports System.Reflection
Imports System.Text.RegularExpressions
Module Example
Public Sub Main()
Dim SentencePattern As New RegexCompilationInfo("\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]",
RegexOptions.Multiline,
"SentencePattern",
"Utilities.RegularExpressions",
True)
Dim regexes() As RegexCompilationInfo = {SentencePattern}
Dim assemName As New AssemblyName("RegexLib, Version=1.0.0.1001, Culture=neutral, PublicKeyToken=null")
Regex.CompileToAssembly(regexes, assemName)
End Sub
End Module
using System;
using System.Reflection;
using System.Text.RegularExpressions;
public class Example
{
public static void Main()
{
RegexCompilationInfo SentencePattern =
new RegexCompilationInfo(@"\b(\w+((\r?\n)|,?\s))*\w+[.?:;!]",
RegexOptions.Multiline,
"SentencePattern",
"Utilities.RegularExpressions",
true);
RegexCompilationInfo[] regexes = { SentencePattern };
AssemblyName assemName = new AssemblyName("RegexLib, Version=1.0.0.1001, Culture=neutral, PublicKeyToken=null");
Regex.CompileToAssembly(regexes, assemName);
}
}
When the example is compiled to an executable and run, it creates an assembly named RegexLib.dll. The regular expression is represented by a class named Utilities.RegularExpressions.SentencePattern that is derived from Regex. The following example then uses the compiled regular expression to extract the sentences from the text of Theodore Dreiser's The Financier.
Imports System.IO
Imports System.Text.RegularExpressions
Imports Utilities.RegularExpressions
Module Example
Public Sub Main()
Dim pattern As New SentencePattern()
Dim inFile As New StreamReader(".\Dreiser_TheFinancier.txt")
Dim input As String = inFile.ReadToEnd()
inFile.Close()
Dim matches As MatchCollection = pattern.Matches(input)
Console.WriteLine("Found {0:N0} sentences.", matches.Count)
End Sub
End Module
' The example displays the following output:
' Found 13,443 sentences.
using System;
using System.IO;
using System.Text.RegularExpressions;
using Utilities.RegularExpressions;
public class Example
{
public static void Main()
{
SentencePattern pattern = new SentencePattern();
StreamReader inFile = new StreamReader(@".\Dreiser_TheFinancier.txt");
string input = inFile.ReadToEnd();
inFile.Close();
MatchCollection matches = pattern.Matches(input);
Console.WriteLine("Found {0:N0} sentences.", matches.Count);
}
}
// The example displays the following output:
// Found 13,443 sentences.
Take Charge of Backtracking
Ordinarily, the regular expression engine uses linear progression to move through an input string and compare it to a regular expression pattern. However, when indeterminate quantifiers such as *, +, and ? are used in a regular expression pattern, the regular expression engine may give up a portion of successful partial matches and return to a previously saved state in order to search for a successful match for the entire pattern. This process is known as backtracking.
Note
For more information on backtracking, see Details of Regular Expression Behavior and Backtracking. For a detailed discussion of backtracking, see Optimizing Regular Expression Performance, Part II: Taking Charge of Backtracking in the BCL Team blog.
Support for backtracking gives regular expressions power and flexibility. It also places the responsibility for controlling the operation of the regular expression engine in the hands of regular expression developers. Because developers are often not aware of this responsibility, their misuse of backtracking or reliance on excessive backtracking often plays the most significant role in degrading regular expression performance. In a worst-case scenario, execution time can double for each additional character in the input string. In fact, by using backtracking excessively, it is easy to create the programmatic equivalent of an endless loop if input nearly matches the regular expression pattern; the regular expression engine may take hours or even days to process a relatively short input string.
Often, applications pay a performance penalty for using backtracking despite the fact that backtracking is not essential for a match. For example, the regular expression \b\p{Lu}\w*\b matches all words that begin with an uppercase character, as the following table shows.
Pattern |
Description |
\b |
Begin the match at a word boundary. |
\p{Lu} |
Match an uppercase character. |
\w* |
Match zero or more word characters. |
\b |
End the match at a word boundary. |
Because a word boundary is not the same as, or a subset of, a word character, there is no possibility that the regular expression engine will cross a word boundary when matching word characters. This means that for this regular expression, backtracking can never contribute to the overall success of any match -- it can only degrade performance, because the regular expression engine is forced to save its state for each successful preliminary match of a word character.
If you determine that backtracking is not necessary, you can disable it by using the (?>subexpression) language element. The following example parses an input string by using two regular expressions. The first, \b\p{Lu}\w*\b, relies on backtracking. The second, \b\p{Lu}(?>\w*)\b, disables backtracking. As the output from the example shows, they both produce the same result.
Imports System.Text.RegularExpressions
Module Example
Public Sub Main()
Dim input As String = "This this word Sentence name Capital"
Dim pattern As String = "\b\p{Lu}\w*\b"
For Each match As Match In Regex.Matches(input, pattern)
Console.WriteLine(match.Value)
Next
Console.WriteLine()
pattern = "\b\p{Lu}(?>\w*)\b"
For Each match As Match In Regex.Matches(input, pattern)
Console.WriteLine(match.Value)
Next
End Sub
End Module
' The example displays the following output:
' This
' Sentence
' Capital
'
' This
' Sentence
' Capital
using System;
using System.Text.RegularExpressions;
public class Example
{
public static void Main()
{
string input = "This this word Sentence name Capital";
string pattern = @"\b\p{Lu}\w*\b";
foreach (Match match in Regex.Matches(input, pattern))
Console.WriteLine(match.Value);
Console.WriteLine();
pattern = @"\b\p{Lu}(?>\w*)\b";
foreach (Match match in Regex.Matches(input, pattern))
Console.WriteLine(match.Value);
}
}
// The example displays the following output:
// This
// Sentence
// Capital
//
// This
// Sentence
// Capital
In many cases, backtracking is essential for matching a regular expression pattern to input text. In these cases, excessive backtracking can severely degrade performance and create the impression that an application has stopped responding. In particular, this happens when quantifiers are nested and the text that matches the outer subexpression is a subset of the text that matches the inner subexpression.
For example, the regular expression pattern ^[0-9A-Z]([-.\w]*[0-9A-Z])*\$$ is intended to match a part number that consists of at least one alphanumeric character. Any additional characters can consist of an alphanumeric character, a hyphen, an underscore, or a period, though the last character must be alphanumeric. A dollar sign terminates the part number. In some cases, this regular expression pattern can exhibit extremely poor performance because quantifiers are nested, and because the subexpression [0-9A-Z] is a subset of the subexpression [-.\w]*.
In these cases, you can optimize regular expression performance by removing the nested quantifiers and replacing the outer subexpression with a zero-width lookahead or lookbehind assertion. Lookahead and lookbehind assertions are anchors; they do not move the pointer in the input string, but instead look ahead or behind to check whether a specified condition is met. For example, the part number regular expression can be rewritten as ^[0-9A-Z][-.\w]*(?<=[0-9A-Z])\$$. This regular expression pattern is defined as shown in the following table.
Pattern |
Description |
|---|---|
^ |
Begin the match at the beginning of the input string. |
[0-9A-Z] |
Match an alphanumeric character. The part number must consist of at least this character. |
[-.\w]* |
Match zero or more occurrences of any word character, hyphen, or period. |
\$ |
Match a dollar sign. |
(?<=[0-9A-Z]) |
Look ahead of the ending dollar sign to ensure that the previous character is alphanumeric. |
$ |
End the match at the end of the input string. |
The following example illustrates the use of this regular expression to match an array containing possible part numbers.
Imports System.Text.RegularExpressions
Module Example
Public Sub Main()
Dim pattern As String = "^[0-9A-Z][-.\w]*(?<=[0-9A-Z])\$$"
Dim partNos() As String = { "A1C$", "A4", "A4$", "A1603D$",
"A1603D#" }
For Each input As String In partNos
Dim match As Match = Regex.Match(input, pattern)
If match.Success Then
Console.WriteLine(match.Value)
Else
Console.WriteLine("Match not found.")
End If
Next
End Sub
End Module
' The example displays the following output:
' A1C$
' Match not found.
' A4$
' A1603D$
' Match not found.
using System;
using System.Text.RegularExpressions;
public class Example
{
public static void Main()
{
string pattern = @"^[0-9A-Z][-.\w]*(?<=[0-9A-Z])\$$";
string[] partNos = { "A1C$", "A4", "A4$", "A1603D$", "A1603D#" };
foreach (var input in partNos) {
Match match = Regex.Match(input, pattern);
if (match.Success)
Console.WriteLine(match.Value);
else
Console.WriteLine("Match not found.");
}
}
}
// The example displays the following output:
// A1C$
// Match not found.
// A4$
// A1603D$
// Match not found.
The regular expression language in the .NET Framework includes the following language elements that you can use to eliminate nested quantifiers. For more information, see Grouping Constructs.
Language element |
Description |
|---|---|
(?=subexpression) |
Zero-width positive lookahead. Look ahead of the current position to determine whether subexpression matches the input string. |
(?!subexpression) |
Zero-width negative lookahead. Look ahead of the current position to determine whether subexpression does not match the input string. |
(?<=subexpression) |
Zero-width positive lookbehind. Look behind the current position to determine whether subexpression matches the input string. |
(?<!subexpression) |
Zero-width negative lookbehind. Look behind the current position to determine whether subexpression does not match the input string. |
Capture Only When Necessary
Regular expressions in the .NET Framework support a number of grouping constructs, which let you group a regular expression pattern into one or more subexpressions. The most commonly used grouping constructs in the .NET Framework regular expression language are (subexpression), which defines a numbered capturing group, and (?<name>subexpression), which defines a named capturing group. Grouping constructs are essential for creating backreferences and for defining a subexpression to which a quantifier is applied.
However, the use of these language elements has a cost. They cause the GroupCollection object returned by the Match.Groups property to be populated with the most recent unnamed or named captures, and if a single grouping construct has captured multiple substrings in the input string, they also populate the CaptureCollection object returned by the Group.Captures property of a particular capturing group with multiple Capture objects.
Often, grouping constructs are used in a regular expression only so that quantifiers can be applied to them, and the groups captured by these subexpressions are not subsequently used. For example, the regular expression \b(\w+[;,]?\s?)+[.?!] is designed to capture an entire sentence. The following table describes the language elements in this regular expression pattern and their effect on the Match object's Match.Groups and Group.Captures collections.
Pattern |
Description |
|---|---|
\b |
Begin the match at a word boundary. |
\w+ |
Match one or more word characters. |
[;,]? |
Match zero or one comma or semicolon. |
\s? |
Match zero or one white-space character. |
(\w+[;,]?\s?)+ |
Match one or more occurrences of one or more word characters followed by an optional comma or semicolon followed by an optional white-space character. This defines the first capturing group, which is necessary so that the combination of multiple word characters (that is, a word) followed by an optional punctuation symbol will be repeated until the regular expression engine reaches the end of a sentence. |
[.?!] |
Match a period, question mark, or exclamation point. |
As the following example shows, when a match is found, both the GroupCollection and CaptureCollection objects are populated with captures from the match. In this case, the capturing group (\w+[;,]?\s?) exists so that the + quantifier can be applied to it, which enables the regular expression pattern to match each word in a sentence. Otherwise, it would match the last word in a sentence.
Imports System.Text.RegularExpressions
Module Example
Public Sub Main()
Dim input As String = "This is one sentence. This is another."
Dim pattern As String = "\b(\w+[;,]?\s?)+[.?!]"
For Each match As Match In Regex.Matches(input, pattern)
Console.WriteLine("Match: '{0}' at index {1}.",
match.Value, match.Index)
Dim grpCtr As Integer = 0
For Each grp As Group In match.Groups
Console.WriteLine(" Group {0}: '{1}' at index {2}.",
grpCtr, grp.Value, grp.Index)
Dim capCtr As Integer = 0
For Each cap As Capture In grp.Captures
Console.WriteLine(" Capture {0}: '{1}' at {2}.",
capCtr, cap.Value, cap.Index)
capCtr += 1
Next
grpCtr += 1
Next
Console.WriteLine()
Next
End Sub
End Module
' The example displays the following output:
' Match: 'This is one sentence.' at index 0.
' Group 0: 'This is one sentence.' at index 0.
' Capture 0: 'This is one sentence.' at 0.
' Group 1: 'sentence' at index 12.
' Capture 0: 'This ' at 0.
' Capture 1: 'is ' at 5.
' Capture 2: 'one ' at 8.
' Capture 3: 'sentence' at 12.
'
' Match: 'This is another.' at index 22.
' Group 0: 'This is another.' at index 22.
' Capture 0: 'This is another.' at 22.
' Group 1: 'another' at index 30.
' Capture 0: 'This ' at 22.
' Capture 1: 'is ' at 27.
' Capture 2: 'another' at 30.
using System;
using System.Text.RegularExpressions;
public class Example
{
public static void Main()
{
string input = "This is one sentence. This is another.";
string pattern = @"\b(\w+[;,]?\s?)+[.?!]";
foreach (Match match in Regex.Matches(input, pattern)) {
Console.WriteLine("Match: '{0}' at index {1}.",
match.Value, match.Index);
int grpCtr = 0;
foreach (Group grp in match.Groups) {
Console.WriteLine(" Group {0}: '{1}' at index {2}.",
grpCtr, grp.Value, grp.Index);
int capCtr = 0;
foreach (Capture cap in grp.Captures) {
Console.WriteLine(" Capture {0}: '{1}' at {2}.",
capCtr, cap.Value, cap.Index);
capCtr++;
}
grpCtr++;
}
Console.WriteLine();
}
}
}
// The example displays the following output:
// Match: 'This is one sentence.' at index 0.
// Group 0: 'This is one sentence.' at index 0.
// Capture 0: 'This is one sentence.' at 0.
// Group 1: 'sentence' at index 12.
// Capture 0: 'This ' at 0.
// Capture 1: 'is ' at 5.
// Capture 2: 'one ' at 8.
// Capture 3: 'sentence' at 12.
//
// Match: 'This is another.' at index 22.
// Group 0: 'This is another.' at index 22.
// Capture 0: 'This is another.' at 22.
// Group 1: 'another' at index 30.
// Capture 0: 'This ' at 22.
// Capture 1: 'is ' at 27.
// Capture 2: 'another' at 30.
When you use subexpressions only to apply quantifiers to them, and you are not interested in the captured text, you should disable group captures. For example, the (?:subexpression) language element prevents the group to which it applies from capturing matched substrings. In the following example, the regular expression pattern from the previous example is changed to \b(?:\w+[;,]?\s?)+[.?!]. As the output shows, it prevents the regular expression engine from populating the GroupCollection and CaptureCollection collections.
Imports System.Text.RegularExpressions
Module Example
Public Sub Main()
Dim input As String = "This is one sentence. This is another."
Dim pattern As String = "\b(?:\w+[;,]?\s?)+[.?!]"
For Each match As Match In Regex.Matches(input, pattern)
Console.WriteLine("Match: '{0}' at index {1}.",
match.Value, match.Index)
Dim grpCtr As Integer = 0
For Each grp As Group In match.Groups
Console.WriteLine(" Group {0}: '{1}' at index {2}.",
grpCtr, grp.Value, grp.Index)
Dim capCtr As Integer = 0
For Each cap As Capture In grp.Captures
Console.WriteLine(" Capture {0}: '{1}' at {2}.",
capCtr, cap.Value, cap.Index)
capCtr += 1
Next
grpCtr += 1
Next
Console.WriteLine()
Next
End Sub
End Module
' The example displays the following output:
' Match: 'This is one sentence.' at index 0.
' Group 0: 'This is one sentence.' at index 0.
' Capture 0: 'This is one sentence.' at 0.
'
' Match: 'This is another.' at index 22.
' Group 0: 'This is another.' at index 22.
' Capture 0: 'This is another.' at 22.
using System;
using System.Text.RegularExpressions;
public class Example
{
public static void Main()
{
string input = "This is one sentence. This is another.";
string pattern = @"\b(?:\w+[;,]?\s?)+[.?!]";
foreach (Match match in Regex.Matches(input, pattern)) {
Console.WriteLine("Match: '{0}' at index {1}.",
match.Value, match.Index);
int grpCtr = 0;
foreach (Group grp in match.Groups) {
Console.WriteLine(" Group {0}: '{1}' at index {2}.",
grpCtr, grp.Value, grp.Index);
int capCtr = 0;
foreach (Capture cap in grp.Captures) {
Console.WriteLine(" Capture {0}: '{1}' at {2}.",
capCtr, cap.Value, cap.Index);
capCtr++;
}
grpCtr++;
}
Console.WriteLine();
}
}
}
// The example displays the following output:
// Match: 'This is one sentence.' at index 0.
// Group 0: 'This is one sentence.' at index 0.
// Capture 0: 'This is one sentence.' at 0.
//
// Match: 'This is another.' at index 22.
// Group 0: 'This is another.' at index 22.
// Capture 0: 'This is another.' at 22.
You can disable captures in one of the following ways:
Use the (?:subexpression) language element. This element prevents the capture of matched substrings in the group to which it applies. It does not disable substring captures in any nested groups.
Use the ExplicitCapture option. It disables all unnamed or implicit captures in the regular expression pattern. When you use this option, only substrings that match named groups defined with the (?<name>subexpression) language element can be captured. The ExplicitCapture flag can be passed to the options parameter of a Regex class constructor or to the options parameter of a Regex static matching method.
Use the n option in the (?imnsx) language element. This option disables all unnamed or implicit captures from the point in the regular expression pattern at which the element appears. Captures are disabled either until the end of the pattern or until the (-n) option enables unnamed or implicit captures. For more information, see Miscellaneous Constructs.
Use the n option in the (?imnsx:subexpression) language element. This option disables all unnamed or implicit captures in subexpression. Captures by any unnamed or implicit nested capturing groups are disabled as well.
Related Topics
Title |
Description |
|---|---|
Examines the implementation of the regular expression engine in the .NET Framework. The topic focuses on the flexibility of regular expressions and explains the developer's responsibility for ensuring the efficient and robust operation of the regular expression engine. |
|
Explains what backtracking is and how it affects regular expression performance, and examines language elements that provide alternatives to backtracking. |
|
Describes the elements of the regular expression language in the .NET Framework and provides links to detailed documentation for each language element. |
Change History
Date |
History |
Reason |
|---|---|---|
|
March 2011 |
Added topic. |
Information enhancement. |