TokenType Enumeration
Specifies the different types of tokens that can be identified and returned from a language service scanner.
Namespace: Microsoft.VisualStudio.Package
Assemblies: Microsoft.VisualStudio.Package.LanguageService.9.0 (in Microsoft.VisualStudio.Package.LanguageService.9.0.dll)
Microsoft.VisualStudio.Package.LanguageService (in Microsoft.VisualStudio.Package.LanguageService.dll)
Microsoft.VisualStudio.Package.LanguageService.10.0 (in Microsoft.VisualStudio.Package.LanguageService.10.0.dll)
Syntax
'Declaration
Public Enumeration TokenType
public enum TokenType
public enum class TokenType
type TokenType
public enum TokenType
Members
| Member name | Description | |
|---|---|---|
| Unknown | The token is an unknown type. This is typically used for any token not recognized by the parser and should be considered an error in the code being parsed. | |
| Text | General text; any text not identified as a specified token type. | |
| Keyword | A language keyword, an identifier that is reserved by the language. For example, in C#, do, while, foreach, if, and else, are all keywords. | |
| Identifier | An identifier or name. For example, the name of a variable, method, or class. In XML, this could be the name of a tag or attribute. | |
| String | A string. Typically defined as zero or more characters bounded by double quotes. | |
| Literal | A literal value (a character or number). For example, in C# or C++, this is a character bounded by single quotes, or a decimal or hexadecimal number. | |
| Operator | A punctuation character that has a specific meaning in a language. For example, in C#, arithmetic operators +, -, *, and /. In C++, pointer dereference operator ->, insertion operator >>, and extraction operation <<. In XML, assignment operator =. | |
| Delimiter | A token that operates as a separator between two language elements. For example, in C#, the period "." between class name and member name. In XML, the angle brackets surrounding a tag, < and >. | |
| WhiteSpace | A space, tab, or newline. Typically, a contiguous run of any whitespace is considered a single whitespace token. For example, the three spaces in "name this" would be treated as one whitespace token. | |
| LineComment | A line comment (comment is terminated at the end of the line). For example, in C# or C++, a comment is preceded by a //. In Visual Basic, this is a single tick '. | |
| Comment | A block comment. For example, in C# or C++, a comment is bounded by /* and */. In XML, the comment is bounded by <!-- and -->. |
Remarks
This enumeration is used in the TokenType structure to identify the type of token parsed. The TokenType structure is used in the IScanner scanner as implemented in a language service.
The types specified in this enumeration cover tokens that can appear in all common computer languages. More importantly, these are the token types that the default managed package framework language service classes understand. You can add additional types that your scanner support but you should first adhere to the meanings of the types described here.