Detect Languages
Updated: February 23, 2017
Detects the language of each line in the input file
Category: Text Analytics
You can use the Detect Languages module to analyze text input and identify the language associated with each record in the input. The language detection algorithm can identify many different languages.
You specify which text column to analyze, and how many languages to detect. The algorithm will analyze each row of text, and assign a language score for each language. The language in the first result column is the language that is most likely.
Add the Detect Languages module to your experiment, and connect a dataset that has a text column including multiple languages.
It is not necessary that the input contain any labels; the language detection algorithm works using morphological and lexical features of the supported languages.
For Text column, choose the column you want to analyze.
For Upper bound on number of languages to detect, indicate the maximum number of languages to detect.
Setting an upper bound on the number of languages can improve performance.
Run the experiment.
The Detect Languages module outputs a language identifier and score for each row. If there are equi-probable language matches, several languages might be listed, with a score for each.
For example, in these results, the first two columns col1 and language label are input columns, already present in your dataset. The remaining columns are generated by the Detect Languages module.
| Col1 | Language label | Col1 Language | Col1 Iso6391 Language | Col1 Iso6391 Language Score |
|---|---|---|---|---|
| It was a wonderful hotel with a friendly staff and good service | English | English | en | 100 |
| Es war ein wunderbares Hotel mit freundlichem Personal und guter service | German | German | de | 100 |
| C’est un magnifique hôtel avec un personnel sympathique et un service de qualité | French | French | fr | 100 |
| Det var et dejligt hotel med et venligt personale og god service | Danish | Danish | nl | 100 |
| Va ser un magnífic hotel amb un personal amable i bon servei | Catalan | Catalan | ca | 92.30769348 |
| とても素敵なホテルで、スタッフは親切で、サービスもよかった | Japanese | (Unknown) | 0 | |
| qu mebpa'mey naQ friendly QaQ chavmoH je | Klingon | French | fr | 77.5 |
For a general idea of the languages that potentially can be detected, refer to Bing Translator.
Many more languages can be detected than Azure Machine Learning currently supports for advanced text analytics. We recommend that you use the results of Detect Languages to filter the results that you send to other modules that require language-specific processing.
| Name | Type | Description |
|---|---|---|
| Dataset | Data Table | The input |
| Name | Type | Range | Optional | Default | Description |
|---|---|---|---|---|---|
| Upper bound on number of languages to detect | Integer | [1;184] | Required | 1 | Upper bound on number of languages to detect. |
| Text column | ColumnSelection | Required | Name or one-based index of text column. |
| Name | Type | Description |
|---|---|---|
| Results dataset | Data Table | The result |
| Exception | Description |
|---|---|
| Error 0003 | Exception occurs if one or more of inputs are null or empty. |
| Error 0010 | Exception occurs if input datasets have column names that should match but do not. |
| Error 0016 | Exception occurs if input datasets passed to the module should have compatible column types but do not. |
| Error 0008 | Exception occurs if parameter is not in range. |