Latent Dirichlet Allocation

 

Updated: July 19, 2017

Use the Vowpal Wabbit library to perform VW LDA

Category: Text Analytics

You can use the Latent Dirichlet Allocation module to group otherwise unclassified text into a number of categories. Latent Dirichlet Allocation (LDA) is often used in natural language processing (NLP) to find texts that are similar. Another common term is topic modeling.

Generally speaking, LDA is not a method for classification per se, but uses a generative approach. What this means is that you don’t need to provide known class labels and then infer the patterns. Instead, the algorithm generates a probabilistic model that is used to identify groups of topics. You can use the probabilistic model to classify either existing training cases, or new cases that you provide to the model as input.

A generative model can be preferable because it avoids making any strong assumptions about the relationship between the text and categories, and uses only the distribution of words to mathematically model topics.

How to use LDA in an experiment

To use this module, you pass in a dataset that contains a column of text, either raw or preprocessed, and indicate how many categories you want to extract from the text. You can also set options for how you want punctuation handled, how large the terms are that you are extracting, and so forth. For details on how to prepare the text and configure the module, see How to Configure Latent Dirichlet Allocation.

When you run the experiment, the LDA module uses Bayes theorem to determine what topics might be associated with individual words. Words are not exclusively associated with any topics or groups; instead, each n-gram has a learned probability of being associated with any of the discovered classes.

Results

The module creates these outputs:

  • The source text, together with a score for each category

  • A feature matrix, containing extracted terms and coefficients for each category

  • A transformation, which you can save and reapply to new text used as input

Because this module uses the Vowpal Wabbit library, it is very fast. For more information about Vowpal Wabbit, see the GitHub repository which includes tutorials and an explanation of the algorithm..

Related Tasks

See the Cortana Analytics Gallery for examples of experiments that use natural language processing in Python, feature hashing, and other text processing techniques.

  1. Add the Latent Dirichlet Allocation module to your experiment.

  2. As input for the module, provide a dataset containing one or more text columns.

  3. For Target columns, choose one or more columns containing text to analyze.

    You can choose multiple columns but they must be of the string data type.

    In general, because LDA creates a large feature matrix from the text, you'll typically analyze a single text column.

  4. For Number of topics to model, type an integer between 1 and 1000 that indicates how many categories or topics you want to derive from the input text.

    By default, 5 topics are created.

  5. For N-grams, specify the maximum length of N-grams generated during hashing.

    The default is 2, meaning that both bigrams and unigrams are generated.

  6. Select the Normalize option to converting output values to probabilities. Therefore, rather than representing the transformed values as integers, values in the output and feature dataset would be transformed as follows:

    • Values in the dataset will be represented as a probability where P(topic|document).

    • Values in the feature topic matrix will be represented as a probability where P(word|topic).

  7. Select the option, Show all options, and then set it to TRUE if you want to view and then set additional advanced parameters.

    These parameters are specific to the Vowpal Wabbit implementation of LDA. There are some good tutorials about LDA in Vowpal Wabbit online, as well as the official Vowpal Wabbit Wiki.

    See this sample for examples in version 8 and use of VW in Azure ML.

    • Rho parameter. Provide a prior probability for the sparsity of topic distributions. Corresponds to VW's lda_rho parameter. You would use the value 1 if you expect that the distribution of words is flat; i.e, all words are assumed equiprobable. If you think most words appear sparsely, you might set it to a much lower value.

    • Alpha parameter. Specify a prior probability for the sparsity of per-document topic weights. Corresponds to VW's lda_alpha parameter.

    • Estimated number of documents. Type a number that represents your best estimate of the number of documents (rows) that will be processed. This lets the module allocate a hash table of sufficient size.

      Corresponds to the lda_D parameter in Vowpal Wabbit

    • Size of the batch. Type a number that indicates how many rows to include in each batch of text sent to Vowpal Wabbit.

      Corresponds to the batch_sz parameter in Vowpal Wabbit.

    • Initial value of iteration used in learning update schedule. Specify the starting value for the learning rate.

      Corresponds to the initial_t parameter in Vowpal Wabbit.

    • Power applied to the iteration during updates. Indicate the level of power applied to the iteration count during online updates.

      Corresponds to the power_t parameter in Vowpal Wabbit.

    • Number of passes over the data. Specify the number of times the algorithm will cycle over the data.

      Corresponds to the epoch_size parameter in Vowpal Wabbit.

  8. Select the option, Build dictionary of ngrams or Build dictionary of ngrams prior to LDA, if you want to create the ngram list in an initial pass, before classifying text.

    If you create the initial dictionary beforehand, you can later use the dictionary when reviewing the model. Being able to map results to text rather than numerical indices is generally easier for interpretation. However, saving the dictionary will take longer and use additional storage.

  9. For Maximum size of ngram dictionary, type the total number of rows that can be created in the ngram dictionary.

    This option is useful for controlling the size of the dictionary. However, if the number of ngrams in the input exceeds this size, collisions may occur.

  10. Run the experiment.

Results

The module has two outputs:

  • Transformed dataset. Contains the input text, and a specified number of discovered categories, together with the scores for each text example for each category.

  • Feature topic matrix. The leftmost column contains the extracted text feature, and there is a column for each category containing the score for that feature in that category.

For details and an example based on customer review text, see Understanding LDA Results.

For examples of how text analytics, see these experiments in the Model Gallery:

By default, the distributions of outputs for transformed dataset and feature-topic matrix are normalized as probabilities.

  • The transformed dataset is normalized as the conditional probability of topics given a document. In this case, the sum of each row equals 1.

  • The feature-topic matrix is normalized as the conditional probability of words given a topic. In this case, the sum of each column equals 1.

Occasionally the module might return an empty topic, which is most often caused by the pseudo-random initialization of the algorithm.

If this happens, you can try changing related parameters, such as the maximum size of the N-gram dictionary or the number of bits to use for feature hashing.

Latent Dirichlet Allocation (LDA) is often used for content-based topic modeling, which basically means learning categories from unclassified text. In content-based topic modeling, a topic is a distribution over words.

For example, assume that you have provided a corpus of customer reviews that includes many, many products. The text of reviews that have been submitted by many customers over time would contain many terms, some of which are used in multiple topics.

A topic that is identified by the LDA process might represent reviews for an individual Product A, or it might represent a group of product reviews. To LDA, the topic itself is just a probability distribution over time for a set of words.

Terms are rarely exclusive to any one product, but can refer to other products, or be general terms that apply to everything (“great”, “awful”). Other terms might be noise words. However, it is important to understand that the LDA method does not purport to capture all words in the universe, or to understand how words are related, aside from proabilities of co-occurrence. It can only group words that were used in the target domain.

After the term indexes have been computed, individual rows of text are compared using a distance-based similarity measure, to determine whether two pieces of text are like each other. For example, you might find that the product has multiple names that are strongly correlated. Or, you might find that strongly negative terms are usually associated with a particular product. You can use the similarity measure both to identify related terms and to create recommendations.

To illustrate how the Latent Dirichlet Allocation module works, the following example applies LDA with the default settings to the Book Review dataset provided in Azure Machine Learning Studio. The dataset contains a rating column, as well as the full comment text provided by users.

Sample Source Text

This table shows only a few representative examples.

During processing, the Latent Dirichlet Allocation module both cleans and analyzes the text, based on parameters you specify. For example, it can automatically tokenize the text and remove punctuation, and at the same time find the text features for each topic.

text
This book has its good points. If anything, it helps you put into words what you want from a supervisor....
I admit, I haven't finished this book. A friend recommended it to me as I have been having problems with insomnia...
Poorly written I tried reading this book but found it so turgid and poorly written that I put it down in frustration. ...
Since borrowing a dog-eared copy from friends who were passing it around a number of years ago, I have not been able to get my hands on this book which became a short-lived cult favorite
The plot of this book was interesting, and it could have been a good book. Unfortunately, it wasn't. The main problem for me was that ...

Transformed Sample Data

The following table contains the transformed dataset, based on the Book Review sample. The output contains the input text, and a specified number of discovered categories, together with the scores for each category.

In this example, we used the default value of 5 for Number of topics to model. Therefore, the LDA module creates five categories, which we can assume will correspond roughly with the original five-scale rating system.

The module also assigns a score to each entry for each of the five categories that represent topics. A score indicates the probability that the row should be assigned to a particular category.

Movie nameTopic 1Topic 2Topic 3Topic 4Topic 5
this book has its good points0.0016528920.0016528920.0016528920.0016528920.9933884
friend recommended it to me0.001980190.0019801980.99207910.0019801980.001980198
tried reading this book0.0024691350.0024691350.99012330.0024691350.002469135
borrowed it from friend0.99012320.0024691350.0024691350.0024691350.002469135
plot of this book was interesting0.0016528920.0016528920.99338840.0016528920.001652892

Feature Topic Matrix

The other output of the module is the feature topic matrix. This is a tabular dataset that contains the featurized text, , in column Feature, along with a score for each of the categories, in the remaining columns Topic 1, Topic 2, ...Topic N. The score represents the coefficient.

FeatureTopic 1Topic 2Topic 3Topic 4Topic 5
interesting0.02402820719831440.03546789547793750.3630518665769140.02766378243158930.660663576149515
was0.01714787295323970.08239690311086690.004529668779507890.04087145103192330.025077322689733
from0.01482242203492170.05050869814921090.004344233224610940.02733891262938240.0171484355106826
plot0.02274158893482120.04087094564893250.1827910413451910.0869370908128191 0.0169680136708971
reading0.02274158893482120.04087094564893250.1827910413451910.08693709081281910.0169680136708971
tried0.02697249791472110.0390262635517670.004437491067850870.06288298160882840.0235340728818033
me0.02626569451401340.03669413027519210.006568379751791380.03292145761600660.0214121851106808
to0.01410261032244620.0433599769192150.003886405318594470.03059259534400550.0228993750526364
it0.02644905471059510.03566744403118470.005417598978643140.03145393862502930.0140606468587681
friend0.01359713229609410.03461181714672340.004349994373507060.06665073218885360.018156863779311
points0.02274158893482120.03962338557190810.004046636014741120.03811565100190250.0337788009496797
good0.6518130738367830.05986463974441080.004468096919856170.03589756946460620.0138989124411206
its0.01853855886470780.1442539867831840.004088764164538660.05830492404414750.015442805566858
of0.01714167802456470.05593611804185860.01006339045449530.0870939301067230.0182573833869842
borrowed0.01714167802456470.05593611804185860.01006339045449530.0870939301067230.0182573833869842
has0.01714167802456470.05593611804185860.01006339045449530.0870939301067230.0182573833869842
book0.01431570479206810.0691459485350520.1840363401709830.05487573378239030.0156837976985903
recommended0.01614868484196890.03991433263995340.005501135302296420.0286371491427640.0147675139039372
this0.01614868484196890.03991433263995340.005501135302296420.0286371491427640.0147675139039372

LDA Transformation

The module also outputs the transformation that applies LDA to the dataset, as an ITransform interface.

You can save this transformation and re-use it for other datasets. This might be useful if you have trained on a large corpus and want to reuse the coefficients or categories.

Because each task has unique requirements and each corpus has different characteristics in terms of the distribution of terms, typically you cannot create a single LDA model that will meet all needs.

Instead, we recommend that you try changing the model parameters, use visualization to understand the results, and get the feedback of subject matter experts to ascertain whether the topics are useful.

Measure Accuracy and Coverage

Qualitative measures can also be useful for assessing the results. Measures often used to evaluate topic modeling include:

  • Accuracy. Are similar items really similar?

  • Diversity. Can the model discriminate between similar items when required for the business problem?

  • Scalability. Does it work on a wide range of text categories or only on a narrow target domain?

Refine Input Text

The accuracy of models based on LDA can often be improved by using natural language processing to clean, summarize and simplify, or categorize text. For example, the following techniques, all supported in Azure Machine Learning, might improve classification accuracy:

  • Stop word removal

  • Case normalization

  • Lemmatization or stemming

  • Named entity recognition

For more information, see Preprocess Text and Named Entity Recognition.

You might also use R or Python libraries for pre-processing of text, by using the Execute R Script or Execute Python Script modules.

NameTypeDescription
DatasetData TableInput dataset
NameTypeRangeOptionalDefaultDescription
Number of hash bitsInteger[1;31]Applies when the Show all options checkbox is not selected12Number of bits to use for feature hashing
Target column(s)Column SelectionRequiredStringFeatureTarget column name or index
Number of topics to modelInteger[1;1000]Required5Model the document distribution against N topics
N-gramsInteger[1;10]Required2Order of N-grams generated during hashing
NormalizeBooleanRequiredtrueNormalize output to probabilities. The transformed dataset will be P(topic|document) and the feature topic matrix will be P(word|topic).
Show all optionsBooleanTrue

False
RequiredFalsePresents additional parameters specific to Vowpal Wabbit online LDA
Rho parameterFloat[0.00001;1.0]Applies when the Show all options checkbox is selected0.01Rho parameter
Alpha parameterFloat[0.00001;1.0]Applies when the Show all options checkbox is selected0.01Alpha parameter
Estimated number of documentsInteger[1;int.MaxValue]Applies when the Show all options checkbox is selected1000Estimated number of documents (Corresponds to lda_D parameter)
Size of the batchInteger[1;1024]Applies when the Show all options checkbox is selected32Size of the batch
Initial value of iteration used in learning rate update scheduleInteger[0;int.MaxValue]Applies when the Show all options checkbox is selected0Initial value of iteration count used in learning rate update schedule (Corresponds to initial_t parameter)
Power applied to the iteration during updatesFloat[0.0;1.0]Applies when the Show all options checkbox is selected0.5Power applied to the iteration count during online updates (Corresponds to power_t parameter)
Number of training iterationsInteger[1;1024]Applies when the Show all options checkbox is selected25Number of training iterations
Build dictionary of ngramsBooleanTrue

False
Applies when the Show all options checkbox is not selectedTrueBuilds a dictionary of ngrams prior to computing LDA. Useful for model inspection and interpretation
Number of bits to use for feature hashingInteger[1;31]Applies when the option Build dictionary of ngrams is False12Number of bits to use during feature hashing
Maximum size of ngram dictionaryInteger[1;int.MaxValue]Applies when the option Build dictionary of ngrams is True20000Maximum size of the ngrams dictionary. If number of tokens in the input exceed this size, collisions may occur
Build dictionary of ngrams prior to LDABooleanTrue

False
Applies when the Show all options checkbox is selectedTrueBuilds a dictionary of ngrams prior to LDA. Useful for model inspection and interpretation
Maximum number of ngrams in dictionaryInteger[1;int.MaxValue]Applies when the option Build dictionary of ngrams is True and the Show all options checkbox is selected20000Maximum size of the dictionary. If number of tokens in the input exceed this size, collisions may occur
NameTypeDescription
Transformed datasetData TableOutput dataset
Feature topic matrixData TableFeature topic matrix produced by LDA
LDA transformationITransform interfaceTransformation that applies LDA to the dataset
ExceptionDescription
Error 0002Exception occurs if one or more specified columns of data set couldn't be found.
Error 0003Exception occurs if one or more of inputs are null or empty.
Error 0004Exception occurs if parameter is less than or equal to specific value.
Error 0017Exception occurs if one or more specified columns have type unsupported by current module.

Text Analytics
Feature Hashing
Named Entity Recognition
Score Vowpal Wabbit 7-4 Model
Train Vowpal Wabbit 7-4 Model
Train Vowpal Wabbit 8 Model

Show: