Ordinal Regression

 

Updated: November 9, 2016

Creates an ordinal regression model

Category: Machine Learning / Initialize Model / Regression

You can use the Ordinal Regression module to create a regression model that can be used to predict ranked values. An example of ranked values might be survey responses that capture user’s preferred brands on a 1 to 5 scale, or the order of finishers in a race.

Ordinal regression is used when the label or target column contains numbers, but the numbers represent a ranking or order rather than a numeric measurement.

Predicting ordinal numbers requires a different algorithm than predicting the values of numbers on a continuous scale, because the numbers assigned to represent rank order do not have intrinsic scale.

For example, to predict students’ test scores, you would use a standard regression model, because students’ test scores vary on a continuous scale and can be measured. However, to predict their class ranking, you must use an ordinal regression model.

For more information about the research behind this algorithm, see this paper (downloadable PDF): http://papers.nips.cc/paper/3125-ordinal-regression-by-extended-binary-classification.pdf

  1. Add the Ordinal Regression Model module to your experiment.

  2. Add one of the binary classification models and connect it as an input to the Ordinal Regression Model module.

    System_CAPS_ICON_warning.jpg Warning

    To solve a ranking problem as a series of classification problems, the algorithm creates a series of extended training examples and trains against that extended set. This operation can be computationally expensive.

  3. You don’t need to set additional parameters on the Ordinal Regression Model itself – the algorithm has been pre-configured with the most effective parameters for solving a ranking problem.

  4. Add a Train Model module and connect the Ordinal Regression Model. For the training dataset, connect a dataset that contains examples of the ordinal values you want to predict.

    System_CAPS_ICON_warning.jpg Warning

    For purposes of processing, the ranks are assumed to have the order 1 to K, where 1 is lowest rank, and K is the highest rank. However, the Train Model module will work even if the semantics of your scale are reversed. For example, if in your original survey, 1 was the highest score and 5 is the lowest, it does not affect the processing of the model.

  5. In the Train Model module, select the column that contains the rank values.

    The rank values must be numerical values, but they need not be integers or positive numbers, as long as they represent a sequence.

  6. After you train the model, you can connect it to the Score Model module to make predictions.

    Alternatively, the untrained model can be passed to Cross-Validate Model for cross-validation against a labeled data set.

For examples of how ordinal regression is used in machine learning, see these sample experiments in the Cortana Intelligence Gallery.

  • In Step 2C of the Predictive Maintenance sample, Ordinal Regression is used to rank values output by a classification model, on the assumption that the value reflects the severity of the failure classification.

The ordinal regression algorithm used in this learner is implemented by extended binary classification, as described by the paper titled Ordinal Regression by Extended Binary Classification, by Ling Li and Hsuan-Tien Lin, in NIPS 2006.

Input Data Restrictions

You can use any numeric column as the target of an ordinal regression model, but in practice you should use only data that represents some sort of order or ranking.

The intervals between ranks are assumed to be unknown and the size of the interval does not matter to the model; however, the model assumes that the sequence of ranks follows the natural ordering of numbers.

The model itself does not assign any meaning to a particular scale. In other words, you might create one model in which 1 is a good rank and 10 is the worst, and in another model assume that 10 is the desired rank and 1 is the worst.

Algorithm

The training set (X,Y) consists of input vectors x and labels y. The labels represents ranks ranging from 1 to k in sequence: 1,2, … , K. The ranks are assumed to be ordered such that 1 is the lowest or the worst rank, and K is the best or highest rank.

The crux of the algorithm lies in modifying the given input features X and labels Y to use extended examples, and then using a binary classifier to solve the ordinal regression problem. The binary classifier is trained to give a yes/no answer to the question, “Is the rank greater than r?”

For example, for each case in the training set there are K-1 extended examples, and the maximum observed rank is K. The extended features are formed by appending the ith row of a K-1 x K-1 identity matrix to the input features for all i. The labels are given +1 for the first r-1 rows if its rank is r and -1 to the rest.

Sample Calculations

Let x1 be the training feature whose rank is 3, where the maximum observed rank is 5. The extended examples corresponding to this feature are as follows:

CaseTestResulting label
X11000Is rank greater than 1?Yes; therefore +1
X10100Is rank greater than 2?Yes; therefore +1
X10010Is rank greater than 3?No; therefore no additional feature
X10001Is rank greater than 4?No; therefore no additional feature
NameTypeDescription
Untrained binary classification modelILearner interfaceAn untrained binary classification model
NameTypeDescription
Untrained modelILearner interfaceAn untrained ordinal regression model

Regression
A-Z Module List

Show: