Types of Learning and Overfitting

Supervised learning uses labeled training data (i.e., the target variable is defined) to guide the ML program toward superior forecasting accuracy. To identify earnings manipulators, for example, a large collection of attributes could be provided for known manipulators and for known nonmanipulators. A computer program could then be used to identify patterns that identify manipulators in another data set.

Multiple regression (discussed in an earlier topic review) is an example of supervised learning. Typical tasks for supervised learning include classification and regression. If the target variable is continuous, the model involved is a regression model.

Classification models are used in cases where the target variable is categorical or ordinal (e.g., ranking). Algorithms can be designed for binary classification (e.g., classifying companies as likely to default vs. not likely to default) or multicategory classification (e.g., a ratings class for bonds).

In unsupervised learning, the ML program is not given labeled training data; instead, inputs (i.e., features) are provided without any conclusions about those inputs. In the absence of any target variable, the program seeks out structure or interrelationships in the data. Clustering is an example of an unsupervised ML program.

Deep learning algorithms are used for complex tasks such as image recognition, natural language processing, and so on. Deep learning algorithms are based on neural networks, a group of ML algorithms applied to problems with significant nonlinearities. Programs that learn from their own output and prediction errors are called reinforcement learning algorithms.

Figure 3.1 summarizes the suitability of various ML algorithms.

Figure 3.1: ML Algorithm Types

Figure 3.2 shows the steps involved in selecting the appropriate ML algorithm to use, based on the problem to be solved and the characteristics of the data.

Figure 3.2: Choice of Appropriate ML Algorithm

We will discuss these ML algorithms in the remainder of this topic review.

Overfitting is an issue with supervised ML that results when a large number of features (i.e., independent variables) are included in the data sample. Overfitting has occurred when the noise in the target variables seems to improve the model fit (i.e., randomness is misperceived to be a pattern, resulting in high in sample R squared). Overfitting the model will decrease the accuracy of model forecasts on other (out of sample) data, overfit models do not generalize well to new data (i.e., out of sample R squared will be low).

To measure how well a model generalizes, data analysts create three nonoverlapping data sets: (1) training sample (used to develop the model), (2) validation sample (used for tuning the model), and (3) test sample (used for evaluating the model using new data). In sample prediction errors occur with the training and validation samples, while prediction errors in the test sample are known as the out of sample error. Data scientists then decompose these errors into the following:

• Bias error. This is the in sample error resulting from models with a poor fit.
• Variance error. This is the out of sample error resulting from overfitted models that do not generalize well. Base error.
• These are residual errors due to random noise.

A learning curve plots the accuracy rate (i.e., 1 − error rate) in the validation or test sample versus the size of the training sample. A robust, well generalizing model will show an improving accuracy rate as the sample size is increased, and the in sample and out of sample error rates will converge toward a desired accuracy level, as shown in the third panel of Figure 3.3. Models with high bias error (first panel) will see the error rates converge, but far below the desired level.

Models with high variance error (second panel) will see only the in sample accuracy rate converge toward the desired level, while the out of sample accuracy rate lags far behind.

Figure 3.3: Accuracy Rate Patterns

Variance error increases with model complexity, while bias error decreases with complexity. Typically, linear models tend to have high bias error while nonlinear models tend to have high variance error. Data scientists often express this as a tradeoff between cost and complexity.

An optimal level of complexity minimizes the total error and is a key part of successful model generalization.

To reduce the problem of overfitting, data scientists use complexity reduction and cross validation. In complexity reduction, a penalty is imposed to exclude features that are not meaningfully contributing to out of sample prediction accuracy. This penalty value increases with the number of independent variables (features) used by the model.

For a model to learn sufficiently, researchers must ensure that the training data set is both large and representative of the population. The validation sample, similarly, should be large and representative to properly test the model. A sampling technique known as cross validation estimates out of sample error rates directly from the validation sample.

In a k fold cross validation, the sample is randomly divided equally into k parts. The training sample comprises (k − 1) parts, with one part left for validation. Error is then measured for the model in each of the parts.

This process is repeated k times, and the average in sample and out of sample error rates are compiled.