Supervised Learning Algorithms

We will now describe some of the common supervised ML algorithms and their applications:

1. Penalized regressions. Penalized regression models reduce the problem of overfitting by imposing a penalty based on the number of features used by the model. The penalty value increases with the number of independent variables (features) used.

Imposing such a penalty can exclude features that are not meaningfully contributing to out of sample prediction accuracy (i.e., it makes the model more parsimonious). Penalized regression models seek to minimize the sum of square errors (SSE) as well as a penalty value. Least absolute shrinkage and selection operator (LASSO).

This is a popular penalized regression model. In addition to minimizing SSE, LASSO minimizes the sum of the absolute values of the slope coefficients. In such a framework, there is a tradeoff between reducing the SSE (by increasing the number of features) and the penalty imposed on the inclusion of more features.

Through optimization, LASSO automatically eliminates the least predictive features. A penalty term, λ (lambda), is the hyperparameter that determines the balance between overfitting the model and keeping it parsimonious.

A related method to reduce statistical variability in a high dimension data estimation problem is regularization. Regularization forces the beta coefficients of nonperforming features toward zero.

Investment analysts use LASSO to build parsimonious models. Regularization can be applied to nonlinear models, such as the estimation of a stable covariance matrix that can be used for mean variance optimization.

Applications of SVM in investment management include classifying debt issuers into likely to default versus not likely to default issuers, stocks to short versus not to short, and even classifying text (from news articles or company press releases) as positive or negative.

1. K nearest neighbor (KNN). More commonly used in classification (but sometimes in regression), this technique is used to classify an observation based on nearness to the observations in the training sample. The researcher specifies the value of k, the hyperparameter, triggering the algorithm to look for the k observations in the sample that are closest to the new observation that is being classified.

The specification of k is important because if it is too small, it will result in a high error rate, and if it is too large, it will dilute the result by averaging across too many outcomes. Also, if k is even, there may be ties, with no clear winner. KNN is a powerful, nonparametric model, but it requires a specification of what it means to be near.

Analysts need to have a clear understanding of the data and the underlying business to be able to specify the distance metric that needs to be optimized. Another issue with KNN is the specification of feature set; inclusion of irrelevant or correlated features can skew the results. Investment applications of KNN include predicting bankruptcy, assigning a bond to a ratings class, predicting stock prices, and creating customized indices.

1. Classification and regression trees (CART). Classification trees are appropriate when the target variable is categorical, and are typically used when the target is binary (e.g., an IPO will be successful vs. not successful). Logit models, discussed in a previous reading, are also used when the target is binary, but are ill suited when there are significant nonlinear relationships among variables.

In such cases, classification trees may be a viable alternative. Regression trees are appropriate when the target is continuous.

Classification trees assign observations to one of two possible classifications at each node. At the top of the tree, the top feature (i.e., the one most important in explaining the target) is selected, and a cutoff value c is estimated. Observations with feature values greater than c are assigned to one classification, and the remainder are assigned to the other classification.

The resulting classes are then evaluated based on a second feature, and again divided into one of two classes. Every successive classification should result in a lower estimation error than the nodes that preceded it. The tree stops when the error cannot be reduced further, resulting in a terminal node as shown in Figure 3.4.

Figure 3.4: Classification Tree Example

It should be noted that a feature may reappear in lower nodes of a tree with a different cutoff value if it helps in classification. The features and cutoff values are learned by the algorithm based on labeled training data.

To avoid overfitting, regularization criteria such as maximum tree depth, maximum number of decision nodes, and so on are specified by the researcher. Alternatively, sections of tree with minimal explanatory power are pruned.

CART is popular because it provides a visual explanation of the prediction process, compared to other algorithms that are often described as black boxes due to their opacity.

Investment applications of CART include detecting fraudulent financial statements and selecting stocks and bonds.

1. Ensemble and random forest. Ensemble learning is the technique of combining predictions from multiple models rather than a single model. The ensemble method results in a lower average error rate because the different models cancel out noise.

Two kinds of ensemble methods are used: aggregation of heterogeneous learners and aggregation of homogenous learners.

Under aggregation of heterogeneous learners, different algorithms are combined together via a voting classifier. The different algorithms each get a vote, and then we go with whichever answer gets the most votes. Ideally, the models selected will have sufficient diversity in approach, resulting in a greater level of confidence in the predictions.

Under aggregation of homogenous learners, the same algorithm is used, but on different training data. The different training data samples (used by the same model) can be derived by bootstrap aggregating or bagging. The process relies on generating random samples (bags) with replacement from the initial training sample.

Random forest is a variant of classification trees whereby a large number of classification trees are trained using data bagged from the same data set. A randomly selected subset of features is used in creating each tree, and each tree is slightly different from the others. The process of using multiple classification trees to determine the final classification is akin to the practice of crowdsourcing.

Because each tree only uses a subset of features, random forests can mitigate the problem of overfitting. Using random forests can increase the signal to noise ratio because errors across different trees tend to cancel each other out. A drawback of random forests is that the transparency of CART is lost, and we are back to the black box category of algorithms.

Investment applications of random forest include factor based asset allocation, and prediction models for the success of an IPO.