Model Training and Evaluation

Before model training, it is important to define the objective(s) of data analysis, identify useful data points, and conceptualize the model. Model conceptualization is the iterative planning phase that lays out the process to be followed. This process gets tweaked until the desired results are achieved.

It is important that ML engineers work with domain experts so as to identify data characteristics and relationships (e.g., the relation between inflation and exchange rates).

Once the unstructured data has been processed and codified in a structured form such as a data matrix, model training is similar to that of structured data. ML seeks to identify patterns in the data set via a set of rules. Model fitting describes how well the model generalizes to new data (i.e., how the model performs out of sample).

Model fitting errors can be caused by:

• Size of the training sample. Small data sets do not provide adequate training and can lead to an underfit model that does not recognize important patterns. Number of features.
• Fewer features can also lead to an underfitting problem; the small number of features may not carry enough information to identify patterns in the training sample. On the other hand, data sets with a large number of features can lead to overfitting due to fewer degrees of freedom. Overfit models do not generalize well in the validation sample.
• The feature selection step discussed earlier is important in mitigating the overfitting and underfitting problems. FE, when properly done, tends to reduce the underfitting problem.

The three tasks of model training are as follows:

1. Method selection is the art and science of choosing the appropriate ML method (i.e., algorithm) given the objectives and data characteristics. Method selection is based on the following factors: to Supervised or unsupervised learning. Supervised learning is used when the training data contains the ground truth or the known outcome (i.e., the target variable).

In such cases, available methods include regression, ensemble trees, support vector machines (SVMs), and neural networks (NNs). Unsupervised learning occurs when there is no target variable.

Unsupervised learning methods include clustering, dimension reduction, and anomaly detection. to Type of data. For numerical data (e.g., predicting earnings) we may use classification and regression tree (CART) methods.

For text data, we can use generalized linear models (GLMs) and SVMs. For image data, neural networks and deep learning methods can be employed.

to Size of data. Large data sets with many observations and features can be handled with SVMs. Neural networks work better with a large number of observations, but few features.

• Once a method is selected, the researcher has to specify appropriate hyperparameters (e.g., the number of hidden layers in a neural network). For mixed data sets (containing numerical and textual data), multiple methods are often used. Sometimes, the output of one method (e.g., classification of financial news text for a company as positive or negative) may be used as an input to another model.
• Sometimes, multiple models are employed, and a weighted average of the forecasts from those models is used.
• For supervised learning, before model training begins, the data set is divided into three parts. The larger part (≈ 60%) is used for model training. A second part (≈ 20%) is used for validation and model tuning.
• The last part (≈ 20%) is the test set, and is
• used to check the out of sample performance of the model. Due to the absence of labeled training data, no splitting of the data set is needed for unsupervised learning.
• For a model to be able to discriminate well, it should be provided with a wide variety of training data. Class imbalance occurs when one class has a large number of observations relative to other classes. For example, in a model for predicting bond default, if the data set has a large number of high grade bonds (i.e., those that would be less likely to default), then the model would be more likely to predict nondefault for a new observation.
• The training data set should have a variety of high and low grade bonds so as to have enough diversity to make correct predictions. One way to overcome class imbalance is to undersample the overrepresented class and oversample the underrepresented class.

1. Performance evaluation is the process of assessing model efficacy; various tools are used to quantify and critique model performance.
2. Tuning is the process of implementing changes to improve model performance.

These steps are recursively applied until a desired level of model performance is attained. We will next explore the performance evaluation and tuning steps in detail.

Techniques to Measure Model Performance

In order to validate a model, we must measure its training performance or goodness of fit. We will next consider a few methods to measure this performance. (These techniques are particularly suited to evaluating binary classification models.)

1. Error analysis. Errors in classification problems can be false positives (Type I error) or false negatives (Type II error). A confusion matrix shows the results of a classification problem, as in Figure 4.2.

Figure 4.2: Classification of Defaulters

Metrics such as precision (the ratio of true positives to all predicted positives) and recall (the ratio of TPs to all actual positives) can be used. High precision is valued when the cost of a Type I error is large, while high recall is valued when the cost of a Type II error is large.

precision (P) = TP / (TP + FP) recall (R) = TP / (TP + FN) While FP and FN are both errors, they may not be equally important. The tradeoff between precision and recall is a business decision, and depends on the model application. For example, a lender may want to avoid lending to potential defaulters, and so will want to maximize recall.

Together, model precision and recall determine model accuracy, which is the proportion of correct forecasts out of a total number of forecasts. The F1 score is the harmonic mean of precision and recall.

accuracy = (TP + TN) / (TP + TN + FP + FN) F1 score = (2 × P × R) / (P + R)

1. Receiver operating characteristic (ROC). Also used for classification problems, the ROC is a curve that plots the tradeoff between FPs and TPs. The true positive rate (TPR) is the same as recall, and is plotted along the Y axis.

The false positive rate (FPR) is the ratio of FPs to all actual negatives, and is plotted along the X axis.

TPR = TP / (TP + FN)

FPR = FP / (FP + TN) Figure 4.3 shows the performance of the three models used to predict defaults. The area under the curve (AUC) is a value from 0 to

1. The closer the value of AUC is to 1, the higher the predictive accuracy of the model. An AUC value of 0.50 (indicated by a straight line, for model 1) indicates that the model makes a random guess. The higher the convexity of the ROC curve, the higher its AUC.

Figure 4.3: ROC Curves and AUC

1. Root mean squared error (RMSE). This is useful for data predictions that are continuous, such as regression models. The RMSE is a single metric summarizing the prediction error in a sample.

Model Tuning

After model evaluation, the model needs to be revised until it reaches an acceptable performance level. Bias error is the prediction error in the training data resulting from underfit models. Bias errors occur from oversimplified models, which don’t learn adequately from the training sample.

Variance error is the prediction error in the validation sample resulting from overfitting models that do not generalize well. Overfitting is an issue with supervised ML that results when too many features are included in the training sample (i.e., the model is too complicated). It is necessary to find an optimum tradeoff between bias and variance errors, such that the model is neither underfitting nor overfitting.

A fitting curve is a plot of training error and cross validation prediction error with varying model complexity (more complex = more features). An example of a fitting curve is shown in Figure 4.4.

Figure 4.4: Fitting Curve

As a model’s complexity increases, it starts overfitting the training sample, and training error (i.e., bias error) declines. However, this decrease in bias error comes at the cost of increasing variance error. Regularization seeks to reduce model complexity by imposing a penalty on features that don’t meaningfully contribute to the predictive power of the model.

Optimal model complexity balances the tradeoff between bias and variance error.

Parameters are estimated by the model (e.g., slope coefficients in a regression model) using an optimization technique on the training sample. Hyperparameters (e.g., the number of hidden layers in a neural network, or the p threshold in logistic regression) are specified by ML engineers, and are independent of the training sample.

Tuning involves altering the hyperparameters until a desirable level of model performance is achieved. For each specification of hyperparameter(s), a confusion matrix is prepared based on the classification results, and accuracy and F1 scores are compiled. Rather than using a trial and error approach, especially if there are multiple hyperparameters in the model, one can use a grid search.

A grid search is an automated process of selecting the best combination of hyperparameters.

Ceiling analysis is an evaluation and tuning of each of the components in the entire model building pipeline. It identifies the weak link in the process, which can be tuned to improve the performance of the model.

Source Notes for Reading 1

1. Arnott, Robert D., and Clifford S. Asness. “Surprise! Higher Dividends = Higher Earnings Growth.” Financial Analysts Journal 59, no. 1 (January/February 2003): 70 to 87.
2. Wang, Fenghua, and Yexiao Xu. “What Determines Chinese (中国) Stock Returns.” Financial Analysts Journal, no. 6 (November/December 2004): 65 to 77.