Evaluating Regression Model Fit and Interpreting Model Results

Quantitative Methods 8 min readModule 2 of 4Save

What you'll learn

LOS 1.d
Evaluate how well a multiple regression model explains the dependent variable by analyzing ANOVA table results and measures of goodness of fit.
LOS 1.e
Formulate hypotheses on the significance of two or more coefficients in a multiple regression model and interpret the results of the joint hypothesis tests.
LOS 1.f
Calculate and interpret a predicted value for the dependent variable, given the estimated regression model and assumed values for the independent variable.

ANOVA Tables

By decomposing the total variation in the dependent variable into the explained and unexplained components, we can evaluate the quality of model fit. Analysis of variance (ANOVA) is a statistical procedure that provides this information.

The results of the ANOVA procedure are presented in an ANOVA table, which accompanies a multiple regression output. A generic ANOVA table is presented in Figure 1.2.

Figure 1.2: ANOVA Table

Source	Degrees of freedom	Sum of squares	Mean square
Regression	k	RSS	MSR
Error	n − k − 1	SSE	MSE
Total	n − 1	SST

The first column indicates the “source” of the variation: regression is the explained component of the variation, while error is the unexplained component.

Coefficient of Determination, R2

$R^{2}$ evaluates the overall effectiveness of the entire set of independent variables in explaining the dependent variable. It is the percentage of variation in the dependent variable that is collectively explained by all of the independent variables. For example, an $R^{2}$ of 0.63 indicates that the model, as a whole, explains 63% of the variation in the dependent variable.

R^{2} = \frac{total variation - unexplained variation}{total variation} = \frac{S S T - S S E}{S S T} = \frac{R S S}{S S T}

Adjusted R2

Unfortunately, $R^{2}$ almost always increases as more independent variables are added to the model even if the marginal contribution of the new variables is not statistically significant. Consequently, a relatively high $R^{2}$ may reflect the impact of a large set of independent variables, rather than how efficiently the set explains the dependent variable. This problem is often referred to as overestimating or overfitting the regression model.

To overcome the problem of overfitting (the impact of additional variables on the explanatory power of a regression model), many researchers recommend adjusting $R^{2}$ for the number of independent variables. The adjusted $R^{2}$ value is expressed as:

R_{a}^{2} = 1 - [\frac{n - 1}{n - k - 1} (1 - R^{2})]

$R_{a}^{2}$ will always be less than or equal to $R^{2}$ . So while adding a new independent variable to the model will increase $R^{2}$ , it may either increase or decrease the $R_{a}^{2}$ . If the new variable has only a small effect on $R^{2}$ (i.e., the absolute value of the coefficient’s t statistic is less than 1), the value of $R_{a}^{2}$ will decrease.

Worked example: Calculating

R^{2}

and adjusted

R^{2}

An analyst runs a regression of monthly stock returns on five independent variables over 60 months. The total sum of squares for the regression is 460, and the sum of squared errors is 170. Calculate the $R^{2}$ and adjusted $R^{2}$ .

Solution

R^{2} = \frac{460 - 170}{460} = 0.630 = 63.0%

R_{a}^{2} = 1 - [\frac{60 - 1}{60 - 5 - 1} (1 - 0.63)] = 0.596 = 59.6%

The $R^{2}$ of 63% suggests that the five independent variables together explain 63% of the variation in monthly stock returns. The $R_{a}^{2}$ is, as expected, a somewhat lower value.

Worked example: Interpreting adjusted

R^{2}

Suppose the analyst now adds four more independent variables to the regression, and the $R^{2}$ increases to 65.0%. Identify which model the analyst is most likely to prefer.

Solution

With nine independent variables, even though the $R^{2}$ has increased from 63% to 65%, the adjusted $R^{2}$ has decreased from 59.6% to 58.7%:

R_{a}^{2} = 1 - [\frac{60 - 1}{60 - 9 - 1} (1 - 0.65)] = 0.587 = 58.7%

The analyst will prefer the first model because the adjusted $R^{2}$ is higher, and the model has five independent variables as opposed to nine.

While the adjusted $R^{2}$ penalizes overfitting, it does not indicate the quality of model fit, nor does it indicate statistical significance of the slope coefficients. We can formally evaluate the overall model fit using an F test (discussed later).

For evaluating a regression model, regression output may include the Akaike’s information criterion (AIC) and the Schwarz’s Bayesian information criteria (BIC). Both AIC and BIC evaluate the quality of model fit among competing models for the same dependent variable. Lower values indicate a better model under either criteria.

AIC is used if the goal is to have a better forecast, while BIC is used if the goal is a better goodness of fit.

These metrics can be calculated as follows:

AIC = n ln (\frac{S S E}{n}) + 2 (k + 1)

BIC = n ln (\frac{S S E}{n}) + ln (n) (k + 1)

The variable k is a penalty parameter in both criteria: higher values of k result in higher values of the criteria. Because ln(n) is greater than 2 for even small sample sizes, the BIC metric imposes a higher penalty for overfitting.

Worked example: Goodness of fit for the rent model

Continuing our example on rental price per square foot, the following shows the results when a single factor, two factors, and all three factors are used in the model.

Independent variables	k	SSR	SSE	R²	Adjusted R²	AIC	BIC
Age	1	3,318.9	32,627.3	9.23%	8.75%	985.9	992.4
Age + Distance	2	20,946.1	15,000.2	58.27%	57.8%	839.4	849.2
Age + Distance + Restaurants	3	22,395.6	13,550.5	62.30%	61.7%	822.0	835.0

A. Which model is the most appropriate for use in generating forecasts?

B. Which model has a better goodness of fit?

Solution

A. The model with all three independent variables has the lowest AIC, and hence is the most appropriate model for generating forecasts.

B. The model with all three independent variables has the lowest BIC and hence the best goodness of fit. Also, the adjusted $R^{2}$ for the three factor model is the highest of the three models.

In addition to AIC and BIC, we can use a formal F test to evaluate nested models. Nested models are models such that one model, called the full model or unrestricted model, has a higher number of independent variables while another model, called the restricted model, has only a subset of the independent variables. Consider a full model with three independent variables that is evaluated relative to a more parsimonious restricted model, which includes only the first variable as the independent variable.

Unrestricted model: $Y_{i}$ = $b_{0}$ + $b_{1}$ $X_{1}$ + $b_{2}$ $X_{2}$ + $b_{3}$ $X_{3}$ + $ε_{i}$

Restricted model: $Y_{i}$ = $b_{0}$ + $b_{1}$ $X_{1}$ + $ε_{i}$

We want to test the following hypothesis:

H_{0} : b_{2} = b_{3} = 0 vs. H_{a} : b_{2} \neq = 0 or b_{3} \neq = 0

We calculate the F statistic to test this hypothesis as:

F = \frac{( S S E _{R} - S S E _{U} ) / q}{S S E _{U} / ( n - k - 1 )} with q and (n - k - 1) degrees of freedom

Decision rule: reject $H_{0}$ if F (test statistic) > $F_{c}$ (critical value)

The F test evaluates whether the relative decrease in SSE due to the inclusion of q additional variables is statistically justified.

We could evaluate the individual slope coefficients using t tests. However, because there is likely to be some amount of correlation among the independent variables, the joint significance of excluded variables F test provides a more meaningful evaluation of the explanatory power of alternative models.

Worked example: Joint hypothesis test

Continuing our previous rental model data, and using the following information, conduct a joint test of hypotheses for the slope coefficients 2 and 3 at a 5% level of significance.

Model	k	SSE	n
Unrestricted	3	13,550.64	191
Restricted	1	32,627.29	191

Partial F table (5% level of significance)

Denominator degrees of freedom	Numerator df: 1	Numerator df: 2	Numerator df: 3
1	161	200	216
30	4.17	3.32	2.92
60	4.00	3.15	2.76
120	3.92	3.07	2.68
190	3.89	3.04	2.65
∞	3.84	3.00	2.60

Solution

F = \frac{( 32627.29 - 13550.64 ) /2}{13550.64/ ( 191 - 3 - 1 )} = 131.63

The critical F value with 2 and 187 degrees of freedom is 3.04, so we reject the null hypothesis and conclude that at least one of the two slope coefficients of the excluded variables is statistically different from 0.

As a general case, we can also use the F test to evaluate the overall model fit. That is, the F statistic can be used to test whether at least one of the independent variables explains a significant portion of the variation of the dependent variable.

For example, if there are three independent variables in the model, the hypothesis is structured as:

H_{0} : b_{1} = b_{2} = b_{3} = 0 vs. H_{a} : at least one b_{j} \neq = 0

In such a situation, all the explanatory variables are excluded in the restricted model (i.e., q = k), and all of the variation in the restricted model is unexplained (i.e., $S S E_{R}$ = $S S T_{U}$ ). The F statistic then becomes:

F = \frac{( S S E _{R} - S S E _{U} ) / q}{S S E _{U} / ( n - k - 1 )} = \frac{( S S T _{U} - S S E _{U} ) / k}{S S E _{U} / ( n - k - 1 )} = \frac{R S S _{U} / k}{S S E _{U} / ( n - k - 1 )}

Predicting the Dependent Variable

We can use the regression equation to make predictions about the dependent variable based on forecasted values of the independent variables. The process is similar to forecasting with simple linear regression, except now we need predicted values for more than one independent variable. The predicted value of dependent variable Y is:

Y^_{i} = b^_{0} + b^_{1} X^_{1 i} + b^_{2} X^_{2 i} + \dots + b^_{k} X^_{k i}

The prediction of the dependent variable uses the estimated intercept and all of the estimated slope coefficients, regardless of whether the estimated coefficients are statistically significantly different from zero. For example, suppose you estimate the following regression equation:, $Y^= 6 + 2 X_{1} + 4 X_{2}$ and you determine that only the first independent variable ( $X_{1}$ ) is statistically significant (i.e., you fail to reject the null that $b_{2}$ = 0). To predict Y given forecasts of $X_{1}$ = 0.6 and $X_{2}$ = 0.8, you would use the complete model: $Y^= 6 + (2 \times 0.6) + (4 \times 0.8) = 10.4$ Alternatively, you could drop $X_{2}$ and re estimate the model using just $X_{1}$ ; but remember that the coefficient of $X_{1}$ will most likely change.

Worked example: Calculating a predicted value for the dependent variable

An analyst would like to use the estimated regression equation from the previous example to calculate the predicted 10 year real earnings growth rate for the S&P 500, assuming the payout ratio of the index is 50%. He observes that the slope of the yield curve is currently 4%. What is the model’s prediction for the index’s real earnings growth rate?

Solution

EG10 = - 11.6% + 0.25 (50%) + 0.14 (4%) = 1.46%

The model predicts a 1.46% real earnings growth rate for the S&P 500, assuming a 50% payout ratio, when the slope of the yield curve is 4%.

Finished this module?

Track your progress across all 171 modules.

Test yourself on Reading 1CFA-style practice questions