Basics of Multiple Regression and Underlying Assumptions

Quantitative Methods 7 min readModule 1 of 4Save

What you'll learn

LOS 1.a
Describe the types of investment problems addressed by multiple linear regression and the regression process.
LOS 1.b
Formulate a multiple linear regression model, describe the relation between the dependent variable and several independent variables, and interpret estimated regression coefficients.
LOS 1.c
Explain the assumptions underlying a multiple linear regression model and interpret residual plots indicating potential violations of these assumptions.

Given the complexities of financial and economic relations, a simple one factor linear regression model is usually inadequate. Multiple regression models allow for consideration of multiple underlying influences (independent variables) on the dependent variable.

We can use multiple regression models to:

Identify relationships between variables: For example, an analyst may perform exploratory analysis of the factors that influence returns on small cap stocks. Or, the analyst may wish to determine if the three factor Fama French model (market, size, and style) actually adequately explains cross sectional returns for a sample time period.
Forecast variables: For example, an analyst may seek to forecast cash flows for a company, or to predict the probability of company default.
Test existing theories: For example, analysts may want to assess if corporate debt issuers with high levels of intangibles on their balance sheet (in addition to other known factors) explain credit risk premiums for those issuers.

Warm Up: Multiple Regression Basics

The general multiple linear regression model is:

Y_{i} = b_{0} + b_{1} X_{1 i} + b_{2} X_{2 i} + \dots + b_{k} X_{k i} + ε_{i}

Symbol	Meaning
Yᵢ	ith observation of the dependent variable Y, i = 1, 2, …, n
Xⱼ	Independent variables, j = 1, 2, …, k
Xⱼᵢ	ith observation of the jth independent variable
b₀	Intercept term
bⱼ	Slope coefficient for each independent variable
εᵢ	Error term for the ith observation
n	Number of observations
k	Number of independent variables

The multiple regression methodology estimates the intercept and slope coefficients such that the sum of the squared error terms, $\sum_{i = 1}^{n} ε_{i}^{2}$ is minimized. The result of this process is the following regression equation:

Y^_{i} = b^_{0} + b^_{1} X_{1 i} + b^_{2} X_{2 i} + \dots + b^_{k} X_{k i}

The residual, $ε^_{i}$ , is the difference between the observed value, $Y_{i}$ , and the predicted value from the regression, $Y^_{i}$ :

ε^_{i} = Y_{i} - Y^_{i} = Y_{i} - (b^_{0} + b^_{1} X_{1 i} + b^_{2} X_{2 i} + \dots + b^_{k} X_{k i})

The Level I curriculum covered simple linear regression and the t test for statistical significance of the slope coefficient. For Level II, in order to interpret regression results, we can alternatively use the p value to evaluate the null hypothesis that a slope coefficient is equal to zero.

The p value is the smallest level of significance for which the null hypothesis can be rejected. We test the significance of coefficients by comparing the p value to the chosen significance level:

If the p value is less than the significance level, the null hypothesis can be rejected.
If the p value is greater than the significance level, the null hypothesis cannot be rejected.

Let’s illustrate multiple regression using research by Arnott and Asness (2003).¹ As part of their research, the authors tested the hypothesis that future 10 year real earnings growth in the S&P 500 (EG10) can be explained by the trailing dividend payout ratio of the stocks in the index (PR) and the yield curve slope (YCS). YCS is calculated as the difference between the 10 year T bond yield and the 3 month T bill yield at the start of the period.

All three variables are measured in percent.

Formulating the Multiple Regression Equation

The authors formulated the following regression equation using annual data (46 observations):

EG10 = b_{0} + b_{1} PR + b_{2} YCS + ε

The results of this regression are shown in Figure 1.1.

Figure 1.1: Coefficient and Standard Error Estimates for Regression of EG10 on PR and YCS

Variable	Coefficient	Standard error
Intercept	−11.6%	1.657%
PR	0.25	0.032
YCS	0.14	0.280

Interpreting the Multiple Regression Results

The interpretation of the estimated regression coefficients from a multiple regression is the same as in simple linear regression for the intercept term, but somewhat different for the slope coefficients:

The intercept term is the value of the dependent variable when the independent variables are all equal to zero. Each slope coefficient is the estimated change in the dependent variable for a 1 unit change in that independent variable, holding the other independent variables constant. For this reason, the slope coefficients in a multiple regression are sometimes called partial slope coefficients.

For example, regarding the real earnings growth model, we can make these interpretations:

Intercept term: If the dividend payout ratio is zero and the slope of the yield curve is zero, we would expect the subsequent 10 year real earnings growth rate to be −11.6%. PR coefficient: If the payout ratio increases by 1%, we would expect the subsequent 10 year earnings growth rate to increase by 0.25%, holding YCS constant. YCS coefficient: If the yield curve slope increases by 1%, we would expect the subsequent 10 year earnings growth rate to increase by 0.14%, holding PR constant.

Let’s discuss the interpretation of the multiple regression slope coefficients in more detail. Suppose we run a regression of the dependent variable Y on a single independent variable $X_{1}$ and get the following result:

Y = 2.0 + 4.5 X_{1}

The appropriate interpretation of the estimated slope coefficient is that if $X_{1}$ increases by 1 unit, we would expect Y to increase by 4.5 units.

Now suppose we add a second independent variable $X_{2}$ to the regression and get the following result:

Y = 1.0 + 2.5 X_{1} + 6.0 X_{2}

Notice that the estimated slope coefficient for $X_{1}$ changed from 4.5 to 2.5 when we added $X_{2}$ to the regression. We expect this to happen when a second variable is included.

Now the interpretation of the estimated slope coefficient for $X_{1}$ is that if $X_{1}$ increases by 1 unit, we would expect Y to increase by 2.5 units, holding $X_{2}$ constant.

Assumptions underlying a multiple regression model include:

A linear relationship exists between the dependent and independent variables.
The residuals are normally distributed. The variance of the error terms is constant for all observations. The residual for one observation is not correlated with that of another observation.
The independent variables are not random, and there is no exact linear relation between any two or more independent variables.

Residual plots allow analysts to get a preliminary indication of violation of regression assumptions. We will discuss formal statistical tests for the detection of violation of (the last three) regression assumptions later.

A normal quantile quantile plot (usually called simply a Q Q plot), is used to compare a variable’s distribution to that of a normal distribution. We can employ a Q Q plot to evaluate the standardized residuals of a regression model: the residuals should lie along a diagonal if they follow a normal distribution. Recall that 5% of normally distributed observations should be below −1.65 standard deviations.

Worked example: Office rent model

An analyst wants to model the determinants of rents for office properties in a large city in the United States. Using a sample of 191 observations, she has estimated the following model:

rent_{i} = b_{0} + b_{1} age_{i} + b_{2} distance_{i} + b_{3} restaurant_{i} + ε_{i}

Variable	Meaning
rent	Monthly rent per square foot ($)
age	Age of the property in years
distance	Distance from the nearest metro station in miles
restaurant	Number of lunch locations within walking distance

Regression Output:

Coefficients	Estimate	Std. error
Intercept	44.67	2.01
Age	−0.31	0.05
Distance	−0.01	0.001
Restaurant	1.29	0.29

Exhibit 1: Plot of Residual vs. Predicted Values

Exhibit 2: Plots of the Residuals vs. the Independent Variables

Exhibit 3: Normal Q Q Plot of Residuals

Interpret the plot of regression residuals versus the predicted value of the dependent variable as shown in Exhibit 1.
Interpret the plot of regression residuals versus the independent variables as shown in Exhibit 2.
Interpret the Normal Q Q plot of regression residuals as shown in Exhibit 3.

Solution

Exhibit 1 does not indicate a systematic pattern or directional relationship between the predicted value of the dependent variable and the model residuals as indicated by the horizontal line centered on 0. This conforms to the requirement that residuals are independent of the predicted value of the dependent variable. Visually, it appears that the residuals have a constant variance and are uncorrelated with each other.
Exhibit 2 does not indicate a directional relationship between the residuals and any of the independent variables. The residuals are scattered around the horizontal line 0 across different values of the independent variables. This is desirable as it conforms to the requirement that the residuals are unrelated to the independent variables.
For a standard normal distribution, only 5% of the observations should be beyond −1.65 standard deviations of 0. We see that a few observations are beyond −2 standard deviations in Exhibit 3, with one outlier beyond −3 standard deviations. Similarly, it appears that there are a higher than expected number of observations beyond +2 standard deviations.

Exhibit 3 also shows skewness in the right tail where the observations are skewed above the line of theoretical distribution. In summary, the distribution of the residuals deviates from a normal distribution as it has fatter tails and right skewness.

Finished this module?

Track your progress across all 171 modules.

Test yourself on Reading 1CFA-style practice questions