Extensions of Multiple Regression

Outliers are extreme observations of the dependent (or ‘Y’) variable, while high leverage points are the extreme observations of the independent (or ‘X’) variables.

A high leverage point would be identified using a measure called leverage ($L_{ij}$) which can be provided by statistical packages. Leverage is a measure of the distance between the jth observation of independent variable i relative to its sample mean. Leverage takes a value between 0 and

1. The higher the value of leverage, the greater the distance and hence the higher the potential influence of the observation on the estimated regression parameters.

The sum of the individual leverages for all observations is k +

1. If a variable’s leverage is higher than three times the average, [3(k + 1) / n], it is considered potentially influential.

We can identify outliers using the studentized residuals. The following steps outline the procedure:

1. Estimate the regression model using the original sample of size n. Delete one observation and re estimate the regression using (n − 1) observations. Perform this sequentially, for all observations, deleting one observation at a time.
2. Compare the actual Y value of the deleted observation i to the predicted Y values using the model parameters estimated with that observation deleted.

1. The studentized residual is the residual in Step 2 divided by its standard deviation.

1. We can then compare this studentized residual to critical values from a t distribution with n − k − 2 degrees of freedom (because we now only have n − 1 observations), to determine if the observation is influential.

Not all outliers or high leverage points are influential. Influential data points are extreme observations that, when excluded, cause a significant change to model coefficients.

Once influential observations are identified, we need to determine whether this was the result of an input error (in which case, the error should be rectified or the observation deleted), or if the observation is valid but the model is incomplete (i.e., important independent variables are omitted). Analysts can also winsorize the dataset to minimize the impact of outliers.

Observations for most independent variables (e.g., firm size, level of GDP, interest rates, etc.) can take on a wide range of values. However, there are occasions when the independent variable is binary in nature it is either “on” or “off” (or true/false). Independent variables that fall into this category are called dummy variables and are often used to quantify the impact of qualitative events.

Dummy variables are assigned a value of “0” or “1.” For example, in a time series regression of monthly stock returns, we could employ a “January” dummy variable that would take on the value of “1” if a stock return occurred in January, and “0” if it occurred in any other month. The purpose of including the January dummy variable would be to see if stock returns in January were significantly different than stock returns in all other months of the year.

Many “January effect” anomaly studies employ this type of regression methodology.

The estimated regression coefficient for dummy variables indicates the difference in the dependent variable for the category represented by the dummy variable relative to the average value of the dependent variable for all classes except the dummy variable class. For example, testing the slope coefficient for the January dummy variable would indicate whether, and by how much, security returns are different in January versus the other months.

An important consideration when performing multiple regression with dummy variables is the choice of the number of dummy variables to include in the model. In the prior example, we had two classes: one set representing the data points that occurred during the month of January, and the other representing the rest of the year. When we want to distinguish between n classes, we must use (n − 1) dummy variables.

Otherwise, we would violate the regression assumption of no exact linear relationship between independent variables.

A dummy variable can be an intercept dummy, a slope dummy, or a combination of the two.

Consider a regression with one continuous X variable and one dummy variable (an intercept dummy):

This regression becomes:

In other words, the intercept of the line shifts from $b_0$ to ($b_0$ + $d_0$) if D =

1. The value of this shift ($d_0$) can be positive or negative.

A slope dummy, on the other hand, changes the slope of the regression equation:

And the regression then becomes:

A slope dummy is the interaction term; it captures the interaction between the dummy variable and the continuous variable.

Finally, we could have dummies in both intercept and slope:

which becomes:

Interpreting the Coefficients in a Dummy Variable Regression

Consider the following regression equation for explaining quarterly EPS in terms of the calendar quarter of their occurrence:

In this equation, the intercept term, $b_0$, represents the average value of EPS for the fourth quarter. The slope coefficient on each dummy variable estimates the difference in earnings per share (on average) between the respective quarter (i.e., quarter 1, 2, or 3) and the omitted quarter (the fourth quarter in this case). Think of the omitted class as the reference point.

For example, suppose we estimate the quarterly EPS regression model with 10 years of data (i.e., 40 quarterly observations) and find that $b_0$ = 1.25, $b_1$ = 0.75, $b_2$ = −0.20, and $b_3$ = 0.10:

We can use the equation to determine the average EPS in each quarter over the past 10 years:

These are also the model’s predictions of future EPS in each quarter of the following year. For example, to use the model to predict EPS in the first quarter of the next year, set $\text{set }{\hat{Q}}_1=1,{\hat{Q}}_2=0,\text{ and }{\hat{Q}}_3=0$. Then:

${\widehat{\mathrm{EPS}}}_{Q1}=1.25+0.75(1)-0.20(0)+0.10(0)=2.00$. This simple model uses average EPS for any specific quarter over the past 10 years as the forecast of EPS in its respective quarter of the following year.

Financial analysis often calls for the use of a model that has a qualitative dependent variable, a categorical variable, usually a binary variable, which takes on a value of either zero or one. An example of an application requiring the use of a qualitative dependent variable is a model that attempts to estimate the probability of default for a bond issuer. In this case, the dependent variable may take on a value of one in the event of default and zero in the event of no default.

An ordinary regression model is not appropriate for situations that require a qualitative dependent variable, because the forecasted values of y using the model can be less than 0 or greater than 1, which are illogical values for probability. Instead, we transform the probability values of the dependent variable into odds: p / (1 − p). For example, if probability = 0.80, then odds = 0.80 / 0.20 or 4 to

1. A logistic transformation involves taking the natural log of the odds: ln[p / (1 − p)].

Logistic regression (logit) models use log odds as the dependent variable.

A logit model assumes that residuals have a logistic distribution, which is similar to a normal distribution but with fatter tails. We use logit models to estimate the probability of a discrete outcome, such as the probability of default of a company, or the probability that a merger will occur.

The coefficients of the logit model are estimated using the maximum likelihood estimation methodology. The interpretation of the intercept term in a logit model is similar to its interpretation in multiple regression: it provides an estimate of log odds when the values of all independent variables is zero. The interpretation of the slope coefficient is, however, not straightforward.

Because the logistic regression model is nonlinear, the change in log odds when one of the independent variables changes is dependent on the curvature of the function at that point. To interpret the slope coefficient, we would typically start with the average values of the independent variables, estimate log odds, then change the value of one of the independent variables by 1 unit, and calculate the change in log odds.

Once the coefficients are estimated, using the regression equation, the predicted value of y (i.e., $\hat{y}$) can be calculated from the values of the X variables. The odds are then calculated as:

And the probability of the event is calculated as: