Backtesting An Investment Strategy

The fundamental steps in backtesting an investment strategy are generally as follows:

Step 1: Strategy design Step 2: Historical investment simulation Step 3: Analysis of backtesting output

Step 1: Strategy Design

The first step in backtesting is to determine our assumptions and investment objectives. For an active investor, our goal is generally to achieve high risk adjusted returns, while managing downside risk. Other factors may also need to be considered, such as turnover, concentration, and our time horizon.

Investment Universe The term investment universe is a reference to all the assets in which we might conceivably take a position while executing our strategy. For exposition, we will use the Russell 3000 Index, a broad U.S. equity index, as our investment universe.

Definition of Return If our investment universe extends beyond our own country’s borders, a number of complications emerge, for example: whether to denominate return in the local currency of the investment, or to translate returns into a single currency (often this decision will depend on whether or not our investments are currency hedged).

Returns also should be reported relative to a particular benchmark. Typically, the benchmark will be an index that reflects the investment universe of our strategy, such as the S&P/TSX Composite Index if our investment universe is the Canadian equity market.

Rebalancing Frequency and Transaction Cost Monthly rebalancing is typical for investment portfolios, though shorter or longer frequencies are not uncommon. When higher frequency rebalancing is used, transaction costs will increase and can become significant. It is important to consider these transaction costs because many apparent profit opportunities vanish once trading costs are considered.

For this reason, performance presentation of a backtest should specify whether or not transaction costs are included.

Start Date and End Date

Portfolio managers generally prefer to use a long data history when backtesting strategies, in order to maximize confidence in the results. However, financial market data tends to be nonstationary, in that it is likely to contain various distinct regimes (a regime is a period of similar macroeconomic conditions, for example, recessions, expansions, or periods of high or low inflation). For this reason, the discrete time intervals within the history should additionally be individually analyzed.

Constructing Portfolios From Factor Portfolios

Equity investment strategies will typically employ factor based models. Factors can be thought of as any variable that can be used to predict future returns or risks and thereby categorize stocks in terms of desirability. Factors are intended to represent unique sources of risk based on some economic fundamental.

The goal of factor based strategies is to identify signals that will allow us to assemble portfolios that will outperform the overall market. Active investment strategies can often be replicated by factors.

Aside from statistical measures, we also want to consider whether a particular factor makes instinctive sense: Is there theoretical support for the factor? Does it make sense from a financial perspective? We use these criteria to try to avoid the data mining trap, which occurs when many different factors are considered, and those that perform well in a backtest are incorporated into the strategy, even if they don’t have any logical or economic reason to be included.

If a factor can successfully pass backtesting, but does not make logical sense, we may want to reject the factor: positive backtesting results alone don’t ensure that future returns will be similar.

Normally, an investor will first develop a theory, for example, that stocks with a high income yield should produce higher returns going forward, and then based on this theory, formulate appropriate investing rules and procedures. After gathering the historical data for each stock (in this case, the earnings yield and return for each security), the investor will partition the data into two subsamples: training (in sample) data, and testing (out of sample) data.

It would be unusual for an investor to rely on a single variable to guide their investment models. Rather, it is more common to construct a stock selection model that linearly combines several factors. Stock screening models will similarly take multiple factors into account.

Two clear and straightforward ways to combine factor portfolios are: first, a benchmark portfolio (BM), which weights factors equally, and second, a risk parity portfolio (RP), which combines factors so that each factor contributes equally to risk.

Investment Styles and Common Factors

For the purpose of exposition, we select several basic variables from each of the following common investment styles:

1. Defensive value - Based on trailing earnings yield.

• High E/P stocks are preferred. 2. Cyclical value - Based on book to market ratio. - High ratio (low market value relative to book value) is preferred.

1. Growth - Based on consensus earnings growth for the coming year. - Stocks with high earnings growth are preferred.
2. Price momentum - Based on 12 month total return, excluding the previous month. - Stocks with momentum are preferred. 5. Analyst sentiment - Based on EPS changes over the past 3 months.

• Stock with positive earnings revisions is preferred. 6. Profitability - Based on return on equity. - High ROE companies are preferred.

1. Leverage - Based on the debt/equity ratio. - Stocks with low leverage are preferred.
2. Earnings quality - Based on level of accruals. - Non cash earnings are seen as low quality; stocks with low levels of accruals are

preferred.

Suppose that we form a portfolio by shorting the “worst” 20% of stocks in the universe according to a particular factor, and buying the “best” 20% of stocks as measured by that factor. Rebalancing for each factor portfolio is performed monthly. We will not explicitly consider trading costs.

The most obvious method of combining these factors is with an equally weighted portfolio, which we will call the benchmark portfolio (BM). This kind of equally- weighted portfolio has been found to perform similarly to portfolios that use more- complex weighting schemes.

An alternative scheme for constructing portfolios is the risk parity (RP) portfolio construction technique, which accounts for the volatility of each factor and the correlations of returns among all the factors to be combined in the portfolio. Under the risk parity technique, each factor will make an equal contribution to overall portfolio risk.

Step 2: Historical Investment Simulation

In this step, we create the portfolio to be evaluated and then rebalance according to our predetermined frequency.

The actual construction of the portfolio will be guided by the investment strategy that we are pursuing, as well as any kind of constraints (such as geography, size, liquidity, limits on shorting, etc.).

Rolling Window Backtesting

Portfolio managers use backtesting to evaluate (for example) the viability of a factor.

In the rolling window backtesting approach, the investor will use a walk forward (rolling window) system rather than dividing the data into just two samples. In this methodology, the investor will calibrate the trade signals or factors based on the moving window, adjust the model and rebalance the portfolio after each period as new information arrives, and then track performance over the long term. In this way, backtesting simulates real investing.

Although the model parameters are continually tuned over time, the primary model methodology is fixed in advance (to avoid overfitting).

To illustrate this process, we will provide an example of backtesting the trailing 12- month earnings yield (i.e., value) factor. Suppose that we begin our backtesting on 30 November 2023. First, we calculate each stock’s trailing earning yield, by taking the earnings per share over the past 12 months (from December 2022 to November 2023), divided by the 30 November 2023 stock price.

Based on this calculation, we will implement our methodology, which in this instance means shorting the lowest quintile of stocks (the 20% of firms with the lowest income yield), while purchasing the 20% of stocks with the highest earnings yield. We then measure the performance of our model versus the first out of sample (OOS) period, which is the following month, December 2023.

We then advance one month in time and repeat the process as of 31 December 2023, and one piece of out of sample data (December 2023) becomes in sample data. We continue until we have covered the entire timespan from November 2023 to April 2024. Using all of this data, we can then calculate performance of our strategy over the period, including monthly average return, maximum drawdown, volatility, and Sharpe ratio.

Our strategy will be accepted if it performs well in the out of test periods (and makes logical sense).

One caveat in using rolling window backtesting to assess investment strategies is that this methodology implicitly assumes that the same pattern of past performance is expected to repeat over time. Another caveat is that rolling window backtesting results may not reflect the dynamic nature of financial markets and possible extreme downside risk.

Backtesting the Performance of Factor Allocation Strategies

Backtesting a multifactor strategy is somewhat complex because we need to execute the rolling window process twice. First, we construct the eight factor portfolios using the walk forward approach. (Each of these factor portfolios is long short.)

Then, we use either the benchmark (BM) or risk parity (RP) weighting schemes to form a multifactor portfolio. Then, a second rolling window process is used to determine the covariances and create the RP portfolio. Both the BM and RP portfolios (which are long only in terms of the factor portfolios) are rebalanced monthly to maintain risk parity or equal weighting, as appropriate.

These portfolios are then analyzed using out of sample data to determine the returns of each portfolio.

Step 3: Analysis of Backtesting Output

Long/Short Hedged Portfolio Approach The conventional approach to employing factor based portfolios is the Fama and French (1993) hedged portfolio methodology. In this approach, we first select a factor and then rank our stock universe by that factor. Then, we rank our universe of stocks based on factor scores, divide this universe into quantiles (and in this case, quintiles, meaning five equal layers based on our factor) and invest in each quintile based on the particular weighting scheme.

(Inside each quintile, we may weigh the stock either equally or by market capitalization.) We then go long the top quintile (i.e., the stocks with the highest factor scores) and short the bottom quintile (i.e., the stocks with the lowest factor scores) in order to form a long/short hedged portfolio. (Note that one drawback of the long/short hedged portfolio approach is that some managers cannot short stocks.)

Rolling window backtesting can then be applied, with the portfolio being rebalanced monthly. This generates a series of out of sample performance data that we can then analyze using metrics such as the Sharpe ratio, maximum drawdown, Sortino ratio, et cetera.

Comparing the Results of Backtesting Methodologies Testing can show that different backtesting methods will produce results that are somewhat different, or sometimes even completely different.

For example, if there is a non linear relationship between a factor and future stock returns, one metric may suggest significant results, while another may not.

No perfect formula exists for determining which methodology to employ. Ideally, multiple methods will suggest similar results.