Metrics, Visuals, and Problems in Backtesting

Metrics

In addition to the familiar Sharpe ratio and the Sortino ratio (which uses target semideviation rather than standard deviation), a number of other measures are commonly used to quantify investment performance.

Value at Risk A portfolio’s value at risk (VaR) is one way to quantify the amount of risk present in the left tail of a portfolio’s return distribution. VaR is a measure of the minimum amount of value losses expected for a given period of time at a particular level of probability. Unfortunately, VaR is sensitive to our assumptions about the shape of the distribution (for example, fat tails versus normal tails).

Conditional VaR

Conditional VaR (CVaR) is an extension of VaR that provides a more comprehensive measure of tail loss than VaR does. CVaR quantifies the weighted average of all returns that exceed the specified value of loss. As an example, if we select a significance level of α = 5%, the CVaR is the expected value of the return within that lowest 5% of outcomes.

If we have a series of historical returns, CVaR is calculated as the average loss once the VaR cutoff is exceeded.

Maximum Drawdown Maximum drawdown is a commonly used measure of downside risk; it represents the greatest amount of loss that an asset has experienced from peak to trough in the past. Maximum drawdown as a risk measure is based on the idea that the best indication of downside risk is actual past loss events. To calculate maximum drawdown, we begin with the asset’s maximum cumulative return, and then subtract from the lowest cumulative return that occurs after that point.

Commodity trading advisors (CTAs) and hedge funds often use maximum drawdown.

Visuals

When evaluating an investment strategy, it can be useful to examine the long term cumulative return performance of an investment strategy by plotting it using a logarithmic scale. This makes equivalent percentage changes span the same vertical distance on the y axis. By examining such a graph, we can often immediately recognize structural breaks and performance decay, and also evaluate downside risk.

Distribution plots of factor returns are an example of a visual often used in assessing the backtesting of a factor based investment strategy.

Problems in Strategy Backtesting

There are several errors that quantitative investors commonly make when backtesting that are likely to introduce behavioral biases or other issues.

Survivorship Bias

Survivorship bias occurs when, at a particular point in time, we consider only those firms or stocks that have remained in existence (i.e., survived) to that time. Survivorship bias is arguably the most well known and yet also the most common error that investors make in backtesting. Despite this issue being familiar to academics and practitioners alike, investors across many kinds of assets will fail to account for the impact of survivorship bias.

Survivorship bias comes about because identifying all securities that exist at the moment is straightforward, while identifying all securities that have existed since a particular point in time (including those that are now defunct) can be much more difficult. The desired solution to this issue is to use point in time data, which represents the specific data that was available at a particular point in history. Point in- time data takes into account what is not known at a given point in time, and thus allows investors to avoid look ahead bias and survivorship bias, in order to implement the most realistic backtesting of an investment strategy.

Stocks and firms emerge and vanish continuously. A company can disappear (and be removed from an index) for a number of reasons: perhaps it was taken private, went bankrupt, was acquired, or simply delisted from an exchange for poor performance. New firms also appear and are added to the major indexes through various mechanisms including successful startups, as well as corporate actions like carve outs or spin offs.

As an example, the UK’s FTSE 100 Index contained 100 stocks when it was initiated on January 3rd, 1984, and it still contains 100 securities more than 40 years later. However, of the original 100 stocks, only 26 of its original members were still in the index as of January 3, 2024.

Backtesting using only stocks that have endured through time can introduce significant bias and can even produce results that are the opposite to the correct conclusion.

Unfortunately, keeping track of hundreds of securities over long periods is not simple, so investors may resort to backtesting using only the securities that exist today. The flaw in this methodology is that we cannot be certain which securities will survive in the future.

One example that we might consider when analyzing survivorship bias would be the low instability anomaly, which suggests that over the long term, low volatility stocks will generally outperform high volatility stocks. If we correctly use point in time data, we would conclude that low volatility stocks do indeed outperform high volatility stocks from 1989 through 2019. However, if we base our analysis only on those stocks that exist today, without considering the stocks that have disappeared over the decades, we will come to the opposite conclusion.

This result is perhaps not surprising: a historically high volatility stock that has survived to the present day likely outperformed other stocks during that time.

This case demonstrates the importance of using point in time data in backtesting, rather than only using existing/surviving stocks.

Look Ahead Bias

Look ahead bias is another common error that investors make when backtesting an investment strategy. This problem arises when an investor makes use of data that would not have been available at the time an investment decision is made. (Survivorship bias is arguably a special case of look ahead bias, because survivorship bias stems from only including securities that will still exist in the future.)

Look ahead can come about due to reporting lag, which refers to a situation where data describing a period is often available only after the period ends (and is often subject to revision).

In backtesting we should ideally use only point in time information. However, not all data vendors provide point in time data, so we may need to make some adjustments to account for look ahead bias. We might adjust the data by adding a lag assumption of one, two, or three months.

For instance, if a firm’s financial quarter ends on December 31, 2023, we should not expect to have that company’s results on that date. Many large- capitalization firms report earnings within 30 to 50 days of quarter end, while a small or midsize company might take even more time.

Our first approach would be to make some assumptions about reporting lag. If we make the assumption that earnings data for all companies is available a month later on January 31, 2024, we would encounter look ahead bias with firms that took longer than a month to report earnings. Conversely, if we used a three month lag and assumed that earnings were available on March 31, 2024, we would be introducing the problem of stale data into our analysis, and our backtest would be overly conservative.

Again, using point in time data is preferable to having to make these kinds of assumptions and adjustments.

A related issue occurs when accounting errors, or changes in accounting standards, causes firms to re state their past financial statements. Similarly, government agencies frequently revise macroeconomic data. Vendors’ data may reflect the current or corrected data or figures, which can be different from the information originally available to the market, meaning that a different decision may have been made based on the data available at that time.

Data Snooping

Suppose that an investment analyst tries many different models, backtests each of them, and then selects the one with the highest numerical results. The practice of backtesting numerous strategies and then selecting the best performing one, rather than constructing a portfolio based on sound theory, is problematic because it is likely to result in false positive results. This process of performing analyses until we “find” a statistically significant strategy is called data snooping (or, sometimes, “p hacking”).

Data snooping can come in various forms. A portfolio manager should be wary of strategies that may have been developed using data snooping to select the model with the highest t statistic and lowest p value.

One solution to data snooping is to use a higher than normal critical t statistic (e.g., 3.0) as a benchmark for declaring a variable to be significant (i.e., for concluding that it adds value to the model).

Another method to deal with data snooping is to use cross validation, a technique that involves testing a hypothesis on a different set of data than the one that was initially used to form the inference or test the hypothesis. To perform cross validation, we divide our data into “training data” versus “validation data” (i.e., testing data). Rolling- window backtesting is actually a type of cross validation, without the random component: in sample periods (i.e., past periods) are used to tune a model, which is then applied to data from out of sample (i.e., later) periods.