"Good" versus "bad" backtests

Suppose you're backtesting two different system you designed. The first system's results show a slightly positive expectacy, 0.05R (5% return per trade). The second system yields a surprisingly negative expectancy of -0.25R (-25% return per trade).

Which backtest yielded better results?

If you're entering the market randomly (e.g. by flipping a coin), your base expectancy is 0.00R. On a long enough time scale, you expect to win as much as you lose. Thus, we should approach all potential systems with an expectancy of 0.00R.

If a system you backtest returns an expectancy close to 0.00R, you can say that the system is "bad", in that you're not much better off than flipping a coin or throwing a dart. If your system returns a significant positive expectancy (I would rate 0.2R or above to be acceptable, from my experience so far), then it is "very good".

But what of a significantly NEGATIVE expectancy? Remember that we approach all systems with a base expectancy of 0.00R. A negative expectancy is telling us what NOT to do. And just as importantly, it's pointing us in the general direction of WHAT to do, i.e. do the "opposite" of the system.

So, we can say that the backtest result of 0.05R to be bad, even though it is slightly positive. We can also say that the backtest result of -0.25R is good, not because that particular system is tradable, but because there's alot more we can learn from. We can use this knowledge to avoid the mistakes highlighted in the system, and go on to create better systems in the future.

Currency personalities

I've been spending the last few weeks backtesting. On a typical day, I'll sink at least three hours into manually backtesting various concepts.

Concepts I've backtested in the last fortnight:

1) Volatility breakout (enter when daily price movement > ATR(14) * 2)

This initially showed promise on volatile pairs like the GBPUSD, but it broke down on quiet pairs like the USDCAD. If I focused on the volatile pairs on the daily TF, my sample size would've been too low for my liking, so decided to abort. Off a sample size of ~265 from five currency pairs, I believe my expectancy was around 0.025R after spread. This was using a 0.75 reward-to-risk ratio, which seemed the most optimal from the various R:R ratios I tested. Very marginal. 

2. Retracement entry system

I used a 21 EMA to measure the trend. Once the MA showed a clear trend and any old counter-trend candle presented itself, I'd enter on the break of the counter-trend candle. The SL was between the break of the counter-trend candle and the previous swing point. TP was set at 1.68 * SL (a Fibonnaci extension level). After 104 samples, expectancy was 0.04R before spread. Not remarkable so gave this a miss.

3. Price Action on Pullback + Elliot Wave system

This is a similar system to #2. I used a 21 MA to measure the trend. If a clear trend was present, I'd wait to see if an Elliot Wave formation presented itself (higher high, higher lows if bullish, lower highs and lower lows if bearish). In the case of a bullish trend, if a price action signal appeared at a higher low (pullback), you'd enter on the break of this signal. I used pinbars, engulfing bars and two-bar reversals as signals.

So far I gathered 125 samples from the AUDUSD and USDJPY on the daily TF from 1996 to 2011. Using a 2:1 reward-to-risk ratio, expectancy was 0.28R per trade, which is promising (28% return per trade). However, this is a discretionary trading system, which is difficult to backtest accurately. Discretionary systems introduce all sorts of biases.

What I learned so far

One lesson I learned over the last fortnight is how every currency pair has their own personality. For example, I noticed that the USDCAD tends to be more docile, making a volatility-based system problematic. I believe the "holy grail" should be able to trade on most currencies and turn a profit, or at least breakeven during adverse conditions. If a "trading system" only works on a small handful of currency pairs, I think what you have is a meaningless curve-fitting expression. A good system should be profitable globally.

Entry on trend retracements

I'm currently backtesting entry on trend retracements.

Whenever a currency pair trends, there will be moments when price will temporarily pullback before resuming the trend. These pullbacks, or retracements, are ideal points to enter a trade and exploit the trend. 

Below is an example of retracements on the AUDUSD during 2003. Here we see a few examples of pinbars, engulfing bars and two bar reversals forming during a retracement of the bullish trend off resistance-turned-support levels. Entering a trade at these points would reward a trader handsomely.

If using line graphs, the basic setup would look like this:

Unfortunately, backtesting a discretionary trading system like this is hard and tedious. The rules aren't completely mechanical, so things like bias and emotion will influence backtesting results. The placement of support and resistance levels are also subjective.

I'm halfway finished with my backtest on the AUDUSD. I hope to move on to the EURUSD, GBPUSD and USDJPY in the next week, then the cross-currency pairs.

12th May 2012 - loss on GBPUSD

GBPUSD

This trade didn't quite work out. I saw a pinbar on what looked like a retracement of the overall bullish trend. I checked with more-experienced traders and all agreed that this was the perfect setup for a sucker rally. Being me, I had to see for myself if this trade would work, and entered long on the break of the pinbar. Price rallied for a short while before reversing and hitting my SL late yesterday. Thankfully I was in demo mode so didn't lose any real money. I think I will stay in demo mode until I am consistently profitable for six months. I need to be serious and treat this as a business, not a hobby.

From what other traders told me, this pinbar:

- closed too weak (bearish body was too big)

- nose of pinbar wasn't long enough

- overall, the pinbar wasn't convincing enough to suggest a continuation of the uptrend.

I'm not so sure about the reasons above. The pinbar formed at an obvious support level in favour of the 21 SMA.  I'm currently backtesting entry on trend retracements and will post examples in my next post.

8th May 2012 - currencies in trend

I'll continue my backtest on "quality" inside bars in the future. I did some more backtesting from 2001 to 2004 and the results weren't pretty. Overall, the trading system looks profitable but there was massive drawdown between 2001 and 2004, along with a massive win spike during 2010 and 2011. The distribution of wins and losses is significantly uneven. I may post details later on and perhaps extend the backtest to the 90s.

Currencies in trend

AUDJPY

Price on the AUDJPY is currently in a bearish trend. If price retraces to the marked resistance level and produce an entry signal, this will look like an excellent trade. The main obstacle would be the VBN 80.000.

USDJPY

Another Yen pair, the USDJPY is likewise in a bearish trend. However, it has already retraced so I may have missed my chance to enter. The quality of this potential setup isn't as good as the AUDJPY.

XAUUSD

Gold is telling an interesting story. We may see the yellow metal hitting $1500 per ounce in the next month or so. I'm expecting a bounce from $1550/oz, and will short if an entry signal is generated around $1610/oz.

"Quality" inside bar backtest - extended

I've spent the last fortnight backtesting inside bars both on the daily and 4HR timeframes. My backtest on the 4HR TF is still incomplete, but I've done further work on the daily TF since my last post and am somewhat satisfied with my results.

Without further ado, here are the final results of my "quality" inside bar backtest on the daily TF.

TRADING STRATEGY

- An inside bar must form within the top or bottom half of the previous "mother" candle.

- Place pending entry at both ends of the inside bar EXACTLY

- Place stop loss at EXACT opposite end of the inside bar

- If one entry is triggered, cancel the remaining pending entry

- Place TP using a 2.5:1 reward : risk ratio (the backtest indicates this is optimal, although lower R:R to 1:1 is also profitable. Please see summary).

- Don't trade bars that form during weekends or global holidays (Easter, Xmas, NY). Nation-specific holidays like US Memorial Day seem okay, though.

What is the rationale behind this trading system?

When an inside bar forms in the top or bottom half of the "mother" candle, it tends to be the result of market indecisiveness where once-strong buying pressure is now giving way to sellers, or vice versa. We trade the break of either end of the inside bar because we believe the inside bar is either:

a) a minor retracement or "rest period" before a breakout (or continuation of breakout)

b) a tipping point for a reversal

What a "quality" inside bar should look like...

Here are a few visual examples of tradable inside bars. Note how they fit in the top or bottom 50% of the "mother" candle.

Backtest results

Eleven currency pairs were backtested from 2004 to 2011 / early 2012. Below is a summary of return on risk for all pairs tested, where:

Return on risk = (average win * win rate (%) - average loss * lose rate (%)) - spread

                                                        size of average stop loss

Comments

The sample size is "only" 322, but spans most of the major pairs (EURUSD, USDJPY, USDCHF, AUDUSD, XAUUSD) as well as some minor ones over a space of seven years. More samples will likely change the average return on risk, but shouldn't destroy profitability.

A R:R of 2.5 seems to be best. However, lower R:R are also profitable, so a scaling-out exit strategy is possible.

$10,000 Scenario

The scenario begins with a $10,000 starting account. Three risk levels were analysed, a moderate risk level (2% risk), aggressive (3%) and ultra-aggressive (5%). The scenario utilised all trades recorded in the backtest. A R:R of 2.5 would be used for all trades.

By the end of 2011, a non-taxed $10,000 account would yield the following:

The average annual growth rate is as follows:

The following is a graph of the running balance for each risk level. 

Possible ways to improve this strategy

- Use support / resistance lines to determine entries and/or exits (very subjective and difficult to backtest)

- Use a moving average and trade with trend, or to stay out of flat markets

- Trade the break of both ends of the inside bar (difficult to backtest due to inaccuracy, makes system susceptible to whip-sawing action)

- Loosen position criteria of inside bar (instead of top or bottom 50% of "mother" candle, extend to top or bottom 61.8% etc)

Weaknesses to this strategy

- Susceptible to ranging markets

- Relatively small sample size (but takes up a large swathe of the 2004-2012 daily TF data)
- "Dumb" strategy, pays little respect to market context (but also makes it impervious to emotional weakness if traded to the letter)
- Variance in annual growth
- Negative growth in first year, indicates annual negative growth is possible