How to calculate profit factor across different currency pairs

I stumbled upon this by accident last month as I was reviewing my systems.

I normally develop my system via a simple process:

1) Develop a hypothesis to base my system on

2) Test my hypothesis against 11 currency pairs (the majors + Yen crosses) from 2001 to 2012

Every currency pair is unique,. Each pair is influenced by a unique set of technical and fundamental factors, so you'll never see perfect replication between two pairs. There will always be some correlation, but this correlation ebbs and flows.

Since each pair is more-or-less unique, I treat the data from each pair as a discrete set.

I will iterate my hypothesis across one set of data (currency pair), which in the end will provide me with important metrics like win% and profit factor.

Do this eleven times, and you'll have eleven profit factors describing each data set. If the hypothesis stands in the vast majority of data sets or currency pairs you test against, then you most likely have a robust tradable system.

In the past, I would then find the average of the eleven profit factors I've calculated, and assume that this averaged profit factor would describe the profitability of my new trading system.

Example:

Suppose you test an hypothesis or idea on two pairs, the USDCAD and AUDUSD. The profit factor for each pair is 1.5 and 2.0. Our averaged profit factor is thus 1.75, and it would make sense to think that the profit factor of our newly-developed trading system is 1.75 if we trade the USDCAD and AUDUSD.

This is errornous once you being to dissect the mathematics involved.

Suppose we test another idea against two pairs, EURUSD and GBPUSD. We collect ten sample trades from each pair using a 1:1 R:R.

With the EURUSD, we find that we win 1 trade, and lose 9. with 1:1 R:R, our profit factor is thus 1/9 = 0.11.

With the GBPUSD, it's the opposite and we find ourselves with 9 wins and 1 loss. Our profit factor is thus 9/1 = 9.

If we find the average profit factor from both pairs, we get (9 + 0.11)/2 = 4.55.

Wow! A trade system with a profict factor of 4.55 would be extremely good.

Intuitively, however, we know this is wrong. If we combine the results from both pairs, we have 10 wins and 10 losses using a 1:1 of R:R.

Therefore, to get an accurate answer, we must combine our wins and losses from each data set, and THEN compute our profit factor. In our case, 10 wins / 10 losses = 1.

This may be highly irrelevant to some traders, depending on how they trade or develop their own trading systems. For me personally, this is an important lesson.

The lesson: when you're combining data from different sets or currency pairs, calculate profit factor at the END. Don't take a shortcut and use the average.