Optimisation vs Curve-Fitting

Analysing past data for patterns to find the probability of them repeating

It is actually quite easy to make any strategy look good by over-optimising it. All you need to do is run enough optimisation tests in MetaTrader and then only show the profitable results – unfortunately this is when people first discover that “past performance should not be relied upon for future results”!

There are many more ways to lose than to win

Nowadays, it seems that the Internet, as well as liberating almost limitless information, has become a minefield for miss-guidance and system vendors – making more money from selling systems, strategies and EAs than from trading them.

There are some gems out there but often they will be “educated” and lose their edge over time as they get over traded and the stop levels become targets for bigger traders – hence the need to create strategies that evolve with the markets, and not to follow the herd when thousands of traders are trading the same EA.

Over optimisation and curve-fitting

It is almost too easy to over-optimise an EA, and the biggest deceiver in looking for the magic settings can often be ourselves.

There’s an excellent ebook by the founder of collective2.com automated systems called How to Lose All Your Money Trading. It explains how unscrupulous system vendors can easily create EAs supplied with pretty back-testing results, showing you how brilliant they are. Plus, how a simplistic view of how to use backtesting optimisation can make people think they have discovered the holy-grail of trading and deceive themselves to ignore all the many permutations that lose money.

Target optimisation results

One important tip is to set your EA to trade 0.1 lots on a micro-account for every trade without any compounding (that is, increasing the lot size with the account size) and use a £/$1,000 starting balance (the same currency your account is in). This will give you results where the £/$ amount balance, minus the original 1,000, will be exactly in pips.

As a rule an EA that can average a minimum of 1,000 pips per year – or 80 pips per month, will give you an account growth rate of 100% year on year. So, 500 pips per year – 50%, 2,000 pips, 200% and so on. It doesn’t sound that difficult but in practice there are of course other factors to consider that will be linked to your personal tolerance for risk to gain those elusive rewards.

This all need needs to be achieved within reasonable drawdown limits, a highly successful strategy will still have drawdowns of about 10% over any year but anything more than 30% and your strategy is in serious danger from changing market conditions eliminating your edge before you realise.

Remember, the bigger you risk per trade, the bigger the drawdowns will be in proportion to your account – this is why it is so often repeated that conservative is is about 1% per trade, most professional systems traders will trade 2-3% risk per trade and 5% risk is considered quite high – not many people have the discipline to trade the kind of drawdowns that high risk will inevitably yield without prematurely losing faith or adjusting (curve fitting) their strategy.

Another metric to look at is the profit ratio, you are aiming to achieve 2 as a minimum and a target of 3 would be fantastic. There are others, which you can read more about at MyFXBook.com.

Genuine Optimisation

So, how do we create and optimise a strategy? Firstly it starts by eliminating strategies that only trade based on averaging indicators alone for entry and exit signals. Why? Because the account size and portfolio necessary to take advantage of these slight edges is normally beyond the funds or regular private traders and the edge normally only proven over such a long period of time you could be better off investing in regular asset with a similar expectancy.

This brings us to price-action strategies based upon patterns and strong levels – they generate stronger and more reasoned entry and exit signals but require more in-depth programming to create.

Once you have a good strategy developed and programmed you can run your optimisation backtests in MetaTrader’s Strategy Tester. What you are looking for is a wide cluster of settings that are profitable, and more settings that are profitable than loss making, then to select the settings in the middle-ground of the profitable group where settings either side are still more or less profitable too.

This will all give us a much stronger edge, and one that can be adapted as we acquire new price data to continually re-test based on recent performance – this is what people mean by adapting to changing market conditions. But be careful not to change things too often, otherwise you are just curve-fitting again.

There is a way to do all of this with our computer modelling techniques before we start risking real money to, which brings us to the next stage of developing automated strategies…

Walk Forward Testing

One way we can avoid over-optimising a strategy, but still optimise it to realistically evaluate future potential is to use the walk-forward approach. This is considered the gold-standard for scientific computer-modelling, so it would be interesting to ask an EA vendor to provide these results if they really are professional.

It involves breaking down your backtesting data into smaller periods, lets say 3 month sections over 4 years, giving us 45 (48 months minus the initial 3 months) opportunities to find settings that work. We start with the first period and then moving forward 1 month to see how they would have worked over the next period and so on. If after repeating this process we can find a reasonably broad range of settings that work over the full period of the tests, then do we have something worth taking further.

The best software we use, designed to do this with MetaTrader, is the Walk Forward Analyser from easyexpertforex.com, created by our good friend, Andrew Young, author of the excellent Expert Advisor Programming book. Again, you’ll need to be confident and have the where-with-all to spend the time doing this but the sort of this compared to potential losses from skipping this important testing method could be very expensive.

Demo account testing

After all that you need to see how your EA will actually trade on new market data. So, get yourself a demo account, or many, load up your EA on a fresh £/$1,000 account balance, link the account to your myfxbook.com account, trade 0.1 lots and gather the statistics on what really happens with future unknown price data.

The frequency that your strategy trades will dictate how long you want to run these tests for but our recommendation is to stick to this stage until you have at least 50-100 trades successfully executed or a month or so of trading history to analyse.

Test live trading at the minimum size first

Demo account trading will not give you real-world feedback on things like widening spreads and slippage, and of course any slight difference in execution limits between demo and live accounts.

Now you need to repeat the above process, again trading the minimum 0.1 lots per trade and gather another 50-100 trades or at least a month of statistics for live trading.

Then, and only then, can you start to increase the lot size that you trade.

Ongoing optimisation

At the end of each reasonable amount of trades, say again 100-300 or 1-3 months worth of trading, you should re-run your backtesting and walk-forward testing including the new data and see if your settings are still in the middle of optimal settings with profitable settings either side.

Then if the middle-ground settings have moved a bit, then you can too but be careful again to only work with settings where those slightly more or less than the values you use would still be profitable – you got it – to avoid curve fitting and deceiving yourself. If you can’t do this then it is back to the strategy development, first make sure it is as simple as possible and then build up each parameter one-by-one to see which have a positive effect, and which might just have worked for random reasons but are not really necessary for an over-all positive edge.

The best strategies are the simplest ones, the less settings there are, the less permutations there are to run optimisation tests on and the more powerful those average settings results will be. This is why we recommend no more that 3 entry and exit signal parameters, other signals, even with many years of backtesting data might still give results that are not statistically significant enough to be a guide to how they may perform in future.

The end of the beginning

There you have it – that’s the secret sauce, the magic formula, not quite the holy grail but this is how we do it. You can now go on, and work through all of this yourself, and you can look further into some of the indicators, strategies and we have created knowing a lot more about how we do it. Hopefully you will have learnt something new and will continue to learn to trade with more confidence, have a better understanding on what it takes to have a long-term edge, and become a successful trader and investor for yourself.

Copyright security with Digiprove certification © 2012All Rights Reserved