This article is a FAQ about the different questions that been asked on forums for the use of the new Walk Forward analysis tool of ProRealTime.
Why using Walk Forward analysis?
Prerequisites: what is the Walk Forward analysis? (to be read for a perfect comprehension of this article please!).
As I already talked here and there about over-fitting, it’s always a better idea to backtest a strategy on 70% of history while you develop it (IS), and then test it on the remaining 30% (OOS). If your result look similar in OOS than your IS test, the strategy have a better chance to continue performing the same way. Why? Because the 70% have certainly included a wide variety of price behaviour of the instrument (ranging, trending, market noises, economic news,..).
With the WF Module you can now multiply this kind of tests in the past history with X repetitions: understand X times making the same test in the past, optimise 70% and test in 30%, and so on until the end of the tests history.
Warning, Walk Forward analysis is not going to make a bad strategy over-optimized, better strategy less over-optimized.
It is a tool that allows us to draw conclusions. In the Walk Forward tool, we have several instances, which allow us to simulate several times as a result of the trading ‘real time’ strategy and this for all the duration of our history. It could be done only once (only one occurrence), but we would only have one validation period. By multiplying the occurrences, we thus have more validation periods, allowing us to check if they have a good WFE.
Please keep in mind that this module is only suppose to help you have a clue about your strategy robustness by testing optimised variables in Out Of Sample periods (OOS), in 1 or more occurrences.
What is the WFE (Walk Forward Efficiency)?
The Walk Forward Efficiency is the “quality” ratio of the test to determine the robustness of the strategy.
It is simply calculated by dividing the annualized return of the out-of-sample period (OOS) by the annualized return of the optimized sample (IS).
Logic, since these two periods do not have the same duration, we must be able to compare them together, so we annualize their performances.
If the WFE is greater than 50%, optimization is considered successful, since the performance of the strategy in the duration excluding the sample was at least 50% with respect to our optimization.
The more important it is, the better it would be. If it was lower, then probably optimization would have failed, the optimization would have led to an “over optimization”: one is too adapted to what happened and without worrying about how the strategy could perform in the future. It is here that takes advantage of this mode of optimization.
Warning: If an OOS period of the walk forward test is making too much profit comparing to the IS period, this is not so good. To be clear, the WFE (walk forward efficiency) must be superior to 50% (made at least 50% annualized profit than the In Sample period), but if it made.. let’s say 300%, this is something you couldn’t expect because it is not the “normal” behaviour of the optimised period (IS). Of course you would have been pleased by this result in real life, but your strategy were not designed to make so much profit! so it is not “normal”.
In conclusion, WFE>50% is good, but it should not be too high too 🙂 What we expect in an OOS test is that the strategy perform them same as in the IS period.
Which optimised variables are the most suitable for my strategy?
All of them are the best for each iteration that give you the best result for the whole backtest period. But it doesn’t mean that all ‘out of sample’ tests are good,
that’s the reason why the WFE (Walk Forward Efficiency) is given as the result of the test for each iteration.
The WFE calculate the ratio of the annual performance of the In Sample period (the optimised period) compared to the Out Of Sample period (the test period of the iteration). A good WFE must be superior to 50% (but not too high too, read paragraph above).
If your strategy is not over-fitted, the OOS period should have a result >= 50%.
You can add as many iterations as you like when you are optimising a strategy,
it can help you determine how much time your optimised variables are good according to the market behaviour and so when you need to optimise it again.
So the last optimised variables of your WF analysis should be the ones to keep for your real trading, if of course this last OOS test have at least a 50% Walk Forward Efficiency score and the whole strategy is considered as a robust one.
Walk forward analysis will tell you how and when you should re-optimise parameters of your strategy and see if it were a good idea to optimise them, because of the past behaviour of each optimised values in the out-of-sample territories of your backtest. I told you that you should take the last optimised values for the real trading.
It is true as long as the last backtest date is yesterday’s date and if you plan to launch your strategy today. Considering that a 70/30% walk forward test has been validated X times over the whole period of the backtest we can conclude that the optimisation should be run every X months.
A robust strategy is a strategy that adapts to the market and that benefits from the re-optimization and that is simulated with these X repetitions (5 by default).
If the WFE had been good on the X out-of-sample occurrences, then it could be said that the strategy is robust because it stalls properly with the market if optimized.
It can therefore be trusted for the future (so the period to be traded is then the Out Of Sample period).
Anchored period or unanchored Out Of Sample?
There is no good or definitive answer to this question. It is up to you. For the anchored period, the beginning of the IN SAMPLE period window will stay fixed at the initial/original starting point of the backtest, while the unanchored one is always shifted forward for each test to encapsulate the chosen percentage of the whole history divided by the repetitions number.
If you asked me, I would tell you that I prefer the unanchored mode because of its accuracy with the recent instrument/security behaviour, while the anchored mode would represent an “average” of the whole history price behaviour.
Analogy with real-time trading
Let’s imagine you have optimised a strategy over the last year (from January to December 2016), this is the IN SAMPLE period. The result seems great to you and then you decide to give it a try with your live account, so since January 1st 2017 you are trading it live, this is the OUT OF SAMPLE period. In March you take a look at the results and they are not the one expected, they are very bad instead. In conclusion, you have lost 3 months and real money.
The WF module would have saved you from this inconvenience by making this job for you, on your computer screen while drinking a cup of coffee : optimize a set of data history and test it in the last set of it.
Because I seen so many times and so many people encountered this situation, I believe this tool was definitely something needed. It is now available, please use it! 🙂
3 different ways to use the new Walk Forward module
So to be clear, this is 3 ways to use the new Walk Forward module:
1/ if you want to optimise the old way, you can! but instead of using the old optimiser, use the WF module and use only 1 iteration with 70/30 (better chance to have a strategy to be not so over-fitted).
2/ you can also use the WF to determine how much (per your customized period, per year, per month, etc..) and so when relaunch a new optimisation by making many iterations in the WF window: if many of the OOS tests are >+50% (WFE=Walk Forward Efficiency), the strategy should be robust and benefit of X optimisations per period.
3/ use it with no optimisation at all. Instead, make an optimisation with a dummy variable that means nothing in your code to get the result of the OOS test, instead of do it manually (develop in 70% period and test in 30% period, like I said in the previous paragraph). This should replace completely the way you are using the backtester now!
To illustrate the point 3. above, I made a simple strategy with a dummy variable to get the benefit of the walk forward module that test the OOS period automatically for me. This dummy variable named “z” is not optimised at all, because its range begin to 1 and finish to … 1 too 🙂
I hope It is now clear for everyone 🙂
- Walk Forward module/analysis: the tool provided in the platform in the optimisation window
- WFE (Walk Forward Efficiency): the result criteria of each OOS period (should be >50%)
- IS (In Sample period): The period of history when the variables have been optimised
- OOS (Out Of Sample period): The period of history when the strategy been tested with the previous IS best optimised variables values