How to use the ProRealTime Walk Forward analysis tool – FAQ

This article is a FAQ about the different questions that been asked on forums for the use of the new Walk Forward analysis tool of ProRealTime.

Why using Walk Forward analysis?

Prerequisites: what is the Walk Forward analysis? (to be read for a perfect comprehension of this article please!).

As I already talked here and there about over-fitting, it’s always a better idea to backtest a strategy on 70% of history while you develop it (IS), and then test it on the remaining 30% (OOS). If your result look similar in OOS than your IS test, the strategy have a better chance to continue performing the same way. Why? Because the 70% have certainly included a wide variety of price behaviour of the instrument (ranging, trending, market noises, economic news,..).

With the WF Module you can now multiply this kind of tests in the past history with X repetitions: understand X times making the same test in the past, optimise 70% and test in 30%, and so on until the end of the tests history.

Warning, Walk Forward analysis is not going to make a bad strategy over-optimized, better strategy less over-optimized.
It is a tool that allows us to draw conclusions. In the Walk Forward tool, we have several instances, which allow us to simulate several times as a result of the trading ‘real time’ strategy and this for all the duration of our history. It could be done only once (only one occurrence), but we would only have one validation period. By multiplying the occurrences, we thus have more validation periods, allowing us to check if they have a good WFE.

Please keep in mind that this module is only suppose to help you have a clue about your strategy robustness by testing optimised variables in Out Of Sample periods (OOS), in 1 or more occurrences.

 

What is the WFE (Walk Forward Efficiency)?

The Walk Forward Efficiency is the “quality” ratio of the test to determine the robustness of the strategy.


It is simply calculated by dividing the annualized return of the out-of-sample period (OOS) by the annualized return of the optimized sample (IS).
Logic, since these two periods do not have the same duration, we must be able to compare them together, so we annualize their performances.
If the WFE is greater than 50%, optimization is considered successful, since the performance of the strategy in the duration excluding the sample was at least 50% with respect to our optimization.
The more important it is, the better it would be. If it was lower, then probably optimization would have failed, the optimization would have led to an “over optimization”: one is too adapted to what happened and without worrying about how the strategy could perform in the future. It is here that takes advantage of this mode of optimization.

Warning: If an OOS period of the walk forward test is making too much profit comparing to the IS period, this is not so good. To be clear, the WFE (walk forward efficiency) must be superior to 50% (made at least 50% annualized profit than the In Sample period), but if it made.. let’s say 300%, this is something you couldn’t expect because it is not the “normal” behaviour of the optimised period (IS). Of course you would have been pleased by this result in real life, but your strategy were not designed to make so much profit! so it is not “normal”.

In conclusion, WFE>50% is good, but it should not be too high too 🙂 What we expect in an OOS test is that the strategy perform them same as in the IS period.

 

Which optimised variables are the most suitable for my strategy?

All of them are the best for each iteration that give you the best result for the whole backtest period. But it doesn’t mean that all ‘out of sample’ tests are good,
that’s the reason why the WFE (Walk Forward Efficiency) is given as the result of the test for each iteration.
The WFE calculate the ratio of the annual performance of the In Sample period (the optimised period) compared to the Out Of Sample period (the test period of the iteration). A good WFE must be superior to 50% (but not too high too, read paragraph above).
If your strategy is not over-fitted, the OOS period should have a result >= 50%.
You can add as many iterations as you like when you are optimising a strategy,
it can help you determine how much time your optimised variables are good according to the market behaviour and so when you need to optimise it again.

So the last optimised variables of your WF analysis should be the ones to keep for your real trading, if of course this last OOS test have at least a 50% Walk Forward Efficiency score and the whole strategy is considered as a robust one.

Walk forward analysis will tell you how and when you should re-optimise parameters of your strategy and see if it were a good idea to optimise them, because of the past behaviour of each optimised values in the out-of-sample territories of your backtest. I told you that you should take the last optimised values for the real trading.
It is true as long as the last backtest date is yesterday’s date and if you plan to launch your strategy today. Considering that a 70/30% walk forward test has been validated X times over the whole period of the backtest we can conclude that the optimisation should be run every X months.

A robust strategy is a strategy that adapts to the market and that benefits from the re-optimization and that is simulated with these X repetitions (5 by default).
If the WFE had been good on the X out-of-sample occurrences, then it could be said that the strategy is robust because it stalls properly with the market if optimized.
It can therefore be trusted for the future (so the period to be traded is then the Out Of Sample period).

 

Anchored period or unanchored Out Of Sample?

There is no good or definitive answer to this question. It is up to you. For the anchored period, the beginning of the IN SAMPLE period window will stay fixed at the initial/original starting point of the backtest, while the unanchored one is always shifted forward for each test to encapsulate the chosen percentage of the whole history divided by the repetitions number.

If you asked me, I would tell you that I prefer the unanchored mode because of its accuracy with the recent instrument/security behaviour, while the anchored mode would represent an “average” of the whole history price behaviour.

Analogy with real-time trading

Let’s imagine you have optimised a strategy over the last year (from January to December 2016), this is the IN SAMPLE period. The result seems great to you and then you decide to give it a try with your live account, so since January 1st 2017 you are trading it live, this is the OUT OF SAMPLE period. In March you take a look at the results and they are not the one expected, they are very bad instead. In conclusion, you have lost 3 months and real money.

The WF module would have saved you from this inconvenience by making this job for you, on your computer screen while drinking a cup of coffee : optimize a set of data history and test it in the last set of it.

Because I seen so many times and so many people encountered this situation, I believe this tool was definitely something needed. It is now available, please use it! 🙂

 

3 different ways to use the new Walk Forward module

So to be clear, this is 3 ways to use the new Walk Forward module:

1/ if you want to optimise the old way, you can! but instead of using the old optimiser, use the WF module and use only 1 iteration with 70/30 (better chance to have a strategy to be not so over-fitted).

2/ you can also use the WF to determine how much (per your customized period, per year, per month, etc..) and so when relaunch a new optimisation by making many iterations in the WF window: if many of the OOS tests are >+50% (WFE=Walk Forward Efficiency), the strategy should be robust and benefit of X optimisations per period.

(lot of good WFE over 10 occurrences give us an estimated time period for this strategy that could perform well and so when we should re-optimised it).

3/ use it with no optimisation at all. Instead, make an optimisation with a dummy variable that means nothing in your code to get the result of the OOS test, instead of do it manually (develop in 70% period and test in 30% period, like I said in the previous paragraph). This should replace completely the way you are using the backtester now!

 

To illustrate the point 3. above, I made a simple strategy with a dummy variable to get the benefit of the walk forward module that test the OOS period automatically for me. This dummy variable named “z” is not optimised at all, because  its range begin to 1 and finish to … 1 too 🙂

I hope It is now clear for everyone 🙂


glossary:

  • Walk Forward module/analysis: the tool provided in the platform in the optimisation window
  • WFE (Walk Forward Efficiency): the result criteria of each OOS period (should be >50%)
  • IS (In Sample period): The period of history when the variables have been optimised
  • OOS (Out Of Sample period): The period of history when the strategy been tested with the previous IS best optimised variables values

Share this

  1. ALE • 114 days ago #

    Thnak you very much Nicolas!

  2. Elsborgtrading • 114 days ago #

    that was nice 🙂

  3. Henrik • 113 days ago #

    Thanks!!

  4. barbagio • 105 days ago #

    Thanks Nicolas, great contribution. Only a point: I suppose the optimal value for each set of parameters is defined only looking at the strategy net profit and is not possible to define a custom “objective function”.

    In any case the WFA is a great improvement of prorealtime.

    • Nicolas • 105 days ago #

      That’s right, I would like to have more sorting criteria to make optimisation instead of only “net profit”. I heard this would be done in a future version of this module, but I couldn’t say how and when..

  5. Kenneth Kvistad • 59 days ago #

    If you optimise with 5 tests, do you simply choose the one with best result out of the 5 test is and oos? When you get resulta between 50~lets say 90% wf is this good? 

    Sometimes I see umbers way higher and this, is this the same as dont bother about them and always focus on wf numbers between 50~90%.

    Also when you have found the best optimisation of 5 is and oos do you then test the same periode with the best parameters for the hole timeframe too confirm again with tick mode ? 

    I use 5 min chart and if I optimise with 5 diffrent points/times I onl test 2 weeks or so. Does this.mean that I should save the first optimised parameters and wait 2 weeks for new information and run test again in.optimisation?

    • Nicolas • 59 days ago #

      Hi Kenneth, all your questions find answers in the article. Please take time to read it again 🙂 I could answer your interrogations but it would be redundant!
      But 5 OOS tests periods above 50% and not too high is very good.

  6. Kenneth Kvistad • 59 days ago #

    Sorry if you got offended by my question. I have read it several times. 

    But what you just answerd: should all 5 test be wf 50% too 90~/100%  or is it suffisiant if only one test is good out of 5 then use this optimalisation on the intiere timeframe? 

    My old crudeoil strategy seems to be working when i test and this makes me sceptic since it stopt working after 4 weeks last time i ran it on 10’2 version. So i feelt i did something wrong. Thats all. Thanks for the answer:)

    • Nicolas • 59 days ago #

      Don’t worry I’m not offend, it proves that I’m not so clear in my article 🙂 

      Walk Forward analysis is just the same way to optimise a strategy than the normal tool, but many times in the whole history to get a clear view of what would happened if you have traded it live through X Out Of Sample periods, that’s all. So you can consider that if most of the OOS performed good with a WFE>50%, your strategy benefit from optimisation and should be considered as a robust strategy to be traded in a live environment. 

avatar
Register or

Top