Walk forward analysis?

Forums › ProRealTime English forum › ProOrder support › Walk forward analysis?

Viewing 15 posts - 1 through 15 (of 16 total)
1 2
  • 03/28/2017 at 6:10 PM #30177
    kg6450

    Hi all,

    I just had a quick question about the new walk-forward analysis feature that has been added.

    I understand why we use walk-forward analysis but what I don’t understand are the actual results given when I run one through the backtest?

    If I simply run a backtest and optimise the moving average period through 5 repetitions, when I look at the analysis results I gives me five different periods of the backtest where different variables were used, how does this show which one is best? The documentation doesn’t explain how to read these result very clearly so I was hoping someone here knew?

    03/29/2017 at 4:15 PM #30225
    Nicolas

    All of them are the best for each iteration that give you the best result for the whole backtest period. But it doesn’t mean that all ‘out of sample’ tests are good, that’s the reason why the WFE (Walk Forward Efficiency) is given as the result of the test for each iteration.

    The WFE calculate the ratio of the annual performance of the In Sample period (the optimised period) compared to the Out Of Sample period (the test period of the iteration). A good WFE must be superior to 50%.

    If your strategy is not overfitted, the OOS period should have a result >= 50%. You can add as many iterations as you like when you are optimising a strategy, it can help you determine how much time your optimised variables are good according to the market behaviour and so when you need to optimise it again.

     

    07/25/2017 at 10:15 AM #41551
    CKW

    Hi Nicholas,

    I read your blog and all other topics/forum related WF but i am still not very sure the correct way to evaluate my system.
    I attatched few screenshots from WF repetition 3, 4 and 5 (All having the same range of parameters, Not anchored, IS 70%, OOS 30%).
    Sorry my igonorance. I still can’t understand which one is better / robust / none of them? some WFE% is too high, some are slightly -%…
    Could please advice what and how you can see from these results and further improve it?

    thank you very much 🙂

    br,

    CKW

     

     

    07/25/2017 at 10:43 AM #41560
    Nicolas

    The WFA is conceived to know if your strategy benefit from optimisation. A strategy is considered as “robust”, understand good enough with optimized variables, if the WFE are good (>50% for the most of them, not too high, etc.).

    If the WFA is good enough, the strategy could be launched in real trading, with another set of optimized variables you should make yourself by optimizing on the same period than an In Sample period (on the most recent history).

    07/25/2017 at 11:37 AM #41563
    CKW

    Thanks Nicholas.

    If WFA is good (all > 50%). Another set of parameters works on recent history (e.g 2016, Same IS) but it may looks not so good in earlier history (e.g. 2015). I think it’s considered fine right ?

    2nd question. It is a must to avoid high WFE% or it’s acceptable ? Read from your blog, it could be due to some events or news. The fact is WFA only gives the highest WFE%. What I meant is other variables could possibly gives WFE% between 50~90 (but we don’t know which parameters). In this assumption, can I say it’s fine to have high WFE%?

    thanks in advance

    Br,

    CKW

     

     

     

    1 user thanked author for this post.
    avatar Despair
    07/25/2017 at 2:22 PM #41578
    Nicolas

    About the first question, we can consider that the strategy is fine, because it has successfully pass the WFA.

    2nd question: not a must, but one or two WFE% should not represent the major part of the whole strategy profit. In fact it is important to understand why the strategy has generated a lot of profit (and so a high WFE%) in these periods, that’s all. If you think it is acceptable and in accordance with the strategy, it’s up to you to consider this is a normal situation.

    07/25/2017 at 3:52 PM #41595
    CKW

    Thanks Nicholas. Fully in-line with your answer!

    05/05/2018 at 5:28 AM #69809
    CKW

    Hi Nicholas,

    May you please further explain “A good WFE must be superior to 50%”? i know how the computation works but not so clear why 50%? and i add several questions….

    1. How do you define this percentage (50%) as target perfect score ?
    2. If we change the percentage of IS and OOS (e.g. 60%, 40%), shall we expect the same WFE target (>50%) ? or different?
    3. For WFE, i assume it’s not recommended to have any parameters related to SL & TP. Otherwise, we will see significant changes on WFE% for each repetition?

    Thanks to clarify

    br,

    CKW

    2 users thanked author for this post.
    avatar Jan, avatar GraHal
    05/07/2018 at 4:12 PM #69945
    Nicolas
    1. 50% mean that the annualized profit of the OOS period has made at least the half of the IS+OOS annualized profit. It is only a simple threshold that tell us that the strategy has performed at least as good as in the IS period.
    2. Yes, because it is “annualized”, so the considered ratio is always the same, even with a IS(20%) and an OOS(80%) for instance..
    3. Why not? A SL and a TP is a way to exit trades, like with the cross of 2 moving averages. So optimization SL/TP is the same as optimizing indicators periods.. at the end 🙂
    1 user thanked author for this post.
    avatar Jan
    05/07/2018 at 5:43 PM #69953
    CKW

    Hi Nicholas,

    Thanks for your clarification

    3. I raised this question because i was a bit unsure when i see major gap %WFE between #1 repetition and # 2 repetition due to change in SL & TP (e.g. #1 WFE 250%, #2 WFE 60%). After few days of testing, i think it’s OK as long as > 50%

    br,

    CKW

     

    06/13/2019 at 11:23 AM #100530
    Jan

    Hi Nicolas,

    If you run the 5 repetitions  AT ONCE in 1 Walk Forward Analysis run, the optimised variables for the second till the fifth repetition may vary significantly from the first one.  However an overall Walk Forward Efficiency Ratio is presented, adding the five OOS results together in an annualized WFE ratio with different optimised variable settings.

    Would it be an idea to also present the SPREAD or Standard Deviation of each used optimized variable from the repetitions together with the overall WFE ratio ?

    Color green as the spread/STD < xx%, color red as the spread/STD > xx% ?

    The spread can be defined as the maximum difference between the optimised values per repetition per variable.

    For example the variable Average is optimised with 5 repetitions,  optimised values appears   85,  115,  120,  75, 125 , the spread is in this example 125 – 75 = 50,  STD is 22,5

     

    I would think this information is important, as the bigger the spread/STD per variable is , the less robust the trading strategy is and the less meaningfull the overall WFE ratio is in my opinion.

     

     

     

    06/13/2019 at 12:38 PM #100540
    Nicolas

    This idea, is just your own assumption of how we should find the best variable setting, a theory.

    Why do you think the average of an optimized variable is a good way to find it is robust? It depends of the strategy and how that variable is used IMO. But you can change my mind if you want 🙂

    The WFA is here to find the best setting for different period and market behavior along the time. It is never meant to average all settings together and to use them in the future or in past backtests. In fact that way of processing would annihilate completely what the WFA is made for. In that case, a simple optimization of the whole data history in a single IS period would have made the job!

     

    06/13/2019 at 3:02 PM #100559
    Jan

    Hallo Nicolas,

    thanks for the quick reaction.

    An overall WFE Ratio of >50% from 5 consecutive runs whereby the optimised values of the variables are more or less in the same order gives me more comfort of a robust system then when the values vary significantly. [given the fact of using sufficient data and generating statistical sufficient trades]

    In the equity graph, when running a walk forward with 5 repetitions, the OOS results are presented in the graph and in the overall WFE Ratio,  but they are based upon 5 consecutive, but completely independent WF runs with different optimised variables. Then my thought was to show the variance of the optimised variables somewhere as additional information besides the WFE Ratio.  By no means I intended to change the current WFA.  The only thing is that I would like to be informed in a way how much the optimised values of used variables varies for the consecutive WF runs when using more repetitions in one run.

    When you have great differences between the optimised values of variables of the consecutive WF runs, I have no clue what variable setting to use in live trading, regardless a positive WFE-ratio outcome.  Then it does not sound robust to me and in this case the WFE Ratio presented is, regrettable, by no means a leading indicator for me.

    Just an Example with 1 variable, Average, optimized in 5 repetitions of the PRT Walk Forward; presented optimised values of Average and OOS results

    Example 1  WFE Ratio 60%    IS  80    OOS1 80   $300     OOS2  70  $-100   OOS2   82   -$100  OOS3  75  $200      OOS4    81 $50  OOS5    82  -$50  Variance between the values is small, WFE Ratio is OK, I would like this strategy

    Example 1  WFE Ratio 60%    IS  80    OOS1 80   $300     OOS2  40  $-100   OOS2   110   -$100  OOS3  150  $200      OOS4    30 $50  OOS5    90  -$50  Variance between the values is big, WFE Ratio is OK, I would doubt this strategy

     

    P.S. By the way I like the possibility to run 5 consecutive Walk Forwards in one run, as it fasten the analysis of robustness significantly.

    KR Jan

     

     

    06/13/2019 at 5:51 PM #100575
    Nicolas

    Analyzing the variance of optimized settings of a variable in a single OOS is a good idea. It is now possible with 2/3D maps in v11, did you have a look?

    06/13/2019 at 6:06 PM #100577
    Jan

    Hi Nicolas,

    No, seems that I have version 10.3  1.8.0_45 from 5 June 2019  (found at Help –> About)

    How do I come to version 11 and how to the 2/3D Maps ?

  • Author
    Posts
Viewing 15 posts - 1 through 15 (of 16 total)
1 2

Find exclusive trading pro-tools on ProRealCode Marketplace

Similar topics:
technical analysis fair

Create your free account now and post your request to benefit from the help of the community
Register or Login