WALK FORWARD ANALYSIS – WHICH PARAMETERS TO USE?

Forums ProRealTime English forum General trading discussions WALK FORWARD ANALYSIS – WHICH PARAMETERS TO USE?

Viewing 13 posts - 1 through 13 (of 13 total)
  • #16411

    Hello

    If I’ve done a walk forward analysis – Auto System optimised over 12 x 3 month in-sample periods and tested over 12 1 x month out of sample periods.

    Optimised values were different for each 3 month period. Out of sample test results are acceptable for all test periods.

    Which set of the 12 x optimised parameters do I use for Live Trading??

    Hope above makes sense, just say if not and I’ll add more information.

    Thank You

    GraHal

    #16413

    The last one. I don’t know how you made your analysis, but the last optimised parameters should be the one that has passed successfully the whole process.

    For your information, PRT is actually working on this feature to be part of the optimisation engine of the platform. I’d be thankful if you could tell me what software you are actually using to make these analysis yourself and how it looks like?

    #16416

    Gret News Nicolas! I was hoping PRT may be developing Walk Forward.

    I am wanting to do it but have not yet, I’m just finding answers to my initial thoughts / questions.

    Let’s say one of the parameters is a MA and for each of the 12 x 3 month periods the MA optimised value is different (by just a few bars + or -) e.g.  the highest 40 and the lowest 20.

    Let’s say the optimised MA value for period 12 gave the worst test results (out all 12 periods) but still above 0.5 / acceptable.  Period 7 gave the best test results.

    Is the logic for using period 12 MA value to Live Trade  … that ‘Live  / period 13’ trading conditions are more likely to approximate to period 12 than, say, period 7, when I got the best test results?

    If I have got Walk Forward testing ‘wrong in my mind’ and it would take too much of your limited time to explain then just say … read the book! ha

    I am on page 1 of ‘The Evaluation and Optimization of Trading Strategies’ by Robert Pardo (the guy who invented Walk Forward testing, so he says!) so I can answer my own question soon (I hope!).

    I though it good to get a discussion going on here as Walk Forward seems to be the way forward?

    Thank You

    GraHal

    1 user thanked author for this post.
    #16417

    Ha … I just scrolled through the whole chapter (in the above book) on Walk Forward Analysis and still couldn’t see details on which period’s optimised parameters to use for Live trading. The Author seems to use 100 words where 10 would be better (exaggeration of course) but I guess he needed to fill the book!?

    Over the last couple of days, I’ve reserched 6 or 7 web sites on Walk Forward and they all seem to ‘stop short’ on this point … unless I’m blind to the details!? ha

    Cheers

    GraHal

    #16420

    Browsed many website to find a good example and explanation, here is what I found the most “simple” at strategyquant.com website:

    What is Walk-Forward optimization/analysis?

    Walk-Forward optimization is generally a special type of backtest that is composed of multiple smaller backtests on optimizaiton periods. These optimization periods are split over the whole backtesting period and are always followed by Out of Sample tests with the optimized parameters.

    It is a technique in which you optimize the parameter values on a past segment of market data, then verify the performance of the system by testing it forward in time on data following the optimization segment, and the process can be repeated over subsequent time segments.

    How Walk-Forward optimization works

    in walk-forward optimization, the data are divided into a configurable number of periods (5 in this example). Each period consists of optimization part and run part.

    The program starts with optimization period 1. It will run the simple optimization on optimization period 1 to find the best parameter values. These parameter values are then applied to run period 1 – strategy is trading with the optimized parameters found in previous step.

    At the end of run period 1, the system again runs simple optimization on a part of data marked as optimization period 2. It finds the best set of parameter values and they are again used for trading in run period 2.

    This continues until the period 5, which is also the end of history data used in test.

    Walk-Forward optimization simulates how you could work with the strategy during real trading – you can optimize it on some historical data and then trade it with the optimal values. After some time you’d want to reoptimize it and let it trade again.

    What Walk-Forward optimization/analysis tells you?

    It basically tells you if the startegy is robust enough and if its performance can be improved by reoptimization.
    If strategy performance is worse during reoptimization than the original non-optimized startegy, it is a big signal to watch for curve fitting.

    On the other side, if Walk-Forward optimized strategy performs better than non-optimized version on the same data, it tells you that :

    1. Your strategy will benefit from optimization, so you should periodically reoptimize it to get the best performance
    2. It also means that the startegy is robust enough to cope with market changes (using reoptimization) and there is a big chance it will work also in future.
    4 users thanked author for this post.
    #16421

    Walk forward analysis will tell you how and when you should re-optimise parameters of your strategy and see if it were a good idea to optimise them, because of the past behaviour of each optimised values in the out-of-sample territories of your backtest.

    I told you that you should take the last optimised values for the real trading. It is true as long as the last backtest date is yesterday’s date and if you plan to launch your strategy today.

    #16425

    Thanks again Nicolas.

    From the strategyquant.com website, this bit confuses me … If strategy performance is worse during reoptimization than the original non-optimized startegy, it is a big signal to watch for curve fitting.

    How could an optimised strategy ever be worse than a non-optimised strategy? Only reason I can think of is if the range of values used for optimising did not include the figure(s) in the non-optimised strategy (something I never do).

    Ignoring Drawdown etc, if MA(40) gives a profit of £10K and values for optimisation used are 5 to 100 then the result cannot be less than £10K (because 40 would show as the optimum value producing £10K profit).

    If values for optimisation used are 50 to 100 then yes the max profit could well end up less than £10k. But why would anyone not want to compare ‘like with like’ during optimisation?

    Or am I missing something here?

    Cheers

    GraHal

    #16427

    Aha I have missed something (above)!

    I was forgetting that the optimised figures are then applied to a sample of date outside the period used for optimisation! doh! 🙂

    To make unambiguous sense, the statement on the strategyquant.com website should have read (with the bold additions) …

    If strategy performance is worse during testing following reoptimization than the original non-optimized startegy, it is a big signal to watch for curve fitting.

    Hey I’m glad I made that blunder … it has helped to fix in my mind the advantage of Walk Forward Analysis … roll on it being available in the PRT Platform!

    GraHal

    #16475

    I woke up this morning with another unanswered question on my mind.

    When I have done my initial Walk Forward Analysis and I then set it to Trade Real or Virtual Money  … how do I know how often / what period to re-optimise over?

    Results for a Walk Forward Matrix are required as illustrated below and more detail here

    http://strategyquant.com/articles/walk%20forward%20matrix

     

    Nicolas, if you attend VIP meetings / have an input (?), might you be able to propose to the ‘PRT Techie Elves’ that Walk Forward Matrix (or alternative to get the same result) is included in the future release of the PR Platform with Walk Forward?

    Many Thanks

    GraHal

     

    1 user thanked author for this post.
    #16478

    Walk Forward Matrix will be part of this project.

    3 users thanked author for this post.
    #20139
    #20144

    Thanks Dowboy. For your information, WFA is currently in development at PRT. I had the chance to participate to the project to give advices and I know it will be a big improvement for the optimization module of probacktest. I hope I’ll get into the beta soon, I’ll give news as soon as possible.

    1 user thanked author for this post.
    #29654

    Hi everyone!

    I found out that Walk-Forward Analysis was just made available on Demo accounts in PRT… I started to use it (I had been waiting for this for a long time on PRT) and it looks nice. I guess that it should become available on Real accounts as well in a few days.

     

    1 user thanked author for this post.
Viewing 13 posts - 1 through 13 (of 13 total)

Create your free account now and post your request to benefit from the help of the community
Register or Login