How to avoid over optimization in algos ?

Forums › ProRealTime English forum › ProOrder support › How to avoid over optimization in algos ?

Viewing 15 posts - 1 through 15 (of 23 total)
1 2
  • 05/13/2020 at 6:07 PM #131544
    zilliq

    Hi Guys,

    Let’s begin a new discussion about Algos 🙂

    Since some days I do a lot a lot of Walk Forward on different algos to test robustess of them

    The conclusion is often the same (Cf picture):

    During the first step, on Backtest, In Sample, the results are very Good, but on the next step on Walk Forward, Out Of Sample, the results are not so good and often bad 🙁

    As we all know, one of the reason is the Sur-Optimization during the Backtest, because PRT test thousands combinaisons and find the best in the past, who is not necessarily the best parameters in the Future

    So for those of you who trade algos, how do you fight against Sur-Optimization and do you think that less Optimization give better result in Out Of Sample data ?

    Zilliq

    PS: And now I will test an idea about the fiability, or not, of the Walk Forward, because it could be one of the problem

    1 user thanked author for this post.
    avatar bertrandpinoy
    05/13/2020 at 7:16 PM #131558
    stefou102

    Hello,

    for me the debate should be split in 4: entry, stop, filters and TP. For entry, for me it’s pretty easy to avoid sur optimisation, as your entry should always be an entry that I would have taken manually too. For SL,  if you based yourself on preceding candles, hard to over optimized. The issue is  more difficult for TP because often you can’t avoid a percentage of gains before activating a trailing or something. Again, basing your system only on candles helps. What is more subtle and very difficult to not optmized are filters, because often you design filters just by trying to remove all losing backtest trades. Maybe sticking to same kind of filters that have demonstrated robustness in the past?

    05/13/2020 at 7:17 PM #131559
    zilliq

    Well I finish the test

    I explain

    I have an algo who works well since 14 april, after a backtest of 2 weeks (31 march/14 april)

    So I did a WF from 31 march to 6 may (2 weeks+3 weeks of WF) to see if the “prediction” of the Walk Forward was good or not

    Surprisely, the results in Live are quite similar to the prediction of the Walk Forward as you see !

    Very interesting

    2 users thanked author for this post.
    avatar Bel, avatar bertrandpinoy
    05/13/2020 at 7:27 PM #131561
    zilliq

    It was the good point, but previously I did the same WF (2 weeks ..) with the SAME algo but from 7 april to 21 april and not 31 march to 14 april and now the results are very bad …:-(

    Very surprising and “confirm” the condition of the market influence the results notably

     

    05/13/2020 at 8:16 PM #131564
    nonetheless
    zilliq wrote:

    works well since 14 april, after a backtest of 2 weeks (31 march/14 april)

    Was there some particular reason you back tested for only 2 weeks? at the development stage I would always be looking at the max data available.

    05/13/2020 at 8:57 PM #131565
    zilliq
    nonetheless wrote:
    zilliq wrote:

    works well since 14 april, after a backtest of 2 weeks (31 march/14 april)

    Was there some particular reason you back tested for only 2 weeks? at the development stage I would always be looking at the max data available.

    Hi, because:

    1/ I want to Backtest+WF on 3 or 5 weeks because for more weeks it’s very very long (2 hours to test only for 2+3 weeks!). I think 100 000 unit will take 6-8 hours

    2/ I search to have the week after the backtest profitable (re-backtest the week-end)

    It’s like a WF but for real with the parameters I want to not only the gain (I’m waiting for we can WF on our indicators)

    3/ In my experience, even with more duration in backtest doesn’t give better results. I search a ratio of 2/1 or 3/1 (2 or 3 weeks of backetst for one week in live)

    4/ The more the duration is, the more the max drawdown you have

    Bye

    05/13/2020 at 9:01 PM #131566
    zilliq

    An example of an algo in live after a good backtest on 50 000 units (I call a good candidate is >95 % winning trades, max drawdown minimum ration winning/loser high and so on ..)

    As you see not better than after a backtest on less weeks

    05/13/2020 at 9:26 PM #131569
    Vonasi

    I think the fact that you keep mentioning weeks for your in sample and weeks for your out of sample is where your issue might be. A few weeks of data is a tiny tiny sample. In reality any strategy developed and optimized on only a few weeks of data stands a very good chance of failing in the future unless you have developed it on a few weeks of data and then OOS tested it on several years of data and it still performed OK.

    Consider it like walking down a very short street and not standing in dog poo – there may have been dog poo but you somehow missed it. Then you walk down another very short street but stand in dog poo. That tells you nothing about the probability of standing in dog poo. You need to walk down a lot of streets, some long, some short and some strewn with dog poo and others with no dog poo at all before you have any idea whether walking down streets with a strategy of not looking where you put your feet is a good idea.

    05/13/2020 at 9:56 PM #131570
    zilliq

    Thanks Vonasi

    As I said, the problem is that WF is a very long period is not feasible because it will take very very long time (More than 6 hours / WF !) and I don’t find difference in my experience if I take 2, 4 or 10 weeks of backtest

    But I can retry to see

    Some of you have some pictures of Backtest and WF as I post to argue this ?

     

    05/13/2020 at 10:10 PM #131571
    nonetheless
    zilliq wrote:

    WF is a very long period is not feasible because it will take very very long time

    Maybe you’re using too many variables in the WF? With any code there are always some values that are carved in stone, you just know you’re not going to change them regardless of the instrument. I usually run it with maybe 2000 permutations, doesn’t take that long. More than that I’d leave it over night.

    05/13/2020 at 11:32 PM #131575
    Vonasi

    It all depends on how you are using walk forward. There is the simple walk forward where you test on an in sample section of data and then test the same strategy with the same fixed variables on an out of sample section of data. Then there is the optimized walk forward where you test and optimize on a small in sample bit of data and then optimize on out of sample bits of data to see if all data samples tell you that your variables are working pretty well within a fairly small range of values on all data samples.

    The walk forward also allows you to see how your strategy performs with varying candles as your starting point so you get to see trades that perhaps the original strategy would have missed because you already had a trade open.

    It is not rocket science it is just testing on different bits of data samples with different starting points and seeing if you get similar results! If you do that is good – if you don’t that is bad!

    1 user thanked author for this post.
    avatar zilliq
    05/14/2020 at 8:38 AM #131612
    GraHal
    Vonasi wrote:

    and then optimize on out of sample bits of data

    I know you meant to say … test on out of sample bits of data … but I thought I’d point it out in case a newbie got confused.

    05/14/2020 at 9:52 AM #131621
    Vonasi
    GraHal wrote:

    I know you meant to say … test on out of sample bits of data … but I thought I’d point it out in case a newbie got confused.

    No – I meant optimize. If you do a walk forward test and allow it to optimize then it will come up with different best variable settings for each overlapping section of OOS data. In the perfect world you want it to come up with all the same numbers for each section of data. The next best is that it comes up with numbers in a fairly close range. In this case the common thought is to go forward in to the future using the best settings from the very latest section of data.

    05/14/2020 at 10:13 AM #131622
    GraHal

    I agree, but you said optimise on OOS data … surely data that is optimised on … then becomes / is described as IS data??

    Anyway that is what is shown in the Optimiser pic attached.

    I guess what you may be talking about is when all of the data chosen to do Walk Forward over is OOS … as in the strategy has never been optimised over this chunk of OOS data which is then used for Walk Forward Optimisation?

    05/14/2020 at 10:25 AM #131630
    zilliq

    Tomorrow I will do differents tests Anchor/Not anchor WF, backtest on hundred of thousands to compare…

     

  • Author
    Posts
Viewing 15 posts - 1 through 15 (of 23 total)
1 2

Find exclusive trading pro-tools on ProRealCode Marketplace

Similar topics:
algos don’t want to start up in auto trading
any way to plot a time based moving average over a tick chart ?
avoid return of a value when “0”
code for over xx bar and close
crosses over vs. “>”
customizing optimization report (adding variable)
how do i speed up backtest optimization?
how to avoid overfitted strategy
is there a easier way to hide the code on algos?
optimization oddities with uneven numbers

Create your free account now and post your request to benefit from the help of the community
Register or Login