Optimizing question

Forums › ProRealTime English forum › General trading discussions › Optimizing question

Viewing 15 posts - 1 through 15 (of 16 total)
1 2
  • 01/28/2019 at 10:19 PM #89872
    jebus89

    So i was thinking, lets say youre optimizing your algo on 50% of the available data so u dont curvefit, what if u optimize the other 50% and just basicly match the best variables that are/look the same in the first and the second part of ur 50% data?

    Anyone done any reading on stuff like that?

    The reason im asking is cus ive seen a couple of times a system where i have written down the “top 5” variable-combination from optimizing, then when im checking on the other 50% of my data, then i might find that 4/5 of the combinations fall apart, but suddenly the last optimized combination of variables look amazing! Which obviously leads me further down the rabbit hole: Is it safe to use that 1 combination?

    I have taken that “1 combination” and run a walk forward with variables close to the ones ive chosen and it passes the test (got it in demo now), but again im wondering if what im doing is actually considered stupid or not. Up until now i have just thrown away the whole thing if the optimized (on 50% of the data) looks like shit when checking the whole 100%, but what if i just chose the wrong combination of variables when i chose to check on the 100%?

     

    Thoughts?

    01/28/2019 at 10:38 PM #89878
    Vonasi

    This is the problem with WF testing. If the first half of the data looks identical to the second half then WF testing is pointless. So we either have to decide to look for the holy grail that fits all the data we have or create a strategy on either all or some of the data we have and then break it down into random sections of all the data we have with random start points and see how the strategy copes. Just because every section is not exactly as profitable as the others or even profitable at all does not mean it is a bad strategy. Unfortunately WF analysis throws us into a whole new world of having to decide what is bad and what is just good enough and what is fantastic – we rarely see the latter but deciding to risk money on the former is sometimes difficult.

    01/28/2019 at 10:40 PM #89879
    GraHal

    I like problems in small bites until I understand.

    So how many variables are there in your System?

    EDIT  / PS

    No need to answer now,  I was asking as you had no replies, but Vonasi has commented now.

    01/28/2019 at 10:54 PM #89883
    Vonasi

    For me WF testing is just taking a strategy and testing it on bite size pieces of history. If you have developed it on all the data then you only have the next future bite size piece of data to forward test it on – but it should cope better because you had more historical data to have developed it on. However if you developed it on a bite size chunk of history then you will be very unlikely to find that it works on the whole of the rest of history so you could be wasting a lot of time. If your strategy has a million indicators and a million variables then it is curve fitted no matter what you developed it on. If you develop it on a five minute time frame then the limited data available in PRT means that you cannot ever test it on say the 2008 market crash and if that happens again next week then all your WF testing has been pointless as it did not include a test such as that.

    01/29/2019 at 8:34 AM #89902
    jebus89
    Vonasi wrote:

    However if you developed it on a bite size chunk of history then you will be very unlikely to find that it works on the whole of the rest of history

    See this is where we differ i think. I like to optimize on a chunk of data (30-50%) and check the rest (100%) of the data. Lets say i optimize on 2006- > 2014, then i want to see the “same” results going from 2014 -> 2019.

     

    Maybe this is just me, but i feel like that is the safest way to creating something robust, basicly meaning u made an algo in “2014” and u went “live” with it for 5 years (2019) now i would feel alot more comfy running this live.

     

    question is tho, if u optimize on that 50% of the data, and find shitty results with 4/5 combinations when checkin then 100% of the data, is that robust or is that 1 combination (that you chose from out of 100’s of combination) just luck?

    01/29/2019 at 10:30 AM #89917
    Vonasi
    jebus89 wrote:

    if you developed it on a bite size chunk of history

    By that I was meaning the extreme of for example developing on a tiny chunk – say 5%. This would make it very difficult to find something that works on the other 95% – but if you did it would be very robust.

    Your 30% to 50% results are very dependent on which 30 to 50% you choose for the development. If you developed it on a 30% section of history with just a solid up trend then sure it will fail if the last 70% has two major market corrections in. It does not mean that it is a bad strategy it just means that you had no market correction filter in it – but if you add one now is that just curve fitting or just adding a needed feature that your development history data didn’t show that you needed?

    Once you know that there have been market corrections in the past then even if we are not developing on a section of data with a market correction in we will have it in our mind and so curve fit by adding some sort of filter even though we are developing on data that does not need one. Once we have spent a few hours staring at a chart it is impressed in our brain and we are curve fitting from then onwards whether we think we are or not.

    I like the fact that the last year has been so different to any of the history that we have previously developed on – it has been a great real life test for any strategies developed before it started to go all volatile and sideways on us.

    01/29/2019 at 8:44 PM #89981
    jebus89

    I agree so much on that this past year has given us some great data haha.

    Also agree with the fact that a 5% in sample and 95% oos code that works great in both is a great algo lol.

     

    And finally yes i agree with the fact that even tho we have the data we have, there is just so much more data to be had (talking about hour/minute charts now). For daily its different, not only do we not have all the data there is, but there is also the issue of electronic trading vs pit-trading. One could make the argument that electronic markets and increasing number of marketmakers and brokers and traders and funds etc, the markets have shifted into a sort of “turbo mode”. Although there are some similar traits that the pre-2000 and after-2000 eras both share, at least in my opinion. The two M’s: Mean reversion and momentum. There are also huge differences in data power and the amount of fucking competition. There used to be a few traders, now everyone with internet access can be a trader.. How this can or should or does effect your strategy or mine, i dont know.. I have some non optimized strategies for mean reverting in US 500 that seems to have delivered profitable results from the 1970’s to this day. Does this mean the same strategies that used to work, still works? Well simple answer is no. If we look at some of the “turtle strategies” that depended a lot on moving averages we can see that there is tons of more noise and choppy markets, making simple moving average strategies more shitty in the sense that they still work for those long nice trends, but they fail alot more often. At least this is my general opinion from comments from other traders, podcasts and just observing it myself, making strategies all day and night. What data to pick and choose from is a whole other level of crazyness, i try to just pick a bunch of data that has up, down, sideways, calm, volatile candles in it. Little bit of everything and hope for the best.

    And i still got the idea of optimizing both the first half, and second half of data at the same time with a copy of your system, stuck in my head. If you can get a pretty similar match of variables in the top 10 best combinations of variables in each of the the 50% data-parts, shouldnt that speak confidence? Not sure..

    I guess if you take that idea further, and leave small OOS samples out of each data set, your actually walk forward testing lol.. So i guess the conclusion i just thought of is that i need to optimize using walk forward and see if i can produce similar combination of variables to increase my confidence..

    01/29/2019 at 10:19 PM #89987
    Vonasi
    jebus89 wrote:

    So i guess the conclusion i just thought of is that i need to optimize using walk forward and see if i can produce similar combination of variables

    That is the ideal robustness test but very difficult to achieve due to the fact that one section of data will be a massive up trend, another a bit of that with half a major pull back and then the third bit will be some sideways chop with a slow start to a trend and so on and so on. Getting a narrow range of variables to work equally well in that mix is a big challenge.

    Also you have to consider that range on most rising indices increases with time so if you buy at the beginning of a day and sell at the end then later sections of data used in WF will provide higher profits than earlier sections so this lower profit does not mean that earlier sections were not as well optimized as later sections.

    01/29/2019 at 10:44 PM #89990
    GraHal

    Please forgive the daft question … how do you optimise with Walk Forward??

    WF only gives one set of results and I rarely choose the top most profitable result with the normal optimiser so I feel like I have blinkers on when looking at the results from WF??

     

    01/29/2019 at 11:38 PM #89992
    Vonasi
    GraHal wrote:

    how do you optimise with Walk Forward??

    The same as a normal optimization except you turn on WF.

    Say you then have 5 iterations – it then splits all the available data into 5 pieces and individually optimizes each section and then returns the results to show which variable values would have been most profitable for each section of data. If they are all the same then you have hit the jackpot – maybe.

    Click on each ‘x’ in the performance report to see the values used or click on the bottom of each section of the equity chart to see them.

    01/29/2019 at 11:43 PM #89993
    Vonasi

    The alternative is to just set a dummy variable from 1 to 1 that is not actually used in the code and then do WF with fixed variable values for those used in the code. This way you get to see a straight comparison of each WF section and get a WF rating for the overall strategy with those fixed values.

    02/01/2019 at 2:54 PM #90249
    O-jay8

    Hi Guys,
    Just to give my 2 cents as well.

    What I usually do when backtesting is to make trades in the past and now comparable.
    For example we take profits respectivly to percentages. A 2% take profit for Dax in 2011 is not as high as 2018 because the Dax was lower at this time.
    That means I have to adjust the position size. I need to adjust the position size depending on the actual price.

    Therefore I just use the formula positionsize = close/current price

    I want avoid evaluating more recent trades more profitable just because the price is much higher.

    02/01/2019 at 3:39 PM #90251
    Vonasi
    O-jay8 wrote:

    Therefore I just use the formula positionsize = close/current price

    That’s an interesting idea. A sort of inflation based position sizing!

    Unfortunately we don’t have inflation based spread sizing to match it in back tests.

    02/02/2019 at 2:07 PM #90301
    GraHal
    Vonasi wrote:

    The same as a normal optimization except you turn on WF.

    Thanks for the explanation Vonasi.

    I have always done it the way you describe, but I thought I was missing a trick!

    Only difference being that I’ve always used WF as a final confidence check not as my main optimiser tool.

    Reason I guess is as I stated below.

    GraHal wrote:

    I feel like I have blinkers on when looking at the results from WF??

    But always open to alternatives, I’ll try optimise JUST with WF and then optimise JUST as normal.

    Anybody done above and got any comments or findings?

     

    02/02/2019 at 2:18 PM #90302
    GraHal

    What do folks prefer  or think is more logical … Anchored or Not Anchored?

  • Author
    Posts
Viewing 15 posts - 1 through 15 (of 16 total)
1 2

Find exclusive trading pro-tools on ProRealCode Marketplace

Similar topics:
a question regarding the sar parabolic indicator
another hedging question
magical trend strategy question
multi timeframe question
question about acumulated positions
rookie question — back testing issues
supertrend movement coding question
wierd optimizing bug

Create your free account now and post your request to benefit from the help of the community
Register or Login