results of backtests changed sometimes from time to time

Forums ProRealTime English forum ProOrder support results of backtests changed sometimes from time to time

Currently, there are 0 users and 1 guest visiting this topic.
View all attachments
Viewing 11 posts - 1 through 11 (of 11 total)
  • #55107 Report

    From long time I felt like the results of backtests changed sometimes from time to time with the same parameters, I tried to assume it was some error in my doings. But now I have started to write apart the results of backtest to compare systems and versions of the same system (with the same dates, no just from the begining to the end of data avaible, because that could be the cause) and I found it´s not my imagination. It doesn´t happen often, I feel like it´s just some days that are prone to. Even in the same day with the same system not changing nothing at all I get different results. Maybe it´s related with the loaded data because I use insample/outsample periods.  Does this happen anyone else or am I getting crazy?

    #55116 Report

    I changed the topic title for something more comprehensible and useful for future research ..

    Of course, the backtests results will change if the start date differ. If you are using optimisation with IS/OOS periods, and if the IS period is a bit different, the best variables found will not be the same and so the result too.

    #55166 Report

    Thanks Nicolas,

    “Of course, the backtests results will change if the start date differ”. Yeah, and of course if anything else changes also the results will change 🙂 I know it´s not because the start date because I put a fixed start date, no the first avaible, a later one, because I don´t want the results to change if I do the backtest the next days.

    I get this quite different results without walkforward or optimization variables, I am still in the insample backtest. And I emphasize it happens rarely, but clearly, I mean it can happen in seconds of difference between backtests without chance to have changed nothing. So it´s really disturbing. 

    Maybe there is a reason for it I can´t see. That´s why I am asking if somebody had the same experience

    #55170 Report

    Ah, before you tell me, it isn´t about the “end date” of the backtest neither, it´s also fixed for the insample test

    #55175 Report

    Today seems to be one of these days…

    I just record one example of what I talked above.

    In the first picture I run an optimization with the varible and the results that appear in the picture. Everything ok

    In the second I fix the varible to the best result (0.2)  but the results differ from the optimization??? and yes, nothing else changes 🙂

    It´s not a onetime thing, I repeated several times and I still getting the same different results. Am I missing something here?

    Attachments:
    #55177 Report

    Here the second picture, I don´t know why didn´t appear in previous message, the rebellion of the machines is coming!!! :):):)

    Attachments:
    #55182 Report

    Thinking about why this is happening.. maybe is because the tickbytick mode. I have checked the tickbytick all the time but maybe in the optimization this  doesn´t work and that´s why the column of “Ticks mode” appear in the optimization window. And that´s the reason of the difference. Could it be?

    #55304 Report

     Is the spread set to the same value for both versions ? You have 1084 positions in both versions, and the difference in gain between the 2 versions is 1106 Euro. So, I guess in one version the spread is set to 1 point, and in the other version to 0. I don’t know how often I made this mistake myself…

    When you start a new strategy from scratch (not by copy and paste), you always have to check the box “spread” and set it to the correct value. Otherwise, it will be 0. 

     

    The rest of the difference probably comes from the slightly different times at which the 2 backtests ended. 

    #55312 Report

    Thinking about why this is happening.. maybe is because the tickbytick mode. I have checked the tickbytick all the time but maybe in the optimization this  doesn´t work and that´s why the column of “Ticks mode” appear in the optimization window. And that´s the reason of the difference. Could it be?

    Yes, this is the answer you are looking for. Tick by tick mode is not active in optimisation, even if the checkbox is checked, and you’re right, that’s why the optimisation result window has a column named “ticks mode” to tell you how much orders could be a “problem” in tick mode (understand how much trades were opened and closed in the same bar).

    #55414 Report

    My finding is if start Prorealtime and run a backtest on 5 minutes it doesn’t take all data (skips 1st month). Switch i.e. to 10 minutes and switch back to 5 and you have that extra month which could change the backtest considerably.

    #55595 Report

    Thanks everybody, by order:

    1. The spread is a good guess but in this case its not, designing the system I never use spread at all
    2. Ok, now I know the backtest dont use tickbytick the results of the pictures makes sense, but still found differences other times that cant be explained by this
    3. Yes,  some times I felt like it isnt loading the same data and thats probably the cause of my other different results. I will try to change TF as you do when this happen. Thanks good to know I am not crazy… or not as much:)
Viewing 11 posts - 1 through 11 (of 11 total)

Create your free account now and post your request to benefit from the help of the community
Register or Login