results of backtests changed sometimes from time to time

Forums › ProRealTime English forum › ProOrder support › results of backtests changed sometimes from time to time

Viewing 11 posts - 1 through 11 (of 11 total)
  • 12/08/2017 at 1:22 PM #55107
    TempusFugit

    From long time I felt like the results of backtests changed sometimes from time to time with the same parameters, I tried to assume it was some error in my doings. But now I have started to write apart the results of backtest to compare systems and versions of the same system (with the same dates, no just from the begining to the end of data avaible, because that could be the cause) and I found it´s not my imagination. It doesn´t happen often, I feel like it´s just some days that are prone to. Even in the same day with the same system not changing nothing at all I get different results. Maybe it´s related with the loaded data because I use insample/outsample periods.  Does this happen anyone else or am I getting crazy?

    12/08/2017 at 1:46 PM #55116
    Nicolas

    I changed the topic title for something more comprehensible and useful for future research ..

    Of course, the backtests results will change if the start date differ. If you are using optimisation with IS/OOS periods, and if the IS period is a bit different, the best variables found will not be the same and so the result too.

    12/08/2017 at 5:37 PM #55166
    TempusFugit

    Thanks Nicolas,

    “Of course, the backtests results will change if the start date differ”. Yeah, and of course if anything else changes also the results will change 🙂 I know it´s not because the start date because I put a fixed start date, no the first avaible, a later one, because I don´t want the results to change if I do the backtest the next days.

    I get this quite different results without walkforward or optimization variables, I am still in the insample backtest. And I emphasize it happens rarely, but clearly, I mean it can happen in seconds of difference between backtests without chance to have changed nothing. So it´s really disturbing. 

    Maybe there is a reason for it I can´t see. That´s why I am asking if somebody had the same experience

    12/08/2017 at 5:41 PM #55170
    TempusFugit

    Ah, before you tell me, it isn´t about the “end date” of the backtest neither, it´s also fixed for the insample test

    12/08/2017 at 6:13 PM #55175
    TempusFugit

    Today seems to be one of these days…

    I just record one example of what I talked above.

    In the first picture I run an optimization with the varible and the results that appear in the picture. Everything ok

    In the second I fix the varible to the best result (0.2)  but the results differ from the optimization??? and yes, nothing else changes 🙂

    It´s not a onetime thing, I repeated several times and I still getting the same different results. Am I missing something here?

    12/08/2017 at 6:16 PM #55177
    TempusFugit

    Here the second picture, I don´t know why didn´t appear in previous message, the rebellion of the machines is coming!!! :):):)

    12/08/2017 at 6:33 PM #55182
    TempusFugit

    Thinking about why this is happening.. maybe is because the tickbytick mode. I have checked the tickbytick all the time but maybe in the optimization this  doesn´t work and that´s why the column of “Ticks mode” appear in the optimization window. And that´s the reason of the difference. Could it be?

    12/11/2017 at 12:51 AM #55304
    verdi55

     Is the spread set to the same value for both versions ? You have 1084 positions in both versions, and the difference in gain between the 2 versions is 1106 Euro. So, I guess in one version the spread is set to 1 point, and in the other version to 0. I don’t know how often I made this mistake myself…

    When you start a new strategy from scratch (not by copy and paste), you always have to check the box “spread” and set it to the correct value. Otherwise, it will be 0. 

     

    The rest of the difference probably comes from the slightly different times at which the 2 backtests ended. 

    12/11/2017 at 8:24 AM #55312
    Nicolas
    TempusFugit wrote:

    Thinking about why this is happening.. maybe is because the tickbytick mode. I have checked the tickbytick all the time but maybe in the optimization this  doesn´t work and that´s why the column of “Ticks mode” appear in the optimization window. And that´s the reason of the difference. Could it be?

    Yes, this is the answer you are looking for. Tick by tick mode is not active in optimisation, even if the checkbox is checked, and you’re right, that’s why the optimisation result window has a column named “ticks mode” to tell you how much orders could be a “problem” in tick mode (understand how much trades were opened and closed in the same bar).

    12/11/2017 at 5:50 PM #55414
    Paul

    My finding is if start Prorealtime and run a backtest on 5 minutes it doesn’t take all data (skips 1st month). Switch i.e. to 10 minutes and switch back to 5 and you have that extra month which could change the backtest considerably.

    12/13/2017 at 4:38 PM #55595
    TempusFugit

    Thanks everybody, by order:

    1. The spread is a good guess but in this case its not, designing the system I never use spread at all
    2. Ok, now I know the backtest dont use tickbytick the results of the pictures makes sense, but still found differences other times that cant be explained by this
    3. Yes,  some times I felt like it isnt loading the same data and thats probably the cause of my other different results. I will try to change TF as you do when this happen. Thanks good to know I am not crazy… or not as much:)
  • Author
    Posts
Viewing 11 posts - 1 through 11 (of 11 total)

Find exclusive trading pro-tools on ProRealCode Marketplace

Similar topics:
indicator matrix series from pinescript to pro real time
“multi time frame macd divergences” screener
any way to plot a time based moving average over a tick chart ?
backtesting does not work which a great amount of time units
backtests not saving results when platform closed and reopened
calculating time range
changing stop loss after a certain time in trade
close on a specific day at a specific time
code conversion easy language to pro real time
coding % of time the price reacts to pivot levels in forex

Create your free account now and post your request to benefit from the help of the community
Register or Login