results of backtests changed sometimes from time to time
12/08/2017 at 1:22 PM #55107
From long time I felt like the results of backtests changed sometimes from time to time with the same parameters, I tried to assume it was some error in my doings. But now I have started to write apart the results of backtest to compare systems and versions of the same system (with the same dates, no just from the begining to the end of data avaible, because that could be the cause) and I found it´s not my imagination. It doesn´t happen often, I feel like it´s just some days that are prone to. Even in the same day with the same system not changing nothing at all I get different results. Maybe it´s related with the loaded data because I use insample/outsample periods. Does this happen anyone else or am I getting crazy?12/08/2017 at 1:46 PM #55116
I changed the topic title for something more comprehensible and useful for future research ..
Of course, the backtests results will change if the start date differ. If you are using optimisation with IS/OOS periods, and if the IS period is a bit different, the best variables found will not be the same and so the result too.12/08/2017 at 5:37 PM #55166
“Of course, the backtests results will change if the start date differ”. Yeah, and of course if anything else changes also the results will change 🙂 I know it´s not because the start date because I put a fixed start date, no the first avaible, a later one, because I don´t want the results to change if I do the backtest the next days.
I get this quite different results without walkforward or optimization variables, I am still in the insample backtest. And I emphasize it happens rarely, but clearly, I mean it can happen in seconds of difference between backtests without chance to have changed nothing. So it´s really disturbing.
Maybe there is a reason for it I can´t see. That´s why I am asking if somebody had the same experience12/08/2017 at 5:41 PM #5517012/08/2017 at 6:13 PM #55175
Today seems to be one of these days…
I just record one example of what I talked above.
In the first picture I run an optimization with the varible and the results that appear in the picture. Everything ok
In the second I fix the varible to the best result (0.2) but the results differ from the optimization??? and yes, nothing else changes 🙂
It´s not a onetime thing, I repeated several times and I still getting the same different results. Am I missing something here?
You must be logged in to access attached files.12/08/2017 at 6:16 PM #5517712/08/2017 at 6:33 PM #55182
Thinking about why this is happening.. maybe is because the tickbytick mode. I have checked the tickbytick all the time but maybe in the optimization this doesn´t work and that´s why the column of “Ticks mode” appear in the optimization window. And that´s the reason of the difference. Could it be?12/11/2017 at 12:51 AM #55304
Is the spread set to the same value for both versions ? You have 1084 positions in both versions, and the difference in gain between the 2 versions is 1106 Euro. So, I guess in one version the spread is set to 1 point, and in the other version to 0. I don’t know how often I made this mistake myself…
When you start a new strategy from scratch (not by copy and paste), you always have to check the box “spread” and set it to the correct value. Otherwise, it will be 0.
The rest of the difference probably comes from the slightly different times at which the 2 backtests ended.12/11/2017 at 8:24 AM #55312
Thinking about why this is happening.. maybe is because the tickbytick mode. I have checked the tickbytick all the time but maybe in the optimization this doesn´t work and that´s why the column of “Ticks mode” appear in the optimization window. And that´s the reason of the difference. Could it be?
Yes, this is the answer you are looking for. Tick by tick mode is not active in optimisation, even if the checkbox is checked, and you’re right, that’s why the optimisation result window has a column named “ticks mode” to tell you how much orders could be a “problem” in tick mode (understand how much trades were opened and closed in the same bar).12/11/2017 at 5:50 PM #55414
My finding is if start Prorealtime and run a backtest on 5 minutes it doesn’t take all data (skips 1st month). Switch i.e. to 10 minutes and switch back to 5 and you have that extra month which could change the backtest considerably.12/13/2017 at 4:38 PM #55595
Thanks everybody, by order:
- The spread is a good guess but in this case its not, designing the system I never use spread at all
- Ok, now I know the backtest dont use tickbytick the results of the pictures makes sense, but still found differences other times that cant be explained by this
- Yes, some times I felt like it isnt loading the same data and thats probably the cause of my other different results. I will try to change TF as you do when this happen. Thanks good to know I am not crazy… or not as much:)