Optimization results different from backtest

Forums ProRealTime English forum ProOrder support Optimization results different from backtest

Viewing 14 posts - 1 through 14 (of 14 total)
  • #224417

    Hi, since yesterday when I do optimization and replace the values of the optimization into the code and backtest the results are vastly different. Any explanation to this? I did a optimization on a simple MA crossover with SP500 mini. The optimization shows profit of $455 with the SMA 40 cross over SMA 55. When I replace the opt values in the code the results show loss of -$29. See code and results below.

     

    #224426

    It seems to be the same case on any instrument which I choose to optimize the moment you replace the values in the code with the optimized values the back test show totally different results. I certainly hope that this will not be the case on the live Pro Order platform. I decide to stop my live algos just for incase there is a underlying problem. Can someone please test this function and advice.

    #224429

    Hi Johann,

    You should post the .itf with the Optimization Variables active so we can try to copy the behavior. Comment out the same variables in the code, so we know what to fix in the code.
    N.b.: Your x and y shown in the result, are not obtained in the code; it may even help yourself to find the culprit (if on your side to begin with).

    Btw, such things happened before, so you can very well be right.

    Which version of the platform is this about (10.3, 11.1, 12) ?
    And which broker (IBKR / IG) ?

    1 user thanked author for this post.
    #224433

    Thanks PeterSt, I will attach the .itf then you can run the opt and use any of the results and replace in the x and y variables and see the different results. I use the v11.1 – 17.0.7

    #224436

    If this type of thing happens with optimization and back testing it will obviously be impossible to develop code accurately. Should I be concerned about existing algos not trading accurately live or will these errors only occur on the back test platform?

    #224437

    Thank you Johann.

    It is not about the variables obtained fixed in code, it is about the iteration of at least y which internally fails. For now too difficult to explain in easy terms, but it looks like an internal rounding issue (I’d need to prove that). But summarized : when the variable y is set to 40, the result is presented of 50.
    To me it feels like the 40 value is internally set to 40.00001 which then is rounded to 50.

    If you set x to 15 (just do that in the Optimization Parameters window) and let iterate y from 30 to 50 with a step of 1, you can better grasp what I am sensing.

    It requires some more digging to provide PRT with the real culprit. I will try to do that …

    #224439

    Exact same for me on v11 …

    1.   Lower curve is the optimisation result.
    2. 2nd curve up is the opti result for x and y entered as fixed in the opti table.
    3. 3rd curve up is the opti result for x and Y entered in the code and opti variables deleted from opti variable table.

    What am I doing different than what you are doing or vice versa?

    #224442

    If this type of thing happens with optimization and back testing it will obviously be impossible to develop code accurately. Should I be concerned about existing algos not trading accurately live or will these errors only occur on the back test platform?

    My opinion : yes you should. But there is one quite crucial matter in order here :

    The way this System comes to its result, is the most over-optimized thinkable (this is not meant to be a negative 🙂 ). This summarized : from one step in the iteration to the next, means do or die. Thus, because 40 (in combination with 15) is coincidentally very good and 50 is as coincidentally about the worst, you know will think that in real life (Live) you might lose all of your money. Well, this probably is so indeed, but not because PRT now contains this bug which I call call a “rounding” bug (not justified yet for the type of but, but a bug is surely there).

    Can you follow what I mean ?

    Example of a more normal situation :
    Suppose your SL is interpreted as 70.0001 instead of 70, that leading to 80 instead of the intended 70, then your SL won’t be right and although it will influence the results, it is not 100% devastating. The bug you have at hand *is seemingly* devastating, because it twists the result completely upside down. But all what happened for real is a different result from very very much over-optimizing. You can test this testis by changing the 10K number of (60 minute) bars I used and which came up with your 15/40 as best result, to 50K bars. Now the best result is at x = 10 and y = 20 with a result which is over 100 times less but which is overall 100 times less, thus also the negative result.
    To me this tells everything about the over-optimizing, which I could “know” in advance, because that is just what will happen 100% always with working with crossing averages like in your example.

    All ‘n all there is a bug, but it is not as devastating as you think it is. How to know that is something else. And wrong (bug) is just that.

    #224444

    Now the best result is at x = 10 and y = 20 with a result which is over 100 times less but which is overall 100 times less, thus also the negative result.

    That was not correct as I abusively selected the wrong instrument. But the story remains the same : it is now x = 10 and y = 70 and the result of 15-40 is heavily negative (the 100 times less etc. does not apply).
    You can check it yourself of course. Apologies for my mistake in selecting the wrong instrument (for my previous post, the other was fine).

     

    1 user thanked author for this post.
    #224445

    Thanks PeterSt, I will do the exercise on 50k bars as well perhaps take the c1 condition out and see what happens…

    #224448

    Hi PeterSt, ok so I run the test on 50k with and without the c1 condition. The first test I did without the c1 I kept the opt variables x and y (5 – 60 with 5 iterations) and (10 – 100 with 10 iterations). The result I chose was a combo of x = 20 and y = 40.  The results after I replaced was different but not as much. Then I tested again but this time I fixed x as 20 and only tested for y (10 – 100 with 10 iterations) and it gave me 40 again. This time around when I replaced y with 40 the results were the same as the optimized result. So it makes sense that there could be a back the moment you have to many variables with iterations. Which is a shame because it makes it a lot harder. I have never experienced this before…

    I did bring back the c1 condition and followed the same principle testing it once again with two variables simultaneously with iterations it was not accurate…one had to be fixed for it to be accurate…

    #224453

    So something just happened now I just did the same tests again and now all is fine everything is working just like before even if I use more variables and iterations….magic

    #224462

    Houston …

    I don’t know what I did or where it happened, but since the last 20 minutes I would say “Johann, what are you talking about  ?!?”. Meaning :

    Suddenly it works correctly and whatever I do, I can’t let it go wrong any more. The only thing I did not do yet, is reloading your .itf from disk. … And that I now did too. It does not help.

    So Johann, what are you talking about eh ? hahaha.
    Unbelievable. There is something wrong beyond imagination. I must give up on this now ?

    I recall that it started to work after I had obtained a Graph command, which was after I tried to make a trading system for ProOrder of it (see second attachment). Notice that both do things;

    The latter has issues to begin with, because look at the x and y you see in there – they are both at 1 and spring from the original loading of the System. They were at 1 just the same when I attempted this ProOrder stuff an hour ago, while long gone I had set them to different values (you can observe that in the first screenshot as well, though this is after reloading the not and not after one hour of messing about). So do notice that if I would hand this System to ProOrder right now, it would use x=1 and y=1 which is always terriblyt wrong of course (because I have filled x=15 and y=40 in the parameters. OK, see third screenshot where you have it combined. N.b.: This can be fixed by exiting the Editor and reloading the System hence Editor again.

    Side note : I never gave PRT this in itself gigantic bug because I kind of gave up on reporting bugs.

    And so my guts tell me that where I originally could copy your behaviour Johann, possibly something in the realm of ProOrder reset stuff, but quite definitely. But look how difficult it is and how may bugs a man has to conquer before he can do his thang :

    I removed the SP500 System from the cloud stuff (“Remove Backtest”) from the list of systems I have in there – Saved and Exited PRT and Restarted it.
    Back was the system. Just like that, including the presenting of the backtest. So it also requires to be removed from the Chart. OK, done, Remove it from the list of systems again, Save – Exit – Restart. Now all was finally clean.
    But now there was so much hassle in between (including the restart of PRT a couple of times, which I already did outside of removing the system) that all has became moot.

    The Graph command is special because I am fairly confident that PRT on their servers cache things with that – or around that or whatever. Graph has given me so so many times issues and all we can do is create backdoor solutions to overcome it. And YES that includes strange backtest results, just because things error out quietly internally. But this was the other way around … I added the Graph command.
    Anyway since adding that command or otherwise going to the “Automatic Trading” tab (2nd attachment) things suddenly were correct – I did both things at the same moment of backtesting.

    Crazy.

    #224466

    So something just happened now I just did the same tests again and now all is fine everything is working just like before even if I use more variables and iterations….magic

    There you go. And I surely did not read your post before posting my last …

Viewing 14 posts - 1 through 14 (of 14 total)

Create your free account now and post your request to benefit from the help of the community
Register or Login