V July 25 what a mess : Backtest Results are all over the place

PeterSt · 2022-09-24T12:44:31+00:00

I call this V July 25, although we’re on August 31 by now, but it is about the new means of “Multi” presentation of Optimization Results. N.b.: I waited with reporting this, watching for at least one other to report the same – but No; nobody. I have several examples, but this is a most clear one because easy to follow. All examples are similar : the selected line in the Optimization Results does not show the result of *that* but of something else. Usually it can not be seen where the result comes from, which makes the result quite wrong and randomly wrong. All ‘n all this is unworkable … The first attachment shows the result of the selected line, which is the blue line. This comes from an explicit backtest. Side note : watch the number of trades, which is usually *the* means to find the real results in the appropriate statistics form (match with one of the result lines). Thus – and then wrongly – what shows in the first attachment are 1008 trades, which clearly does belong to the first line, and not the second which I selected. But promise in advance : would I click the first line then a statistics form with 996 trades will show. So let’s do that … After a bit of waiting, now the 2nd attachment shows. Big fun, because this belongs to the 2nd line – the one selected with the story above. —————- This is only half or even less than half of the story, because both result lines should be the very same to begin with. I can tell you : in this super simple case the first line shown is the one which was automatically performed when the chart was changed from the default 10K to 200K and the 2nd line is the result of an explicit backtest (via the Editor window) and to be clear : with “In tick by tock mode” active (which is not related anyway, but still saying because it looks related surely). I by now experienced 100+ of these situations during a normal backtest session, where at some stage the statistics start to show too optimistically or the other way around : that something which had been too pessimistic so far, now starts to show more real results. In other words, I have no clue what the truth is, except for the observation that at some stage things “knack” and wont restore to normal. When then PRT is restarted, it boots up with the knacked result, and so I feel like having wasted several hours before the restart. It seriously is a big mess. I think I can recognize that things start to change when I move a varying Parameter to fixed, which somehow fixates error or good, but which I can’t restore to any previous situation by means of activating the fixed value to the same varying again. The result just won’t be the same any more, whatever I do, including the restart of PRT. In the third attachment you see an example of two of these situations occurring. In this case the right-most varying parameter “causes” a vary different situation in a first attempt vs a third attempt, the first optimization line consisting of the very same parameter values for all the parameters compared to the third line. But the number of trades and the profit (“winst”) are not equal at all. And so I can’t tell whether even one of the lines is correct (as they could both be wrong). So look for the 11,9 (and all else) being equal in line 1 and line 3 for the first example, and the 2300 (and all else) being equal in line 1 and 3 for the second example. Notice that the “+” sign (4th column) in the 2nd attachment relates to optimization results over varying parameters, but that the expanded content never leads to “seeing through” what the system does for real or what it does wrongly. This is contrary to the 1st attachment where things are obviously swapped, while at the first result in there (1st attachment) the 2nd result is not even there, thus it is *not* a matter of “things are swapped”. It just goes wrong all together and everywhere, and I can’t see the sense in anything. A hint for others – or possible help – could be that the results in both 1st and 2nd attachments (1st and 2nd example) mighty much look like anomalies in the area of tick by tick mode – especially the 2nd example (people might recognize the erroneous situation in such smooth curves). However, no 0 bar trades (or more than one in-bar trade) occur anywhere and the tick by tick mode situations (max 2 and 3 respectively) are too low anyway for the influence of often dozens of trades of difference between two situations which should be the same. And otherwise I explicitly created the curves like this. The curve from the 2nd example just does not comply towards the end, so it could be there where things go wrong and that *that* is the wrong curve (and the other one the correct one). If nobody has ideas, I will use this topic to link to in a ticket from PRT Support (starting with a Technical Report). Thanks …

Viewing 3 posts - 16 through 18 (of 18 total)

← 1 2

Author

Posts
09/25/2022 at 1:56 PM #201363 quote

PeterSt
Participant

Master

GraHal wrote:

Are you making the pont that you have to do above in order to be able to achieve meaningful results?

I am making the point that backtest results s*ck. No more and no less. Sorry …

09/25/2022 at 2:41 PM #201365 quote

GraHal
Participant

Legend

GraHal wrote:

we can both do the same

If you want to do a dual-run / test on the same Algo at any time let me know.

PeterSt thanked this post

09/26/2022 at 1:13 PM #201431 quote

GraHal
Participant

Legend

Weird stuff happening to me now!!

To get the Detailed Report and equity curve to show same value as a row in the Table, I have to have (some) fixed values in the optimiser. ALSO have the full results of any particular ‘opti-run’ open in the Table, then click on a row in the Table as shown in Peter2 attached at red arrowhead

Prior to today, all I had to do was click on the top row (with all remaining results and opti-runs closed) as shown in Peter3 attached at red arrowhead.

Peter2.jpg Peter3.jpg
Author

Posts