Backtest NOT ‘Tick by Tick Mode’ even if Box is Ticked!

Viewing 15 posts - 31 through 45 (of 46 total)

← 1 2 3 4 →

Author

Posts
05/27/2022 at 6:44 PM #194014 quote

GraHal
Participant

Master

If we have to write the values in the code … I’m giving up!! 🙂

This is one thing I love that came with v11 … PRT sets the values in the code using the ONCE function at the top of the code (after DefParam).

05/27/2022 at 7:34 PM #194015 quote

phoentzs
Participant

Master

This feature makes things a lot easier.

05/28/2022 at 5:37 AM #194019 quote

PeterSt
Participant

Master

All right (not so), it is no different than what I already knew, despite it just took me an hour to prove to myself (and now you) what exactly the situation is. Anyway it is not related to variables needing to be obtained in the code instead of leaving them as Optimization Parameters.

I said something along these lines :

When the optimization result is there, you can click a result line in the list and it will show the backtesting result in the Statistics form, including Tick by Tick mode.

I also said :

When the result is there, usually PRT simulates you clicking on the best result (the result line at the top) and that result will show in the Statistics form and it incorporates Tick by Tick mode.

Now we add a new one, which dangerously let all newbies go under :

When we don’t have any Optimizations to enumerate, hence we don’t have Optimization Parameters *or* they are all (!!) fixed, no Optimization result is shown, and there is no result line for PRT to simulate a click on. And thus it now shows the result in the Statistics form without incorporating Tick by Tick mode.

That simple it is.

First attachment shows the latter situation. Notice that I deliberately implied a lot of in-bar trades, so the Tick by Tick mode would be a requirement (it actually always is !!).
Thus, this is just a Strategy without Optimization Parameters involved. This can be read as : all is fixed.

Second attachment shows how I now use a Dummy Optimization Parameter, which each self-respecting Backtester will have in there (I bet). This now causes the

Third attachment. This should be the result in reality; the 41 situations where in-bar trades occurred, should now incorporated correctly. PRT could click the first line which causes Tick by Tick mode to apply (also see message at the bottom). But in this example it did not perform that click (mind my mentioned “usually” in this post and in a previous post).

The fourth attachment shows what happened after I clicked the result line.

image_2022-05-28_063141565.png image_2022-05-28_063213725.png image_2022-05-28_063621417.png image_2022-05-28_064055947.png

05/28/2022 at 5:57 AM #194024 quote

PeterSt
Participant

Master

GraHal, we already knew about situations that Tick by Tick mode it not performed by PRT. At the start of PRT is one situation. Using the wrong Backtest button is another. Getting in this situation – the default for everyone new to PRT – is yet another.

When we last time talked about this, I announced these kind of situations at PRT support. What I persistently heard was : yeah, you should always have that checkbox ticked. The other situations are neglected because over there in Paris nobody seems to have a clue about USERS.

This is just bug 388 from my hand, and nothing is solved. Unless, of course, Nicolas arranges for it. So on behalf of his own customers ;-), he hopefully does.

PS: We can see that again I will survive. Don’t ask about the struggle, but I will survive. But will others ? People lost money on this (I’m not making this up). There is no single way that no-coders (who still try) can comprehend this idiocy.
And yes, I start to be in that mood again. Why was that ? ah, because more than 2 months ago I gave this bug which is general to local support.

🙂

05/28/2022 at 8:44 AM #194029 quote

GraHal
Participant

Master

PeterSt wrote:
I now use a Dummy Optimization Parameter, which each self-respecting Backtester will have in there (I bet)
I’ve only included a Dummy before when I use Walk Forward, but yes I agree … I woke this morning thinking … the work around (to the Issue discussed in this Topic) is to ALWAYS include a Dummy variable. But why-oh-why doesn’t the PRT guidance Note that we see when we hover over the blue button alongside the Tick by Tick box include …
NOTE : ALWAYS include a ‘Dummy Variable’ to ensure Tick-by-Tick backtest works correctly when using the optimiser??
Above is so simple a solution as an Alert to users and would take a PRT Techie < 1/10 time (to implement) that it has taken us here discussing this anomaly!!?? 🙁

05/28/2022 at 8:59 AM #194031 quote
PeterSt
Participant

Master
🙂 But I just came back here to contemplate this : When I am fully correct in what I wrote in the previous two posts, in a situation that Tick by Tick interpretation would be necessary, I would never be able to have fixed variables in the code for the solution; my backtest of that would always fail. But, I would not be able to tell. Thus, no solution available ? ** (next post) I always run one last backtest when I moved the Optimization Parameters to the code, in order to check whether I made typo’s in the declarations. They always ARE equal. But what does it really tell ?
1. That my backtest of Live is close to the result in “test” with all the Optimization Variables still in;
2. That “close” in my case is that because I cause the tests of Strategies to have good results and not dependent on faults I make;
3. That I am used to “not equal” because I saw that a million times by now. But “not equal” is close and close is say 1000 on 50000.
So I’m afraid nobody’s backtest can ever be correct. Disclaimer (kind of) : It still is so that 3 months or so ago I started to notice differences regarding this. I did not read back into it, but I now think that this topic woke me up on that one : https://www.prorealcode.com/topic/backtest-profit-not-consistent/
05/28/2022 at 9:05 AM #194034 quote

PeterSt
Participant

Master

**): Last week I was at a private seminar from IG. This dealt with Autotrading and PRT. En masse people shouted that no backtest can be equal to reality. “For me it is fine !” I shouted back. But I really was and remained alone. So it really really depends on how one observes things and how he works on solutions until he drops dead. I do that. But I also don’t take “no” for an answer.

05/28/2022 at 1:50 PM #194071 quote

PeterSt
Participant

Master

From the other topic :

phoentzs wrote:
I think many errors tend to happen in small TF, maybe in the seconds range, and are therefore hardly noticeable in larger TF like M5 or M15? I think there aren’t many people using the very small TF, maybe because there isn’t enough history?
This is good thinking. But in practice maybe not so. Below you see the same as I showed earlier on, but now with one of the Parameters not set so that it implies Tick by Tick error. But watch the number of trades and take it from me that this is for the period of one month. So it is about the number of trades, not the very short TF. Thus TF is 1 minute here (I know, also short for many). The shorter the TF, the less the errors can happen (less chance for in-bar happenings). Ideas of others for mine !
image_2022-05-28_144818491.png image_2022-05-28_145210080.png

05/29/2022 at 10:24 AM #194102 quote

GraHal
Participant

Master

Both screenshots above are the same? Both show 771 trades and other stats are the same.

05/29/2022 at 10:48 AM #194104 quote

PeterSt
Participant

Master

They are indeed. But the first overwrote the Z score (if important at all) and since we can’t remove attachments, I added the “improved” version. Sorry for the confusion.

05/29/2022 at 11:50 AM #194108 quote

GraHal
Participant

Master

PeterSt wrote:
one of the Parameters not set so that it implies Tick by Tick error
By not set, do you mean not fixed? It is when parameters / variable values ARE fixed (in the optimiser) that we may get Tick by Tick error.

05/29/2022 at 12:07 PM #194112 quote

PeterSt
Participant

Master

GraHal wrote:
By not set, do you mean not fixed?
No, that was clumsy semantics. Read it as “not-set so that” it implies. This refers to the earlier post where I said that I set-a-parameter-so-that it implied many tick by tick errors. I suppose “such” instead of “so” would be better English ? Not that such that it implied …

05/29/2022 at 12:08 PM #194113 quote

PeterSt
Participant

Master

GraHal wrote:
It is when parameters / variable values ARE fixed (in the optimiser) that we may get Tick by Tick error.
It is not related to that at all. Re-reading my posts would be killing, though. 😉
GraHal thanked this post

05/30/2022 at 2:47 PM #194197 quote

Nicolas
Keymaster

Master

PeterSt wrote:
When we don’t have any Optimizations to enumerate, hence we don’t have Optimization Parameters *or* they are all (!!) fixed, no Optimization result is shown, and there is no result line for PRT to simulate a click on. And thus it now shows the result in the Statistics form without incorporating Tick by Tick mode.
Your assumption is right, this is how it works by now. A modification of this behavior, that can lead to misunderstanding of the backtest results, is about to be changed, so that we can be sure that the backtest is made with tick by tick in any case (with a single occurrence in the optimization window, because of use of fixed variables).
GraHal thanked this post

06/20/2022 at 11:48 AM #195616 quote

PeterSt
Participant

Master

No time to read my posts ? then don’t. Go on to something useful. 🙂 But ProRealTime should read it. There is definitely more to this than we described until now; First there is a notable difference between a result springing from an optimization “range” (columns 2 and 3) versus obtaining a result in the fixed column (the first column). I saw this happening yesterday and could repeat it over and over again. I started investigating because out of the blue the result I had been working on the whole weekend, suddenly dropped with 5K or so (on the 55K). Btw, this was never found back and I don’t even know what was correct, the 55K or the 50K. I have screenshots of everything, including the optimization result window with the parameters all shown (hence, all was the same but the result suddenly changed). Then I UNticked the Tick by Tick mode (Editor-)checkbox and saw no difference whatsoever. This came to me as wrong because a difference should show. I obtained the parameter I was testing with (result from range vs fixed in optimizer-parameters form) directly in the code and saw no difference. I restarted PRT and saw no difference. I commented-out the parameter in the code and … after that the result in the optimizer for range vs fixed had disappeared.
I am almost certain that the results seen are related to the ticks being respected properly or not. But I can not prove that.
By now I have the clear hunch that optimizer results / backtest results are being cached (possibly by proxy) and that this is too smart to do; it can go wrong and we won’t notice until something snaps. Probably after that is is correct again, but we think it is wrong now (my 5K being disappeared goes as “wrong” into my brain).
What I said earlier on (possibly not in this topic) is that it is very clear to me when something with the ticks go bananas because a same backtest repeatedly takes twice as long. Again, no proof, but after many months dealing with it, it has become intuition. What this leaves is a feeling of totally unreliable backtests. What it surely leaves is the complete waste of time of about a whole weekend; I make screenshots of the progress – or perceived steps back I make, left for later to work out. If suddenly 5K disappears I can only see that it disappeared – not how it happened. It just happened. Because I can not revert to the situation with the 5K more again, I am 100% sure that all the time I was deceived. It is always related to the “storage” of the Optimization Parameters. Always. This can be proven by the fact that a. copying the Strategy does not help (it copies along the Parameters) and b. the virtual copy by means of creating a new Strategy and next add the Parameters manually again (type-over from the old), solves whatever the issue is.
Is someone at ProRealTime working on this ?
Author

Posts