Have you ever seen that the Worst Parameters are the first results on Backtest ?

Viewing 13 posts - 16 through 28 (of 28 total)

← 1 2

Author

Posts
11/26/2020 at 11:50 AM #151732 quote

zilliq
Participant

Master

A picture to explain..

First step Backtest on 6 weeks

In this case, with this combination var1, var2, var3 the Algo would have win 43462 euros (whatever the amount)

And above, a backtest on 8 weeks

With the same combination var1, var2, var3 we see the Algos would have win maximum +19.17 % during this 2 weeks (it simulates OOS results). Not bad at all on 2 weeks

If I remember @GraHal had explained this many times on the forum

As I said the purpose is to determine which combinations work and which don’t work by simulating many many OOS like that

2020-11-26_11h44_34.jpg

11/26/2020 at 7:38 PM #151807 quote

Vonasi
Moderator

Master

So that is a totally standard in sample and out of sample walk forward test then. I’m not sure why you didn’t just say that in the first place and save us all the confusion! 🙂

11/26/2020 at 8:52 PM #151816 quote

zilliq
Participant

Master

Because it’s not à Walk forward test 😁

It was to simplify the explanations

It’s more than 80-90 one pass Wf test in one time

Like that it’s like i did 80-90 OOS test in one test, and do 80-90 analysis

11/26/2020 at 10:40 PM #151829 quote

Vonasi
Moderator

Master

What you showed was a test on some data (that’s in sample) then a test on some data plus some extra data (that’s in sample with some out of sample).

If there is more to it than that then you need to provide coding examples of how it is more than that and/or optimisation examples because at the moment I am just seeing some in sample and out of sample forward testing from what you have so far said and shown.

11/27/2020 at 8:46 AM #151852 quote

zilliq
Participant

Master

For now it’s just an explanation on how I do to have more and more OOS test and test robustess

The purpose as I said is now to determine which factors/indicators/stats are predictive of a correlation between IS/OOS

I have some good ideas but need to do some others IS/OOS tests to confirm

Have a good day

11/27/2020 at 9:52 AM #151877 quote

GraHal
Participant

Master

zilliq wrote:

more than 80-90

More than 80-90 what?
I want to understand what you have to say, I might want to do the same.

I do try honest … I read your posts 3 or 4 times and mostly I still don’t know what you are saying.

Have you ever tried writing in French and then use the Select Language box at the top of the page to translate to English and then post your comment?

It would be worth a try as surely it will be easier for you and more effective to share your good work with us all?

11/27/2020 at 6:46 PM #151954 quote

zilliq
Participant

Master

I think the most important problem is that it’s not very easy to explain

Just remember it’s just a method to test the robustess and to find the best factors/indicator to obtain a correlation between IS/OOS results

I will re-try next week to explain if I have some time

Happy week-end

GraHal thanked this post

12/02/2020 at 3:28 PM #152414 quote

zilliq
Participant

Master

Well, after thousands of simulation on OOS, on differents codes, I have some first results…

I study 15-20 Pearson correlations between IS and OOS results (Gain, drawdown, runup and so on..)

First : It’s absolutely not easy ! 🙂 and seems not to be reproductible in all algos

In general it confirms what I said, there is a quite good correlation between the rankings in IS and OOS results, may be at 70- 80 %

But it confirms what I sais in this post the best variatiions are not the first in the ranking in the IS Backtest= the classical backtest !

The better evolution +15, +20 or +30 % are between rank 80-100 in IS Backtest results

Yes you can have a very bad curve and a wonderful OOS evolution !

The 1-20 rank in IS Backtest results give less than 10 % and often -5 %, -10 %

It confirms what we already know, the more curvefit we do (A lot of variable, a very long code and so on..), the worst results we will have in OOS !

If it’s possible to “predict” the first results in the ranking on OOS, at this time I find no correlation to predict the variation (Some ideas but need to be confirmed) and differentiate a good and bad evolution.

In other way, Max drawdown, % winning trades, runup and so on..can not predict a good evolution in OOS results = You can have a high % winning trade, small Max drawdown, high runup and so on and have an evolution of -10 % …

To be continued

[attachment file=”152415″]

2020-12-02_15h10_45.jpg

12/02/2020 at 3:37 PM #152416 quote

zilliq
Participant

Master

An example (OK it’s only on 33 transactions on IS backtest, but it’s the first I found)

Good % winning trade = 78,79 %, good average win 574 dollars, very small max drawndown, the first in the ranking in IS Backtest Results

BUT

Number 47 in the OOS results/rank (after 2 weeks), and a loss 18965 dollars ->17172 dollars = – 10 %

[attachment file=”152417″]

2020-12-02_15h32_58.jpg

12/02/2020 at 4:45 PM #152422 quote

GraHal
Participant

Master

It would be useful to show the backtest equity curve for the example above.

12/02/2020 at 4:58 PM #152423 quote

zilliq
Participant

Master

I need to do the same backtest because I delete this backtest

12/02/2020 at 5:25 PM #152424 quote

GraHal
Participant

Master

Don’t duplicate work as it takes too long, but next time the equity curve would be useful.

I use backtest equity curves, positions and price curve (all shown under each other) to make useful judgements about Algo likely OOS performance.

12/02/2020 at 6:17 PM #152430 quote

Dow Jones
Participant

Veteran

zilliq wrote:

It confirms what we already know, the more curvefit we do (A lot of variable, a very long code and so on..), the worst results we will have in OOS !

I agree partially, just sharing that I have system with plenty variables and long code and so far performing well in live for 100% of the IS period with 65% win rate, and each month so far is gaining without changing the parameters.

I think bottom line is depending how you use and optimize the variables, I have much more variables because of MTF, market structure, trailing, breakeven, price action parameters, distance to the nearest SL, overbought/oversold, volatility, etc…And I don’t always pick the best result but more on stable result and each variable undergone the 5 iterations 60/40 (or 70/30) WFA and I optimize long and short separately (with WFA too). I also conduct the Vonasi’s robustness tester and monte carlo analysis, if result not so good, I repeat whole thing again. All these consume time and may tend to let people impatiently assign the most optimized value. Btw, one can have few variables, but if 1 variable is used on multiple items/indicators, for me, it is like using multiple variables too.

One thing I didn’t optimize though is the SL, I assign it with 0.3% of the close price and fixed 1 contract only. Due to low SL, so generally it open trades on low to medium volatility only, i.e. it will open trade and wait for increasing volatility to take profit or hit SL if wrong direction. It generally has 2-3 trades per day, so fast for recovery too, it means I won’t be worrying if losing 1-2 trades, and I have to wait long to recover. And more affordable for me to diversify with other assets too with lower SL.

In any case, I appreciate your sharing, just adding my experience so it can act as additional source of thought for others.

GraHal and zilliq thanked this post
Author

Posts