like Dow Jones said, our tests must me run on a same number of trades
I agree cannot be too low, but I kind of misleading you, I guess the usage depends on each people. I was accepting it when it produces 25% of trades from the total trades of the strategy (I forget to mention, AND also to have at least 100 trades), generally I set this rule for myself because I trade low time frame and high frequency (2-4 trades/days). But this tool will be hardly to have no. of trades too close to the total trades, because the mechanism it uses is restricting certain bar that is allowed to trade, so it provides opportunity to test those signals that were blocked due to on-going trade.
I take it if strategy pass the VST with flying color, then it means the strategy can be robust to all kind of market structure, when market change, it can handle it well, so when it open trades, each trade is consistent in terms of average gain and win rate which can matter less even if the trades are little, that’s why the tool is not assessing the total gain. Of course if it doesn’t pass with flying color, it doesn’t mean it is a bad strategy, like you said, the strategy has different mean, like certain period not performing but overall still acceptable. So the VST tool maybe just not suitable or the Qty should be adjusted…or to rely on other robust test methodology.
I hope it make sense.
Thanks Dow Jones, i finaly understand ! All my bad
Gonna work on what you sent and VST to try to get a protocol and ratios we can use with low time frame strategies.
See you!
Moreover, i would like to make an assumption: we are in a lucky historical period.
We have the possibility to backtest a system that has gone through “standard” and “chaotic” phases such as covid in our previous 200k bars (> 5m tf).
Being able to make robust a system that has successfully gone through these phases (and therefore a good variability of phases for what can be a general vision) can ensure us some good stability for the future?
can ensure us some good stability for the future?
If only before the 2008 crash we had had a similar crash before to test on! Any back/forward /robustness test results are pretty much made pointless when something extreme that has never happened before happens. All we can do is see how our strategy would have coped and hope that the next extreme event isn’t really, really extreme. A big stop loss and quit is the only thing that can save us in extreme events.
PaulParticipant
Master
The point of my ‘Vonasi Robustness Tester’ and also my ‘Strategy Every Bar’ tests is to see trades that we might not see in a normal back test.
There is also another way. Taking cumulative positions with defparam cumulateorders = true and make sure each position handles it’s own stoploss & profit target (as it was only 1 position on market).
Partial closing works in backtesting and it could be usefull this way.
It would be interesting to see the effects on optimising, which in part avoids curvefitting because it tests every signal.
The problem is to to have a compact code to handle an unlimited cumulative positions, each position independently.
And second, how to adapt a trailing stop to suit this test, besides sl&pt?
After optimising parameters on every signal, then with a switch turn it off to see the results of a normal backtest which has less trades because it skips signals when in position.
It may result into a more reliable strategy.
Good point regarding CUMULATEORDERS = TRUE Paul though as you point out it does not work for all strategy exit styles. Still it is another weapon in our armoury if the strategy suits.
PaulParticipant
Master
If you want to control different positions with different levels then you will need different stop loss and take profit variables for each position. I haven’t ever bothered to code something like that because we can’t use it in real life because IG consider two positions of 1 to be one position of 2 and won’t let us close half of it using pending orders
you wrote this in another topic and this gave me hope it might be possible! Only for sl & pt with stop/limit orders or is that a stretch to far in the limits of prt because of the unknown number of cumulative positions?
It would be a great base to test/optimise all signals with a simple r/r ratio, settle on some values and extend the code with ts etc further.
After optimising parameters on every signal, then with a switch turn it off to see the results of a normal backtest which has less trades because it skips signals when in position. It may result into a more reliable strategy
Thanks, Paul. I had thought about that before, but too complicated for me to come up with any workable solution. So exactly why I started this article, maybe someone explore it and can share.
Personally I have doubt whether it is possible because we don’t have command to know the tradeprice, set SL and set TP for individual trade, but just the average price (positionprice) and the SET STOP LOSS is for the last trade and close all trades when the SL is hit (same for TP), please correct me if I’m wrong. Today the most genius workaround is the VRT tool only.
Like @vonasi said, easier if PRT introduce such mechanism in probacktest. I imagine with a checkbox to select possibility to run “multiple instances” of back test that will consolidate all trades, i.e. when signal trigger, open it as individual trade until it close, if another signal came up before first trade close, open it as another individual trade instance. At the end, consolidate all the trades.
I think this alternative also make back test more reliable without have to rely on huge back test history.
PaulParticipant
Master
to know the tradeprice,
Another angle, what if you don’t need the tradeprice, instead remember the next barnumber and it’s openingprice after a signal? It doesn’t take into account spread for the entry/exit, but it could be added manually later.
Are arrays needed for this & v11? Maybe it can be coded up to a fixed small number of cumulative positions, but then it will be much better if there’s no limit.
Whenever a SET STOP pPROFIT or SET TARGET pPROFIT instruction is encountered in a strategy code it adjusts the stop and target distance for all open positions. So positions will close out individually at different levels. The downside is that we cannot control each position individually – if you send a new stop or target distance then it is applied to all open positions – they cannot have different distances.
Hi Guys,
For me the best way to avoid overfitting (and so have the best correlation Backtest/OOS) is
1/ Not more than 3-4 variables (I see on the forum some strategies with 10-12 variables…The more variables we have in the strategy, the more linear will be the equity curve, but the worst OOS we will have…Kind of polynomial regression)
2/ Keep it simple ! The more simple is the strategy, the best OOS results we will have
3/ A long long length of the backtest. The problem for me is that if we have 100 000 units it will take hours to have a backtest. I test on 1 million unit with 3 variable but I stop because of the time it takes. Ideally we need more than 100 trades in the backtest. The more you have, the best it is
Exemple of results on OOS we should have if you have no (or minor) overfitting (Demo Strategy “1000 dollars / day” on forex EUR/USD since mid august after a 200 000 units backtest )
Zilliq
[attachment file=”143211″]
@zilliq how come some Long entries are a red down arrow and some Short entries are also a red down arrow??
I guess you are teasing us and you are not going to share your excellent looking Strategy?? 🙂
Hi @Grahal
No it’s normal green arrow for long and red for short
Simple strategy based on Bollinger Boll and ATR (Keep It Simple As Always).
There are some months/Years I had codes with hundreds lines of code and I did not a lof of backtest IS/OOS. Now codes of my “incubator” have no more than 50 lines may be, but I do more and more backtests and robust tests
Have a nice evening
[attachment file=”143225″]
No it’s normal green arrow for long and red for short
Am I losing it 🙂 … see attached ?
I see @Grahal it’s a bug on graphs of PRT
On small view there is a bug, but on a big view it’s correct
[attachment file=”143232″]
[attachment file=”143233″]
