Strategy Robustness Tester

Vonasi · Sun, 01 Sep 2019 12:34:14 +0000

Following on from a discussion elsewhere regarding walk forward testing I had the idea to create a bit of code that you can add to any strategy and see how it would have performed if you had traded on only odd days or only on even days, or odd or even months or odd and even years or any combination of the three. By comparing the output of all 23 tests we can get an idea of how consistently a strategy makes profit. The gain figure in the optimisation results should be ignored as the number of trades for each test will be very different. The average gain per trade, win rate and equity curve for each test can tell us a lot about the strategies robustness though. You just add the code to the start of your strategy code and then insert the extra condition ‘and tradeon’ in the conditions whenever you send buy or sellshort orders to the market. Then just optimise with a variable ‘choice’ with values 0 to 22 step 1. Then just click on each result in the optimisation report to see the equity curve for each test. Here is the code to add to the start of your strategy in case it is of use to anyone: //Day Month and Year Robustness Tester //By Vonasi //20190901 //0 = all days //1 = odd days only //2 = even days only //3 = odd months only //4 = even months only //5 = even years only //6 = odd years only //7 = odd days in odd months only //8 = even days in even months only //9 = even days in odd months only //10 = odd days in even months only //11 = even months in even years only //12 = odd months in odd years only //13 = even months in odd years only //14 = odd months in even years only //15 = odd days in odd months in odd years only //16 = even days in even months in even years only //17 = even days in odd months in odd years only //18 = odd days in even months in even years only //19 = odd days in even months in odd years only //20 = even days in odd months in even years only //21 = even days in even months in odd years only //22 = odd days in odd months in even years only oddday = opendayofweek mod 2 = 1 evenday = opendayofweek mod 2 = 0 oddmonth = openmonth mod 2 = 1 evenmonth = openmonth mod 2 = 0 oddyear = openyear mod 2 = 1 evenyear = openyear mod 2 = 0 tradeon = 0 if choice = 0 then tradeon = 1 endif //odd days only if choice = 1 and oddday then tradeon = 1 endif //even days only if choice = 2 and evenday then tradeon = 1 endif //odd months only if choice = 3 and oddmonth then tradeon = 1 endif //even months only if choice = 4 and evenmonth then tradeon = 1 endif //even years only if choice = 5 and evenyear then tradeon = 1 endif //odd years only if choice = 6 and oddyear then tradeon = 1 endif //odd days in odd months only if choice = 7 and oddday and oddmonth then tradeon = 1 endif //even days in even months only if choice = 8 and evenday and evenmonth then tradeon = 1 endif //even days in odd months only if choice = 9 and evenday and oddmonth then tradeon = 1 endif //odd days in even months only if choice = 10 and oddday and evenmonth then tradeon = 1 endif //even months in even years only if choice = 11 and evenmonth and evenyear then tradeon = 1 endif //odd months in odd years only if choice = 12 and oddmonth and oddyear then tradeon = 1 endif //even months in odd years only if choice = 13 and evenmonth and oddyear then tradeon = 1 endif //odd months in even years only if choice = 14 and oddmonth and evenyear then tradeon = 1 endif //odd days in odd months in odd years only if choice = 15 and oddday and oddmonth and oddyear then tradeon = 1 endif //even days in even months in even years only if choice = 16 and evenday and evenmonth and evenyear then tradeon = 1 endif //even days in odd months in odd years only if choice = 17 and evenday and oddmonth and oddyear then tradeon = 1 endif //odd days in even months in even years only if choice = 18 and oddday and evenmonth and evenyear then tradeon = 1 endif //odd days in even months in odd years only if choice = 19 and oddday and evenmonth and oddyear then tradeon = 1 endif //even days in odd months in even years only if choice = 20 and evenday and oddmonth and evenyear then tradeon = 1 endif //even days in even months in odd years only if choice = 21 and evenday and evenmonth and oddyear then tradeon = 1 endif //odd days in odd months in even years only if choice = 22 and oddday and oddmonth and evenyear then tradeon = 1 endif Then alter your strategy code to use the TRADEON variable so that trades are only opened if the day, month and year meet the test criteria. if (your long entry conditions) and tradeon then buy 1 contract at market endif if (your short entry conditions) and tradeon then sellshort 1 contract at market endif

This topic has 161 replies, 20 voices, and was last updated 3 years ago by Roger.

Tagged: casino, robustness, walk forward

Viewing 15 posts - 61 through 75 (of 162 total)

← 1 2 3 4 5 6 7 8 9 10 11 →

09/29/2019 at 7:42 PM #108805

GraHal

Vonasi

Is the the logic behind using Qty / Random combo to give more than 100 results so that we input the worst 100 results into the Robustness Tester and so we have a worst case scenario??
What do you think of the result attached for the code attached discussed on the link

https://www.prorealcode.com/topic/backtest-results-change-when-i-stop-using-variables/page/2/

GraHal

Participant

Topics: 204

Replies: 8401

Been thanked: 2042 times

09/29/2019 at 9:07 PM #108813

Vonasi

No – we should never have more than 100 tests as we need to see the bad ones – the bad ones tell us more about a strategy than the good ones! The optimiser results only ever give us the best 100 results.
I had a look at that strategy and started checking it out but then dinner, a few drinks and the girlfriend wanting to watch some American drama on the laptop sort of got in the way!

1 user thanked author for this post.

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1810 times

09/30/2019 at 9:00 AM #108863

Vonasi

I have created yet another robustness tester. I call this one the ‘Robustness Tester – WF Style’ because it splits the history being tested into lots of smaller sections and tests just on those sections.

To make the tester work it needs to know the value of barindex for the last bar on the chart. You can simply get this with the indicator RETURN BARINDEX and then enter the last value into the tester as BARSONCHART. You can also set a start date. Set this to zero if you want to test on all of history.

As with all the other robustness testers add the tester to the start of your strategy (under any DEFPARAMs) and ensure that all variables in the strategy are fixed values. Then add TRADEON to your entry conditions (being aware of how to exactly do this as previously discussed in this thread). Then you optimise the strategy with two variables DIVI and DIVI2.

DIVI one is how many bigger pieces the history being tested is divided into and DIVI2 is how many smaller pieces these bigger pieces are then divided in to. So with DIVI set to optimise from 1 to 0 step 1 and DIVI2 set to optimise from 1 to 4 step 1 we will have 40 different sections of data to test on. Be careful of setting the maximum values for DIVI and DIVI2 too high as otherwise the data is split into very small data samples that will not tell us very much at all when tested.

The results can then be dragged into Excel and analysed exactly the same as for the other testers.

//Robustness Tester - WF Style
//By Vonasi
//20190930

barsonchart= 2286// total number of bars on chart
startdate = 19950101// earliest start date of test period - set to zero to test all of history


if opendate = startdate then
testrange = barsonchart - barindex
testsize = (testrange / divi)
startindex = barindex - (testrange / divi)
endif

startbar = startindex + (testsize * divi)
endbar = startbar + ((barsonchart - startbar) / divi2)

tradeon = opendate > startdate and barindex >= startbar and barindex <= endbar

//graph barindex

//Robustness Tester - WF Style

//By Vonasi

//20190930

barsonchart= 2286// total number of bars on chart

startdate = 19950101// earliest start date of test period - set to zero to test all of history

if opendate = startdate then

testrange = barsonchart - barindex

testsize = (testrange / divi)

startindex = barindex - (testrange / divi)

endif

startbar = startindex + (testsize * divi)

endbar = startbar + ((barsonchart - startbar) / divi2)

tradeon = opendate > startdate and barindex >= startbar and barindex <= endbar

//graph barindex

2 users thanked author for this post.

O-jay8,

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1810 times

10/26/2019 at 8:39 PM #111287

fifi743

Hello ,
is it possible to make a robustness test of this algorithm.
I remain at your disposal
cordially

fifi743

Participant

Topics: 9

Replies: 851

Been thanked: 401 times

10/26/2019 at 9:30 PM #111298

Vonasi

Yes it is but that strategy has more BUY and SELLSHORT orders than I think I have ever seen in one strategy and it looks like it has the possibility to swap directions too and so adding the TRADEON condition to all the right places and still allow direction changes is going to be a bit of a mammoth task (possibly better done when I haven’t had a couple of glasses of wine!)

Changing trade direction always dilutes any robustness test a little – the more you change direction the less the robustness test can be trusted. Also because it is a 5 minute strategy the amount of data available for robustness testing is very limited compared to all of history so I am not sure exactly how conclusive the test will be but it is worth a look.

I’ll try to find time!

1 user thanked author for this post.

fifi743

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1810 times

10/26/2019 at 9:47 PM #111299

fifi743

thank you vonasi,
strongly version 11 prt

fifi743

Participant

Topics: 9

Replies: 851

Been thanked: 401 times

10/28/2019 at 9:43 PM #111465

Vonasi

I finally managed to find some time to try to add the ‘Robustness Tester – Random’ to fifi743 ‘s Vectorial DAX M5 code. I think that I added it correctly and it turned out not to be as difficult as I first thought despite the masses of BUY and SELLSHORT instructions in the code! In reality not many of them were buys or sellshort when not reversing direction.

The test on this strategy is very slow!!! – so results took quite a lot of time to get.

I tested with settings of 10/5 and 20/5 for QTY and RANDOM. So that is 45 random samples and 95 random samples. See the attached images for the results. The equity curves are for the lowest resulting average gain per trade in each test.

At first glance the results look pretty good. In the 20/5 test the average gain score is not very high but all tests in both tests resulted in positive equity curves and equity curves that closely resembled the original equity curve for the back test on all candles. The combined score in the 20/5 test was still pretty good thanks to the win rate% score being very good.

It would seem that the strategy on the data sample tested is pretty robust especially considering that the sample is very up and down and not just a bull market or a bear market. My only reservation being that the data sample of 100k 5 minute candles is a very small data sample considering the amount of history that precedes it that we have not tested on! I have attached the ITF file for the strategy modified to include the robustness tester random code. If anyone has 200k bars and wants to test it then it would be interesting to see the results.

3 users thanked author for this post.

Tony87,

MrCool,

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1810 times

10/28/2019 at 10:08 PM #111475

fifi743

hello vonasi
a big thank you for your work.
cordially

1 user thanked author for this post.

Tony87

fifi743

Participant

Topics: 9

Replies: 851

Been thanked: 401 times

10/28/2019 at 10:43 PM #111478

josef1604

GraHal wrote:

I found myself copying and pasting the VRT snippet from one System to another and it got on my nerves, so …

Here’s the VRT snippet as an .itf … if you make it available in your System backtesting list then it’s real easy to copy into the various strategy codes.

All thanks goes to Vonasi of course!

Hi GraHal, could you give us this bot to try it on demo? please? Thank you

josef1604

Participant

Topics: 1

Replies: 50

Been thanked: 2 times

10/28/2019 at 11:48 PM #111479

GraHal

josef1604 wrote:

could you give us this bot

What bot are you referring to?

I included the VRT.itf in the post that you quoted.

1 user thanked author for this post.

josef1604

GraHal

Participant

Topics: 204

Replies: 8401

Been thanked: 2042 times

10/29/2019 at 11:11 AM #111517

josef1604

Hi, GraHal! If the vrt.itf is there, but I don’t know how to remove the robustness test and have the bot to test it in demo. Could you hang it please? many thanks

josef1604

Participant

Topics: 1

Replies: 50

Been thanked: 2 times

10/29/2019 at 11:19 AM #111520

GraHal

Are you saying that you do not know how to use the vrt.itf in a bot in order to test your own bot’s robustness?

GraHal

Participant

Topics: 204

Replies: 8401

Been thanked: 2042 times

10/29/2019 at 11:48 AM #111523

Vonasi

To stop the robustness tester from working just change the line in the code that says

tradeon = 0

1	tradeon = 0

tradeon = 1

1	tradeon = 1

1 user thanked author for this post.

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1810 times

10/29/2019 at 1:52 PM #111540

josef1604

GraHal wrote:

Are you saying that you do not know how to use the vrt.itf in a bot in order to test your own bot’s robustness?

No, what I want is to put the bot in demo mode and it gives me an error

josef1604

Participant

Topics: 1

Replies: 50

Been thanked: 2 times

10/29/2019 at 2:39 PM #111550

GraHal

@josef1604 … did you follow Vonasi advice in the post above your last one?

If you want you could post your bot (.itf file be best) on here and I would try and run it for you on my Demo Platform?

GraHal

Participant

Topics: 204

Replies: 8401

Been thanked: 2042 times

Author

Posts

Viewing 15 posts - 61 through 75 (of 162 total)

← 1 2 3 4 5 6 7 8 9 10 11 →

Find exclusive trading pro-tools on