Strategy Robustness Tester

Vonasi · Sun, 01 Sep 2019 12:34:14 +0000

Following on from a discussion elsewhere regarding walk forward testing I had the idea to create a bit of code that you can add to any strategy and see how it would have performed if you had traded on only odd days or only on even days, or odd or even months or odd and even years or any combination of the three. By comparing the output of all 23 tests we can get an idea of how consistently a strategy makes profit. The gain figure in the optimisation results should be ignored as the number of trades for each test will be very different. The average gain per trade, win rate and equity curve for each test can tell us a lot about the strategies robustness though. You just add the code to the start of your strategy code and then insert the extra condition ‘and tradeon’ in the conditions whenever you send buy or sellshort orders to the market. Then just optimise with a variable ‘choice’ with values 0 to 22 step 1. Then just click on each result in the optimisation report to see the equity curve for each test. Here is the code to add to the start of your strategy in case it is of use to anyone: //Day Month and Year Robustness Tester //By Vonasi //20190901 //0 = all days //1 = odd days only //2 = even days only //3 = odd months only //4 = even months only //5 = even years only //6 = odd years only //7 = odd days in odd months only //8 = even days in even months only //9 = even days in odd months only //10 = odd days in even months only //11 = even months in even years only //12 = odd months in odd years only //13 = even months in odd years only //14 = odd months in even years only //15 = odd days in odd months in odd years only //16 = even days in even months in even years only //17 = even days in odd months in odd years only //18 = odd days in even months in even years only //19 = odd days in even months in odd years only //20 = even days in odd months in even years only //21 = even days in even months in odd years only //22 = odd days in odd months in even years only oddday = opendayofweek mod 2 = 1 evenday = opendayofweek mod 2 = 0 oddmonth = openmonth mod 2 = 1 evenmonth = openmonth mod 2 = 0 oddyear = openyear mod 2 = 1 evenyear = openyear mod 2 = 0 tradeon = 0 if choice = 0 then tradeon = 1 endif //odd days only if choice = 1 and oddday then tradeon = 1 endif //even days only if choice = 2 and evenday then tradeon = 1 endif //odd months only if choice = 3 and oddmonth then tradeon = 1 endif //even months only if choice = 4 and evenmonth then tradeon = 1 endif //even years only if choice = 5 and evenyear then tradeon = 1 endif //odd years only if choice = 6 and oddyear then tradeon = 1 endif //odd days in odd months only if choice = 7 and oddday and oddmonth then tradeon = 1 endif //even days in even months only if choice = 8 and evenday and evenmonth then tradeon = 1 endif //even days in odd months only if choice = 9 and evenday and oddmonth then tradeon = 1 endif //odd days in even months only if choice = 10 and oddday and evenmonth then tradeon = 1 endif //even months in even years only if choice = 11 and evenmonth and evenyear then tradeon = 1 endif //odd months in odd years only if choice = 12 and oddmonth and oddyear then tradeon = 1 endif //even months in odd years only if choice = 13 and evenmonth and oddyear then tradeon = 1 endif //odd months in even years only if choice = 14 and oddmonth and evenyear then tradeon = 1 endif //odd days in odd months in odd years only if choice = 15 and oddday and oddmonth and oddyear then tradeon = 1 endif //even days in even months in even years only if choice = 16 and evenday and evenmonth and evenyear then tradeon = 1 endif //even days in odd months in odd years only if choice = 17 and evenday and oddmonth and oddyear then tradeon = 1 endif //odd days in even months in even years only if choice = 18 and oddday and evenmonth and evenyear then tradeon = 1 endif //odd days in even months in odd years only if choice = 19 and oddday and evenmonth and oddyear then tradeon = 1 endif //even days in odd months in even years only if choice = 20 and evenday and oddmonth and evenyear then tradeon = 1 endif //even days in even months in odd years only if choice = 21 and evenday and evenmonth and oddyear then tradeon = 1 endif //odd days in odd months in even years only if choice = 22 and oddday and oddmonth and evenyear then tradeon = 1 endif Then alter your strategy code to use the TRADEON variable so that trades are only opened if the day, month and year meet the test criteria. if (your long entry conditions) and tradeon then buy 1 contract at market endif if (your short entry conditions) and tradeon then sellshort 1 contract at market endif

This topic has 161 replies, 20 voices, and was last updated 3 years ago by Roger.

Tagged: casino, robustness, walk forward

Viewing 15 posts - 136 through 150 (of 162 total)

← 1 2 3 4 5 6 7 8 9 10 11 →

07/12/2020 at 3:45 PM #139086

Vonasi

No robustness test is a bad test compared to not robustness testing at all. However we should always keep in mind to ignore the extremes especially those test results that have very few trades in them. Better to delete them from the spread sheet results and not include them in the the analysis charts at all. Set a minimum trade quantity level and delete all test results that don’t meet it.

It is always a nice idea to include GRAPH TRADEON at the end of your strategy so you can see the sorts of trading sequences that your QTY and RANDOM settings are producing.

The need to take an average of lots of different robustness tests and the fact that they still might not see every possible trade along with this need to filter out the extreme test results is why I much prefer my ‘strategy every bar’ testing. Unfortunately this SEB method does not work easily on faster time frame strategies due to the massive number of strategies you are required to run to ensure that you miss as few trades as possible.

1 user thanked author for this post.

Dow Jones

Vonasi
Moderator

Topics: 192
Replies: 5088
Been thanked: 1810 times

07/13/2020 at 8:38 AM #139111

Dow Jones

Thank you, @Vonasi. I concur on the extreme result too. I’m not proud in case having few trades that produce good result. Thanks for the advice about the GRAPH TRADEON

Dow Jones
Participant

Topics: 10
Replies: 135
Been thanked: 75 times

09/24/2020 at 8:55 PM #145278

Francesco

Dumb question: if the strategy have “not onmarket” before entry conditions, is RT useless?

Francesco
Participant

Topics: 32
Replies: 439
Been thanked: 200 times

09/24/2020 at 10:58 PM #145285

Vonasi

Francesco wrote:

Dumb question: if the strategy have “not onmarket” before entry conditions, is RT useless?

No. The TRADEON condition is the condition that decides whether a trade is allowed to open or not and it can be used with any other conditions including only opening a trade only if “not onmarket”. TRADEON must be checked only for the first market entry and not for following market entries if accumulating positions are part of the strategy.

1 user thanked author for this post.

Francesco

Vonasi
Moderator

Topics: 192
Replies: 5088
Been thanked: 1810 times

10/25/2020 at 7:08 PM #148437

ullle73

would this be an okay result according to you vonasi?

ullle73
Participant

Topics: 51
Replies: 178
Been thanked: 7 times

10/26/2020 at 12:53 PM #148502

nonetheless

@ullle73, those are v unusual results. Either you have produced the most robust system ever … or there’s an error in your testing. It’s quite rare for the av gain per trade to go above 50, and I don’t think it’s possible for either result to go above 100 (i may be wrong about that).

nonetheless
Participant

Topics: 83
Replies: 1237
Been thanked: 978 times

10/26/2020 at 5:29 PM #148548

Francesco

ullle73 wrote:

would this be an okay result according to you vonasi?

There’s clearly something wrong, check it again

Francesco
Participant

Topics: 32
Replies: 439
Been thanked: 200 times

10/27/2020 at 9:48 AM #148630

Vonasi

As others have said – those results are so good that there is a possibility that they could be too good to be true. You would need to double check how you have applied the robustness code. Not all strategies are suitable for this type of robustness testing – for example the robustness results for a strategy that opens a position and then is always on the market switching from one direction to the other will be meaningless.

Vonasi
Moderator

Topics: 192
Replies: 5088
Been thanked: 1810 times

10/27/2020 at 11:54 AM #148649

ullle73

oh i see, i just did the optimization and drag the result into excel. Could not get the code to work in the strategy, will try again 🙂 thanks

ullle73
Participant

Topics: 51
Replies: 178
Been thanked: 7 times

01/02/2021 at 3:54 PM #155977

Monobrow

Hi

Phew, been through this thread today and found it incredibly interesting and useful….

I’ve put one of my algos through this and have a result below – thanks to Vonsai, Paul (and everyone) for the algos and templates.

Can somebody please take a look and give me their feedback please. I’m not looking for a “go ahead” or “dont go ahead”, just an opinion from someone who has done this before as it will help me read results better in the future.

I used QTY 2-21, RANDOM 1-5 and used all 100 rows of data from each.

many thanks!

Monobrow
Participant

Topics: 15
Replies: 157
Been thanked: 23 times

01/02/2021 at 4:12 PM #155981

nonetheless

Sorry to bring bad news but that is a v poor score.

The evaluation is always going to be subjective but personally I like to see something above 35 in the top box and preferably in the 40-50 range. It rarely goes above 50. The % win is always a higher score and usually less problematic but 56 is not great – I’m usually looking for 80-90+ there.

Sorry to say but I think it’s back to the drawing board on this one. Hey ho … 🤔

NB i usually run it qty 2-10 and random 1-10 so results may vary, but not that much.

nonetheless
Participant

Topics: 83
Replies: 1237
Been thanked: 978 times

01/02/2021 at 4:24 PM #155983

Monobrow

nonetheless wrote:

Sorry to say but I think it’s back to the drawing board on this one. Hey ho … 🤔

Hey, no worries! and thanks for the feedback

Monobrow
Participant

Topics: 15
Replies: 157
Been thanked: 23 times

01/03/2021 at 12:18 PM #156067

nonetheless

Vonasi wrote:

to ignore the extremes especially those test results that have very few trades in them. Better to delete them from the spread sheet results and not include them in the the analysis charts at all

With this in mind, I’m thinking it might be better to run it with something like qty 2-12 / random 1-10, giving 110 iterations of which PRT shows us the top 100? This would automatically remove the outliers, which in my experience tend to be at the bottom of the Avg Gain results.

What do you think?

nonetheless
Participant

Topics: 83
Replies: 1237
Been thanked: 978 times

01/03/2021 at 12:23 PM #156069

Vonasi

That’s an option but I prefer just to see all 100 results and then manually delete any with hardly any trades from the spreadsheet rather than possibly lose 10% of results that could all also be fantastic and give me even more confidence in the strategy.

Vonasi
Moderator

Topics: 192
Replies: 5088
Been thanked: 1810 times

01/20/2021 at 10:48 AM #158640

XORANDNOT

How about doing the parameter optimization (i.e., all the curve fitting) with the aid of a position entry scrambler like the robustness tester ? That means, during parameter optimization, you block several entries randomly, ideally using different blocking patterns as the robustness tester generates ? Shouldn’t this result in more robust systems from the start ?

So, could it be possible to apply position scrambling already during parameter optimization, not only in hindsight, as the robustnesss tester does ? I haven’t thought about how to code this, but would this be a feasible way to obtain more robust parameter sets for a given system ?

XORANDNOT
Participant

Topics: 12
Replies: 178
Been thanked: 38 times
Author

Posts