How to avoid over optimization in algos ?

• 05/14/2020 at 11:45 AM #131643
  Vonasi

  When doing a WF with optimization each data sample has its own IS and OOS bits of data so in reality it is just 5 separate overlapping IS and OOS WF tests and you are hoping that they all give similar results and all work best with exactly the same or similar settings. That then tells you how robust your strategy is.  The ideal development is to develop a strategy on a section of IS data and once you are happy it works as you want it to then test it on the rest of the data available. Then if it works the same or similar on that data then break it down into 5 overlapping IS and OOS sections of data and then WF optimize on that. If all the results are similar and the setting similar then that is good. If happy then set it live and forward test on demo with the settings from the very latest data sample.

  I think a common mistake is to WF optimize with massive amounts of variables and with massive ranges for those variables. You should have an idea of a small range that you think your strategy should work ok in and not fall apart just because some extreme value is not available to it for one section of data. Not only should this give more confidence that your strategy is pretty robust within a small range of values but it also considerably speeds up WF testing.

  2 users thanked author for this post.

  MWALKER, nonetheless

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1811 times
  05/14/2020 at 11:48 AM #131644
  Vonasi

  You also have to consider the bigger picture. If a strategy is say long only and has a filter to sit out major downturns such as the 2008 crash then if that crash falls into any data sample then the results for that test will look very poor compared to the others and could make you think the strategy has failed the test when in fact it hasn’t.

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1811 times
  05/14/2020 at 12:27 PM #131651
  nonetheless

  Vonasi wrote:

  common mistake is to WF optimize with massive amounts of variables and with massive ranges for those variables.

  Thanks very much for clarifying. This has been my instinctive approach (also speeds up the WF process), but part of me felt that maybe I was cheating the results by only giving it a small range to choose from. Good to know I was doing the right thing for once.

  

  nonetheless

  Participant

  Topics: 83

  Replies: 1237

  Been thanked: 978 times
  05/14/2020 at 12:48 PM #131652
  zilliq

  Vonasi wrote:

  When doing a WF with optimization each data sample has its own IS and OOS bits of data so in reality it is just 5 separate overlapping IS and OOS WF tests and you are hoping that they all give similar results and all work best with exactly the same or similar settings. That then tells you how robust your strategy is. The ideal development is to develop a strategy on a section of IS data and once you are happy it works as you want it to then test it on the rest of the data available. Then if it works the same or similar on that data then break it down into 5 overlapping IS and OOS sections of data and then WF optimize on that. If all the results are similar and the setting similar then that is good. If happy then set it live and forward test on demo with the settings from the very latest data sample.

  I think a common mistake is to WF optimize with massive amounts of variables and with massive ranges for those variables. You should have an idea of a small range that you think your strategy should work ok in and not fall apart just because some extreme value is not available to it for one section of data. Not only should this give more confidence that your strategy is pretty robust within a small range of values but it also considerably speeds up WF testing.

  Agree to you too. It seems that less variable you are, less suroptimization you have because you have less variability

  But always difficult to create an algo with only one or 2 variables and machine learning is another type of variable you add

  But not completely agree with you on one point. I don’t think even with one or two variable that they will work for x weeks or y months and I think we need to reajust them periodically

  Have a nice day

  

  zilliq

  Participant

  Topics: 58

  Replies: 683

  Been thanked: 152 times
  05/14/2020 at 12:51 PM #131653
  zilliq

  Probably we can use the results of IS test to find a small range of each variable (Kind of what you do with your robustess test I think)

  Perhaps can we construct the range for each variable with range who include 95 % of the “best results” (For each variable) or median +/- 5 %  or  90 centile +/- 5 %?

  I will test it too tomorrow

   

  

  zilliq

  Participant

  Topics: 58

  Replies: 683

  Been thanked: 152 times
  05/14/2020 at 1:25 PM #131660
  Vonasi

  We have to consider the standing on a cliff edge scenario when choosing our WF value range. When creating a strategy I always like to drag the optimized results window into Excel so that I can plot some graphs to see if there is a range of values that work similarly and to check that my best performing value is not on the edge of this range just before a cliff edge or that it is not just a rogue value out in the wilderness on its own. So this can help decide on the range of values to use in a WF optimization. In v11 the 2D and 3D charts mean that we can do this within the platform now.

  If you have loads of variables then the above analysis becomes quite challenging because you have too many combinations to consider which is why I like to keep strategies as simple as possible with only one or two variables to optimize. Far less chance of a curve fit and easier to analyse that way.

  1 user thanked author for this post.

  nonetheless

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1811 times
  05/14/2020 at 5:36 PM #131691
  Mauro T. “Algorithm System”

  Usually a less variable system has more you have the possibility to last over time … well as Vonasi I do not put more than two or three variables … if instead you have the idea of ​​re-optimizing very often there is the risk of see exceptional performances … but only on statistics …. But everyone has their own ideas and way of working, so it is difficult to give advice on how to avoid excessive optimization ….

  

  Mauro T. “Algorithm System”

  Participant

  Topics: 18

  Replies: 302

  Been thanked: 75 times
  05/14/2020 at 6:38 PM #131697
  Vonasi

  zilliq wrote:

  Probably we can use the results of IS test to find a small range of each variable (Kind of what you do with your robustess test I think)

  Are you referring to my robustness tester – the one a lot of people now call VRT?

  If so then no. The VRT is purely done to produce strategy results that you might not see by just starting at one bar and running your strategy through the data just once. While a trade is open you will be missing others so by ‘randomizing’ when we allow trades to open or not we get to ensure that no trades are missed. It is not infallible though as some strategies just might not have enough trades for a meaningful test and others will have lots of very quick trades which can also make the results less conclusive. It is however just one more tool in our armoury to stop us going live with over fitted strategies – but like all tools you have to understand how it works and know how to use it to get the best out of it. It is the same with WF testing – it is not infallible and as a tool it works better on some strategies than others.

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1811 times
• Author
  Posts