Best ways to compare 2 versions of 1 system?

• 02/18/2020 at 9:28 PM #119890
  jebus89

  Hi all.

  I need some help. I have 2 versions of the same system, the difference is:

  System A (Original system) – Buys 1 contract no matter what, if X, Y , Z is true.
  System B – Buys 2 contracts if C1 is true, 1 contract if C1 is false, if X, Y , Z is true.

  So its basicly the same systems but with 1 filter added where i will buy more contracts if the filter is true, for example c1 can be close > MA200.

  The difference in system A and system B is not huge. System B gives a little bit better drawdown periods, but also just a tiny bit less total profit.

  On one hand i would like to switch to system B because it looks just slightly better, a little bit reduced drawdown. But the change is not big and huge.

  On the other hand though we have all read and heard about “one more tweak to my system will fix all my trouble” and years later its obvious that you should have let your system be as it is..

   

  I guess my questions to you guys are the following:

  If you change your system/tweak it/add to it and get a better version, how much better does it need to be for you to replace your current version? If you had to put it into a percentage: 10% “better”? 50% “better”?

  Do you try to change/tweak your systems often? I find myself often whipping up a currently running system and keep trying different things, either to keep testing/breaking the entry/exit or to try to improve it. I very rarely find anything that would make me want to actually change what version i run, but sometimes i find something that instantly appeals to me. Everything just looks a little bit better with an added C1 or maybe a tweaked C2… I dont want to fuck a working system, at the same time, sometimes i see that if i had only added C1 last month my system would have performed alot better at the same time as the rest of the backtest looks just as good or better all over the 10+ years.

  When you compare System A and System B what are you mainly looking for? improved drawdown? improved total profit? both?

  Any special statistics your looking at?

  

  jebus89

  Participant

  Topics: 118

  Replies: 586

  Been thanked: 238 times
  02/18/2020 at 9:55 PM #119891
  GraHal

  jebus89 wrote:

  and years later its obvious that you should have let your system be as it is..

  Or even weeks / months later? 🙂

  I often have 4 to 6 versions of the same System on Forward Test and it is not often that the forked / improved versions giver continued better results than the original.  But the few that do (give better results) make the effort worthwhile.

  I look for improvements on all fronts … Gain to Loss Ratio, % of winning trades, drawdown, etc.  I would settle for much improved profit at the expense of the other stats … providing trades enter and exit at logical price action points.

  What spooks me is profits due to trades sitting there for ever and ever – with big drawdown – but coming up good in the end … good luck which likely will turn bad very soon!?

  

  GraHal

  Participant

  Topics: 204

  Replies: 8397

  Been thanked: 2039 times
  02/18/2020 at 10:15 PM #119893
  jebus89

  GraHal wrote:

  What spooks me is profits due to trades sitting there for ever and ever – with big drawdown – but coming up good in the end … good luck which likely will turn bad very soon!?

   

  I dont trust the type of trades that your talking about there. The ones with the big drawdowns and coming up in the end. In my opinion thats probably luck. The way i trade is that if im wrong about my setup, i want to get out as soon as possible, in other words i want a stop loss/exit thats not too far away from my entry. The way i see it is that my setup, or my entry, is either valid or its not. If that makes sense.

  My drawdowns tend to be multiple failing trades, rather than 1 trade that “never” closes until the market bounces up again.

  If im Long only and my drawdowns occur when the markets go down (which they have done so far), i want to make sure that my multiple failing trades would give me a smaller drawdown rather then holding 1 loosing trade longer and hoping for a bounce up. One reason is that you never know when the market will turn up again, it might just drop for the next weeks/months (a long time, depending on ur timeframe) the other reason is that if i have multiple long only systems which are all in drawdown periods at the same time, i would rather want 10 systems that have failed 3-5 times in a row but are all ready for the next signal to go long, then being stuck with 10 negative trades which would eat up my margin and i think at least for that would feel more emotional stressful than just taking the losses.

  

  jebus89

  Participant

  Topics: 118

  Replies: 586

  Been thanked: 238 times
  02/18/2020 at 10:40 PM #119895
  Vonasi

  I always look at what can be taken away to make a strategy less curve fitted rather than what I can add which will only ever make it more curve fitted.

  I totally get what you say about not wanting to hold on too long to negative trades. Holding is fine if you start with a big bank but then if you have a big bank then just buy stocks and hold on to them through thick and thin – don’t spread bet! I would rather look for a lot of strategies that are very simple and have a slight edge and just buy today and sell tomorrow or buy today and sell in a weeks time. Trade a lot of these uncorrelated and you will make a profit – but the important thing is to keep it very simple – less is more when it comes to conditions in a strategy in my humble opinion.

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1810 times
  02/19/2020 at 8:16 AM #119918
  jebus89

  Vonasi wrote:

  Trade a lot of these uncorrelated and you will make a profit – but the important thing is to keep it very simple – less is more when it comes to conditions in a strategy in my humble opinion.

  For sure, this is my belief as well. Still tho, im not really sure how to address my issue here..

  How about you Vonasi? If you got System A and System B and theres 1 filter in place, that dosnt stop u from taking trades, but rather wants you to buy more. Meaning you dont exclude trades from your backtest, meaning u dont really curvefit the trades, the trades still happen exactly the same place and time they would have happened in your original system. But you do of course kind of curvefit the backtest equity curve in that it will look different and for the better.. Would you agree with my statement here? That my example above is “less curvefitting” than adding a C1 filter that exludes alot of trades thus making it kinda a “whole new system” maybe?

   

  Edit: If C1 is a “bad” filter in the future this could mean disaster for your whole system of course. If 90% of your winning trades are happening when C1 is false = 1 contract, which would mean that 90% of you loosing trades are happening when C1 is true = 2 contracts and BAM your whole backtest MIGHT look like shit..

  

  jebus89

  Participant

  Topics: 118

  Replies: 586

  Been thanked: 238 times
  02/19/2020 at 11:36 AM #119935
  Vonasi

  I think that you always have to justify every condition by analysing it before you allow it to stay in a strategy.

  Take the humble average as a long or short filter for example. I would want to test the strategy with every average period of a large range and then plot the results in Excel charts. I would want to see a pretty large area in the chart where the results are not too dissimilar in returns and win rates. This means we now have a large range of average periods that have worked pretty well historically even if not all the equity curves would be to our liking. I would then choose a period for our average that is smack bang in the middle of this working range of periods or the one with our nicest results if it is near the middle of the range which should minimise our new filter being stood on a cliff edge.

  You could even consider running three or four identical strategies but just with different average periods from across this range of working periods and then average out the equity curve.

  If we add a condition that gives a more binary result – i.e. it works with a certain setting but does not work with any other setting then I really would want to delve deep and see if I can understand why it works. This is where robustness testing is useful – especially if done on OOS data.

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1810 times
  02/21/2020 at 11:55 AM #120154
  deletedaccount100622

  For me the new system doesn’t have to be too much better as over the long term the gains mount up especially if using money management and I often learn things about stops or adding or removing and indicator that can be applied across multiple systems

  I agree with @Vonasi about the importance of diversification across products and time frames

  I would also look at the work involved, so for example I have a system Alpha 8  that uses RSI divergence but with varying Stops and TP levels is live on:

  NASDAQ @ 30 mins

  EUR/USD @ 10, 15, 30 mins and probably a bunch more markets that I haven’t tested yet so retesting each of the live systems and redoing the research for the other contenders would require a big improvement where as a system being used in one live strategy I would me more open for an update

  

  deletedaccount100622

  Participant

  Topics: 13

  Replies: 116

  Been thanked: 60 times
• Author
  Posts