Have you ever seen that the Worst Parameters are the first results on Backtest ?

• 11/24/2020 at 3:35 PM #151434
  zilliq

  Hi Guys,

  As I said previously I work a lot with excel now to find some interesting results, optimizing and so on..

  And this morning I find something very strange

  I took an algo, take a backtest on 80 000 unit, copy the results on Excel, and do the same backtest on 100 000 units to search what combinations works better

  And I calculate a ratio Gain after 2 weeks / Gain before 2 weeks I put on excel and apply a color red is bad, blue is good

  And it’s very strange, but as you see on the picture, the best evolutions were with the last results on backtest ??? There is 10- 20 % difference

  I do this “test” on 2 algo, verify there is no error, and obtain the same conclusion 2 times

  Have you ever see that ? And if so, it will mean that when we take the first result on backtets to put in real/OOS this is in fact the worst combination …

  Bye

  Zilliq

  

  zilliq

  Participant

  Topics: 58

  Replies: 683

  Been thanked: 152 times
  11/24/2020 at 3:52 PM #151437
  GraHal

  yeah I sure have noticed … it’s always the red losing results appear first then the white text 0 results then the green winning results start to populate.

  

  GraHal

  Participant

  Topics: 222

  Replies: 9290

  Been thanked: 2460 times
  11/24/2020 at 11:48 PM #151464
  M.Nicolas 81

  Hi zillip

  Hello GraHal

  Forgive me to intrude.

  How do you do it to be able to pass the data to Excel?

  I have tried but they always copy me as an image.

  Thanks a lot.

  

  M.Nicolas 81

  Participant

  Topics: 13

  Replies: 33

  Been thanked: 1 time
  11/24/2020 at 11:59 PM #151466
  Vonasi

  M.Nicolas 81 wrote:

  How do you do it to be able to pass the data to Excel?

  Drag and drop.

  2 users thanked author for this post.

  zilliq, M.Nicolas 81

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1812 times
  11/25/2020 at 1:34 AM #151476
  M.Nicolas 81

  Thanks Vonasi.

  I had tried by downloading the document and converting it.

  I just did it and everything is moved correctly except for Entry Date and Exit Date which appears like this:

  ######

  You authorize me to include your answer in the Topic: Tips for beginners?

   

  

  M.Nicolas 81

  Participant

  Topics: 13

  Replies: 33

  Been thanked: 1 time
  11/25/2020 at 8:09 AM #151480
  Vonasi

  Widen the column by dragging the right edge over at the top of the column and the date will be shown correctly.

  No problem to include anything I write in your tips.

  1 user thanked author for this post.

  M.Nicolas 81

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1812 times
  11/25/2020 at 2:01 PM #151557
  Nicolas

  Export data results into Excel with v11: https://www.prorealcode.com/topic/drag-and-drop-detailed-reports-v11-1/

  1 user thanked author for this post.

  M.Nicolas 81

  

  Nicolas

  Keymaster

  Topics: 83

  Replies: 20898

  Been thanked: 4866 times
  11/25/2020 at 8:13 PM #151631
  zilliq

  Since yesterday, I did many many trials and it seems to confirm what I see on different algos as you see on this other trial

  The first results on a backtest are probably the one with the worst evolutions (20 % vs 116 % after 2 weeks on OOS for the last)

  Same thing for the absolute variation. In this trial, the first results before the OOS (1 to 5 first results) win 5500 euros in 2 weeks versus 11 800 euros for the number 96 and 97 before OOS test (See picture)

  The problem is that if I can determine/predict the first best results after 2 weeks on OOS, I don’t now how to determine the one with the best evolutions (because they are the worst results before the OOS period I repeat)

  Very strange but it’s like that

  

  zilliq

  Participant

  Topics: 58

  Replies: 683

  Been thanked: 152 times
  11/25/2020 at 8:34 PM #151639
  GraHal

  Zilliq … I am confused by your reference to 2 weeks??

  No need to answer, but I don’t like to see posts with no response and maybe others are confused also?

  Ah I think you may be referring to Walk Forward??

  Topic title and all your other posts in this Topic have referred to normal backtests (not Walk Forward)?

  

  GraHal

  Participant

  Topics: 222

  Replies: 9290

  Been thanked: 2460 times
  11/25/2020 at 9:37 PM #151648
  zilliq

  Hi @GraHal

  On PRT complete we have 2 months of historic data with 1 minute

  I want to do a backtest on 25 % of this time, equal 2 weeks

  (It could be one week (may be better) whatever)

  It’s to know if I need to re-backtest every one or 2 weeks

  And after, I do what you know, a backtest on 6 weeks, and OOS test on 2 weeks to see how it evolve during this 2 weeks 6->8 weeks

  It’s like a WF one pass 25 % 75 % in fact

  Bye

  1 user thanked author for this post.

  GraHal

  

  zilliq

  Participant

  Topics: 58

  Replies: 683

  Been thanked: 152 times
  11/25/2020 at 10:44 PM #151660
  Vonasi

  GraHal wrote:

  and maybe others are confused also?

  Yep. Total gobbeledygoop to me – I have absolutely no idea what ‘The first results on a backtest are probably the one with the worst evolutions’ is meant to mean.

  Sorry Zilliq but you lost me at the doors of the sausage factory for three legged donkeys that want to be race horses.

  1 user thanked author for this post.

  eckaw

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1812 times
  11/25/2020 at 11:39 PM #151673
  zilliq

  LOL sorry @vonasi and @GraHal, it’s probably me 🙂

  Well, different steps

  First step, I do a backtest on 6 weeks

  You obtain 100 results from better one to worst, 1 to 100

  Second step, I do a backtest on 8 weeks

  I have 100 results too

  And now I look for the evolution of the different combinations of the first step on the results of the second step

  Fo example, if the 1st result of the first step was a=12 b=23 c=25 I look for on the results of the second step this combination and see how it progress (Win or loss)

  For example if with this combination I obtain an IS backtest gain of 10 000 euros and with this combinations on the second step I obtain a gain of 15 000 euos I can say that on an hypothetic OOS of this combination, it would win 5000 euros

  And I do that for all 100 results of the 1st step = how progress the different combinations of parameters. Like that  I  “simulate” quite 100 OOS for all algos to test the robustess. +/- a kind of Monte carlo simulation (I know it’s not that)

  Hope to be clear. If not I will post pictures tomorrow

  Bye

  

  zilliq

  Participant

  Topics: 58

  Replies: 683

  Been thanked: 152 times
  11/25/2020 at 11:47 PM #151674
  Vonasi

  Still confused

  Do a backtest on 6 weeks of data and see which variable values work best. Then fix those values in the strategy and test them on 2 weeks of data that does not include the original 6 weeks of data in any way. Compare your in sample to your out of sample and that is it- but always remembering that 8 weeks of data is bugger all data!

  If your OOS holds good then look at values either side of your chosen variable values to see if you just got lucky or if your chosen values are on a cliff edge.

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1812 times
  11/26/2020 at 12:05 AM #151676
  zilliq

  No it’s not completely that

  It’s “just” a comparison of results on 2 backtests,  6 weeks and 8 weeks (It could be 7 and 8 weeks)

  In the 8 weeks results it included some combinations of the 6 weeks but not all

  And I compare the identical combinations. Like that it’s like I do 80-90 OOS tests on 2 weeks with this algo

  We can probably say it’s like a one pass work forward 75/25 but I test 80-90 (identical) combinations. A real WF one pass will only give you only the “best” combination (But as I previously said not the best one in fact), me I analyze and test 80-90 on 100

  The purpose is to determine why some combinations works and some not by analyzing these combinations who work

  Hope to be clear

  Have a good night

  1 user thanked author for this post.

  eckaw

  

  zilliq

  Participant

  Topics: 58

  Replies: 683

  Been thanked: 152 times
  11/26/2020 at 12:20 AM #151679
  eckaw

  Vonasi wrote:

  GraHal wrote:

  and maybe others are confused also?

  Yep. Total gobbeledygoop to me – I have absolutely no idea what ‘The first results on a backtest are probably the one with the worst evolutions’ is meant to mean.

  Sorry Zilliq but you lost me at the doors of the sausage factory for three legged donkeys that want to be race horses.

  I think this forum needs a ‘haha’ button, not just quote / thanks /reply buttons

  

  eckaw

  Participant

  Topics: 13

  Replies: 118

  Been thanked: 61 times
• Author
  Posts